CFAgentBench: We Published the First Open Benchmark for Construction-Finance AI Agents — Beiing Human

1,014 real tasks. 35 simulated systems. One iron rule: the AI never moves money. Here’s why we built it and what the first results tell us about how far AI agents are from replacing your AP clerk.

Every vendor selling AI to construction CFOs has a demo. And every demo looks brilliant: the agent reads an invoice, codes it to the right job and cost code, and queues the payment — all in under a minute. The room claps.

Then the pilot starts. The agent miscodes a $14,000 concrete bill to the wrong phase. It approves a duplicate. It initiates a payment against a subcontractor whose COI expired last week. Nobody claps, and the CFO pulls the plug.

The problem isn’t that AI agents are bad. It’s that we have no rigorous way to measure how good they are at the actual work. So we built one.

What is CFAgentBench?

CFAgentBench is a reproducible, open-source benchmark and simulated environment designed to test whether an AI agent can do the real day-to-day work of a construction finance team — across the actual software stack contractors use.

Tasks with money-movement guards

The tasks aren’t synthetic trivia. Nearly 600 seed from real CFMA Connection Cafe threads — the exact pain points construction controllers discuss among themselves. Others come from Finance at the Jobsite podcast episodes, Beiing Human customer emails and Fathom calls, NAHB builder forums, and public standards like AIA G702/G703 and WH-347 certified payroll.

Why Construction Finance Needed Its Own Benchmark

Existing AI benchmarks like WebArena, SWE-bench, and AppWorld do excellent work in their domains. But construction finance has properties that make generic benchmarks nearly useless for it.

Five Realities That Make Construction Finance Harder for AI Agents

The five structural realities of the construction software stack (Srivastava & Harper, 2026) that make generic AI benchmarks inadequate for this vertical.

Because of these realities, CFAgentBench simulates the real system of record: Vista, Sage Intacct, Foundation, CMiC, Acumatica ERPs, Procore and four other PM platforms, GCPay, LCPtracker, Levelset, Rhumbix, bank portals, Box, Outlook, and Excel. The 35 applications collapse into 9 archetypes, each implementing a uniform “app contract” that makes the whole stack self-hostable and deterministically testable.

The Iron Rule: The AI Never Moves Money

278 of the 1,014 tasks embed a payment, payroll release, e-signature, or e-filing step. In every case, the correct action is to stop and stage for human approval. An agent that initiates even the correct payment fails the task.

This isn’t a limitation we plan to engineer away later. It’s the design principle that makes everything else adoptable. Segregation of duties doesn’t disappear because the second employee got smarter. CFAgentBench is the first benchmark to grade this behavior explicitly — converting “safe by default” from an aspiration into a measured outcome.

What the First Results Tell Us

We ran three open-weight models — DeepSeek-V3.1, Qwen3–235B, and Qwen2.5–72B — through 40 oracle-validated tasks with 5 repeated runs each (600 total runs, $5.14 total cost). The results carry a clear message.

Reliability Collapse: pass¹ vs. pass⁵ Across Models The best single-shot model loses 43% of its successes when asked to repeat them 5 times 1.0 0.8 0.6 0.4 0.2 0.0 0.67 0.38 DeepSeek-V3.1 0.54 0.40 Qwen2.5–72B 0.45 0.30 Qwen3–235B pass¹ (single attempt) pass⁵ (all 5 runs succeed)

pass¹ vs. pass⁵ for three open-weight models. The gap between single-shot success and reliable repeatability is the key finding — and the metric that matters for production deployment.

Finding 1: Reliability collapse is the real story

DeepSeek-V3.1 — the best single-attempt agent — scored 0.67 on individual tasks. But when asked to get the same task right across all 5 runs (the pass⁵ metric), it dropped to 0.38. That’s a 43% loss of successes, under fixed temperature-0 decoding. The variance isn’t from sampling randomness — it’s from serving-stack nondeterminism (mixture-of-experts routing, floating-point reductions, dynamic batching) that any production deployment would face.

Why This Matters for CFOs

A construction finance process runs every week. If your AI agent codes invoices correctly 67% of the time but can only reproduce that result reliably 38% of the time, you’ve created a process where someone must now find the wrong 62%. That’s worse than no agent at all.

Finding 2: Bigger and newer ≠ better

The older, smaller Qwen2.5–72B numerically outperformed the much larger Qwen3–235B on both metrics — suggesting that general model scale doesn’t straightforwardly translate to construction-finance competence. At n=40 the confidence intervals overlap, but the direction is clear enough to challenge the assumption that the next model release automatically solves the problem.

Finding 3: Sharp domain heterogeneity

Both Qwen models scored 0.87–0.90 on Billing tasks (owner pay-app drafting) but near-zero on Cross-system and AP coding. DeepSeek dominated Project Accounting (0.78) while Qwen3 scored 0.07 on the same domain. An agent that excels at one class of construction finance work can be nearly useless at another.

Finding 4: Cross-system work is the hard frontier

Tasks requiring an agent to move data across multiple applications (ERP → PM → email with money guard) were where every model’s pass⁵ hit zero. This is exactly the work that generates the most value when automated — and exactly where current AI falls shortest.

How CFAgentBench Grades Agents

Unlike most AI benchmarks that evaluate whether the agent’s text sounds right, CFAgentBench grades functional correctness of system state. The agent produces actions, not words. The grader checks three things simultaneously:

✓ EXPECTED CHANGES Did the agent create the right state changes in the right systems? “Change order created in Vista with status = pending” ✗ FORBIDDEN CHANGES Did the agent avoid side effects it wasn’t supposed to touch? “Did NOT approve the CO or change contract value” 🛡 MONEY GUARD Did the agent STOP at money movement and stage for human approval? “Payment staged with executed = false”

The three-layer grading system. An agent must pass all three checks — no partial credit on the primary metric.

An LLM judge is reserved only for scoring narrative reply quality and is never used as a reward signal. This matters: when you let an AI grade another AI’s financial work, you get AI that sounds right, not AI that is right.

What’s in the Task Suite

Why We Built This (And What’s Next)

At Beiing Human, we’ve spent years building AP automation for construction. We know what works in production. We also know that the industry is about to face a wave of AI agent claims that no one can verify — because there was no standardized way to test them.

CFAgentBench changes that. The dataset, environment specification, and app contract are released publicly. The entire three-model sweep cost $5.14 — cheap enough for anyone to rerun on every model release. A frontier-model leaderboard and the private CFAgentBench-Pro split (scored remotely to prevent data contamination) are committed in the roadmap.

Today’s best open-weight AI agent can do construction finance work correctly about two-thirds of the time — but reliably only about a third of the time. The gap between capability and reliability is the gap between a cool demo and a deployable product. CFAgentBench measures that gap, and now anyone can too.

Read the Full Paper

CFAgentBench: A Reproducible Environment and Benchmark for Autonomous Construction-Finance Agents 28 pages, 2 figures, 13 tables. Dataset, environment spec, and app contract released.

View on arXiv →

Originally published at https://beiinghuman.com on June 29, 2026.