Seat Risk

Sixteen Developers, 246 Tasks, and the AI Coding Budget Trap

In 2025, 16 experienced open-source developers finished 246 real tasks slower with AI, turning seat expansion into a measurement problem.

Share free access to this member-only story with a friend: Read it free here.

Figure 1. Sixteen Developers, 246 Tasks, and the AI Coding Budget Trap. Image created by the author with diffusion-synthesis and Python post-processing.

METR put a stopwatch next to the code diff, and the invoice started looking less simple. For a founder approving AI coding seats in 2026, the bad bet is not just another SaaS line; it can become slower delivery, a thicker backlog, and senior review time nobody built into the budget.

This was not a demo reel. In a 2025 randomized controlled trial, Model Evaluation & Threat Research watched 16 experienced open-source developers complete 246 real tasks inside mature repositories they had known for an average of 5 years. Some got AI access, some did not, assigned at random. Before the work, developers expected a 24% speedup. Afterward, they felt 20% faster. The measured completion time moved the other direction: 19% slower.

The keyboard felt quicker, but the merge took longer.

The trial broke the sales-room spell

The METR setup matters because it did not ask developers to solve toy prompts under studio lighting. The tasks were bug fixes, features, and refactors from real open-source projects. The tools were also familiar to the 2025 market: developers primarily used Cursor Pro and Claude 3.5 or 3.7 Sonnet, according to the arXiv paper.

That makes the finding painful. The study did not prove that AI coding tools are useless. It proved something more useful for operators: when skilled people work inside codebases full of local history, measured output can split from felt speed.

METR also published anonymized data and regression code. That is the part founders should respect. The scary object is not a quote from a skeptical engineer. It is a dataset where the developer’s confidence, the forecast, and the completion clock disagree.

Confidence is not throughput.

If your company only asks engineers whether AI feels helpful, you can buy the 20% story and miss the 19% cost. That is how a clean demo becomes a messy operating expense.

Familiar repositories punish generic fluency

A mature repository is not just files. It is old migrations, quiet conventions, weird test fixtures, half-written docs, release scars, and decisions that live in one senior engineer’s head because the last refactor happened during a customer fire.

AI tools are good at producing plausible code. Familiar codebases punish plausibility.

The hidden labor arrives after the first draft. A developer reads the generated diff, checks whether the naming fits the house style, runs tests, inspects edge cases, repairs imports, rewrites the part that misunderstood a local abstraction, and then re-reads the whole thing because the tool was almost right. Almost right is where expensive review time goes to hide.

This is the verification tax: review and repair time that eats the minutes saved during generation. A simple way to write it is:

Verification tax = review time + rework time + context repair — generation time saved.

That equation is crude, but it changes the buying conversation. AI creates real advantage when the task has clear boundaries, strong tests, visible examples, and low local context. It gets expensive when the answer depends on buried product rules, implicit architecture, or a maintainer’s private map of what breaks in production.

Figure 2. Proof of source-backed strategy: The cited source, Abs 2507.09089, gives the article a primary source readers can inspect. Source: Arxiv

The incentives are asymmetric. Vendors get paid when the first draft looks impressive. Founders feel relief when the backlog looks less brutal. Senior engineers absorb the residue when the generated code has to survive review, tests, security, and users.

Seat licenses are cheap props until the review bill arrives.

DORA turns one slowdown into a system warning

Google’s 2025 DORA report gives the METR result a wider operating frame. DORA surveyed nearly 5,000 technology professionals and included more than 100 hours of qualitative data. Its central finding is not that AI magically fixes software delivery. It is that AI amplifies the system already around it.

That is the link founders should not skip. METR shows a task-level trap in familiar repositories. DORA shows why the trap appears unevenly across companies.

DORA’s AI model points to boring assets with brutal consequences: strong version control, AI-accessible internal data, small batches, a clear AI stance, and a quality internal platform. Those are not conference-stage nouns. They are the rails that decide whether generated code becomes a shipped improvement or a review pile.

A team with small pull requests, reliable tests, searchable internal docs, clean ownership, and fast rollback has places where AI can remove work. A team with oversized tickets, flaky tests, missing context, and heroic reviewers gives AI a bigger room to create uncertainty.

That is why the 2026 decision is not “AI or no AI.” The better question is where AI reduces the total path from ticket to safe production. The answer changes by task type, repository maturity, test coverage, and reviewer load.

The 2026 buying rule starts before the next seat

METR’s own February 2026 update complicates the story in the right way. The group said later experiments became harder to interpret as more developers adopted agentic tools, avoided non-AI conditions, selected tasks differently, and worked on other things while agents ran. METR also said late-2025 tools appeared stronger in some settings, but the signal was weakened by selection effects.

That does not rescue careless buying. It makes measurement more urgent.

Before expanding seats, run a small internal trial on work that actually matters. Pick 20 to 40 real tickets. Track start-to-merge time, review cycles, test failures, rework, and any production cleanup. Compare AI-assisted tickets against matched tickets without AI access, or randomize the condition when the team can tolerate it. Count waiting time and repair time, not just typing time.

Then route work with discipline. Low-context tasks with strong tests can go AI-first. High-context core code should use AI for narrow helpers, test scaffolds, or explanation before touching the main diff. Product logic with unclear acceptance criteria needs sharper specs and tests before it needs another model subscription.

This is a game of second-order effects. If every engineer generates more code than the review system can absorb, the bottleneck moves downstream. If AI turns senior engineers into auditors of plausible mistakes, your most expensive people stop doing their highest-value work. If the company measures only output volume, it rewards the behavior that creates the next delay.

The METR trial should not make technical founders timid. It should make them precise. Buy AI where saved minutes survive review, tests, and release. Hold back where the tool only moves labor into invisible verification.

The next budget decision is sitting in the same place as METR’s proof object: a real task, a real repository, and a clock that does not care how fast the work felt. Buy the seats only where the saved minutes survive the merge; thank you for reading.

Related reading

• GitHub Says 55% Faster. METR Found The Opposite.

• Stack Overflow’s 2025 Trust Gap Hits Autonomous Coding Agents

• Copilot’s 2026 Handoff Starts With an Issue and Ends in Review

• OpenAI’s 2025 Operator Makes Checkout Failure an Agent Problem