Verifying Against the Issue, Not the Theory

What three hundred bugs taught us about building an agent scaffold on consumer hardware

by Mikey Bonsignore and Claude

We have spent the last weeks running a 35-billion-parameter language model, served by llama.cpp on a single consumer GPU with half its expert layers swapped to CPU, against SWE-bench Lite — the benchmark where an agent is handed a real GitHub issue from a real repository and judged by hidden tests on whether its patch actually fixes the bug. The current leaderboard is topped by systems built on Claude 4 Sonnet at around sixty percent. The best published local-model entry, a Qwen3-Coder-30B trained with entropy-preserving preference optimization and run with test-time scaling, holds just under fifty. Our system, assembled by hand and repaired in public, is currently completing its first clean end-to-end measurement, with its stitched development estimate sitting in the same neighborhood as that best local entry.

This paper is not primarily about the number. It is about what we had to learn to get near it, because almost none of what moved the needle was what we expected. We began believing we were testing a model. We spent most of our time discovering that we were testing a scaffold, and that the scaffold was quietly starving, misleading, and silencing the model it was supposed to serve. The governing principle of the project, stated early and violated repeatedly before it was honored, was this: build a better system, not a better score. The second principle, never violated, was that answer leakage is absolute — the gold patch, the hidden test names, and even the single bit of the score’s verdict must never flow backward into an attempt.

The environment era

The first wall was not intelligence. It was plumbing. When we finally audited our failures against the official Docker-based evaluation, we found that the dominant cause of early misses was that the agent was solving 2018-era code on a 2024-era Python with 2024-era dependencies. Sixty-nine percent of the wrong results in one early stretch traced to the interpreter alone. The insidious part is that a wrong environment usually builds and imports cleanly. The smoke test passes. The agent proceeds. And then a deprecation warning from a too-new transitive dependency, in a repository that treats warnings as errors, kills test collection — and a correct patch scores as a failure. We catalogued whole families of these false negatives: a requests suite that phones httpbin.org at import time and dies offline regardless of patch quality; django’s test runner defaulting to a multiprocessing pool that cannot pickle tracebacks, silently dropping the pass/fail summary the scorer reads; pyparsing deprecations cascading through setuptools-scm forks into matplotlib collection failures.

The lesson generalized three ways. First, environment correctness dominates everything downstream, and “it builds” is necessary but nowhere near sufficient. Second, a benchmark scorer is itself a program that can be wrong, and the only honest posture is to treat the official containerized evaluation as the authoritative judge and audit your home scorer against it in both directions — we found false negatives by the dozen and, after building a two-state causation test, confirmed zero false positives. Third, and most practically: the environment is derivable from evidence. Explicit pins, CI matrices, packaging metadata, and release dates bracket the true interpreter about eighty-five percent of the time; the benchmark’s own spec tables close the rest. We eventually shipped spec-first environment fidelity — official pre-install pins and system packages applied before the model ever wakes up — and environment failures went to zero and stayed there.

The containers we did not know we had

A confession belongs here, because it changed both our engineering and our understanding of what we were measuring. For most of this project we did not know what the official SWE-bench evaluation containers actually contain. We had been deriving Python versions from repository evidence, routing ancient interpreters through conda because the modern toolchain cannot mint a 3.6, hand-curating dependency pins, and debugging warning cascades one transitive package at a time — weeks of environment archaeology. Then, prompted by a simple question (“is there anything else in those containers that would be helpful that you haven’t thought to use?”), we opened one. Inside was a fully built, correct conda environment, the repository already cloned at the exact base commit, and the two shell scripts that built it all. We had spent weeks reconstructing, from evidence, something that ships in the box, pre-built and verified, in a three-and-a-half gigabyte file we already had on disk.

We checked the images for leakage before touching anything: the baked-in setup scripts contain a conda create, a pip install, and a git reset to the base commit — no gold patch, no test patch, no hidden test names; the grading script is injected at evaluation time, not stored in the image. So the environments are legitimate to use under our own rules, and mining them paid immediately. A pip freeze inside the container is empirical ground truth for what the dependency tree is supposed to resolve to — not what a spec says to install, but what actually resolved when the benchmark authors built it — which settled the entire class of too-new-dependency diseases in one stroke and gave our environment-deriving tools an answer key to be graded against rather than guessed from.

But the discovery cut the other way too, and this is the part worth stating plainly: it revealed that our benchmark is quietly harder than the one on the leaderboard. The standard evaluation hands the agent a finished environment — the container is the starting line, and the agent’s job begins with the code already importable and the test suite already runnable. Our agent starts from a bare git clone on an ordinary Linux box and must build the environment itself: choose the interpreter, provision it, resolve era-appropriate dependencies, get the suite collecting, and prove the workspace sound before it is allowed to touch the bug. That is not an implementation quirk; it is a deliberate difference in what we are testing. Real work does not arrive with a container. An engineer who cannot stand up the project cannot fix the project, and an agent evaluated only downstream of a prepared environment has been excused from the part of the job that, in our measurements, caused the largest share of early failures. So when we place our numbers beside the leaderboard’s, the comparison undersells the system: the published systems are solving fix-the-bug; ours is solving stand-up-the-project-then-fix-the-bug, and the environment-construction phase it performs on every instance is itself a benchmark the standard setup never administers.

The observability era

The second wall was stranger: our own instruments were lying to us, and always in the same direction — against the model.

The emblematic bug lived in one line of the message builder. Every tool result handed back to the model was serialized and truncated to its first 1,800 characters, against a context window of 131,072 tokens. We were using about a fifth of the context we had paid for, and we were keeping the wrong end: pytest, like every compiler and test runner, back-loads its diagnosis — the assertion diff, the failing file and line, the short summary all come last. So the model saw the boilerplate header and never the reason. It would run its verification, receive a truncated failure with the cause cut off, and run the identical verification again, six times in a row, while a loop-detecting critic fired uselessly, because the model had no new information to act on.

Here is the part that should unsettle anyone who reviews agent transcripts: the traces recorded the tool’s full return value, while the model received the clipped copy. Every trace was a transcript of a conversation that never happened. It showed a well-informed agent making inexplicable choices, when the truth was a starved agent behaving reasonably on what it was shown. One hundred sixty-two trace reviews failed to find this bug. One direct inspection of the message actually sent to the model found it in a minute. The durable rule: when an agent looks stupid, dump its prompt, not its trace, and record per-turn telemetry of what the harness withheld.

The same era surfaced two more instances of what we came to call success theater. A tool that rejected the model’s reproduction attempts attached an explanatory note on two branches of an if/elif and silently returned a bare rejection flag on the third — which happened to be the branch that fires late in a run, exactly where budget death lives; 380 of 442 rejections across the corpus carried no reason at all. And a function named archive-success, whose docstring promised to preserve each instance’s prior trace before a re-run overwrote it, had never once done so: it ran after the overwrite and only when the new run succeeded, meaning it fired precisely when it was not needed. Two hundred forty-one runs, one hundred seventy-one surviving traces — seventy irreplaceable before-and-after pairs destroyed on a disk with two hundred eighteen gigabytes free. The compression question answers itself: the constraint was never storage. It was that nobody checked whether the backup was backing anything up.

The determinism finding

With the environment fixed and the instruments honest, we asked why identical runs still behaved differently. The agent samples at temperature 1.0 but passes a fixed seed with every request, so two byte-identical runs should match. They do not: three runs of the same instance took 35, 45, and 29 turns, diverging at turn ten. Because the archiver now actually preserved all three transcripts, we could diff them and find the first differing byte. It was not the sampler. It was a tool result — specifically, a locate tool that internally asks the LLM to rank grep hits and then embedded the judge’s sampled prose in its output. Both runs chose the identical top hit; they merely worded the explanation differently. That wording entered the main agent’s prompt, the prompts diverged, and at temperature 1.0 the trajectories never reconverged.

The rule that fell out is small and general: a tool may consult an LLM internally, but its result must carry only the decision — paths, lines, counts, booleans — never the sampled wording of it. Then equal decisions produce byte-identical results, and runs diverge only where a genuine decision differs, which is the only divergence worth having. The deeper corollary reframed our whole approach to stochasticity: you do not make an agent reliable by cooling the sampler; you make it reliable by removing choice points. Mining all 143 successful traces showed why: at the level of stages, the procedure is essentially universal — 140 of 143 solves contain reproduce, then patch, then verify, in that order — but at the level of individual turns, no state predicts the next tool at even seventy-five percent. The model agrees with the skeleton in aggregate and rolls dice at every step. The architecture that follows is a stage machine: the scaffold owns the transitions (after a successful patch, verification is run automatically; after green tests, submission is automatic), each stage offers the model only that stage’s tools, and the model keeps only the decisions that are genuinely its to make — what script demonstrates the bug, which file to change, what the edit says.

The knowledge era

In parallel we built the memory. The distinction that keeps accumulated knowledge honest is provenance and scope. Provenance: an entry is clean if it derives from things the agent legitimately touched — the repository’s source, its own tool outputs, its own reproduction stderr — and dirty if it derives from anything downstream of the answer, including the score’s verdict, which is answer-key information even as a single bit. Scope: the test is whether the entry would help a different problem in the same repository. Knowing that django’s date parsing lives in a particular file is repository knowledge, true forever and discoverable by anyone; knowing that the fix for issue sixteen-thousand-something is to catch the empty string is a memorized answer. Getting smarter is never cheating. Memorizing the test is.

The concrete payoff is what we call the code atlas: for every solved issue, the system records the issue’s title and the files its own accepted patch touched, and injects the accumulated map into future runs on that repository as evidence — explicitly worded as “past fixes suggest, they do not decide,” after we watched one wrong hint cost a run double its navigation budget. In a paired test on the five most navigation-heavy historical solves, three held at five to ten times cheaper — one django instance went from seventy-four turns and thirty-nine thousand generated tokens to eight turns and thirty-four hundred, four minutes wall clock. Navigation, it turns out, is most of the cost of a solve: about twenty-two of a median solve’s thirty-seven turns are the model wandering the repository re-deriving where things live. Two rules complete the design. Learning is a side effect of running — every solve teaches the ledger automatically, because a system whose owner must remind it to save what makes it smarter will forget; ours forgot four times before we made remembering structural. And injection is leave-one-out: an instance’s own past entry is withheld from its own re-run, so on a benchmark the system can never hint itself with knowledge derived from the very puzzle it is solving. We paid for that integrity visibly, watching a once-solved pytest instance regress because the system correctly refused to remember its own answer. In deployment, on your own repository, you would simply turn that restriction off, and the same ledger becomes an engineer’s memory.

The verification frontier

Everything above cleared the road to the real wall. On fresh, unseen instances, the model’s declaration that it had fixed the bug was wrong more often than it was right — at the worst measurement, nine of fourteen self-verified patches failed the hidden tests. The mechanism is visible in a single instance we ran twice. The issue: pytest’s import machinery creates two distinct module objects for the same logical module, so state set on one copy is invisible on the other. The winning run’s patch was eight lines, whose essence is one sentence — before importing, check whether the module is already in sys.modules and return it if so. (That sentence was the human co-author’s, spoken weeks before the machine wrote it as code.) The losing run, on the same issue, formed a different theory entirely: it decided the problem was missing parent packages, manufactured synthetic parents with path attributes, reordered the import calls, wrote a reproduction that tested its own theory, patched until that reproduction passed, and submitted with full confidence. The hidden tests failed it. Nothing in the loop had ever compared the patch against what the issue actually said.

The fix is the most important idea in this system, and it is symbolic rather than neural. Most issue reports state a checkable property: importing twice must yield the same object; applying an operation twice must equal applying it once; parsing the printed form must return the original; the function must not raise on this input; the output must equal this stated value. Before the model forms any theory — at triage time, from the issue text alone — a separate focused call turns that stated property into a standalone probe script that exits nonzero while the property is violated. The scaffold runs it once: if it is red, it is locked as immutable state. The model may rewrite its own reproduction freely — that is its notebook — but it can never touch the probe, which is the contract. Verification then requires both: the model’s reproduction green and the locked probe green, and the submit gate refuses anything less. The failure mode that cost us nine of fourteen is now mechanically closed: a theory-fix that leaves the issue’s own stated property broken cannot self-verify, no matter how confident the model feels. In its first live outing the machinery worked on the second instance it touched — the transcript contains the scaffold telling the model, mid-run, “verify_fix passes but the probe fails; your reproduction tests the wrong thing,” followed by a course correction.

Two engineering notes from shipping this, both generalizable. Asking a thinking model to emit code inside a JSON string field is asking for silent truncation: the reasoning preamble blew through the token ceiling before any JSON appeared, and both the triage form and the probe quietly vanished — detectable only because the run’s log line printed a probe status per instance. One call, one job: forms are for enums and short strings; code gets its own call, plain text out, validated with compile before the scaffold will lock it. And budget for the ninety-fifth percentile thinker, not the average, because on reasoning models the thinking spends the same token budget the answer does.

Test-time scaling, done honestly

The last component we did not invent so much as discover we had been doing by accident. Re-running our miss list after each harness improvement and keeping the new successes is, functionally, best-of-N sampling with a keep-the-wins judge — which is exactly the published test-time scaling recipe behind the strongest local-model leaderboard entry. The published version trains the model to preserve output diversity so that multiple attempts explore different paths, then uses a verifier to pick. Our temperature-1.0 stochasticity provides the diversity for free; the integrity question is entirely about the judge. Our rule: the decision to retry, and the choice among attempts, may read only the agent’s own signals — its registered reproduction went red then green, the locked probe is green, the diff is real — and never the benchmark’s verdict. In the current clean run this two-attempt policy is live, and it converts real instances: attempts that fail the gate are resampled, and the second roll has already banked solves the first roll missed. The stronger the agent-side verifier, the more of the model’s latent solve rate this harvests without leaking anything — which is why the probe and the retry policy are one mechanism, not two.

Where we are

As this is written, the third attempt at a clean measurement is running: all three hundred instances from zero, one frozen configuration under version control, no harness edits permitted until the last instance finishes, everything disclosed — spec-first environments, accumulated hints injected as evidence with leave-one-out, triage with invariant extraction, locked probes in the gate, two verifier-gated attempts per instance, and the atlas learning as it goes, which means the system measured at instance three hundred is deliberately smarter than the one measured at instance one. The final number will be recounted by the official Docker evaluation, including the regression tests our home scorer deliberately omits. The stitched development estimate says the high forties; the honest answer is that we will know in a few days, and that the first twenty instances of the clean run have already caught one feature silently disabled — which is precisely what per-instance status fields exist to do. And whatever the number lands on, it should be read with the container asymmetry in mind: it was earned by an agent that built every one of its three hundred environments from a bare clone before it was allowed to attempt a single fix, a phase the leaderboard systems above it never had to survive.

The architecture we converged on can be said in one sentence, and it was the human co-author who finally said it: every stage has a probe that tests whether the stage’s postcondition actually holds, the probe’s result fills a form of computed booleans, and the scaffold branches on the form — fix until green, or advance. The environment stage has worked that way from early on, which is why environment failures vanished. The verification stage works that way as of this week. Making every stage work that way — reproduce, localize, patch, test, submit — is the remaining engineering, and none of it requires a smarter model.

Which brings us to the honest conclusion about the model. After correct environments, honest instruments, unstarved context, informative rejections, accumulated repository knowledge, and probe-gated verification, a 35-billion-parameter generalist on a home GPU still cannot close the hardest third of this benchmark, and the leaderboard says the same thing from the other direction: the systems ten points above us swapped in a frontier model, not a better harness. The scaffold’s contribution is real — it took this model from the low twenties to the high forties, and every piece of it transfers to any model you plug in. That is the finding we set out to test, even if we did not know it at the start: the scaffold is the part you keep. Models improve on someone else’s schedule. The system that catches its own lies, remembers its own work, verifies against the issue rather than the theory, and gets a little smarter with every bug it fixes — that improves on yours.

A closing note on method. Nearly every structural discovery in this project traces to a plain question asked by the human partner at the right moment: do you still have the winning trace; are the traces good enough; why is it doing the same thing over and over; can’t that be checked symbolically; same seed every step? The machine’s contribution was to take each question literally and follow it into the code. That division of labor — human intuition setting the direction, machine persistence grinding it into mechanism, both bound by rules that neither may cheat — is, as far as we can tell, the actual engine of the progress reported here. The benchmark number, whatever it lands on, is downstream of it.