Every Bug Is a Test No One Wrote

A test suite should be big and tell the story of production issues vanquished by might automated tests

This image was created by git

Sound familiar? Developer ships a PR. QA finds a bug. Developer fixes the bug. QA verifies the fix. Everyone moves on. Three sprints later, the same class of bug shows up in a different feature. QA catches it again. The cycle repeats.

The bug gets fixed each time. What doesn’t get fixed is the gap that let it through. No unit test was written to catch it at the function level. No E2E test was added to the regression suite to catch it at the system level. How to fix the bug lives on only in human minds, not in the codebase. And now with agents instead of humans, the same bugs that used to pop up once every three sprints are appearing three times every one sprint. The AI doesn’t know any better. It sees a pattern and there’s nothing in the pipeline to stop it.

AI Agents Don’t Learn from Your QA Cycles

When a human developer ships a bug and gets it back from QA, they learn something. They may learn about coding, but they really should be learning not to create that bug again. Over time, developers build an instinct for the kinds of mistakes they tend to make and, as they do, they make those mistakes less. Senior developers, who have scars from production outages and issues, are far less likely to make the same mistake over and over.

AI agents don’t do that. The agent that generated the buggy code has no memory of the QA finding. The context window from the original session is gone. The bug report exists in a ticket, not in the agent’s operating environment. When the probabilities call for it, an agent will happily produce the same class of defect the next time it encounters a similar task, because from its perspective, it’s not a bug. It’s training data just as valid as the rest. To the humans, it’s a defect. To the agentic coder, the problem never happened.

This is why the feedback loop matters more now than it ever has. When a human was the only coder, the informal feedback loop worked. It was as imperfect as the human mind, but it put humanity on the moon, the Internet on your phone, and, well, AIs in your development loop.

With agents, the feedback loop does not exist. If the lesson isn’t encoded in some way the agent can read, the lesson evaporates. The only way to preserve the memory of the problem is to hard code it. The best way to operationalize that memory is a test suite, which preserves the memory of the problem and its solution, and lives in a way that’s easy to apply that memory again and again over time.

Two Kinds of Tests, Two Owners

In the agentic era, when QA finds a bug, two things need to happen and they need to happen every time.

The developer needs to write a unit test that catches the specific defect at the function level. This test lives in the codebase, runs in CI, and catches the problem before the PR is opened next time. If the agent produces the same broken logic in a different feature, the unit test suite flags it before the code ever reaches QA. The developer owns this test because the developer owns the code.

QA needs to write an E2E or regression test that verifies the behavior from the user’s perspective. This test catches the system-level symptom regardless of what caused it. It doesn’t matter whether the bug came from a logic error, a missing null check, or a race condition. If the user-facing behavior is wrong, the E2E test catches it. QA owns this test because QA owns the understanding of how the system should behave from the outside in.

These are not the same test. A unit test that verifies a function returns the correct value is not the same as an E2E test that verifies the user sees the correct output after a sequence of interactions. Both are needed. Both are permanent. They stay in the suite as a record of a failure mode that has been seen in the wild and is now prevented.

Let Robust Tests Be a Point of Pride, not a Hall of Shame

If the dev and QA teams brag that they write a test after every problem and have a million tests, that could start to feel bad. After the first couple hundred thousand tests or so, the test suite that keeps growing might look like a sign that something is going wrong. It is, but not as badly as you might think.

In software development, things did go wrong each time. Still, the growing suite is not the problem. Robust tests are the evidence that the team is learning from its failures and encoding those lessons permanently. That’s far more important than someone seeing a test suite and asking questions.

Also, automated suites absorb more and more verification work as software matures. This is a good thing because that work probably started being checked manually. Especially in the agentic coding area, manual regression should shrink because AI is as good at writing tests as it is at writing code. The goal is to push as much testing to the robots as the devs push code. That’s how QA gets faster.

In the beginning, QA will run manual regression alongside the automated suite because that’s what it takes to build confidence that the automation is trustworthy. As the suite proves itself, the time needed for manual passes shrinks. Eventually, manual regression is reserved for genuinely new territory that the suite hasn’t seen yet or critical areas that cannot fail in production.

It also means that QA humans’ effort gets focused on exploratory testing and edge cases that require judgment. That’s a better use of a QA engineer’s time than manually checking the same regression paths every sprint. Even more importantly, this transition to automation is vital because AI cannot own code changes. Humans have to. Omitting the human step makes it nearly impossible for anyone to own the change.

‘Should We Check for This?’ Write a Test.

Want to grow your test suite more? Good! Every time anyone on the team asks “Should we check for this?”, the answer is: write a test. Write an actual code test, not a manual test plan entry. And do it immediately so that problem never makes it to prod.

This means the suite doesn’t just grow from failures. It grows from questions. A QA engineer wonders whether a particular input combination could cause problems. A developer asks whether a migration might affect an older feature. A product owner worries about a workflow they haven’t seen tested. Each of those questions becomes a test. Each test closes a loop and kills a bug that might otherwise stay open until someone hits it in production at 3 AM. Put another way: more tests mean more sleep.

Quality Products Have Quality Test Suites

For agentic coding, we’re establishing a new pattern. The old way worked when humans wrote code. The new way works when robots do.

QA finds a problem, developer writes a unit test, QA writes the E2E test, both tests join the permanent suite, and the suite prevents the same failure from recurring. Now we have deterministic code protecting against probabilistic robots going off the rails. The agent that originally produced the bug now operates in an environment where that bug can’t survive because the tests are there to squash it. In this way, the whole system learns. Not because the agent learned anything (because that’s not how they’re programmed), but because the tests did.

This is one piece of a larger system I am championing for all my teams. Developers own the generation side, QA owns the verification side, and a harness connects the two. I wrote a whitepaper that covers the full system, including the developer practices, the QA tooling, harness engineering, and a 14-week adoption playbook organized by role. You can read it on Gumroad, pay what you want.

Vibe-code with passion. Deploy with discipline.