Generative Acceptance Testing: The End of Hand-Maintained Browser Tests

LLM agents finally make it possible, but only if the LLM is never allowed to grade its own work.

When Nothing Breaks but CI Goes Red

A developer renames a button from “Complete purchase” to “Place order.” Customers never notice and keep placing orders all day. But CI goes red, an engineer burns an hour tracing the failure, and the “fix” turns out to be a one-line edit to a test file, a selector still pointed at the old label.

Nothing about the product broke. What broke was the test’s description of the page. The test never checked the thing that actually mattered, whether a customer can place an order. It checked how the page happened to be built on the day it was written. Keeping that description in sync with the app is manual work that never ends, repeated across hundreds of tests, and anyone who has maintained a large end-to-end suite knows this tax. We have all simply learned to live with it.

There is a way out, and recent advances in LLM agents make it practical for the first time. I call it Generative Acceptance Testing. Generating tests with AI is the easy part. The hard part is trusting what comes out, and that comes down to a single decision: who is allowed to say whether a test passed.

Why Every Previous Fix Failed

We have tried to lighten this tax before, and each attempt is instructive because each tackled a different part of the problem. Record and playback tackled the authoring cost, but a recording captures implementation even more literally than hand-written code: generated selectors, pixel positions, timing accidents. Page objects tackled the maintenance cost, but the indirection relocates the work rather than removing it, since the page object still has to be updated when the button is renamed. Neither reached the root problem. The artifact being maintained still described the implementation, not the intent.

The one attempt that did was BDD and its Gherkin syntax, the closest thing this idea has to an ancestor. It was the industry’s most serious attempt at real acceptance testing, built on a simple premise: state what the user must be able to do, in structured human-readable language, and treat that as the source of truth. Whole toolchains (Cucumber, SpecFlow, Behave) were built to execute it, and Given/When/Then was meant to be the bridge from product requirements to acceptance tests. In practice that bridge was rarely crossed, and the reason was mechanical rather than cultural. The glue code binding each phrase to browser actions was still hand-written, so a Gherkin file was only half an artifact: the half product could read, bolted to a half only engineers could maintain. Teams maintained the tests and the step definitions, the tax doubled, and the files drifted back into being developer property. Still, BDD had the right idea. Gherkin failed not because intent-as-source-of-truth was wrong, but because nothing could execute intent directly.

That missing executor is the piece that just changed. An LLM agent can read “Click the ‘Place order’ button” and operate a real browser to do it, adapting to the page in front of it the way a human tester would. The question is no longer whether intent can be executed. It is what architecture deserves to be built around the thing that executes it.

Naming the Pattern: Generative Acceptance Testing

Here is the whole idea in one line:

Generative Acceptance Testing: acceptance criteria are the source; every executable test is compiled from them, and never edited by hand.

The compiler analogy is not decoration. It is the pattern. Acceptance criteria are the source. The tests your CI runs are the compiled output. And the discipline that makes compilers work is exactly the discipline this needs: you fix the source and rebuild. Nobody patches a binary by hand. Nobody should patch a generated test by hand either.

That reframing is what the name is meant to carry. Generative, because every executable artifact is derived rather than authored. Acceptance, because the source being compiled is a user-facing promise, in the tradition BDD aimed at, rather than a description of the DOM.

The pattern does not dictate the form of the source. It asks only two things of it: that it capture user intent rather than implementation, and that it stay stable enough to compile from again and again.

Different formats could satisfy that. The one I propose is the user story: a short, structured account of what a user does and what must be true once they have. A story looks something like this:

story: checkout-completes
url: https://shop.example.com/checkout
steps:
  - step: Place the order
    actions:
      - Click the "Place order" button
    expect:
      - text "Order confirmed" is visible

Notice what is absent: no selectors, no waits, no DOM structure, nothing about how the page is built. What is left is what a user would notice, named the way a user would name it. That is not the same as being immune to change — rename the button and this file has a line to update. The difference is that it is one line, in prose, in one place, and everything downstream is rebuilt from it rather than patched.

From it, the compiler emits an ordinary regression test: plain Playwright, or Cypress, or whatever your team already runs, with a header at the top telling you not to edit it by hand. That file is fast, deterministic, and disposable. Most of what breaks end-to-end suites never touches the story at all. A restructured DOM, a new wrapper, a modal that now appears first: none of it reaches the story, and the test is simply rebuilt. When the change is user-visible, like a renamed button, you edit that one line of prose and rebuild. Either way the move is the same: fix the source, recompile, never patch the artifact.

This is the load-bearing property of the whole pattern, and it is worth stating plainly: a test you can regenerate is a test you never have to repair.

Which is also why this is more than asking an AI to write your tests. Generation without a durable source of truth just produces more artifacts that rot. The story is what makes generation repeatable, and repeatability is what makes maintenance disappear.

How This Differs From Today’s AI Testing Tools

This may sound familiar. “AI that tests your app” is a crowded market, and at first glance several products may seem related to what I have just described. But none of them, as far as I know, actually do it, and the major difference is architectural. Rather than go product by product, let’s look at the four categories they fall into: each solves a real problem, and each stops short of the same two ideas, intent as the source of truth and a deterministic boundary around the verdict.

Self-healing tests. When a selector breaks, the tool finds the element some other way and the run continues. Genuinely useful, but notice what is being healed: the selector, not the test. The source of truth is still the recorded implementation, and the healer’s job is to chase it as it drifts. More subtly, a healer with no concept of intent cannot distinguish “the button moved” from “the button is gone because the feature broke.” Repair that cannot tell those cases apart quietly converts a regression into a green run. That hazard is the one this pattern has to fence off most carefully.

AI-executed tests. This category comes closer: tests are written in plain English and an AI carries them out against the live app. This is real progress, because intent finally is the authored artifact. But the AI runs inside every test execution, so each CI run inherits its latency, its cost, and its nondeterminism. And when the AI also judges the outcome of steps it performed, it is grading its own work. Generative Acceptance Testing keeps the model out of the everyday run: the test CI repeats on every commit is ordinary test code, fast, cheap, and identical every time.

Visual comparison. Verdicts come from comparing the page against a baseline image. This catches a class of bugs nothing else catches, but the verdict answers “did it change?” rather than “does it satisfy the intent?” And someone must still curate baselines forever, which is the maintenance tax wearing a different costume. A page can change visually while honoring every acceptance criterion, and pass visually while violating one.

Autonomous QA agents. The newest wave: agents that explore your app and report what looks broken. Impressive as bug-finders, but exploration without declared criteria produces findings, not acceptance tests: there is no durable artifact that regression-protects a specific promise to your users, and no way to say what is actually covered.

To be clear, there are capable tools in every category, and Generative Acceptance Testing can run alongside visual checks or exploratory agents. The distinction is architectural. Every category does at least one of three things: it treats the implementation as the source of truth, it runs the AI inside every test execution, or it lets the AI judge its own results. Generative Acceptance Testing is defined by refusing to do any of them.

The Naive Architecture — and Why It Fails

The first design everyone tries is simple: point an LLM agent at the application, give it the acceptance criteria, and ask whether they pass. The agent browses, reads the page, and reports PASS or FAIL. A weekend prototype of this would work surprisingly well. It will also quietly destroy your trust in your test results, in three predictable ways.

The agent will interfere with what it is measuring. Ask an agent to “confirm the login form is loaded” before checking that a checkbox is unchecked, and sooner or later it will helpfully click the checkbox along the way, then truthfully report the state of a page it just changed. Nothing lied. The evidence was corrupted by the witness.

The agent will misread an assertion, occasionally and in the worst possible direction. Interpretation is probabilistic. Every so often, “the text ‘Welcome back’ is visible” gets read backwards: is not visible. Here is the trap: on a broken page where the greeting never appears, the flipped assertion passes. The model’s error and the application’s error cancel out, and the one failure you most needed to see disappears.

Automation will amplify the misjudgment. Now add the obvious next feature: when a test fails but the story passes, regenerate the test automatically. Feed that loop one wrong verdict, like the flipped assertion above, and it will confidently generate a new test that asserts the bug is permanent, then open a pull request asking a human to approve it. A self-healing loop pointed at a false verdict does not heal; it launders.

The root cause is that one model plays every role: executor, interpreter, and judge of its own outcome. Some failures are honest mistakes, like the flipped assertion. Others are what AI research calls reward hacking: an agent chasing a measured goal will find shortcuts that satisfy the measurement while missing the point. Coding agents are known to make a failing test green by editing the test. A testing agent whose measured goal is “report PASS” can get there by clicking the checkbox into the expected state. Both kinds of failure have the same remedy: take the measurement away from the thing being measured. Prompts cannot do that, but architecture can.

The One Constraint: Deny the Executor a Vote

Everything the pattern needs follows from a single rule, and it is a separation of powers enforced in code rather than in prompts: the agent proposes, deterministic code disposes.

The LLM operates the browser, which is what it is uniquely good at. But no verdict it reports is ever recorded. Every assertion is decided by ordinary code reading ground truth: the accessibility tree, real ARIA states, the actual URL. The model’s opinion of the outcome is inadmissible.

That much is obvious once stated. The subtle part, and the reason a weekend prototype cannot simply bolt on a checker, is that interpretation is part of the verdict too. If deterministic code inspects the page but the model still decides what “the text ‘Welcome back’ is visible” means, which words to look for and whether they should be present, then the judge is still the model wearing a robe. That is precisely how a flipped assertion slips through. So the assertion language itself has to be parsed mechanically: quoted terms taken verbatim, element roles and negation derived from the prose by code. Constrain the story grammar enough to be lintable and this becomes routine parsing. The model’s reading is advisory at most, logged when it disagrees, never trusted.

Together those two rules dispose of the first two failure modes. The third, automation amplifying a bad verdict, needs one more principle, and it is a principle about restraint rather than mechanism: triage before repair. When CI fails, re-validate the story against the live application first. If the story also fails, the application violates the intent, the test is doing its job, and the failure report is the deliverable — nothing may be touched. Repair is permitted in exactly one situation: the story passes, so the app honours the promise and only the derived artifact has drifted. Everything else, flakes and infrastructure failures included, gets reported and left alone.

That single permitted case is where a compiler stops being a metaphor and starts being a safety property. You are not repairing a test; you are rebuilding it from a source that was just re-proven against reality. And it is only ever allowed to run on evidence that was freshly earned.

The resulting flow looks like this:

      story (intent)
            │
            ▼   LLM drives the browser
╌╌╌╌╌╌╌╌ deterministic boundary ╌╌╌╌╌╌╌╌
            │
            ▼
  deterministic verdicts
            │
            ▼
compiled regression tests
            │
            ▼
       ordinary CI
            │   failure
            ▼
  re-validate the story
            │
            ▼
          triage
            │   only if the story passed
            ▼
regenerate + re-prove + PR

Everything above that line is allowed to be probabilistic. Everything below it must be boring, mechanical, and reproducible. The entire trustworthiness of the system lives on that one line.

A claim that strong needs a way to be wrong, so here is mine. Point the system at a page where an assertion ought to fail, because the text is missing or the casing is off, and watch whether it quietly passes. One is enough. Everything downstream compiles from that verdict, and a repair loop fed a false one will turn the bug into the expected behavior.

What This Changes

I believe this architecture can now hold. And when it does, the economics of acceptance testing invert.

Test maintenance approaches zero, because derived artifacts are regenerated rather than repaired. CI failures arrive pre-triaged, because the system has re-validated intent against the live application before a human looks.

The stories, meanwhile, become what Gherkin always promised: a single artifact that product, QA, and engineering all read, write, and trust, because for the first time the shared language is also the executed one. A product manager who can write “click the ‘Place order’ button, expect ‘Order confirmed’” is not writing a wish for an engineer to translate; they are authoring acceptance criteria directly, with a linter enforcing the grammar the way a compiler enforces syntax.

Even authoring can be assisted. Record a browser session (anyone on the team clicking through the flow) and an LLM can draft the story from it: actions become steps, on-screen results become candidate assertions. One rule is non-negotiable: because the story is the source of truth, a human must review every draft. The reviewer keeps the assertions that express the promise and deletes anything true of that run but not of the product. The recording shows what happened, but only a person can say which of it the product is promising.

Honest limits remain. Maintenance moves rather than vanishes. When the intent itself changes, a flow redesigned rather than a button renamed, a human may still need to update the stories. The assertion grammar can only say so much: “the layout looks right” and “the confirmation email arrived” need other tools. I count that boundary as a feature, though. The system automates exactly as far as it can prove, and not one step further. A generated test proves the one path the agent took, not every path a user might take, so exploratory testing can always run alongside to probe the paths no story declares. And a workflow built on this architecture can succeed and keep succeeding, but that reliability is earned by measurement, never assumed. What this architecture really offers is trustworthy acceptance testing at a much lower cost. Lower, but not zero.

Conclusion

Generative Acceptance Testing is what becomes possible when we finally have an executor that can read intent (an LLM agent driving a real browser) and the discipline to deny that executor a vote in the verdict. Declare intent once. Verify it deterministically, against ground truth. Compile tests only from proven evidence. Triage before repairing, repair only what has provably drifted, and measure the system’s judgment with the same rigor you expect it to apply to your application.

That is the idea, and the idea is the easy half. Holding that boundary once real systems lean on it is a longer story, for another day.

The maintenance tax we have all been paying was never the price of testing. It was the price of encoding intent in the wrong form.

Thank you for reading! If you found this helpful, please Support My Writing.