Introducing AgentVerity: What Does a Green Agent Test Prove?

I wrote six labelled evaluation cases for a payment-dispute routing workflow. In the deployed version, an LLM classifies each complaint into one of six specialist workflows. Every expected label came back correct, so an exact-match quality check scored 6/6. Then I looked at what those six cases had actually done. All of them took the same route. An agent that always returned duplicate_charge would have earned the identical green result.

That is a vacuous green result: green for the tested route, silent about the other five.

This article introduces AgentVerity, an open-source Python library built around a narrower question than existing evaluators ask. When tests pass, did the target agent make a repeatable decision, and did the cases reach more than one of its choices? When the application declares its required decisions, did the suite include and observe all of them?

DeepEval, Ragas, and LangSmith score output and process quality during development and CI. AgentVerity checks a different condition, before a team trusts a run beyond the cases it contains. Currently alpha, the library belongs in controlled evaluation, not the customer request path.

This is narrower than “any agent”. Suitable targets expose decision points with a finite, reviewed set of labels or tool paths, such as routers, approval gates, policy checks, and multi-agent supervisors. An open-ended chat, coding, or research agent fits only where it also exposes one of those bounded decisions.

Why a router is hard to test

In conventional software, a router is usually explicit code. Static tools can inspect its declared branches, while branch coverage can show which paths the tests executed. Such checks do not guarantee good tests, but they reveal much of the router’s structure.

Mocking the LLM is right for unit-testing the surrounding workflow. Force each label, then check dispatch, schemas, hand-offs, and fallbacks. Those tests remove the classification behaviour itself. They cannot show how the deployed LLM maps complaints to routes or whether it repeats the choice.

LLM-based routing adds another test layer. Source code can declare six allowed routes and what follows each one, but it cannot establish how a hosted model maps a complaint to a route. Static analysis still helps with orchestration, schemas, and permissions. Runtime model choice remains outside source analysis.

So the blind spot in my six tests is invisible to the tooling that would catch it in ordinary code. Those cases were six differently worded duplicate-charge complaints, which is easy to overlook when a dashboard reports only a green aggregate.

Compare that set with a repaired one. Here, a baseline means a reviewed set of expected routes that future versions should preserve.

The narrow run says what the 6/6 score omits:

NOT TRUSTWORTHY - the agent answered 'duplicate_charge' on 100% of the
probes, so a pass says more about the probe set than about the agent.

The regression trap

Saving a baseline turns today’s output into tomorrow’s regression test, so it deserves a higher bar than one passing run.

Freezing the narrow set creates a blind spot. A prompt refactor or model upgrade could make the router send unrelated disputes to duplicate_charge. Those six duplicate-charge cases would still pass because they never test the other routes. Standard assertions correctly check the cases they contain. Missing cases remain invisible.

Better tests remain the first repair. Add reviewed cases for every important route and boundary. AgentVerity does not generate those cases or replace branch coverage. It makes one-path evidence visible before a team promotes it into a router-wide baseline. Declared decision contracts can now name the required routes and block that promotion when one is missing.

Why AgentVerity

That gap suggested a small, enforceable rule: do not freeze a baseline until the evidence supports it. Three questions follow:

• Decision coverage: do deliberately varied cases reach more than one route?
• Decision stability: does the same case reach the same route across isolated reruns?
• Declared coverage: did the suite intend to reach every required route, did the agent return them, and did any call return an unknown route?

Without a contract, coverage remains a minimum dynamic check for a suite that collapses onto one route. With one, the report separates routes intended by the test designer from routes observed at runtime. Per-case correctness and important boundaries remain outside that result. Labelled cases and quality evaluators carry that job.

Repeat counts cannot be picked by habit. Provider controls can reduce model variation without guaranteeing identical results. OpenAI’s reproducibility guidance, for example, describes seeded output consistency as best-effort.

To test the rule without model variation, I used a Python router with no randomness at all. Six cases at 12 repeats produced 36 non-overlapping comparisons, and not one of them changed. A small yes-or-no helper still returned NOT STABLE. The helper was not broken. It had no way to separate a result proven stable from one it could not resolve, and 36 unchanged comparisons only bound the change rate at 9.6%, above my 5% limit. That run was inconclusive rather than unstable. Reaching 5% with no observed changes required 73 comparisons.

AgentVerity turns that calculation into a three-outcome rule. It forms non-overlapping rerun pairs, then places a 95% Wilson interval around the observed change rate. An upper bound below the tolerated rate means stable enough. A lower bound above it means variable. Anything between becomes not enough evidence. Wilson scoring is established statistics. The design choice is to calculate the call budget before execution and refuse a binary release claim when the evidence cannot support one. The technical note contains the executable arithmetic.

An evidence gate applies the higher bar. It refuses to save the baseline unless every call completed, the route was stable enough, the test set reached more than one route, any declared contract was satisfied, and a person approved the expected routes.

Deployment and proof

AgentVerity is an evaluation runner, not middleware. Run it during development, on a pull request, before release, or as a scheduled synthetic canary using controlled test requests. Do not repeat live customer requests. Results can feed the terminal, JSON or JUnit XML, or an existing OpenTelemetry pipeline.

Local execution demonstrates the idea without cloud setup. I also ran the same six-route task as a Strands LLM agent using Amazon Nova Micro on Bedrock AgentCore Runtime in London. DeepEval checked route correctness, AgentVerity checked stability and coverage, and CloudWatch recorded runtime health.

The first live run caught a mistake in my own release workflow. AgentVerity found stable decisions across several routes, while DeepEval found one wrong route and scored quality at 5/6. Both tools were right. I had combined them wrongly, allowing stable evidence to proceed towards a baseline even when a reviewed answer was incorrect. A reliably wrong route is still wrong. Now the public example stops on failed quality before paying for repeated calls or saving a baseline.

After the fix, the canary recorded:

• 6/6 reviewed routes correct
• no route changes across 36 non-overlapping rerun comparisons, which bounds the change rate at 9.6%
• all 6 routes reached
• 78 successful calls, with no errors or throttles

After declared contracts were added, a second London run went directly through Bedrock. That run again scored 6/6 with no changes across 36 pairs, while the contract reported all six required routes intended and observed with no unknown decision. Both redacted results remain separate.

Two more lessons came out of the cloud run.

Isolation costs what it buys. Every trial created and stopped its own session, because trials sharing a session are not independent and the statistics assume they are. End-to-end p50 went from 0.498 seconds locally to 5.869 seconds through AgentCore, while AgentCore’s own runtime stayed at 0.576 seconds. Session setup accounts for that gap, not the model. So this belongs in a release gate or a nightly canary, never a per-commit hook.

My own numbers needed the same scepticism. Zero changes across 36 pairs sounds settled. That evidence supports a 95% upper bound of 9.6%, which clears a 10% tolerance and proves nothing about 5%. Quoting the bare count would be the reading this whole article argues against, so the canary above quotes the bound instead.

Where it adds value

This check sits beside broader evaluation rather than replacing it. It adds one gate before quality and process results become a baseline, and refuses that promotion when any part of the evidence cannot support it.

Green badges should tell you what passed. Before a passing run becomes evidence for the next release, it should also tell you what actually ran and whether the result would repeat.

Find AgentVerity on GitHub and on PyPI under the Apache 2.0 licence.