Testing LLM tools — consistency runs and evaluation

Mirror, mirror… are you sure this time?

LLM responses are non-deterministic — even with identical input and a well-written prompt, the model will occasionally produce a different answer. This means a test that passes once proves very little. To confidently say a tool works, you need to run each case multiple times and track how often it produces the correct result.

Two types of tests

A smoke test is a single run per test case. It is fast and cheap. Use it to verify that your approach is basically correct — that the prompt reaches the model, the output is in the expected format, and nothing is obviously broken. A smoke test will not catch low-frequency errors or instability.

A consistency run sends the same input multiple times and measures how often the model gets it right. This is what you need to confidently deploy a tool. The standard approach is ten runs per test case. Ten runs are enough for failures to surface — a case that passes eight out of ten times is telling you something important. You do not need to target 100% accuracy, but with good prompts and well-structured documentation you should consistently get close to it.

Scenario one — single input, single expected output

The simplest pattern: you send a specific input with a prompt and a reference document, and you expect a specific answer. Build a sample of test cases — typically around a hundred — and run each one ten times. With a hundred cases that is one thousand total calls.

A practical example, drawn from a tool I was testing, is a Jira ticket-routing tool: it took an incoming support ticket and routed it to the correct team. You collect a set of real tickets with known correct destinations, send each ticket to the model ten times alongside the routing-rules document, and measure how consistently it routes to the right team.

At this scale prompt caching makes the cost manageable. The routing-rules document stays fixed across all calls — once it is cached, subsequent calls pay a fraction of the normal input price.

Scenario two — free-text output

When a tool returns a free-text answer rather than a single value, you cannot compare results with exact string matching. Two correct answers will often be worded completely differently. This applies to any tool that produces natural-language output — vector database tools are the most common case.

The approach here involves two calls per test run:

First, build your set of test questions in natural, human-friendly language. For each question, generate an expected answer by having Claude work through the documentation. If you are not an expert in the topic, have a domain expert review and validate the expected answers — the expert defines what information is mandatory in a correct answer.

The second call is a judge call — using the model to score the tool’s output. You send both the expected answer and the real answer to the model and ask it to do a content comparison: does the real answer contain all the information that the expected answer marks as mandatory, regardless of how it is worded?

For the judge call, Haiku is usually sufficient when the comparison is straightforward. For complex domain-specific answers where nuance matters, Sonnet may be needed. Opus is not practical for judge calls in most cases.

Scenario three — multi-turn dialogue

Some tools conduct a conversation. After receiving the initial input, the tool assesses the quality of the information provided and asks follow-up questions before producing a final result. The number of questions and their order are not predictable.

Running a consistency test on a dialogue tool without a live user requires a different setup. Each test case consists of two parts: the original input data, and a hidden knowledge bank — a set of facts that would be available to a real user.

The test runner works as follows: when the tool asks a follow-up question and presents response options, a cheap Haiku call is made — it receives the question, the suggested options, and the knowledge bank, and identifies which option best matches the facts. The test runner sends that answer back to the tool. This loop repeats until the dialogue ends.

Because Haiku evaluates each question independently against the knowledge bank, the test is completely order-agnostic and wording-agnostic.

Once the dialogue concludes, the final output is evaluated using the same content-comparison judge call as scenario two.

Reporting

Good reporting is as important as the tests themselves. Raw data storage should be built into the test runner so it saves automatically. Raw data means everything: all fields in every request you send to the model, and the full uncut response you receive back. If a failure appears and you do not have the complete raw data, you will need to rerun the entire test — which defeats the purpose.

For the report itself, Confluence or an HTML file both work well. The report structure should include at minimum: what you sent, what you expected, what you actually received, the consistency percentage per test case, and a root cause analysis column for failures.

Monitoring token usage

Consistency runs are also a useful tool for predicting and controlling production costs. While running tests, collect token usage data alongside results. The metrics that matter most: number of calls made, cost per call, and cache hit percentage.

Cache hit percentage is particularly important. If it is unexpectedly low during a consistency run, that is a signal that your caching strategy is not behaving as expected — worth catching before you scale to production.

Further reading

Part of a series on working with AI as a non-developer:

• From skeptic to builder — a product owner’s story
• The quality gate — trusting code you don’t read
• Running the team while you’re away
• Running analytical work with AI
• How a non-developer runs an AI dev team
• How an AI project remembers things