Building an eval harness for AI voice agents

Lessons learned from prototype to production

As part of a co-development project with Amadeus, a tech leader in the travel industry, we recently started rolling out an AI call center voice agent to production. Getting there required building an evaluation harness that let us iterate with confidence, fixing dozens of issues in weeks without breaking what already worked.

Good resources exist on *why* evaluations matter for agents and on general guidelines to follow. Anthropic’s Demystifying Evals for AI Agents covers the conceptual foundations. OpenAI’s Realtime Eval Guide walks through voice-specific challenges.

This article focuses on the how, drawing on lessons from a real project that weren’t obvious at the outset. This not only includes the evaluation system we built but, equally important, how to leverage it as an organization. While our system is voice-to-voice, the ideas apply broadly to conversational agents — including text or voice LLM agents that use tools to read from or write to external systems.

Setting the stage

For illustrative purposes, this article frames our experience around an order management hotline, with an autonomous AI voice agent that customers can call to either modify or cancel orders. Behind the agent sits a database for the orders, and each order has a status, items, shipping address, and refund eligibility.

Primary success criteria for the agent is that it should be able to:

• Understand customer intent and verify it can help.
• Gather required information from the customer (e.g., name, order number).
• Call the right tools in the right order with the right arguments.
• Adhere to company dialogue policy (e.g., greeting, flow, escalation scripts).
• Provide a pleasant, useful experience.
• Escalate to human agent appropriately when it cannot help.

Two challenges make this difficult:

1. Everything is coupled. The only levers we have to improve the agent are free-form text: the prompt and the tools with their names, argument names, and descriptions. Any modification to fix an issue can easily break something else. It is impossible, by design, to add new isolated functionality. How do we know we’re moving forward and not playing whack-a-mole?
2. The agent has write access. This is not a read-only assistant. It can modify orders, process refunds, and cancel items. If the agent is not reliable, it does not just annoy customers, it can wreak havoc on the business. How do we build confidence that this agent won’t go rogue once deployed?

The answer to both is the same: An evaluation harness that catches regressions before we ship and proves reliability before we deploy. The rest of this article is about how we built and used ours.

Our early bets

Early on we decided to make two architectural decisions that would be expensive to change down the line, to use simulation only and to have real test systems (no mocks).

Simulation only

Before we could build anything, we had to decide what to test.

The field seems to have settled on three levels for evaluating conversational agents: single-turn (one message in, one out), replay (populate conversation history up to turn N–1, test the next response), and full simulation (use a second LLM to act as a user through a complete conversation). Many resources advocate for layering all three.

After a few spikes and learnings from colleagues, our conclusion was simple: Ignore single-turn and replay, focus on simulation only. These were our reasons:

• Context drift. Consider that a recorded conversation was generated with a specific system context. After changing the prompt or tools, that conversation state may no longer be reachable. We’d be testing a scenario that can’t happen anymore.
• The interesting turns are deep. Most conversations for these agents start the same way (e.g., Hello! Can you please provide your order number?). The complexity comes from turns answered deep into the conversation. Single-turn testing does not reach where it matters.
• Outcomes matter, not exact paths. We care whether the customer’s order got cancelled, not the exact wording or sequence of turns to get there. Replay forces you to lock in exact paths, creating brittle tests and combinatorial explosion.

We weren’t sure simulated users would capture enough nuance long-term, particularly with audio. But it simplified everything: one testing paradigm, one set of tooling, one mental model.

Real test systems, no mocks

Agents interact with external systems through tools. A natural instinct is to mock those tools for testing: control the responses, avoid setup complexity, move fast. For simple agents with minimal tooling, this can work fine.

But an agent like our example usually sits on top of an existing system with real complexity: order states, eligibility rules, refund logic, error conditions. Early spikes showed us where this was heading: mocking meant reimplementing that complexity. Every scenario needed its own mock behavior. Every tool change (e.g., rename a function, adjust a parameter, tweak a return type) meant updating mocks across many scenarios.

Worse, mocking hides one of the agent’s most valuable capabilities: self-healing. Real systems return meaningful errors. An agent can call a tool incorrectly, receive an error message, and course correct. With mocks, you either reimplement that error logic or lose visibility into how the agent recovers.

Therefore, we landed on using a real test system, not mocks:

• Tool iteration freedom. We wanted to experiment with tool design — names, arguments, granularity — without rewriting test infrastructure each time.
• Self-healing visibility. Real systems give real errors. We could observe and rely on the agent’s ability to recover from mistakes.
• Complexity scales with the system, not your mocks. The system already encodes its own logic. Why duplicate it?

This came with real costs. Setting up a test environment is not free. Each scenario needs proper initialization and you need isolation so scenarios don’t collide (e.g., reusing order IDs or modifying shared records). We invested in setup and teardown logic per scenario.

The upfront investment was significant, but we believed the ongoing cost of mock maintenance would exceed the one-time cost of proper infrastructure.

The testing unit: Scenarios

At this point, we knew we wanted to test only through simulated conversations. The testing unit concept we landed on was the scenario. A scenario encodes:

• Who the user is (persona, information they have, emotional state).
• What they’re trying to accomplish (intent, goal, behavior).
• Where they’re starting from (system state).
• How success is measured (outcomes, side-effects, style).

While equivalent, we found that, in this domain, scenario better conveys its meaning for both product and technical people than Anthropic’s generic “task.”

We found that encoding scenarios as YAML files was useful. They display high content density and are very human readable which makes them easier to discuss with the broader team. An example of what they might look like is below.

name: "refund_full_order"
description: "Refund a full order that is eligible for it"
system_spec:
  first_name: "Alex"
  last_name: "Doe"
  order_id: 123
  order_items:
    - "product"
  refundable: true
user_spec:
  voice: "marin"
  info: "Your name is Alex Doe."
  goal: "You want to cancel and get your order refunded."
eval_spec:
  outcome_spec:
    order_id: 123
    refunded: true
  expected_exit: "user_ended"
  judges:
  - name: "goal_achieved"
    context: "Agent refunded order and communicated it to user"
  - name: "stays_on_flow"

We initially aimed for testing all situations we could think of. Most didn’t surface new failures because they were permutations of the same flows. We settled on a few dozen scenarios, aiming for orthogonality: Each scenario should test something meaningfully different. For voice, where each simulation runs in real time and costs money, fewer well-designed scenarios beat a sprawling test suite.

Due to the nature of scenarios, it is easy to end up repeating data (e.g., the name of the simulated user is used to create a record in the system and to tell the user what their name is). To ease maintainability and avoid inconsistencies, we used Jinja templates and rendering functionality to go from scenario.template.yaml to scenario.yaml which is what is read by the simulation engine.

The evaluation engine

The scenario becomes the specification for the simulation it should run. The following diagram outlines what constitutes a scenario evaluation and the subsections below elaborate on each.

Execution flow for scenario simulation

Initialize the system

Each scenario specifies the system state it needs. Before simulating the conversation, we set up the system accordingly. This step is highly dependent on use case. In the system_spec example above, this step would create a refundable order for Alex Doe.

This already illustrates the benefits of using a real system. While initialization logic may be heavy, it is centralized and the tools will work as is once the system is properly populated without extra work.

There are two key considerations:

• Isolation. Multiple runs should not collide with each other, whether the same scenario running twice or different scenarios in parallel. That is, data modified by one run should not affect another run. Cleanup (discussed below) is the other half of this.
• Parity with production. The test system should behave like production. If it diverges, you’re testing something that doesn’t exist.

Initializing the simulated user

The simulated user is simply another real-time voice agent. Its only tool is end_call, which it uses to terminate the conversation, so nearly all of its behavior is defined by its prompt. We started with specifying full user prompts in each scenario until we realized the commonalities. Then, we extracted the prompt into a template that contains the following parts:

1. Base instructions reused across all users, which simply contain the basic instructions on how to be a user in this system.
2. Information the user has and can volunteer: their name, their order number, and so on.
3. Goal the user aims to fulfill with this conversation.
4. Extra instructions placeholder. We used this for some more complex scenarios where we wanted the user to follow more exotic behavior such as changing information mid-conversation.

In our experience, simple simulated users created from rather minimal instructions were enough. An illustrative example for the rendered prompt:

You are a regular person calling the customer service center.
Always start your first answer with "Hello! I would like to…".
You are provided two things:

1. Information you have that you may disclose upon request by the agent
2. Your goal for this conversation.

<information>
Your name is Alex Doe
</information>

<goal>
You want to cancel and get your order refunded.
</goal>

Once your goal is achieved or you are sure it cannot be achieved, thank the agent and end the call.

Initializing the agent

No special magic here. This should create an instance of the agent we aim to evaluate that matches exactly what is used in production: same code, same prompt, same tools. The only difference should be the system to which the tools have access (dev instead of prod).

Simulating the conversation

The engine triggers an initial response from the agent, which typically starts with a greeting and a request for information (e.g., What do you need help with?). From there, we let the conversation unfold. For real-time audio, we do not explicitly control turn-taking; the models handle it themselves via built-in voice activity detection. The conversation continues until one of three exit criteria is met:

1. Escalate to a human agent. Triggered by the agent (and usually confirmed by the user) when it cannot fulfill the request. This is represented by a tool call. The specific details on what happens afterward are heavily dependent on the use case.
2. User ends the conversation. When its goal is fulfilled or unfulfillable, as a human would. This is represented by the single tool the user has access to: end_call.
3. A limit is reached. These models can be expensive on long conversations (or context can be exhausted), so we need guardrails to ensure things do not get out of hand. While the limit is an exit criterion, we displayed it as an escalation to the user: “I am sorry but we have reached our conversation limit, a human agent will help you shortly.”

Regarding the third point, we identified three limits:

• Time limit. Triggered when time since first message exceeds a threshold. This proved to be important because if one agent crashed or went silent, the other would keep waiting and the simulation never ended.
• Turn limit. Triggered when a certain number of turns are reached.
• Token limit. Triggered when total tokens consumed by the conversation reach a predefined limit. We derived this limit from analyzing many conversations and finding clear patterns on the token budget required to fulfill most goals. We found this to be the most meaningful, as it represents the information content of the conversation. The assumption is that if the agent goes beyond this limit, it is probably too confused to end up succeeding. Additionally, this makes it much easier to anticipate costs.

Treaceability

It is critical for this process to support full traceability. We stored the following for every single run:

• Traces from both user and agent point of view: messages received, messages sent, tool calls, and tool results. Timestamps are important to reconstruct the conversation in the correct order, especially for audio. Without them, debugging turn-ordering issues in async voice pipelines is guesswork.
• Recordings of all audio. Being able to listen to the conversations reveals issues that traces alone cannot: unnatural pauses, talking over each other, tone problems. This proved indispensable for voice.
• Logs. Invest in quality logging and centralize all logs for a conversation in a single file that can be associated with the run. This helps in understanding weird behavior. For instance, an agent was going silent mid-simulation on occasion. Thanks to the logs we discovered a content filter was being hit in specific situations. After that, it was easy to account for and build resilience around it.

On simulating voice

The simulation of conversations with audio models is more complex than with text. One key challenge is that we cannot easily speed up conversations. Built-in turn detection in models like gpt-realtimeexpects audio at regular speed, and it is unclear if accelerated audio would produce representative results. We decided not to attempt it and settled for running many conversations in parallel. In practice, quota easily becomes the bottleneck: these models are token-hungry and expensive to run, which means quotas tend to be restrictive — particularly for non-production payloads. In our internal test setup, a typical run (approximately 40 scenarios, three trials each) took roughly two hours and cost around $60, though this varies significantly by configuration and model.

A couple of technical details that enabled voice simulation:

Both agent and user share an asyncio.Eventpassed to escalateand end_calltools at initialization time. The conversation loops monitor this event and when it is set by either tool, it gracefully terminates the process of both agents.

Audio bridge. gpt-realtimeexpects audio at regular speed but returns audio bytes much faster. To create meaningful interactions between two models, we built an audio bridge that collects audio bytes as they come but feeds them to the other model at a normal rate. It also supported flushing of the queued bytes to handle possible interruptions that could arise from silences in the produced audio.

Evaluating the results

Once the simulation is finished, we evaluate the outputs to produce performance metrics. In our case, we focused the automated evaluation on traces (text) only, which was enough. We had four different evaluation levels:

• Outcome-based evaluation. Does the state of the system at the end match the desired one? For example: Is the order refunded?
• Exit-based evaluation. Did the conversation end for the reason we expect? This validates we escalate when we have to or that user exits as expected. It also detects quickly if we hit a limit.
• Performance-based evaluation. Consumed tokens, total turns, tool calls, total conversation time, and so on. We did not set hard thresholds on these initially, but tracked them to detect regressions and inform optimization later.
• LLM-as-judge evaluation. This enables us to test things we cannot easily code. For example: Did the agent skip the confirmation step before processing the refund?

Designing judges

Using LLMs for evaluation is fiddly and can easily turn into a gun pointing to your foot. In our experience, the following design decisions greatly diminish the risk:

• Make judges binary. In our experience, multi-scale judges are too nuanced to be useful. You need very detailed rubrics and even different humans would struggle to provide the same scoring for ambiguous situations. Favor judges with only binary outcomes. Did it do X or didn’t it. This makes it much easier to reason about them and validate that they behave as expected.
• Make judges provide a reason for their verdict. This makes debugging significantly easier. In our experience, these reasons are quite good and help in understanding why the decision was made. It tells you what to look for on the traces to see what is going on or what is wrong in your judge prompt if the result is unexpected.
• Add judges reactively. Do not start writing the judges you think you will need. Develop judges as consequences of observed failure modes. For instance, we only added a flow judge when we realized in many cases our agent was not staying true to the flow on the script. This allowed us to measure the problem and solve it.

We developed two types of judges: judges that require ground truth and judges that do not. For instance, a user-behavior judge could serve to quickly detect if simulated users are behaving as they should and help in iteration. This judge, though, needs a ground truth of what is the expected behavior. For all of them, the conversation (including tool calls and their results) is formatted to text and passed to the judge as input.

Judges that do not require ground truth can be reused in production. Because they need only the conversation transcript, they can run as a post-process step on real conversations to provide the same insights. For instance, to validate whether the agent followed the flow defined by our policy. This, if properly monitored, can quickly help identify problematic conversations.

Who guards the guardians: Aligning LLM judges

LLMs as judges can be tremendously useful, but they suffer from the same problem as the agent we are evaluating: the only lever we have is its prompt. Thus, the metrics the judges output are going to guide us in defining whether or not some solution is good enough. To trust them, we need to evaluate them too.

The beauty of this is that because we create judges only reactively when we have observed failure modes already — from production, internal testing, or the simulations themselves — we have a good realistic dataset to start from. Beyond that, if you have some data, it is easy to create more synthetic data using an LLM manually that represents more cases (of course, human-check it!). The idea then is to build some infrastructure that makes it extremely easy to test them. For instance, for the flow judge mentioned, we had conversations that were good, and others that were not. Even 10 conversations could be a solid starting point. Then, you label them as True or False (what the judge should output) and evaluate them. It is important to have either balanced datasets — the same number of True and False — or use metrics that do not suffer if that is not the case. What provides the best picture is a confusion matrix. If we want to track improvement over time, the confusion matrix can be summarized as True Positive Rate (TP / (TP + FN)) and True Negative Rate (TN / (TN + FP)).

A nice benefit of setting this system in place is that every time we encounter an example of the judge flagging a false positive or a false negative, we can take the conversation transcript, add it to the judge eval set and iterate the prompt until all previous cases pass, and the new one too.

If you have enough data, you can split it into an eval set and a test set, one for iterating the prompt and the other to validate that it is generalizing. That being said, we decided not to do that. Initially we had limited data and we wanted to aim for perfect scoring. This, paired with the strategy of adding any failure example to the set and then diligently improving it worked well for us.

What about tool call evaluation?

It is tempting to evaluate whether the agent is calling the expected tools in the expected order. We tried that but we ended up dropping it. These are the problems that justified the decision:

• Brittle and enforcing the wrong patterns. A big win of having a good test suite, either for this kind of project or with regular software, is that you can refactor at peace. If your tests are heavily coupled to the implementation details, you keep having to drag test updates every time you want to change something.
• Argument management. If you change the name of the tool, the name of an argument, or the signature, you have to change all your checks. If arguments are not predefined and instead set as the result of previous tool calls (e.g., the ID of the options provided by a search is used for the confirmation), it can easily be a nightmare to implement logic that can detect maybe the type but not the value.
• Non-obvious and hard to reason about. There are too many degrees of freedom: tool name correctness, argument names, argument values, call order, extra calls. Agents are good at fixing their own errors, so extra wrong calls followed by corrections are not necessarily a problem. Defining what “correct” means here is surprisingly hard.

The immediate benefit after dropping tool call evaluation was a major refactor on the tools that we could run through the suite without having to rewrite all the expectations. More broadly, we realized it was making us miss the forest for the trees: We were spending a disproportionate amount of time keeping this evaluation updated while largely ignoring its results. The other evaluations were already catching the problems we cared about.

Cleaning up the system

At the end of a simulated conversation, teardown logic reverses whatever the conversation did: delete created orders, cancel transactions, restore modified records. The goal is maximizing isolation. Without it, state leaks across runs. For example, if your system has limited inventory and scenarios create orders that consume stock, after enough runs the system is exhausted and unrelated scenarios start failing for reasons that have nothing to do with the agent (been there done that). Granted, simulations running in parallel could consume that same stock but the magnitude makes it easier to manage.

Aggregating results

So far, we have defined the steps to evaluate a single conversation stemming from a single scenario. This is insufficient to evaluate the consistency of the results. Because of the probabilistic nature of both agents, two conversations from the same scenario are rarely exactly the same. A scenario that passes once might fail next. Thus, results need to be aggregated at two levels:

• Multiple trials of the same scenario. We run multiple trials (in our case, three) for the same scenario. Only if all trials pass do we consider the scenario successful. A failing trial means it is likely to fail in production.
• Metrics across all scenarios. While usually we work on one scenario at a time, we want to ensure no regressions on the rest. Thus, we are interested in global metrics aggregation across all scenarios in a run.

The diagram below illustrates the full aggregation pattern: individual metric results roll up into trial pass/fail, trials roll up into scenario pass/fail, and scenarios roll up into the overall run result.

Results aggregation flow

To know which runs are comparable, we versioned the dataset by hashing the contents of each scenario and then hashing the scenario hashes together for the full harness. If the harness hash matched between two runs, we knew they used the same scenarios and could be compared meaningfully.

We exported all metrics into a single run_evaluation.jsonwith a hierarchical structure: aggregated metrics at the run level (e.g., scenario pass rate), then for each scenario the aggregated metrics (e.g., all outcomes passed), and for each trial each individual metric with its result and, for judges, the reasoning. While manual inspection is not ideal in such an object, it is very easy to consume downstream by visualization tools, which we discuss later.

Critical versus non-critical metrics

For a trial to be considered successful, all its critical metrics have to pass. This benefits from most metrics being binary (as discussed above). A trial passes if and only if every critical metric passes.

We consider non-critical metrics those that could improve the user experience but the scenario could be solved even if they failed. For instance, the agent asking twice whether the user wants to escalate to a human. That is not ideal, but the user’s goal can still be fulfilled. Equally, performance metrics (tokens consumed, number of turns, conversation time) are largely considered non-critical but it is valuable to track and improve them. The classification of what is critical evolved over time as we better understood which failures truly blocked users and which were rough edges.

From engine to workflow: Eval-driven development

The evaluation engine on its own does not buy you much. Without organizational discipline around it, it is just infrastructure. The real value comes from the process that leverages it. This is where we landed:

Eval-driven development flow

1. A failure is detected. Something is not working as it should. Ideally these are identified by UX or product teams as they own the vision. These issues may come from previous simulation results, manual testing, red teaming, or observed from production.
2. A ticket is opened. Whatever ticket system the development team is already using suffices here. We found it valuable to have a simple template: expected behavior, observed behavior, and a link to the conversation trace or recording so anyone can inspect it directly. This is why observability is so important.
3. A ticket is prioritized. Once the ticket is prioritized it is eventually picked up by the owning development team.
4. Create a scenario. The idea is to create a scenario that systematically reproduces the issue so we can feed it into the evaluation engine.
5. Validate failure can be detected. When running this new scenario through the engine, something needs to fail. If nothing fails, we need a new grader which often is a new LLM judge. The judge is developed and iterated until we can indeed reproduce and detect the issue consistently. In a way, this is the equivalent of the Red from Red-Green-Refactor of Test-Driven Development. At this stage we have a failing test. For instance, when we discovered the agent was asking for escalation confirmation multiple times, we added a binary judge: Did the agent ask for confirmation only once?
6. Iterate the agent. This usually means iterate the prompt and/or the tools through their arguments and their descriptions. We do that until our failing test passes. Now we are in the green.
7. Validate nothing broke. We run the rest of the suite and validate that the rest of the scenarios continue passing.
8. Merge and close the ticket. As standard procedure, a PR had to have two linked runs: one that shows we detect the issue and another that validates the issue is solved. For this, of course, it is important to have a solid experimentation framework in place that allows traceability, reproducibility, and sharing of results.

If it ain’t broken, don’t fix it. From this process it derives that all changes must be justified. With these systems it is easy to end up with complex prompts and tool configurations. It is tempting to aim to simplify them or clean them. But it is important to resist the temptation. Precisely because the main levers for behaviors reside in the prompt and the context in general, any change in wording can have unexpected side effects. If your evaluation harness — which should have evolved from observed problems and desired behaviors — works, there is no reason to change it. Even if it looks ugly.

This is what makes the process eval-driven: we never changed the agent speculatively. Every modification originated from a failing evaluation that reproduced a real problem. The eval fails first, then we fix. Not the other way around.

Invest in visualization

I cannot stress this enough: Invest in custom visualization tools. The return on investment is simply huge. We built a webapp to load runs and dig into them. The following is a list of features that were valuable:

• See metrics at a glance.
• Display a matrix of success per trial and their metrics.
• Expose justification of judges.
• Read the conversation, including tool calls and their results.
• Listen to the recording.
• Annotate feedback per scenario.
• Explore logs for each conversation.
• Compare runs at the scenario level.

This was not a nice-to-have. Steps 5 through 7 of the process above require rapidly inspecting results, comparing runs, and spotting regressions. Without a good visualization layer, this becomes a bottleneck that slows down every iteration cycle. Shown below is a screenshot of a slice of what we built for this:

Scenario evaluation inspection tool

Conclusion

Setting up this evaluation system was a significant upfront investment. It took weeks of greenfield development before the flywheel was running smoothly. At the time, it was unclear if the bet would pay off or just become expensive shelfware.

It paid off. We solved dozens of issues in a few weeks and built enough confidence to put the agent in front of real users. By the time we got there, we had addressed most of what we could reasonably anticipate without seeing real interactions.

There are things we deliberately did not tackle: richer user personas (different personalities, voices, accents) and noisier simulations (background noise, poor connections). These decisions followed the same principle as everything else: Fix things after they’re proven to be a problem. We are still early in the production journey, but the goal was never (and could never) be to deploy a perfect solution on the first go. The idea was to build the machinery to measure what matters and iterate based on what we observe. As real conversations surface new failure modes, they feed back into the engine the same way everything else did.

These are open questions, and we’ll address them if and when the data tells us to. The system matters, but the process matters more. The discipline of not changing anything without a failing eval first is what kept us moving forward instead of in circles.

This reflects the experience of a single co‑development project between Microsoft and a customer and should not be interpreted as a general statement of readiness for similar systems.

Bernat Puig Camps is on LinkedIn.

References

• Demystifying evals for AI agents — Anthopic
• Realtime Eval Guide — OpenAI
• Hypothesis Driven Development for AI Products — Bernat Puig Camps
• Who watches the Watchers? (Deep dive on LLM judges) — James Asher
• Taming complexity: An intuitive evaluation framework for agentic chatbots in business-critical environments — Karol Zak & Marc Gomez
• LLM Evals: Everything you need to know — Hamel Husain & Shreya Shankar