Why Your AI Agent Observability Is Lying to You

89% of teams have monitoring but only 52% have evaluation. Here’s how to build real testing frameworks.

Photo by Nima van Ghavim on Unsplash

You have dashboards. Latency charts. Token usage graphs. Error rate monitors. Within seconds you can tell if the agent is up or down.

You cannot tell if it is right or wrong.

This is not unusual. According to LangChain’s State of AI Agents survey of over 1,300 practitioners, 89% of teams running AI agents have observability tooling. Only 52% have evaluation frameworks. That 37-point gap explains why teams say: “After the last model upgrade, the agent feels worse, but we can’t prove it.”

Observability shows you what happened. It does not show you whether what happened was correct. An agent that took ten tool calls instead of three still looks fine in your dashboards if all the calls succeeded. The output looked plausible. The error rate stayed low. But the trajectory was fragile. Next model update or next week’s data might break it.

You cannot improve what you cannot measure. Uptime dashboards do not measure agent quality.

Why Standard Testing Breaks with AI Agents

Software testing assumes determinism. Call a function with input X, you get output Y every time. If you get Z, fail the test. Rerun it, get the same result.

Traditional ML evaluation assumes fixed pairs. Feed a model test data, compare predictions to ground truth, compute accuracy. Same input produces same output.

AI agents break both assumptions at the same time.

An agent asked to “find the best restaurant nearby” might call search first, then reviews, then maps. Or maps first, then reviews, then search. Both paths can work. Both can produce different correct results. Same input, different execution paths, different valid outputs. You cannot test this with assertions.

A multi-step workflow is not atomic. It is a chain of five to twenty decisions, each contingent on the previous result. Testing the final output tells you if the chain worked. It does not tell you where it broke. If an agent calls the wrong tool at step three, the remaining steps might still produce something that looks plausible. The output is wrong but it does not look wrong.

When an agent calls an API, it triggers side effects. Sending an email. Creating a record. Initiating a payment. You cannot rerun these tests freely. You need mocking, sandboxing, or idempotency design. Your tests verify not just that the agent called the right tool, but that it called it with the right parameters.

Three Evaluation Layers Every Agent Needs

A single pass or fail grade on the output is not enough. You need evaluation at three layers. Each catches different classes of failure.

Unit-level tool call testing checks whether the agent called the right tool with correct parameters and handled the response. This is the one layer where deterministic testing works reliably. Tool calls have defined schemas. You can assert exact values, types, and fields.

Did the agent call get_order_status with the right order ID? Did it pass the value as an integer or string when the API expects an integer? When the tool returned an error, did the agent retry, try an alternative, or escalate? Mock the tools. Feed the agent a known input. Capture the tool call. Assert against expected parameters. This is fast and cheap. Coverage target: five test cases per tool (three happy paths, one error case, one edge case).

Trajectory-level workflow testing asks whether the agent took a reasonable path. Not “did it get the right answer” but “did it get there sensibly?”

Multiple paths can be correct. The agent that calls search-then-filter is as correct as the agent that calls filter-then-search if both reach the right answer. Your evaluation must allow for valid diversity. Did the agent solve the task in a reasonable number of steps? Did it call tools in logical order? When a step failed, did it recover gracefully? Did it avoid calling irrelevant tools? Capture the full trace and evaluate it against trajectory criteria. Some criteria are deterministic (step count thresholds, prohibited sequences). Others require semantic judgment.

An agent can produce the correct output through a fragile path. It got lucky. Trajectory evals catch fragile success before it becomes a production failure.

Outcome-level end-to-end grading asks whether the agent achieved the goal. Not “did it produce text that looks correct” but “did the outcome match the success criteria?”

This is hardest because outcomes are often semantic. A correct response to “What is our return policy?” can be worded many ways. You cannot use string matching. Use semantic evaluation. Is the information factually accurate? Did the output address all parts of the question? If the agent was supposed to create a ticket or send an email, did it complete correctly? Did the output avoid harmful or misleading content? Grade with an LLM-as-judge. A separate model evaluates the agent’s output against success criteria. The judge receives the original input, the expected outcome, and the agent’s actual output, then grades it on defined rubrics.

Critical detail: use a different model family for judging than the one powering your agent. If your agent runs on Claude, use GPT-4o as the judge. If your agent runs on GPT-4o, use Claude as the judge. Same-model judging introduces systematic bias. The judge shares the agent’s biases and will rate its mistakes as acceptable.

Starting With Real Failures, Not Synthetic Tasks

Anthropic recommends building task datasets with 20 to 50 cases from real production failures. Not synthetic examples. Not benchmarks. Real cases where the agent got it wrong.

Go through your support tickets, escalation logs, and user complaints. Find where the agent failed. Write each as a test case: the exact user input that caused failure, any relevant context (user profile, account data, conversation history), the semantic outcome it should have produced, and the root cause of the failure.

These 20 to 50 failure cases will catch more real problems than 500 synthetic cases. Failures cluster around patterns. Fix the pattern and you fix a category of problems.

Expand over time. Week one: 20 to 50 tasks from real failures. Month one: 100 tasks (add happy paths, edge cases). Month three: 200 to 300 tasks (add adversarial inputs, regression cases). Ongoing: add every new production failure as a test case.

Standard benchmarks like GAIA have a top score of just 61% on difficult tasks. That means top-ranked agents fail 39% of the time. High benchmark scores do not correlate reliably with production success. Your eval suite must track real tasks your users actually give the agent, not academic benchmarks.

Building an Eval Pipeline in Two Weeks

Most teams delay eval infrastructure because it feels expensive. It does not have to be.

Week one: foundation. Days one and two, collect failures. Go through escalation logs and support tickets from the last 30 to 60 days. Pull 20 to 50 cases where the agent got it wrong. Write each as structured test: input, context, expected outcome, failure reason. Store as YAML or JSON.

Days three and four, build the tap. A minimal tap needs a runner that executes your agent against a task and captures the trace, a mock layer for tools with side effects, and a results file storing results and traces. This is 200 to 400 lines of code from scratch. Platforms like Braintrust, Langfuse, or Galileo provide pre-built taps.

Day five, baseline. Run all 20 to 50 tasks and grade them manually. You need ground truth. Record your current pass rate. Every improvement is measured against this.

Week two: automation. Days six and seven, build LLM-as-judge grading. Create a grading prompt that evaluates agent outputs against expected outcomes. Include rubrics: what makes a pass, what makes a fail. Test the judge against your manual grades. If agreement is below 85%, refine until reaching 90% or higher.

Days eight and nine, add happy-path test cases. Reach 50 to 75 tasks total. Set up scheduled runs: daily for production agents, on every PR for agents under development.

Day ten, wire into CI/CD. Define quality gates: if outcome accuracy drops below 85 to 90%, block the deployment. These gates prevent regressions from reaching production.

After two weeks you have a failure-derived dataset of 50 to 75 tasks, an automated tap that captures traces, LLM-as-judge grading that correlates 90% or higher with human judgment, and CI/CD gates blocking regressions. This is not complete. It is a working foundation that catches failures users already reported.

Model Swap Regression Testing

“After the last model upgrade, the agent feels worse, but we can’t prove it.”

Model providers ship updates continuously. GPT-4o-2024–08 behaves differently from GPT-4o-2025–01. Claude 3.5 Sonnet routes tool calls differently than Claude 4. Without regression testing, model upgrades are a coin flip.

Baseline your current model against your eval suite. Record scores at all three layers. Pin the new model version and run the same suite. Compare layer by layer. An outcome accuracy drop over two percentage points fails the upgrade. Trajectory efficiency degradation (agents taking 30% more steps for the same tasks) also fails it.

For tasks that changed from pass to fail, review traces manually. New models sometimes fix one category of failures while introducing another. Net score may stay similar but failure distribution shifts. Do not swap models globally. Route 10% of traffic to the new model. Monitor for 48 hours. Expand to 50%, then 100%.

A smaller, cheaper model that scores 91% on your eval suite might be better than a larger model scoring 95% but costing four times more per call. Your eval suite gives you data for this decision. Without it, you are guessing.

Common Mistakes That Kill Eval Programs

Testing with synthetic data only. Synthetic tasks test what you imagine going wrong. Real failures test what actually goes wrong. Overlap is smaller than you think.

Grading outputs, not trajectories. An agent producing the right answer through fragile paths fails unpredictably. Grade the path, not just the destination.

Running evals manually. Manual eval runs happen once, get discussed in a meeting, and disappear. Automate from day one.

Skipping the baseline. Without baseline data, every eval run is isolated. You cannot detect regressions without knowing where you started.

Using the same model as judge and agent. The judge will rate the agent’s mistakes as acceptable because they share the same biases.

The evaluation gap explains why 32% of organizations cite quality as their top barrier to production agents. Teams investing in evaluation catch regressions before users do. They make model upgrade decisions with data instead of intuition. They build agents that improve systematically over time.

Your agent testing foundation comes down to three layers, real production failures, and LLM-as-judge grading. Build it in two weeks. Expand it continuously. Every production failure becomes a test that prevents recurrence. Teams deploying agents without eval frameworks pay the cost in time spent investigating why the agent did something after each model update. When you build evaluation as a core deliverable, not an afterthought, you have confidence in what your dashboards cannot show you.

Originally published at https://www.raftlabs.com/blog/ai-agent-testing-evaluation-guide