Your Agent’s Eval Should Tell You Where It Broke

Why a single pass/fail score hides your agent’s real problem, and how to write outcome and process checks that point straight at it. With runnable Agent SDK code.

Your eval goes red. Yesterday it passed, today it fails, and all you have is a number that dropped. Something regressed. But what? So you start reading transcripts, guessing, opening files at random.

That hunt is avoidable, and the fix is a decision you make before you write a single check: what to measure. Get it right and a failing eval will not just tell you the agent got worse. It will tell you exactly where to look.

By the end of this article you will have a runnable eval, built on the Claude Agent SDK (Anthropic’s Python toolkit for building tool-using agents), that scores an agent on five separate checks and turns any failure into a pointer.

One number is a check-engine light

Think about the most useless dashboard warning there is: a single light that says “something is wrong.” It is technically true and tells you nothing. You still have to pop the hood and start guessing.

A single pass/fail score for your agent is that light. “Good answer or not” tells you the run was bad without telling you which part was bad. What you want instead is the diagnostic readout, the one that says “cylinder three is misfiring,” so you know exactly where to put the wrench.

To get that readout, you have to measure more than the output. An agent gives you three different things worth scoring:

• Outcome: what it produced at the end. Did the final answer actually help?
• Process: how it got there. Which tools it called, in what order, how many turns it took.
• Cost: what it spent. Turns, tokens, latency, money.

The output is only the first one. The other two are where most agent failures quietly hide, and a single score throws them both away.

The agent we will measure

Let us make this concrete with a real agent you can run. It is a support agent for an equipment dealer, and it has exactly two tools: search_manual, which looks up guidance in a small knowledge base, and lookup_part, which confirms a part number. Its system prompt tells it to search the manual first and to verify any part before recommending it.

AGENT_OPTIONS = ClaudeAgentOptions(
    system_prompt="You are a support agent. Use search_manual to find "
                  "guidance and lookup_part to confirm parts before recommending.",
    allowed_tools=["mcp__support__search_manual", "mcp__support__lookup_part"],
    max_turns=6,   # a turn cap so a confused run cannot spin forever
)

That is the whole agent: two tools, a system prompt, and a turn cap. The mcp__support__ prefix on the tool names is just how the Agent SDK namespaces in-process tools. Every eval we write points at this same AGENT_OPTIONS, so the thing under test never drifts between runs.

One run, both axes

Here is the small piece of plumbing that makes the rest possible. We run the agent once and pull out both the outcome and the process in a single pass.

async def run_agent(prompt, options):
    tool_calls = []
    async for msg in query(prompt=prompt, options=options):
        if isinstance(msg, AssistantMessage):
            for block in msg.content:
                if isinstance(block, ToolUseBlock):
                    tool_calls.append(block.name)   # the process
        elif isinstance(msg, ResultMessage):
            final = msg                              # the outcome and cost
    return final, tool_calls

Notice what comes back. final carries the outcome on final.result, plus the cost signals: final.num_turns, final.total_cost_usd, and final.duration_ms. The tool_calls list is the process, the exact sequence of tools the agent reached for. One run, both axes, no extra work. Capturing them together matters more than it looks: the agent is nondeterministic, so the outcome and the process have to come from the same run to be comparable. Score them from two separate runs and you are quietly judging two different agents.

The eval that points at the failure

Now the payoff. We score one real case on five checks at once, and we group them on purpose.

final, tool_calls = await run_agent(
    "The hydraulic pump on a 4830 keeps tripping the overload. What should I check?",
    AGENT_OPTIONS,
)
answer = (final.result or "").lower()

checks = {
    "mentions_strainer":  "strainer" in answer,                              # outcome
    "cites_correct_part": "7y-2210" in answer,                               # outcome
    "searched_first":     tool_calls[:1] == ["mcp__support__search_manual"], # process
    "verified_part":      "mcp__support__lookup_part" in tool_calls,         # process
    "stayed_efficient":   final.num_turns <= 4,                              # cost
}

Read the grouping, because the grouping is the whole lesson. The first two checks are outcome: did the answer name the right component (a strainer) and the right part number (7Y-2210)? The next two are process: did the agent search before it spoke, and did it verify the part instead of inventing one? The last is cost: did it stay efficient? Five signals, three categories, one run.

You could have collapsed all of that into one boolean, “good answer or not.” Resist that. The split is what turns a failure into a pointer.

Why the split pays off

Picture this eval failing tomorrow on just one check.

Say cites_correct_part fails and everything else passes. You instantly know the agent searched and verified correctly but lost the part number somewhere between retrieval and the final write-up. That is a summarization problem, not a tool problem. The separated checks pointed straight at the file to open.

Now say searched_first fails while both outcome checks pass. The agent answered correctly without ever consulting the manual, which means it leaned on the model's own memory instead of your knowledge base. That run looks fine today, and it will quietly make things up the moment a question falls outside what the model already knows. A single pass/fail score would have hidden that landmine completely.

So here is the rule worth designing around: write your checks so a failure tells you where to look. A score that only says “worse” sends you hunting. A score that says “process is fine, outcome regressed on part numbers” hands you the file.

Testing one part in isolation

Everything above runs the whole agent end to end. There is one more move worth knowing: when a single component is noisy, test it alone. For our agent, that part is retrieval. You can call search_manual directly with twenty real queries and check whether the right manual entry comes back, with no model in the loop at all. If retrieval is broken, no amount of agent-level testing will explain why, and tuning the prompt will not save you. Isolate the noisy part when the end-to-end results get hard to read.

Go try this

You now have a runnable agent, a harness that captures outcome and process in one pass, and an eval that scores both and tells you where a failure lives. That is further than most agent projects ever get.

Do not rush to fifty checks. Pick the two or three failures you would be most embarrassed to ship, write one check for each, and run them on every change, so a regression (something that used to work quietly breaking) shows up the moment it happens. Once your checks point at the failure instead of just flagging it, the next question is whether you are running them on the right cases, and that is where you go next.