Evaluating AI Agents: Test Decisions, Not Just Answers

Caravaggio, The Incredulity of Saint Thomas (c. 1601). Public domain. — Don’t trust the answer; inspect the evidence.

You built an agent. You asked it a question. It searched a few sources, wrote a confident, well-cited answer, and returned something that looked useful. Nice. It works. Or does it?

That’s the uncomfortable question hiding underneath nearly every successful agent demo. Most teams can build an agent that appears to work. Far fewer can prove it works consistently — on the next question, after the next prompt change, after the next model upgrade. That gap is what agent evaluation is really about.

And unlike traditional software testing, the answer alone usually isn’t enough.

Agents Fail Quietly

Ordinary software fails loudly. A broken function throws an exception. A crashed application shows an error screen. The failure announces itself.

Agents are different. When an agent fails, it usually still produces an answer — fluent, confident, well-formatted, and wrong. The failure hides under good writing.

Take a healthcare research agent. Ask it a clinical question and it might answer straight from memory, skipping retrieval entirely:

Question: What are the symptoms of a heart attack?
Route: answered directly (no retrieval)
Sources consulted: none
Answer: Chest pain, shortness of breath, nausea… (correct)

The answer happens to be correct. But notice what actually happened: the agent never searched for evidence. You only know the answer is right because you already know the answer. And when the question gets harder and the agent follows the exact same behavior, you’ll have no way to distinguish genuine knowledge from hallucination — unless you inspect the path.

1. Before You Evaluate Behavior, Make Behavior Visible

You can’t evaluate an agent’s decisions if you can’t see them. Before thinking about judges, benchmarks, or metrics, make the agent’s trajectory inspectable. If you’re building with LangGraph, the simplest approach is to store decisions directly in state.

from typing import TypedDict, Optional, List, Dict
from langgraph.graph import StateGraph, START, END

class AgentState(TypedDict):
    question: str
    route: Optional[str]
    sources: Optional[List[str]]
    search_results: Optional[Dict[str, str]]
    answer: Optional[str]
    steps: int

def decide(state): ...
def search(state): ...
def answer(state): ...

g = StateGraph(AgentState)

g.add_node("decide", decide)
g.add_node("search", search)
g.add_node("answer", answer)

g.add_edge(START, "decide")

g.add_conditional_edges(
    "decide",
    lambda s: s["route"],
    {
        "search": "search",
        "answer": "answer"
    }
)

g.add_edge("search", "answer")
g.add_edge("answer", END)

graph = g.compile()
Now every execution returns a trace:
final = graph.invoke({
    "question": q,
    "route": None,
    "sources": None,
    "search_results": None,
    "answer": None,
    "steps": 0
})

final["route"]    # "search" or "answer"  -> did it decide to look things up?
final["sources"]  # ["pubmed", "nature"]  -> which tools did it reach for?
final["steps"]    # 4                      -> how much did it cost?

That returned final is the trace. Everything we evaluate from here reads those fields.

The Biggest Mistake in Agent Evaluation

The entire field can be summarized in one sentence: evaluate the decisions, not just the answer. There are two fundamentally different things you can measure in an agent.

1. Output Evaluation: Did the agent produce a good answer? Accurate, Grounded, Clear and Helpful?

2. Output evaluation — did the agent produce a good answer? Accurate, grounded, clear, helpful?

3. Trajectory evaluation — did the agent behave correctly?

• Did it search when it should?
• Did it use the right tools?
• Did it gather sufficient evidence?
• Did it stay within budget?
• Did it use the evidence it retrieved?

Most teams only evaluate the output. That’s exactly where silent failures hide. An agent can arrive at a correct answer through a broken trajectory and simply get lucky. Eventually the luck runs out. Some checks a machine can do instantly; some need judgment.

2. Start With Deterministic Checks

Before involving another LLM, take advantage of something engineers already know how to trust: deterministic code. Many important agent behaviors can be verified instantly.

assert final["route"] == "search"
assert final["steps"] <= 6
assert "not medical advice" in final["answer"].lower()

These checks are fast, cheap, and objective. They verify architectural requirements:

· Did the agent search? · Did it stay within budget? · Did it satisfy compliance constraints?

Every rule you can encode as code is one less judgment call for a model. Use models only where deterministic validation ends.

3. Write Down What “Good” Looks Like (define test Cases)

A test set is simply your definition of “working” which can run the same way every time.. Each case includes an input, an expected behavior and any judgments that still require review.

from dataclasses import dataclass, field

@dataclass
class Expectation:
    expected_route: str                 # "search" or "answer"
    appropriate_sources: set            # which sources are OK here
    must_include: list = field(default_factory=list)  # e.g. a disclaimer
    step_budget: int = 6

A useful evaluation suite deliberately includes different categories of test cases:

· Happy Paths: Normal expected usage (e.g. Does metformin reduce mortality in type 2 diabetes?) · Edge Cases: Unusual but valid requests. · Adversarial Cases: Inputs designed to break assumptions. · Known Failures: Previously discovered bugs.

The last category is the most valuable. Every bug should become a permanent test case. Once fixed, it should never silently return. That’s regression testing for agent behavior.

4. Automating Human Judgment — Carefully

Deterministic checks scale beautifully. Human review does not. Eventually you need to evaluate hundreds or thousands of runs. That’s where LLM-as-a-judge becomes useful. But there’s an important caveat: A judge is just another model. It comes with its own biases. Naive judges consistently overrate:

• Long answers
• Confident answers
• Answers containing citations
• Answers with polished formatting

Ask a vague question like: Is this answer grounded? Rate it 1–5 and many judges will reward style rather than evidence. Instead of asking for an opinion, force them into audits and constrain the evaluation process.

def validate(judge, cases):
    agree = sum(judge(c).grounded == c.human_grounded for c in cases)
    return agree / len(cases)   # naive: 57% · careful: 100%

A judge you haven’t validated isn’t a judge — it’s just another unverified model running loose in your pipeline.

5. Evaluate Actions, Not Just Words

Agents don’t only generate text. They take actions. Tool use is behavior, and behavior can be evaluated directly.

ToolExpectation(
 expect_search=True,
 required_sources={"pubmed", "nature"},
 forbidden_sources={"web"},
 call_budget=3,
 )

The goal isn’t to enforce a single path. Many valid trajectories may exist. Instead, validate constraints for required tools were used, forbidden tools were avoided, resource budgets were respected. This catches failures hidden behind seemingly correct answers.

6. Grounding Isn’t Binary

Correctness and grounding are not the same thing. Ask an agent, “Has HRT guidance changed?” It may return a fluent, fully‑cited, accurate answer — and still be untrustworthy. Why? Because the evidence came from a telehealth marketing blog instead of the peer‑reviewed guideline it also retrieved. The answer was right, but for the wrong reasons.

Grounding isn’t a yes/no property. An answer can be correct, grounded, and still unreliable if the supporting sources are weak — blogs, SEO content, marketing pages. That’s why source quality needs its own evaluation layer. A simple framework may look like below:

Treat these as separate dimensions. A correct answer built on weak evidence is still technical debt.

Add Source Authority Checks

Not all evidence should carry equal weight. A claim about mortality rates should not be supported solely by a marketing blog. You can encode this directly:

HIGH   = ("pubmed", "ncbi.nlm.nih.gov", ".gov", ".edu", "cochrane", "nature.com")
STRONG = ("mortality", "fda", "reduces", "cure", "prevents")

def assess_sources(answer: str, evidence_domains: list[str]) -> dict:
    strong = [w for w in STRONG if w in answer.lower()]
    weak   = [d for d in evidence_domains
              if not any(h in d for h in HIGH)]
    has_high = any(any(h in d for h in HIGH) for d in evidence_domains)
    flagged  = bool(strong) and not has_high      # strong claim, no solid source
    return {"ok": not flagged, "strong_claims": strong, "weak_sources": weak}

assess_sources("HRT reduces breast cancer mortality",
               ["allarahealth.com", "whdatx.com"])
# {'ok': False,
#  'strong_claims': ['mortality', 'reduces'],
#  'weak_sources':  ['allarahealth.com', 'whdatx.com']}

The answer was correct. The check still flags it — because a strong medical claim is resting entirely on marketing pages, with no peer-reviewed source behind it. That’s the failure mode worth catching: the agent is right today, but its reasoning process is unreliable.

Two things to keep honest about this:

• It’s a separate dimension, not a replacement. Source authority sits beside your accuracy and grounding checks as a third axis — a correct, grounded answer can still fail it.
• It’s a smoke alarm, not a surgeon. A keyword-and-domain heuristic only flags that strong claims and weak sources co-occur; it can’t prove a specific claim came from a specific source. For true per-claim attribution you need an LLM judge or retrieval with source provenance. In practice the HIGH and STRONG lists are longer and domain-specific — keep them short here to see the shape of the idea.

7. Tooling Worth Knowing

You don’t need specialized tooling to begin. A spreadsheet, a test set, and a for-loop can take you surprisingly far. Eventually, though, dedicated platforms become useful. Common options include:

· LangSmith for trace inspection and evaluation workflows · Braintrust for large-scale evaluation and experiment tracking · Arize Phoenix for observability and debugging · Weights & Biases (Weave) for experiment management · OpenAI Evals for lightweight evaluation pipelines · DeepEval / Ragas — eval libraries with batteries-included metrics

The tooling matters less than the discipline. The fundamentals remain the same regardless of platform.

The Takeaway

1. Traditional software engineering taught us to test outputs, Agent engineering requires us to test decisions.
2. The answer is merely the artifact users see, the trajectory is the system you’re actually shipping. That’s where routing mistakes happen, that’s where bad tool choices happen and hallucinations begin. and that’s where most failures hide.
3. Evaluate both what the agent says and what the agent does.
4. Build a test set. Run it continuously. Read the diff.

Full runnable code: github.com/YOUR_USERNAME/trajectory-first-eval — every snippet above is a simplified excerpt; the repo has the complete, working versions.