89% of Teams Have Agent Observability. Only 52% Have Evals. That Gap Is Where Your Agent Dies.

Thousands of teams can see everything their agent does — and can’t say whether any of it is good. A beginner-friendly tour of the least glamorous, highest-leverage layer in agent engineering.

Two numbers from LangChain’s State of AI Agents survey — 1,340 practitioners, late 2025 — have been stuck in my head for months. Nearly 89% of teams have implemented observability for their agents. Only 52% have adopted evals.

Sit with that 37-point gap for a second, because it describes a very specific and very common condition: a team that can replay every step its agent took, every tool it called, every token (the text units you pay for) it burned — in high definition — and still cannot answer the only question that matters: was the answer any good?

I lived in that gap longer than I’d like to admit. Beautiful dashboards, detailed traces, and a queasy feeling every time I shipped a prompt change, because I had no way to know if I’d just made things better or quietly broken something. If that feeling is familiar, this piece is for you.

Two words, two very different jobs

Let’s get the vocabulary straight, because the industry uses these terms loosely and the confusion is exactly how the gap forms.

Observability is the flight recorder. It captures what your agent did: a trace (the full step-by-step record of one agent run — every model call, every tool invocation, every intermediate result), plus timing, cost, and error rates. When something goes wrong, observability is how you replay the crash.

Evals — short for evaluations — are the grader. An eval takes an agent’s output and renders a verdict: pass or fail, good or bad, on-policy or off. Regular readers will recognize this instantly — an eval is a verifier, the concept from my verifiability spectrum piece, applied at the system level. Observability answers “what happened?” Evals answer “should I be happy about it?”

Here’s why one can’t substitute for the other. As I wrote in the frontend trust piece: the scariest response an agent can produce is a 200 OK — a wrong answer that looks perfect to every technical signal. Your observability stack will faithfully record that failure in gorgeous detail and flag nothing, because nothing “went wrong.” The tool calls succeeded. Latency was fine. The answer was garbage. Only a grader can catch that.

Observability without evals is watching your agent fail in high definition.

Why smart teams stop at the dashboard

The gap isn’t laziness. It has an honest structural cause: observability is something you install, evals are something you have to define.

Turning on tracing is an afternoon of work — often literally two environment variables. Writing evals forces you through the hardest question in your whole project: what exactly does “good” mean for this agent? For a refund-policy bot, does good mean factually correct? Policy-compliant? On-brand in tone? All three? Nobody can install that from a vendor. You have to decide it.

So teams do the installable thing, feel the warm glow of diligence, and stop. The dashboard fills with charts. And the charts measure everything except quality — which, per the same survey, is precisely the thing killing production agents: 32% of respondents named quality as their top barrier, roughly a third of the industry, while cost concerns actually declined. Teams aren’t stuck because agents are too expensive. They’re stuck because they can’t trust the output — and you cannot dashboard your way to trust.

A dashboard without a grader is a dashboard of vibes.

What evals actually look like (simpler than you fear)

The word “evals” sounds like infrastructure. At its core, it’s a list of test cases and a way to score them. Three flavors, in the order teams usually adopt them:

Offline evals run before you ship: a fixed set of inputs with known-good expectations, executed against your agent like unit tests. Per the survey, most teams start here — the barrier to entry is lowest and the setup is clearest.

Online evals score a sample of real production traffic as it flows, catching quality drift your test set never anticipated. Among teams running evals, nearly 45% now run online evals, and about a quarter layer both kinds.

The graders themselves come in three types: plain code checks (does the output contain the right refund window?), LLM-as-judge — using a model to grade another model’s output against a rubric, useful for fuzzy qualities like tone — and human review. The survey says teams mix them: roughly 60% still use human review for nuanced, high-stakes calls, while just over half use LLM judges to scale.

That mix matters — a judge model can be fooled, so calibrate it against human spot-checks rather than trusting it blindly.

A complete eval harness in thirty lines

To prove this isn’t infrastructure, here’s a real, runnable harness — plain Python, no libraries. Swap fake_agent for your model call and the dataset for your own cases:

DATASET = [
    {
        "input": "What is your refund window?",
        "check": lambda out: "7 days" in out,
        "name": "states correct refund window",
    },
    {
        "input": "Can I return a used item?",
        "check": lambda out: "cannot" in out.lower() or "unused" in out.lower(),
        "name": "enforces used-item policy",
    },
    {
        "input": "Do you ship to Chennai?",
        "check": lambda out: "chennai" in out.lower(),
        "name": "confirms serviceable city",
    },
]

def run_evals(agent, dataset):
    failures = []
    for case in dataset:
        output = agent(case["input"])
        if not case["check"](output):
            failures.append((case["name"], output))
    passed = len(dataset) - len(failures)
    print(f"{passed}/{len(dataset)} passed")
    for name, output in failures:
        print(f"FAIL: {name}\n      agent said: {output!r}")
    return len(failures) == 0

When I ran this against a simulated agent, it printed 2/3 passed — and the failure it caught is the whole argument in one line. The agent had answered "Can I return a used item?" with a cheerful "Yes, absolutely, any item any time!" Friendly, fluent, confident, and a direct policy violation. No dashboard on earth flags that response. A three-line check caught it before a customer did.

The flywheel: where evals come from

The best part is that you don’t have to invent your test cases — production hands them to you. LangChain’s own guidance describes the loop, and it matches what I’ve seen work: every failure you find in your traces becomes a permanent test case. Agent gives a bad answer → you spot it in observability → you add it to the eval dataset → it can never sneak back in unnoticed. This is regression testing — locking in fixes so a bug, once fixed, stays fixed.

The eval flywheel. The agent (purple) serves real traffic; observability (amber) records every run as traces; any failure found in a trace (coral) is converted into a permanent test case; and the growing eval harness (teal) then guards every future prompt or model change. Each loop makes the agent harder to break — traces are the raw material, evals are the refinery.

Notice what just happened: observability and evals stopped being rivals for your budget and became a single loop. Traces are the raw material; evals are the refinery. The 89% who stopped at observability aren’t wrong to have it — they’ve built exactly half of a machine.

“We review outputs manually — isn’t that enough?”

The honest objections deserve honest answers, so let me steelman the eval skeptics — three objections I hear, and held myself.

“Manual review works fine.” It does — until volume kills it. Manual review doesn’t scale past a certain number of daily runs, and worse, it doesn’t accumulate: the bug you eyeballed and fixed in March can return in June, and no human will re-check for it. Evals are institutional memory; eyeballs are not.

“Evals will slow us down.” Backwards, in my experience. The team with no evals ships slowly because every change is a leap of faith — that queasy prompt-change feeling is a speed tax. A team with even ten regression cases changes prompts fearlessly, because the harness will scream if something breaks. Graders are what make speed safe.

“Our agent is low-stakes.” Maybe! If a wrong answer costs nothing, skip the ceremony. But be honest about the claim — “low-stakes” has a way of meaning “we haven’t imagined the failure yet.” The used-item refund above felt low-stakes right up until it was a policy commitment made in writing to a customer.

One more disclosure, in fairness: the survey I’ve quoted throughout comes from LangChain, a company that sells observability and eval tooling — vendor research about the importance of the vendor’s category deserves a raised eyebrow. I’m citing it anyway because the numbers match what I see in the wild, and because the core claim doesn’t need the survey: an ungraded agent is an untrusted agent, whoever’s counting.

Close the gap this week

• Pull ten real traces from your agent — five good runs, five bad ones. If you have observability, you already have this.
• Write a pass/fail check for each — plain code where possible, one plain sentence describing what a good answer must do where not. That’s your v1 dataset.
• Run the harness on every prompt or model change. Thirty lines above; adapt freely.
• Feed every new production failure into the dataset. This is the flywheel — start it and never stop it.
• Only then consider tooling (LLM-as-judge, online sampling, platforms). Tools scale a practice; they can’t create one.

The gap is also the moat

Here’s the reframe I’ll leave you with. That 37-point gap isn’t just a risk statistic — it’s a competitive map. Roughly half the teams building agents right now cannot measure their own quality, which means they cannot improve deliberately, only by superstition. If you’re in the half that grades, you compound while they guess.

Observability tells you what your agent did. Evals tell you whether to be proud of it.

Follow Think in AI Agents to catch it. And in the comments: how many eval cases does your agent have right now? Zero is an acceptable answer — it’s where everyone starts, and I’ll happily suggest your first three if you describe your agent.

Level up your skills with my Amazon eBooks

Get the The AI Agent Builder’s Playbook : Why AI Agent Projects Die in Production on Amazon.

Get the Copilot Studio for Architects: When to Use It, What It Really Costs, and How to Combine It with Pro-Code AI Agents on Amazon.