Evaluations in AI Applications

The 2 a.m. question every AI team eventually asks

Every team building with large language models eventually hits the same wall. The demo worked. The investor loved it. The Slack channel was full of fire emojis. Then it shipped, and three weeks later someone asks the question that ends the celebration: “How do we actually know it’s working?”

Not “does it look like it’s working” — anyone can scroll through ten chat transcripts and feel good. The real question is whether the system performs reliably across thousands of unseen inputs, whether last week’s prompt tweak made things better or quietly worse, and whether the multi-step agent that books flights and edits spreadsheets on someone’s behalf will still be doing the right thing in production at 2 a.m. when no one is watching.

That question has a name: evaluation, or “evals” for short. It is, without much exaggeration, the most underrated discipline in applied AI today. Model architecture gets the conference talks. Evals get the 2 a.m. pages. This article is a practical, end-to-end tour of evals — for generative AI and for the harder, newer problem of agentic AI — covering the major eval types, how to write eval code that actually scales, where evals help and where they hurt, the shape of a healthy eval pipeline, and where the discipline is heading in 2026 and beyond.

What evals actually are

An eval is a structured, repeatable test that scores how well an AI system performs against a defined task. That’s the whole definition, but three words in it carry all the weight: structured, repeatable, and scored.

• Structured — there’s a defined input, an expected behavior or outcome, and a method for judging the actual output against it. “Vibes” is not a method.

• Repeatable — the same eval can be re-run against a new model version, a new prompt, or a new agent configuration, and produce a comparable score. If you can’t compare today’s number to last Tuesday’s number, it isn’t an eval yet.

• Scored — somewhere a number, a pass/fail, or a categorical label comes out the other end, because without a score there’s nothing to aggregate, chart, or gate a release on.

Evals borrow heavily from classical machine learning, where test sets, precision, recall, and F1 scores have existed for decades. What’s different with generative and agentic AI is that the “correct answer” is frequently not a single label anymore — it’s an essay, a piece of code, a multi-turn conversation, or a sequence of tool calls. Grading free-form, open-ended output is a fundamentally harder problem than grading a classification, and that difficulty is exactly why evals have become their own specialty rather than a footnote in a training pipeline.

Evals for generative AI

Generative AI evals assess a single model’s output for a given input — text completions, summaries, translations, code, or images. The core question is narrower than it is for agents: given this prompt, is this output good?

What gets measured

• Correctness and factuality — does the output match ground truth, or is it free of hallucinated facts, names, dates, or citations?

• Relevance and helpfulness — does the response actually address what was asked, without padding or going off-topic?

• Coherence and fluency — is the output well-structured, grammatically sound, and logically consistent across sentences?

• Safety and policy compliance — does the output avoid toxic, biased, or disallowed content, and does it refuse appropriately when it should?

• Style and tone adherence — does the output match a required voice, format, reading level, or brand guideline?

• Groundedness — for RAG systems specifically, is every claim in the answer traceable back to a retrieved source document?

For RAG pipelines in particular — which show up constantly in claims processing, fraud investigation, and internal knowledge assistants — evals typically split into retrieval metrics and generation metrics. Retrieval metrics ask whether the right documents were even fetched (context precision, context recall, hit rate). Generation metrics ask whether the model used those documents faithfully (faithfulness, answer relevance, groundedness). A RAG system can retrieve perfectly and still hallucinate, or retrieve poorly and still happen to generate a correct answer from parametric memory — which is precisely why both halves need separate evals rather than one blended score.

Evals for agentic AI

Agentic systems are a different animal, and evaluating them is considerably harder than evaluating a single generation. An agent built with LangGraph, CrewAI, or AutoGen doesn’t just produce one output — it plans, calls tools, reads results, revises its plan, calls more tools, and eventually terminates. The output you actually care about is often the trace, not just the final message.

What gets measured

• Task completion rate — did the agent actually accomplish the goal, end to end, not just produce plausible-sounding text about the goal?

• Tool selection accuracy — did it call the correct tool, with the correct arguments, at the correct point in the plan?

• Trajectory quality — was the sequence of steps efficient, or did the agent loop, backtrack, or take an unnecessarily expensive path to the same result?

• Groundedness of intermediate reasoning — did each step follow logically from the previous tool result, rather than ignoring it?

• Multi-agent coordination — in crew- or graph-based systems, did agents hand off context correctly, or did information get lost between roles?

• Recovery and robustness — when a tool call fails or returns malformed data, does the agent retry sensibly, or does it hallucinate a result and continue?

• Cost and latency per task — a technically correct agent that takes 40 tool calls and 90 seconds may still fail a production eval if the budget was 5 calls and 10 seconds.

This is where most teams get surprised. A chatbot eval is largely about the final text. An agent eval has to additionally reconstruct and judge a full execution trace — which tools were called, in what order, with what arguments, and whether the plan that produced that trace was sound even if the final answer happened to look fine. Two agents can reach the identical final answer through very different paths, one of which is robust and one of which is a near-miss that will fail the moment the environment changes slightly. Trajectory-level evaluation is the only way to tell those two apart, and it’s the single biggest methodological shift between gen AI evals and agentic AI evals.

Types of evals, matched to the application

There isn’t one eval method — there’s a toolbox, and the right tool depends heavily on what’s being built. Below is a working map from application type to the eval approaches that tend to fit best in practice.

The three grading mechanisms underneath every eval type

Whatever the application, almost every eval ultimately uses one of three grading mechanisms, often in combination.

1. Code-based (deterministic) evals

These compare output against an exact or pattern-based expectation: string match, regex, JSON schema validation, unit test execution, SQL query equivalence. They’re fast, free, perfectly reproducible, and the gold standard whenever the task has a checkable right answer — code that should pass tests, a function call that should hit a specific API with specific arguments, a classification that should match a label. Their weakness is that most generative tasks don’t have a single checkable right answer; “write a polite decline email” has no regex.

2. Model-graded evals (LLM-as-judge)

A second LLM — often a stronger or differently-prompted model — reads the output and a rubric, then produces a score or verdict. This is the workhorse for open-ended generation, because it scales far better than human review and can apply nuanced criteria a regex never could. The catch is that judge models have their own biases: they tend to favor longer answers, can be swayed by the order in which two outputs are presented, and inherit blind spots from their own training. Well-designed model-graded evals mitigate this with clear rubrics, few-shot examples of good and bad outputs, randomized ordering, and periodic calibration against human judgments.

3. Human evals

People reading outputs and scoring them against guidelines remain the ground truth that the other two methods are ultimately validated against. Human evals are slow and expensive, which is exactly why teams use them sparingly — to build the gold-standard dataset, to calibrate an LLM judge, and to spot-check production samples — rather than as the main gate in a CI pipeline.

Where evals are headed: 2026 and beyond

Evaluation is shifting from an afterthought bolted onto the end of model development to a first-class layer of the AI stack in its own right, and a few directions are becoming clear.

From static benchmarks to continuous, live evaluation

Fixed test sets, refreshed quarterly, are giving way to evals that run continuously against sampled production traffic — closer to how application performance monitoring works in traditional software than to how academic NLP benchmarking has historically worked.

Trajectory and process evaluation become standard, not optional

As agentic systems multiply — across customer support, coding, finance, and operations — evaluating the path an agent takes, not merely its final answer, is moving from a niche research concern to a baseline requirement, with emerging standards for trace formats and step-level grading.

Multi-agent and cross-system evals

As systems built from CrewAI-style role-based crews, LangGraph state machines, and A2A-protocol-style cross-agent communication become more common, eval frameworks are extending to score coordination quality between agents and systems, not just the performance of any single agent in isolation.

Eval-as-a-service and standardized eval marketplaces

Expect more shared, domain-specific eval suites — for legal, medical, financial, and coding tasks — that organizations can adopt rather than rebuild from scratch, similar to how shared unit-testing frameworks displaced bespoke testing harnesses in traditional software engineering.

Tighter integration with observability and tracing

Evals are converging with observability tooling, so that a single trace of an agent’s execution feeds both a debugging dashboard and an automated eval score, rather than living in two disconnected systems that engineers have to manually reconcile.

Regulation will make evals a compliance requirement, not just an engineering best practice

As AI governance frameworks mature globally, demonstrable, documented evaluation — particularly for safety, fairness, and reliability — is increasingly likely to become a legal and audit requirement in high-stakes domains, not merely something careful teams choose to do.

Self-improving eval loops

The frontier emerging now is systems where production failures automatically generate new eval cases, which then automatically retrain or fine-tune the system, closing the loop shown earlier in this article with far less manual curation than today’s process requires.

What an eval pipeline looks like in practice

Most production eval setups converge on the same five-stage loop, regardless of whether the underlying system is a single-turn chatbot or a multi-agent trading platform.

Figure 1 — the eval loop: golden dataset, model run, scoring, aggregation, and the gate that decides whether a change ships. Failures discovered in production get added back into the golden dataset, so the eval suite compounds in coverage over time

The detail that separates teams with a real eval culture from teams that just “have some evals” is that last dashed arrow. Every production failure, every edge case a user stumbles into, every weird tool-call loop an agent gets stuck in — all of it should become a new row in the golden dataset, not just a bug ticket that gets closed and forgotten. Eval suites that don’t grow from production incidents go stale within a few months and stop catching the regressions that actually matter.

Writing eval code that scales

The fastest way for an eval suite to die is for it to be slow, flaky, or expensive to run — because a slow eval suite gets skipped, and a skipped eval suite provides exactly zero protection. A few patterns from production systems make a meaningful difference.

1. Separate the dataset, the runner, and the scorer

Treat these as three independent, swappable pieces rather than one monolithic script. The dataset is just data — inputs, expected outputs or rubrics, metadata, ideally versioned in source control or a database. The runner executes the system under test against each dataset row. The scorer takes a (input, output, expected) triple and returns a score. Decoupling them means a model upgrade only touches the runner, a new grading approach only touches the scorer, and new test cases only touch the dataset — instead of every change requiring a rewrite of one giant function.

# A minimal but real structure for a generative AI eval

from dataclasses import dataclass
from typing import Callable
import asyncio

@dataclass
class EvalCase:
    id: str
    input: str
    expected: str | None = None
    rubric: str | None = None
    metadata: dict | None = None

@dataclass
class EvalResult:
    case_id: str
    score: float
    passed: bool
    output: str
    latency_ms: float
    cost_usd: float
    details: dict | None = None

async def run_eval_suite(
    cases: list[EvalCase],
    runner: Callable,
    scorer: Callable,
    concurrency: int = 8,
) -> list[EvalResult]:
    semaphore = asyncio.Semaphore(concurrency)

    async def run_one(case: EvalCase) -> EvalResult:
        async with semaphore:
            output, latency_ms, cost_usd = await runner(case.input)
            score, passed, details = await scorer(case, output)
            return EvalResult(
                case_id=case.id,
                score=score,
                passed=passed,
                output=output,
                latency_ms=latency_ms,
                cost_usd=cost_usd,
                details=details,
            )

    return await asyncio.gather(*(run_one(c) for c in cases))

2. Run cases concurrently, but cap concurrency deliberately

LLM calls are I/O-bound, so sequential evaluation of a few hundred test cases wastes enormous amounts of wall-clock time waiting on network round-trips. Async execution with a bounded semaphore, as in the snippet above, is usually the single biggest speedup available — turning a 40-minute eval run into a 4-minute one. The cap matters too: providers rate-limit aggressively, and an uncapped fan-out just trades wall-clock time for a wall of 429 errors.

3. Cache aggressively, but key the cache correctly

Re-running an eval suite after only changing the scorer shouldn’t require regenerating every model output again. Cache the (input, model, prompt version, temperature) tuple to the model’s output, and invalidate only the relevant cache entries when something in that key actually changes. This is what makes it cheap to iterate on a rubric ten times in an afternoon instead of once.

4. Make scorers deterministic where possible, and seed the rest

Code-based scorers should be pure functions — same input, same output, every time, with no hidden network calls or wall-clock dependence. For model-graded scorers, pin the judge model’s version, set temperature to zero or near-zero, and log the exact rubric and prompt used for grading alongside the result. Without this, a noisy regression three weeks from now is indistinguishable from judge-model drift, and debugging becomes guesswork.

5. Aggregate with the right statistic, not just a mean

A single average score across a hundred test cases hides exactly the information that matters most: which subgroup is failing. Segment scores by category, difficulty, and known edge cases, and track tail metrics (worst 10%, pass rate at a strict threshold) alongside the mean. A system that scores 92% on average but drops to 40% on multi-step financial calculations is a very different system from one that scores a uniform 92% everywhere — and a bare mean cannot tell those two apart.

# Aggregation that surfaces the failure mode, not just a headline number

import pandas as pd

def summarize(results: list[EvalResult], cases_by_id: dict[str, EvalCase]) -> pd.DataFrame:
    rows = []
    for r in results:
        case = cases_by_id[r.case_id]
        category = (case.metadata or {}).get("category", "uncategorized")
        rows.append({
            "category": category,
            "score": r.score,
            "passed": r.passed,
            "latency_ms": r.latency_ms,
            "cost_usd": r.cost_usd,
        })

    df = pd.DataFrame(rows)
    summary = df.groupby("category").agg(
        n=("score", "count"),
        mean_score=("score", "mean"),
        pass_rate=("passed", "mean"),
        p90_latency_ms=("latency_ms", lambda s: s.quantile(0.9)),
        total_cost_usd=("cost_usd", "sum"),
    ).reset_index()

    summary["regression_flag"] = summary["pass_rate"] < 0.85
    return summary.sort_values("pass_rate")

6. Gate releases automatically, but with a human escape hatch

In CI/CD, an eval suite is most useful when it fails the build the same way a unit test failure does — below-threshold pass rate blocks the merge. But agentic and generative evals carry more judgment-call ambiguity than a typical unit test, so pair the automated gate with a clear, fast path for a human to review and override a borderline failure, rather than either blocking everything rigidly or, worse, making the gate advisory-only and routinely ignored.

Where evals actually get used

• Pre-release regression testing — catching the silent quality drop when a prompt, a model version, or a retrieval index changes, before it reaches users.

• Model and vendor selection — comparing GPT, Claude, Gemini, or open-weight models on the same task-specific benchmark instead of relying on generic leaderboards that may not reflect the actual workload.

• Prompt and pipeline iteration — turning prompt engineering from trial-and-error into a measurable optimization loop with a real scoreboard.

• Production monitoring — running lightweight evals continuously on a sample of live traffic to catch drift, not just at deploy time.

• Safety and compliance sign-off — in regulated domains like finance and healthcare, evals double as the audit trail that demonstrates a system was tested against defined criteria before going live.

• Agent debugging — trajectory-level evals are often the only practical way to find where in a forty-step agent run things actually went wrong.

Pros and cons, honestly stated

What evals genuinely give you

• Objective, repeatable comparisons across model versions, prompts, and architectures — replacing gut feel with a number you can defend.

• Early detection of regressions before they reach users, which is dramatically cheaper than detecting them after a customer complains.

• A shared, falsifiable definition of “good” across a team, instead of every engineer privately judging quality by a different standard.

• A defensible audit trail in regulated industries, where being able to show what was tested matters as much as the system actually working.

• Faster iteration, because a good eval suite turns subjective debates about prompt wording into a five-minute experiment with a clear winner.

Where evals fall short

• Building a genuinely representative golden dataset is slow, manual work, and a sloppy dataset produces a confidently wrong eval score.

• LLM-as-judge evals inherit the judge model’s blind spots and biases, and can be gamed by outputs that are persuasive rather than correct.

• Trajectory and multi-agent evaluation remain immature compared to single-turn evals — tooling and shared standards are still catching up to how fast agentic systems are being built.

• Eval suites can create a false sense of security: a system that scores well on a fixed test set can still fail in unanticipated, long-tail real-world scenarios the dataset never covered.

• Running thorough evals, especially with strong judge models or human reviewers, has real cost and latency, and teams under deadline pressure are tempted to cut corners exactly when evals matter most.

• Metrics can be optimized for their own sake (Goodhart’s law) — a team chasing a benchmark number can ship a system that’s technically higher-scoring but subjectively worse to actually use.

Where evals are headed: 2026 and beyond

Evaluation is shifting from an afterthought bolted onto the end of model development to a first-class layer of the AI stack in its own right, and a few directions are becoming clear.

From static benchmarks to continuous, live evaluation

Fixed test sets, refreshed quarterly, are giving way to evals that run continuously against sampled production traffic — closer to how application performance monitoring works in traditional software than to how academic NLP benchmarking has historically worked.

Trajectory and process evaluation become standard, not optional

As agentic systems multiply — across customer support, coding, finance, and operations — evaluating the path an agent takes, not merely its final answer, is moving from a niche research concern to a baseline requirement, with emerging standards for trace formats and step-level grading.

Multi-agent and cross-system evals

As systems built from CrewAI-style role-based crews, LangGraph state machines, and A2A-protocol-style cross-agent communication become more common, eval frameworks are extending to score coordination quality between agents and systems, not just the performance of any single agent in isolation.

Eval-as-a-service and standardized eval marketplaces

Expect more shared, domain-specific eval suites — for legal, medical, financial, and coding tasks — that organizations can adopt rather than rebuild from scratch, similar to how shared unit-testing frameworks displaced bespoke testing harnesses in traditional software engineering.

Tighter integration with observability and tracing

Evals are converging with observability tooling, so that a single trace of an agent’s execution feeds both a debugging dashboard and an automated eval score, rather than living in two disconnected systems that engineers have to manually reconcile.

Regulation will make evals a compliance requirement, not just an engineering best practice

As AI governance frameworks mature globally, demonstrable, documented evaluation — particularly for safety, fairness, and reliability — is increasingly likely to become a legal and audit requirement in high-stakes domains, not merely something careful teams choose to do.

Self-improving eval loops

The frontier emerging now is systems where production failures automatically generate new eval cases, which then automatically retrain or fine-tune the system, closing the loop shown earlier in this article with far less manual curation than today’s process requires.

The takeaway

You cannot improve what you do not measure, and in generative and agentic AI, the thing you’re measuring is no longer a single number — it’s a distribution of behaviors across an open-ended space of possible inputs.

Evals are not a tax on shipping fast. They are what makes shipping fast survivable. The teams that treat eval infrastructure with the same seriousness as the model or the agent architecture itself are the ones whose systems keep working after the demo ends — and as generative and agentic AI keep moving from impressive prototypes into systems that touch real money, real claims, and real decisions, that seriousness stops being optional.

Build the golden dataset before you need it. Decouple the runner from the scorer. Score the trajectory, not just the destination. Let production failures feed the next eval cycle. Do that consistently, and the 2 a.m. question gets a real answer instead of a shrug.

Thank you for diving into this post. I hope this content helps in better understanding. Also publised ebook on Gumroad for Agentic AI and AI production Issues bible. If the content helped you, your claps and subscribe me on Medium that means a lot — they help this knowledge reach more readers and keep me motivated to write more. Really appreciate your time and support!!!