ALL ABOUT AI EVALUATIONS— AGENT EVAL S— EVAL DRIVEN DEVELOPMENT

As Agents and Agentic AI Development is picking up traction, the need of AI Evals is also increasing. Good evaluations helps businesses and teams ship AI agents more confidently. Without them, it’s easy to get stuck in reactive loops catching issues only in production, where fixing one failure creates others.

Evals make problems and behavioral changes in agents visible before they affect users. Agents operate over many turns: calling tools, modifying state, and adapting based on intermediate results. These capabilities that make AI agents useful, autonomous, intelligent and flexibility also make them harder to evaluate.

An evaluation (“eval”) is a test for an AI system: give AI an input, then apply grading logic to its output to measure success. Here in this blog we focus on automated evals that can be run during development without real users.

SINGLE-TURN EVALUATIONS:

Single-turn evaluations are straightforward: a prompt, a response, and grading logic. For earlier LLMs, single-turn, non-agentic evals were the main evaluation method.

Single Turn Eval

AGENT EVALUATIONS:

As AI capabilities have advanced, Agent Evaluation have become increasingly common. Multi-turn evaluation a subset of Agent Evaluation is the process of assessing how well an AI or LLM system performs across an entire conversational thread, rather than judging each response in isolation.

Agent Eval

Before diving deep into Agent Evaluation lets us first understand the terms mostly used in Evaluation:

• A task (a.k.a problem or test case) is a single test with defined inputs and success criteria.
• Each attempt at a task is a trial. Because model outputs vary between runs, we run multiple trials to produce more consistent results.
• A grader is logic that scores some aspect of the agent’s performance. A task can have multiple graders, each containing multiple assertions (sometimes called checks).
• A transcript (also called a trace or trajectory) is the complete record of a trial, including outputs, tool calls, reasoning, intermediate results, and any other interactions.
• The outcome is the final state in the environment at the end of the trial. A flight-booking agent might say “Your flight has been booked” at the end of the transcript, but the outcome is whether a reservation exists in the environment’s SQL database.
• An evaluation harness is the infrastructure that runs evals end-to-end. It provides instructions and tools, runs tasks concurrently, records all the steps, grades outputs, and aggregates results.
• An agent harness (or scaffold) is the system that enables a model to act as an agent: it processes inputs, orchestrates tool calls, and returns results. When we evaluate “an agent,” we’re evaluating the harness and the model working together.
• An evaluation suite is a collection of tasks designed to measure specific capabilities or behaviors. Tasks in a suite typically share a broad goal. For instance, a customer support eval suite might test refunds, cancellations, and escalations.

Following diagram makes it easy for you to understand the components of evals:

Components Of Evals

REAL WORLD EXAMPLES OF COMPANIES USING EVALS:

Claude Code started with fast iteration based on feedback from Anthropic employees and external users. Later, they added evals — first for narrow areas like concision and file edits, and then for more complex behaviors like over engineering. These evals helped identify issues, guide improvements, and focus research product collaborations. Combined with production monitoring, A/B tests, user research, and more, evals provide signals to continue improving Claude Code as it scales.

Descript’s agent helps users edit videos, so they built evals around three dimensions of a successful editing workflow: don’t break things, do what I asked, and do it well. They evolved from manual grading to LLM graders with criteria defined by the product team and periodic human calibration, and now regularly run two separate suites for quality benchmarking and regression testing.

Bolt AI team started building evals later, after they already had a widely used agent. In 3 months, they built an eval system that runs their agent and grades outputs with static analysis, uses browser agents to test apps, and employs LLM judges for behaviors like instruction following.

Some teams create evals at the start of development; others add them once at scale when evals become a bottleneck for improving the agent. Evals are especially useful at the start of agent development to explicitly encode expected behavior. Two engineers reading the same initial spec could come away with different interpretations on how the AI should handle edge cases. An eval suite resolves this ambiguity. Regardless of when they’re created, evals help accelerate development.

Evals also shape how quickly you can adopt new models. When more powerful models come out, teams without evals face weeks of testing while competitors with evals can quickly determine the model’s strengths, tune their prompts, and upgrade in days.

Once evals exist, you get baselines and regression tests for free: latency, token usage, cost per task, and error rates can be tracked on a static bank of tasks. Evals can also become the highest-bandwidth communication channel between product and research teams, defining metrics researchers can optimize against.

HOW TO EVALUATE AI AGENTS?

We see several common types of agents deployed at scale today, including coding agents, research agents, computer use agents, and conversational agents. Each type may be deployed across a wide variety of industries, but they can be evaluated using similar techniques. You don’t need to invent an evaluation from scratch. The sections below describe proven techniques for several agent types. Use these methods as a foundation, then extend them to your domain.

Agent evaluations typically combine three types of graders: code-based, model-based, and human. Each grader evaluates some portion of either the transcript or the outcome. An essential component of effective evaluation design is to choose the right graders for the job.

Code Based Graders

Model Based Graders

Human Graders

For each task, scoring can be weighted (combined grader scores must hit a threshold), binary (all graders must pass), or a hybrid.

After an agent is launched and optimized, capability evals (“What can this agent do well?” They should start at a low pass rate, targeting tasks the agent struggles with and giving teams a hill to climb ) with high pass rates can “graduate” to become a regression suite (regression evals is “Does the agent still handle all the tasks it used to?” and should have a nearly 100% pass rate. They protect against backsliding, as a decline in score signals that something is broken and needs to be improved. As teams hill-climb on capability evals, it’s important to also run regression evals to make sure changes don’t cause issues elsewhere) that is run continuously to catch any drift. Tasks that once measured “Can we do this at all?” then measure “Can we still do this reliably?”

EVALUATING CODING AGENT:

Coding Agents are usually evaluated on the basis of benchmark, whether the agent works well on a certain benchmark. For example SWE-Bench is a benchmark designed to evaluate how well a coding agent can solve real software engineering tasks. You run your agent on the benchmark tasks. The benchmark checks whether your agent’s generated code actually fixes the issue (typically by running the project’s tests). The final score tells you how many tasks your agent successfully solved. So your workflow is basically: Give your coding agent a SWE-Bench task (usually a GitHub issue). The agent analyzes the repository. The agent edits the code. The benchmark runs the project’s test suite. If the relevant tests pass (and the benchmark’s evaluation criteria are met), the task is considered solved. Repeat across many tasks to obtain an overall success rate. For example: SWE-Bench contains 500 tasks. Your agent solves 180 correctly. Your score is 36%. You can then compare this score against: previous versions of your own agent. Similarly one another benchmark i.e. Terminal-Bench takes a different track: it tests end-to-end technical tasks, such as building a Linux kernel from source or training an ML model.

Once you have a set of pass-or-fail tests for validating the key outcomes of a coding task, it’s often useful to also grade the transcript.

EVALUATING CONVERSTATIONAL AGENTS:

Conversational agents interact with users in domains like support, sales, or coaching. Unlike traditional chatbots, they maintain state, use tools, and take actions mid-conversation. While coding and research agents can also involve many turns of interaction with the user, conversational agents present a distinct challenge: the quality of the interaction itself is part of what you’re evaluating. Effective evals for conversational agents usually rely on verifiable end-state outcomes and rubrics that capture both task completion and interaction quality. Unlike most other evals, they often require a second LLM to simulate the user. Anthropic uses alignment auditing agents approach to stress-test models through extended, adversarial conversations.

Success for conversational agents can be multidimensional: is the ticket resolved (state check), did it finish in <10 turns (transcript constraint), and was the tone appropriate (LLM rubric)? Two benchmarks that incorporate multidimensionality are 𝜏-Bench and its successor, τ2-Bench. These simulate multi-turn interactions across domains like retail support and airline booking, where one model plays a user persona while the agent navigates realistic scenarios.

In practice, conversational agent evaluations typically use model-based graders to assess both communication quality and goal completion, because many tasks — like answering a question — may have multiple “correct” solutions.

EVALUATING RESEARCH AGENTS:

Research agents gather, synthesize, and analyze information, then produce outputs like an answer or report. Unlike coding agents where unit tests provide binary pass/fail signals, research quality can only be judged relative to the task. What counts as “comprehensive,” “well-sourced,” or even “correct” depends on context: a market scan, due diligence for an acquisition, and a scientific report each require different standards.

Research evals face unique challenges: experts may disagree on whether a synthesis is comprehensive, ground truth shifts as reference content changes constantly, and longer, more open-ended outputs create more room for mistakes. A benchmark like BrowseComp, for example, tests whether AI agents can find needles in haystacks across the open web — questions designed to be easy to verify but hard to solve.

One strategy to build research agent evals is to combine grader types. Groundedness checks verify that claims are supported by retrieved sources, coverage checks define key facts a good answer must include, and source quality checks confirm the consulted sources are authoritative, rather than simply the first retrieved. For tasks with objectively correct answers (“What was Company X’s Q3 revenue?”), exact match works. An LLM can flag unsupported claims and gaps in coverage but also verify the open-ended synthesis for coherence and completeness.

As we know that Agents are non-deterministic (they return different result on every try which makes evaluation results harder to interpret than they first appear) so how to think about non-determinism in evaluations for agents?

When agents runs tasks each task has its own success rate — maybe 90% on one task, 50% on another — and a task that passed on one eval run might fail on the next. Sometimes, what we want to measure is how often (what proportion of the trials) an agent succeeds for a task.

Two metrics help capture this nuance:

pass@k measures the likelihood that an agent gets at least one correct solution in k attempts. As k increases, pass@k score rises: more “shots on goal” means higher odds of at least 1 success. A score of 50% pass@1 means that a model succeeds at half the tasks in the eval on its first try. In coding, we’re often most interested in the agent finding the solution on the first try — pass@1. In other cases, proposing many solutions is valid as long as one works.

pass^k measures the probability that all k trials succeed. As k increases, pass^k falls since demanding consistency across more trials is a harder bar to clear. If your agent has a 75% per-trial success rate and you run 3 trials, the probability of passing all three is (0.75)³ ≈ 42%. This metric especially matters for customer-facing agents where users expect reliable behavior every time.

Both metrics are useful, and which to use depends on product requirements: pass@k for tools where one success matters, pass^k for agents where consistency is essential.

This is all about Evals in AI. I hope this article provides an introduction, better understanding and helpful tips for evaluating AI agents. In the next article i will be summarizing a roadmap for great evals for agents. See you in the next one.