How to Debug AI Agents with Traces and Evals

Your AI agent failed, but the chat transcript doesn’t explain why.

This image was created using an AI image generation program.

So someone edits the prompt, reruns one example, and calls it fixed.

That is how agent quality turns into guesswork.

A better workflow is slower at first and faster later: capture traces, label what actually went wrong, convert those labels into evals, and only then change the prompt, tools, routing, guardrails, or harness. OpenAI’s Agents SDK tracing docs say traces can capture LLM generations, tool calls, handoffs, guardrails, and custom events during an agent run.

This article is about that loop.

Not observability as decoration.

Not dashboards for screenshots.

A real trace-to-eval loop.

Do not rewrite the prompt until you can replay the failure.

The common mistake: treating the prompt as the whole system

When an agent fails, the prompt is the easiest thing to blame.

It is visible. It is editable. It feels like the control panel.

But most agent failures are not only prompt failures.

A bad answer can come from weak instructions, yes. It can also come from the wrong tool call, missing context, a bad retrieval result, a handoff to the wrong specialist, a schema mismatch, a guardrail that fired too late, or a tool result the model interpreted incorrectly.

OpenAI’s current Agents SDK guide describes agents as applications that plan, call tools, collaborate across specialists, and keep enough state to complete multi-step work. It also says the SDK path is for applications that own orchestration, tool execution, approvals, and state.

That matters.

If the system has tools, state, approvals, routing, and handoffs, then a prompt diff is not enough evidence.

You need to see the run.

A trace is not an eval

This is the first line to draw.

A trace tells you what happened.

An eval tells you whether it was good enough.

Both are needed.

A trace can show that the agent called a search tool, passed a query, received results, handed off to another agent, and produced an answer. That is useful. It can still leave the hardest question unanswered:

Was that the right behavior?

OpenAI’s May 12, 2026 agent improvement loop cookbook makes this distinction practical. It starts with real traces, adds human and model feedback, turns that feedback into evals, and uses the resulting evidence to propose harness changes.

That shape is the important part.

Trace first. Feedback second. Eval third. Change fourth.

Most teams want to jump from trace to change.

That is where regressions hide.

The harness is bigger than the prompt

One useful phrase from OpenAI’s cookbook is “harness.” In that notebook, the harness means the full contract around the model: instructions, tools, routing, output requirements, and validation checks.

That is a better mental model than “prompt.”

A prompt is one part of the harness.

The harness also includes:

• which tools the agent can call
• when a tool call is allowed
• how tool results are shaped
• when a human approval is required
• how output is validated
• how handoffs are selected
• how state is carried across steps
• what the agent must refuse or escalate
• what counts as a passing answer

If an agent calls the wrong tool, the fix might be tool descriptions.

If it calls the right tool with a weak argument, the fix might be schema design.

If it hands off too early, the fix might be routing logic.

If it writes valid JSON with useless content, the fix might be eval criteria.

If it passes a safety check but still creates risk, the fix might be guardrail placement.

The prompt is not the product. The whole harness is the product.

What a trace-to-eval loop looks like

Here is a practical version.

A trace-to-eval loop turns one agent failure into reusable quality evidence before the harness is changed.

Step one: capture the run.

Do not start with only success cases. Capture real failures, confusing outputs, tool loops, long runs, unexpected refusals, missing citations, incorrect route choices, or cases where the answer looked confident but thin.

Step two: inspect the trace.

Look for the first bad decision, not just the final bad answer.

The final answer may be where the user noticed the problem. The earlier tool call may be where the system created it.

Step three: label the failure.

Use plain labels first:

• wrong tool
• missing retrieval
• bad citation
• unsafe action
• weak refusal
• hallucinated field
• wrong handoff
• invalid format
• excessive tool loop
• ignored user constraint

Do not make the labels too fancy.

Fancy labels are hard to reuse.

Step four: turn the label into an eval.

The eval should answer a narrow question:

• Did the agent choose the right tool?
• Did it cite only supported claims?
• Did it preserve the requested format?
• Did it refuse an unsafe tool call?
• Did it ask for approval before a write action?
• Did it avoid calling tools when no tool was needed?

Step five: change the harness.

Only after the failure is reproducible should you edit the system prompt, tool schema, routing rule, guardrail, evaluator, or approval flow.

Step six: rerun.

A change that fixes one trace but breaks three older examples is not a fix.

It is a tradeoff you did not see early enough.

Use traces to find the first wrong turn

Agent failures often look mysterious because the visible answer is far away from the original mistake.

Example pattern:

The user asks for a deployment summary. The agent searches logs. The log tool returns noisy matches. The agent picks the wrong service name. The handoff agent assumes the service name is correct. The final answer summarizes the wrong incident.

A chat transcript may make the final answer look like the problem.

A trace can show the earlier wrong turn.

That is why trace shape matters. OpenAI’s tracing docs describe traces that include LLM generations, tool calls, handoffs, guardrails, and custom events.

A useful trace review asks:

Where did the agent first lose the task? Was the wrong tool available? Was the right tool unavailable? Did the tool return ambiguous data? Did the agent ignore a constraint? Did a handoff remove context? Did a guardrail run too late? Did the final answer hide uncertainty?

That last one matters.

Many agent failures are not loud.

They are quiet confidence.

Metrics are useful, but they are not judgment

OpenTelemetry’s GenAI semantic conventions are useful because they give teams a more standard vocabulary for GenAI telemetry. The GenAI docs describe conventions for events, exceptions, metrics, model spans, agent spans, and technology-specific systems such as OpenAI, Anthropic, Azure AI Inference, AWS Bedrock, and MCP. The page currently marks the GenAI conventions as Development and describes a transition plan around experimental versions.

That helps with plumbing.

It does not replace judgment.

Token usage, latency, error type, and request counts can tell you a run was expensive, slow, broken, or unusual. OpenTelemetry’s GenAI metrics docs include a recommended gen_ai.client.token.usage metric when token usage is available.

But telemetry cannot automatically tell you whether the answer helped the user.

That is where evals enter.

A practical split:

• Observability: What happened?
• Evaluation: Was it acceptable?
• Experimentation: Did the change improve the right cases?
• Governance: Was the action allowed?

Do not collapse those into one dashboard.

You will end up measuring what is easy and missing what matters.

Be careful with prompt and response capture

Capturing full prompts, completions, tool calls, and tool results can be useful for debugging.

It can also create a privacy problem.

OpenTelemetry’s GenAI events docs say instrumentations may capture user inputs and model responses as events, but those events are opt-in and still in development across some languages.

That is the right caution.

Before storing full content, ask:

• Does this include customer data?
• Does it include secrets?
• Does it include internal documents?
• Does it include regulated information?
• Who can read the traces?
• How long are traces retained?
• Can sensitive fields be redacted?
• Are prompts and tool outputs needed, or are structured summaries enough?

Trace everything is not a strategy.

Capture enough to debug.

Redact enough to sleep.

The minimum eval set for a real agent

A small agent does not need a huge benchmark on day one.

It needs a starter eval set that covers the ways it can hurt trust.

A practical minimum:

Golden task evals

A small set of representative user tasks with expected behavior.

Tool choice evals

Cases where the agent should call one tool, no tool, or ask for approval.

Grounding evals

Cases where the answer must cite or rely only on provided context.

Format evals

Cases where the output must match JSON, Markdown, schema, or ticket format.

Safety evals

Cases where the agent must refuse, escalate, or avoid side effects.

Regression evals

Past failures that should never return.

This is not glamorous.

It is quality control.

Langfuse describes itself as an open-source LLM engineering platform for monitoring, evaluating, debugging AI applications, prompt management, datasets, and traces. That is one example of the kind of tool category teams use to build this workflow, not a requirement to use that specific product.

The workflow matters more than the logo.

A useful failure record

When a run fails, do not only save the bad output.

Save a failure record.

A lightweight record can include:

task: user-facing description of the request
expected_behavior: what should have happened
actual_behavior: what happened instead
first_wrong_turn: trace step where the mistake began
failure_label: wrong_tool | bad_citation | unsafe_action | etc.
harness_area: prompt | tool_schema | routing | guardrail | validation
eval_added: yes/no
fix_candidate: short description

That record turns one annoying bug into reusable engineering knowledge.

The key field is first_wrong_turn.

Without it, people argue about the final answer.

With it, the team can fix the system.

Do not tune on vibes

A common anti-pattern:

1. Run agent.
2. See bad answer.
3. Edit prompt.
4. Try same question.
5. See better answer.
6. Ship.

This feels productive.

It is fragile.

A better anti-pattern detector:

• Did you add the failed case to an eval set?
• Did you rerun older passing cases?
• Did you check tool calls, not just final text?
• Did you check cost or latency impact?
• Did you check whether the fix made refusals too aggressive?
• Did you check whether the answer is more grounded or just more confident?

If the answer is no, you did not improve the agent.

You edited it.

That may still be useful.

But it is not evidence yet.

A prompt change without an eval is just a confident guess.

Where code belongs

A full implementation depends on your stack, so this draft should not pretend one snippet covers every agent framework.

But the structure is simple enough to express:

run agent -> capture trace -> label failure -> create eval case -> change harness -> rerun evals

For a real post, this is where Sukhpinder should add one concrete example from a tool he actually used.

Possible additions:

• one OpenAI Agents SDK trace screenshot
• one Langfuse or Phoenix trace screenshot
• one Promptfoo eval file
• one before/after prompt diff
• one tool schema change
• one failed run converted into a regression test

The practical opinion

Do not start agent reliability work with a better prompt.

Start with a better feedback loop.

This is easy to disagree with because a prompt edit is faster. Sometimes it is enough for a demo, a prototype, or a private workflow.

But once an agent has tools, handoffs, approvals, files, retrieval, or user-facing impact, the prompt is no longer the whole system.

A trace-to-eval loop gives you a way to learn without relying on memory.

It gives your future self a way to ask:

Did this change actually help?

That is the difference between agent hacking and agent engineering.