What Breaks First: A Postmortem Framework for AI Agents in Production

Every team running AI agents in production eventually has the same meeting. Something broke. Nobody’s quite sure why. The logs show a wall of tokens, not a stack trace. And the postmortem template your org has used for a decade — the one built for deterministic systems — doesn’t have a box to check for “the model decided to do something else.”

Five failure classes and a practical postmortem framework for diagnosing AI agent incidents in production.

I’ve spent the last year building and running agentic systems in production — an on-call engineering agent, a multi-agent orchestrator, memory layers for long-running agents — and I keep coming back to the same conclusion: we’re debugging non-deterministic systems with deterministic postmortem tools, and it’s not working.

This is an attempt at a better framework. Not a checklist for writing better prompts — a structural way to think about where agent systems fail, so you can find the failure class before you find the root cause.

Why Traditional Postmortems Fall Apart

A classic incident postmortem assumes a few things: the system does what it was told, the failure is reproducible, and the chain of causation is a straight line from trigger to symptom. Agent systems violate all three.

An agent doesn’t execute instructions — it interprets them, probabilistically, against a context window that changed shape three tool calls ago. The same input can produce different outputs. And the “root cause” is often not a bug at all — it’s a reasonable decision made with incomplete or misleading context, which is a category traditional engineering has no vocabulary for.

That last point matters most. In classical systems, we ask “what broke?” In agent systems, the more useful question is often “what did the agent believe, and where did that belief diverge from reality?”

The Five Failure Classes

After enough incidents, patterns emerge. I’ve found almost every agent failure sorts into one of five classes. Naming the class is 80% of the diagnosis.

1. Context Collapse The agent’s working context no longer reflects the actual state of the world. This happens through context window truncation, stale tool outputs, or memory retrieval pulling in outdated facts. Symptom: the agent confidently acts on something that was true five steps ago and isn’t anymore.

2. Tool Misuse The agent has the right tool but the wrong mental model of it — wrong parameters, wrong assumptions about side effects, or calling a tool that’s technically available but semantically wrong for the situation. This is the agent equivalent of a junior engineer running a destructive command because the man page was ambiguous.

3. Goal Drift Over a long-running task, the agent’s interpretation of the objective shifts. Small reasonable-seeming steps compound into an outcome nobody asked for. This is the hardest class to catch in real time because every individual step looked defensible.

4. Cascading Delegation Failure In multi-agent systems, one agent’s error becomes another agent’s premise. Nobody downstream questions it because from their vantage point, it arrived as ground truth. This is a distinctly architectural failure — it’s about system topology, not model quality.

5. Cost/Latency Blowout Not a “wrong answer” failure — a resource failure. The agent finds a valid but absurdly inefficient path to the goal: excessive retries, redundant tool calls, or reasoning loops that technically terminate but burn budget or SLA doing it.

A Framework, Not a Checklist

Here’s the structure I now use for every agent incident review. It’s deliberately architect-facing — built for the person who has to explain to leadership why this will happen again if we don’t change the system, not just patch the instance.

Step 1 — Classify before you diagnose. Which of the five classes (or which combination) does this match? This alone reframes the conversation from “the AI made a mistake” to “here’s the structural gap that made this mistake possible.”

Step 2 — Reconstruct the belief state, not just the event timeline. At the moment of failure, what did the agent’s context actually contain? Not what you think it contained — what was literally in the window or retrieved from memory. This is usually where the surprise lives.

Step 3 — Identify the decision point, not the error point. The visible failure is rarely where things went wrong. Trace backward to the last point where the agent had a fork and took the wrong branch with plausible-looking justification. That fork is your real incident.

Step 4 — Ask the systems question, not the prompt question. “Could better prompting have prevented this?” is almost always the wrong first question — it treats a systems problem as a wording problem. The better question: “What structural guardrail — validation layer, tool constraint, human checkpoint, memory boundary — would have made this class of failure impossible, not just less likely?”

Step 5 — Write the postmortem in terms of the failure class, not the incident. This is the part that actually compounds. If your postmortems are indexed by failure class instead of by date, your fifth incident of “Context Collapse” should take five minutes to diagnose because you’ve already built the vocabulary and the fix pattern for that class.

Why This Matters Beyond the Incident

The organizations getting AI agents right in production aren’t the ones with the fewest failures — they’re the ones who’ve built institutional memory around why agents fail, structured well enough that failure classes become predictable and, eventually, designed around.

That’s an architecture problem before it’s a model problem. Better base models will reduce the frequency of these failures. They will not eliminate the need for this kind of thinking — because the five classes above aren’t model limitations, they’re consequences of building systems that reason under uncertainty and act autonomously. That’s the job. The postmortem framework has to match the job.

If your team is running agents in production and doesn’t yet have a shared vocabulary for how they fail, that gap is worth closing before the next incident, not after.