Harness Engineering: The Retry That Booked Marta’s Flight Twice V

Part 5: Why safe agent recovery is not “just retry,” and how idempotency guards, durable checkpoints, resume policies, and replay streams prevent one failed run from becoming a second real-world side

The scariest agent failures are not the ones where the model gets confused. They are the ones where everything looks correct, the retry fires automatically, and the customer pays for the same mistake twice.

In this article, you will learn why wrapping an agent loop in try/except is not recovery, and what to build instead. We cover the idempotency guard that records intent before any side effect fires, durable checkpointing in both frameworks, the resume-branch-abandon policy you decide before an incident rather than during one, and the replay stream that turns your checkpoints into a debugger. By the end, you will know exactly where deduplication lives and why the framework cannot do it for you.

Part 5 of “Harness Engineering, Two Frameworks,” an eight-part guide to the jobs a harness does that the model cannot, was built twice in the Claude Agent SDK and LangChain Deep Agents.

Please consider subscribing to my Substack newsletter. If this kind of practical, production-minded AI work is useful to you, a subscribe, like, comment, or share really helps grow the channel and supports the work.

Share

Your agent’s retry logic can be perfectly correct and still result in a double charge to a customer. Here is why naive retry breaks a non-idempotent agent, and the small set of controls that fix it in both the Claude Agent SDK and LangChain Deep Agents.

Marta booked a flight to Barcelona at 10:42 a.m. She watched the spinner turn for thirty-eight minutes while the agent worked through her constraints. Then the container hosting the harness restarted on a routine cloud-scheduler preemption, and her partial progress was gone. The retry was automatic. It was also catastrophic: the book_flight call from the first run had already succeeded and charged her card before the crash. The retry charged her again. Two seats, two confirmation emails, one thirty-minute phone call with the airline’s billing department to undo it.

Here is the uncomfortable part. The retry logic was correct on its own. Nothing in it was buggy. The missing piece was a record of what had already happened. This article builds that record, and the recovery policy that uses it, the same way twice: once in the Claude Agent SDK and once in LangChain Deep Agents.

Why “just retry” fails

The night-before-release instinct is to wrap the loop in try/except and re-run. That works for stateless functions. A multi-step agent is not a stateless function. Run search_flights, the model picks one and calls book_flight, the container crashes mid-write on the confirmation step. A naive retry re-runs search_flights, then calls book_flight again. The reservation system is stateful. It already received the first call. The second book a second seat. The problem is not the retry. The problem is that the state has already drifted by the time the retry fires.

Pat Helland named this in 2012: idempotency is a non-negotiable prerequisite for retry in loosely coupled systems. A retry that re-executes side effects is not a recovery; it is a second execution of the same harm.

The problem also compounds. With five sequential tool calls, each at 95% success, the run succeeds 77% of the time. At twenty steps, 36%. Every non-idempotent step multiplies the blast radius of a careless retry. The field is littered with evidence: an onboarding agent that sent the same email fourteen times in nine minutes after a retry loop fired during a network partition, and a CRM agent that looped for sixty-three hours and $4,200 while retrying an upsert against 429 responses. The retry code was fine in both. Nobody recorded which steps had already succeeded.

The fix is one ordering rule:

Write the checkpoint before the side effect fires. Record the result after it lands. On retry, if the checkpoint already holds a result, skip the call.

The idempotency guard

Here is the load-bearing control; it is framework-agnostic because neither framework knows that book_flight is non-idempotent. The guard wraps any side-effecting call. It records intent before the call, the result after, and refuses to blindly re-fire a call that was in flight when the process died.

import json, hashlib
from pathlib import Path

class NeedsReconcile(Exception): ...

  def _ckpt(session_id: str, step: str) -> Path:
    return Path(f"checkpoints/{session_id}/{step}.json")

  def idem_key(session_id: str, step: str, args: dict) -> str:
    raw = f"{session_id}:{step}:{json.dumps(args, sort_keys=True)}"
    return hashlib.sha256(raw.encode()).hexdigest()[:32]   # stable intent fingerprint

  def guarded(session_id: str, step: str, fn, args: dict):
    """Run a non-idempotent side effect at most once across retries."""
    path = _ckpt(session_id, step)
    if path.exists():
        rec = json.loads(path.read_text())
        if rec["status"] == "done":                        # ① already succeeded
            return rec["result"]                           #    skip; return recorded result
        raise NeedsReconcile(step, rec["idem_key"])        # ② in-flight at crash: reconcile,
                                                           #    do not blindly re-fire
    key = idem_key(session_id, step, args)
    path.parent.mkdir(parents=True, exist_ok=True)
    path.write_text(json.dumps({"status": "in_flight", "idem_key": key}))  # ③ BEFORE
    result = fn(**args, idem_key=key)                      # ④ pass key to the downstream API
    path.write_text(json.dumps({"status": "done", "idem_key": key,
                                "result": result}))        # ⑤ AFTER it lands
    return result

① A completed step returns its recorded result instead of running again; this is what spares Marta the second charge. ② A step that was in flight at crash time is not retried blindly, because the side effect may have landed before the process died; it goes to reconciliation. ③ and ⑤ are the ordering rule in code: intent first, result last. ④ passes an idempotency key downstream so APIs that support deduplication (the Stripe pattern) catch the duplicate too. Wrap book_flight in guarded and the double-booking cannot happen, in either framework.

Note: The full extracted listing at code/harness_engineering/part-5-recovery/listings/01-idempotency-guard.py shows a runnable version with a demonstration book_flight and crash/resume harness.

The three states the guard moves through, and what a crash does in each, are easier to see laid out:

Read as a single call across a crash, the same logic looks like this: the guard checks the store, and either returns a recorded result, fires once and records, or raises for reconciliation.

The guard handles the side effect. The next question is what makes resuming the rest of the run cheap, and that is where the two frameworks diverge.

Durable state under the guard

In Deep Agents (on LangGraph), durability is built in. A checkpointer writes a state snapshot at every super-step, so a crashed run resumes from its last good step instead of the beginning. Use InMemorySaver for tests, SqliteSaver for a single process, and PostgresSaver for anything that must survive a container restart, which is exactly Marta’s case.

from langgraph.checkpoint.postgres import PostgresSaver
from deepagents import create_deep_agent

with PostgresSaver.from_conn_string("postgresql://localhost/agents") as cp:
    cp.setup()
    agent = create_deep_agent(
        model="anthropic:claude-sonnet-4-6",
        tools=[search_flights, book_flight],   # book_flight is wrapped by `guarded`
        checkpointer=cp,                        # ① per-step durable state in Postgres
    )
    config = {"configurable": {"thread_id": "marta-1042"}}
    # The first run crashes after book_flight. On restart, re-invoke the SAME thread:
    agent.invoke({"messages": [{"role": "user",
        "content": "Continue Marta's booking."}]}, config)  # ② resumes from last checkpoint

① The checkpoint survives the crash in the database. ② Re-invoking the same thread_id resumes from the last committed step; the steps that already succeeded are not re-run. A useful detail: when parallel steps run and one fails, the successful sibling is still checkpointed, so partial progress is preserved.

Note: The full extracted listing at code/harness_engineering/part-5-recovery/listings/02-deepagents-postgres-resume.py shows the tool stubs and guarded wrapping elided here.

The Claude Agent SDK persists the conversation, not arbitrary graph state, and exposes recovery through sessions. Capture the session_id, and resume continues that conversation. The SDK does not know book_flight is non-idempotent, so side-effect deduplication stays the guard’s job. What the SDK adds on top is conversation resume and, separately, file rollback.

import asyncio
from claude_agent_sdk import query, ClaudeAgentOptions, ResultMessage

async def run(prompt: str) -> str | None:
    session_id = None
    async for m in query(prompt=prompt,
                         options=ClaudeAgentOptions(allowed_tools=["Read", "Write"])):
        if isinstance(m, ResultMessage):
            session_id = m.session_id          # ① capture for later resume
    return session_id

async def resume(session_id: str):
    async for m in query(prompt="Continue where you left off.",
                         options=ClaudeAgentOptions(resume=session_id)):  # ② same conversation
        if isinstance(m, ResultMessage):
            print("resumed:", m.subtype)

① Every ResultMessage carries the session_id. ② resume=session_id reconstructs the conversation so the agent continues with full context. For files the agent wrote, the SDK adds file checkpointing: it backs up files before Write/Edit/NotebookEdit and lets you call rewind_files() to restore a known-good state on disk. Note the boundary clearly: file rewind restores files, not the conversation, and the guard, not the SDK, prevents the second charge.

Note: The full extracted listing at code/harness_engineering/part-5-recovery/listings/03-claude-sdk-session-resume.py shows a runnable asyncio.run entry point around these two coroutines.

The recovery triad: resume, branch, abandon

A checkpoint gives you something to recover from. The policy decides how. There are three options, and the selection is mechanical, not a 2 a.m. judgment call.

• Resume continues from the last checkpoint. Use it for transient failures: timeouts, restarts, cleared rate limits. This is Marta’s case: book_flight has a recorded result, so resume skips it and finishes the confirmation step.
• Branch forks a new run from a prior checkpoint to try a different path without disturbing the original. Use it for debugging and alternative exploration.
• Abandon discards the in-flight state and starts clean. Use it for corruption, fatal logic errors, or budget exhaustion, where resuming would only re-enter the bad state.

Encode the choice so it is made before production, not invented during an incident.

from enum import Enum

class Recovery(Enum):
   RESUME = "resume"; BRANCH = "branch"; ABANDON = "abandon"

   def select_recovery(error_type: str, budget_exhausted: bool,
                    checkpoint_corrupt: bool, is_debugging: bool) -> Recovery:
    if checkpoint_corrupt or budget_exhausted:        # ① non-recoverable by definition
        return Recovery.ABANDON
    if is_debugging:                                  # ② explore without disturbing the run
        return Recovery.BRANCH
    if error_type in {"timeout", "rate_limit", "network", "restart"}:
        return Recovery.RESUME                        # ③ transient -> resume
    return Recovery.ABANDON                           # ④ unknown/non-transient -> escalate

① Corruption and budget exhaustion cannot be resumed; the state that caused them is the problem. ② Debugging branches. ③ Transient failures resume. ④ Anything else escalates rather than retries, which is the rule that would have stopped the sixty-three-hour loop.

Note: The full extracted listing at code/harness_engineering/part-5-recovery/listings/04-select-recovery.py shows the same policy with a small driver that exercises each branch.

The whole decision is a short tree, and the point of writing it down is that an on-call engineer never has to guess:

Both frameworks implement all three. Branching is the one worth seeing paired, since it is how you explore a fix without losing the run that got you somewhere useful.

# Claude Agent SDK: fork leaves the original session untouched.
options = ClaudeAgentOptions(resume=session_id, fork_session=True)  # new session id

# LangGraph (Deep Agents): update_state writes a diverging checkpoint from a prior config;
# delete_thread is the abandon path.
agent.update_state(config, {"messages": [alt_message]})  # branch from a checkpoint

Resume is the default path, branch is for exploration, and abandon is for the unrecoverable. The triad is policy. Decide it now.

Replay: the checkpoint stream is also your debugger

One more payoff falls out for free. Because each checkpoint records the tool inputs and outputs, the model version, and the transcript hash, the same stream that powers recovery lets you replay an incident offline. You reconstruct exactly what the agent saw at each step instead of guessing from a log narrative, and you can confirm a fix against the real sequence before shipping it.

Recording the model version matters here. A replay can detect when the model shifted underneath a run, which is otherwise an invisible source of “it worked yesterday.”

Do this today

• Wrap one non-idempotent tool in guarded. Run it, kill the process right after the call, and re-run. Confirm the recorded result returns and no second side effect fires.
• Force an in-flight crash. Kill the process between the intent write and the result write. Confirm the guard raises NeedsReconcile instead of blindly re-firing.
• Drive select_recovery through each branch. Feed it a rate_limit error, then a corrupt checkpoint, then a budget-exhausted flag. Confirm resume, abandon, abandon.
• Pick your durable store deliberately. Use InMemorySaver in tests, but move to PostgresSaver (Deep Agents) or captured session_id resume (Claude SDK) for anything that must survive a container restart.
• Add the model version to every checkpoint record. It is one field now, and it is the only thing that will tell you a run broke because the model shifted under it.

The bug was never the retry

Marta’s agent did not fail because the model was wrong. It dispatched exactly the right booking. It failed because nothing recorded that the booking had already happened before the retry fired. The guard records it. The checkpointer and session resume make continuing cheap. The triad decides resume versus branch versus abandon. And the same checkpoint stream replays the incident when you need to debug it.

The lesson generalizes past flights and charges. Any time an agent takes an action, the outside world remembers, your retry is only as safe as your record of what already happened. Write the checkpoint before the side effect, decide the recovery policy before the incident, and retry booking Marta’s flight twice to book it exactly once.

This is Part 5 of “Harness Engineering, Two Frameworks,” an eight-part guide to the jobs a harness does that the model cannot, built twice in the Claude Agent SDK and LangChain Deep Agents.

If this helped you, please consider subscribing to my Substack newsletter.

Likes, comments, and shares really do make a difference. They help grow the channel, support the work, and get these ideas in front of more builders who are trying to understand AI agents, harness engineering, and production AI systems.

Thank you for reading and for helping the work reach more people.

Share

Leave a comment

Share Hightower’s AI Harness Engineering

About the Author — Claude Certified Architect

Rick Hightower is a former Senior Distinguished Engineer at a Fortune 100 company, focusing on delivering ML / AI insights to front-line applications, and a practitioner building multi-agent production systems. Follow him on SubStack and Medium for more hands-on agent engineering content. You can also book him to speak and train your team: Check out Rick Hightower’s SpeakerHub.

Rick Hightower helps companies become AI-first through practical mentoring, executive and team training, and custom AI solution development. He is a former Senior Distinguished Engineer at a Fortune 100 company, where he focused on bringing ML and AI insights into real front-line business applications.

Subscribe to Rick’s newsletter to see videos and guides.

Rick is a Claude Certified Architect, AI systems practitioner, and builder of production multi-agent systems. He is currently working on authoring a book on Harness Engineering with Manning Publishing. He created Skilz, a universal agent skill installer supporting 30+ coding agents, including Claude Code, Gemini, Copilot, and Cursor, and co-founded one of the largest agentic skill marketplaces.

Today, Rick and the Spillwave team work with leaders and teams who want to move beyond AI experiments and build real AI capability inside their companies. He helps organizations adopt AI safely, train their people, redesign workflows, and build practical AI systems that create measurable business value.

Ready to make your company AI-first? Connect with Rick on LinkedIn, Substack, or Medium, book him to speak or train your team, or visit Spillwave to explore mentoring, training, and custom AI solutions for your organization.

Originally published at https://rickhigh.substack.com.