The Replay Advantage: How Banks Turn an Agent’s Execution Path Into Audit-Grade Evidence

Photo by Den Harrson on Unsplash

A validation lead asked me something last month that I could not answer cleanly. Her agent had passed every accuracy test she threw at it. Then she asked whether, if she ran the same case again tomorrow, she would get the same work back. Not the same answer. The same work.

I did not have a good answer. Most of us do not.

There is now a number for it. A preprint posted in July 2026 analyzed 4,157 replayed episodes across 719 synthetic compliance and financial data operations groups. In an argument-aware extension covering 570 of those episodes across 190 groups, the decisions the agents reached matched on replay between 94.2 and 95.1 percent of the time. Comforting. Then the author looked at how they got there. Exact tool paths matched only 66.9 to 69.4 percent of the time. Full trajectories, meaning the arguments passed in and the results handed back at each step, matched between 45.0 and 51.5 percent.

Sit with that gap. Stable output, unstable process.

If the answer agrees 95 percent of the time and the route agrees half the time, the outcome was never your control point. It was a coincidence you were grading.

The artifact is a run record, not a log

Most teams already log agent activity. Logs are not the thing.

A log is exhaust. It is written for whoever is debugging at 2am, retained for as long as storage policy says, owned by nobody in particular. What second line actually needs is a run record: a versioned, replayable trace of one execution, treated as a product deliverable rather than a byproduct.

That means it gets the things deliverables get. A schema. A named owner. A retention period tied to the complaint and dispute window, not to the observability bill. And a release gate that fails when it is missing.

Five fields carry most of the weight:

• Tool path: which systems it called, in what order, how many times
• Permissions in force: what it could touch at that moment, not what policy says today
• Retrieval sources: the specific records it actually read, pinned by version
• Model and prompt build: both move underneath you, often without a ticket
• Human approval points: where a person said yes, and what was on screen when they did

None of this is exotic engineering. It is logging with intent, and a decision about who owns the output.

Golden trajectories give change control something to test

Here is the mechanism I would build first, and it is the part almost nobody has.

Freeze a small set of representative episodes for each process. Twenty is often enough. Then re-run that set every time something moves underneath the agent: a prompt edit, a tool swap, a model version bump, a refreshed retrieval index. Score the result on path agreement, not just answer agreement.

Suddenly three broken things start working again. Change control has an actual test to run, instead of assuming changes arrive through a change request. Model validation, the independent check that a system does what it claims, has something it can reproduce rather than re-benchmark. And complaint handling can reconstruct what happened, not restate what was decided.

A golden-trajectory suite is a regression test for behavior. Without one, every prompt tweak is an untested production change.

Be honest about the cost, because it is real. Replay needs storage, it needs compute, and it is only credible if you can pin the state of the tools and the data the agent saw. That last one is the hard part, and it is worth saying out loud in the business case rather than discovering it in month four.

Budget the variance, do not outlaw it

The instinct is to chase determinism everywhere. Resist it.

Forcing a single fixed route through a summarization or retrieval flow buys you brittleness and cost, and protects nothing anyone will ever ask about. Variance is only dangerous where the consequence is.

So set a tiered path budget, the same way you already run latency and error budgets. High consequence flows get a tight band and mandatory replay before release: credit adverse action, sanctions escalation, customer review outcomes, anything a regulator or an ombudsman can demand you reconstruct. Lower consequence flows get a wide band and sampled replay: internal research, drafting, triage, summarization where a human reads the output before it moves.

Which process sits in which band is not a technical call. It is a program decision, made by people who understand consequence, and it should be signed the way a risk appetite statement is signed.

Path variance is not a defect to eliminate. It is a quantity to budget, tier and gate.

This is a capability your GCC should own

One more thing, and it is the part I think is genuinely underplayed.

Run records, golden-trajectory suites and replay gates are a manufacturing capability. They are repeatable, they are measurable, they improve with volume, and they produce something the bank needs continuously rather than once. That is precisely the profile of work a global capability center is good at owning end to end.

Most GCC conversations I sit in are still about moving cost or building talent density. Evidence production is a better claim. It puts the center on the critical path of every agent release, not adjacent to it.

The timing pressure is real, and it is worth exactly one paragraph. In April 2026 the Federal Reserve, OCC and FDIC issued SR 26–2, replacing SR 11–7, and a footnote places generative and agentic AI outside its scope while keeping traditional and non-generative AI models inside. The FSB published 12 sound practices in June and closed consultation on July 22, with a final report due in October. India’s RBI closed comments on its draft model risk guidance on July 24, and it is explicit that a bank’s own independent validation stands notwithstanding any certification a third-party provider supplies. Read together, my inference is that lifecycle evidence is where this lands. Nobody has yet told you what format it takes.

The firms that scale agents fastest over the next two years will not be the ones with the best models. They will be the ones whose second line can say yes, because the work can be replayed and not merely the answer restated. That capability is buildable this quarter, it costs a fraction of what retrofitting it will, and right now the format is still yours to define.

References

Khatchadourian, R., “DFAH-Bench: Benchmarking Observable Agent Instability in Financial Decision-Making”, July 2026. https://arxiv.org/abs/2607.20491

Federal Reserve, “SR 26–2: Revised Guidance on Model Risk Management”, April 2026. https://www.federalreserve.gov/supervisionreg/srletters/SR2602.htm

Financial Stability Board, “Sound Practices for Responsible Adoption of Artificial Intelligence (AI)”, June 2026. https://www.fsb.org/2026/06/fsb-consults-on-sound-practices-for-the-responsible-adoption-of-artificial-intelligence-ai/

Reserve Bank of India, “Draft Guidance on Regulatory Principles for Model Risk Management, 2026”, June 2026. https://www.rbi.org.in/scripts/BS_PressReleaseDisplay.aspx?prid=63006

BankingTransformation #AgenticAI #AIGovernance #GlobalCapabilityCenters #ResponsibleAI #FutureOfBanking #TransformationLeadership