How We Turned 5 Hours of RCA Writing Into 10 Minutes of Review

At 4 AM, after firefighting a production incident for three hours, the last thing any engineer wants to do is open a blank Google Doc and start writing.

Yet that’s the reality of thorough incident documentation. After resolving high-severity incidents, producing detailed Root Cause Analysis documents is essential for organizational learning and preventing future occurrences. In practice, this means spending 5+ hours gathering information scattered across Slack threads, Zoom call recordings, and internal dashboards. Often this happens days after the incident, when critical details have already faded from memory and engineers are juggling their regular work alongside documentation responsibilities.

The quality varied wildly. Some teams produced exceptional RCAs with deep analysis. Others rushed through them, treating them as a checkbox. The 5-Whys section, arguably the most important part for organizational learning, was consistently the weakest. “Why did it fail? Because the pod ran out of memory. Why? Because the limit was too low.” That’s two shallow whys that teach us almost nothing.

Here’s the paradox: the better your incident response process becomes, the more valuable your RCAs should be. But the more incidents you handle, the less time anyone has to write thorough documentation. The learning artifact competes with the learning itself.

That’s why we built RCA-GPT, an AI tool that turns incidents into complete first-draft RCAs in 6–10 minutes. Not to replace the human introspection and judgment that makes RCAs valuable, but to eliminate the mechanical work of gathering information and filling out template structures.

The Information Scattering Problem

A strong RCA goes beyond structure, it captures a minute-by-minute timeline of events, clearly quantifies impact, and validates root causes across systems, processes, and recent changes. It should highlight missed signals, document real-time mitigation steps, identify repeat patterns, and define concrete, owned actions to prevent recurrence. It must also call out gaps in detection, monitoring, or change management, ensuring systemic issues are addressed; not just symptoms. This granularity isn’t bureaucratic; it’s how organizations learn from incidents and prevent recurrence. The challenge is that creating this level of detail requires gathering context from multiple essential sources.

Slack threads capture the real-time decision-making process: what hypotheses the team explored, which approaches were tried and discarded, when the breakthrough insight happened. This context explains not just what was done but why, which is critical for understanding the incident fully. However, these threads span 100 to 200+ messages across multiple linked conversations. Understanding what happened means scrolling through everything, following every thread link, and reconstructing the narrative from timestamps.

Zoom war room recordings preserve verbal discussions that never make it to Slack: the tone of urgency, the reasoning behind choosing approach A over approach B, the moment someone realized the actual root cause. These nuances are essential for complete documentation. But watching or even skimming 30 to 90 minutes or more of recording to extract relevant quotes is time-consuming.

Internal dashboards provide the factual foundation: metrics showing when the issue started, logs revealing error patterns, deployment history establishing what changed. This objective data anchors the RCA timeline with verifiable facts and timestamps.

Past RCAs offer valuable precedent: how we handled similar incidents, what corrective actions worked, patterns we’ve seen before. This institutional knowledge prevents repeating mistakes and ensures consistency. But finding similar incidents requires searching scattered Google Drive folders and reading through multiple documents.

All of this information is genuinely necessary for high-quality RCAs that drive organizational learning. The challenge isn’t that we’re collecting too much; it’s that gathering and organizing this essential information consumed hours before anyone even started the actual analysis. This is exactly the kind of structured, necessary-but-mechanical work that AI systems handle well.

Enter RCA-GPT: Automating the Mechanical Work

That’s where RCA-GPT comes in. It’s an AI system designed to handle the structured, time-consuming parts of RCA creation: gathering scattered information from Slack and Zoom, organizing it into chronological timelines, filling out the required sections in the Razorpay template structure, and generating initial 5-Whys analysis based on similar past incidents.

The goal isn’t to replace human analysis and judgment. It’s to eliminate the 5+ hours engineers spend on mechanical work so they can focus their time on the intellectual work: validating root causes, identifying systemic improvements, and ensuring the lessons actually get applied.

RCA-GPT generates complete first drafts in 6–10 minutes. Engineers then only spend time reviewing, editing, and adding the human insights that only someone who lived through the incident can provide.

How RCA-GPT Works Under the Hood

The current system uses a four-agent pipeline where each agent has a focused responsibility.

Agent 1: Search Query Formulation. Takes the incident summary and formulates an optimal search query for retrieving similar historical RCAs from our vector database. If Agent 2 provides feedback that retrieved documents weren’t relevant enough, Agent 1 reformulates with that feedback incorporated. This loop runs at most twice.

Agent 2: Document Critique. Evaluates the retrieved historical RCAs for relevance to the current incident. Ranks each document on a 0 to 1 scale and decides whether the set is sufficient for style reference. If not, it sends structured feedback back to Agent 1 for another retrieval attempt. This quality gate prevents using irrelevant past RCAs as references.

Agent 3: Style Extraction. This agent does something specific and deliberate. It extracts the writing style from the top-matching RCAs, not their content. How detailed is the Sequence of Events? How deep does the 5-Whys analysis go? How are corrective actions phrased? The output is a style guide that Agent 4 uses as a reference.

This separation is critical. We want new RCAs to read like Razorpay’s best RCAs without copying content from past incidents. Early versions without this separation would occasionally bleed details from a 2023 database incident into a 2025 deployment failure.

Agent 4: Final Synthesis. Takes all collected data (Slack messages, Zoom transcript, timeline), the style guide from Agent 3, the curated similar RCAs from Agent 2, and the official template. It generates the complete RCA using Claude on AWS Bedrock.

The Technical Foundation

The system is built on several key architectural decisions that make it reliable and improvable over time.

ICL-RAG over fine-tuning. Instead of fine-tuning a model on Razorpay RCAs (expensive, requires retraining when we want to improve, prone to overfitting), we retrieve similar past RCAs at query time and use them as in-context examples. This means the system improves automatically as more high-quality RCAs are added to the vector database. No retraining required.

Vector database for institutional memory. Every historical RCA is embedded using all-mpnet-base-v2 (a sentence transformer producing 768-dimensional vectors) and stored in Qdrant. When a new incident occurs, we search for the top 5 most similar past incidents filtered by a similarity threshold of 0.7. This retrieval happens in milliseconds and provides relevant context the AI uses for generation.

We also maintain a separate Qdrant collection for infrastructure knowledge: Kubernetes components, monitoring configurations, database architectures. This gives the system context about our specific stack, not just generic cloud infrastructure.

ML model for 5-Whys quality scoring. A trained neural network (scikit-learn’s MLPRegressor) predicts the quality of generated 5-Whys analysis on a 0 to 1 scale. This scoring helps identify shallow attempts that need deeper analysis. The model was trained on manually labeled historical RCAs where we scored 5-Whys depth and completeness.

Template compliance enforcement. A dedicated validation engine checks every generated RCA against the official template: all sections present, title format correct (Priority Inc #X: [Title]), Sequence of Events using proper Event/Action/Missing/Missed structure, priority classification (P0/P1) accurate. The system targets 95%+ compliance scores.

What Actually Changed

The shift is straightforward but impactful. Engineers spend their time reviewing an RCA instead of writing one from scratch. Once an incident is resolved, RCA-GPT automatically generates a first draft and posts it in Slack, so the context is captured while it’s still fresh. It also suggests initial action items based on the incident data and similar past RCAs, giving teams a strong starting point instead of a blank page.

Template compliance problems largely disappeared. Every generated RCA has all of the required sections because the system enforces it structurally. The shallow 5-Whys problem reduced because the system uses similar past RCAs as depth references and the ML model flags superficial analysis. Context loss went away because the system captures Slack and Zoom data immediately, before details fade.

RCA submission timelines moved from days to same-day. When the draft is already written, the barrier to completing the final document drops dramatically.

The consistency improved because every RCA follows the same structural pattern and depth expectations learned from our best historical examples. Junior engineers now produce RCAs that match the quality bar set by senior engineers.

The Deliberate Limitation

RCA-GPT is positioned as a draft generator, not a final document producer. This was a product decision, not a technical limitation.

RCAs are organizational learning artifacts. The introspection (we should have escalated sooner, this runbook was outdated, the on-call engineer didn’t have dashboard access) requires human judgment from someone who was in the war room. The nuance of what went wrong culturally or procedurally can’t be extracted from Slack messages and Zoom transcripts.

The system generates structure, timeline, factual sequence of events, and suggested 5-Whys and corrective actions. Engineers review, edit, keep what makes sense, discard what doesn’t. The AI handles grunt work. Humans handle thinking.

This positioning has proven important for adoption. Teams don’t feel like the AI is replacing their expertise; it’s removing the tedious parts so they can focus on the insights that actually matter.

The Broader Lesson

This isn’t just about incident documentation. It’s about recognizing which parts of complex knowledge work can be automated and which require human insight.

Writing RCAs involves both mechanical work (collecting information, formatting timelines, filling templates) and intellectual work (analyzing root causes, identifying systemic issues, proposing improvements). Traditional automation struggles with knowledge work because it’s all bundled together. AI systems like RCA-GPT succeed when they cleanly separate the mechanical from the intellectual and automate only the former.

The pattern applies broadly. Any process that combines structured data gathering with unstructured analysis, template filling with creative thinking, or factual documentation with interpretive judgment can benefit from similar approaches. Legal document preparation, medical case documentation, research literature reviews, technical specification writing. These all have mechanical components that AI can handle and intellectual components that require human expertise.

The key is designing systems that augment rather than replace, that handle grunt work so humans can focus on insight work, and that improve continuously by learning from the best human examples.

Looking ahead, we’re exploring and making headway on how Project Viveka (our AI-powered incident investigation system) can enhance RCA-GPT’s 5-Whys analysis. Viveka already investigates incidents in real-time, correlating evidence across observability tools and building causal timelines. By feeding Viveka’s investigation findings directly into RCA-GPT, we can generate deeper, more evidence-based 5-Whys that trace from surface symptoms through intermediate causes to systemic root causes. This integration would combine Viveka’s real-time investigative intelligence with RCA-GPT’s documentation capabilities, creating RCAs that are both thorough and technically grounded.

RCA-GPT is in production at Razorpay, generating first-draft RCAs for incidents across engineering teams. The system continues to improve as more high-quality RCAs are added to the vector database and engineers provide feedback on generation quality.

Building AI for knowledge work? The balance between automation and human judgment is always interesting. What processes are you augmenting? Where are you drawing the line between AI assistance and human oversight?

Editor: Parth Sawhney