I Analyzed 200 Production Incidents. 80% Could Have Been a 40-Line Script.

Not the same script. The same shape of script. Forty lines that almost nobody had bothered to write because everyone was too busy debugging instead.

It started as a boring project.

The CTO at a company I was consulting for asked me to read every post-mortem from the last two years and tell her what the team should invest in. She expected a list of fragile systems. Refactor this. Rewrite that. Hire here.

I read 200 incidents. I expected to find ten fragile systems. I found something worse and more useful, which is that almost all of them — about 80%, which I’ll show you the math on — were not really separate incidents. They were the same thing wearing different costumes.

And the thing they had in common was a script almost nobody had bothered to write. Roughly forty lines. Boring. Unsexy. Would not get anyone promoted. Would have prevented or contained 160 of those 200 incidents, including two that made the news inside the company.

I want to walk through what 200 incidents actually look like in aggregate, because almost nobody gets to read 200 in one sitting, and the view from there is different from the view from inside any one of them.

How I read 200 post-mortems

Two months. A spreadsheet. Every incident got a row.

Columns: date, system affected, time to detect, time to resolve, dollars lost (where they’d estimated it), root cause in their own words, root cause in my words after reading the timeline, and one column I added partway through that ended up being the only one that mattered.

I called it “what would have caught this in under a minute.”

For every incident, I wrote down the smallest possible monitoring check that would have caught it before the page. Not “better monitoring.” Not “improve observability.” The literal sentence: if there had been a script doing X every Y seconds, this incident would have been a sixty-second alert instead of a three-hour outage.

By incident 30 I noticed something I did not believe yet. By incident 80 I was sure. By incident 200 I had counted.

162 of 200 incidents — 81% — had the same column entry, in shape. A scheduled check comparing two numbers that should agree, alerting on divergence. That was it. That was the script. Forty lines or fewer in every language. The two numbers changed per incident. The shape did not.

The shape of the script

Before I tell you why this works, let me show you what the script actually is, because the surprise is how unsexy it is.

The script does one thing. On a schedule — usually every minute or every five minutes — it asks: does what should be equal still equal?

The two numbers vary by system, but the categories are repetitive. Across the 162 incidents this shape applied to, it was always one of:

• Messages produced upstream vs. messages provably processed downstream
• Orders placed vs. orders with a terminal status (paid, fulfilled, refunded, failed — anything terminal)
• Users who reached the “confirm” step vs. users who received a confirmation
• Rows inserted into a table vs. rows visible in the read replica
• API requests accepted vs. API requests that produced a logged result
• Cache writes vs. expected cache reads of those keys
• Money charged to customers vs. money the ledger says was charged
• Jobs scheduled vs. jobs that emitted a “done” or “failed” record

The script counts the left side and the right side, in a defined window, and alerts when the gap exceeds a tolerance you derive from normal noise.

That’s it. That’s the script. The implementation is, in most languages, between twenty and forty lines depending on how you query the two sources. The runbook for what to do when it fires is another twenty or thirty lines. So call it a hundred lines total, including the alert wiring, generously.

A hundred lines that, across those 200 incidents, would have changed the outcome of 162 of them.

Why the script catches what dashboards don’t

This is the part that took me 200 incidents to fully believe, even though I’d been telling people some version of it for years.

Dashboards monitor symptoms you have already imagined. Error rate up. Latency up. Queue depth up. CPU pinned. Each panel exists because somebody, at some point, said “this could go wrong” and built a panel for it. The dashboard is a museum of failures you already anticipated.

The script doesn’t monitor symptoms. It monitors equalities that must hold for the system to be correct. And here’s the structural difference, which is the entire reason the script catches things dashboards cannot:

A symptom-based monitor fires only if you predicted the failure mode. No prediction, no panel, no alert. The novel failure — the one you’ve never seen — produces no symptom you’re watching for.

An equality-based monitor fires whenever the equality breaks, regardless of why. You did not have to predict the failure. You only had to know that two numbers must agree. When they stop agreeing, the alert fires, and you do not need to have imagined the cause in advance.

In the 162 incidents this would have caught, the dashboard was green during all of them. Not green-and-ignored. Green-and-accurate, because the panels were measuring things that were genuinely fine while the equality was silently breaking somewhere the panels could not see.

Read that twice. The dashboard was correctly green while the system was failing. The failure was not in anything any reasonable panel would have shown. The failure was in the gap between two numbers nobody had thought to compare.

What the other 38 incidents looked like

I want to be honest, because “80% of incidents” sounds like marketing, and I don’t want to pretend the script is a silver bullet.

The other 38 incidents — 19% — were genuinely something else.

Some were security incidents where the issue was deliberate, hidden, and not subject to an “should this number equal that number” check. A leaked credential is not a reconciliation gap.

Some were performance incidents where the system was technically working correctly but slowly enough that customers experienced it as broken. Equality held; latency didn’t. A different category of monitoring catches those, and most teams already do.

Some were genuine novel hardware or infrastructure failures — a cloud provider had a bad afternoon, a network partition was real — where no script in our codebase could have helped. Those are the cost of running on top of physical machines you do not own.

And a handful — maybe six of the 200 — were the kind of weird, specific, never-going-to-happen-again incidents that every long-running system accumulates. A leap-year date bug. A character encoding issue that surfaced only when a customer in one specific locale submitted one specific kind of input. Those don’t have a script.

But the other 162 were the script. Different equalities. Same shape. Same forty lines.

Why almost nobody writes the script

The script is not a secret. The pattern has names — reconciliation check, end-to-end audit, conservation invariant — depending on which subculture you ask. There are books that mention it. There are blog posts older than my career that recommend it.

Almost nobody writes it. I went looking, in the 200 incidents, for cases where the team had attempted some version of it. I found maybe a dozen. Of those, half were broken in production at the time of the incident and the team didn’t know, because no one had bothered to alert on whether the reconciliation check itself was running.

There are three reasons I think this happens, and they all matter because they are also why your team probably doesn’t have one either.

It’s boring. A reconciliation script does not look impressive on a resume. Nobody gets a promotion for shipping “a forty-line cron job that compares two numbers.” Engineers ship the interesting thing. The interesting thing is rarely the high-value thing. Boring and high-value is the entire ecological niche the script lives in, and the incentive structure of most engineering orgs makes that niche uninhabited.

It feels redundant. “We already have monitoring.” Yes, you have monitoring of symptoms. You don’t have monitoring of equality. The team that has both is rare. The team that thinks they have both because they have one of them is the majority. The dashboard’s confident greenness is the thing that argues against writing the script, and it argues persuasively until the day the dashboard is green and the system is on fire.

It requires admitting your dashboards are insufficient. This is the political reason and it is the strongest one. To justify writing the script, someone has to say out loud “our current monitoring would not catch the next bad incident.” Saying that out loud is a critique of whoever owns the dashboards. In most organizations, this is a senior engineer or a platform team, and they do not love being told their work has a gap. So the script doesn’t get prioritized, and the next silent incident happens, and the post-mortem says “missing detection,” and a panel is added to the dashboard, and the underlying gap remains.

What I told the CTO

I went into the readout meeting with my spreadsheet and one slide.

The slide had the number 162 on it, and the words “the same script would have caught all of these.”

She read it. Then she said, and I will remember this for the rest of my career: “We’ve been writing a sentence in our post-mortems for two years that I never connected. ‘Detection took N hours.’ Every time. We’ve been measuring our own failure to detect for two years and treating it as an act of God.”

That is the entire thing. The script is not a technical breakthrough. It is a structural acknowledgment that detection is the bottleneck, not resolution. The 200 incidents averaged something like 11 minutes to fix once understood. The same 200 averaged something like 80 minutes to detect from the moment things started going wrong. The script does not make you faster at fixing. It collapses the detection window, which is where the cost actually lives.

Her team shipped the first reconciliation check inside two weeks. It found a real divergence in the first month. Not a big one. A small, ongoing, silent data inconsistency that had probably been bleeding for years and that no symptom-based monitor had ever shown. They fixed it. Then they wrote the second check. And the third. The script proliferates once a team understands what it is.

What I would tell you if you have not written it yet

Pick one critical equality in your system. Just one. The one where, if it broke silently, you would lose money or trust or both.

For most backend teams, it is one of three: requests we accepted vs. requests we provably completed, charges we recorded vs. money the payment provider says we collected, or messages we received vs. messages we provably processed.

Write the script this week. Not next quarter. This week, while it is small. Forty lines. A scheduled job that compares the two numbers in a defined window, alerts on divergence beyond normal tolerance, pages on it. Treat the alert with the seriousness of a production-down page, because by the time it fires, you are losing the thing you most need to protect, and you only have minutes before someone outside the team notices.

When that script catches its first real divergence — and it will, probably within the first month, because the gap was almost certainly already there — you will understand why the 162 incidents I read had the same column entry. You will also become the engineer who installed the highest-leverage monitoring on the team, even though it is forty lines and looks like nothing, and that fact is going to matter to your career more than the impressive thing you were going to ship instead.

The harder lesson

The hardest part of this for me, personally, was not the technical realization. I had been telling people about reconciliation checks for years. I had written some myself.

The hardest part was reading 200 post-mortems and realizing that, in maybe four of them, the team had explicitly identified the missing check during their post-mortem and still not built it. The action item said something like “consider adding reconciliation between X and Y.” It went into the backlog. It died there. The next incident was the same shape and the post-mortem said the same thing and nothing got built.

Engineers know about this script. Teams know. Tech leads know. It still doesn’t get built. The technical knowledge is not the bottleneck. The bottleneck is whether anyone is willing to spend two days on something that does not look impressive in exchange for not losing eight hours to detection lag every quarter.

The math of that trade is obviously favorable. The politics of that trade, inside most engineering orgs, are obviously unfavorable. That is why the script does not get written. Not because no one knows about it. Because the people who would write it are busy doing the work that gets them promoted, and the work that prevents the next outage they will be blamed for, oddly, isn’t it.

If you are senior enough to choose what gets built, the script is the highest-leverage thing on your list and it does not look like the highest-leverage thing on your list. That gap — between what is actually valuable and what looks valuable — is the gap most production systems are quietly losing to, every quarter, in the incidents nobody connects.

Mine connected at 162. Yours might be lower if you start counting now.

I keep the full pattern library — the eight equalities the script applies to, the exact shape of each one, what to compare against what, the alerting thresholds I’ve watched work and not work — in the playbook I built from these 200 and the years before them.

**The Production Incident Playbook — 80+ Real Incidents Across 9 Documents →** — every entry has the equality, the divergence pattern, and the script shape that would have caught it. $99 for the bundle, $174 if you bought the parts separately, and the whole thing exists because the script you should be writing this week is the same script for 162 incidents that have already happened to teams who didn’t write it.

The longer monthly teardowns — one real incident apart in full, including the reconciliation check that ends it permanently — live on Substack:

**substack.com/@devrimozcay1 →**

[embed]Devrim's Engineering Notes | Substack Founder @ProdRescue AI | https://www.prodrescueai.com/ | I analyze real production failures. What breaks, why it…substack.com

The first teardown is the silent-consumer incident, which is incident 1 of 162. Same shape. Same forty lines. Read in twenty minutes, prevent the version of it that’s already scheduled in your codebase.

I write about what actually happens in production — including the script you should have written six months ago, that you still haven’t, that you know about, that I know you know about, and that will still not be there the next time the dashboard is green and the system is on fire.

What’s the equality in your system that, if it broke silently, would cost the most? Tell me in the comments. Mine took me 200 incidents to name properly. Yours can take less time than that.