Every Postmortem I’ve Read Tells Me One Thing First

Before the timeline, before the impact, one section tells me whether it’s safe to be wrong on this team.

Before the timeline, before the impact, one section tells me whether it’s safe to be wrong on this team.

It happens often enough now that I have a routine for it. A postmortem lands in my inbox, a day or two after the dashboards went green and everyone went back to bed, and I do the thing I always do with a postmortem.

I don’t start at the top.

I scroll past the summary, past the timeline, past the customer impact numbers, and stop at the section labeled contributing factors. That is the only part I need to read first. Everything else can wait, because that section tells me the one thing I actually want to know about the team that wrote it.

Whether they went looking for a cause or went looking for a person.

The document is a confession before it is a record

A postmortem pretends to be a record of what happened. It is also, whether the authors intend it or not, a record of how the team handles being wrong.

You can feel the difference in the first sentence of the root cause. “The deploy script did not validate the config, so an empty value reached production”, reads one way. “An engineer pushed a bad config”, reads another. Same outage. Same forty minutes. Two completely different teams.

The first team is describing a system that lets a mistake through. The second team has already found someone to be annoyed at. And once a team has someone to be annoyed at, the investigation tends to stop. Why keep digging when you already have a name?

That is what I am reading for. Not the cause. The appetite for finding one.

The language gives it away before the conclusions do

Before the conclusions, the grammar tells the team.

The most useful word in any postmortem is “should”. When a contributing factor leans on the word should, the investigation has usually stopped at that sentence. “The engineer should have noticed the failed check.” “The reviewer should have caught the missing validation.” Should is the sound of hindsight wearing the costume of analysis. The person in the moment did not have the view I have now, reading calmly two days later with the answer already known. Should imports my hindsight into their past and then holds them responsible for not having had it.

Passive voice is trickier, because it is not automatically a tell. “An empty value reached production” is passive and honest: it describes a state of the system. “Mistakes were made” is passive and evasive: it describes nothing at all. The test is what the sentence does next. Honest passive is followed by a mechanism: reached production because the deploy step did not validate it. Evasive passive is followed by silence, or by a name.

And then there is the counterfactual, the “if only”. If only the config had been checked. If only the alert had fired. Every if only is a small fantasy in which exactly one thing changes and the outage evaporates, and every one of them is a lie, because real failures are never one missing thing. They are five things that each had to go wrong in sequence. The postmortem that fixates on the last link in that chain, the human one, is choosing the most satisfying villain over the most useful explanation.

Human error is not a root cause

This is the part I feel strongest about, so let me be plain about it.

“Human error” is never a root cause. It is the place where an investigation gives up. A human made an error, yes. Humans make errors constantly, all day, on every system that has ever existed. That is the single most predictable fact in all of engineering. Designing as if it were surprising is the actual failure.

So when the contributing factors terminate at a person, what I am looking at is not an explanation. It is an unanswered question wearing the costume of an answer. The real questions are still sitting there untouched. Why was it possible to deploy that config at all? Why did nothing catch the empty value? Why did it take a customer to notice instead of an alert? Why was one tired person at 2am the last line of defense for something that mattered?

A postmortem that names a person answers none of those. A postmortem that names a system answers most of them by construction, because to describe the system honestly, you have to walk through every gate that should have stopped the failure and didn’t.

I joined a team once where every postmortem had a section called “responsible party”. I want to be fair. The people who built that template were not cruel. They thought they were creating accountability. What they had actually built was a machine for teaching everyone to be careful about what they wrote down.

The action items are the other tell

If the contributing factors tell me where a team looked, the action items tell me what they believe will keep it from happening again. That belief is even more revealing than the diagnosis.

Blame cultures produce action items aimed at people. Remind the team to be careful with config changes. Add a line to the checklist. The engineer will double-check values before deploying going forward. Read enough of these and a pattern surfaces: every fix is a promise that a human will try harder next time.

Learning cultures produce action items aimed at the machine. Validate config in the deploy pipeline and fail closed on an empty value. Make the alert that was silently broken page loudly, and add a test that fails if it ever stops firing. Block the deploy entirely when the check does not run.

The difference is not effort, and it is not sincerity. Both teams want the outage not to recur. The difference is that one team believes the answer is more vigilance, and the other believes the answer is less reliance on vigilance. And vigilance is not a fix. You cannot ship more carefulness. The entire reason the system exists is that human attention is finite and fails predictably under load, at 2am, after nineteen hours. An action item that asks for more of the exact thing that already ran out is not a correction. It is the same outage with a calendar reminder attached.

So I read the action items, asking one question. Does this change the system, or does it change a person’s future mood? Only one of those survives the next tired night.

Blame makes people quiet

And quiet is the most expensive thing you can have in a system that fails.

The whole value of a postmortem culture is that people tell you about the near-misses, the weird thing they saw, the shortcut they took that worked but scared them. That information is the early warning system. Blame turns it off. Nobody volunteers the scary shortcut to a team that is going to write their name next to the next outage.

It compounds quietly, too. The first blamed engineer goes a little quieter. The people who watched it happen take the lesson without needing to be named. Within a quarter, the postmortems get shorter, cleaner, and almost entirely useless, because everyone has learned that the document is evidence, and you do not hand evidence to a process that is looking for someone to charge.

The one I got wrong

I have written the bad kind. Early in my career, after a long night, I wrote a postmortem that said, in slightly more polite words, that the on-call engineer should have caught it.

He had been awake for nineteen hours. The alert that should have paged him had been silently broken for a month. I knew both of those things when I wrote it, and I wrote the sentence about him anyway, because it was easier and because some part of me wanted it on record that I would have caught it.

He left the team that quarter. I’m not going to claim my postmortem was the reason, I genuinely don’t know if it was, that would be a tidy story and the truth is probably messier. But I think about that sentence more than I think about almost anything else I’ve written. It was the moment I understood that a postmortem is not a neutral document. It does something to the people named in it.

The obvious objection

The pushback writes itself. Sometimes a person really did make the mistake. Sometimes there was negligence, or someone ignored a process that existed for exactly this reason.

That is true, and the honest postmortem still doesn’t stop at the person. It asks why the process was ignorable. A process that depends on everyone choosing to follow it under pressure is a system with a known failure mode, and the failure mode is human, and you already knew humans were in the loop. If one person ignoring a step can take you down, the step was never really protecting you. The person just made that visible.

Accountability and blame look similar from a distance. They are opposites up close. Accountability asks what we will change. Blame asks who we will be annoyed at. Only one of them makes the next outage less likely.

What I’m really checking

So no, the first thing I read isn’t really the root cause. It looks like that from the outside.

What I’m checking is whether it’s safe to be honest on this team. Because if it’s safe to be honest, the postmortems are real, and if the postmortems are real, the system gets better every time it breaks. And if it isn’t safe, then every document this team produces is partly fiction, and the outages will keep arriving with the same three causes wearing different names.

A postmortem that records what broke is useful. A postmortem that records why the system allowed it is valuable. A postmortem that the person responsible could have written about themselves, without fear, is the only kind that makes a team safe to fail in.

That last one is rare. When I find it, I stop reading for the cause. I already know the most important thing about the people who wrote it. And then, usually, I go read the rest. The timeline is genuinely interesting once you trust the document it’s sitting in.