The Postmortem Culture That Works
Postmortems are easy to do badly and hard to do well. Here's what I've learned about running them so they actually prevent the next incident.
The Point of a Postmortem
The point of a postmortem is not to assign blame — it's to change the system so the same incident doesn't happen again. That framing sounds obvious, but it's surprisingly hard to maintain in practice, because the easiest thing to write in a postmortem is "engineer X made a mistake" and the hardest thing to write is "our process allowed engineer X to make a mistake, and here's how we're changing the process." The first framing is easier because it identifies a clear actor and a clear action; the second is harder because it requires examining the system that surrounded the actor, which is more work and which implicates decisions made by people who aren't in the room. But the first framing doesn't prevent the next incident, because the next incident will involve a different engineer making a different mistake, and if the process hasn't changed, the outcome will be the same.
A blameless postmortem doesn't mean nobody is responsible — it means the responsibility is understood to be systemic rather than personal. The engineer who deployed the breaking change is responsible for the deploy, but the system that allowed a breaking change to reach production is everyone's responsibility, and fixing that system is the actual work. That distinction is what makes blameless postmortems productive: they focus attention on the system, where changes have lasting impact, rather than on the individual, where changes (firing, retraining) have limited impact on future incidents. A culture that fires engineers for making mistakes is a culture that hides mistakes, because engineers who are afraid of being fired will cover up their errors rather than reporting them, and covered-up errors don't get fixed.
What a Good Postmortem Contains
A good postmortem has five sections: a timeline of what happened, an impact assessment, a root cause analysis, action items, and lessons learned. The timeline is the most important part, because it's the factual record that everything else is built on, and getting it right requires gathering information from everyone involved while the events are fresh. The timeline should include when the incident was detected, how it was detected, what was tried, what worked, what didn't, and when the incident was resolved. Writing the timeline is also the first place where the "why" questions surface — "why did it take thirty minutes to detect this?" — and those questions drive the rest of the postmortem.
The root cause analysis is where most postmortems go wrong, because the temptation is to stop at the first cause that feels satisfying — "the deploy introduced a bug." But the deploy introducing a bug isn't the root cause; it's a proximate cause. The root cause is deeper: why was the bug not caught in testing? Why was the monitoring not alerting on the symptom? Why was the deploy not staged behind a feature flag? Each of those questions has an answer that points to a systemic issue, and fixing the systemic issue is what prevents the next incident, even if the next incident involves a completely different bug. The "five whys" technique — asking "why" repeatedly until you reach a systemic cause — is simple but effective, and it's the technique I use in every postmortem.
Action Items That Get Done
Action items are where postmortems have their real impact, and they're also where postmortems most commonly fail. The failure mode is generating a long list of action items that nobody owns, nobody prioritizes, and nobody tracks — a list that feels productive at the postmortem but that quietly decays into a document nobody looks at again. The fix is to treat action items like any other work: assign an owner, estimate the effort, prioritize against other work, and track to completion. An action item without an owner is a wish; an action item with an owner but no timeline is a suggestion; an action item with an owner and a timeline is a commitment, and commitments are what actually change the system.
The other principle is to be selective about action items. A postmortem that generates twenty action items will complete none of them, because the team can't absorb that much additional work alongside feature development. A postmortem that generates three action items — the three highest-impact, most-feasible changes — will complete all of them, and the system will be meaningfully better as a result. Selectivity is hard because every potential action item feels important in the moment, but the discipline of choosing three forces the team to identify the changes that will have the largest impact on preventing the next incident, which is the entire point of the postmortem.
