Saturday, 3:47. Production has been down a fourth hour. Six people on a call, red eyes, clients in the chat, the finance lead counting: $87,000 and rising.
Monday, 9:00. The CTO, red in the face: who allowed this.
The senior is silent. The mid looks at the floor. The junior whose commit it was is not breathing. A week later the same hole. Again nobody knows anything.
I have seen this scene dozens of times. The team learns nothing, the incident returns, people burn out. The write-up after a fall is not for a verdict. It is so the system does not let the same hand reach the same button next time.
A postmortem is a teardown: what happened, why the system allowed it, what stops a repeat, who owns the step and by when. It is not an interrogation, not a fine, and not a wiki page "for the record".
Without a teardown, Google SRE numbers say about 60% of incidents come back. With a formal paper, about 30%. With steps, names and a check a month later, about 5%. Seventy percent of action items from teardowns never happen if nobody returns to the list. That is the real reason for "but we wrote a postmortem".
Tone first, template second
"Who deployed the bug" breeds silence next Monday. "What allowed the bug to land" breeds a process.
The developer: no time for tests, the deadline. Next is not "your responsibility", it is why the estimate had no room for tests. Items: plus 20% for tests in the estimate, a check in CI, staging mandatory before production. A person could break prod with one action. So the hole is in the system.
If the lead hunts a surname in the room, there will be no culture. First line to an honest "I pressed it": thanks for saying, let us close the hole. Punishing a confession teaches people to hide the next incident.
We write the paper when downtime is over half an hour, data is gone, there is a security hole, the loss is visible, a pile of complaints, an SLA is broken. A small thing that arrives a third time we also tear down. A teardown after a good release happens too: which risks we took on purpose and guessed right. Then the slot stops smelling like court.
The person who fought the fire writes it, or the lead who was in the channel. Not "the guilty one": they will defend, not unpack.
A paper that fits on the table
Below is the frame I carry. The numbers in the example are live: disk, logs, a migration.
# Disk at 100%, prod 2025-12-15
Start: 22:47 UTC. Restored: 03:15. Length: 4 h 28 min. P0.
## Short
The database could not write WAL: the disk was full.
Everyone down. ~100,000 users. ~$87,000. 247 angry reviews.
Root: logrotate vanished after a migration three months ago, logs grew ~20 GB a day, 300 GB died in 15 days.
How we got up: wiped old logs, restarted the database.
To not repeat: logrotate everywhere, alert on disk > 80%, config check after migration.A timeline as a table, from logs and Slack, not from memory a week later. 22:47 errors to the database, 22:52 the alert arrived, 23:15 disk 100%, 23:45 wiping tmp did not help, 00:30 the DBA saw application logs, 01:15 50 GB free and the database breathing, 03:15 we closed.
Five whys, until you hit a missing check, not a surname.
- Why did prod fall? PostgreSQL did not write WAL.
- Why not? Zero disk.
- Why was the disk full? Application logs were not rotated.
- Why not rotated? The logrotate config died in a server migration.
- Why did nobody see the loss? No check of critical configs after a move, and no alert on disk fill.
Root: a migration with no automatic check. Next to it, helpers: the DBA not on the first on-call line, no runbook for "disk full", nobody looked at logs for three months.
What worked goes in too. The on-call answered in five minutes against a fifteen-minute SLA. The DBA escalation did not get lost. Clients got the truth on the status page. Otherwise the teardown reads as a flogging.
Items on four shelves, each with a name, a date, a status.
Prevent: turn logrotate on, alert at 80%, config check after migration, DBA on the first line.
See it earlier: a disk dashboard, a weekly look at log growth, an alert on more than 10 GB a day.
Fight faster: a "disk full" runbook, a cleanup script, a drill on staging.
Learn: a teardown with the team, a slice in onboarding about monitoring.
Without a name and a date an item is a wish. A team cannot carry more than seven or eight live items. Three closed beat fifteen in a backlog of shame.
I write the summary so someone without the stack can read it. The business reads only that. Then links: the thread, the dashboard, the logs, the status. Sign-off from the lead, the on-call, the manager. For a P0, a 60-90 minute talk with the people who fought: summary, timeline, five whys, items, priority, names. The moderator cuts "it was Vasya". They rephrase: what in review and in monitoring let it through.
While memory is still warm
Right after we are up: a stub of the paper, a timeline from logs, dashboard shots, the thread, commits next to the hole. Logs have a habit of rotating exactly when you need them.
22:47 monitoring: errors to the database
22:52 on-call took the alert (SLA 15 min, fact 5)
23:15 disk 100%
23:45 wiped tmp, no space
00:30 DBA: application logs, 20 GB/day
01:15 50 GB free, database breathing
03:15 replication live, incident closed
What worked that night: the on-call did not sleep through it, the DBA escalation did not get lost, clients got the truth. What did not: three months without rotation, a disk with no alert, DBA not on the first line, no runbook for "disk full". If you write only the second list, the teardown smells like a flogging and next Monday they will be silent again.
The meeting in a day or two, not in a month. In a month everyone is sure it was the neighbour's release.
After the meeting the author updates the doc in a week, sends it to the engineering feed, wider if it was critical. Then a weekly walk of the items: done, in progress, planned, stuck (and why). Stuck more than two weeks: escalate. Target: more than 80% of items closed in a month, average age two to three weeks, a repeat of this hole zero.
Once a quarter I look at a pile of teardowns together. The same root three times is no longer an incident. It is a hole in the process. A runbook you wrote and never ran on staging will not help at three in the morning. After the disk we filled a volume on a copy and checked: the alert came, the on-call understood, the script ran, the runbook text is readable by a sleepy person.
A public write-up for clients is right when you have already lied to them with silence. An honest internal one matters more than a pretty external one.
How people usually spoil it
They write it in half an hour to close a ticket and drop it in the wiki. They hunt a surname. They collect twenty items and do none. They do not count whether the hole came back. They meet a month later, when the timeline is already a myth.
Public write-ups from big shops I read for the move, not the moral. In 2017 an AWS typo in a command took down too many S3 servers: the tooling no longer lets you pull the whole pen at once. At GitLab an admin killed prod instead of staging, backups were untested, they lost six hours of data and published everything, including the chat. At Knight Capital old code woke up after a crooked deploy and burned $440 million in 45 minutes. In all three the hole was a system that let one motion through without a belt. The engineer's surname fixed nothing.
Four things are worth counting. Minutes from fall to standing. How often we fall. What share of incidents already sat in older papers. What share of items is alive a month later. If the teardown starts three weeks later, you are reviewing a legend.
A rollout starts with one template and the next fall. Not with a course and not with a five-figure platform. You take the next P0 through the frame above, on Monday you do not ask "who", on Friday you open the item list. If the items are alive, the culture has already started. If the paper went back into the wiki and the disk filled again, you bought yourself another Saturday at 3:47.
