Anatomy of Famous Postmortems — Learning From the Failures of Cloudflare, Fastly, AWS, Knight Capital, and GitLab

Prologue — "Failure Is the Best Teacher"

April 2025, your service went down at 3 a.m.

Cause: unknown
Recovery: 2 hours
Person responsible: whoever committed last night
Mood: "who did this?"

This is Blame culture — an environment where you cannot learn from failure.

The opposite: world-class companies publish their postmortems. Cloudflare, Google, AWS, GitLab. The transparency of "here is how we broke it, and here is what we learned" teaches the entire industry.

If Season 3 Ep 1 looked at the success architectures of these companies, Ep 2 learns from failure.

This piece is the practical lessons extracted from published postmortems. Let us avoid the same mistakes at someone else's expense.

Part 1 — What a Postmortem Is

Definition

Postmortem (after-the-fact analysis): documenting the cause, impact, timeline, and lessons after an incident.

The Five Elements From the Google SRE Book

1. Incident summary
2. Impact (who, how much)
3. Root cause (technical + process)
4. Timeline (minute by minute)
5. Action items (concrete)

The Core of a Blameless Postmortem

Bad:  "Alice pushed a bad config"
Good: "Config validation was missing.
      Alice's change getting through is a gap in the system."

Philosophy: fix the system, not the person. Blame people and the next one hides it.

Part 2 — Cloudflare 2022-06-21 — Half of It Went Down Over BGP

What Happened

June 21, 2022, 06:27 UTC
19 Cloudflare data centers (half of the world's traffic) down
Customer services affected for 1 hour 30 minutes

Cause

A mistake while changing BGP routes:

Cloudflare was standardizing its network
BGP configuration rolled out in some data centers
Config change: some prefixes stopped being advertised → traffic had nowhere to go

Why All of It?

A central control plane propagates the config
A structure where many DCs are hit at the same time
Fast propagation with no canary

Lessons

A config change is a deploy too: canary is mandatory
The double edge of central control: speed vs. blast radius
BGP is dangerous: a mistake plays out at internet scale
A transparent postmortem: Cloudflare published within hours

Action items (as announced by Cloudflare)

Redesign the config change workflow
A stronger staging environment
Enforce progressive rollout
Improve alerting

Part 3 — Cloudflare 2019-07-02 — A Regex Took CPU to 100%

What Happened

July 2, 2019
Cloudflare WAF rule update → CPU at 100% worldwide
HTTP 50x elevated for 27 minutes

Cause — Catastrophic Backtracking

The regex that was in the WAF rule:
(?:(?:\"|'|\]|\}|\\|\d|(?:nan|infinity|true|false|null|undefined|symbol|math)|
`|\-|\+)+[)]*;?((?:\s|-|~|!|{}|\|\||\+)*.*(?:.*=.*)))

Triggers "catastrophic backtracking"
Runs in exponential time on some inputs
→ CPU 100% → HTTP cannot be served

Lessons

Regex is dangerous: once it gets complex, ReDoS attacks become possible
A WAF rule is a global change: one edit hits the whole world
CPU testing is required: latency + throughput too
Kill switch: turn it off immediately when there is trouble

How Cloudflare Responded

Evaluating linear-time regex engines such as Re2 (Google)
CPU profiling on WAF changes
Rolling deployment (no more whole world at once)

Part 4 — Fastly 2021-06-08 — One Customer Stopped Half the Internet

What Happened

June 8, 2021, 10:00 UTC
Fastly CDN down worldwide (about 1 hour)
Affected: Amazon, Reddit, CNN, NYT, PayPal, UK government websites
Reported as "half the internet stopped"

Cause

One customer changed a particular config
That config triggered a software bug inside Fastly
→ cache miss on edge servers worldwide + system errors
→ 85% of traffic affected

A Bug Shipped Back in May

May 12: software deployed (bug planted)
June 8: a specific customer config triggers the bug
  → the dormant bug detonates

Lessons

The danger of multi-tenancy: one customer affects everyone
Dormant bugs: they blow up weeks or months later
Fast recovery: Fastly recovered within an hour (impressive)
The limits of customer testing: you cannot simulate the real production mix

Part 5 — AWS S3 2017-02-28 — One Typo Stopped the Internet

What Happened

February 28, 2017, 09:37 PST
AWS us-east-1 S3 down (about 4 hours)
Affected: Slack, Quora, Trello, Medium, thousands of sites
The AWS "Service Health Dashboard" could not be updated either, because it writes to S3

Cause

An engineer was debugging the billing system:

The plan was to intentionally restart some of the S3 subsystem servers
But a typo restarted many more servers

Specifically:
- Core servers of the Index subsystem + Placement subsystem removed
- These subsystems are the S3 core
- A full restart was required → 4 hours

Lessons

Human error is unavoidable: you have to defend with tooling
The empty seat where automation should be: no confirmation on destructive commands
Blast radius: every subsystem in the same AZ
Dashboard dependency: the status page depended on the same system

How AWS Responded

Added safeguards for dangerous operations
Improved the command-line tool (things like --confirm)
Redesigned service dependencies
Made the status page independent

Part 6 — Knight Capital 2012-08-01 — $440M Lost in 8 Minutes

What Happened

August 1, 2012, 9:30 a.m. (New York stock market open)
Knight Capital (a major market maker in the US stock market)
$440M lost in 8 minutes
→ the company nearly went bankrupt, acquired 3 months later

Cause — Dead Code Resurrection

Background:
- A "Power Peg" feature unused for years in Knight's trading system
- Left behind as dead code
- During a new feature deploy, a deploy mistake on 1 of 8 servers
- That server interpreted the "Power Peg" code as a re-enable flag
- At market open it began generating orders frantically

Over 8 minutes:
- Millions of orders generated
- Buy high, sell low, regardless of price
- Market distortion
- $440M lost in total

Lessons

Remove dead code: code you do not use is dangerous
Automate deploys: updating 8 servers by hand is a breeding ground for errors
Never reuse a feature flag: the same flag with a different meaning is a catastrophe
Circuit breaker: no automatic blocking of abnormal orders
Testing gap: the problem should have been caught in pre-production

Part 7 — GitLab 2017-01-31 — DB wipe

What Happened

The evening of January 31, 2017
GitLab.com database wiped
300GB of data nearly lost (recoverable: about 6 hours of transactions lost)

Cause — Human Error + Backups That Did Not Work

Situation: debugging a DB replication problem
  Replication got tangled on the primary DB
  An engineer tried to clean up the replica
  
Mistake 1: ran rm -rf against the wrong DB
  → part of the production DB deleted

Recovery attempts:
  Backup 1 (daily pg_dump): file was 0 bytes because of an error
  Backup 2 (LVM snapshot): 6 hours old
  Backup 3 (S3 upload): never enabled
  Backup 4 (replica): the very thing that caused the problem
  Backup 5 (Azure disk snapshot): recoverable

→ in the end, 6 hours of transactions lost

An Innovative Response

Live-stream postmortem: GitLab broadcast the recovery process live on YouTube. Tens of thousands watched. The height of transparency.

Lessons

Test your backups: if you have never restored one, it is not a backup
Several kinds of backup: when one fails, another covers it
Prevent human error: destructive commands need a double-check
Transparency: trust in GitLab actually went up

Part 8 — Facebook 2021-10-04 — BGP Is the Culprit Again

What Happened

October 4, 2021, 15:40 UTC
Facebook, Instagram, WhatsApp down worldwide for 6 hours
Facebook staff could not enter the buildings either (the badge system depended on it too)

Cause

BGP config change → Facebook's DNS servers vanished from the internet
Internal tools all depended on the internet too → engineers could not recover
Physical badges affected too → no way into the data center

Recovery:
- Physically entered the data center
- Manually reset the network equipment
- Restored the BGP advertisements

Lessons

Out-of-band access: a way in even when the main network is dead
Dependency loop: depend on the internet for everything and you self-lockout
Physical access: digital infrastructure still needs physical space
The danger of BGP: a pattern that keeps repeating

Part 9 — Heroku 2021-04-13 — GitHub OAuth Token Leak

What Happened

April 2021
Tokens from Heroku's GitHub OAuth integration leaked
Attackers reached the GitHub repos of thousands of Heroku accounts
Heroku did not disclose it properly at first and lost trust

Lessons

Secret management: an OAuth token is a secret too
Rotation: rotate on a schedule
Fast disclosure: hide it and the crisis grows
Third-party risk: an integration is a window for vulnerabilities

Part 10 — Roblox 2021-10-28 — 73 Hours Down

What Happened

73 hours down starting October 28, 2021
A game platform with 50 million DAU
Black Friday weekend

Cause

Load on the HashiCorp Consul (service discovery) cluster servers
A new feature created excessive write load on Consul
Consul consensus broke → total downtime

What Made It Special

A joint HashiCorp + Roblox postmortem. Vendor and customer analyzed it together.

Lessons

Service discovery dependency: the core has to be extremely stable
Chaos testing: load testing should have caught it
Vendor collaboration: a shared-responsibility model

Part 11 — Atlassian 2022-04-05 — 14 Days Down

What Happened

April 2022
775 Atlassian customers (Jira, Confluence, and so on)
14 days spent recovering
Data deleted by a script error

Cause

A script deprecating a legacy feature
It should have deleted at the "site" level, but the argument was wrong at the "app" level
→ entire customer site data deleted
Recovery: from backups, but since it is customer data, each one by hand

Lessons

A destructive script needs a dry-run
Backup per tenant: the key to SaaS multi-tenant recovery
Restore speed: backups mean half as much if restoring is slow
Customer communication: Atlassian was criticized for a poor initial response

Part 12 — Ten Failure Patterns

The patterns visible in every postmortem:

Manual work + mistakes (Knight, AWS, GitLab)
Config change without staging (Cloudflare)
Dead code / unused features (Knight)
Global rollout without canary (Cloudflare, Fastly)
BGP (Facebook, Cloudflare)
Backups that did not work (GitLab)
Dependency loop (Facebook, AWS S3 dashboard)
Multi-tenant blast radius (Fastly, Atlassian)
Third-party integration risk (Heroku)
Chaos testing gap (Roblox)

Part 13 — The Postmortem Template

A Practical Template

# Incident: [Title]
Date: 2026-04-15
Severity: SEV1/SEV2/SEV3
Duration: 09:30-11:45 UTC (2h 15m)
Impact: X users affected, Y revenue lost

## Summary
[2-3 sentences]

## Impact
- Users: ...
- Revenue: ...
- Services: ...
- Customer communications: ...

## Timeline
- 09:30 UTC: Alert triggered
- 09:35 UTC: Engineer paged
- ...

## Root Cause
[Technical explanation]
Contributing factors:
- ...

## Detection
- How was it detected?
- Time to detect: X min
- Could it have been detected faster?

## Response
- Time to mitigate: X min
- What worked?
- What didn't?

## Recovery
[Steps taken to restore service]

## Lessons Learned
### What went well
- ...

### What went wrong
- ...

### Where we got lucky
- ...

## Action Items
| Item | Owner | Priority | Due |
|---|---|---|---|
| Add canary deployment | @alice | P0 | 2026-04-30 |
| ... | | | |

## Supporting Data
- Graphs, logs, screenshots

Writing Tips

A draft within 48 hours of the incident
Review by several people (everyone involved)
Action items are SMART (Specific, Measurable, ...)
Follow-up: check status after 2 weeks and 4 weeks
Consider publishing: share at least part of it externally

Part 14 — A 12-Item Checklist

Part 15 — Ten Antipatterns

Hunting for who did it → blame culture
Not publishing the postmortem → the same team makes the same mistake
No action item tracking → a paper tiger
Wrapping up in a one-hour meeting → no depth
Forcing root cause into one → a complex incident has several causes
"Just human error" → dodging the system design problem
No postmortem unless it is SEV1 → the small ones teach you too
Relaxing because backups exist → never testing the restore
Postponing Chaos Engineering → production becomes the chaos
Legal/PR editing the postmortem → the truth gets diluted

Closing — "Failure Is Expensive, but Learning Is Not More Expensive"

Cloudflare, GitLab, AWS — the industry's best teach the entire industry through public postmortems. What they lost: 100M+ dollars. What we get: free lessons.

The next time your service goes down:

Instead of spending time blaming someone, fix the system
Instead of spending time being embarrassed, write it down
Instead of spending time hiding it, share it with the team

"An organization that can learn from failure is the one with the edge". That is what the Google SRE book has been saying for 25 years.

The next post is Season 3 Ep 3 — The Complete Guide to Scaling Inflection Points. Architecture changes at 1 → 100 → 10K → 100K → 1M → 10M users. When do you shard the DB? When microservices? When a CDN? The concrete thresholds.

Next Up — "The Complete Guide to Scaling Inflection Points: Architecture Evolution From 100 to 10K to 1M Users"

Season 3 Ep 3 covers:

100 users: a single server, SQLite
10K users: Rails + Postgres + Redis
100K users: read replicas, caching, CDN
1M users: read/write split, sharding
10M users: considering microservices
When do you change languages?
The inflection points at real companies

See you in the next post.