Chaos and Order

💡 왼쪽 원문을 읽으면서 오른쪽에 따라 써보세요. Tab 키로 힌트를 받을 수 있습니다.

Introduction — When Someone Else's Postmortem Beats the Textbook
What Happened
The Hidden Premise in Quorum Arithmetic — Failure Independence
And Yet Coinbase Isn't Walking the Choice Back
Code That Only Runs During a Disaster Is Code That Has Never Been Tested
Managed Services Don't Remove Risk, They Relocate It
The Most Honest Part — Even 3-AZ Wouldn't Have Stopped This
Reread the Timeline and the Single-Cause Story Falls Apart
An AZ Is Not the Smallest Unit of Failure
So What Should Change In Practice
Closing — And What This Post Doesn't Know
References

Introduction — When Someone Else's Postmortem Beats the Textbook

Eighty percent of distributed-systems writing starts with the same sentence: "Raft tolerates f failures with 2f+1 nodes." It's true, it's on the exam, and in practice nobody dies from not knowing it. People die from the assumption that sentence never writes down.

On June 1, 2026, Coinbase published a postmortem of its May 7 outage. This post is worth reading not because it's a "we made a dumb mistake" story. It's the opposite — a deliberate, documented, defensible-in-its-domain architecture choice running head-on into a failure mode that same choice had assumed away. And Coinbase still hasn't walked the choice back.

This post reads that postmortem as its source. Every timestamp and figure here is Coinbase's own, and anywhere the source differs I'll say so.

What Happened

It started with physics, not software. By Coinbase's account, at 7:20 PM Eastern on May 7, several chillers failed simultaneously inside a single data hall within AWS us-east-1's availability zone use1-az4. The cooling loss sent those racks into thermal-protection shutdown, and the building's EC2 instances and EBS volumes went offline.

AWS's own account paints the same picture. Per the AWS status dashboard language quoted by The Register and Network World, AWS wrote in a 5:25 PM Pacific update that EC2 instances and EBS volumes on the affected hardware were impacted by "power loss during a thermal event," and in a next-morning update explained that temperatures exceeded operating thresholds, triggering automatic server shutdowns to protect the hardware. This wasn't the kind of failure software can route around — cooling had to come back before the hardware could be powered back up.

Coinbase's published timeline, all times Eastern:

May 7
  7:20 PM  Multiple chillers fail simultaneously in a single data hall in use1-az4
           Cooling loss -> thermal-protection shutdown of the racks -> EC2 / EBS offline
  7:48 PM  Most Coinbase trading halts
  9:29 PM  AWS terminates EC2 instances within the placement group
           3 of 5 matching-engine nodes down -> quorum lost

May 8
 12:06 AM  Quorum restored
  2:25 AM  Matching engine opened in cancel-only mode
  3:00 AM  Manual partition reassignment to pull topics off the damaged brokers
  3:49 AM  Trading resumes normally across all products
  5:30 AM  coinbase.com / mobile app trading restored at reduced capacity
  9:30 AM  P0 / P1 topics fully recovered
  9:53 AM  coinbase.com / mobile app fully recovered
  2:00 PM  Remaining event-streaming topic backlog cleared

Coinbase's own summary: most of the surfaces customers depend on — trading, deposits, and withdrawals among them — were unavailable or degraded for "roughly 8 hours," and full recovery of every system took "an additional 12 hours." For reference, from 7:48 PM (trading halt) to 3:49 AM (all products resumed) is exactly 8 hours 1 minute, so that "roughly 8 hours" checks out against the timeline.

The Hidden Premise in Quorum Arithmetic — Failure Independence

Coinbase Exchange's matching engine is a Raft-based replicated cluster running inside an AWS Cluster Placement Group. It has 5 nodes, and by Raft's arithmetic that tolerates up to 2 failures. With 3 alive, that's a majority, so it can keep going.

Three died. All at once.

What matters here isn't "3 died, so 5 nodes wasn't enough." The same thing would have happened with 7 nodes. The real problem is the premise behind that arithmetic. The precise form of "5 nodes tolerates 2 failures" is 5 nodes tolerates 2 independent failures. And a cluster placement group is a thing that deliberately removes that independence.

AWS's own placement-groups documentation states this tradeoff plainly. The docs break placement strategies into:

Cluster — "packs instances close together" inside a single availability zone. The goal is the low-latency network performance that HPC-style, tightly-coupled inter-node workloads need.
Partition — splits instances into logical partitions, so instance groups in different partitions don't share the same hardware. The docs cite Hadoop, Cassandra, and Kafka as examples.
Spread — strictly places a small number of instances on distinct hardware to "reduce correlated failures."

And the docs add: without any placement group, EC2 by default tries to spread instances across hardware to "minimize correlated failures."

In other words, the phrase "correlated failure" shows up in AWS's docs when describing Spread and the default behavior — not when describing Cluster. Cluster is the strategy that lives on the opposite side. It's a trade of independence for latency, and that trade is stated on the docs' first page.

So the one-line summary of this incident is this: quorum counts nodes, but failure doesn't count nodes. Failure counts failure domains. The 5 nodes inside a placement group look like 5 to Raft, and like 1 to a chiller.

And Yet Coinbase Isn't Walking the Choice Back

This is the most interesting part of the postmortem. Posts like this usually go "we created a single point of failure and we regret it." Coinbase didn't write it that way.

The postmortem states co-location was a deliberate choice, and gives the reason: a matching engine that meets the latency and throughput demands of a serious market can't absorb network hops across availability zones between voting cluster members. It states that the physics of distributed consensus and the economics of running a fair, liquid order book point to the same answer — co-location.

That claim deserves to be taken seriously. In Raft, commit latency is bound to how long it takes the leader to hear back from a majority of followers — which is the round-trip latency among voting members. Crossing AZs lengthens that round trip, and the cost is paid on every single commit. For people shaving microseconds through kernel tuning in a low-latency trading system, that's not a negotiable constraint. (Coinbase didn't publish specific latency figures, and I won't invent any here either.)

So Coinbase's own diagnosis sits next to co-location, not on top of it. In the postmortem's words, the architectural problem was the lack of ability to automatically fail over to another availability zone. The problem wasn't using a placement group — it was not having an automated way out of one.

That's not wordplay; it leads to a genuinely different prescription in practice. "Don't use placement groups" is useless advice for this team — do that and the matching engine can't hit its performance requirements. "If you've deliberately coupled a failure domain, build the door out and open it on a schedule" is actionable. And that's exactly what tops Coinbase's list of remediation items — improving the matching engine's warm cross-zone standby design and running production failover drills on a regular cadence, with the largest exchanges included during maintenance windows and the rest run on a sandbox cadence.

Code That Only Runs During a Disaster Is Code That Has Never Been Tested

The sentence in the postmortem I stared at the longest is this one. To restore quorum, Coinbase had to deploy an emergency code change during the incident, one that removed a boot-time assumption that all five nodes must be resolvable.

Read that slowly. They're trying to bring the cluster back up, and the cluster won't come up — because the boot code treated the topology it was born into as fact. All five nodes must resolve by name before it comes up: true in normal operation, true during deploys, true in staging. False only on the one night three nodes turned to ash. Which makes this assumption the kind of code that's correct on every other day, and wrong only on the one day it matters.

Why this happens is no mystery. The happy path runs thousands of times a second; the recovery path runs once every few years. And code that never runs is code that's never tested. When we say "we have a recovery procedure," what we usually mean is "we have a document that describes a recovery procedure." Documents don't compile.

The list of what Coinbase actually did that night makes the point concrete — deploy an emergency code change, stand up a new node group outside the damaged placement group, and step carefully through a sequence to restore 3-of-5 quorum. None of these three was a single button. All three were close to being done for the first time, that night.

That's why "regular production failover drills" made the remediation list. The point of the drill isn't mainly that people get familiar with the procedure — it's making sure the recovery path is code that actually runs. A failover you've never run isn't a failover; it's a hypothesis.

Managed Services Don't Remove Risk, They Relocate It

The second failure mode came from an entirely different layer. A large share of Coinbase's event streaming runs on AWS's managed Kafka, MSK.

The promise of managed Kafka is clear. In the postmortem's own summary, if an individual broker dies, the service is supposed to automatically re-elect partition leaders and keep serving traffic from the surviving brokers; if an entire availability zone goes down, the result is supposed to be reduced capacity, not unavailability.

That's not what happened. By Coinbase's account, a defect in the AWS MSK control plane blocked automatic partition-leader re-election, and two MSK clusters got stuck in a "healing" state where producers couldn't write.

Two things are worth flagging here.

First, this is a gray failure, not a crash. "Healing" isn't death. Death would actually be easier — health checks go red, alerts fire, failover kicks in. "Healing" is a system telling itself "I'm currently recovering," and that statement can be true, or it can never come true. Meanwhile producers can't write. The dashboard isn't green or red, it's yellow, and yellow doesn't come with automation attached.

Second, not being able to fix it is the real cost. With self-managed Kafka, you'd have had the option to force a leader re-election by hand, or restart a controller. On MSK, that lever isn't yours. What Coinbase ended up doing was performing manual partition reassignment live with AWS engineering at 3 AM to move topics off the damaged brokers. And one of the remediation items is to build tools and runbooks for "Kafka control levers not made available to us through MSK." That sentence is a remarkably precise summary of a managed service's real bill — hand off the operational burden to a vendor, and when the vendor can't fix it, neither can you. Your recovery time is then set by the vendor's response speed, not your own engineering capability.

And how expensive this second failure mode actually was shows up right in the timeline. Quorum was restored at 12:06 AM, but the market didn't open until 3:49 AM. That's a 3-hour-43-minute gap. Coinbase states directly that the delay was mostly the MSK issue. Reviving the consensus cluster and reviving the service were not the same event.

The Most Honest Part — Even 3-AZ Wouldn't Have Stopped This

There's one sentence in the postmortem that most secondary coverage quietly dropped. Coinbase admits one of its Kafka clusters ran a 2-AZ configuration that widened the blast radius and lengthened recovery — and then immediately adds: the MSK control-plane defect affected 2-AZ and 3-AZ clusters similarly.

That's not a defense; it's accuracy. And cutting that sentence would cut the incident's lessons in half.

Here's the lesson it carries. Data-plane redundancy doesn't fix a control-plane failure. Adding a third AZ means "there's one more place with an extra broker." But if the problem isn't a missing broker, but rather the thing that's supposed to elect a leader failing to elect one, bolting on another AZ just gives you one more pile of leaderless brokers. Redundancy only pays off while the controller is alive.

Notably, Coinbase says it's still moving the 2-AZ cluster to 3-AZ. That's the right call — 3-AZ is still better than 2-AZ, and "it didn't help this time" doesn't mean "it won't help next time." But they didn't package it as "the fix for this incident." A postmortem whose remediation items don't pretend to have solved the root cause is rarer than you'd think.

Reread the Timeline and the Single-Cause Story Falls Apart

This incident is mostly told this way right now: "the Raft cluster lost quorum, so Coinbase went down." Clean, memorable, and doesn't match the timeline.

Trading halted at 7:48 PM. The matching engine lost quorum at 9:29 PM. Trading had already stopped 1 hour and 41 minutes before quorum broke.

So what stopped trading at 7:48 PM? The postmortem doesn't say explicitly. But in describing the MSK issue, it notes that the fallout blocked the fee service, blocking the fee service blocked quote generation, and so most customers experienced this incident not as a Kafka outage but as broken matching and broken quotes. Part of the ledger pipeline, settlement, and several data pipelines were affected the same way.

The circumstantial evidence points that direction, but since the postmortem doesn't state the causation for 7:48 PM explicitly, I won't assert it either. What can be said with confidence is more modest, and more useful — the timeline alone is enough to disprove the single-cause story. And that's exactly how Coinbase itself frames it: two failure modes, each independently recoverable, turned a localized cloud-provider event into a multi-hour platform outage when they overlapped and amplified each other.

This is how large outages actually happen. Not one root cause, but two defects that normally know nothing about each other meeting on the same night. And afterward we pick whichever one makes the better picture ("the consensus cluster lost quorum") and use it as the name for the whole incident. The moment we do, the other half of the lesson disappears. This is, incidentally, exactly the kind of after-the-fact reconstruction that blameless postmortem culture is meant to guard against.

An AZ Is Not the Smallest Unit of Failure

There's one more mental model this incident shook loose.

Coinbase states, explicitly in the postmortem, the design principle it — and most AWS customers — build against: a whole availability zone can die, and a properly designed system should keep serving from the remaining zones. And that this class of failure is the kind hyperscale providers are supposed to absorb at the availability-zone boundary.

But what actually died wasn't an availability zone. It was a single data hall — smaller than an AZ.

There's an irony here. The failure was smaller than an AZ, and it still took out 3 of 5 nodes in the cluster. Because the placement group's blast radius was also smaller than an AZ. Two small circles overlapped. The habit of "thinking in AZs" blinds you not just to failures bigger than an AZ (a region-wide control-plane problem) but also to failures smaller than one. The question that needs answering isn't how many AZs your replicas are spread across — it's how many actual failure domains your replicas are spread across. And the moment you use a cluster placement group, that number gets far smaller than your AZ count.

This is exactly why abstractions like the CAP theorem often go powerless in practice. In theory, P (partition) is one random variable. In reality, it's the plumbing layout of a chiller.

So What Should Change In Practice

Here's what can be honestly taken from this incident.

Count failure domains, not replicas. "Our etcd is 5 nodes" isn't an answer. The question is "does a single event exist that can kill all 5 at once." Same rack, same switch, same power, same data hall, same placement group, same Kubernetes node group, same AMI, same deploy pipeline. Overlap anywhere on that list and quorum arithmetic becomes a lie by that much.

If you deliberately coupled a failure domain, build the door out — and open it. Co-location itself isn't the sin; that's Coinbase's own diagnosis. The problem was coupling it and never automating the escape route. If you sold independence to buy latency, what you bought with it is "a night you'll eventually have to unwind by hand." Reserve that night in advance and rehearse it.

Hunt down and delete topology assumptions in boot code. A condition like "all N nodes must resolve before it comes up" looks like a health check in normal times and turns into a lock during a disaster. The only way to know whether it boots on a degraded cluster is to actually boot it on a degraded cluster.

Ask managed services this question: if the control plane is what's broken, what can I actually do? If the answer is "open a support ticket," that's your RTO. Whether you can live with that RTO is the real line between managed and self-operated — a far more concrete question than the phrase "operational burden" tends to collapse it into.

Separate data-plane redundancy from control-plane redundancy. Adding AZs is the former. It's worth nothing against a failure where the controller can't move.

And don't over-learn from this. This might be the most important one. Coinbase is an exchange running a matching engine that can't absorb AZ hops between voting members. If your service doesn't have that constraint — and most don't — there was never a reason to use a cluster placement group in the first place, and AWS's default behavior (spread across hardware to minimize correlated failure) is already the right choice. If what you take from this postmortem is "pack nodes tightly for low latency," you've learned exactly the wrong lesson. What's worth learning is the opposite — even someone with a genuine reason to pack nodes tightly paid this price for it.

Closing — And What This Post Doesn't Know

The last paragraph of Coinbase's postmortem carries a sentence to this effect: their bar isn't whether AWS had an outage, it rests entirely on whether they built a system sturdy enough to withstand one. That sentence summarizes this whole post. A cloud provider's outage is not an excuse for your outage — it's only a grade on the assumptions you wrote into your design doc.

Now, this post's limits.

We only have one side of the story. Checking AWS's list of post-event summaries turns up no official summary for this May 2026 thermal incident, and none for the MSK control-plane defect either (the most recent entry is the October 2025 DynamoDB incident). So everything in this post about the MSK defect — that leader re-election was blocked, that clusters got stuck "healing," that 2-AZ and 3-AZ were affected similarly — is Coinbase's account. It is not a fact AWS has verified or confirmed. Coinbase itself, as of June 1, wrote that it was still conducting root-cause analysis of this defect together with AWS. So this part is fair to read as provisional.

And we don't know what happened at 7:48 PM. As noted above, the postmortem doesn't state the causation of the first trading halt. There's circumstantial evidence, but no confirmation.

Gaps like these are the reality of this genre. A postmortem isn't a copy of the truth — it's a copy of what the organization that wrote it knew, and could say, at that moment. Even so, Coinbase's writeup earns its read — it defends its own choice while writing down the price of that choice, and even states that its own remediation items don't fix the root cause. A piece that does both at once is rarer than it should be.