Chaos and Order

💡 왼쪽 원문을 읽으면서 오른쪽에 따라 써보세요. Tab 키로 힌트를 받을 수 있습니다.

Introduction — The Mean Got 9% Worse While the Median Got 46% Better
One Dataset, Four Conclusions
Why the Mean Fails — Multimodal Distributions
The Tail That Load Generators Erase — Coordinated Omission
What to Plot Instead — Five Lenses
A Diagnostic Walkthrough — When the Mean Is Flat but the Distribution Splits
Tools for Plotting This in Production
Conclusion — Summary Statistics Are a Hypothesis, Not a Conclusion
References

Introduction — The Mean Got 9% Worse While the Median Got 46% Better

On July 27, 2026, Farid Zakaria published The mean means nothing, and two days later it sat near the top of a Hacker News thread for a while, gathering 73 points. The setup is familiar — a caching layer was rolled out gradually over a week, and the dashboard's average latency rose 9%, from 112ms to 122ms. It's exactly the kind of picture that gets a rollback meeting scheduled.

Yet in the same period, from the same request logs, the median fell 46%, from 99ms to 54ms. p95 rose from 224ms to 454ms, and p99 rose from 309ms to 678ms — more than doubling. Caching succeeded and failed at the same time. Neither statement is wrong.

One thing should be made clear up front. The data in the original post is not a record of a real incident; it's synthetic data generated with a fixed seed by a published script. The comment at the top of the script says as much, and the author notes that AI assisted in generating the charts. So it's more accurate to read this piece not as an incident report saying "this outage happened," but as teaching material on "what lens should you use to look at data shaped like this." And as teaching material, it's put together quite well — a bimodal distribution created by a mix of cache hits and misses is a shape that almost any system with even one cache bolted on will eventually run into.

Taking that situation as a starting point, this post lays out why the mean fails, how load generators erase the tail, and what you should plot instead.

One Dataset, Four Conclusions

Here are the summary statistics the original post presents.

Statistic	Before Rollout	After Rollout	Change
Mean	112ms	122ms	+9%
p50 (median)	99ms	54ms	−46%
p95	224ms	454ms	+103%
p99	309ms	678ms	+119%

Each of these four numbers produces a different meeting. A team looking only at the mean decides to roll back; a team looking only at the median presents it as a success story; a team looking only at p99 opens an incident. All three looked at the same log.

The arithmetic reason the mean produces an ambiguous number is simple. The mean puts the many requests that got faster and the few that got slower on the same scale and cancels them out. If half the traffic gets 45ms faster and 5% gets 370ms slower, the sum lands close to zero. The sign of that canceled-out result tells you nothing about the system. In that sense, the mean isn't so much a wrong metric as a metric that doesn't contain a single question.

Why the Mean Fails — Multimodal Distributions

The core issue is that the caching layer split the request population into two. Requests that hit the cache skip the backend and come back faster than the old baseline. Requests that miss only reach the original path after one extra round trip to check the cache. So the single peak from before the rollout becomes two peaks after it.

This is exactly where the classics of statistics education apply. The most-cited references in the Hacker News comments were Anscombe's quartet and its modern successor, Same Stats, Different Graphs. The point of both is that you can construct data sets whose mean, variance, correlation coefficient, and regression line are all identical, yet whose plots look completely different. One of the data sets in the latter, when plotted as a scatter plot, forms the shape of a dinosaur.

The causes of multimodality in latency data tend to fall into a fixed set.

Cache hits vs. misses
Cold starts vs. warm instances
A connection obtained instantly from the pool vs. a freshly negotiated TLS handshake
The leader region vs. a cross-region fallback
Small responses vs. large responses shipped whole with no pagination
Requests caught by a GC pause or compaction

The original post's final diagnosis lands right near the last item on this list — in a scatter plot of response size against latency, large responses weren't fitting into the cache and were forming the miss-side peak. So the prescription isn't "roll back" but "increase cache capacity or split up large responses." You only get to a prescription once you find the axis that split the distribution.

The Tail That Load Generators Erase — Coordinated Omission

If what you're looking at is load-test results rather than production metrics, there's one more thing to check before you plot the distribution: the measurement itself may have erased the tail.

Here's the structure of what Gil Tene named coordinated omission. Say a load generator is configured to send 1,000 requests per second, and one response happens to take 2 seconds. A synchronous generator simply does not send the 2,000 requests it should have sent during those 2 seconds. Each of those 2,000 requests should have waited up to 2 seconds, but because they were never fired at all, they never enter the histogram. The end result is that the samples from the system's slowest stretch vanish entirely, and the load generator ends up "cooperating" with the backpressure it created itself.

The symptoms of this flaw are distinctive.

p99 barely moves even as you push more load. The queue is actually exploding, but the measured tail looks calm.
Reported throughput comes in below the configured target, yet the latency distribution looks as if it were measured at the target.
Observing the same system under production traffic shows a tail that's far worse than what the load test measured.

There are two ways to fix this. One is to use a generator like wrk2 that holds a constant target throughput while recording latency against the scheduled send time. wrk2 replaces its per-request sample buffer with HdrHistogram and computes response latency starting from "the moment the request should have gone out." The other is to use the correction API that HdrHistogram provides: when the expected interval is known, it synthesizes and fills in the missing samples.

// When recording with an expected interval of 1ms (=1,000,000ns), if a sample
// exceeds that interval, the histogram fills in the intermediate samples that would have been missed.
Histogram h = new Histogram(3600L * 1000 * 1000 * 1000, 3);
h.recordValueWithExpectedInterval(latencyNanos, 1_000_000L);

Production observation doesn't have this problem, because real users don't politely delay their next request just because the previous one was slow. For exactly that reason, when load-test results and production metrics disagree, it's usually the load test you should suspect.

What to Plot Instead — Five Lenses

The real value of the original post is in showing the same numbers plotted several different ways. Each chart answers a different question.

The density plot answers how many peaks there are. Draw this one first — if there's only one peak, you can skip most of the rest of the analysis. Latency trails off to the right, so a log-scale x-axis is the default.

The CDF answers "what percentage finished within how many ms." When you overlay the before and after CDFs on the same axes, wherever the two curves cross, that point marks the boundary between improvement and regression. In the original post they cross around 140ms. A crossing CDF is the visual proof that "no single percentile can summarize this change." A Hacker News comment took this a step further: instead of the CDF, plot one minus it (the CCDF — the fraction of requests not yet finished) on a log-log scale, and the tail that gets crushed into a horizontal line at 1 on a CDF spreads out across the full range. This version is the better one for looking at tails.

The shift function answers "up to which percentile is this a win, and from where does it become a loss." For each percentile p, it plots the after-value minus the before-value. In the original post, the curve stays negative (improvement) up to around p76, then crosses steeply into positive territory above that. This one chart alone produces the sentence "76% of users benefited, and the rest lost out" — and it's exactly the point where the deploy decision splits, depending on where your SLO is set.

The ridgeline plot answers "since when has this been happening." Stacking the per-day density plots vertically shows the second peak growing as the rollout percentage climbs from 0 to 100%. The practical value of a ridgeline is in distinguishing whether a regression arrived with the deploy or predates it.

The heatmap packs the same information as a ridgeline into a grid. The x-axis is time, the y-axis is a latency bucket, and color is the amount of traffic that landed in that bucket. It has a higher information density than a ridgeline, and above all, you can leave it running permanently on a dashboard. Once the time axis stretches beyond a few days, a ridgeline turns into an unreadable overlapping mess, but a heatmap stays legible.

Add the two things the original post uses at the end, and the diagnosis is complete — the conditional CDF (split by cache hit/miss and plot each separately, and it becomes unimodal again) and the scatter plot (latency against response size). The first confirms "what split the distribution," and the second explains "why that grouping."

A Diagnostic Walkthrough — When the Mean Is Flat but the Distribution Splits

The original post's case actually stood out precisely because the mean moved at all. A nastier case is one where the mean stays completely flat while only the distribution splits. This happens when the amount one half sped up happens to match the amount the other half slowed down, and in that case, no alarm goes off at all.

Here's the order to work through.

Look at the density or heatmap first. Check how many peaks there are and when they started splitting. If it's unimodal, skip step 2 and jump straight to step 6 (endpoint breakdown).
Overlay the before/after CDFs. Find the crossing point. If they cross, drop any plan to "report this with one of mean/median/p99."
Use the shift function to pin down the boundary percentile. Which side of that boundary your SLO threshold sits on determines the deploy decision.
Find the axis that split the distribution. Test candidates one at a time, plotting conditionally on each — cache hit or miss, endpoint, region, instance, client version, response-size bucket. The correct axis is the one where each split becomes unimodal.
Look at the relationship between that axis and other variables. In the original post, it was response size. This is where the prescription comes from.
Break it down by endpoint and look again. The overall metric is a traffic-weighted average, which can completely hide a regression in a low-traffic, slow endpoint.

There's one trick worth using when no candidate axis comes to mind in step 4: sample traces only from requests belonging to the slow peak. If you're already running distributed tracing, filtering by latency range and comparing span composition is the fastest route. This connects to the trace-sampling strategy covered in The Three Pillars of Observability and LLM Workloads.

And there's one thing to check before step 4 — in a system with fan-out, the median lies even more badly than p99 does. In an architecture where a single user request scatters across 100 leaf servers and the response only goes out once all of them return, each leaf's p99 dominates the user-facing median. This is exactly what Jeff Dean's The Tail at Scale laid out, and the same point came up in the Hacker News comments. In a fan-out system, "the tail of a component" becomes "the average for the user."

Tools for Plotting This in Production

The teaching-material charts were drawn with plotnine, but the charts you need to look at every day in production run through a different pipeline.

Prometheus native histograms are the most realistic foundation right now. Classic histograms require you to fix bucket boundaries in advance, and the tighter you pack those boundaries, the more linearly your time series count grows. Native histograms remove this trade-off by placing buckets automatically on an exponential scale. They became a stable feature in v3.8.0, and because the data model maps onto OpenTelemetry's exponential histogram, there's also a path for handing data off to an OTLP backend. Turning it on, though, touches several things — scrape protocol negotiation, storage, dashboard queries — enough that I wrote it up separately in Native Histograms Went Stable — So Why Can't We Turn Them On Yet.

Here's what the queries look like.

# a single percentile — look at only this value and you fall straight into this post's trap
histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

# for a heatmap: emit the per-bucket rate as-is and map it to color on the graphing side
sum by (le) (rate(http_request_duration_seconds_bucket[5m]))

# with a native histogram there's no le label. the series itself is the distribution
histogram_quantile(0.99, sum(rate(http_request_duration_seconds[5m])))
histogram_fraction(0, 0.14, sum(rate(http_request_duration_seconds[5m])))

The histogram_fraction on the last line is especially useful in this post's context. It pulls out "the fraction of requests that finished within 140ms" directly, so you can wire up a specific point on the CDF as a time series. If you know the crossing point, turning that single point into an alert is far more accurate than a p99 alert.

Grafana's heatmap panel takes the second query above as-is. Setting the format to Heatmap and switching the y-axis to a log scale is, in practice, a required setting. A heatmap with a linear y-axis crushes all the lower buckets into a single line and can't show a bimodal shape.

When you need the kernel-side distribution, bpftrace's hist() is the cheapest option. With no application instrumentation at all, it prints the latency distribution of a specific system call or function directly into log2 buckets.

# block I/O completion latency distribution (usec, log2 buckets)
sudo bpftrace -e '
kprobe:blk_account_io_start { @s[arg0] = nsecs; }
kprobe:blk_account_io_done  /@s[arg0]/ {
  @us = hist((nsecs - @s[arg0]) / 1000); delete(@s[arg0]);
}'

The broader approach of capturing distributions with eBPF is covered in more depth in eBPF Is Eating Observability.

For load testing, use something in the wrk2 family mentioned earlier, or a generator that embeds HdrHistogram and maintains a target throughput. Whatever tool you use, there's exactly one thing to check when you look at the results — does the reported actual throughput match the configured target throughput. If it doesn't, the latency distribution from that run can't be trusted.

Conclusion — Summary Statistics Are a Hypothesis, Not a Conclusion

To sum up.

The mean cancels out the many requests that got faster against the few that got slower. The sign of that canceled-out value tells you nothing about the system.
If two CDFs cross, no single percentile can summarize the change. In that case, plot a shift function instead of reaching for one more percentile.
If the distribution has split, finding the axis that split it is the whole diagnosis. The cause is whichever axis makes each conditional split unimodal.
If a load test's tail looks suspiciously well-behaved, suspect coordinated omission first. A mismatch between target and actual throughput is the tell.
In a fan-out system, the tail of a component becomes the average for the user.

The cost of drawing one chart before deciding to roll back based on a single number is, if you're already collecting histograms, just one line of query. The problem isn't the tooling — it's the habit.