필사 모드: Why Average Response Time Lies — How to Read p50, p95, and p99 Properly

• Introduction — The Average Is 80ms, So Why Are the Complaints Coming In
• What the Average Hides — The Arithmetic of a Long Tail
• The Question Each of p50, p95, p99, and p99.9 Answers
• p99 Is Not "One User in a Hundred" — The Multiplicative Effect of Latency
• Percentiles Cannot Be Averaged — Which Is Why You Need Histograms
• Prometheus Histograms — Bucket Boundaries and Interpolation Error
• Practical Criteria for Defining an SLO with Percentiles
• Closing — Look at the Distribution, Not a Single Number

Introduction — The Average Is 80ms, So Why Are the Complaints Coming In

The dashboard says the average response time is 80ms. The graph has been flat for a month and no alert has fired. Meanwhile the support team receives a ticket every single day saying the checkout screen freezes for seconds at a time.

The two facts are not contradictory. The average is simply a metric that behaves this way. It flattens the shape of a distribution into one number, and the first thing it discards is precisely the region where users get angry.

This post is about digging that region back out. What percentiles answer, why percentiles cannot be averaged with one another, and what numbers a Prometheus histogram actually hands back — all the way down to a level where you can do the arithmetic yourself.

What the Average Hides — The Arithmetic of a Long Tail

Response time distributions are not normal distributions. There is a wall on the left (nothing can be faster than 0ms) and a long stretch to the right. GC pauses, cold caches, connection pool waits, retries, noisy neighbours — there are many ways to get slower and none to get faster.

Let us look at the numbers. Suppose that out of 10,000 requests, 9,900 finish in 50ms and 100 take 3,000ms.

# Pull only the response time (ms) column out of the access log and compute the distribution directly
awk '{print $NF}' access.log | sort -n > sorted.txt

wc -l < sorted.txt
# 10000

awk '{s+=$1} END {printf "mean %.1f\n", s/NR}' sorted.txt
# mean 79.5

for n in 5000 9500 9900 9990; do
  printf "n=%s\t%s\n" "$n" "$(awk -v r=$n 'NR==r' sorted.txt)"
done
# n=5000	50      <- p50
# n=9500	50      <- p95
# n=9900	50      <- p99  (sitting right on the boundary)
# n=9990	3000    <- p99.9

The average is 79.5ms. Not a single request was actually served in 79.5ms. The average describes a user who does not exist.

Sensitivity is worse still. If those 100 slow requests double from 3,000ms to 6,000ms, the average moves only from 79.5ms to 109.5ms. A 30ms rise crosses no alert threshold anywhere. To the users who hit those 100 requests, however, the service looks completely broken. The average barely detects the tail getting worse.

The Question Each of p50, p95, p99, and p99.9 Answers

Every percentile answers a different question. Looking at only one is worse than looking at none.

Here is how to read them in practice.

• If p50 gets worse, the whole system slowed down. Suspect capacity shortfalls or a shared dependency.
• If p50 is unchanged but only p99 degrades, the problem occurs under specific conditions. Look at a particular tenant, a particular query path, a particular node, lock contention, or GC.
• If the gap between p99 and p99.9 widens, a rare but very expensive path has appeared. Check for retry storms or timeout settings.
• If p50 and p99 are nearly identical, the distribution has been compressed. That may be a good sign, but also consider the possibility that something upstream is cutting requests off with a timeout.

p99.9 only means something when traffic volume supports it. In a service taking 1,000 requests every five minutes, p99.9 is a single request. Alert on a one-request signal and the alert becomes a random number generator.

p99 Is Not "One User in a Hundred" — The Multiplicative Effect of Latency

This is the most common and most expensive misunderstanding. A p99 of 2 seconds does not mean "1% of users experience 2 seconds", it means "1 out of every 100 requests experiences 2 seconds". If a single user generates many requests, the probability they feel it multiplies by the number of requests.

// When one screen issues N backend calls,
// the probability that "at least one exceeds p99"
const p = 0.99
for (const n of [1, 5, 20, 50, 200]) {
  const hit = 1 - Math.pow(p, n)
  console.log(`${String(n).padStart(3)} calls → ${(hit * 100).toFixed(1)}%`)
}
//   1 calls → 1.0%
//   5 calls → 4.9%
//  20 calls → 18.2%
//  50 calls → 39.5%
// 200 calls → 86.6%

If one dashboard screen calls 20 APIs, the chance of meeting p99 latency each time that screen opens is 18 percent. For an active user who generates 200 requests a day, 87 percent will hit the worst-case region at least once a day. p99 is not a handful of unlucky users, it is the daily reality of nearly all of them.

Two conclusions follow. First, the larger the fan-out in an architecture, the more tail latency management matters. Improving the p99 of a single service has a bigger effect than improving the p50 of the whole screen. Second, you need to watch client-side metrics separately. What the user actually experiences is the percentile at the moment the screen is complete, not the server p99.

Percentiles Cannot Be Averaged — Which Is Why You Need Histograms

This section is the most practically important part of the post. Sums and counts can be added together, but percentiles cannot be added or averaged.

Take two instances as an example.

• Server A: 9,900 requests, all 50ms → p99(A) = 50ms
• Server B: 100 requests, all 3,000ms → p99(B) = 3,000ms

Averaging the two p99 values gives 1,525ms. Weighting the average by request count gives 79.5ms. But the true p99, computed by lining up all 10,000 requests from both servers, is 3,000ms. All three numbers differ, and the first two mean nothing at all.

That is why the following PromQL is wrong.

# Wrong: compute p99 per instance and then average the results
avg(
  histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{job="checkout-api"}[5m]))
)

The correct order is to sum the bucket counters first and then compute the percentile from the combined distribution. Counters are aggregatable; percentiles are not.

# Right: sum the rates by the le label first, then compute the percentile
histogram_quantile(
  0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket{job="checkout-api"}[5m]))
)

If you want to break the view down by service, put that label into the sum by clause alongside le. le must never be dropped.

# p99 per route — keep le and route together
histogram_quantile(
  0.99,
  sum by (le, route) (rate(http_request_duration_seconds_bucket{job="checkout-api"}[5m]))
)

For the same reason, the Prometheus summary type computes percentiles inside the instance before exporting them, so there is no way to combine multiple instances or multiple routes. It is usable for local diagnosis of a single process but not as a service-level metric. A histogram gives up accuracy to the extent of its bucket resolution and gains aggregatability in return. In a distributed system, that is almost always the right trade.

Prometheus Histograms — Bucket Boundaries and Interpolation Error

The value histogram_quantile returns is not an observed value. It is an estimate produced by drawing a straight line between bucket boundaries.

curl -s localhost:8080/metrics | grep '^http_request_duration_seconds_bucket'
# http_request_duration_seconds_bucket{le="0.005"} 0
# http_request_duration_seconds_bucket{le="0.01"}  0
# http_request_duration_seconds_bucket{le="0.025"} 12
# http_request_duration_seconds_bucket{le="0.05"}  4103
# http_request_duration_seconds_bucket{le="0.1"}   8890
# http_request_duration_seconds_bucket{le="0.25"}  9604
# http_request_duration_seconds_bucket{le="0.5"}   9740
# http_request_duration_seconds_bucket{le="1"}     9812
# http_request_duration_seconds_bucket{le="2.5"}   9993
# http_request_duration_seconds_bucket{le="5"}     10000
# http_request_duration_seconds_bucket{le="+Inf"}  10000

p99 is the 9,900th observation. That value sits between the le=1 bucket (cumulative 9,812) and the le=2.5 bucket (cumulative 9,993). Linear interpolation computes it like this.

python3 - <<'PY'
lo, hi = 1.0, 2.5
c_lo, c_hi = 9812, 9993
target = 0.99 * 10000
est = lo + (target - c_lo) / (c_hi - c_lo) * (hi - lo)
print(f"p99 estimate = {est:.3f}s")
PY
# p99 estimate = 1.729s

Out comes 1.729 seconds, a number that looks precise, yet nobody knows where inside that bucket the 181 observations actually fall. They could all be 1.05 seconds or all 2.4 seconds. When buckets are wide, the output of histogram_quantile is a guess reported to three decimal places.

Three practical rules follow.

First, always include a bucket boundary that exactly matches your SLO threshold. If you are going to use 300ms as the reference, there must be an le=0.3 bucket. With the boundary in place you can count the exact ratio without interpolation.

Second, place buckets densely in the region you care about. The default buckets are far too sparse above 0.1 seconds for a web API.

// prom-client — set the boundaries yourself to match the real distribution of the service
const { Histogram } = require('prom-client')

const httpDuration = new Histogram({
  name: 'http_request_duration_seconds',
  help: 'HTTP request processing time',
  labelNames: ['method', 'route', 'code'],
  // Include the SLO threshold 0.3 as a boundary and pack the 100ms~1s region densely
  buckets: [0.01, 0.025, 0.05, 0.075, 0.1, 0.15, 0.2, 0.3, 0.4, 0.6, 0.8, 1, 2, 5, 10],
})

Third, set the highest finite bucket above the actual timeout. Once p99 pushes past the last finite boundary, histogram_quantile either returns that boundary value itself (depending on the implementation) or pins to the ceiling, and you lose all information about how bad things really are.

If choosing bucket boundaries by hand is a burden in itself, Prometheus native histograms are worth evaluating. They generate exponentially spaced buckets automatically, which removes much of the boundary-selection and resolution problem. Note, though, that the storage format, query path, and remote-storage compatibility differ from classic histograms, so check support across your whole stack first.

Practical Criteria for Defining an SLO with Percentiles

Here comes the final twist. Defining a latency SLO as "p99 at or below 300ms" is not a good choice in practice.

The reasons all appeared above. Percentiles cannot be aggregated, and they cannot be combined along the time axis either. Collecting 30 days of five-minute-window p99 values does not give you the p99 of 30 days. You cannot compute an error budget or derive a burn rate from them.

Express the same content as a ratio instead. The sentence "99% of requests complete within 300ms" is computed directly from the fraction of requests that crossed the threshold, and that fraction can be added.

# Latency SLI: the fraction of requests completed within 300ms
sum(rate(http_request_duration_seconds_bucket{job="checkout-api", le="0.3"}[30d]))
/
sum(rate(http_request_duration_seconds_count{job="checkout-api"}[30d]))

# It leads straight into error budget burn (target 99%, i.e. a 1% budget)
(
  1 -
  sum(rate(http_request_duration_seconds_bucket{job="checkout-api", le="0.3"}[1h]))
  /
  sum(rate(http_request_duration_seconds_count{job="checkout-api"}[1h]))
) / 0.01

This form has three advantages. There is no interpolation error (because it counts the le=0.3 bucket directly). It can be recomputed over any time window. And it plugs straight into burn rate alerts.

Here are some practical reference values for setting the criteria.

• Choose the target percentile with the fan-out of the user journey in mind. If one screen makes 20 calls, the target for a single service has to be 99.9% rather than 99% to yield 98% at the screen level.
• Derive the threshold backwards from the limits of human perception. The 100ms that feels instantaneous, the 1 second that keeps flow unbroken, and the 10 seconds where attention departs are old baselines and still valid.
• Set the measurement window as a rolling 28 or 30 days, longer than your deployment cycle. If the window is short, one mistake in one deployment burns the entire budget immediately.
• Keep watching percentile dashboards. Manage the SLO as a ratio, but start root-cause investigation from a graph that overlays p50, p99, and p99.9 together.

Closing — Look at the Distribution, Not a Single Number

There is one sentence to remember. The average flattens a distribution into one number and discards the part where users get angry first, while percentiles show you that part but can never be combined again.

So the practical order becomes this. Preserve the raw distribution with a histogram, plant the SLO threshold in a bucket boundary, use percentiles only on dashboards that humans read, and define alerts and SLOs as aggregatable ratios. You can delete the average response time graph. Simply putting p50 and p99 side by side in that spot is enough for most teams to start seeing incidents that were invisible before.

Further reading.

• Prometheus: Histograms and summaries
• Prometheus: the histogram_quantile function
• Google SRE Book: Service Level Objectives