Skip to content
←

필사 모드: GPU Serving SLOs and Alert Design — What to Promise and What to Page On

English
0%
정확도 0%
💡 왼쪽 원문을 읽으면서 오른쪽에 따라 써보세요. Tab 키로 힌트를 받을 수 있습니다.

Introduction — What Are You Promising

The first proposal in any meeting about setting an SLO is availability: let us make the rate of endpoints returning 200 be 99.9 percent. For an inference service that proposal is close to useless. The server is returning 200 while the first character shows up thirty seconds later, and the user calls that an outage.

What is worth promising in GPU serving is not the existence of a response but its speed. And there are two ways to measure speed, which eat each other.

Metric names and configuration were verified against the official documentation and repositories on 2026-08-12. They can differ between releases, so check again against the version you are running.

Choosing the Metric That Represents the User

There are broadly three candidates: time to first token, inter-token latency, and throughput.

For an interactive service, time to first token is the answer, because it is the time a human spends staring at a screen. In vLLM, vllm:time_to_first_token_seconds measures it. Next in importance is vllm:inter_token_latency_seconds: a fast first character followed by stuttering still feels bad.

For batch or offline work, throughput is the answer. That is the rate of increase of vllm:generation_tokens_total, and individual request latency hardly matters, because there is no one waiting.

The problem appears when both targets get set at once. The way to raise throughput is to grow the batch, and growing the batch lengthens how long an individual request waits to be included. So setting aggressive targets on both throughput and time to first token means missing both. The honest design picks one as the target and merely observes the other.

Having chosen a metric, define the SLI. There is one practically important choice here. Rather than a quantile, use the fraction of requests below a bucket boundary. The reason lies in the bucket list from the previous post. The boundaries of vllm:time_to_first_token_seconds include 1.0, and the next one is 2.5. Nothing exists in between, so if the quantile function answers 1.4 seconds, that is an interpolation and not a measurement. The fraction of requests at or under 1.0 second, by contrast, is read straight out of the histogram.

# SLI: fraction of requests whose first token arrived within 1 second
sum by (model_name) (rate(vllm:time_to_first_token_seconds_bucket{le="1.0"}[5m]))
  / sum by (model_name) (rate(vllm:time_to_first_token_seconds_count[5m]))

# For reference: keep quantiles for dashboards, not for the SLI
histogram_quantile(0.95,
  sum by (le, model_name) (rate(vllm:time_to_first_token_seconds_bucket[5m])))

When picking the threshold, always choose from among the real bucket boundaries. 0.5, 0.75, 1.0, and 2.5 exist; 1.5 and 2.0 do not.

Reading the Saturation Signals

Signals appear before the SLO breaks, and they appear in a fixed order. Knowing that order shortens cause tracing.

Stage one is the KV cache. vllm:kv_cache_usage_perc pins to the ceiling. On its own, though, this value can be high in perfectly healthy operation. It is context, not a warning.

Stage two is preemption. The rate of vllm:num_preemptions_total leaves zero. From this moment latency is already degrading, because a rolled-back request has to be recomputed and that work is wasted.

Stage three is the queue. vllm:num_requests_waiting grows and the capacity side of vllm:num_requests_waiting_by_reason swells. If instead the deferred side is growing, the problem is more likely configuration than capacity.

Stage four is user latency. Now the SLI drops below target.

GPU-side metrics narrow down the cause within that chain. A long queue alongside a low DCGM_FI_PROF_SM_ACTIVE means the GPU is idle and the cause lies in memory or scheduling rather than compute. A DCGM_FI_DEV_FB_USED at the ceiling means the model or cache configuration has to shrink.

# The saturation chain on one screen
max by (model_name) (vllm:kv_cache_usage_perc)
sum by (model_name) (rate(vllm:num_preemptions_total[5m]))
sum by (model_name, reason) (vllm:num_requests_waiting_by_reason)

# GPU idle while the queue is long
(sum by (pod) (vllm:num_requests_waiting) > 10)
  and on(pod) (avg by (pod) (DCGM_FI_PROF_SM_ACTIVE) < 0.3)

Alert Design — Page on Symptoms, Diagnose on Dashboards

There is one principle. An alert that wakes a human fires only on the fact that users are hurting; why they are hurting is found on a dashboard. Put an alert on every cause metric and a single incident rings eight alerts, and time is spent deciding which one is real.

Error budget burn rate splits severity naturally. Requiring a long window and a short window together catches sustained failures while letting momentary wobble pass.

groups:
  - name: vllm-slo
    rules:
      # Pin the SLI in a recording rule first
      - record: vllm:ttft_good_ratio:rate5m
        expr: |
          sum by (model_name) (rate(vllm:time_to_first_token_seconds_bucket{le="1.0"}[5m]))
            / sum by (model_name) (rate(vllm:time_to_first_token_seconds_count[5m]))

      - record: vllm:ttft_good_ratio:rate1h
        expr: |
          sum by (model_name) (rate(vllm:time_to_first_token_seconds_bucket{le="1.0"}[1h]))
            / sum by (model_name) (rate(vllm:time_to_first_token_seconds_count[1h]))

      # 99% target, 14.4x burn = only when both the 1h and 5m windows exceed it
      - alert: VLLMTTFTBudgetBurnFast
        expr: |
          (1 - vllm:ttft_good_ratio:rate1h) > (14.4 * 0.01)
          and (1 - vllm:ttft_good_ratio:rate5m) > (14.4 * 0.01)
        for: 2m
        labels:
          severity: page
        annotations:
          summary: 'TTFT error budget burning fast for {{ $labels.model_name }}'

      # Slow burn goes to a ticket, not a page
      - alert: VLLMTTFTBudgetBurnSlow
        expr: (1 - vllm:ttft_good_ratio:rate1h) > (3 * 0.01)
        for: 1h
        labels:
          severity: ticket
        annotations:
          summary: 'TTFT error budget burning slowly for {{ $labels.model_name }}'

Three choices here are deliberate. Pinning the SLI in a recording rule first makes the alert and the dashboard share one definition. Requiring both windows for the fast-burn alert avoids paging for an incident that already recovered. And slow burn drops in severity and becomes a ticket.

Four Recurring False Positives

Some false positives show up over and over in GPU serving.

First, quantiles over short windows. Inference request lengths have a wide distribution, so a p99 over a five minute window is decided by a handful of requests. During low-traffic hours the alert fires simply because two requests were long. Avoid it by lengthening the window, using a ratio-based SLI instead of a quantile, and attaching a minimum request count condition.

Second, alerting on GPU utilization. DCGM_FI_DEV_GPU_UTIL measures only whether a kernel was resident, so on a serving GPU it is nearly always high. Put a threshold on it and it fires constantly in a healthy state, or set it low and it misses real problems. This metric is context, not an alert.

Third, alerting on KV cache usage. A high value here is generally the intended state, because reserving a large cache and using it is normal operation. What deserves an alert is not the usage but its consequences, preemption and queueing.

Fourth, reading missing data as an outage. This one is especially common in GPU environments. As the DCGM documentation warns, profiling metrics can only be read in certain groups due to hardware constraints and are automatically multiplexed, and collecting at too high a frequency returns zeroes. Compare those zeroes directly against a threshold and you manufacture an outage that never happened.

# Only evaluate when there is enough traffic to be meaningful
(
  sum by (model_name) (rate(vllm:time_to_first_token_seconds_count[30m])) > 0.1
)
and (
  1 - vllm:ttft_good_ratio:rate1h > 0.05
)

# Distinguish zero from missing
absent(vllm:num_requests_running) or (vllm:num_requests_running >= 0)

Closing — Most of the Outcome Is Decided When You Pick the Metric

The hard part of SLO work is not choosing thresholds. It is choosing the metric. Start from availability and however carefully you compute burn rates, you end up managing a number unrelated to user complaints.

The order, summarized: if it is interactive, target time to first token and merely observe throughput. Pick thresholds from bucket boundaries that actually exist in the histogram. Alert only on symptoms and send every cause metric to a dashboard. Follow those three lines and the alert count drops sharply while the remaining alerts become trustworthy.

The last post covers what to check, and in what order, once an alert does fire.

Try It Yourself

Series

References

현재 단락 (1/79)

The first proposal in any meeting about setting an SLO is availability: let us make the rate of endp...

작성 글자: 0자원문 글자: 8,579자작성 단락: 0/79