Skip to content
Published on

vLLM Metrics — What to Chart and What to Alert On

🇰🇷KO🇺🇸EN🇯🇵JA🇨🇳ZH
Split View필사 모드
Share
Authors

Introduction — Scrape the Endpoint Once

The previous post separated the questions GPU metrics answer from the ones they do not. DCGM tells you the card is hot and how full memory is; it does not know how many requests are backed up right now or how long a user waited for the first character. Only the application can answer that.

vLLM publishes those answers in Prometheus format. The endpoint path is /metrics.

kubectl -n serving port-forward svc/vllm 8000:8000

curl -s localhost:8000/metrics | grep '^vllm:' | head -30

Metric names and configuration were verified against the official documentation and repositories on 2026-08-12. They can differ between releases, so check again against the version you are running.

Scheduler State — How Many Are Running, How Many Are Waiting

Two gauges come first. vllm:num_requests_running is, in the words of the documentation, the number of requests in model execution batches, and vllm:num_requests_waiting is the number of requests waiting to be processed. The ratio between them says nearly everything about the server's condition. When the running count pins at some value and stops climbing while the waiting count keeps rising, that is saturation.

There is also a gauge that breaks down the reason for waiting. It is vllm:num_requests_waiting_by_reason, carrying a reason label. The documentation names two values: capacity for waiting on scheduling capacity, and deferred for requests deferred by transient constraints such as the LoRA budget or KV transfer. That distinction earns its keep in practice. The former is usually a capacity problem and the latter usually a configuration one.

One counter must be read alongside these: vllm:num_preemptions, documented as the cumulative number of preemptions from the engine. Preemption is what happens when the KV cache runs short and a running request gets rolled back, so a rising counter means capacity pressure is already affecting latency.

# Running and waiting
sum by (model_name) (vllm:num_requests_running)
sum by (model_name) (vllm:num_requests_waiting)

# Waiting, broken down by reason
sum by (model_name, reason) (vllm:num_requests_waiting_by_reason)

# Preemption rate
sum by (model_name) (rate(vllm:num_preemptions_total[5m]))

KV Cache and Prefix Cache

vllm:kv_cache_usage_perc is KV cache usage, and the documentation states outright that 1 means 100 percent usage. When this sticks to the ceiling, the queue grows and preemption begins. Read together with the three metrics above, the causal chain fits on one screen: cache fills, preemption starts, waiting grows, latency degrades.

The prefix cache counts queries and hits separately, as vllm:prefix_cache_queries and vllm:prefix_cache_hits, and the documentation is explicit that both are measured in tokens queried and tokens cached. Tokens, not requests. Compute a hit rate without knowing that and the number comes out strange. If you use cross-instance cache sharing through a KV connector, vllm:external_prefix_cache_queries and vllm:external_prefix_cache_hits exist separately.

For token counters, vllm:prompt_tokens is prefill tokens processed and vllm:generation_tokens is generation tokens processed. Cached prompt tokens are counted separately as vllm:prompt_tokens_cached.

# KV cache usage
max by (model_name) (vllm:kv_cache_usage_perc)

# Prefix cache hit rate (token based)
sum by (model_name) (rate(vllm:prefix_cache_hits_total[10m]))
  / sum by (model_name) (rate(vllm:prefix_cache_queries_total[10m]))

# Generated tokens per second
sum by (model_name) (rate(vllm:generation_tokens_total[5m]))

Latency Histograms — TTFT and Everything After

The latency family is entirely histograms. Following one request through its life maps onto them in order.

vllm:request_queue_time_seconds is time spent in the waiting phase. vllm:time_to_first_token_seconds is time to first token. vllm:request_prefill_time_seconds and vllm:request_decode_time_seconds are time spent in the prefill and decode phases respectively, and vllm:request_inference_time_seconds covers the running phase as a whole. vllm:inter_token_latency_seconds is inter-token latency, vllm:request_time_per_output_token_seconds is time per output token per request, and the whole thing is vllm:e2e_request_latency_seconds.

Knowing the bucket boundaries helps when interpreting quantiles. Confirmed in the source, the buckets for vllm:time_to_first_token_seconds are 0.001, 0.005, 0.01, 0.02, 0.04, 0.06, 0.08, 0.1, 0.25, 0.5, 0.75, 1.0, 2.5, 5.0, 7.5, 10.0, 20.0, 40.0, 80.0, 160.0, 640.0, 2560.0. Note that there is no bucket between 1 and 2.5 seconds. A quantile landing in that range is an interpolation, not a measurement.

# TTFT p95
histogram_quantile(0.95,
  sum by (le, model_name) (rate(vllm:time_to_first_token_seconds_bucket[5m])))

# Inter-token latency p99
histogram_quantile(0.99,
  sum by (le, model_name) (rate(vllm:inter_token_latency_seconds_bucket[5m])))

# Share of end-to-end latency spent queueing
sum by (model_name) (rate(vllm:request_queue_time_seconds_sum[5m]))
  / sum by (model_name) (rate(vllm:e2e_request_latency_seconds_sum[5m]))

Request completions are counted by vllm:request_success, which carries a finished_reason label so you can separate normal completion from length-capped completion. And every series carries model_name and engine labels by default.

The Problem of Names Changing Between Versions

There is a trap here that must be named. Two documentation pages spell the same metric differently.

The usage page lists vllm:prompt_tokens and vllm:generation_tokens as Counters. The design page, meanwhile, shows vllm:prompt_tokens_total and vllm:generation_tokens_total. Checking the source, the registered names carry no suffix and the _total appears in the Prometheus exposition format. So your queries must use the _total names, and copying the documentation list verbatim yields no data at all. That is why only the counters in the examples above carry _total.

Version drift does not stop there. The design page explicitly marks several metrics for deprecation. vllm:num_requests_swapped and vllm:cpu_cache_usage_perc are described as no longer relevant in V1, and vllm:time_in_queue_requests is a deprecated duplicate of vllm:request_queue_time_seconds. Plenty of old dashboards still floating around the internet use those names as-is.

Conversely, some metrics appear only conditionally. The KV block lifetime family, vllm:kv_block_lifetime_seconds, vllm:kv_block_idle_before_evict_seconds, and vllm:kv_block_reuse_gap_seconds, is registered in the source only when the KV cache metrics option in the observability configuration is enabled. The same holds for vllm:spec_decode_num_accepted_tokens_per_pos, which appears only with speculative decoding, and the NIXL family, which appears only with a KV connector.

So before building a dashboard, scrape your own instance once and capture the real list of names. An unverified name becomes a quietly empty panel on a dashboard, and in an alert rule it becomes a rule that never fires.

What Belongs on a Dashboard Versus in an Alert

Draw a line between the two. A dashboard is where you find the cause; an alert is where you learn the user is hurting. Mixing them turns alerts into noise.

On the dashboard goes the whole causal chain. From the top: KV cache usage, preemption rate, waiting request count with its reason breakdown, running request count, prefix cache hit rate, generated tokens per second, and the latency histogram quantiles. Put the previous post's GPU metrics, SM activity and framebuffer usage, on the same time axis and cause tracing finishes on one screen.

In alerts goes far less. Two things the user actually feels are usually enough: the TTFT quantile crossing its target, and the request failure rate climbing.

groups:
  - name: vllm-serving
    rules:
      - alert: VLLMHighTTFT
        expr: |
          histogram_quantile(0.95,
            sum by (le, model_name) (rate(vllm:time_to_first_token_seconds_bucket[5m]))) > 2
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: 'TTFT p95 above 2s for {{ $labels.model_name }}'

      - alert: VLLMQueueGrowing
        expr: |
          sum by (model_name) (vllm:num_requests_waiting) > 50
          and sum by (model_name) (rate(vllm:num_preemptions_total[5m])) > 0
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: 'Queue growing with preemptions for {{ $labels.model_name }}'

Note that both rules carry a long for clause. Request lengths vary wildly on an inference server, so quantiles swing considerably over short windows even in a healthy steady state. The details of alert design continue in the next post.

Closing — Application Metrics Come First

However well you build a GPU dashboard, it cannot tell you whether users are slow. A busy GPU and a waiting user are separate facts, and they sometimes move in opposite directions. An idle card with a long queue is usually a KV cache or batching configuration problem.

So the order runs like this. Judge whether the user is hurting from application metrics first, then find out why from GPU metrics. Reverse the order and you end up staring at a green dashboard unable to explain the complaints.

The next post covers turning both layers of metrics into an actual promise: SLOs and alert design.

Try It Yourself

Series

References

Discuss on Twitter • View on GitHub