Skip to content
Published on

Inside vLLM (4) — The Scheduler and Preemption, Where Throughput Collapses

🇰🇷KO🇺🇸EN🇯🇵JA🇨🇳ZH
Split View필사 모드
Share
Authors

The Problem the Scheduler Solves Every Step

Part 3 showed that vLLM reassembles the batch on every step. The scheduler is what actually makes that call. On every step, the scheduler has to find a combination that satisfies two limits at the same time.

One is the token budget. The total number of tokens processed this step must not exceed the cap. Looking at the V1 scheduler source, every time a request is admitted, that amount gets subtracted from the budget, and the running total is checked against the cap.

The other is KV cache blocks, and this one is much trickier. The token budget is just arithmetic, but blocks have to actually be free right now. And the requests that are already running also ask for more blocks every step as they generate new tokens. In other words, the resource the scheduler is working against does not hold still. It keeps shrinking.

Everything here was verified against the official documentation and source on 2026-08-12. vLLM moves fast, so re-check settings and behavior against the documentation for the version you are running.

The Waiting Queue and the Running Queue

The structure itself is simple. The V1 scheduler source has a queue holding waiting requests sitting alongside a list of running requests. The running list is an ordinary list, while the waiting queue is built as a different structure depending on the scheduling policy.

[Waiting queue]  requests that have never run yet, or were preempted
     │
     │  promoted once blocks are secured
     ▼
[Running list]  requests that go into this step's batch
     │
     │  preempted and sent back if blocks run short
     └──────────────▶ back to the waiting queue (at the front)

That arrow going back is the real subject of this post. And there is one important detail. A preempted request is placed at the front of the waiting queue, not the back. This can be confirmed in the source by the operation that reinserts the request at the front of the queue. It means a request that was just pushed out becomes a candidate again immediately, which avoids a situation where a specific request keeps getting pushed back and starves.

Preemption: Pushing Back a Request That Was Already Running

When the KV cache runs short, the scheduler does not stop at simply refusing new requests. It forces a request that is already running to step back. This is preemption.

The reason it goes this far is that there is no alternative. If every running request needs more blocks to produce its next token and there are no blocks left, someone has to leave so the rest can move forward. If no one leaves, everything stalls.

The official optimization documentation shows an example of the warning this produces.

WARNING 05-09 00:49:33 scheduler.py:1057 Sequence group 0 is preempted by
PreemptionMode.RECOMPUTE mode because there is not enough KV cache space.

The log format and line number vary by version, but the part to focus on is the last clause: that there is not enough KV cache space. If this line keeps showing up repeatedly in the logs, that deployment is already running past its capacity.

What makes preemption dangerous is that its cost is hard to see. The request does not fail. No error gets raised. It just gets slower, and not on average but out at the tail. This creates a situation where the average latency on the dashboard looks fine while a subset of users end up waiting unusually long.

Recompute and Swap

What happens to the KV cache of a request that got pushed back? Historically, there have been two approaches.

Recompute is the approach of simply discarding it. When the request runs again later, computation starts over from the prompt. It recovers memory fully and immediately, at the cost of throwing away all the computation done so far. Looking at the V1 scheduler source, it marks the preempted request's state and resets the number of tokens computed so far back to 0. That one line lays bare the fact that progress gets reset.

Swap is the approach of moving the KV cache to CPU memory and bringing it back later. It saves the computation, but it costs the price of shuttling data back and forth between GPU and CPU.

Here is what matters as of now. The official documentation states that vLLM V1's default preemption mode is recompute, not swap, and that the overhead of recompute is smaller under the V1 structure. The V1 guide also lists GPU-CPU KV cache swap among the features that have been removed. So do not try to apply a swap-related setting you saw in an older post as-is. Check first whether that argument actually exists in the version you are running.

Scheduling Policy: fcfs and priority

The scheduler settings include a policy field. In the SchedulerConfig source on the main branch, the default value of policy is fcfs, meaning first-come, first-served. The other option is priority, which follows the priority assigned to a request. priority also appears in the list of extra parameters for the OpenAI-compatible server.

The policy also changes how the target of preemption gets chosen. Looking at the V1 scheduler source, under the priority policy, the request ranked lowest by priority and arrival time among the running requests is the one pushed back, while under other policies, the last request in the list is pulled out.

In practice, where this choice matters is clear. If every request is equally important, first-come-first-served is enough. But if interactive requests and large batch jobs share one endpoint, the batch job will push out interactive users unless priority is used. Before switching policy, though, there is a question to ask first: is the problem you are seeing an ordering problem or a capacity problem. If capacity is insufficient, changing only the order just changes who loses out. The total amount of work does not change.

What to Do When You See Preemption

The official optimization documentation lays out four responses: raise gpu_memory_utilization to give the KV cache more room, lower max_num_seqs or max_num_batched_tokens to reduce how many requests are alive in a step, raise tensor_parallel_size, or raise pipeline_parallel_size.

The first two and the last two are different in nature. The first two change how allocation is done within the GPU you already have, while the last two mean using more GPU. It is better to try them in that order. In most deployments, preemption goes away just by fixing the allocation. It may seem like lowering max_num_seqs would hurt throughput, but the opposite is often true in practice. A state where preemption keeps repeating is already a state of wasting work on recompute, so reducing the number of concurrent requests to eliminate preemption tends to favor net throughput.

Try It Yourself

References

Discuss on Twitter • View on GitHub