Skip to content
Published on

Inside vLLM (3) — How Continuous Batching Keeps the GPU Busy

🇰🇷KO🇺🇸EN🇯🇵JA🇨🇳ZH
Split View필사 모드
Share
Authors

How Static Batching Leaves the GPU Idle

Part 2 looked at memory, so this part looks at time: the question of how to keep the GPU from ever resting.

A GPU is efficient only when it has a lot of work to do at once, so requests get grouped into batches. Everyone does this much the same way. The problem starts after that. A naive implementation forms a batch once and does not change its composition until every request in it is done.

Static batching — the slot stays empty until the whole batch finishes

Step →     1  2  3  4  5  6  7  8  9 10 11 12
Request A  ■  ■  ■  ✓
Request B  ■  ■  ■  ■  ■  ■  ■  ■  ■  ■  ■  ✓
Request C  ■  ■  ■  ■  ■  ✓
Request D  ■  ■  ✓

■ computing   ✓ done   blank = GPU doing nothing in that slot

Request D finished in 3 steps, but its slot sits empty all the way through step 12. Worse, even if request E is already sitting in the queue, it cannot even start until this batch finishes. This loss is especially large for LLMs because response length varies to an extreme degree. Within the same service, one answer might be 20 tokens and another 2000.

Everything here was verified against the official documentation and source on 2026-08-12. vLLM moves fast, so re-check settings and behavior against the documentation for the version you are running.

Iteration-Level Scheduling

The idea behind continuous batching is simple. Batch composition gets redecided per step, not per request. Every time the model runs once, the scheduler asks again: is anything finished? If so, pull it out, and put a waiting request into that slot.

Continuous batching — the next request slots in the moment a spot opens up

Step →     1  2  3  4  5  6  7  8  9 10 11 12
Request A  ■  ■  ■  ✓
Request B  ■  ■  ■  ■  ■  ■  ■  ■  ■  ■  ■  ✓
Request C  ■  ■  ■  ■  ■  ✓
Request D  ■  ■  ✓
Request E         ■  ■  ■  ■  ✓
Request F            ■  ■  ■  ■  ■  ■  ✓
Request G                     ■  ■  ■  ■  ■  ✓

Three more requests fit into the same 12 steps. The total amount of work the GPU did is roughly the same, but the number of requests it processed went up. This is the core of vLLM's throughput.

One constraint follows naturally from this. Slotting a new request in mid-stream requires being able to secure that request's KV cache blocks on the spot. Without the block structure from part 2, this flexibility does not hold. Continuous batching and PagedAttention are, in effect, a matched set.

Prefill and Decode Have Different Characters

Everything up to this point shows up in the description of any serving engine. Real-world tuning turns on the fact that comes next. The lifetime of a single request has two phases with completely different characters.

Prefill is the phase that computes the KV for the entire input prompt at once. A 4000-token prompt means processing all 4000 tokens in parallel. The amount of computation is large, and the GPU's compute units stay busy.

Decode is the phase after that, where tokens come out one at a time. Each request processes exactly one new token per step. The amount of computation is small, but the model weights and the entire KV cache still have to be read. Reading memory becomes the bottleneck rather than computation.

In practice, this difference shows up like this. If someone pastes in a 40,000-token document and asks for a summary, other users' token output visibly stutters while that prefill runs. A single prefill chunk ends up taking over an entire step by itself.

Chunked Prefill: Mixing Both Characters Into One Batch

The fix is to stop processing a long prefill in one piece and instead cut it up across multiple steps. This is chunked prefill. A single step's batch then becomes a mix: part of a long prompt's prefill, plus decode tokens from several other requests.

Without chunked prefill
  Step N   : [40000-token prefill]                   ← every other request waits
  Step N+1 : [decode · decode · decode · decode]

With chunked prefill
  Step N   : [prefill chunk · decode · decode · decode]
  Step N+1 : [prefill chunk · decode · decode · decode]
  Step N+2 : [prefill chunk · decode · decode · decode]

The reason V1 does this naturally comes down to scheduler design. According to the official V1 guide, V1's unified scheduler treats prompt tokens and output tokens the same way, without distinguishing between them, and divides up a fixed token budget by deciding how many tokens each request gets this step. Because it does not draw a hard line between a prefill phase and a decode phase, features like chunked prefill and prefix caching run within a single framework.

The default changed too. The V1 guide states that chunked prefill is now on by default whenever possible, and contrasts this with V0, where it was enabled conditionally depending on model characteristics. In the SchedulerConfig source on the main branch, the declared default for enable_chunked_prefill is also true.

Token Budget: One Single Dial

So the value that sets the total number of tokens allowed into a step is max_num_batched_tokens, and the cap on the number of sequences processed at once is max_num_seqs. In SchedulerConfig on the main branch, the class defaults for these are declared as 2048 and 128 respectively. That said, the value actually applied can be adjusted depending on usage context and environment, so check your startup logs.

The official optimization documentation points in this direction. Keep max_num_batched_tokens small, around 2048, and the gap between tokens improves, meaning streaming feels less choppy. Set it large, and time to first token improves. If throughput is the priority, the documentation recommends setting this value above 8192, and adds that this especially applies when running a small model on a large GPU.

The documentation also spells out a condition to watch for. With chunked prefill turned off, max_num_batched_tokens has to be larger than max_model_len. Otherwise there is no way to fit a maximum-length prompt into a single step when one arrives.

What to Adjust First

To sum up, here is the order to work through.

If the complaint is that streaming feels choppy, look at lowering max_num_batched_tokens. There is a good chance a long prompt is blocking shorter requests.

If the complaint is that the first response is slow, look at raising it instead. Just be aware that individual users' token output may get a bit less smooth as a result. The two metrics share the same budget, so improving one pushes back on the other.

If concurrency is high but throughput will not climb, look at max_num_seqs together with how much KV cache headroom is left. If requests are getting blocked by insufficient cache rather than the sequence cap, the preemption covered in the next part may already be happening.

Try It Yourself

References

Discuss on Twitter • View on GitHub