Skip to content
Published on

Inside vLLM (1) — The Full Path From One Request to One Token

🇰🇷KO🇺🇸EN🇯🇵JA🇨🇳ZH
Split View필사 모드
Share
Authors

What This Series Covers

Anyone who has served an LLM in production tends to hit the same walls in the same order. First you are just glad it runs at all. Then you start wondering why it is so slow. Eventually you need to know why memory suddenly blows up. The answers to all three questions live inside vLLM.

This series treats vLLM as a structure, not as the end result of a fast server. There are seven parts total, and this first one is the map. It walks the full path a single request takes from arrival to token output, start to finish, and the remaining parts zoom into each segment.

Before starting, a note on versions is necessary. vLLM moves fast and has gone through a major internal rewrite. The official V1 guide states that the scheduler, KV cache manager, worker, sampler, and API server were all redesigned. So this series describes the structure as it stands after V1. If an older post disagrees with something here, it is usually describing the V0 era.

Everything here was verified against the official documentation and source on 2026-08-12. vLLM moves fast, so re-check settings and behavior against the documentation for the version you are running.

The Full Path of One Request

Before walking through it in words, here is the picture.

Client HTTP request
        │
        ▼
[API server process]   Receive HTTP, process input, tokenize
        │
        ▼
[Engine core process]  Scheduler + KV cache manager
        │             "who gets how many tokens this step"
        ▼
[GPU worker process]   ModelRunner → model forward → logits
        │
        ▼
      Sampler           Picks one next token from the logits
        │
        ├───────────▶ Stream one piece back to the client
        │
        └───────────▶ Unfinished requests go back to the scheduler

The most important part of this picture is the last arrow. After a single token comes out, the request is not done. It goes back to the scheduler. This is not a structure that finishes one request before looking at the next. Every live request advances a little on every step. The story of vLLM performance ultimately comes down to how tightly this loop gets packed.

The API Server: Where Requests Become the Engine's Language

vLLM has two entry points. According to the official architecture documentation, offline inference is handled by the LLM class, and online serving is handled by the vllm serve command or the OpenAI-compatible API server. That OpenAI-compatible server uses AsyncLLMEngine. Because the async wrapper runs a background loop, multiple concurrent HTTP requests do not have to wait on each other.

What happens here is not glamorous. It applies the chat template, turns strings into token sequences, and packages the sampling settings into a request object the engine understands. But performance already starts to diverge at this stage. How the prompt is assembled determines whether it hits the prefix cache discussed in part 5, and the input length runs straight into the length limits covered in part 6.

Engine Core: Scheduler and KV Cache Manager

The official documentation lays out the V1 process layout like this: an API server process that receives HTTP, one engine core process per data-parallel rank that runs the scheduler and manages the KV cache, and one worker process per GPU. In the example the documentation gives, a deployment with 4 GPUs comes out to 1 API server + 1 engine core + 4 GPU workers, 6 processes total.

The reason for splitting processes is simple. HTTP parsing and tokenization are Python work that uses the CPU, and model execution uses the GPU. If both live in the same process, CPU work blocks the GPU loop. Look inside a deployment that is not hitting expected throughput and it is common to find low GPU utilization alongside one CPU core pegged at 100 percent.

Inside the engine core, the scheduler and the KV cache manager move as one unit. Before the scheduler can decide that a request goes into this step, it has to actually be able to secure that many KV cache blocks. When it cannot, the scheduler will even push back a request that is already running. That preemption behavior is the topic of part 4.

Worker and Sampler: The Layer That Actually Touches the GPU

The architecture documentation's definitions are terse. Worker is the process that runs model inference, ModelRunner is responsible for loading and running the model, and the Model inside it is the actual torch module instance.

One forward pass from the worker produces next-token logits for every request in the batch. The sampler then applies settings like temperature, top_p, and top_k to pick a single token. What matters is that requests with different sampling settings can still sit in the same batch. The batch only shares the model computation; what gets picked out is decided separately per request.

Why Configuration Collapses Into One Object

One design the documentation emphasizes is VllmConfig. The idea is to build a single config object that holds everything needed and pass that around. This makes it possible to apply sharding and quantization together at initialization time, instead of loading the model and then patching its weights afterward. For huge models this difference is decisive, because you cannot load a model that does not fit on one GPU and then split it up after the fact.

This structure also affects operations. It is why values like --max-model-len and --gpu-memory-utilization freeze in place at startup and then affect everything downstream. That is why tuning vLLM is, for the most part, a matter of designing the startup arguments rather than adjusting things at runtime.

The map is drawn now. Starting with the next part, we zoom into the segments of this path where the most people trip.

Try It Yourself

References

Discuss on Twitter • View on GitHub