Split View: vLLM 내부 구조 (6) — 컨텍스트 윈도우, max_model_len, max_tokens 완전 정리

Inside vLLM (6) — Context Window vs max_model_len vs max_tokens, Fully Explained

• Why These Five Get Confused
• The Comparison Table at a Glance
• Context Window and max_model_len
• max_tokens and the Renaming Story
• max_num_batched_tokens and max_num_seqs
• Scenario 1 — An 8k Model With 6k Input and max_tokens 4000
• Scenario 2 — Why a 128k Model Gets Blocked at 8192
• Scenario 3 — Blocked at Startup, Before Any Request
• What Actually Sets the Limit Is KV Cache Memory
• Tracing the Cause From the Error Message
• Try It Yourself
• References

Why These Five Get Confused

The most common question when running vLLM in production is not about performance — it is about length. Why does a model with a 128k context window get blocked at 8192? Why does a request with max_tokens set to 4000 get rejected? And what exactly is max_num_batched_tokens?

The reason for the confusion is clear. The names all sound similar, they get set in different places, and the symptom when you exceed each one is different. Exceeding one gets the request rejected, exceeding another quietly truncates the response, and exceeding a third keeps the server from starting at all.

This post sorts out all five at once. Here is the order: first a comparison table to get the whole map, then an explanation of each one, then three scenarios you actually run into, worked through with real numbers, and finally a method for tracing an error message back to its cause.

Everything here was verified against the official documentation and source on 2026-08-12. vLLM moves fast, so re-check settings and behavior against the documentation for the version you are running.

The Comparison Table at a Glance

The most important row in the table is the second one. What max_model_len limits is not the input — it is the sum of input and output. The official engine arguments documentation states explicitly that this value is the model context length combining the prompt and the output. Half of all the confusion about length comes from this one point.

Context Window and max_model_len

The two live at different layers.

The context window is a property of the model. It is a structural limit that comes from how the model was trained, so it is not something you can extend by editing a config file. It is the number written as 128k on the model card.

max_model_len is a setting on the engine. According to the official documentation, if you do not specify this value it is derived automatically from the model config. In other words, if you give it nothing, it generally follows the model's context window. But if you specify it, that value becomes the actual limit.

This clears up the first misconception. The fact that a model supports 128k does not mean your server accepts 128k. What your server accepts is max_model_len. And keeping this value lower than the model's maximum is usually not a mistake — it is usually a deliberate choice. The reason is covered later.

Here is also why this post does not write down a default value as a number. The official documentation does not give this argument a fixed default — it only explains that the value is derived from the model config. So check your own startup log to see what number your deployment actually landed on.

max_tokens and the Renaming Story

max_tokens is a value you put in the request body, and it is the upper bound on the number of tokens that request will newly generate. Input length does not count toward it. So this value on its own only sets how long the answer can be — it has nothing to do with how long a question it can accept.

Here is one piece of current information worth noting. In the chat completion request definition on the main branch, the max_tokens field is marked deprecated, with guidance to use max_completion_tokens instead. And if both values are supplied, max_completion_tokens takes priority. This change follows the direction the OpenAI-compatible API has moved in. If you are writing new code, it is better to use max_completion_tokens. If existing code still uses max_tokens, it will not break right away, but you can treat it as debt to clean up eventually.

What happens if you do not supply this value at all? Generation continues using whatever budget remains, until the model ends the response on its own or hits the limit. From a service standpoint, specifying an upper bound is almost always better. Without one, a single user can end up holding the KV cache for a long time, and that leads to the preemption covered in part 4.

max_num_batched_tokens and max_num_seqs

These two look like length limits because of their names, but they are a completely different kind of thing. They are not a limit on any single request — they are a limit on how much work the engine processes in one step.

max_num_batched_tokens is the upper bound on how many tokens the whole batch processes in this step. A single request is not rejected for exceeding this value. If chunked prefill is on, it is simply split across multiple steps.

max_num_seqs is the upper bound on how many sequences can be packed in at once, in one step. Exceeding it means waiting, not failing.

In SchedulerConfig on the main branch, the class defaults are declared as 2048 and 128 respectively. That said, the value actually applied can be adjusted based on usage context and environment, so check your startup log.

There is exactly one point where these two intersect with the length question. The official optimization documentation states that when chunked prefill is off, max_num_batched_tokens must be larger than max_model_len. The reasoning: if the engine cannot split the work across steps, and a maximum-length prompt cannot fit into one step, there is no way forward.

Scenario 1 — An 8k Model With 6k Input and max_tokens 4000

Start with the most common case. You have a model deployed with a context of 8192, and a user pastes in a 6000-token document while setting max_tokens to 4000.

max_model_len = 8192  (the budget input + output share)

input   6000 ██████████████████████████████
output  4000 ████████████████████
total  10000 ──────────────────────────────────────────  exceeds 8192

result: the request is rejected before generation even starts.

The most common assumption beginners make here is "input 6000 is less than 8192, so it will go through, and the answer will get cut off somewhere around 2192 tokens." That is not what happens. vLLM checks the sum of the input and max_tokens at the moment it receives the request, and rejects it if that sum is over the limit.

Here is the actual form the message takes. This is the original text as reported in a public vLLM issue, copied verbatim.

This model's maximum context length is 16384 tokens. However, you requested
122946 tokens (112946 in the messages, 10000 in the completion).
Please reduce the length of the messages or completion.

Pulling the message apart, the structure is right there: the parenthetical splits into input and output, and the sum of the two is what gets compared against the limit stated earlier. This one line is the clearest possible proof that input and output draw from the same budget.

So there are three possible fixes. Reduce the input, reduce max_tokens, or increase max_model_len. The first two are changes on the request side; the last one requires restarting the server. As a side note, there is an enhancement request issue open in the vLLM repository about this behavior of rejecting instead of automatically truncating. In other words, this is an area that can change between versions, so check it directly on the version you are running.

One practical tip. If your application hardcodes max_tokens as a constant, you will hit this error every time a long input comes in. It is more robust to count the input tokens first, then compute that value as whatever budget remains, minus a safety margin.

Scenario 2 — Why a 128k Model Gets Blocked at 8192

The model card says 128k, but the server rejects requests at a much smaller value. Here is the order to check things in.

First, check whether --max-model-len was ever passed explicitly in the startup command. If it was, that value is the answer. Regardless of what the model is capable of, the limit the engine was given takes priority.

Second, if it was not, look at the value the engine derived from the model config. As the official documentation states, if you do not specify it, the value is automatically derived from the model config, and this value can differ from the marketing language on the model card. Models that support long context through some extension technique sometimes have a smaller default window written into their config file.

Third, trust the number the error message states. In the message shown earlier, the value that follows "maximum context length is" is the effective limit of this engine right now. That number is the truth, more than the model card is.

Fourth, if you genuinely need 128k anyway, start by calculating whether your KV cache can support it. And that leads into the third scenario.

Scenario 3 — Blocked at Startup, Before Any Request

This is the case where you set --max-model-len to a large value and the server does not start at all. Here is the form reported in a vLLM issue.

ValueError: The model's max seq len (4096) is larger than the maximum number
of tokens that can be stored in KV cache (3664). Try increasing
gpu_memory_utilization or decreasing max_model_len when initializing the engine.

This message might be the single most important thing in this post, because it exposes the real mechanism underneath. At startup, the engine loads the weights and allocates whatever memory remains to the KV cache. Then it calculates how many tokens that cache can hold. If that number is smaller than max_model_len, it means the engine could not even complete a single maximum-length request, so it aborts startup.

In other words, the final authority on the length limit is neither the model nor the configuration — it is memory.

What Actually Sets the Limit Is KV Cache Memory

Generalizing the scenarios above gives this picture.

Total GPU memory
  └─ vLLM uses the fraction set by gpu_memory_utilization
       ├─ model weights          (determined by model size and quantization)
       ├─ activations and other overhead
       └─ everything left = KV cache    ← this is what eats length and concurrency together

The KV cache faces two demands at the same time. One is length: the longer a single request is, the more blocks it uses. The other is concurrency: the more requests there are, the more blocks get used. So if you double max_model_len, the number of concurrent requests the same memory can sustain roughly gets cut in half.

In practice, this relationship shows up like this: leaving 128k open reduces concurrent capacity even if no user is actually using 128k, because the engine only starts if it can handle the worst case. That is why most production deployments set max_model_len to the upper percentile of the actual workload, not to the maximum the model supports. This is the reason stated earlier for why keeping this value low is not a mistake.

Tracing the Cause From the Error Message

Last, a table for working backward from the symptom.

To sum up, there is one procedure. First, work out whether the error happened at the request stage or the startup stage. If it is the request stage, look at the sum of input and the generation cap. If it is the startup stage, look at the relationship between memory and max_model_len. Just separating these two branches resolves most length problems within ten minutes.

Try It Yourself

• LLM GPU Memory (VRAM) Calculator — How high you can set max_model_len is ultimately decided by the KV cache. Enter your model and GPU, and try changing the length. This reproduces scenario 3 from this post with real numbers.
• LLM API Cost Calculator — Compare how input and output tokens each factor into cost. Designing for length is not just a performance decision — it is a cost decision too.
• Previous: Inside vLLM (5) — Prefix Caching, and Why System Prompt Design Is Performance
• Next: Inside vLLM (7) — Deployment Tuning, Common Pitfalls, and OOM Triage

References

• vLLM Engine Arguments (docs.vllm.ai) — Source for the statement that max-model-len is the length combining the prompt and the output, and that it is derived from the model config when not specified.
• vLLM Optimization and Tuning (docs.vllm.ai) — Source for the condition that max_num_batched_tokens must be larger than max_model_len when chunked prefill is off.
• vLLM chat completion request definition source (GitHub main) — Where the deprecation marker on max_tokens and the priority given to max_completion_tokens were read.
• vLLM issue 20409 — Source for the real error message showing that input and the generation cap are checked as a sum.
• vLLM issue 2418 — Source for the error message showing startup failing due to insufficient KV cache capacity.
• vLLM issue 42474 — Enhancement request about the behavior of rejecting instead of truncating when the generation cap is exceeded.