Split View: vLLM을 고쳐서 빠르게 만들기 — 설정, 내부 구조, 그리고 코드를 건드릴 지점

Making vLLM Fast — Configuration, Internals, and Where to Actually Touch the Code

• Introduction — Six Lines to Check Before You Open the Code
• First, What You Can Fix Without Touching Code
  • Eliminate preemption first
  • Trading off TTFT against ITL with the token budget
  • Startup time is performance too
  • Don't starve the CPU
  • The attention backend defaults to auto-selection
• PagedAttention — The Idea That Eliminates Fragmentation
• What the Continuous-Batching Scheduler Actually Decides
• Prefill and Decode Are Fundamentally Different Kinds of Work
• Benchmarking — What to Hold Fixed, and What to Measure
  • The exact definitions of what you're measuring
  • The actual commands
  • What must be held fixed
  • Reading it as a curve
  • When you want to pin down the bottleneck
• Places Actually Worth Touching the Code
  • 1. Custom logits processors — the safest place to touch
  • 2. Custom attention backends — high value, high cost
  • 3. Scheduler policy — the last resort
  • The cost of tracking upstream
• Closing — Tuning Without Measurement Is Just a Statement of Taste
• References

Introduction — Six Lines to Check Before You Open the Code

When someone asks "vLLM feels slow, should I patch the source," I always check the same things first: whether preemption warnings are showing up in the logs, what max_num_batched_tokens is set to, whether prefix caching is on, what the GPU memory utilization ratio is, how many physical cores there are, and exactly what you're measuring right now.

Most of the time, the answer is in these six lines. Situations that actually require touching the source are rarer than you'd think, and even when you do need to, there are only a handful of places worth touching.

This post follows that order. The reference is vLLM v0.26.0 (PyPI release on July 25, 2026, Python 3.10 up to but not including 3.15), and the internals sections were written from directly reading that tag's source. vLLM ships a minor version roughly every two weeks, so the file paths and argument names in this post can drift out of date within a few months. Get in the habit of checking against your own version's source.

One thing up front: the V0 engine is fully deprecated. The official docs state "We have fully deprecated V0" and point to RFC #18571. Most V0-era tuning posts still floating around the internet no longer apply.

First, What You Can Fix Without Touching Code

Ordered by priority. The higher up the list, the bigger the payoff and the cheaper the cost.

Eliminate preemption first

The single most common root cause. When the KV cache runs short, vLLM preempts a running request and recomputes it later. If a line like this repeats in the logs, no other tuning matters yet.

WARNING ... Sequence group 0 is preempted by PreemptionMode.RECOMPUTE mode
because there is not enough KV cache space. This can affect the end-to-end
performance. Increase gpu_memory_utilization or tensor_parallel_size ...

V1's default preemption mode is recompute, not swap. The preempted request's prefill is redone from scratch. So frequent preemption collapses both throughput and latency together. The docs spell out the fix directly: raise gpu_memory_utilization, lower max_num_seqs and max_num_batched_tokens, or raise tensor_parallel_size to increase KV space per GPU.

Trading off TTFT against ITL with the token budget

max_num_batched_tokens is the total number of tokens that can be scheduled in one step. The official docs state the direction clearly.

• A smaller value (e.g. 2048) improves ITL. The prefill chunk that delays decoding gets smaller.
• A larger value improves TTFT. You can push more prefill tokens into a single batch.
• If throughput is the goal, the docs recommend going above 8192 — especially when a small model sits on a large GPU.

There's one trap to watch for. With chunked prefill turned off, max_num_batched_tokens must be larger than max_model_len, or the server can die at startup.

from vllm import LLM

# Conversational service: prioritize per-token latency
llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct", max_num_batched_tokens=2048)

# Batch processing: prioritize throughput
llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct", max_num_batched_tokens=16384)

Startup time is performance too

If you repeatedly spin up the same model with the same settings, the docs offer three levers.

• Reuse the compilation cache. torch.compile artifacts are stored under VLLM_CACHE_ROOT (a cache directory under home by default), and you can bake this directory into your container image or copy it between machines. Setting VLLM_FORCE_AOT_LOAD=1 makes a cache miss fail explicitly instead of silently recompiling. The cache is invalidated if any of the model, config, related environment variables, torch build, or GPU model changes.
• Skip memory profiling with --kv-cache-memory. On startup, vLLM logs a value that reproduces the current allocation. Passing that value on the next boot skips the measurement step. There's a cost, though: the KV cache is now pinned to the specified value instead of the measured one, so setting it conservatively cuts concurrency and setting it optimistically causes allocation failures. Valid only for the same GPU with the same initial free memory.
• --enforce-eager. Skips both compilation and CUDA graph capture entirely. Startup is fastest, and steady-state decode performance is worse. This is for the dev loop, and also useful for measuring how much of boot time is spent on compilation.

Don't starve the CPU

An easily overlooked item worth calling out separately. vLLM V1 is a multi-process architecture. With N GPUs, you get one API server, one engine core, and N GPU workers — at least N plus 2 processes competing for CPU.

The docs nail the minimum down to physical cores. If hyperthreading is on, one vCPU is half a physical core, so you need twice the vCPUs. The engine-core process in particular is sensitive to CPU starvation because it runs a busy-wait loop. If GPU utilization is inexplicably low in a virtualized environment, this is the first thing to suspect.

The attention backend defaults to auto-selection

vLLM looks at the GPU architecture, the model, and the config, then picks the first compatible backend from a priority list. As of v0.26.0, the priority order for standard attention is this.

To override it manually, do this. Specifying an incompatible backend raises an error with the reason attached.

vllm serve Qwen/Qwen3-8B --attention-backend FLASH_ATTN
# or via structured config
vllm serve Qwen/Qwen3-8B -ac.backend FLASH_ATTN

PagedAttention — The Idea That Eliminates Fragmentation

From here on we're in internals territory. The observation vLLM started from is simple: if you allocate the KV cache as one big contiguous chunk per request, most of that memory is wasted.

The waste comes in three flavors: internal reservation, allocated ahead of time in case a request runs to its max length; over-allocation, set aside but never actually used before the request ends; and external fragmentation, created by returning and re-acquiring variously sized chunks.

The fix is lifted straight from OS virtual memory. Cut the KV cache into fixed-size blocks, and keep a block table that maps a logically contiguous sequence onto physically scattered blocks.

Request A's logical KV:  [t0 t1 t2 t3] [t4 t5 t6 t7] [t8 t9 __ __]
                              │              │              │
Block table A         →   block 7        block 3        block 12

Request B's logical KV:  [t0 t1 t2 t3] [t4 t5 __ __]
                              │              │
Block table B         →   block 7        block 5
                              ↑
                    shared prefix means sharing the same block (prefix caching)

Two results follow. First, there's no waste except in the leftover space of the last block. The original paper describes this as "near-zero waste in KV cache memory." Second, block-level sharing becomes possible. Requests using the same system prompt physically share the prefix blocks, and this is prefix caching. The paper reports, from these two effects, 2x to 4x throughput at the same latency compared to FasterTransformer and Orca (Kwon et al., SOSP 2023).

In v0.26.0 this logic lives under vllm/v1/core/. kv_cache_manager.py handles per-request block allocation, block_pool.py handles the block pool and hash-based reuse, and kv_cache_coordinator.py coordinates models that use several kinds of cache together (hybrid attention).

The practical implication is this: prefix caching only wins when the system prompt is long and shared. If each request's prefix differs, all you're left with is the cost of hash computation and block management. Measuring before and after turning it on is the only correct way to judge.

What the Continuous-Batching Scheduler Actually Decides

This is what the schedule() method in vllm/v1/core/sched/scheduler.py does on every engine step. The comment at the top of the source captures the design exactly.

There's no "decoding phase" nor "prefill phase" in the scheduler. Each request just has the num_computed_tokens and num_tokens_with_spec. At each step, the scheduler tries to assign tokens to the requests so that each request's num_computed_tokens can catch up its num_tokens_with_spec.

This sentence is the key to understanding the V1 scheduler. The scheduler doesn't distinguish prefill from decode. Each request holds only "tokens computed so far" and "tokens that need to be computed," and every step allocates tokens so the former catches up to the latter. This single abstraction expresses chunked prefill, prefix caching, and speculative decoding all without any special-cased logic.

Here's the skeleton of the actual loop.

# A summary of the structure of vllm/v1/core/sched/scheduler.py (not the real code)
def schedule(self):
    token_budget = self.max_num_scheduled_tokens   # = max_num_batched_tokens

    # 1) RUNNING requests get served first. Decode has priority.
    for request in self.running:
        if token_budget <= 0:
            break
        num_new = min(need(request), token_budget)
        if not kv_cache_has_room(request, num_new):
            preempt(self.running.pop())     # preempt from the back
            continue
        schedule_tokens(request, num_new)
        token_budget -= num_new

    # 2) Attach WAITING requests with whatever budget remains. Prefill comes later.
    while self.waiting and token_budget > 0:
        if len(self.running) >= self.max_num_running_reqs:   # = max_num_seqs
            break
        request = self.waiting.peek()
        num_new = min(need(request), token_budget)
        # If it doesn't all fit, slice it in → this is chunked prefill
        schedule_tokens(request, num_new)
        token_budget -= num_new

You can see exactly where the values we set as config plug in.

• max_num_batched_tokens is the initial value of token_budget. The total work for one step.
• max_num_seqs is max_num_running_reqs. The cap on the length of the running queue.
• long_prefill_token_threshold is the cap on how many tokens a single prefill request can take in one step. It stops one long prompt from monopolizing the budget and starving other requests.

The important part is that decode gets allocated first. The policy is to take care of users already streaming a response first, then start new requests' prefill with whatever budget remains. So under load, TTFT degrades first and ITL holds up comparatively better.

Prefill and Decode Are Fundamentally Different Kinds of Work

The two phases use hardware in opposite ways.

The reason decode is memory-bound is simple: producing one token requires reading the entire model's weights once. With batch size 1, that read gets you 1 token; with batch size 64, the same read gets you 64. So decode throughput scales almost linearly with batch size, up until it hits the bandwidth wall.

The problem is that running the two phases separately loses on both sides. On a prefill-only step, the tensor cores saturate while the memory pipe idles. On a decode-only step, it's the opposite.

Chunked prefill addresses this head-on. A long prefill gets sliced and mixed into the same batch as decode requests. With a compute-bound task and a memory-bound task coexisting in one batch, both resources get used at the same time. In V1 it's on by default whenever possible.

The trade-off here is explicit. Smaller chunks disturb decode less, so ITL improves, but prefill gets spread across more steps, so TTFT gets worse. Larger chunks do the opposite. Which side is right is for the service to decide, not vLLM.

Benchmarking — What to Hold Fixed, and What to Measure

This is the most important section in this post. Everything above only means something if the measurement is honest.

The exact definitions of what you're measuring

The metrics vLLM's benchmark tool prints have their definitions pinned in the source.

Distinguishing TPOT from ITL matters. TPOT is the average for one request, and ITL is the distribution of individual gaps. If mean TPOT is 20ms but p99 ITL is 400ms, users experience "it stutters sometimes." Looking only at the mean, this doesn't show up at all.

The actual commands

# 1) Start the server. Pin the arguments you're tuning here.
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --max-num-batched-tokens 8192 \
  --max-num-seqs 256 \
  --gpu-memory-utilization 0.90 &

# 2) Apply load. Vary the request rate to plot several points.
vllm bench serve \
  --backend vllm \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --dataset-name sharegpt --dataset-path sharegpt.json \
  --num-prompts 500 \
  --request-rate 8 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,90,99 \
  --save-result --result-filename rate8.json

# 3) If you want the offline throughput ceiling, use this instead
vllm bench throughput --model meta-llama/Llama-3.1-8B-Instruct \
  --dataset-name sharegpt --dataset-path sharegpt.json --num-prompts 1000

What must be held fixed

For a single measurement to be trustworthy, all of the following must be held fixed. If even one shifts, the comparison becomes meaningless.

• Input and output length distributions. Same dataset, same seed. If you're using synthetic data, pin input and output lengths explicitly. Throughput can vary arbitrarily if the length distribution differs.
• Request arrival rate. Measuring under infinite load gives you the throughput ceiling but makes latency meaningless (it's all queue wait). Plotting a curve by varying the request rate is the only useful form. A single point tells you nothing.
• Warmup. The first requests get compilation, CUDA graph capture, and cache warming mixed in. Exclude them from the stats.
• Prefix cache state. If it's on and you repeat the same prompt, TTFT improves dramatically from the second request onward. Reporting this as an optimization result is a lie. Either flush the cache and measure again, or measure the steady state with the cache warm — pick one and stick to it.
• GPU clocks and neighbors. On shared hardware, make sure nothing else runs in the same window. Hitting a power limit drops clocks.
• Version. Record vLLM, PyTorch, driver, and image tag alongside the results.

Reading it as a curve

Instead of one number, measure while raising the request rate through 5, 10, 15, 20, and you get a shape like this.

   p99 TTFT
      ^
      |                                    ╱  ← the queue starts building up here
      |                                  ╱
      |                          ______╱
      |     ____________________╱
      +--------------------------------------> request rate (req/s)
                              the knee

  Set your operating point to the left of the knee.
  To the right of the knee is a region of "throughput exists,
  but latency is out of control."

The goal of tuning isn't maximum throughput, it's maximum throughput that satisfies your SLO. If your condition is p99 TTFT under 500ms, the metric is the request rate you can sustain while holding that condition. Plot several of these curves while varying max_num_batched_tokens, and you can see with your own eyes which value fits your service.

When you want to pin down the bottleneck

If the numbers are bad and you don't know why, use a profiler. But the docs lead with a warning: profiling is for developers, it slows inference down significantly, so end users should never turn it on. If you need low overhead, use Nsight Systems; if you need stack traces and tensor shapes too, use the PyTorch profiler.

# Attach a profiler to the server on startup (--profiler-config is v0.13.0+)
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --profiler-config '{"profiler": "torch", "torch_profiler_dir": "./vllm_profile"}'

# Collect over a bounded window
curl -X POST http://localhost:8000/start_profile
# ... send only a few requests. The trace gets very large ...
curl -X POST http://localhost:8000/stop_profile

# Can also be used together with a benchmark
vllm bench serve --backend vllm --model ... --profile --num-prompts 2

Collected traces are viewed in the Perfetto UI. Keeping the request count low matters. The docs note that dumping 100 requests' worth of trace on a 70B-class model takes about 10 minutes on an H100.

Places Actually Worth Touching the Code

If you've done all of the above and it still isn't enough, that's when you go to the source. There are realistically three places worth the effort.

1. Custom logits processors — the safest place to touch

The reason to recommend this is that it attaches as a plugin without modifying the source. You don't need to fork vLLM. Logits processors operate at the batch level: they take a logits tensor shaped [number of requests x vocab size], transform it, and hand it off to softmax.

Subclass vllm.v1.sample.logits_processor.LogitsProcessor and implement five methods.

# my_pkg/procs.py
import torch
from vllm.config import VllmConfig
from vllm.sampling_params import SamplingParams
from vllm.v1.sample.logits_processor import BatchUpdate, LogitsProcessor


class BanTokenAfterN(LogitsProcessor):
    """Bans a specific token once output has passed N tokens (example)."""

    @classmethod
    def validate_params(cls, params: SamplingParams):
        # Filter out bad arguments at the entrypoint ahead of time. If you
        # don't implement this, a bad value just flows straight into the kernel.
        v = params.extra_args and params.extra_args.get("ban_after")
        if v is not None and not isinstance(v, int):
            raise ValueError("ban_after must be int")

    def __init__(self, vllm_config: VllmConfig, device: torch.device,
                 is_pin_memory: bool):
        self.device = device
        # batch index -> (banned token, threshold, reference to the output token list)
        self.req: dict[int, tuple[int, int, list[int]]] = {}

    def is_argmax_invariant(self) -> bool:
        # Can change the argmax token, so False.
        # If set to True, vLLM skips this processor entirely whenever the
        # whole batch is doing greedy sampling.
        return False

    def update_state(self, batch_update: BatchUpdate | None) -> None:
        if batch_update is None:
            return
        # Must be processed in the order removed -> added -> moved.
        for idx in batch_update.removed:
            self.req.pop(idx, None)
        for idx, params, _prompt_ids, output_ids in batch_update.added:
            self.validate_params(params)
            n = params.extra_args and params.extra_args.get("ban_after")
            if n is None:
                self.req.pop(idx, None)
            else:
                # output_ids is a live list reference, so it always
                # reflects the latest output at every step.
                self.req[idx] = (params.extra_args["ban_token"], n, output_ids)
        for a, b, direction in batch_update.moved:
            va, vb = self.req.pop(a, None), self.req.pop(b, None)
            if vb is not None:
                self.req[a] = vb
            if va is not None and direction.name == "SWAP":
                self.req[b] = va

    def apply(self, logits: torch.Tensor) -> torch.Tensor:
        for idx, (tok, n, out_ids) in self.req.items():
            if len(out_ids) >= n:
                logits[idx, tok] = float("-inf")   # in-place is memory-friendlier
        return logits

There are two ways to attach it: pass the fully qualified class name, or register it as a package entrypoint.

vllm serve facebook/opt-125m --logits_processors my_pkg.procs:BanTokenAfterN

# pyproject.toml — loads automatically once installed
[project.entry-points."vllm.logits_processors"]
ban_after = "my_pkg.procs:BanTokenAfterN"

Three things to watch out for.

• The set of logits processors is fixed at engine initialization. You can't add one per request afterward. The only way to toggle one on or off per request is to branch on SamplingParams.extra_args inside apply.
• Answer is_argmax_invariant() honestly. Setting it to True lets the batch skip your processor entirely, for free, whenever the whole batch is greedy — but returning True for a processor that actually changes the argmax silently produces wrong results.
• apply runs on every step, over the whole batch. A Python loop over the number of requests becomes the bottleneck by itself. Vectorize with tensor ops wherever you can.
• The docs themselves state explicitly that for this API "design changes are still in progress and the API may change in the near future." It's safer to use this while pinning your version.

2. Custom attention backends — high value, high cost

The backends live under vllm/v1/attention/backends/, and the common interface is in vllm/v1/attention/backend.py. v0.26.0 includes flash_attn.py, flashinfer.py, triton_attn, flex_attention.py, and dedicated MLA implementations.

The situations that justify writing your own are narrow: you need a variant of standard attention that isn't in any existing backend, and that variant is decisive for performance. For example, a domain-specific sparse mask where only 10 percent of full attention actually needs computing.

Be honest about the cost. A backend has to get right: the metadata builder, CUDA graph compatibility, the prefill and decode paths, interaction with chunked prefill, and interaction with prefix caching. Writing the kernel alone doesn't finish the job. And this interface changes with every vLLM release.

3. Scheduler policy — the last resort

Under vllm/v1/core/sched/ are scheduler.py, interface.py, and request_queue.py. You'll end up here when you want to replace request priority with domain rules (e.g. paid tier first, short requests first).

But there's an order to follow. vLLM already ships a request priority feature, so first check whether it can express what you need. Modifying the scheduler is the highest-risk option. As we saw above, this code is entangled with the token budget, preemption, chunked prefill, prefix caching, and speculative decoding all at once.

The cost of tracking upstream

A cost common to all three places: a forked vLLM ages automatically.

vLLM ships a minor release roughly every two weeks. New model support, new kernels, new quantization formats, and performance improvements land in between. Leave a fork unattended for six months and you can't run the newest models, can't use the newest attention kernels, and are stuck carrying bugs that were already fixed upstream. By that point, the rebase cost has become several times the cost of the original modification.

So the cost ordering looks like this.

I want to underline the third row. If what you need is also useful to other people, sending it upstream is the cheapest maintenance strategy there is. The moment it merges, your maintenance cost drops to zero.

Closing — Tuning Without Measurement Is Just a Statement of Taste

The order this post covered is itself the conclusion. Eliminate preemption logs, weigh TTFT against ITL with the token budget, don't starve the CPU, and measure whether prefix caching is actually winning. Only after that do you need to understand how PagedAttention and the scheduler behave, and only once you understand that do you know where to touch the code.

And at no stage in this sequence should you skip measuring before and after. If raising max_num_batched_tokens from 8192 to 16384 grows throughput by 12 percent and doubles p99 TTFT, only your service's SLO can answer whether that's an improvement or a regression. Making that call without numbers isn't tuning, it's a statement of taste.

One last line. The most-used optimization technique in vLLM is still "make the batch bigger," and most teams try to open the source before they've used up that room. Check the six lines first.

References

• vLLM Optimization and Tuning Guide: https://docs.vllm.ai/en/latest/configuration/optimization.html
• vLLM V1 User Guide (includes the V0 deprecation notice): https://docs.vllm.ai/en/latest/usage/v1_guide.html
• vLLM Custom Logits Processors documentation: https://docs.vllm.ai/en/latest/features/custom_logitsprocs.html
• vLLM Attention Backends documentation: https://docs.vllm.ai/en/latest/design/attention_backends.html
• vLLM Profiling documentation: https://docs.vllm.ai/en/latest/contributing/profiling.html
• vLLM Benchmark CLI: https://docs.vllm.ai/en/latest/cli/bench/serve.html
• Kwon et al., Efficient Memory Management for LLM Serving with PagedAttention (SOSP 2023): https://arxiv.org/abs/2309.06180
• Agrawal et al., Sarathi (the paper behind chunked prefill): https://arxiv.org/pdf/2308.16369
• vLLM source (scheduler): https://github.com/vllm-project/vllm/blob/main/vllm/v1/core/sched/scheduler.py