Skip to content
Published on

Inside vLLM (5) — Prefix Caching, and Why System Prompt Design Is Performance

🇰🇷KO🇺🇸EN🇯🇵JA🇨🇳ZH
Split View필사 모드
Share
Authors

Not Computing the Same Prefix Twice

Part 2 explained that block-level management is what makes sharing possible. This part covers the feature that extends that sharing across requests: prefix caching.

Requests in a real service overlap more than you would expect. In a chatbot, every request starts with the same system prompt. In a tool-using agent, the same bundle of tool definitions is prepended every time. In a multi-turn conversation, the second request contains the entire first request as a prefix. Recomputing the KV for this overlapping prefix every single time is pure waste.

Prefix caching eliminates this waste. The effect shows up first in time to first token rather than in throughput, because the user waits less by exactly as much prefill gets skipped. In the CacheConfig source on the main branch, the declared default for enable_prefix_caching is true. In other words, in recent versions the default is to work without turning it on separately.

Everything here was verified against the official documentation and source on 2026-08-12. vLLM moves fast, so re-check settings and behavior against the documentation for the version you are running.

How Block Hashes Are Built

The core question is how a block gets identified. The vLLM design document is explicit about this. When hashing each KV cache block, it does not use only the tokens inside that block — it also folds in the tokens of the prefix that came before it. In the implementation, the hash of the previous block, the tokens of the current block, and a set of extra factors go in together. Those extra factors include the LoRA identifier, the hash of any multimodal input, and the cache salt.

That chain structure is the whole mechanism. Every practical conclusion you need follows from it.

When block size is 4

Prompt:    [You are a helpful]  [assistant. Today]  [the date is Aug]  [12th. The question]
              block0                 block1               block2              block3

hash0 = H(none,        block0 tokens, extras)
hash1 = H(hash0,       block1 tokens, extras)
hash2 = H(hash1,       block2 tokens, extras)
hash3 = H(hash2,       block3 tokens, extras)

→ If block2's tokens change even a day later, hash2 changes,
  and hash3 changes with it, since it uses hash2 as an input.
  block0 and block1 survive untouched.

Once one block diverges, everything after it diverges too. Put the other way, everything up to the point of divergence stays alive. So the performance of prefix caching is decided not by how much of the content is the same, but by how many tokens from the very front are identical.

Why Only Full Blocks Get Cached

The design document nails down one more rule. Only full blocks are cached. A partially filled block cannot be pulled from the cache until it is completely filled.

The example in the document shows this rule clearly. With a block size of 4, one request hits the cache only up through the first two blocks — that is, 8 tokens. The third block matches on only 2 of its 4 tokens, so it does not count as a hit.

This is where the trade-off previewed in part 2 becomes concrete. Larger blocks make this rounding loss bigger. If the block size is 32 and the shared prefix is 40 tokens, the hit only covers 32 tokens and the remaining 8 get recomputed. This loss is negligible for workloads with very long shared prefixes, but it is noticeable in services with a lot of short prompts.

Why Prompt Design Is Performance

This all comes down to one practical rule. Put what never changes at the front, and what changes every time at the back.

Why this matters so much is obvious once you look at the hash chain. If you put the current time on the very first line of the prompt, everything after it is invalidated — no matter how good or how long the shared instructions that follow are. The hash of the first block changes on every request, and every hash after it uses that hash as an input.

Bad layout — 0 cache hits
[Current time 09:31:07] [User ID 8823] [Long system instructions …] [Tool definitions …] [Question]
    different every time     different every time         always the same          always the same

Good layout — the first two chunks hit the cache whole
[Long system instructions …] [Tool definitions …] [User ID 8823] [Current time 09:31:07] [Question]
         always the same            always the same      different every time      different every time

The content is identical — only the order changed — and the outcome is completely different. Saying that prompt design is performance is not a figure of speech. It is a direct description of this mechanism.

One more thing. The shared prefix has to match down to the last character. What the tokenizer sees is a sequence of tokens, so even a single space or a single line break can produce a different token. If you are building the system prompt in code by concatenating strings, it is worth checking whether the concatenated result really is byte-identical every time.

Common Reasons the Cache Does Not Hit

When the hit rate is lower than expected, here is a list to work through in order.

First, a variable element near the front. A timestamp, a session identifier, a user name, a randomly shuffled order of examples — any of these near the front kills the cache.

Second, the shared prefix does not fill even a single block. If the system prompt is short, there is no full block to cache in the first place. In this case caching is not broken — there is simply nothing to cache.

Third, eviction. Cached blocks do not stay around forever. When a new request needs a block and there is no room, the oldest blocks are reclaimed first. In a deployment where concurrent requests are so high that the KV cache is always tight, there is no room to keep anything cached in the first place. Here the fix is not the prompt — it is the capacity question covered in part 4.

Fourth, whatever splits requests into different groups. As the design document states, the LoRA identifier, multimodal input, and cache salt are also inputs to the hash. Different adapters mean the same text does not use the same cache.

Security: Why cache_salt Exists

Prefix caching has a shadow side. A cache hit makes the response faster, and that speedup is observable. In an environment where multiple users share one engine, an attacker who varies the prompt and measures response time can narrow down which prefixes are already in the cache.

This is why the OpenAI-compatible server has a cache_salt parameter. The source is explicit about it. When specified, it mixes the given string into the prefix cache so that, in a multi-user environment, an attacker cannot guess prompts. The salt should be random, never exposed externally, and long enough that it cannot be guessed. The source gives 43 base64 characters, equivalent to 256 bits, as an example.

Applying this in practice is simple. Giving each tenant a different salt keeps the cache from crossing tenant boundaries. The trade-off is that you give up any sharing benefit between tenants, so a reasonable balance is per-tenant within the same organization, or per-customer when customers are fully separate. As a side note, the hash algorithm itself is also a configuration item. The declared default for prefix_caching_hash_algo in CacheConfig on the main branch was sha256.

Try It Yourself

References

Discuss on Twitter • View on GitHub