필사 모드: LLM Inference VRAM Math — the KV Cache Blows Up Before the Weights Do

Introduction — "Will this model fit on our GPU"

Answering that question with "we will have to try it and see" leaves you with nothing to say in an infrastructure budget meeting. Fortunately, inference memory — unlike training memory — is almost entirely predictable with arithmetic. There are no optimizer states and no gradients.

The trouble is that most of these calculations stop halfway. "A 7B model is 14GB in fp16, so it fits comfortably on a 24GB card" is correct as far as it goes. Then you raise the batch size to 32 and the context to four thousand tokens, and the process dies out of memory. The missing term is the KV cache, and unlike the weights, that term grows with the shape of your user traffic.

This post splits inference VRAM into four chunks (weights, KV cache, activations, framework overhead) and shows how to compute each one. Every number is presented as a formula you can check yourself, so read it with your own model config file open.

Weight memory — the part that ends with one multiplication

Weights are honest. Multiply the parameter count by bytes per parameter and you are done.

BYTES_PER_PARAM = {
    "fp32": 4.0,
    "fp16": 2.0,   # same for bf16
    "int8": 1.0,
    "fp8": 1.0,
    "int4": 0.5,   # 4-bit; in practice scales/zero-points add a little more
}

GIB = 1024 ** 3

def weight_gib(params_billions: float, dtype: str = "fp16") -> float:
    return params_billions * 1e9 * BYTES_PER_PARAM[dtype] / GIB

for n in (7, 13, 70):
    print(n, [round(weight_gib(n, d), 1) for d in ("fp16", "int8", "int4")])
# 7  [13.0, 6.5, 3.3]
# 13 [24.2, 12.1, 6.1]
# 70 [130.4, 65.2, 32.6]

A word about units before we go further. Seven billion times 2 bytes is 14 billion bytes; divide by 10 to the ninth and you get 14GB, divide by 2 to the thirtieth and you get 13.0GiB. The "80GB" on a card spec sheet is usually decimal, while the number nvidia-smi shows you is binary, and that seven percent gap is why a calculation that looked tight sometimes fails to add up. Everything below is in GiB.

The 4-bit row is an idealized number. A real 4-bit checkpoint carries an fp16 scale and zero-point per group, so with a group size of 128 you are paying roughly an extra 0.5 bit per parameter. Budgeting about 0.53 bytes per parameter rather than 0.5 is the safer move. Many implementations also leave the embedding and output layers unquantized, so the gap widens as the vocabulary grows.

KV cache — where it actually blows up

Autoregressive decoding computes attention over every previous token each time it produces one more. Throwing away the keys and values you already computed would mean recomputing the whole sequence at every step, so you keep all of them. That is the KV cache.

Here is what a single token costs.

def kv_bytes_per_token(layers: int, kv_heads: int, head_dim: int,
                       bytes_per_elem: float = 2.0) -> float:
    # the 2 is because K and V are both stored
    return 2 * layers * kv_heads * head_dim * bytes_per_elem

# 7B-class MHA config: 32 layers, 32 KV heads, head dim 128
print(kv_bytes_per_token(32, 32, 128) / 1024, "KiB")   # 512.0 KiB
# 70B-class GQA config: 80 layers, 8 KV heads, head dim 128
print(kv_bytes_per_token(80, 8, 128) / 1024, "KiB")    # 320.0 KiB

512KiB per token. If that number does not land, look at it this way. On a 7B-class MHA model, one request with a 4096-token context spends 2GiB on KV cache. Thirty-two concurrent requests is 64GiB. Five times the 13GiB of weights.

Read this table across, not down. The weights column is a constant and the last column is proportional to traffic. Stretch the context from 4K to 32K and only the last column changes, by a factor of eight. That is why services that advertise long context quietly run with far lower concurrency.

One caution. The KV cache is proportional to the total number of live tokens, not to the number of active sequences. A hundred requests of 500 tokens each cost the same memory as five requests of 10,000 tokens each. Capacity planning has to be done in concurrent tokens, not requests per second.

GQA and MQA — a structural change that only shrinks KV heads

The 7B MHA row and the 7B GQA row differ by exactly four times because the KV heads dropped from 32 to 8. There are still 32 query heads; they are grouped four at a time and share one KV head.

# query heads stay the same, only KV heads shrink
n_heads, head_dim, layers = 32, 128, 32

for kv_heads, name in [(32, "MHA"), (8, "GQA-8"), (1, "MQA")]:
    per_token = kv_bytes_per_token(layers, kv_heads, head_dim)
    ratio = n_heads // kv_heads
    print(f"{name:6} kv_heads={kv_heads:2}  {per_token/1024:6.1f} KiB/token  ({ratio}배 절감)")
# MHA    kv_heads=32   512.0 KiB/token  (1배 절감)
# GQA-8  kv_heads= 8   128.0 KiB/token  (4배 절감)
# MQA    kv_heads= 1    16.0 KiB/token  (32배 절감)

The saving ratio equals the query head count divided by the KV head count, exactly. This is not an approximation; it follows from the definition.

On quality, GQA is generally regarded as a cheap trade. The original paper reported quality close to MHA with only a handful of KV heads, and many open models since have settled on around eight. MQA, with a single KV head, has drawn more reports of measurable degradation. That said, this is a pretraining-time architecture choice, not a knob you can turn on an already-published model. What you get to choose is "pick a model that uses GQA."

If num_key_value_heads is smaller than num_attention_heads in the model config.json, it is GQA. Confuse those two while calculating and your answer is off by four times.

Activations, overhead, and the full calculator

The two remaining chunks vary enough with circumstance that an exact formula is hard to give. Knowing what they scale with is enough.

Activation memory is barely an issue during decoding, since only one token per batch element flows through per step. Prefill is the problem. The whole prompt passes through at once, so intermediate tensors scale with batch times prompt length.

def prefill_activation_gib(batch, prompt_len, hidden, intermediate,
                           live_buffers=4, bytes_per_elem=2.0):
    """A crude lower bound on prefill peak activations. live_buffers varies by framework, so measure it."""
    per_token = (hidden + intermediate) * bytes_per_elem
    return batch * prompt_len * per_token * live_buffers / GIB

# 7B class: hidden 4096, intermediate 11008
print(round(prefill_activation_gib(8, 4096, 4096, 11008), 2), "GiB")  # 3.69 GiB

live_buffers=4 is a constant I made up. The number of buffers actually alive at once depends on how aggressively kernels are fused and how the memory planner is implemented, so this value has to be corrected by measurement. The only thing that is certain is that it scales with batch times prompt length — which is why chunked prefill (feeding a long prompt in several pieces) flattens this term to a constant.

Framework overhead is the CUDA context, cuBLAS workspaces, communication buffers and allocator slack added together. Budgeting around 1GiB per process is usually right. On top of that, multiplying the total by 10 to 15 percent for allocator fragmentation is the practical move.

Now put it all together into a calculator.

def total_vram_gib(params_b, weight_dtype, layers, kv_heads, head_dim,
                   seq_len, batch, kv_bytes=2.0, overhead=1.15, fixed_gib=1.0):
    w = weight_gib(params_b, weight_dtype)
    kv = kv_bytes_per_token(layers, kv_heads, head_dim, kv_bytes) * seq_len * batch / GIB
    return (w + kv) * overhead + fixed_gib

# 70B, 4-bit weights, GQA-8, 8K context, 16 concurrent
print(round(total_vram_gib(70, "int4", 80, 8, 128, 8192, 16), 1), "GiB")
# 84.6 GiB  → does not fit on a single 80GiB card

The weights are 32.6GiB and the KV cache is 40GiB. Squeeze the weights down to 4-bit and the KV cache is still larger. Miss that point and you get stuck in the "we quantized it, why does it still not fit" state.

The calculation people actually use in practice runs the other way. Given a fixed card, how many requests can you take at once?

def max_concurrent(vram_gib, params_b, weight_dtype, layers, kv_heads, head_dim,
                   seq_len, util=0.9, kv_bytes=2.0, fixed_gib=1.0):
    usable = vram_gib * util - fixed_gib - weight_gib(params_b, weight_dtype)
    if usable <= 0:
        return 0
    per_seq_gib = kv_bytes_per_token(layers, kv_heads, head_dim, kv_bytes) * seq_len / GIB
    return int(usable / per_seq_gib)

print(max_concurrent(80, 70, "int4", 80, 8, 128, seq_len=8192))   # 15
print(max_concurrent(80, 70, "int4", 80, 8, 128, seq_len=2048))   # 61
print(max_concurrent(80, 70, "int4", 80, 8, 128, seq_len=8192, kv_bytes=1.0))  # 30

Each of those three lines is an operational decision. Lowering the context ceiling from 8K to 2K quadruples throughput; storing the KV cache in fp8 doubles it. Both decisions affect quality, but at least you can see what you gain and what you are wagering.

PagedAttention — memory you cannot use even when the math is right

Everything so far assumed "we use exactly as much as we need." Early serving implementations did not. When a request arrived, they reserved a contiguous KV cache region large enough for the longest output that request could reach. If the ceiling was 4096 and the actual output was 200 tokens, the rest was simply thrown away.

The vLLM paper splits this waste into two kinds: internal fragmentation, reserved and never used, and external fragmentation, stranded between blocks where nobody can reach it. On the workloads the paper measured, the share of KV memory that was effectively used came out quite low, with the rest lost to those two. The specific figures are workload-dependent, so rather than quoting them, remember the structure: the wider the spread of output lengths, the bigger the waste.

PagedAttention solves this the way an operating system would. It splits the KV cache into fixed-size blocks (typically 16 tokens) and maps a logically contiguous sequence onto physically scattered blocks. Waste is then bounded by the empty tail of the last block per sequence — at most 15 tokens.

There is a side effect, and it is in fact the bigger one. Managing memory in blocks lets several sequences share the same block. A hundred requests using the same system prompt keep a single copy of the blocks for that prefix and share it by reference count. In services with long prefixes and high request volume, that prefix sharing saves more than the defragmentation does.

# the knobs you actually end up tuning in vLLM
vllm serve <모델경로> \
  --max-model-len 8192 \            # context ceiling; sets the KV cache ceiling directly
  --gpu-memory-utilization 0.90 \   # share of total VRAM to use; the rest is for overhead
  --kv-cache-dtype fp8 \            # KV cache only, in 8-bit; halves the capacity cost
  --max-num-seqs 64 \               # concurrent sequence ceiling
  --enable-prefix-caching           # reuse shared prefix blocks

# check these lines in the startup log. They should agree with the math above.
#   "GPU KV cache size: 129,024 tokens"
#   "Maximum concurrency for 8192 tokens per request: 15.75x"

If the KV cache token count in the startup log differs sharply from your max_concurrent calculation, one of the two is wrong. Usually it is the KV head count being mistaken for the query head count.

Quantization is not free

Drop the weights to 4-bit and memory falls to a quarter. That is true. The problem is that the sentence usually stops there.

First, quality loss hides from average metrics. A change summarized as "wikitext perplexity rose by 0.1" often shows up far larger in the tails — the back half of long outputs, code generation, multilingual work, format compliance. Perplexity is average token prediction difficulty, not task success rate. When deciding whether to quantize, you have to measure on an evaluation set for your own task. How to build that set is covered separately in LLM evaluation without the vibes.

Second, 4-bit is not always faster. Weight-only quantization stores in 4-bit but computes in fp16, so every matrix multiply carries a dequantization step. When the batch is small and memory bandwidth is the bottleneck, reading a quarter as many bytes is a clear win. When the batch is large and compute becomes the bottleneck, all that is left is the dequantization overhead, and it can end up slower than fp16. "Quantization makes it faster" is a batch-size-1 benchmark statement.

Third, KV cache quantization is a separate decision from weight quantization, and it is generally safer. There are many reports that fp8 KV cache costs little, while 4-bit KV has been reported to degrade at long context. As the math above showed, memory in a long-context service is dominated by the KV side, so flipping the order — take the KV cache down to fp8 first and leave the weights at fp16 or 8-bit — is frequently the better combination.

So the order of choices is this. Look at the KV cache dtype first, the context ceiling second, and weight quantization last. Most teams approach it in exactly the opposite order and lose time.

Closing — weights are a constant, the KV cache is a variable

There is one line to remember. Weight memory is a constant fixed the moment you pick a model, and the KV cache is a variable determined by the shape of the traffic you agreed to take. Capacity shortfalls almost always come from the variable.

So two numbers are enough to compute before deployment: KV bytes per token (2 times layers times KV heads times head dim times bytes per element) and the total token count you get by dividing your remaining VRAM by that value. With those two, you can work out the trade between context ceiling and concurrency ceiling live in the meeting room. Only after that is it time to talk about quantization.