- Introduction
- Why the KV Cache Is the Problem
- From MHA to MQA, Then GQA
- MLA: Changing What Gets Cached
- One Table
- What You Give Up
- Choosing to Cut Head Count Itself
- Closing
- References
- Try It Yourself
- Series
Introduction
Anyone who has served a long-context model knows how fast the KV cache eats memory. Weights are fixed, but the KV cache grows with the number of users and the length of the context. The attention variants of the last few years are, in effect, a history of solving this one problem.
This post follows the line from MHA through MQA and GQA to MLA using real config values. Every calculation comes from the public settings files verified in the previous posts.
All figures were verified directly against papers, official reports, and config.json files on 2026-08-12. Models get updated, so check the originals again.
Why the KV Cache Is the Problem
In autoregressive generation, producing each token requires the keys and values of every preceding token. Rather than recomputing them each time, you store them, and that store is the KV cache. Its size is set by this formula.
KV cache bytes
= 2 x layers x key/value heads x head_dim x context length x bytes per element
The leading 2 stands for the two sets, keys and values.
Of these terms, the model designer can realistically move only one: the number of key/value heads. Layer count and head dimension bear directly on quality, and context length is chosen by the user.
From MHA to MQA, Then GQA
The original transformer gave every query head its own key/value head. That is MHA. MQA goes to the opposite extreme and has all query heads share a single key/value head. The cache shrinks by the head count, but the loss in representational capacity is large.
GQA sits in between. The GQA paper defines the technique as a generalization of multi-query attention using an intermediate number of key/value heads, more than one and fewer than the query heads, and reports that converting an existing multi-head checkpoint takes 5 percent of the original pre-training compute (Ainslie et al., arXiv:2305.13245).
Most dense models shipping today use GQA. Here are the ratios confirmed in the actual configs.
model query heads / kv heads ratio
Qwen3-8B 32 / 8 4:1
Qwen3-32B 64 / 8 8:1
Qwen3-235B-A22B 64 / 4 16:1
Qwen2.5-7B 28 / 4 7:1
Mixtral-8x7B 32 / 8 4:1
GLM-4.5 96 / 8 12:1
Llama 3.1 (8B/70B/405B) 32,64,128 / 8 4:1, 8:1, 16:1
The Llama 3 figures come from Table 3 of the technical report (arXiv:2407.21783). Notice that the ratio grows more aggressive as models get larger, because a bigger model has more query heads and therefore more cache pressure.
MLA: Changing What Gets Cached
MLA, introduced by DeepSeek-V2 and carried forward by DeepSeek-V3 and Kimi K2, takes a different approach. Instead of cutting head counts, it jointly compresses keys and values into a low-dimensional latent vector and caches only that vector.
The hyper-parameters in the DeepSeek-V3 report give 128 attention heads, a per-head dimension of 128, a KV compression dimension of 512, a query compression dimension of 1536, and a per-head dimension of 64 for the decoupled queries and key (arXiv:2412.19437). What remains in the cache is only the compressed latent vector and the decoupled key.
MLA cache elements/token = layers x (kv_lora_rank + qk_rope_head_dim)
DeepSeek-V3 = 61 x (512 + 64) = 35,136
Had the same model been built as MHA
= 2 x 61 x 128 x 128 = 1,998,848
Ratio = 1,998,848 / 35,136 = about 56.9x
The DeepSeek-V2 paper reports that this design reduced the KV cache by 93.3 percent and raised maximum generation throughput to 5.76 times, compared with DeepSeek 67B (arXiv:2405.04434).
One Table
Sizes for caching a single 32,768-token context at fp16. All computed by substituting config values into the formula above.
model scheme elem/token vs MHA 32K cache (fp16)
Qwen3-8B GQA 73,728 4.0x 4.50 GiB
Qwen3-32B GQA 131,072 8.0x 8.00 GiB
Qwen3-235B-A22B GQA 96,256 16.0x 5.88 GiB
Mixtral-8x7B GQA 65,536 4.0x 4.00 GiB
GLM-4.5 GQA 188,416 12.0x 11.50 GiB
DeepSeek-V3 MLA 35,136 56.9x 2.14 GiB
Kimi K2 MLA 35,136 28.4x 2.14 GiB
Two things stand out. First, Qwen3-235B-A22B is a 235B-parameter model yet its KV cache is only about 30 percent larger than that of the 8B Qwen3-8B, because its key/value heads are grouped down to 4. Second, DeepSeek-V3 and Kimi K2 have identical per-token cache, since their layer counts and compression dimensions match. Their reduction multiples differ only because Kimi K2 has 64 attention heads, which halves the MHA baseline it is measured against.
What You Give Up
This is the part that matters. Shrinking the cache always costs something.
GQA gives up representational capacity. Several query heads look at the same keys and values, so the ability of individual heads to attend to different information is reduced. That is why the GQA paper frames it as a tradeoff between quality and speed.
MLA gives up compute and complexity. Extra matrices are needed for compression and reconstruction, and part of the query and key must be decoupled and handled separately to carry positional information. The DeepSeek-V3 report noting that additional RMSNorm layers are placed after the compressed latent vectors, and that scaling factors are multiplied in at the width bottlenecks, shows that this structure does not come for free.
Choosing to Cut Head Count Itself
Kimi K2 moved a different lever. According to Table 2 of its report, it halved the 128 attention heads of DeepSeek-V3 down to 64. The reason is stated concretely in the same section: at a sequence length of 128k, with the total expert count fixed at 384, raising attention heads from 64 to 128 leads to an 83 percent increase in inference FLOPs.
The gain on the other side was measured too. The same report finds that doubling the number of heads lowered validation loss by only about 0.5 to 1.2 percent, and concludes that such marginal gains do not justify the inference cost (arXiv:2507.20534). It is a good example of the evidence a design decision can rest on.
Closing
Reading attention variants is straightforward. Look at the ratio of num_attention_heads to num_key_value_heads in the config, and if kv_lora_rank is present, you are looking at MLA. Then put the numbers into the formula and compute per-token cache yourself. How a model intends to handle long context is captured in that single number.
References
- GQA (Ainslie et al., arXiv:2305.13245): https://arxiv.org/abs/2305.13245
- DeepSeek-V2 (arXiv:2405.04434): https://arxiv.org/abs/2405.04434
- DeepSeek-V3 Technical Report (arXiv:2412.19437): https://arxiv.org/abs/2412.19437
- Kimi K2 (arXiv:2507.20534): https://arxiv.org/abs/2507.20534
- The Llama 3 Herd of Models (arXiv:2407.21783): https://arxiv.org/abs/2407.21783
- GLM-4.5 config.json: https://huggingface.co/zai-org/GLM-4.5/raw/main/config.json
Try It Yourself
- VRAM calculator — vary context length and watch the KV cache grow.
- Neural network architecture explorer — inspect the relationship between head count and layer count.
- Neural net lab — run the basic mechanics of attention yourself.
Series
현재 단락 (1/55)
Anyone who has served a long-context model knows how fast the KV cache eats memory. Weights are fixe...