Introduction

The largest bottleneck in the inference cost of Large Language Models (LLMs) is the memory consumption of the **KV Cache (Key-Value Cache)**. When a GPT-4 class model processes hundreds of concurrent requests with a 128K context length, the KV Cache alone can consume hundreds of GB of GPU memory. This memory constraint directly limits throughput and response latency.

In Transformer Self-Attention, recomputing Key and Value vectors for all previous tokens at each decoding step is inefficient, so storing and reusing them in a cache is the fundamental principle of KV Cache. The problem is that this cache grows linearly with sequence length.

Various attention mechanisms have been proposed to address this problem. **Multi-Query Attention (MQA)** dramatically reduces memory by having all Query heads share a single KV head, but at the cost of quality degradation. **Grouped Query Attention (GQA)** serves as a compromise between MHA and MQA and was adopted in Llama 2/3. **Multi-Head Latent Attention (MLA)** achieves remarkable efficiency in DeepSeek-V2/V3 by compressing KV into a low-dimensional latent space.

This article comprehensively covers the mathematical principles and memory analysis of each attention mechanism, KV Cache compression techniques, PagedAttention (vLLM), PyTorch implementation examples, real-world OOM failure cases and recovery, and an optimization checklist.

Transformer Self-Attention and KV Cache Fundamentals

Self-Attention Operation

Transformer's Scaled Dot-Product Attention is defined as follows:

$$

\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V

$$

Here, Q (Query), K (Key), and V (Value) are obtained by linearly transforming the input token embeddings.

class ScaledDotProductAttention(nn.Module):

def __init__(self, d_model, n_heads):

super().__init__()

self.d_k = d_model // n_heads

self.n_heads = n_heads

self.W_q = nn.Linear(d_model, d_model)

self.W_k = nn.Linear(d_model, d_model)

self.W_v = nn.Linear(d_model, d_model)

self.W_o = nn.Linear(d_model, d_model)

def forward(self, x, kv_cache=None):

B, T, C = x.shape

Q, K, V projections

q = self.W_q(x).view(B, T, self.n_heads, self.d_k).transpose(1, 2)

k = self.W_k(x).view(B, T, self.n_heads, self.d_k).transpose(1, 2)

v = self.W_v(x).view(B, T, self.n_heads, self.d_k).transpose(1, 2)

Use KV Cache

if kv_cache is not None:

k_cache, v_cache = kv_cache

k = torch.cat([k_cache, k], dim=2)

v = torch.cat([v_cache, v], dim=2)

Attention computation

scores = torch.matmul(q, k.transpose(-2, -1)) / math.sqrt(self.d_k)

attn = torch.softmax(scores, dim=-1)

output = torch.matmul(attn, v)

output = output.transpose(1, 2).contiguous().view(B, T, C)

return self.W_o(output), (k, v)

KV Cache Memory Analysis

The memory usage of KV Cache is calculated using the following formula:

**KV Cache Memory** = 2 (K and V) x Number of layers (L) x Number of heads (H) x Sequence length (S) x Head dimension (d_k) x Byte size

For example, with the Llama 2 70B model:

- Number of layers: 80

- Number of heads: 64 (before GQA)

- Head dimension: 128

- Sequence length: 4096

- FP16 (2 bytes)

KV Cache = 2 x 80 x 64 x 4096 x 128 x 2 = **10.7 GB** (single request)

With batch size 32, this becomes **342.4 GB**, far exceeding model weight memory.

Multi-Head Attention (MHA) Memory Analysis

MHA Structure

In standard Multi-Head Attention, each attention head has independent Q, K, V projections. With H heads each maintaining K, V of d_k dimensions, the KV Cache reaches its maximum size.

class MultiHeadAttention(nn.Module):

"""Standard Multi-Head Attention (MHA)

Each head maintains an independent KV pair"""

def __init__(self, d_model, n_heads):

super().__init__()

self.n_heads = n_heads

self.d_k = d_model // n_heads

self.W_q = nn.Linear(d_model, d_model)

self.W_k = nn.Linear(d_model, d_model) # H * d_k dimensions

self.W_v = nn.Linear(d_model, d_model) # H * d_k dimensions

self.W_o = nn.Linear(d_model, d_model)

def forward(self, x, kv_cache=None):

B, T, _ = x.shape

q = self.W_q(x).view(B, T, self.n_heads, self.d_k).transpose(1, 2)

k = self.W_k(x).view(B, T, self.n_heads, self.d_k).transpose(1, 2)

v = self.W_v(x).view(B, T, self.n_heads, self.d_k).transpose(1, 2)

if kv_cache is not None:

k = torch.cat([kv_cache[0], k], dim=2)

v = torch.cat([kv_cache[1], v], dim=2)

KV Cache size: [B, n_heads, S, d_k] x 2

Memory: 2 * B * n_heads * S * d_k * sizeof(dtype)

attn = torch.matmul(q, k.transpose(-2, -1)) / math.sqrt(self.d_k)

attn = torch.softmax(attn, dim=-1)

out = torch.matmul(attn, v)

out = out.transpose(1, 2).contiguous().view(B, T, -1)

return self.W_o(out), (k, v)

MHA KV Cache per token per layer = 2 x H x d_k x sizeof(dtype)

Multi-Query Attention (MQA)

MQA Structure and Savings

MQA, proposed by Shazeer (2019), has all Query heads **share a single KV head**. This reduces KV Cache by a factor of H but introduces quality degradation.

class MultiQueryAttention(nn.Module):

"""Multi-Query Attention (MQA)

All Query heads share a single KV head"""

def __init__(self, d_model, n_heads):

super().__init__()

self.n_heads = n_heads

self.d_k = d_model // n_heads

self.W_q = nn.Linear(d_model, d_model) # H * d_k

self.W_k = nn.Linear(d_model, self.d_k) # 1 * d_k (single head)

self.W_v = nn.Linear(d_model, self.d_k) # 1 * d_k (single head)

self.W_o = nn.Linear(d_model, d_model)

def forward(self, x, kv_cache=None):

B, T, _ = x.shape

q = self.W_q(x).view(B, T, self.n_heads, self.d_k).transpose(1, 2)

k = self.W_k(x).view(B, T, 1, self.d_k).transpose(1, 2)

v = self.W_v(x).view(B, T, 1, self.d_k).transpose(1, 2)

if kv_cache is not None:

k = torch.cat([kv_cache[0], k], dim=2)

v = torch.cat([kv_cache[1], v], dim=2)

Broadcast K, V to all heads

k = k.expand(-1, self.n_heads, -1, -1)

v = v.expand(-1, self.n_heads, -1, -1)

KV Cache size: [B, 1, S, d_k] x 2

Reduced to 1/H compared to MHA

attn = torch.matmul(q, k.transpose(-2, -1)) / math.sqrt(self.d_k)

attn = torch.softmax(attn, dim=-1)

out = torch.matmul(attn, v)

out = out.transpose(1, 2).contiguous().view(B, T, -1)

return self.W_o(out), (k[:, :1], v[:, :1])

MQA KV Cache per token per layer = 2 x 1 x d_k x sizeof(dtype), **1/H reduction** compared to MHA.

Grouped Query Attention (GQA)

GQA Architecture (Llama 2/3)

GQA, proposed by Ainslie et al. (2023) and adopted in Llama 2 and Llama 3, divides Query heads into G groups where each group shares one KV head. When G=1, it is equivalent to MQA; when G=H, it is equivalent to MHA.

class GroupedQueryAttention(nn.Module):

"""Grouped Query Attention (GQA)

Divides Query heads into G groups, each sharing KV

Adopted in Llama 2/3"""

def __init__(self, d_model, n_heads, n_kv_heads):

super().__init__()

self.n_heads = n_heads # Number of Query heads

self.n_kv_heads = n_kv_heads # Number of KV heads (groups)

self.d_k = d_model // n_heads

self.n_rep = n_heads // n_kv_heads # Query heads per group

self.W_q = nn.Linear(d_model, n_heads * self.d_k)

self.W_k = nn.Linear(d_model, n_kv_heads * self.d_k)

self.W_v = nn.Linear(d_model, n_kv_heads * self.d_k)

self.W_o = nn.Linear(d_model, d_model)

def repeat_kv(self, x):

"""Repeat KV heads to match Query head count"""

B, H_kv, S, D = x.shape

if self.n_rep == 1:

return x

return (

x[:, :, None, :, :]

.expand(B, H_kv, self.n_rep, S, D)

.reshape(B, self.n_heads, S, D)

)

def forward(self, x, kv_cache=None):

B, T, _ = x.shape

q = self.W_q(x).view(B, T, self.n_heads, self.d_k).transpose(1, 2)

k = self.W_k(x).view(B, T, self.n_kv_heads, self.d_k).transpose(1, 2)

v = self.W_v(x).view(B, T, self.n_kv_heads, self.d_k).transpose(1, 2)

if kv_cache is not None:

k = torch.cat([kv_cache[0], k], dim=2)

v = torch.cat([kv_cache[1], v], dim=2)

KV Cache size: [B, n_kv_heads, S, d_k] x 2

new_cache = (k, v)

Repeat KV heads for broadcast

k = self.repeat_kv(k)

v = self.repeat_kv(v)

attn = torch.matmul(q, k.transpose(-2, -1)) / math.sqrt(self.d_k)

attn = torch.softmax(attn, dim=-1)

out = torch.matmul(attn, v)

out = out.transpose(1, 2).contiguous().view(B, T, -1)

return self.W_o(out), new_cache

Llama 3 70B GQA configuration: n_heads=64, n_kv_heads=8. KV Cache is **reduced to 1/8** compared to MHA with virtually no quality loss.

Multi-Head Latent Attention (MLA)

MLA Architecture (DeepSeek-V2/V3)

MLA, proposed in DeepSeek-V2 (2024), is an innovative method that compresses KV vectors into a low-dimensional **latent vector** for cache storage. During decoding, K and V are reconstructed from the latent vector.

The core idea is:

1. Compress input x into a low-dimensional latent vector c: c = W_down \* x

2. Reconstruct K, V from the latent vector: K = W_uk _ c, V = W_uv _ c

3. Only the low-dimensional c is stored in the cache

class MultiHeadLatentAttention(nn.Module):

"""Multi-Head Latent Attention (MLA)

Used in DeepSeek-V2/V3

Compresses KV into low-dimensional latent space for caching"""

def __init__(self, d_model, n_heads, d_latent, rope_dim=64):

super().__init__()

self.n_heads = n_heads

self.d_k = d_model // n_heads

self.d_latent = d_latent # Latent dimension (much smaller than d_model)

self.rope_dim = rope_dim

Query projection

self.W_q = nn.Linear(d_model, n_heads * self.d_k)

KV compression (Down-projection)

self.W_dkv = nn.Linear(d_model, d_latent)

KV reconstruction (Up-projection)

self.W_uk = nn.Linear(d_latent, n_heads * self.d_k)

self.W_uv = nn.Linear(d_latent, n_heads * self.d_k)

Separate projection for RoPE

self.W_kr = nn.Linear(d_model, rope_dim)

self.W_o = nn.Linear(d_model, d_model)

def forward(self, x, kv_cache=None):

B, T, _ = x.shape

Query

q = self.W_q(x).view(B, T, self.n_heads, self.d_k).transpose(1, 2)

KV compression: d_model -> d_latent

c_kv = self.W_dkv(x) # [B, T, d_latent]

RoPE key

k_rope = self.W_kr(x) # [B, T, rope_dim]

if kv_cache is not None:

c_kv_cached, k_rope_cached = kv_cache

c_kv = torch.cat([c_kv_cached, c_kv], dim=1)

k_rope = torch.cat([k_rope_cached, k_rope], dim=1)

KV Cache: only stores low-dimensional c_kv and k_rope

Memory: B * S * (d_latent + rope_dim) * sizeof(dtype)

Compared to MHA: d_latent / (2 * n_heads * d_k) ratio reduction

new_cache = (c_kv, k_rope)

KV reconstruction: d_latent -> n_heads * d_k

S = c_kv.shape[1]

k = self.W_uk(c_kv).view(B, S, self.n_heads, self.d_k).transpose(1, 2)

v = self.W_uv(c_kv).view(B, S, self.n_heads, self.d_k).transpose(1, 2)

Attention computation

attn = torch.matmul(q, k.transpose(-2, -1)) / math.sqrt(self.d_k)

attn = torch.softmax(attn, dim=-1)

out = torch.matmul(attn, v)

out = out.transpose(1, 2).contiguous().view(B, T, -1)

return self.W_o(out), new_cache

DeepSeek-V2 MLA configuration: d_model=5120, d_latent=512, n_heads=128. KV Cache is reduced by approximately **93.7%** compared to MHA while achieving performance equivalent to MHA.

MHA vs MQA vs GQA vs MLA Comparison

| Feature | MHA | MQA | GQA | MLA |

| ----------------------------------- | ----------- | ------------------ | ------------------ | -------------------------- |

| **KV Heads** | H | 1 | G (1 \< G \< H) | - (latent vector) |

| **Cache per token per layer** | 2Hd_k | 2d_k | 2Gd_k | d_latent + d_rope |

| **Cache ratio vs MHA** | 100% | 1/H | G/H | d_latent/(2Hd_k) |

| **Llama 3 70B example (H=64, G=8)** | 16,384B | 256B | 2,048B | - |

| **DeepSeek-V2 example** | 163,840B | - | - | ~4,608B |

| **Quality Impact** | Baseline | Slight degradation | Nearly identical | Nearly identical |

| **Inference Speed** | Baseline | Fastest | Good | Good (reconstruction cost) |

| **Training Stability** | High | Moderate | High | High |

| **Representative Models** | GPT-3, BERT | PaLM, Falcon | Llama 2/3, Mistral | DeepSeek-V2/V3 |

| **Implementation Complexity** | Low | Low | Medium | High |

KV Cache Compression Techniques

Quantization

Quantizing KV Cache from FP16 to INT8 or INT4 can reduce memory by 2-4x.

class QuantizedKVCache:

"""KV Cache Quantization

FP16 -> INT8 conversion for 50% memory reduction"""

def __init__(self, n_layers, n_heads, max_seq_len, d_k, dtype=torch.int8):

self.n_layers = n_layers

self.scales = {} # Quantization scale factors

self.zero_points = {}

self.cache_k = {}

self.cache_v = {}

def quantize(self, tensor):

"""Per-channel INT8 quantization"""

min_val = tensor.min(dim=-1, keepdim=True)[0]

max_val = tensor.max(dim=-1, keepdim=True)[0]

scale = (max_val - min_val) / 255.0

zero_point = (-min_val / scale).round().clamp(0, 255)

quantized = ((tensor / scale) + zero_point).round().clamp(0, 255).to(torch.uint8)

return quantized, scale, zero_point

def dequantize(self, quantized, scale, zero_point):

"""INT8 -> FP16 dequantization"""

return (quantized.float() - zero_point) * scale

def update(self, layer_idx, k, v):

q_k, s_k, z_k = self.quantize(k)

q_v, s_v, z_v = self.quantize(v)

if layer_idx in self.cache_k:

self.cache_k[layer_idx] = torch.cat([self.cache_k[layer_idx], q_k], dim=2)

self.cache_v[layer_idx] = torch.cat([self.cache_v[layer_idx], q_v], dim=2)

else:

self.cache_k[layer_idx] = q_k

self.cache_v[layer_idx] = q_v

self.scales[layer_idx] = (s_k, s_v)

self.zero_points[layer_idx] = (z_k, z_v)

def get(self, layer_idx):

k = self.dequantize(

self.cache_k[layer_idx],

self.scales[layer_idx][0],

self.zero_points[layer_idx][0]

)

v = self.dequantize(

self.cache_v[layer_idx],

self.scales[layer_idx][1],

self.zero_points[layer_idx][1]

)

return k, v

Eviction Policy

A strategy that selectively removes old token KV pairs as the sequence grows longer.

The **H2O (Heavy-Hitter Oracle)** approach retains only the KV pairs of tokens with high attention scores.

class H2OKVCache:

"""Heavy-Hitter Oracle (H2O) KV Cache Eviction

Retains only tokens with high attention scores"""

def __init__(self, max_cache_size, n_heads, d_k):

self.max_cache_size = max_cache_size

self.attention_scores = None

def update(self, k, v, attn_weights):

"""Accumulate importance based on attention weights and perform eviction"""

B, H, S_new, _ = k.shape

if self.attention_scores is None:

self.attention_scores = attn_weights.sum(dim=2) # [B, H, S]

else:

Accumulate attention scores for new tokens

new_scores = attn_weights.sum(dim=2)

self.attention_scores = torch.cat(

[self.attention_scores, new_scores[:, :, -S_new:]], dim=2

)

Evict when cache size is exceeded

current_size = self.attention_scores.shape[2]

if current_size > self.max_cache_size:

Identify low-importance tokens (always keep first token)

scores = self.attention_scores[:, :, 1:] # Exclude first token

_, indices = scores.topk(

self.max_cache_size - 1, dim=2, sorted=False

)

indices = indices + 1 # Offset correction

Add first token index

first_token = torch.zeros(B, H, 1, dtype=torch.long, device=k.device)

keep_indices = torch.cat([first_token, indices], dim=2)

Keep only selected tokens

k = k.gather(2, keep_indices.unsqueeze(-1).expand(-1, -1, -1, k.shape[-1]))

v = v.gather(2, keep_indices.unsqueeze(-1).expand(-1, -1, -1, v.shape[-1]))

self.attention_scores = self.attention_scores.gather(2, keep_indices)

return k, v

Sliding Window Attention

A fixed-size window-based attention used in models like Mistral.

class SlidingWindowAttention(nn.Module):

"""Sliding Window Attention

Applies attention only to the most recent W tokens

Used in Mistral, Gemma, etc."""

def __init__(self, d_model, n_heads, window_size=4096):

super().__init__()

self.window_size = window_size

self.n_heads = n_heads

self.d_k = d_model // n_heads

self.W_q = nn.Linear(d_model, d_model)

self.W_k = nn.Linear(d_model, d_model)

self.W_v = nn.Linear(d_model, d_model)

self.W_o = nn.Linear(d_model, d_model)

def forward(self, x, kv_cache=None):

B, T, _ = x.shape

q = self.W_q(x).view(B, T, self.n_heads, self.d_k).transpose(1, 2)

k = self.W_k(x).view(B, T, self.n_heads, self.d_k).transpose(1, 2)

v = self.W_v(x).view(B, T, self.n_heads, self.d_k).transpose(1, 2)

if kv_cache is not None:

k = torch.cat([kv_cache[0], k], dim=2)

v = torch.cat([kv_cache[1], v], dim=2)

Limit KV Cache to window size

if k.shape[2] > self.window_size:

k = k[:, :, -self.window_size:]

v = v[:, :, -self.window_size:]

attn = torch.matmul(q, k.transpose(-2, -1)) / math.sqrt(self.d_k)

attn = torch.softmax(attn, dim=-1)

out = torch.matmul(attn, v)

out = out.transpose(1, 2).contiguous().view(B, T, -1)

return self.W_o(out), (k, v)

PagedAttention (vLLM)

Virtual Memory-Based KV Cache Management

vLLM's PagedAttention, inspired by OS virtual memory systems, manages KV Cache in **fixed-size blocks (pages)**. This eliminates the inefficiency of contiguous memory allocation (internal fragmentation, external fragmentation).

class PagedKVCache:

"""vLLM PagedAttention concept implementation

Manages KV Cache in page units"""

def __init__(self, block_size=16, n_blocks=1024, n_heads=32, d_k=128):

self.block_size = block_size

self.n_blocks = n_blocks

self.n_heads = n_heads

self.d_k = d_k

Physical block pool (pre-allocated)

self.k_blocks = torch.zeros(n_blocks, n_heads, block_size, d_k)

self.v_blocks = torch.zeros(n_blocks, n_heads, block_size, d_k)

Free block management

self.free_blocks = list(range(n_blocks))

Per-sequence block tables (virtual -> physical mapping)

self.block_tables = {} # seq_id -> list of physical block indices

self.seq_lengths = {}

def allocate(self, seq_id):

"""Allocate first block for a new sequence"""

if not self.free_blocks:

raise RuntimeError("OOM: No free blocks available")

block_idx = self.free_blocks.pop(0)

self.block_tables[seq_id] = [block_idx]

self.seq_lengths[seq_id] = 0

def append_token(self, seq_id, k, v):

"""Append new token KV to a sequence"""

seq_len = self.seq_lengths[seq_id]

block_idx_in_seq = seq_len // self.block_size

offset = seq_len % self.block_size

Allocate new block when current block is full

if offset == 0 and block_idx_in_seq > 0:

if not self.free_blocks:

raise RuntimeError("OOM: No free blocks available")

new_block = self.free_blocks.pop(0)

self.block_tables[seq_id].append(new_block)

Store KV in physical block

physical_block = self.block_tables[seq_id][block_idx_in_seq]

self.k_blocks[physical_block, :, offset] = k

self.v_blocks[physical_block, :, offset] = v

self.seq_lengths[seq_id] += 1

def get_kv(self, seq_id):

"""Assemble complete KV Cache for a sequence"""

blocks = self.block_tables[seq_id]

seq_len = self.seq_lengths[seq_id]

k_list, v_list = [], []

for i, block_idx in enumerate(blocks):

if i == len(blocks) - 1:

Last block only up to actual token count

remaining = seq_len % self.block_size or self.block_size

k_list.append(self.k_blocks[block_idx, :, :remaining])

v_list.append(self.v_blocks[block_idx, :, :remaining])

else:

k_list.append(self.k_blocks[block_idx])

v_list.append(self.v_blocks[block_idx])

return torch.cat(k_list, dim=1), torch.cat(v_list, dim=1)

def free(self, seq_id):

"""Return blocks after sequence completion"""

for block_idx in self.block_tables[seq_id]:

self.free_blocks.append(block_idx)

del self.block_tables[seq_id]

del self.seq_lengths[seq_id]

The key benefits of PagedAttention are:

- **Eliminates memory fragmentation**: No contiguous memory required, so no external fragmentation

- **Improved memory utilization**: Blocks are allocated only as needed. vLLM achieves up to 24x higher throughput compared to previous systems

- **Copy-on-Write**: Shares KV Cache in beam search and parallel sampling to save memory

- **Prefix caching**: Shares common prefix (system prompt) KV Cache across multiple requests

Failure Cases and Recovery Procedures

Case 1: OOM During Long Context Inference

**Situation**: Attempted document summarization with a 128K context length using the Llama 3 70B model, but OOM occurred at batch size 4, shutting down the entire inference service.

**Memory Analysis**:

- Model weights (FP16): ~140 GB

- KV Cache per request (GQA, H=64, G=8): 2 x 80 x 8 x 128K x 128 x 2 = ~33.5 GB

- Total KV Cache for batch 4: ~134 GB

- Total memory needed: ~274 GB (risky even on 8x A100 80GB = 640 GB when considering activation memory and temporary buffers)

**Recovery Procedure**:

1. Set memory limits when restarting the vLLM server

python -m vllm.entrypoints.openai.api_server \

--model meta-llama/Llama-3-70B \

--tensor-parallel-size 8 \

--max-model-len 65536 \

--gpu-memory-utilization 0.9 \

--max-num-seqs 8 \

--enforce-eager

2. Enable KV Cache quantization

python -m vllm.entrypoints.openai.api_server \

--model meta-llama/Llama-3-70B \

--tensor-parallel-size 8 \

--kv-cache-dtype fp8_e5m2 \

--max-model-len 128000

3. Dynamic batch size limiting

Reduce max-num-seqs to limit concurrent request count

Case 2: Quality Degradation from KV Cache Quantization

**Situation**: KV Cache was quantized to INT4 for inference cost reduction, but response quality degraded significantly in long conversations. Error rates increased particularly in math calculations and code generation tasks.

**Symptoms**:

- Increased hallucinations after 10+ conversation turns

- 15% accuracy drop on math problems

- Frequent syntax errors in code generation

**Recovery Procedure**:

1. Upgrade quantization precision from INT4 to INT8

In vLLM, change kv-cache-dtype

--kv-cache-dtype fp8_e5m2 (use FP8)

2. Keep only critical layers at high precision (mixed quantization)

Initial and final layers at FP16, middle layers at INT8

kv_cache_config = {

"default_dtype": "int8",

"high_precision_layers": [0, 1, 2, 77, 78, 79], # First/last 3 layers

"high_precision_dtype": "fp16"

}

3. Re-run benchmarks to verify quality

Validate with standard benchmarks like MMLU, HumanEval, GSM8K

Case 3: Response Mix-up from Prefix Caching Error

**Situation**: After enabling vLLM's prefix caching feature, incorrect contexts were applied to different users' requests. KV Cache was incorrectly shared between different sessions with the same system prompt.

**Recovery Procedure**:

1. Temporarily disable prefix caching

python -m vllm.entrypoints.openai.api_server \

--model meta-llama/Llama-3-70B \

--enable-prefix-caching false

2. Modify prefix hash key to include session ID

Different sessions should use separate caches even with the same prefix

3. Set TTL when re-enabling prefix caching

--prefix-cache-ttl 300 (auto-expire after 5 minutes)

Optimization Checklist

Model Selection and Configuration

- [ ] Verify the model's attention mechanism (MHA/MQA/GQA/MLA)

- [ ] Validate optimal n_kv_heads value when using GQA models (typically n_heads/8)

- [ ] Limit maximum sequence length to match actual usage patterns

KV Cache Memory Management

- [ ] Calculate KV Cache allocation ratio relative to GPU memory (60-80% of total recommended)

- [ ] Decide on KV Cache quantization (quality/memory tradeoff: FP8 > INT8 > INT4)

- [ ] Decide on PagedAttention adoption (vLLM, TensorRT-LLM, etc.)

- [ ] Verify key collision prevention when enabling prefix caching

Inference Performance Optimization

- [ ] Enable Continuous Batching (immediately recycle slots upon request completion)

- [ ] Evaluate Speculative Decoding adoption (maximize KV Cache utilization with draft model)

- [ ] Assess Sliding Window Attention applicability (long document summarization, etc.)

- [ ] Choose between Tensor Parallelism vs Pipeline Parallelism

Monitoring

- [ ] Collect KV Cache utilization metrics (vLLM: `vllm:cache_usage_percent`)

- [ ] Track per-request KV Cache memory usage

- [ ] Set up automatic OOM event alerts

- [ ] Profile throughput and latency by batch size

Operational Notes

- [ ] Set concurrent request upper limits (max-num-seqs)

- [ ] Set GPU memory utilization upper limits (gpu-memory-utilization)

- [ ] Implement graceful degradation on OOM (request queuing, batch reduction)

- [ ] Perform regular KV Cache profiling to check for memory leaks

Conclusion

KV Cache optimization is the key to LLM inference efficiency. The evolution of attention mechanisms from MHA to MQA, GQA, and MLA has successfully reduced KV Cache memory dramatically while maintaining model quality. GQA is a practical approach proven in the Llama series, and MLA enables even more aggressive compression as demonstrated in DeepSeek-V2/V3.

KV Cache quantization, eviction policies, and Sliding Window Attention are additional techniques for handling long sequences under memory constraints, while PagedAttention innovates memory management itself to maximize inference system throughput.

In real production environments, these techniques must be used in combination, and continuous profiling and benchmarking are essential to find optimal configurations that match model characteristics, workload patterns, and GPU resources.

References

- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (Ainslie et al., 2023)](https://arxiv.org/abs/2305.13245)

- [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model (2024)](https://arxiv.org/abs/2405.04434)

- [KV Caching Explained - Hugging Face Blog](https://huggingface.co/blog/not-lain/kv-caching)

- [KV Cache Optimization via Multi-Head Latent Attention - PyImageSearch](https://pyimagesearch.com/2025/10/13/kv-cache-optimization-via-multi-head-latent-attention/)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon et al., 2023)](https://arxiv.org/abs/2309.06180)