Complete Guide to vLLM Production Serving Optimization: From PagedAttention to Kubernetes Deployment

Traditional LLM Serving (HuggingFace Transformers, etc.):
├─ Pre-allocates contiguous memory for max sequence length per request
├─ Max length memory is occupied even when actual sequence is short → internal fragmentation
├─ No memory sharing between requests → external fragmentation
└─ Result: 60-80% of GPU memory is wasted

vLLM PagedAttention:
├─ Divides KV Cache into fixed-size blocks (e.g., 16 tokens)
├─ Blocks can be stored in non-contiguous memory
├─ Dynamically allocates/frees blocks as needed
├─ Manages logical→physical mapping via block tables
└─ Result: Reduces GPU memory waste to under 5%

# PagedAttention Block Table Concept
# Logical Block → Physical Block Mapping

# Request 1: "The cat sat on the mat"
# Logical:  [Block 0] [Block 1]
# Physical: [GPU Block 3] [GPU Block 7]

# Request 2: "Hello world"
# Logical:  [Block 0]
# Physical: [GPU Block 1]

# Copy-on-Write when system prompts are identical:
# Request 3 and Request 4 use the same system prompt
# → Share physical blocks for system prompt (no additional memory consumption)

Static Batching:
Time →
Req A: [████████████████████]     ← Generation complete
Req B: [████████]                 ← Finished early but waits for A
Req C: [████████████]             ← Longer than B but waits for A
       ↑ Batch start    ↑ Batch end (when A completes)

Continuous Batching:
Time →
Req A: [████████████████████]
Req B: [████████] → Req D starts immediately: [████████████]
Req C: [████████████] → Req E starts immediately: [██████]
       ↑ Slot is replaced with new request as soon as one finishes

┌─────────────────────────────────────────────┐
│                 vLLM Engine                  │
├─────────────────────────────────────────────┤
│  API Server (OpenAI Compatible)              │
│  ├─ /v1/completions                          │
│  ├─ /v1/chat/completions                     │
│  └─ /v1/embeddings                           │
├─────────────────────────────────────────────┤
│  Scheduler                                   │
│  ├─ Continuous Batching                      │
│  ├─ Priority Queue                           │
│  └─ Preemption (Swap/Recompute)             │
├─────────────────────────────────────────────┤
│  KV Cache Manager (PagedAttention)           │
│  ├─ Block Allocator                          │
│  ├─ Block Table                              │
│  └─ Copy-on-Write                            │
├─────────────────────────────────────────────┤
│  Model Executor                              │
│  ├─ Tensor Parallelism (Ray/NCCL)           │
│  ├─ Quantization (GPTQ/AWQ/FP8)            │
│  └─ Speculative Decoding                    │
└─────────────────────────────────────────────┘

# Single GPU (default)
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 1

# 4 GPU tensor parallelism
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4

# 8 GPU (A100 80GB x 8 node)
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-405B-Instruct \
  --tensor-parallel-size 8 \
  --pipeline-parallel-size 1

# Enable Speculative Decoding
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --speculative-model meta-llama/Llama-3.2-1B-Instruct \
  --num-speculative-tokens 5 \
  --speculative-draft-tensor-parallel-size 1

Traditional autoregressive generation (1 token/step):
Step 1 → "The"
Step 2 → "weather"
Step 3 → "is"
Step 4 → "sunny"
Step 5 → "today"
= 5 forward passes (large model)

Speculative Decoding:
Draft model (fast, small): "The weather is sunny today" (5 tokens speculated)
Target model (1 forward pass): "The weather is sunny today" ✓ All accepted!
= 1 forward pass (small model) + 1 forward pass (large model)
→ Potentially 2.5-3x speed improvement

# Enable Prefix Caching
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --enable-prefix-caching

Scenario: All requests use the same 2000-token system prompt

Prefix Caching disabled:
Req 1: [Process 2000-token system prompt] + [Process user input] → TTFT: 500ms
Req 2: [Reprocess 2000-token system prompt] + [Process user input] → TTFT: 500ms
Req 3: [Reprocess 2000-token system prompt] + [Process user input] → TTFT: 500ms

Prefix Caching enabled:
Req 1: [Process 2000-token system prompt] + [Process user input] → TTFT: 500ms
Req 2: [Cache hit!] + [Process user input] → TTFT: 50ms (10x improvement!)
Req 3: [Cache hit!] + [Process user input] → TTFT: 50ms

# Enable Chunked Prefill
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --enable-chunked-prefill \
  --max-num-batched-tokens 2048

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  \
  # GPU memory settings
  --gpu-memory-utilization 0.90 \          # GPU memory usage ratio (0.0-1.0)
  --max-model-len 32768 \                  # Maximum context length
  \
  # Batch and concurrency settings
  --max-num-seqs 256 \                     # Maximum concurrent sequences
  --max-num-batched-tokens 32768 \         # Maximum tokens per batch
  \
  # Parallelism settings
  --tensor-parallel-size 1 \               # Number of tensor parallel GPUs
  --pipeline-parallel-size 1 \             # Number of pipeline parallel stages
  \
  # Quantization settings
  --quantization awq \                     # Quantization method (awq, gptq, fp8, etc.)
  --dtype auto \                           # Data type (auto, float16, bfloat16)
  \
  # Server settings
  --host 0.0.0.0 \
  --port 8000 \
  --api-key "your-secret-key"

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --gpu-memory-utilization 0.95 \
  --max-num-seqs 512 \
  --max-model-len 4096 \
  --enable-prefix-caching \
  --enable-chunked-prefill

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --gpu-memory-utilization 0.85 \
  --max-num-seqs 32 \
  --max-model-len 8192 \
  --num-scheduler-steps 1

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --gpu-memory-utilization 0.92 \
  --max-num-seqs 16 \
  --max-model-len 131072 \
  --enable-chunked-prefill \
  --max-num-batched-tokens 4096

Want to get started quickly?
→ vLLM (pip install vllm → serve immediately)

Need maximum performance?
→ TensorRT-LLM (requires significant effort for model conversion and configuration)

Already using the HuggingFace ecosystem?
→ TGI (natural integration with HuggingFace Hub)

Need enterprise deployment?
→ Triton + TensorRT-LLM (official NVIDIA support, multi-model serving)

# Single GPU deployment with Docker
docker run --runtime nvidia --gpus '"device=0"' \
  -v /path/to/model:/model \
  -p 8000:8000 \
  vllm/vllm-openai:latest \
  --model /model \
  --gpu-memory-utilization 0.9 \
  --max-model-len 8192

# 4 GPU tensor parallel deployment
docker run --runtime nvidia --gpus '"device=0,1,2,3"' \
  -v /path/to/model:/model \
  -p 8000:8000 \
  vllm/vllm-openai:latest \
  --model /model \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.9

# vllm-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-serving
  namespace: ml-serving
spec:
  replicas: 2
  selector:
    matchLabels:
      app: vllm-serving
  template:
    metadata:
      labels:
        app: vllm-serving
    spec:
      containers:
        - name: vllm
          image: vllm/vllm-openai:latest
          args:
            - '--model'
            - 'meta-llama/Llama-3.1-8B-Instruct'
            - '--gpu-memory-utilization'
            - '0.9'
            - '--max-model-len'
            - '8192'
            - '--max-num-seqs'
            - '256'
            - '--enable-prefix-caching'
            - '--port'
            - '8000'
          ports:
            - containerPort: 8000
              name: http
          resources:
            limits:
              nvidia.com/gpu: 1
              memory: '32Gi'
              cpu: '8'
            requests:
              nvidia.com/gpu: 1
              memory: '16Gi'
              cpu: '4'
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 120
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 180
            periodSeconds: 30
          env:
            - name: VLLM_USAGE_SOURCE
              value: 'production'
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-service
  namespace: ml-serving
spec:
  selector:
    app: vllm-serving
  ports:
    - port: 80
      targetPort: 8000
      name: http
  type: ClusterIP
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: vllm-hpa
  namespace: ml-serving
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: vllm-serving
  minReplicas: 2
  maxReplicas: 8
  metrics:
    - type: Pods
      pods:
        metric:
          name: gpu_utilization
        target:
          type: AverageValue
          averageValue: '80'

# When serving multiple models on the same cluster
# Deploy separate vLLM instances per model + router in front

# router.py (simple example)
from fastapi import FastAPI, Request
import httpx

app = FastAPI()

MODEL_ENDPOINTS = {
    "llama-8b": "http://vllm-8b:8000",
    "llama-70b": "http://vllm-70b:8000",
    "codellama-34b": "http://vllm-code:8000",
}

@app.post("/v1/chat/completions")
async def route_request(request: Request):
    body = await request.json()
    model = body.get("model", "llama-8b")
    endpoint = MODEL_ENDPOINTS.get(model)

    async with httpx.AsyncClient() as client:
        response = await client.post(
            f"{endpoint}/v1/chat/completions",
            json=body,
            timeout=120.0
        )
        return response.json()

# Key Prometheus metrics
# vllm:num_requests_running - Currently processing requests
# vllm:num_requests_waiting - Waiting requests
# vllm:gpu_cache_usage_perc - KV Cache GPU utilization
# vllm:cpu_cache_usage_perc - KV Cache CPU swap utilization
# vllm:avg_prompt_throughput_toks_per_s - Prompt processing throughput
# vllm:avg_generation_throughput_toks_per_s - Generation throughput
# vllm:e2e_request_latency_seconds - E2E request latency histogram
# vllm:time_to_first_token_seconds - TTFT histogram
# vllm:time_per_output_token_seconds - TBT histogram

# prometheus-scrape-config.yaml
scrape_configs:
  - job_name: 'vllm'
    scrape_interval: 15s
    static_configs:
      - targets: ['vllm-service:8000']
    metrics_path: '/metrics'

# prometheus-alert-rules.yaml
groups:
  - name: vllm_alerts
    rules:
      - alert: HighKVCacheUsage
        expr: vllm_gpu_cache_usage_perc > 0.95
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'KV Cache usage exceeds 95%'

      - alert: HighRequestLatency
        expr: histogram_quantile(0.99, vllm_e2e_request_latency_seconds_bucket) > 30
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: 'P99 request latency exceeds 30 seconds'

      - alert: HighQueueDepth
        expr: vllm_num_requests_waiting > 100
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: 'Waiting requests exceed 100'

# 1. Lower gpu-memory-utilization
--gpu-memory-utilization 0.80  # Reduce from default 0.9

# 2. Reduce max-model-len
--max-model-len 4096  # Limit unnecessarily long contexts

# 3. Reduce max-num-seqs
--max-num-seqs 64  # Decrease concurrent processing

# 4. Apply quantization
--quantization awq  # Or gptq, fp8

# 5. Tensor parallelism (add GPUs)
--tensor-parallel-size 2

# 1. Long prompts → Enable Chunked Prefill
--enable-chunked-prefill
--max-num-batched-tokens 2048

# 2. Enable Prefix Caching (when repeated prompts exist)
--enable-prefix-caching

# 3. Check CUDA graph optimization
# Remove --enforce-eager (enable CUDA graphs)

# 4. Optimize model loading
--load-format auto  # Use safetensors when possible

# 1. Check batch size
--max-num-seqs 256  # Too small leads to low GPU utilization

# 2. Check memory utilization
--gpu-memory-utilization 0.92  # Too conservative limits batch size

# 3. Try Speculative Decoding
--speculative-model <small-model> --num-speculative-tokens 5

# 4. Apply quantization
--quantization awq  # Or fp8 (A100/H100)

# 1. Limit maximum tokens
# Set max_tokens appropriately in API requests

# 2. Allocate swap space
--swap-space 16  # Allow swapping to CPU memory

# 3. Set preemption strategy
--preemption-mode recompute  # Or swap

# Leverage FP8 quantization on NVIDIA H100
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --quantization fp8 \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.92

# FP8 compared to INT8:
# - Higher accuracy (maintains FP range)
# - Maximum performance with H100 FP8 Tensor Cores
# - No separate calibration required

# Serve base model + multiple LoRA adapters simultaneously
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --enable-lora \
  --lora-modules \
    korean-chat=/path/to/korean-lora \
    code-assist=/path/to/code-lora \
    medical-qa=/path/to/medical-lora \
  --max-loras 3 \
  --max-lora-rank 64

# vLLM built-in benchmark tools
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct &

# Throughput benchmark
python -m vllm.benchmark_throughput \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --input-len 512 \
  --output-len 128 \
  --num-prompts 1000

# Latency benchmark
python -m vllm.benchmark_latency \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --input-len 512 \
  --output-len 128 \
  --batch-size 1