The New Throne Beyond vLLM: 5 Reasons SGLang Is Changing the LLM Inference Landscape

1. Introduction: Curing LLM Inference's 'Amnesia'
2. [Reason 1] RadixAttention: A Paradigm Shift in KV Cache
3. [Reason 2] The 29% Gap: Hyper-Specialized Design
4. [Reason 3] The 4,000-Line Miracle: Python Zero-Overhead Scheduler
5. [Reason 4] Prefill-Decode Disaggregation: Division of Labor in Computation
6. [Reason 5] Structured Generation: The Innovation of Compressed FSM
7. Comprehensive LLM Inference Framework Comparison
- 7.1 SGLang vs vLLM vs TGI vs TensorRT-LLM
- 7.2 Selection Guide
8. SGLang Installation and Quick Start
9. API Usage and Code Examples
10. Key Configuration Parameter Guide
11. Deployment Guide
12. SGLang 2026 Roadmap and Ecosystem
13. Conclusion: Breaking Hardware Limits with Software Intelligence
14. References

1. Introduction: Curing LLM Inference's 'Amnesia'

1.1 Repeated Computation, Wasted GPUs

In the second half of 2024, as LLM-based agents and RAG pipelines were being deployed into production environments in earnest, an inconvenient truth surfaced: the problem of LLM inference's 'amnesia'.

Every time a user sends the same system prompt and few-shot examples repeatedly, the serving engine discards previously computed KV caches and recomputes everything from scratch. In multi-turn conversations, context from previous turns cannot be reused, and in RAG pipelines, the encoding of shared document chunks is performed anew each time. In an era where H100 GPU costs reach $3-4 per hour, such redundant computation translates directly into wasted money.

Existing inference engines only partially addressed this problem. vLLM's PagedAttention revolutionized KV cache memory management efficiency, but failed to fundamentally solve the bigger problem of inter-request cache reuse. TensorRT-LLM delivers extreme kernel optimization on NVIDIA hardware, but showed limitations in integration with flexible prompt programming.

1.2 SGLang: DSL + Runtime Co-Design

This is where SGLang (Structured Generation Language) enters the picture. Developed primarily by the LMSYS team at UC Berkeley, SGLang is not just another inference engine. It is an integrated system that co-designs the frontend language (DSL) and the backend runtime to optimally execute complex LLM programs.

"SGLang is a system for efficient execution of complex language model programs. It consists of a frontend language and a runtime. The frontend simplifies programming with primitives for generation and parallelism control. The runtime accelerates execution with novel optimizations." — Zheng et al., NeurIPS 2024

To summarize SGLang's core design philosophy in one phrase: it is an 'LLM Inference Operating System'. Just as an operating system integrates process scheduling, memory management, and I/O optimization, SGLang unifies KV cache management (RadixAttention), CPU-GPU pipelining (Zero-Overhead Scheduler), distributed execution (PD Disaggregation), and structured output (Compressed FSM) into a single coherent architecture.

1.3 SGLang by the Numbers

The benchmark results reported in the paper are impressive.

Metric	SGLang Performance
Agent Tasks	Up to 6.4x throughput vs vLLM
H100 Token Throughput	16,215 tok/s (vLLM: 12,553)
Multi-turn Cache Hit Rate	85-95% (vLLM: 15-25%)
JSON Structured Decoding	Up to 3x speed improvement
PD Disaggregation	52.3K input tok/s per node

In this article, we provide an in-depth technical analysis of the 5 key reasons SGLang is changing the LLM inference landscape. Each reason is not a simple feature addition but an architectural innovation that redesigns the entire inference pipeline.

2. [Reason 1] RadixAttention: A Paradigm Shift in KV Cache

2.1 Limitations of PagedAttention

PagedAttention, introduced by vLLM, applied the concept of virtual memory from operating systems to KV cache management, dramatically solving memory fragmentation problems. By managing KV caches in fixed-size block units, it minimized both Internal and External Fragmentation.

However, PagedAttention has a fundamental limitation: inter-request KV cache sharing is difficult.

[Request A] System Prompt(500 tokens) + User Query A(50 tokens)
[Request B] System Prompt(500 tokens) + User Query B(80 tokens)
[Request C] System Prompt(500 tokens) + User Query C(30 tokens)

All three requests share the same 500-token system prompt, but vLLM's default PagedAttention independently computes and stores the KV cache of the system prompt for each request. This causes massive waste in both GPU computation and memory. While vLLM also introduced Automatic Prefix Caching, it relies on exact prompt matching and has limitations with partial prefix sharing.

2.2 Radix Tree: Innovation in Data Structures

SGLang's RadixAttention solves this problem with a fundamentally different approach. The key is introducing the Radix Tree data structure for KV cache management.

A Radix Tree (also called a Patricia Trie) is a tree structure that compresses and stores common prefixes of strings. SGLang's core insight lies in applying this data structure -- widely used in network routing tables (IP Prefix Matching) and string dictionaries -- to KV cache management.

RadixAttention KV Cache Tree Structure
=================================

                          [ROOT]
                            |
                +-----------+-----------+
                |                       |
        [System Prompt]           [Few-shot Prefix]
        "You are a helpful       "Translate the following
         AI assistant..."         examples..."
         (KV: 500 tokens)         (KV: 800 tokens)
                |                       |
        +-------+-------+        +-----+-----+
        |               |        |           |
   [User A]        [User B]  [Example 1]  [Example 2]
   "What is       "Explain   "Hello->     "Good->
    Docker?"      K8s pods"   Annyeong"    Joeun"
   (KV: 50 tok)  (KV: 80 tok) (KV: 30 tok) (KV: 25 tok)
        |               |
   [Turn 2-A]     [Turn 2-B]
   "How to       "What about
    install?"     services?"
   (KV: 40 tok)  (KV: 60 tok)
        |
   [Turn 3-A]
   "Configure
    networking"
   (KV: 45 tok)

As shown in the diagram above, each node in the Radix Tree stores a token sequence and its corresponding KV cache pages. When a new request arrives, the runtime traverses the tree and automatically detects the Longest Common Prefix. The KV cache for the matched prefix is reused, and only the remaining unmatched portion is newly computed.

2.3 How RadixAttention Works

Let's examine the core operation flow of RadixAttention in detail.

Step 1: Prefix Matching

New Request: "You are a helpful AI assistant... What is Kubernetes?"

Radix Tree Search:
  [ROOT] -> [System Prompt] Match! (500 tokens reused)
                 -> [User A: "What is Docker?"] Mismatch
                 -> [User B: "Explain K8s pods"] Mismatch
                 -> New branch needed

Result: 500 tokens of KV cache reused, only remaining 50 tokens newly computed

Step 2: Cache Insertion

Newly computed KV caches are inserted at the appropriate position in the tree. If there is a common prefix with an existing node, the node is split; if not, a new child node is added.

Step 3: LRU Eviction + Tree Pruning

When GPU memory runs low, an intelligent eviction combining LRU (Least Recently Used) policy and tree structure is performed.

LRU Eviction Process:
=================
Memory shortage detected!

1. Select the least recently accessed leaf node
   -> [Turn 3-A] (last access: 30 minutes ago)

2. Free that node's KV cache (45 tokens reclaimed)

3. Inspect parent node: Does [Turn 2-A] have other children?
   -> No: Parent is promoted to eviction candidate
   -> Yes: Parent is retained

4. Is memory sufficient? -> If not, repeat with next LRU leaf node

The key mechanism here is that eviction starts from leaf nodes. Upper tree nodes (system prompts, etc.) are naturally preserved because they are shared by more requests, while individual user turns at lower levels are evicted first. This effectively automatically prioritizes preserving nodes with higher cache reuse value.

2.4 Power in Agent Workflows

RadixAttention's power is dramatically demonstrated in the rapidly growing agent workflows. Agents repeat tool calls, observations, and reasoning, sending the entire conversation history as context at each step.

Agent Execution Flow (10-step reasoning):
================================
[Step 1] System + Tools + Query               -> 1000 tokens (newly computed)
[Step 2] System + Tools + Query + Obs1         -> 1000 reused + 200 newly computed
[Step 3] System + Tools + Query + Obs1 + Obs2  -> 1200 reused + 200 newly computed
...
[Step 10] System + Tools + Query + Obs1~Obs9   -> 2800 reused + 200 newly computed

RadixAttention: Total new computation = 1000 + 200*9 = 2,800 tokens
PagedAttention: Total new computation = 1000 + 1200 + 1400 + ... + 2800 = 19,000 tokens

Savings: approximately 85%

According to Zheng et al.'s paper, in such agent scenarios, SGLang achieves up to 5x throughput improvement compared to vLLM. Notably, in few-shot learning benchmarks, SGLang's cache hit rate reaches 85-95%, while vLLM remains at 15-25%.

2.5 Cache-Aware Load Balancer

The Cache-Aware Load Balancer introduced in SGLang v0.4 extends RadixAttention's effectiveness to multi-instance environments. When multiple SGLang server instances exist, the load balancer routes requests by considering each instance's Radix Tree state instead of simple round-robin.

Cache-Aware Load Balancing:

Request: "System Prompt A + User Query X"

Instance 1: Radix Tree has "System Prompt A" cached <- Route here!
Instance 2: Radix Tree has "System Prompt B" cached
Instance 3: Radix Tree has no "System Prompt A" cache

Result: Cache hit rate maximized -> 1.9x throughput improvement, 3.8x cache hit rate improvement

This feature allows SGLang to maximize KV cache reuse even at the cluster level.

3. [Reason 2] The 29% Gap: Hyper-Specialized Design

3.1 Same Kernels, Different Performance

There is a frequently overlooked fact when comparing LLM inference engine performance: SGLang and vLLM actually use the same FlashInfer kernels for GPU computation. FlashInfer is an inference-specialized variant of FlashAttention, and the GPU kernels for Attention operations themselves are identical.

Yet benchmark results on H100 GPUs show a surprising gap.

Engine	Throughput (tok/s)	Relative Performance
SGLang	16,215	100%
LMDeploy	16,132	99.5%
vLLM	12,553	77.4%

A 29% throughput gap occurs despite using the same GPU kernels. Where does this gap come from?

3.2 The Difference in Architectural Philosophy

The answer lies in the fundamental difference in architectural design philosophy.

vLLM: Flexible Plugin Architecture

vLLM designed thick plugin-based abstraction layers to support various hardware (NVIDIA, AMD, Intel, TPU) and diverse model architectures. It maximized extensibility by designing Attention Backend, Executor, Worker, etc., as swappable modules.

vLLM Architecture (Simplified):
================================
[Request] -> [Scheduler] -> [Executor Interface]
                              |
                    [Backend Abstraction Layer]
                              |
                    [Attention Backend Plugin]
                              |
                    [FlashInfer / FlashAttention / ...]
                              |
                          [GPU Kernel]

The advantages of this design are clear: adding new hardware support, integrating new Attention algorithms, and accepting community contributions are straightforward. However, these abstraction layers incur overhead from indirection, memory copies, and type conversions. Each layer's overhead is negligible individually, but accumulates over thousands to tens of thousands of iterations in autoregressive decoding.

SGLang: Hyper-Specialized Integration

SGLang took the opposite approach: narrowing the scope of support while optimizing performance to the extreme on supported paths.

SGLang Architecture (Simplified):
=================================
[Request] -> [Zero-Overhead Scheduler]
                    |
            [Direct FlashInfer Integration]
                    |
            [TMA-Optimized GPU Kernel]
                    |
                [GPU HBM]

SGLang directly integrates FlashInfer kernels, minimizing intermediate abstraction layers. It directly implements optimized memory access patterns at the kernel level, leveraging NVIDIA Hopper architecture's TMA (Tensor Memory Accelerator).

3.3 Micro-Benchmark Analysis

Breaking down the 29% gap reveals the following components.

Overhead Source	Est. Contribution	Description
Scheduler Overhead	~10%	vLLM's complex scheduling logic vs SGLang's zero-overhead scheduler
Memory Management	~8%	Block table management, metadata synchronization
Abstraction Layer Cost	~6%	Backend dispatch, type conversion
Cache Management	~5%	RadixAttention's tree-based vs hash-based approach

Each item is individually small, but accumulates over thousands of autoregressive decoding iterations to form the meaningful 29% gap.

3.4 Real-World Benchmarks: Per-Model Comparison

Synthesizing benchmark results across various models:

Model	GPU	SGLang (tok/s)	vLLM (tok/s)	TRT-LLM (tok/s)	TGI (tok/s)
Llama-3.1-8B	1x H100	16,215	12,553	14,800	11,200
Llama-3.1-70B	4x H100	8,500	6,800	8,200	5,900
Mixtral-8x7B	2x H100	12,800	10,100	11,500	8,700
Qwen-2.5-72B	4x H100	7,900	6,200	7,500	5,500
DeepSeek-V3 (EP)	8x H100	6,200	4,800	-	-

SGLang records the highest throughput in most scenarios, with additional advantages in MoE (Mixture of Experts) models through Expert Parallelism support.

4. [Reason 3] The 4,000-Line Miracle: Python Zero-Overhead Scheduler

4.1 "Python Control, Native Compute" Paradigm

The hidden bottleneck of LLM inference is not the GPU but the CPU. While the GPU performs the current batch's Forward Pass, the CPU must prepare metadata for the next task. There are numerous CPU tasks: batch composition, memory allocation, prefix matching, request queue management, etc.

An unoptimized inference engine can spend up to 50% of total execution time on CPU overhead. To solve this, SGLang adopts the "Python Control, Native Compute" paradigm.

Traditional Approach (Sequential):
========================
Time ->  [CPU: Batch N prep] [GPU: Batch N compute] [CPU: Batch N+1 prep] [GPU: Batch N+1 compute]
                                ^ GPU idle                              ^ GPU idle

SGLang Zero-Overhead (Pipelined):
==================================
Time ->  [CPU: Batch N prep] [CPU: Batch N+1 prep] [CPU: Batch N+2 prep] [CPU: Batch N+3 prep]
         [                 ] [GPU: Batch N compute] [GPU: Batch N+1 compute] [GPU: Batch N+2 compute]
                              ^ No GPU idle!         ^ No GPU idle!          ^ No GPU idle!

4.2 Asynchronous CPU-GPU Pipelining

SGLang's scheduler implements asynchronous pipelining where the CPU prepares Batch N+1's metadata while the GPU processes Batch N.

# Core loop of SGLang's scheduler (conceptual pseudocode)
class ZeroOverheadScheduler:
    def run_event_loop(self):
        while True:
            # 1. Asynchronously receive previous batch results from GPU (non-blocking)
            completed = self.recv_from_gpu(blocking=False)
            if completed:
                self.process_completed_tokens(completed)

            # 2. Compose next batch (performed on CPU)
            next_batch = self.schedule_next_batch()

            # 3. Prefix matching in Radix Tree (performed on CPU)
            self.match_prefixes(next_batch)

            # 4. GPU memory allocation (performed on CPU)
            self.allocate_kv_cache(next_batch)

            # 5. Send batch to GPU (non-blocking)
            self.send_to_gpu(next_batch)

            # GPU is processing the previous batch throughout this entire process!

The key is that all CPU work runs in parallel with GPU computation. SGLang separates forward_stream and copy_stream to execute Forward Pass GPU computation and Data-to-Host (D2H) memory transfers independently, maximizing overlap.

4.3 Iterative Scheduling

Another innovation of SGLang is its Iterative Scheduling approach. While traditional batch schedulers wait until all requests in a batch complete, SGLang reconstructs the batch at every Forward Iteration.

Traditional Static Batching:
=====================
Batch: [Req A(100 tok), Req B(50 tok), Req C(200 tok)]

Step 1-50:   A, B, C all processed
Step 51-100: A, C processed (B completed, slot wasted)
Step 101-200: Only C processed (A also completed, 2 slots wasted)

SGLang Iterative Scheduling:
============================
Step 1-50:   [A, B, C] processed
Step 51:     B completed -> immediately [A, C, D] (new request D inserted)
Step 101:    A completed -> immediately [C, D, E, F] (new requests E, F inserted)

-> GPU utilization maximized!

4.4 Codebase Lightness

SGLang's core scheduler is implemented in approximately 4,000 lines of pure Python code. This number is significant.

Engine	Scheduler Code Size	Language
SGLang	~4,000 lines	Python
vLLM	~30,000+ lines	Python + C++
TensorRT-LLM	~50,000+ lines	C++ + Python
TGI	~20,000+ lines	Rust + Python

The fact that a Python-written scheduler matches or outperforms C++ or Rust-based schedulers proves that algorithm design matters more than implementation language. SGLang's asynchronous pipelining completely hides CPU scheduling time behind GPU computation time, so Python's relative slowness does not become a bottleneck.

This lightness also offers practical advantages. Debugging, profiling, and adding custom scheduling logic are overwhelmingly easier compared to C++-based engines. The environment that enables researchers and engineers to quickly experiment and contribute drives SGLang's rapid pace of development.

4.5 Micro-Batching Event Loop

The Micro-Batching Event Loop introduced after SGLang v0.4 takes pipelining one step further. In Pipeline Parallelism (PP) environments, it overlaps GPU computation, CPU metadata processing, and PP communication through asynchronous P2P (Peer-to-Peer) communication.

Micro-Batching Event Loop (PP=2):
==================================
GPU Stage 0: [Fwd mb0] [Fwd mb2] [Fwd mb4] ...
GPU Stage 1:           [Fwd mb0] [Fwd mb2] [Fwd mb4] ...
P2P Comm:      [Send mb0->S1] [Send mb2->S1] ...
CPU:         [Sched mb2] [Sched mb4] [Sched mb6] ...

mb = micro-batch, S = Stage, Fwd = Forward Pass
-> All resources active simultaneously!

Through this design, SGLang minimizes bubbles in Pipeline Parallelism environments, maintaining high efficiency even in multi-node serving of large-scale models.

5. [Reason 4] Prefill-Decode Disaggregation: Division of Labor in Computation

5.1 The Fundamental Difference Between Prefill and Decode

The two phases of LLM inference -- Prefill and Decode -- have completely different hardware requirements.

Characteristic	Prefill Phase	Decode Phase
Computation Type	Compute-Bound	Memory-Bound
Bottleneck	GPU FLOPS	GPU Memory Bandwidth
Batch Nature	Large input, parallel	1 token, sequential
GPU Utilization	High (60-80%)	Low (10-30%)
Optimal Hardware	High FLOPS	High HBM Bandwidth
Latency Impact	TTFT	TPOT

Existing inference engines process both phases through time-sharing on the same GPU. While techniques like Continuous Batching provide some optimization, the fundamental inefficiency of different computational patterns sharing a single GPU remains unresolved.

Interference Between Prefill and Decode:

Time-sharing on the same GPU:
========================
Time -> [Prefill Req A] [Decode Batch 1-5] [Prefill Req B] [Decode Batch 1-8]
                          ^                                   ^
                   Decode latency irregular           New Prefill disrupts existing Decode
                   (Prefill occupies GPU)             (TPOT spike occurs)

When a long prompt's Prefill runs, the TPOT (Time Per Output Token) of already-decoding requests surges -- an interference phenomenon that severely degrades user experience.

5.2 SGLang's PD Disaggregation Architecture

SGLang supports a Disaggregation architecture that places Prefill and Decode on physically separated GPU groups.

PD Disaggregation Architecture:
================================

   +-----------------------------------------------------+
   |                    Router / Gateway                  |
   |      (Receives requests and distributes to Prefill) |
   +------------------+----------------------------------+
                      |
          +-----------+-----------+
          v                       v
   +--------------+      +--------------+
   |  Prefill GPU  |      |  Prefill GPU  |
   |   Group #1    |      |   Group #2    |
   |  (H100 x4)   |      |  (H100 x4)   |
   |              |      |              |
   | Compute-     |      | Compute-     |
   | Optimized    |      | Optimized    |
   +------+-------+      +------+-------+
          | NIXL/RDMA            | NIXL/RDMA
          | KV Cache Transfer    | KV Cache Transfer
          v                       v
   +--------------+      +--------------+
   |  Decode GPU   |      |  Decode GPU   |
   |   Group #1    |      |   Group #2    |
   |  (H100 x4)   |      |  (H100 x4)   |
   |              |      |              |
   | Bandwidth-   |      | Bandwidth-   |
   | Optimized    |      | Optimized    |
   +--------------+      +--------------+

5.3 NIXL: High-Speed KV Cache Transfer

Separating Prefill and Decode creates a new challenge: transferring KV caches generated during Prefill to Decode GPUs. SGLang uses NVIDIA's NIXL (NVIDIA Inference Transfer Library) as the transfer backend.

NIXL is a low-latency point-to-point transfer library that unifies diverse fabrics (NVLink, InfiniBand, PCIe, SSD) into a single abstraction layer.

NIXL KV Cache Transfer Flow:
============================

1. Bootstrap: Decode -> Prefill passes bootstrap_room ID
2. Memory Alloc: Decode pre-allocates GPU memory pages
3. Prefill Exec: Prefill Worker processes prompt -> generates KV Cache
4. RDMA Write: Prefill writes directly to Decode GPU memory via RDMA
                (CPU bypass, Zero-Copy)
5. Completion: Decode polls for transfer completion -> immediately starts Decode

NVLink transfer bandwidth: ~900 GB/s (GB200 NVL72)
InfiniBand transfer bandwidth: ~400 Gb/s (HDR)

Through RDMA (Remote Direct Memory Access), direct transfer between GPU memories bypasses the CPU, minimizing transfer latency.

5.4 Blackwell GB200/B200 Optimization

SGLang's PD Disaggregation delivers particularly powerful performance on the NVIDIA Blackwell architecture.

Hardware	Prefill Performance	Decode Performance	Notes
H100 SXM	Baseline	Baseline	FP8
B200 SXM	~2.5x	~2.0x	NVFP4 support
GB200 NVL72	~3.8x	~4.8x	NVLink 900GB/s

On GB200 NVL72, SGLang serves DeepSeek V3/R1 models with FP8 Attention + NVFP4 MoE configuration, achieving 26,156 input tokens/s and 13,386 output tokens/s per GPU. This represents 3.8x prefill and 4.8x decode improvement over H100.

5.5 The Value of Independent Scaling

The most important practical benefit of PD Disaggregation is independent scaling.

Scaling by traffic pattern:

[Scenario 1: Long prompts, short responses (document summarization)]
-> Prefill GPU: 8 units  /  Decode GPU: 2 units

[Scenario 2: Short prompts, long responses (code generation)]
-> Prefill GPU: 2 units  /  Decode GPU: 8 units

[Scenario 3: Balanced load (general chatbot)]
-> Prefill GPU: 4 units  /  Decode GPU: 4 units

Since Prefill and Decode resources can be independently adjusted according to workload characteristics, GPU utilization can be maximized while optimizing costs. This enables natural integration with autoscaling in cloud environments.

6. [Reason 5] Structured Generation: The Innovation of Compressed FSM

6.1 The Need for Structured Output

In production LLM applications, forcing model output to conform to a specific format (JSON, XML, SQL, etc.) is essential. Agent system tool call parameters must be valid JSON, and data extraction pipeline outputs must follow a predefined schema.

Existing structured generation methods are implemented through token-level masking. At each token generation, only tokens allowed by the current FSM (Finite State Machine) state are kept, and the logits of the rest are set to -infinity. While accurate, this approach incurs significant overhead as FSM state transitions must be computed for every token.

6.2 How Compressed FSM Works

SGLang's Compressed FSM converts a JSON schema (or regular expression) into an FSM, then applies an optimization that compresses adjacent Singular Transition Edges.

JSON Schema Example:
{
  "name": string,
  "age": integer,
  "city": string
}

Regular FSM (per-token transitions):
========================
S0 ->'{' -> S1 ->'"' -> S2 ->'n' -> S3 ->'a' -> S4 ->'m' -> S5 ->'e' -> S6 ->'"' -> S7 ->':' -> S8

Each transition requires a GPU forward pass -> 8 forward passes

Compressed FSM (Jump-Forward):
=============================
S0 -> '{"name":' -> S8  (Singular transitions compressed!)

Detection: From S0 to S8, each state has exactly one possible transition
-> 8 tokens generated in 1 forward pass!

6.3 Jump-Forward Mechanism

The core of Compressed FSM is Jump-Forward Decoding. It analyzes the FSM to pre-identify segments where a unique path (singular transition path) exists from the current state to the next state.

Jump-Forward Decoding Process:
===========================

1. FSM Pre-analysis:
   - Calculate possible transitions from each state
   - Identify and compress singular path segments

2. Decoding Execution:
   State S0: Possible transitions = {'{'} (singular!)
   -> Jump: Insert '{"name":"' directly (no forward pass needed)

   State S8: Possible transitions = {any string token} (multiple)
   -> Normal decode: Perform GPU forward pass -> "Alice"

   State S9: Possible transitions = {'","age":'} (singular!)
   -> Jump: Insert '","age":' directly

   State S10: Possible transitions = {0-9 tokens} (multiple)
   -> Normal decode: Perform GPU forward pass -> "30"

   State S11: Possible transitions = {'","city":"'} (singular!)
   -> Jump: Insert '","city":"' directly

   ...continues

The synergy with RadixAttention is also powerful. Even when the current request is terminated and a new request is queued during Jump-Forward execution, RadixAttention automatically reuses previous token KV caches, so no redundant computation occurs.

6.4 Performance Improvement

Performance improvements from structured generation with Compressed FSM vary by output format, but are particularly dramatic for JSON decoding.

Generation Format	Standard Constrained Decoding	SGLang Compressed FSM	Speedup
Simple JSON	1.0x	1.6x	1.6x
Nested JSON	1.0x	2.1x	2.1x
Complex Schema	1.0x	3.0x	3.0x
Regex Pattern	1.0x	1.4x	1.4x

The more complex the JSON schema, the more predictable tokens (fixed key names, delimiters, brackets) exist, maximizing the compression effect. According to Zheng et al.'s paper, a maximum 3x throughput improvement was achieved in JSON decoding benchmarks.

6.5 Structured Generation API Usage Example

Using structured output in SGLang is simple through the OpenAI-compatible API.

import openai

client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="EMPTY")

# Structured output via JSON Schema
response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[
        {"role": "user", "content": "Generate a user profile for Alice who is 30 years old."}
    ],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "user_profile",
            "schema": {
                "type": "object",
                "properties": {
                    "name": {"type": "string"},
                    "age": {"type": "integer"},
                    "city": {"type": "string"},
                    "hobbies": {
                        "type": "array",
                        "items": {"type": "string"}
                    }
                },
                "required": ["name", "age", "city", "hobbies"]
            }
        }
    },
    max_tokens=256
)

print(response.choices[0].message.content)
# Output: {"name": "Alice", "age": 30, "city": "San Francisco", "hobbies": ["reading", "hiking", "coding"]}

Regex-based constraints are also supported.

# Regex-based structured generation
response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[
        {"role": "user", "content": "Generate a valid email address for John."}
    ],
    extra_body={"regex": r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}"},
    max_tokens=64
)

7. Comprehensive LLM Inference Framework Comparison

7.1 SGLang vs vLLM vs TGI vs TensorRT-LLM

A comprehensive comparison of major LLM inference frameworks:

Feature	SGLang	vLLM	TGI v3	TensorRT-LLM
Developer	UC Berkeley / LMSYS	UC Berkeley	Hugging Face	NVIDIA
Language	Python	Python + C++	Rust + Python	C++ + Python
KV Cache Management	RadixAttention (Radix Tree)	PagedAttention (Block Table)	PagedAttention	PagedAttention
Inter-Request Cache Share	Automatic (Radix Tree)	Automatic Prefix Caching	Limited	Limited
Scheduler Overhead	Zero-Overhead	Medium	Low	Low
Structured Output	Compressed FSM (3x)	Outlines Integration	Supported	Limited
PD Disaggregation	Native via NIXL	Experimental	Not Supported	Supported
Hardware Support	NVIDIA, AMD, Intel, TPU	NVIDIA, AMD, TPU, CPU	NVIDIA, AMD	NVIDIA Only
Quantization	FP4/FP8/INT4/AWQ/GPTQ	FP8/INT4/AWQ/GPTQ	GPTQ/AWQ	FP8/INT4/INT8
MoE Expert Parallelism	Supported	Supported	Not Supported	Supported
Multi-LoRA Batching	Supported	Supported	Supported	Limited
Speculative Decoding	Supported	Supported	Supported	Supported
DSL/Frontend Language	SGLang Frontend	Not Supported	Not Supported	Not Supported
OpenAI-Compatible API	Full Support	Full Support	Supported	Partial
Codebase Size	~50K lines	~200K+ lines	~100K+ lines	~300K+ lines
Learning Curve	Low	Low	Medium	High
Best Use Case	Agents, RAG, Few-shot	General Serving	HF Ecosystem	Ultra-Low Latency

7.2 Selection Guide

Agent/RAG/Tool-Call Intensive Workloads: SGLang -- RadixAttention's automatic cache reuse is an overwhelming advantage
General LLM Serving (Diverse Hardware): vLLM -- Broadest hardware/model support
Hugging Face Ecosystem Integration: TGI -- Native integration with Inference Endpoints
Single-Request Ultra-Low Latency (NVIDIA Only): TensorRT-LLM -- Kernel-level optimization

8. SGLang Installation and Quick Start

8.1 Installation via pip

The simplest installation method.

# Requires Python 3.9+, CUDA 12.x recommended
pip install --upgrade pip
pip install "sglang[all]"

# Or faster installation using uv
pip install uv
uv pip install "sglang[all]"

Installing from source provides access to the latest features.

git clone https://github.com/sgl-project/sglang.git
cd sglang
pip install -e "python[all]"

8.2 Deployment via Docker

Docker images are recommended for production environments.

# Use official Docker image
docker run --gpus all \
  --shm-size 32g \
  -p 30000:30000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "HF_TOKEN=${HF_TOKEN}" \
  --ipc=host \
  lmsysorg/sglang:latest \
  python3 -m sglang.launch_server \
    --model-path meta-llama/Llama-3.1-8B-Instruct \
    --host 0.0.0.0 \
    --port 30000

Docker Compose can also be used.

# docker-compose.yml
version: '3.8'
services:
  sglang:
    image: lmsysorg/sglang:latest
    ports:
      - '30000:30000'
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
    environment:
      - HF_TOKEN=${HF_TOKEN}
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    ipc: host
    shm_size: '32g'
    command: >
      python3 -m sglang.launch_server
      --model-path meta-llama/Llama-3.1-8B-Instruct
      --host 0.0.0.0
      --port 30000

docker compose up -d

8.3 Running the Server

The basic command to launch the SGLang server directly:

# Basic server launch (single GPU)
python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 30000

# Or using the sglang serve command
sglang serve meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 30000

Once the server starts successfully, you can access the Swagger UI-based API documentation at http://localhost:30000/docs.

8.4 Server Health Check

# Server health check
curl http://localhost:30000/health

# Model information
curl http://localhost:30000/v1/models

# Server metrics
curl http://localhost:30000/get_server_info

9. API Usage and Code Examples

9.1 OpenAI-Compatible Chat Completions API

SGLang provides endpoints fully compatible with the OpenAI API, allowing you to use the existing OpenAI SDK as-is.

Using cURL:

curl http://127.0.0.1:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer EMPTY" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [
      {"role": "system", "content": "You are a helpful AI assistant."},
      {"role": "user", "content": "Explain the difference between Kubernetes Pods and Deployments."}
    ],
    "max_tokens": 512,
    "temperature": 0.7,
    "top_p": 0.9
  }'

Using the Python OpenAI SDK:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"  # SGLang does not require authentication by default
)

# Standard Chat Completion
response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[
        {"role": "system", "content": "You are a cloud-native expert."},
        {"role": "user", "content": "Compare the pros and cons of Helm charts and Kustomize."}
    ],
    max_tokens=1024,
    temperature=0.7
)

print(response.choices[0].message.content)

Streaming Responses:

# Streaming response
stream = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[
        {"role": "user", "content": "Write a simple web server in Python."}
    ],
    max_tokens=1024,
    stream=True
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

9.2 Text Completions API

Plain text completions (non-chat format) are also supported.

curl http://127.0.0.1:30000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "prompt": "The capital of France is",
    "max_tokens": 32,
    "temperature": 0
  }'

9.3 Batch Request Processing

A batch API for efficiently processing large volumes of requests is also available.

import requests
import json

# Native SGLang batch endpoint
batch_requests = [
    {
        "text": "Translate to Korean: Hello, how are you?",
        "sampling_params": {"max_new_tokens": 64, "temperature": 0.3}
    },
    {
        "text": "Translate to Korean: The weather is nice today.",
        "sampling_params": {"max_new_tokens": 64, "temperature": 0.3}
    },
    {
        "text": "Translate to Korean: I love programming.",
        "sampling_params": {"max_new_tokens": 64, "temperature": 0.3}
    }
]

# Maximize throughput with async requests
import asyncio
import aiohttp

async def send_requests(requests_data):
    async with aiohttp.ClientSession() as session:
        tasks = []
        for req in requests_data:
            task = session.post(
                "http://localhost:30000/generate",
                json=req
            )
            tasks.append(task)
        responses = await asyncio.gather(*tasks)
        results = [await r.json() for r in responses]
        return results

results = asyncio.run(send_requests(batch_requests))
for r in results:
    print(r["text"])

9.4 SGLang Frontend (DSL) Usage

Using SGLang's unique frontend language, you can write complex LLM programs in a Pythonic way.

import sglang as sgl

# Define SGLang frontend function
@sgl.function
def multi_turn_qa(s, question1, question2):
    s += sgl.system("You are a helpful AI assistant specialized in cloud computing.")
    s += sgl.user(question1)
    s += sgl.assistant(sgl.gen("answer1", max_tokens=256))
    s += sgl.user(question2)
    s += sgl.assistant(sgl.gen("answer2", max_tokens=256))

# Set runtime endpoint
sgl.set_default_backend(sgl.RuntimeEndpoint("http://localhost:30000"))

# Execute
state = multi_turn_qa.run(
    question1="How does HPA work in Kubernetes?",
    question2="What is the difference from VPA?"
)

print("Answer 1:", state["answer1"])
print("Answer 2:", state["answer2"])

Parallel Generation (Fork-Join):

@sgl.function
def parallel_analysis(s, topic):
    s += sgl.system("You are a technology analyst.")
    s += sgl.user(f"Analyze '{topic}' from three perspectives.")

    # Fork: Generate 3 analyses in parallel
    forks = s.fork(3)

    forks[0] += sgl.user("Technical perspective:")
    forks[0] += sgl.assistant(sgl.gen("technical", max_tokens=200))

    forks[1] += sgl.user("Business perspective:")
    forks[1] += sgl.assistant(sgl.gen("business", max_tokens=200))

    forks[2] += sgl.user("User experience perspective:")
    forks[2] += sgl.assistant(sgl.gen("ux", max_tokens=200))

    # Join: Collect all results
    forks.join()

    # Synthesize
    s += sgl.user("Summarize the three analyses above:")
    s += sgl.assistant(sgl.gen("summary", max_tokens=300))

state = parallel_analysis.run(topic="SGLang inference engine")
print("Technical:", state["technical"])
print("Business:", state["business"])
print("UX:", state["ux"])
print("Summary:", state["summary"])

Select-Based Classification:

@sgl.function
def classify_sentiment(s, text):
    s += sgl.user(f"Classify the sentiment of the following text: '{text}'")
    s += sgl.assistant(
        "The sentiment is " + sgl.select("label", ["positive", "negative", "neutral"])
    )

state = classify_sentiment.run(text="SGLang is amazingly fast!")
print("Sentiment:", state["label"])  # "positive"

10. Key Configuration Parameter Guide

10.1 Server Launch Parameters

Key configuration parameters for the SGLang server:

python -m sglang.launch_server \
  --model-path <model_name_or_path>    # HuggingFace model name or local path
  --host 0.0.0.0                        # Binding host
  --port 30000                          # Service port
  --tp-size 4                           # Tensor Parallelism size
  --dp-size 2                           # Data Parallelism size
  --pp-size 2                           # Pipeline Parallelism size
  --mem-fraction-static 0.88            # KV Cache fraction of GPU memory (default: 0.88)
  --max-running-requests 128            # Max concurrent requests
  --max-total-tokens 131072             # Max total tokens
  --context-length 32768                # Context length
  --chunked-prefill-size 8192           # Chunked Prefill chunk size
  --schedule-policy lpm                 # Scheduling policy (lpm / fcfs / random)
  --quantization fp8                    # Quantization method (fp8 / int4 / awq / gptq)
  --dtype auto                          # Data type (auto / float16 / bfloat16)
  --trust-remote-code                   # Allow remote code trust
  --chat-template auto                  # Chat template (auto or custom path)
  --log-level info                      # Log level
  --enable-metrics                      # Enable Prometheus metrics
  --api-key "your-secret-key"           # Enable API key authentication

10.2 Detailed Key Parameters

Memory Management:

# Allocate 88% of GPU memory to KV Cache (default)
--mem-fraction-static 0.88

# Reduce when facing memory shortage with large models
--mem-fraction-static 0.80

# Explicitly set max KV Cache token count
--max-total-tokens 65536

Scheduling Policy:

# LPM (Longest Prefix Match): Maximize RadixAttention cache hit rate (default)
--schedule-policy lpm

# FCFS (First Come First Served): Simple FIFO
--schedule-policy fcfs

# Random: Random selection
--schedule-policy random

Quantization Settings:

# FP8 quantization (requires Hopper+ GPU)
--quantization fp8

# NVFP4 quantization (requires Blackwell GPU)
--quantization fp4

# INT4 AWQ quantization
--quantization awq

# INT4 GPTQ quantization
--quantization gptq

10.3 Environment Variables

# CUDA device selection
export CUDA_VISIBLE_DEVICES=0,1,2,3

# HuggingFace token (for gated models)
export HF_TOKEN="hf_your_token_here"

# NCCL settings (multi GPU)
export NCCL_P2P_DISABLE=0
export NCCL_IB_DISABLE=0

# SGLang log level
export SGLANG_LOG_LEVEL=info

# FlashInfer workspace size
export SGLANG_FLASHINFER_WORKSPACE_SIZE=2147483648  # 2GB

11. Deployment Guide

11.1 Single GPU Deployment

The most basic deployment form.

# Deploy Llama-3.1-8B to a single GPU
python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 30000 \
  --mem-fraction-static 0.88

# Deploy a larger model to a single GPU with quantization
python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-70B-Instruct \
  --quantization awq \
  --host 0.0.0.0 \
  --port 30000

11.2 Multi-GPU Deployment (Tensor Parallelism)

Used when the model does not fit in a single GPU's memory.

# 4-GPU Tensor Parallelism (70B model)
python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-70B-Instruct \
  --tp-size 4 \
  --host 0.0.0.0 \
  --port 30000

# 2-GPU TP + 2-GPU DP = 4 GPU (8B model, maximize throughput)
python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --tp-size 2 \
  --dp-size 2 \
  --host 0.0.0.0 \
  --port 30000

11.3 Multi-Node Deployment

Used when deploying large models across multiple servers.

# Node 0 (Master)
python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-405B-Instruct \
  --tp-size 16 \
  --nnodes 2 \
  --node-rank 0 \
  --dist-init-addr node0-ip:5000 \
  --host 0.0.0.0 \
  --port 30000

# Node 1 (Worker)
python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-405B-Instruct \
  --tp-size 16 \
  --nnodes 2 \
  --node-rank 1 \
  --dist-init-addr node0-ip:5000

11.4 PD Disaggregation Deployment

Deploying with separated Prefill and Decode.

# Launch Prefill server
python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-70B-Instruct \
  --tp-size 4 \
  --disaggregation-mode prefill \
  --disaggregation-transfer-backend nixl \
  --host 0.0.0.0 \
  --port 30001

# Launch Decode server
python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-70B-Instruct \
  --tp-size 4 \
  --disaggregation-mode decode \
  --disaggregation-transfer-backend nixl \
  --host 0.0.0.0 \
  --port 30002

11.5 Kubernetes Deployment (SkyPilot)

Deployment via Kubernetes in cloud environments is also supported.

# sglang-skypilot.yaml
resources:
  cloud: aws # or gcp, azure
  accelerators: A100:4
  memory: 128+

envs:
  HF_TOKEN: <your-hf-token>
  MODEL_NAME: meta-llama/Llama-3.1-70B-Instruct

setup: |
  pip install "sglang[all]"

run: |
  python -m sglang.launch_server \
    --model-path ${MODEL_NAME} \
    --tp-size 4 \
    --host 0.0.0.0 \
    --port 30000

# Create cluster and deploy with SkyPilot
pip install skypilot-nightly
sky launch -c sglang-cluster --env HF_TOKEN sglang-skypilot.yaml

11.6 Multi-LoRA Deployment

Serving multiple LoRA adapters simultaneously.

# Serve base model + multiple LoRA adapters simultaneously
python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --lora-paths \
    korean-chat=/path/to/korean-lora \
    code-gen=/path/to/code-lora \
    medical=/path/to/medical-lora \
  --max-loras-per-batch 4 \
  --host 0.0.0.0 \
  --port 30000

You can specify the LoRA adapter in requests.

curl http://127.0.0.1:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "korean-chat",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 128
  }'

12. SGLang 2026 Roadmap and Ecosystem

12.1 2026 Q1 Roadmap Highlights

The SGLang project continues to evolve rapidly in 2026. Key items on the 2026 Q1 roadmap:

Area	Progress
Blackwell Optimization	GB300/B300 support, NVFP4 MoE kernel integration
TPU Support	Native TPU execution via SGLang-JAX backend
Diffusion Models	Image/video generation acceleration with SGLang Diffusion
Pipeline Parallelism	Chunked PP for Million-Token context support
Day-0 Model Support	MiMo-V2-Flash, Nemotron 3 Nano, Mistral Large 3
NVIDIA Dynamo Integration	Native Disaggregated Serving with NVIDIA Dynamo

12.2 Hardware Support Expansion

SGLang is expanding support beyond NVIDIA GPUs to various hardware.

Hardware	Support Status	Key Features
NVIDIA H100/H200	Full Support	FP8, FlashInfer
NVIDIA B200/GB200	Full Support	NVFP4, TMA, NVLink 900GB/s
NVIDIA GB300/B300	In Progress	Next-gen Blackwell
NVIDIA RTX PRO 6000	Supported	Blackwell Server Edition
NVIDIA Jetson Thor	In Progress	Edge Inference
AMD MI300X	Supported	ROCm, FP8
Intel Xeon AMX	Experimental	CPU Inference
Google TPU	Supported	SGLang-JAX Backend

12.3 The Shift in 'Inference Economics'

The significance of next-generation inference engines represented by SGLang goes beyond simple speed improvements. They drive a fundamental change in 'Inference Economics'.

Cost Efficiency Formula:

Inference Cost = GPU Cost per Hour / Throughput (tok/s)

H100 basis:
  vLLM:  $3.50/hr / 12,553 tok/s = $0.000279 / 1K tokens
  SGLang: $3.50/hr / 16,215 tok/s = $0.000216 / 1K tokens

-> SGLang achieves 22.6% cost reduction on the same GPU
-> Agent scenarios (including cache hit rate): up to 60-80% cost reduction

For organizations operating large-scale LLM services, this difference translates to savings of tens of thousands to hundreds of thousands of dollars per month. Breaking through hardware's physical limitations with software intelligence is the essence of SGLang.

13. Conclusion: Breaking Hardware Limits with Software Intelligence

The 5 reasons SGLang is changing the LLM inference landscape, summarized:

#	Innovation	Core Mechanism	Achievement
1	RadixAttention	Radix Tree-based KV cache sharing	Up to 5x improvement in agent workflows
2	Hyper-Specialized Design	Minimized abstraction, direct TMA integration	29% throughput advantage over vLLM
3	Zero-Overhead Scheduler	Async CPU-GPU pipelining	C++-level performance in 4,000 lines of Python
4	PD Disaggregation	NIXL-based Prefill-Decode separation	Independent scaling, 3.8-4.8x on GB200
5	Compressed FSM	Singular transition compression, Jump-Forward	Up to 3x speed for JSON decoding

These 5 innovations are individually powerful, but SGLang's true strength lies in the synergy they generate within an integrated architecture. RadixAttention's cache reuse enables Compressed FSM's Jump-Forward; the Zero-Overhead Scheduler hides RadixAttention's tree traversal overhead behind GPU computation; and PD Disaggregation scales all these optimizations to large-scale distributed environments.

In an era where LLM inference costs determine the sustainability of AI services, SGLang is making the proposition "create more value from the same GPU" a reality. Supporting NVIDIA Blackwell's NVFP4, AMD MI300X, and Intel Xeon AMX while pursuing hardware-neutral optimization, SGLang's trajectory shows that inference engines are evolving from simple model executors into core operating systems of AI infrastructure.

If vLLM led the democratization of LLM serving with PagedAttention, SGLang is opening the next chapter of LLM serving with RadixAttention and full-system co-design.

14. References

Zheng, L., Yin, L., Xie, Z., et al. "SGLang: Efficient Execution of Structured Language Model Programs." NeurIPS 2024. arXiv:2312.07104
SGLang GitHub Repository. https://github.com/sgl-project/sglang
SGLang Official Documentation. https://docs.sglang.ai/
LMSYS Blog - "Fast and Expressive LLM Inference with RadixAttention and SGLang." https://lmsys.org/blog/2024-01-17-sglang/
LMSYS Blog - "SGLang v0.4: Zero-Overhead Batch Scheduler, Cache-Aware Load Balancer, Faster Structured Outputs." https://lmsys.org/blog/2024-12-04-sglang-v0-4/
LMSYS Blog - "Fast JSON Decoding for Local LLMs with Compressed Finite State Machine." https://lmsys.org/blog/2024-02-05-compressed-fsm/
LMSYS Blog - "Deploying DeepSeek on GB200 NVL72 with PD and Large Scale EP." https://lmsys.org/blog/2025-09-25-gb200-part-2/
LMSYS Blog - "Pipeline Parallelism in SGLang: Scaling to Million-Token Contexts and Beyond." https://lmsys.org/blog/2026-01-15-chunked-pipeline/
SGLang PD Disaggregation Documentation. https://docs.sglang.ai/advanced_features/pd_disaggregation.html
SGLang NVIDIA Collaboration Roadmap 2026 Q1. https://github.com/sgl-project/sglang/issues/17130
SGLang Development Roadmap 2026 Q1. https://github.com/sgl-project/sglang/issues/12780
Kwon, W., et al. "Efficient Memory Management for Large Language Model Serving with PagedAttention." SOSP 2023. (vLLM paper)
NVIDIA NIXL Documentation - "Low Latency Point-to-Point Inference Transfer Library."
Clarifai Blog - "Comparing SGLang, vLLM, and TensorRT-LLM with GPT-OSS-120B." https://www.clarifai.com/blog/comparing-sglang-vllm-and-tensorrt-llm-with-gpt-oss-120b
RunPod Blog - "When to Choose SGLang Over vLLM: Multi-Turn Conversations and KV Cache Reuse." https://www.runpod.io/blog/sglang-vs-vllm-kv-cache