Skip to content
←

필사 모드: PD Disaggregation Doesn't Increase Throughput — What Prefill/Decode Disaggregation Actually Buys You

English
0%
정확도 0%
💡 왼쪽 원문을 읽으면서 오른쪽에 따라 써보세요. Tab 키로 힌트를 받을 수 있습니다.

Introduction — Two Jobs With Opposite Personalities, Living on One GPU

It's common knowledge by now that LLM inference splits into two stages: prefill, which pushes the entire prompt through a single forward pass to build the KV cache, and decode, which pulls out tokens one at a time. These are the same two stages that come up every time we talk about the KV cache.

What matters is that the two have opposite personalities. In the SGLang documentation's own words, prefill is a computation-intensive job that processes the entire input sequence, while decode is a memory-intensive job that manages the KV cache. Prefill burns the GPU's compute units, while decode is bound by memory bandwidth because it has to re-read the entire model weights and KV cache for every single token.

What happens when you put two jobs with opposite personalities on the same GPU and run them under the same scheduler? And does splitting them really make things better? This post is about how far that "split" — prefill/decode disaggregation (PD disaggregation) — has come as of 2026, and why you shouldn't take the multiplier numbers floating around the internet at face value.

Why Putting Both on One GPU Is a Problem — Interference

The SGLang documentation pins down exactly two problems that arise in a unified engine.

First, Prefill Interruption. In the documentation's own words, "incoming prefill batches frequently interrupt ongoing decode batches, causing significant delays in token generation." Picture it and it's obvious: a user is streaming tokens one by one, and meanwhile an 8,000-token prompt arrives from someone else. That prefill occupies the entire GPU for a while, and the streaming user's next token comes out that much later. From the user's side, it's the moment the text flow suddenly stops.

Second, DP Attention Imbalance. In data-parallel attention, if one DP worker processes a prefill batch while another simultaneously processes a decode batch, decode latency grows.

NVIDIA's TensorRT-LLM technical blog makes the same diagnosis — in unified serving, "context processing delays token generation," producing interference that degrades interactivity. And the blog sums up the essence of the problem in one line: "optimizing for one metric, like TTFT, frequently comes at the cost of another, like TPOT."

This is the crux. On a single GPU, prefill and decode are two conflicting goals competing for the same resource. Grow the batch and throughput rises but tail latency falls apart; slice prefill into small pieces (chunked prefill) and interference drops but prefill efficiency suffers. Trying to satisfy both goals with a single scheduler is structurally a compromise, full stop.

So You Disaggregate — The Structure of PD Disaggregation

The idea is simple: since the two have different personalities, give them separate hardware.

By the vLLM documentation's definition, disaggregated prefill runs the prefill and decode stages on separate vLLM instances and moves the KV cache between them via a connector. An actual deployment looks roughly like this.

                     [ Proxy / Router ]
                        |          |
            Request --->|          |
                        v          v
              [ Prefill Instance ]  [ Decode Instance ]
               GPU 0-3                GPU 4-7
               Prompt -> KV           KV -> Token stream
                        |          ^
                        |  KV Cache |
                        +--- RDMA --+
                          (Connector)

You need three pieces.

  1. Prefill instance — takes the prompt and builds the KV cache.
  2. Decode instance — takes that KV cache and pulls out tokens.
  3. Connector and proxy — carries the KV cache across and coordinates which side a request goes to.

Now you can apply a separate parallelization strategy to each stage — low TP for prefill, high TP for decode, or the reverse. This is exactly the first benefit the vLLM documentation lists for this feature: "tuning TTFT and ITL separately." The second is "controlling tail ITL." Since a prefill job can no longer interrupt decode, tokens come out of the decode GPU at a steady rhythm.

In vLLM, this feature is still marked experimental. In the documentation's own words, "this feature is experimental and subject to change."

How to Read the Numbers — Goodput, Not Throughput

This is the heart of the post.

Search for PD disaggregation and numbers like "2x" and "7x" pour out. Yet the official vLLM documentation has this sentence pinned down in capital letters.

"Disaggregated prefill DOES NOT improve throughput."

It looks like a contradiction. But both are true. Once you understand why, it changes how you see PD disaggregation completely.

Think about it and it's obvious. PD disaggregation doesn't conjure up more GPUs. Splitting 8 GPUs into 4 for prefill and 4 for decode can't possibly increase total compute. If anything, moving the KV cache over the network is new work, so in terms of pure total work, it actually increases. The TensorRT-LLM blog admits as much — "disaggregation incurs the overhead of transferring KV cache blocks."

So what are those multipliers actually measuring? The answer is goodput.

The concept traces back to DistServe (Zhong et al., OSDI 2024), the original PD disaggregation paper — the title itself says "Goodput-optimized." Goodput is defined roughly as the maximum request rate the system can sustain while satisfying both TTFT and TPOT constraints. Where plain throughput is "how many did it spit out per second," goodput is "how many did it spit out per second at the quality it promised."

How big this difference is comes through clearly in the MORI-IO connector blog post that landed in vLLM. That post defines goodput this way.

"Goodput = maximum request rate (req/s) such that requests satisfy both TTFT < T_ttft and ITL < T_itl."

In other words, a request that misses its SLO doesn't get counted. In a throughput calculation, a TTFT that took 3 seconds still counts as one request; in a goodput calculation, it counts as zero — because the user has probably already closed the tab.

So to sum up:

  • Throughput (pure tokens/s, latency ignored): PD disaggregation does not improve it. If anything, transfer overhead makes it slightly worse. This is what the vLLM documentation is talking about.
  • Goodput (SLO-compliant throughput): PD disaggregation improves it substantially. This is what the vendor blogs are measuring.

Same system, different ruler. Which is exactly why, whenever you see a multiplier on the internet, the first thing to check is which ruler was used.

Taking the Multipliers Apart

Let's re-read the published numbers through this lens. Everything below is a vendor/author self-reported measurement, and I've noted the conditions for each — the conditions are the meaning of the number.

DistServe (Zhong et al., OSDI 2024, author self-reported). The paper's headline figures are "7.4x more requests" or "12.6x tighter SLO." But a condition immediately follows — "while staying within latency constraints for over 90% of requests." This isn't a throughput multiplier; it's a request-rate multiplier with the SLO-attainment rate held fixed. And the paper itself says the multiplier varies by model, application, and latency requirement.

TensorRT-LLM (NVIDIA self-reported, GB200). These numbers are especially instructive. On DeepSeek R1 with 4,400 input / 1,200 output tokens: 1.4-1.8x without MTP, 1.6-2.5x with MTP on. At 8,192 input / 256 output: up to 1.73x on 4 GPUs, 2x on 8 GPUs. And the decisive entry — at 4,096 input / 1,024 output, "1.7x improvement at the 50 tokens/s/user point." Read that last phrase again: it's a throughput multiplier with a latency target of 50 tokens/s per user held fixed. Relax the latency constraint and this multiplier does not stay put. For Qwen 3 at 8,192 input / 1,024 output, a wide range of 1.7-6.11x was reported — and a range that wide is itself evidence that "the conditions are everything."

And NVIDIA appends an honest footnote: there's a 0-25% gap between the theoretical "rate-matched" analysis and actual end-to-end benchmarks, attributed to "idealized assumptions" and "KV cache transfer overhead." In effect, the vendor is telling you, in its own words, not to take the theoretical multiplier at face value.

llm-d on AWS (AWS self-reported, March 16, 2026). GPT-OSS model, ml.p6-b200.48xlarge instance, 4 prefill pods (TP1) and 1 decode pod (TP4), 1,024 tokens each for input and output, concurrency up to 128. The result: "as concurrency increases, tokens per second increase by up to 70% over standard vLLM deployment." Here too the condition "as concurrency increases" is attached — meaning there's no gain at low load. And the AWS authors add: "but it's not for every workload. Try it on larger models, longer input sequences, and sparse MoE architectures."

Notice the pattern. Every single number has a fixed latency target or load condition attached. That's not a coincidence — it's because that's the product PD disaggregation is selling.

The Plumbing That Moves KV — The Real Engineering Is Here

The idea behind PD disaggregation fits in one paragraph. The hard part is the plumbing. The KV cache built by the prefill GPU has to move to the decode GPU, and it isn't small — and this transfer lands directly on top of TTFT.

So the real competition in 2026 is playing out at the transfer layer.

NIXL (NVIDIA Inference Xfer Library) is currently the most widely used axis. In the repository's own description, it's "designed to accelerate point-to-point communication in AI inference frameworks such as NVIDIA Dynamo... while providing a consistent abstraction across diverse memory (CPU, GPU) and storage (file, block, object store) tiers via a modular plug-in architecture." It's Apache 2.0-licensed, with backend plugins for UCX, POSIX, OBJ, AZURE_BLOB, HF3FS, MOONCAKE, GUSLI, UCCL, GDS, GPUNETIO, LIBFABRIC, and more. In short, it's an abstraction layer that lets the KV cache move over InfiniBand, NVMe, or even S3.

Mooncake is SGLang's default transfer engine, and SGLang supports three backends: Mooncake, NIXL, and ASCEND. The Mooncake path requires specifying the InfiniBand device via --disaggregation-ib-device.

On the vLLM side, the current connector list is: NixlConnector, LMCacheConnectorV1, MooncakeConnector, MoRIIOConnector (ROCm-only), MultiConnector, OffloadingConnector, FlexKVConnectorV1, and the example ExampleConnector. The internal abstraction has three pieces — the Connector (KV lookup between producer and consumer), the LookupBuffer (insert is non-blocking, drop_select is blocking), and the Pipe (a unidirectional FIFO).

Don't miss the point where the plumbing turns back into a requirement. An RDMA fabric is effectively a precondition. That's why AWS talks about EFA and the libfabric plugin in the same breath as running llm-d, and why SGLang requires specifying an InfiniBand device. Layer this design on top of plain Ethernet, and you hand back the ITL you saved as transfer latency.

April 2026, MORI-IO — One Concrete Measurement

Let's bring the abstract discussion down to one concrete measurement. On April 7, 2026, the vLLM blog published a post on the MORI-IO connector, written by the AMD and Embedded LLM teams, with the measurement carried out on March 12, 2026. Let me flag up front that this is an AMD self-reported measurement.

MORI-IO is an RDMA-based KV cache connector built on top of the MORI (Modular RDMA Interface) framework and contributed to vLLM. Think of it as filling the role NIXL plays, but on the AMD GPU (ROCm) side.

The measurement conditions:

  • Hardware: 8x AMD Instinct MI300X (192GB each), 2x AMD EPYC 9654 96-core sockets
  • Software: Ubuntu 22.04 LTS, ROCm 7.0.51831, vLLM 0.16.0rc1.dev1
  • Model: Qwen/Qwen3-235B-A22B-FP8
  • Workload: 2,000 input tokens, 1,000 output tokens, 100 total requests, request rate 0.5-10 req/s
  • SLO: TTFT under 1 second, ITL under 50ms per token
  • Configuration: prefill on GPU 0-3, decode on GPU 4-7, plus a proxy server

Results reported at the 8 req/s request-rate point:

ConfigurationSLO-satisfying requestsRelative goodput
Standard (1x TP8)26/1000.9x
Standard (2x TP4)30/1001x (baseline)
MORI-IO Read (1P+1D)70/1002.4x
MORI-IO Write (1P+1D)73/1002.5x

The most important thing in this table isn't the multiplier — it's why the requests failed. Quoting the blog post directly:

"Both disaggregated modes completely eliminate ITL violations. The remaining failures are TTFT overruns that occur as the request rate increases."

This sums up the true nature of PD disaggregation in one line. ITL violations went to zero. That's exactly what you'd expect, since prefill can no longer intrude on the decode GPU. But the failures didn't disappear — they moved over to TTFT. The bottleneck wasn't eliminated, it was relocated, and the trade just happened to be a profitable one.

The difference between Read mode and Write mode is an extension of the same story.

  • Read mode: decode pulls the KV from prefill over RDMA. Per the blog's description, "TTFT increases by at least one full prefill forward pass (proxy serialization) plus the RDMA transfer time."
  • Write mode: prefill pushes the KV to decode while it's still computing. "TTFT increases only by the RDMA transfer time, and because this transfer overlaps with prefill compute, the net penalty is smaller."

That's why Write scores 73 and Read scores 70 — per the blog, because "concurrent proxy dispatch lowers TTFT." In other words, the 3-point gap is ultimately about the size of the TTFT penalty.

And one footnote is quietly attached: "prefix caching disabled, as required by the MORI-IO connector." This holds for both disaggregated configurations. This is not a small footnote — I'll cover it in the next section.

It's Not Free — The Honest Costs

Now for the bill.

TTFT gets worse. Exactly as we saw above. The time it takes the KV cache to cross the network lands on top of TTFT, and Read mode adds proxy serialization on top of that. Remember that in the MORI-IO measurement, all 27-30 requests that missed SLO were TTFT overruns. PD disaggregation is a trade that converts an ITL problem into a TTFT problem. If your bottleneck is TTFT, this trade is a net loss.

It can conflict with prefix caching. In the MORI-IO measurement, prefix caching was turned off as a connector requirement. This hurts because prefix caching, which wipes out the prefill cost entirely for long system prompts or multi-turn conversations, is the single biggest optimization in most real-world workloads. If your traffic is chatbot traffic sharing a long common prefix, whether the 2.5x goodput you get from turning prefix caching off actually beats unified serving with prefix caching on is not obvious at all. This is something you have to measure directly with your own traffic. (This varies by connector and version, so check the current constraints of the combination you intend to use.)

It requires an RDMA fabric. InfiniBand or RoCE — EFA if you're on AWS. This is a hardware-procurement and network-design problem, not a matter of flipping one flag.

It needs multiple GPUs. The bare minimum is 1 for prefill + 1 for decode, and real-world configurations run larger — the MORI-IO measurement split 8 GPUs 4:4, and the AWS case used 4 prefill pods against 1 decode pod. This isn't a topic worth discussing for a service running on a couple of GPUs.

Operational complexity goes up. One instance becomes two instance types plus a proxy/router plus a transfer layer. You have to tune the prefill-to-decode ratio for your workload (this is the exact point the AWS post raises), and failure modes multiply accordingly. Even profiling gets more cumbersome — the SGLang documentation notes that you need to profile prefill workers and decode workers separately.

It's sensitive to model architecture. SGLang's staging-buffer optimization is for non-MLA models (GQA, MHA), and the documentation explicitly states "MLA models (e.g., DeepSeek-V2/V3) should not enable this flag." TensorRT-LLM says this approach benefits "long input sequences and medium-length outputs" workloads the most. With a short prompt and a long output, prefill was never the bottleneck to begin with, so there's less reason to split it out.

It's still experimental. At least in vLLM.

So, Should You Use It?

The decision criterion turns out to be surprisingly clear-cut. The MORI-IO blog gives you the answer directly.

"If ITL p99 exceeds SLO under production load: disaggregate — this is the primary use case."

"Low request rate with short prompts: standard serving is sufficient."

Layering everything covered so far on top of this gives the following.

When it earns its keep

  • Tail ITL is the actual cause of SLO violations. You have complaints about streaming stalling, and measuring p99 ITL points the finger at prefill interference.
  • It's a long-input, medium-length-output workload. (The condition TensorRT-LLM explicitly names.)
  • Load is genuinely high. Remember that the gains in the AWS measurement showed up "as concurrency increases."
  • You already have an RDMA fabric and GPU headroom.
  • Large models, especially sparse MoE. (The condition AWS recommends.)

When it's overkill or a net loss

  • Your bottleneck is TTFT. Disaggregation makes TTFT worse.
  • Prefix caching is your primary optimization — especially if you're a chatbot sharing a long common system prompt.
  • Load is low, or prompts are short.
  • You have few GPUs, or no RDMA.
  • You just want more throughput, full stop. In this case disaggregation isn't the answer — exactly what the vLLM documentation says, in capital letters.

That last item is the most common misunderstanding. If you came to evaluate PD disaggregation motivated by "I want to cut GPU cost," you're pointed the wrong direction. That's a problem quantization or batch tuning solves. What PD disaggregation sells is predictability, not capacity.

Closing

To sum up: prefill is compute-intensive and decode is memory-intensive, and putting both on one GPU means prefill cuts decode off. PD disaggregation splits the two onto separate GPUs, eliminating that interference at the source and letting you tune each stage's parallelization independently. The complete disappearance of ITL violations in the MORI-IO measurement is the evidence.

In exchange, you have to move the KV cache over the network, that time lands on TTFT, it demands an RDMA fabric, multiple GPUs, and more operational complexity, and it can collide with existing optimizations like prefix caching. And pure throughput — in the vLLM documentation's own words — does not increase.

So PD disaggregation isn't "faster serving" — it's "serving that keeps its promises." It's not a technique for spitting out more from the same GPUs; it's a technique for making sure what you do spit out lands within SLO. The moment you blur that distinction, you fall for the multiplier numbers on the internet — because every one of them is goodput measured with a latency target held fixed.

So the order of operations is this: measure your p99 ITL first. If it's exceeding SLO and the culprit is prefill interference, that's when you reach for disaggregation. Reaching for disaggregation before you've even measured is treating a disease you don't have with a very expensive drug.

References

현재 단락 (1/109)

It's common knowledge by now that LLM inference splits into two stages: **prefill**, which pushes th...

작성 글자: 0자원문 글자: 17,920자작성 단락: 0/109