The Four Kinds of Multi-GPU Parallelism — What You Split and What You Communicate

• Introduction — Why Eight 80GB Cards Still Can't Fit a 7B Model
• The Memory Ledger — 16 Bytes per Parameter, and the Larger Activation Bill
  • State Memory: 16 Bytes per Parameter
  • Activations Are What Actually Blow Up the Card
• Data Parallelism and ZeRO — Which Terms of the Ledger Get Divided
• Tensor Parallelism — Split the Matrix, Communicate on Every Layer
• Pipeline Parallelism — Cheapest Communication, Most Expensive Bubble
• Sequence and Context Parallelism — the Fourth Axis
• Picking a Combination — Reading the Table by Size and GPU Count
• Common Failure Modes
• Closing — Choosing a Parallelism Is Spending a Bandwidth Budget

Introduction — Why Eight 80GB Cards Still Can't Fit a 7B Model

"A 7B model is 14GB in bf16, so it'll fit comfortably on one 80GB card" — that math only holds for inference. Try to train the same model and a single card won't even get you started; even eight of them can still blow up depending on your configuration.

The reason is that training has to hold three more chunks of memory beyond the parameters: gradients, optimizer state, and activations. Which of these four chunks you split, and how, is the entirety of a parallelism strategy — and what the GPUs have to exchange as the price of that split is the entirety of its performance.

This post organizes the four parallelisms along those two axes: what gets split and how much gets communicated. Every number here is presented so you can verify it yourself. Inference-side memory math was already covered in LLM Inference VRAM Math, so this post looks at training only.

The Memory Ledger — 16 Bytes per Parameter, and the Larger Activation Bill

State Memory: 16 Bytes per Parameter

In mixed-precision training with an Adam-family optimizer, the bytes a single parameter occupies break down as follows.

This 16-byte figure matches the ledger set out in the ZeRO paper. For a 7B model that's 112GB; for 70B it's 1,120GB. Which is exactly why a single 80GB card can't even train a 7B model.

Change the optimizer and this ledger changes with it. With SGD with momentum, the fp32 state is just master weights plus momentum, so it drops to 12 bytes per parameter; 8-bit Adam or Adafactor shrink it further still. That said, swapping optimizers at pretraining scale is a convergence problem, not a memory problem, so lower-precision optimizers are mostly used for fine-tuning.

Activations Are What Actually Blow Up the Card

The ledger above leaves activations out. Yet activations are usually the actual culprit behind OOMs during pretraining. Korthikanti et al.'s Reducing Activation Recomputation in Large Transformer Models works out that, with no parallelism, the activation memory of a single transformer layer is s·b·h·(34 + 5as/h) bytes, in terms of sequence length s, batch b, hidden dimension h, and attention head count a.

def act_gb(s: int, b: int, h: int, a: int, layers: int, mode: str) -> float:
    sbh = s * b * h
    per_layer = {
        "none": sbh * (34 + 5 * a * s / h),   # store everything
        "selective": sbh * 34,                # recompute only the attention matrix
        "full": sbh * 2,                      # store only the layer input, recompute everything
    }[mode]
    return per_layer * layers / 1e9


s, b, h, a, L = 4096, 1, 4096, 32, 32   # 7B-scale config, one sample
for mode in ("none", "selective", "full"):
    print(f"{mode:10} {act_gb(s, b, h, a, L, mode):7.2f} GB / sample")
# none        104.15 GB / sample
# selective    18.25 GB / sample
# full          1.07 GB / sample

That's 104GB for a single sample. Set the micro-batch to just 4 and it's 416GB. Recompute everything and it falls to 1.07GB, at the cost of roughly a third more compute from the extra forward pass.

Look at the 5as/h term riding alongside the 34. It scales with the square of sequence length. Take s from 4K to 32K and the 34 term grows 8x, but the 5as/h term grows 64x. That is exactly where long-context training suddenly gets hard, and exactly why context parallelism becomes necessary.

Data Parallelism and ZeRO — Which Terms of the Ledger Get Divided

The simplest parallelism, DDP, has every GPU hold a full copy of the 16 bytes above. Only the batch gets split; gradients are all-reduced and every GPU applies the same update. It's the simplest to implement, but it saves no memory at all.

ZeRO strips this duplication away in stages.

def per_gpu_state_gb(params_b: float, world: int, stage: int) -> float:
    """State memory held by a single GPU under Adam mixed precision (GB, base 10^9).
    stage 0 = DDP, 1 = optimizer sharded, 2 = +gradient, 3 = +parameter
    Activations are not included."""
    p = params_b * 1e9
    param, grad, opt = 2 * p, 2 * p, 12 * p
    if stage >= 1:
        opt /= world
    if stage >= 2:
        grad /= world
    if stage >= 3:
        param /= world
    return (param + grad + opt) / 1e9


for stage in (0, 1, 2, 3):
    row = [round(per_gpu_state_gb(n, 8, stage), 1) for n in (7, 13, 70)]
    print(f"stage {stage}  world=8  7B/13B/70B -> {row} GB")
# stage 0  world=8  7B/13B/70B -> [112.0, 208.0, 1120.0] GB
# stage 1  world=8  7B/13B/70B -> [38.5, 71.5, 385.0] GB
# stage 2  world=8  7B/13B/70B -> [26.2, 48.8, 262.5] GB
# stage 3  world=8  7B/13B/70B -> [14.0, 26.0, 140.0] GB

Here's how to read it. Stage 3 divides the full 16 bytes by the GPU count, so it shrinks without limit as you add GPUs. Stage 1, by contrast, leaves the 4 bytes of parameters and gradients undivided, so no matter how many GPUs you add, a 7B model never drops below 28GB. If adding more GPUs isn't shrinking memory, the ZeRO stage is usually too low.

In PyTorch, FSDP is effectively ZeRO stage 3. As of August 2, 2026, the official PyTorch tutorial states plainly that "FSDP1 is deprecated" and recommends FSDP2, built on torch.distributed.fully_shard. The tutorial doesn't say as of which version it was deprecated, so it's safer to confirm that against the release notes of whatever PyTorch version you're running. The stable PyTorch version confirmed here is 2.13.0, released July 8, 2026.

# FSDP2 in its minimal form — shards in place instead of wrapping the module
import torch
from torch.distributed.device_mesh import init_device_mesh
from torch.distributed.fsdp import fully_shard, MixedPrecisionPolicy

mesh = init_device_mesh("cuda", (torch.distributed.get_world_size(),), mesh_dim_names=("dp",))
policy = MixedPrecisionPolicy(param_dtype=torch.bfloat16, reduce_dtype=torch.float32)

# Apply to submodules too, not just the root, so the communication unit stays small
for block in model.layers:
    fully_shard(block, mesh=mesh, mp_policy=policy)
fully_shard(model, mesh=mesh, mp_policy=policy)

Write only the last line and skip the loop, and the entire model becomes a single communication unit — it gathers all parameters at once at the start of the forward pass. The memory savings vanish and so does the overlap between communication and compute. A good share of "I turned on FSDP, why isn't memory dropping" questions come down to exactly this mistake.

Tensor Parallelism — Split the Matrix, Communicate on Every Layer

Tensor parallelism splits a single matrix multiplication across multiple GPUs. The classic layout from the Megatron-LM paper splits the MLP's first matrix column-wise and the second row-wise, so they chain together with no communication in between, and only one all-reduce happens at the end of the block. The attention block is split the same way, head by head.

The result is two all-reduces on the forward pass and two on the backward pass, per transformer layer. What's being communicated isn't parameters but activation tensors, sized s·b·h.

def tp_bytes_per_step(s, b, h, layers, tp, dtype_bytes=2):
    """Bytes moved by a single GPU per step under tensor parallelism (ring all-reduce approximation)."""
    tensor = s * b * h * dtype_bytes
    ring = 2 * (tp - 1) / tp          # per-GPU traffic coefficient for ring all-reduce
    per_layer = 4 * tensor * ring     # forward 2 + backward 2
    return per_layer * layers


print(tp_bytes_per_step(4096, 1, 4096, 32, tp=8) / 1e9, "GB/step")
# 7.516192768 GB/step

That's 7.5GB to move per step. An H100 SXM with NVLink 4 advertises 900GB/s bidirectional per GPU, so this communication finishes in around 10 milliseconds. Do the same communication over a single 400Gb/s InfiniBand port — that's 50GB/s — and it becomes 150 milliseconds. That arithmetic is exactly why training speed drops by a full order of magnitude the moment a tensor-parallel group crosses a node boundary.

The figures above are vendor-advertised numbers; measured effective bandwidth runs lower. You're better off getting the real number by running nccl-tests's all_reduce_perf directly on your cluster. The practical rule is simple: never let the tensor-parallel degree exceed the number of GPUs in a node. On an 8-GPU node, TP should be 8 or less.

Pipeline Parallelism — Cheapest Communication, Most Expensive Bubble

Pipeline parallelism cuts the layers into stages and places them on different GPUs. Activations are only exchanged at stage boundaries, so the communication volume is dramatically lower. Cut 32 layers into 4 stages and there are only 3 boundaries, and at each boundary a single s·b·h tensor moves point-to-point. That's why pipelines are allowed to cross node boundaries.

The price is the bubble. While the first stage processes the first micro-batch, every other stage sits idle; and the earlier stages sit idle again waiting for the backward pass of the last micro-batch to finish.

def bubble_ratio(stages: int, micro_batches: int) -> float:
    return (stages - 1) / micro_batches

for m in (4, 8, 16, 32, 64):
    print(f"micro_batches={m:3}  8-stage bubble {bubble_ratio(8, m):.1%}")
# micro_batches=  4  8-stage bubble 175.0%
# micro_batches=  8  8-stage bubble 87.5%
# micro_batches= 16  8-stage bubble 43.8%
# micro_batches= 32  8-stage bubble 21.9%
# micro_batches= 64  8-stage bubble 10.9%

Pushing the bubble down to around 10% takes at least 8x as many micro-batches as stages. But raising the micro-batch count grows activation memory and inflates the global batch size, which in turn affects the learning-rate schedule. These three values being tied to each other is exactly what makes pipeline tuning hard.

Interleaved schedules and DeepSeek-V3's DualPipe are among several schedules developed to cut the bubble. DualPipe overlaps the compute and communication phases of the forward and backward passes to shrink the pipeline bubble, and it's described in the DeepSeek-V3 technical report. On the PyTorch side, torch.distributed.pipelining is offered as the standard API.

Sequence and Context Parallelism — the Fourth Axis

The three axes above all split along parameters and batch. There's one more axis that splits along sequence length. The terminology gets used loosely, so let's keep the two apart:

• Sequence parallelism: paired with tensor parallelism in the Megatron family. It splits, along the sequence axis, the LayerNorm and dropout activations that tensor parallelism leaves un-split and duplicated, eliminating that redundant storage. It operates inside the same group as tensor parallelism.
• Context parallelism: splits the attention computation itself along the sequence axis. Each GPU holds only part of the sequence, and passes keys and values around in a ring to accumulate partial attention. Ring-attention variants belong to this family. DeepSpeed's Ulysses achieves a similar goal through a different implementation, using all-to-all communication.

The rule for when to reach for context parallelism is clear-cut: if you've raised the sequence length, dropped the micro-batch to 1, and recomputed every activation, and you're still hitting OOM — that's the moment to turn on context parallelism. Llama 3 405B training used 4D parallelism (TP, CP, PP, DP) precisely because of its 128K context extension stage.

Training an MoE model adds one more axis on top: expert parallelism. It places experts across GPUs and routes tokens between them with all-to-all. Because the communication pattern is all-to-all rather than all-reduce, it's far more sensitive to network topology, and if load balancing breaks down, a handful of GPUs end up doing all the work while the rest wait.

Picking a Combination — Reading the Table by Size and GPU Count

The real decision in practice isn't "which parallelism is best" but "what order do you stack them in at this scale." The generally accepted rule is to place the most communication-expensive axis innermost: TP inside the node, CP next, PP between nodes, and DP on the outside of everything.

This table is a starting point, not an answer. In practice, your target global batch size constrains the combination heavily. Global batch equals micro-batch × gradient accumulation × data-parallel degree, and the DP degree is fixed as the total GPU count divided by the TP, PP, and CP degrees. Change TP from 8 to 4 and DP doubles, which doubles the global batch too, which means you have to re-tune the learning rate. Changing a parallelism configuration isn't pure infrastructure work — it's a hyperparameter change.

Common Failure Modes

Just the ones that keep showing up in practice.

1. A TP group spans nodes. Stretch a 16-GPU TP group across two 8-GPU nodes and throughput collapses. Check the actual links with nvidia-smi topo -m, and check what order your launcher places ranks on nodes.
2. FSDP applied only to the root. The mistake covered above. You need to wrap at the submodule level for communication to overlap with compute.
3. OOM even with activation checkpointing on. What's left over is the embedding, the output layer, and the logits tensor — none of which get recomputed. In a model with a large vocabulary, the logits tensor is sized s·b·V, which alone can run to tens of gigabytes. You need to split the loss computation into chunks.
4. Shrinking the micro-batch to dodge OOM while leaving the bubble unaddressed. In a PP setup, shrinking the micro-batch solves the memory problem but grows the bubble, and throughput drops by half. The two are opposite ends of the same lever.
5. Gradient accumulation and normalization mismatched. Get the point at which you divide the loss by the accumulation step count wrong, and the effective learning rate shifts. Every time the parallel configuration changes, run a short training run and check that the loss curve overlaps the previous configuration's.
6. Papering over problems by raising the NCCL timeout. A timeout usually means a specific rank has died or there's one slow node. Check NCCL_DEBUG=INFO first to see which rank stalled.

Closing — Choosing a Parallelism Is Spending a Bandwidth Budget

The four parallelisms aren't substitutes for each other; each one spends a different resource. ZeRO and FSDP trade memory for bandwidth. Tensor parallelism trades memory for in-node bandwidth. Pipeline parallelism trades memory for idle time. Context parallelism trades sequence-axis memory for ring communication.

So the thing to do before picking a configuration isn't reading framework docs — it's drawing your cluster's bandwidth map. Measured in-node GPU-to-GPU bandwidth, measured inter-node bandwidth, and the ratio between the two. Once you know those three numbers, the table above practically decides itself. The next post surveys the training framework stack that turns these decisions into actual code, organized by lineage.