Chaos and Order

💡 왼쪽 원문을 읽으면서 오른쪽에 따라 써보세요. Tab 키로 힌트를 받을 수 있습니다.

Introduction — The Moment GPUs Multiply, the Problem Becomes Software
Part 1 — A Map of the AI Library and Framework Ecosystem
Part 2 — Parallelization Strategies: Start by Asking What Does Not Fit
Part 3 — torchrun: The Common Denominator of Everything
Part 4 — A Slurm Guide: The King of HPC
Part 5 — A Kubeflow Guide: The Kubernetes Camp's Answer
Part 6 — Selection Criteria: Slurm vs Kubeflow vs Ray
Closing
References

Introduction — The Moment GPUs Multiply, the Problem Becomes Software

Training on a single GPU is something the framework handles for you. The moment it becomes eight GPUs across four nodes, an infrastructure problem begins: how do you launch the processes, how do they find each other, and how do you restart when something fails? This post covers that entire terrain — which libraries exist (the ecosystem map), how to split the model (parallelization strategies), and where to submit the job (Slurm vs Kubeflow vs Ray). It expands the "distributed training stack" section of the AI model development lifecycle post to a hands-on level.

Part 1 — A Map of the AI Library and Framework Ecosystem

First, the cast of characters. Split them into layers and the terrain becomes clear.

Layer                Key players                   One-line summary
───────────────────  ────────────────────────────  ─────────────────────────────
Base frameworks      PyTorch 2.x                   The de facto standard; compiled speedups via torch.compile
                     JAX                           Functional + XLA; the center of TPU and Google-sphere research
Model & data hub     HuggingFace Transformers      The lingua franca of model definitions
                     Datasets / Tokenizers         The standard for data loading and tokenization
Fine-tuning layer    PEFT (LoRA), TRL (SFT/DPO)    Tuning and alignment on modest resources
                     Accelerate                    One abstraction for single-GPU and distributed code
Distributed engines  PyTorch DDP/FSDP              Framework-native parallelism
                     DeepSpeed (ZeRO)              Synonymous with memory optimization
                     Megatron-LM                   The standard for giant pretraining, TP/PP included
Orchestration        Slurm                         The king of HPC clusters
                     Kubeflow / Ray                The Kubernetes and Python-native camps
Serving (reference)  vLLM / SGLang / TensorRT-LLM  Where you go after training — see the serving post

A sense for choosing: research and fine-tuning default to the PyTorch + HF stack, TPU and large-scale parallelism research point to JAX, giant pretraining belongs to the Megatron family, and everywhere in between you will find DeepSpeed and FSDP.

Part 2 — Parallelization Strategies: Start by Asking What Does Not Fit

The choice of parallelization is decided by "what does not fit on the GPU."

DDP (Data Parallel) — the model fits, but you want to train faster. Replicate the model on every GPU, split the batch, and synchronize gradients with all-reduce after the backward pass. It is the simplest and scales well, so whenever possible DDP is the first choice.
ZeRO / FSDP (Fully Sharded Data Parallel) — the model (more precisely, parameters + gradients + optimizer state) does not fit. Shard those three across the GPUs and gather them only at the moment they are needed. The stages of DeepSpeed ZeRO (1: optimizer, 2: +gradients, 3: +parameters) and PyTorch FSDP are two implementations of the same idea. This is the workhorse of 7B–70B fine-tuning.
TP (Tensor Parallel) — not even a single layer fits, or latency matters. Split the matrix multiplications themselves across GPUs. Communication is frequent, so the rule is to stay within a node (NVLink).
PP (Pipeline Parallel) — distribute the layers across GPU groups and stream data through like an assembly line. Micro-batches shrink the pipeline bubbles. A good fit for splits that cross node boundaries.
Real-world giant training combines these into 3D parallelism (DP × TP × PP), and MoE models add EP (Expert Parallel) on top.

The formula to memorize: fits → DDP; does not fit → FSDP/ZeRO; still stuck → add TP and PP. And whatever the combination, mixed precision (bf16) and gradient checkpointing come as standard equipment.

Part 3 — torchrun: The Common Denominator of Everything

Whether it is Slurm or Kubeflow, at the bottom it is usually torchrun (PyTorch's distributed launcher) doing the work. There are only three concepts — the total number of processes (world size), your position in line (rank), and the meeting point (rendezvous).

# Single node, 8 GPUs
torchrun --nproc_per_node=8 train.py

# Multi-node (e.g., 2 nodes × 8 GPUs = world size 16)
# Run the same command on every node, changing only node_rank
torchrun \
  --nnodes=2 --nproc_per_node=8 --node_rank=0 \
  --rdzv_backend=c10d --rdzv_endpoint=10.0.0.1:29500 \
  train.py

The training code just calls torch.distributed.init_process_group() and then grabs its GPU from the LOCAL_RANK environment variable. With HF Accelerate or Lightning, the library does this wiring for you. One performance precondition for multi-node: the fabric between nodes must be RDMA (InfiniBand/RoCE), not plain Ethernet, or all-reduce becomes the bottleneck — NCCL handles the communication on top of it.

Part 4 — A Slurm Guide: The King of HPC

Slurm is a batch scheduler polished over decades of HPC history, and the de facto standard for GPU cluster training. Versions follow a year.month scheme (24.11, 25.05, ...) maintained by SchedMD. Four concepts are enough: node (a machine) — partition (a group of nodes, i.e., a queue) — job (a resource request plus a script) — GRES (generic resources such as GPUs).

# Survey the cluster
sinfo                      # Partition and node status
squeue --me                # My job queue
scontrol show node node01  # Node details

# The three ways to submit work
sbatch job.sh              # Batch submission (the standard)
srun --pty bash            # Interactive shell (for debugging)
salloc --gres=gpu:2        # Reserve resources, then run by hand

The heart of multi-node training is the sbatch script. A 2-node × 8-GPU torchrun example:

#!/bin/bash
#SBATCH --job-name=llm-train
#SBATCH --partition=gpu
#SBATCH --nodes=2                  # 2 nodes
#SBATCH --ntasks-per-node=1        # 1 torchrun per node (torchrun spawns the 8 processes)
#SBATCH --gres=gpu:8               # 8 GPUs per node
#SBATCH --cpus-per-task=64
#SBATCH --time=48:00:00
#SBATCH --output=logs/%x-%j.out    # %x=job name %j=job ID

# Rendezvous address = the first node
export MASTER_ADDR=$(scontrol show hostnames "$SLURM_JOB_NODELIST" | head -n 1)
export MASTER_PORT=29500

# Run torchrun on each node — SLURM variables automate the wiring
srun torchrun \
  --nnodes=$SLURM_JOB_NUM_NODES \
  --nproc_per_node=8 \
  --node_rank=$SLURM_NODEID \
  --rdzv_backend=c10d \
  --rdzv_endpoint=$MASTER_ADDR:$MASTER_PORT \
  train.py --config config.yaml

How to read it: srun runs torchrun once on each node assigned to the job, SLURM_NODEID supplies the node index, and scontrol show hostnames fills in the master address automatically. This one pattern is 80% of Slurm multi-node training. The remaining 20% is operational craft — write checkpoints more often than the --time limit forces you to (to survive preemption and timeouts), catch a signal two minutes before termination with --signal=SIGUSR1@120 to save one last checkpoint, and run hyperparameter sweeps as array jobs (--array).

Part 5 — A Kubeflow Guide: The Kubernetes Camp's Answer

If your GPU cluster is already Kubernetes (the very cluster set up with the GPU Operator), it is natural to submit training jobs as Kubernetes resources too. Kubeflow is that camp's all-in-one platform.

A map of the components — you do not need all of them:

Component          Role                                        Do you need it?
─────────────────  ─────────────────────────────────────────   ────────────────────────────
Training Operator  Distributed training job CRDs (the core!)   ★ Enough on its own to start
Pipelines          ML workflow DAG orchestration                When you pipeline things
Katib              Hyperparameter tuning / AutoML               When you need it
Notebooks          Jupyter environments inside the cluster      A convenience
KServe             Model serving (now its own project)          At the serving stage

The core is the Training Operator. A single PyTorchJob CRD declares multi-node training:

apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata:
  name: llm-train
spec:
  nprocPerNode: "8"                # Processes per node (pod) = number of GPUs
  pytorchReplicaSpecs:
    Master:
      replicas: 1
      template:
        spec:
          containers:
            - name: pytorch
              image: my-registry/train:latest
              command: ["torchrun", "train.py"]
              resources:
                limits:
                  nvidia.com/gpu: 8
    Worker:
      replicas: 3                  # 1 master + 3 workers = 4 nodes
      template:
        spec:
          containers:
            - name: pytorch
              image: my-registry/train:latest
              command: ["torchrun", "train.py"]
              resources:
                limits:
                  nvidia.com/gpu: 8

The Training Operator creates the pods and automatically injects the MASTER_ADDR, WORLD_SIZE, and RANK environment variables, so the wiring you did by hand on Slurm disappears. Check job status with kubectl get pytorchjobs. Note that the next-generation Kubeflow Trainer (v2) is evolving toward unifying the per-framework CRDs into a single TrainJob — if you are adopting fresh, check the v2 support status in the docs before you start. Gang scheduling (the job starts only when all nodes are secured at once) is solved by attaching a scheduler such as Volcano or Kueue.

Part 6 — Selection Criteria: Slurm vs Kubeflow vs Ray

Situation                                                Recommendation
───────────────────────────────────────────────────────  ─────────────────────────
Organization with an HPC tradition, bare-metal GPU farm  Slurm
Everything already runs on Kubernetes                    Kubeflow (Training Operator)
Flexible distributed training + serving inside Python    Ray (Train/Serve)
Small research lab, 2~4 nodes                            Plain torchrun + pdsh/tmux is enough
Prefer cloud-managed services                            SageMaker, Vertex, etc. (same concepts)

To keep only one essence — the floor is the same on every platform: torchrun (or its equivalent) launches the processes, NCCL carries the GPU-to-GPU communication, and checkpoints survive the failures. Platforms differ only in "who lends you the nodes and who fills in the environment variables." That is why, once you properly understand multi-node on one platform, the rest becomes a translation problem.

Closing

Lining up the decisions you face as GPUs grow from one card to many cards and many nodes: parallelization strategy (fits → DDP; does not fit → FSDP/ZeRO; still stuck → TP and PP) → launcher (torchrun) → orchestrator (Slurm or Kubeflow) → operations (checkpoints, RDMA, gang scheduling). Once training runs on top of this stack, the next bottleneck is data — that story continues in the LLM training data preprocessing post.