Split View: Slurm으로 GPU 클러스터 쓰기 — 제출보다 중요한 것은 왜 안 도는지 아는 일

Running a GPU Cluster with Slurm — Knowing Why a Job Will Not Run Matters More Than Submitting

• Introduction — Submitting Takes Five Minutes, Waiting Takes Three Days
• The Coordinate System — Partition, QoS, Account
• Anatomy of an sbatch Script
• Multi-Node Training — srun and Rendezvous
• Array Jobs and Dependency Chains
• Anatomy of PENDING
• Autopsy of a Dead Job — And Preparing for Preemption
• Slurm or Kubernetes
• Closing — The Scheduler Isn't a Queue, It's a Contract

Introduction — Submitting Takes Five Minutes, Waiting Takes Three Days

What someone learns first with Slurm is usually a single sbatch line. But what actually eats up time isn't submission. It's a submitted job sitting in PENDING for three days with no idea why, a job that ran for eight hours vanishing with no message, or a resumed job whose loss starts at some bizarre value.

So this post spends about half its length on failure diagnosis. The first half is the coordinate system and script structure; the second half is the autopsy.

Let me state the reference up front. As of August 2, 2026, per the SchedMD repository release tags, the current stable line is 26.05, with the latest tag at 26.05.2, released July 14, 2026. The previous line, 25.11, also got a 25.11.7 release the same day and is under maintenance. The installed version varies by cluster, so before applying anything below, check your actual version with sinfo --version and consult that version's official documentation. Option behavior has changed across minor versions more than once.

The Coordinate System — Partition, QoS, Account

Before submitting a job, you need to know three axes. Without these three, you can never interpret a PENDING reason.

• A partition is a grouping of nodes. H100 nodes and A100 nodes live in different partitions, and max runtime and accessible accounts differ by partition.
• An account is the accounting unit against which resource usage is logged. It forms a tree similar to an org chart, and fair-share priority is calculated along that tree.
• QoS is a policy tag. Priority weight, concurrent-run limits, preemptibility, and total resource-usage caps all attach here.

# Partitions I can use, and their limits
sinfo -o "%20P %5a %10l %6D %10T %N"

# My account and QoS associations
sacctmgr show assoc user="$USER" format=Account,Partition,QOS,GrpTRES,MaxJobs

# Policies by QoS
sacctmgr show qos format=Name,Priority,MaxTRESPU%30,MaxJobsPU,Flags%30

# How many of which GPU a partition has
sinfo -p gpu-h100 -o "%20N %10c %10m %30G"

The last column of that final command is the GRES definition. If you see a string like gpu:h100:8, the type name is h100, and this name differs by cluster. Copying an example straight out of the docs usually fails because of this name.

Anatomy of an sbatch Script

Here's the minimal shape for submitting a training job. Why each line is there is explained below.

#!/bin/bash
#SBATCH --job-name=llm-pretrain
#SBATCH --account=research
#SBATCH --partition=gpu-h100
#SBATCH --qos=normal
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=1
#SBATCH --gpus-per-node=8
#SBATCH --cpus-per-task=64
#SBATCH --mem=0
#SBATCH --time=24:00:00
#SBATCH --exclusive
#SBATCH --output=/scratch/%u/logs/%x-%j.out
#SBATCH --error=/scratch/%u/logs/%x-%j.err

set -euo pipefail
echo "job=$SLURM_JOB_ID nodes=$SLURM_JOB_NODELIST start=$(date -Is)"
srun --cpus-per-task="$SLURM_CPUS_PER_TASK" python -c "import torch; print(torch.cuda.device_count())"

There are three ways to request GPUs, and you shouldn't mix them.

With --gpus-per-task, Slurm implicitly applies GPU binding, so each task only sees its own single GPU. That behavior is sometimes convenient and sometimes in the way. For instance, if you're launching torchrun as one task per node but accidentally use --gpus-per-task=1, that process ends up seeing only one GPU, and torch.cuda.device_count() returns 1.

You need to request CPU and memory together too. The data-loader workers use CPU, and tokenized data gets loaded into host memory. --mem=0 means "request the entire memory of the node," commonly used for training jobs that take a node exclusively. The CPU cores per GPU determines your data-loader worker count, so base OMP_NUM_THREADS and num_workers on --cpus-per-task divided by GPU count.

There's a trap here. Whether srun inherits sbatch's CPU request has varied by Slurm version. In 22.05, srun stopped inheriting it and introduced the separate SRUN_CPUS_PER_TASK environment variable, and behavior was adjusted again in later versions. Since it can behave differently across cluster versions, explicitly re-passing --cpus-per-task to srun, as in the script above, is safe regardless of version. If this value falls to 1, the data loader gets pinned to a single core and GPU utilization plateaus in the 30s.

Binding is also worth checking. An 8-GPU node is usually split across two NUMA domains, with GPUs 0-3 attached to socket 0 and 4-7 to socket 1. If a process gets placed on cores on the opposite socket, bandwidth from host to GPU drops.

# GPU interconnect topology and NUMA placement within a node
nvidia-smi topo -m

# Which cores things actually got bound to
srun --cpu-bind=verbose,cores --gpu-bind=verbose,closest hostname

Multi-Node Training — srun and Rendezvous

There are only two patterns.

Pattern A: one torchrun per node. srun launches exactly one task per node, and torchrun spawns as many processes as there are GPUs inside it. Rendezvous is handled by torchrun's c10d backend.

#!/bin/bash
#SBATCH --job-name=pretrain-7b
#SBATCH --partition=gpu-h100
#SBATCH --account=research
#SBATCH --nodes=8
#SBATCH --ntasks-per-node=1
#SBATCH --gpus-per-node=8
#SBATCH --cpus-per-task=96
#SBATCH --mem=0
#SBATCH --exclusive
#SBATCH --time=48:00:00
#SBATCH --output=/scratch/%u/logs/%x-%j.out

set -euo pipefail

# Use the first node as the rendezvous point
MASTER_ADDR=$(scontrol show hostnames "$SLURM_JOB_NODELIST" | head -n1)
# Derive the port from the job number so concurrent jobs don't collide
MASTER_PORT=$(( 20000 + SLURM_JOB_ID % 20000 ))
export MASTER_ADDR MASTER_PORT
export OMP_NUM_THREADS=$(( SLURM_CPUS_PER_TASK / 8 ))
export NCCL_DEBUG=WARN
export TORCH_NCCL_ASYNC_ERROR_HANDLING=1

srun --cpus-per-task="$SLURM_CPUS_PER_TASK" --kill-on-bad-exit=1 \
  torchrun \
    --nnodes="$SLURM_NNODES" \
    --nproc-per-node=8 \
    --rdzv-backend=c10d \
    --rdzv-endpoint="$MASTER_ADDR:$MASTER_PORT" \
    --rdzv-id="$SLURM_JOB_ID" \
    train.py --config configs/7b.yaml

Pattern B: srun launches one process per rank. No torchrun; Slurm's own task becomes the rank directly. This removes one layer of process management, keeps logs cleaner, and lets Slurm directly see a dead rank.

#SBATCH --nodes=8
#SBATCH --ntasks-per-node=8
#SBATCH --gpus-per-task=1
#SBATCH --cpus-per-task=12

srun python train.py --config configs/7b.yaml

# Top of train.py — carries Slurm env vars over into distributed init vars
import os
import torch
import torch.distributed as dist

if "SLURM_PROCID" in os.environ and "RANK" not in os.environ:
    os.environ["RANK"] = os.environ["SLURM_PROCID"]
    os.environ["WORLD_SIZE"] = os.environ["SLURM_NTASKS"]
    os.environ["LOCAL_RANK"] = os.environ["SLURM_LOCALID"]

torch.cuda.set_device(int(os.environ["LOCAL_RANK"]))
dist.init_process_group(backend="nccl")
print(f"rank {dist.get_rank()}/{dist.get_world_size()} "
      f"on {os.uname().nodename} gpu {torch.cuda.current_device()}")

If you're running in a container, use pyxis and enroot. As of August 2, 2026, the versions confirmed are pyxis v0.24.0 (2026-05-12) and enroot v4.2.1 (2026-06-09). Once installed, container options become available on srun.

srun --container-image=/scratch/images/train-2026-08.sqsh \
     --container-mounts=/scratch:/scratch,/data:/data:ro \
     --container-workdir=/workspace \
     python train.py

Three problems come up often with rendezvous. First, if you pin MASTER_PORT to a fixed constant, it collides whenever another job lands on the same node. Derive it from the job number as shown above. Second, if a node has several interfaces (management, storage, and a fast fabric), NCCL might pick the slow one. Specify it explicitly with NCCL_SOCKET_IFNAME and confirm the actual choice with NCCL_DEBUG=INFO. Third, without --kill-on-bad-exit=1, one dead rank leaves the rest alive for up to 30 minutes until an NCCL timeout, burning GPU-hours the whole time.

Array Jobs and Dependency Chains

A hyperparameter sweep gets thrown as an array job. It becomes not one job but a bundle of indexed jobs, which the scheduler fills into open slots on its own.

#!/bin/bash
#SBATCH --job-name=lr-sweep
#SBATCH --array=0-11%4          # 12 jobs, at most 4 running at once
#SBATCH --gpus-per-node=1
#SBATCH --cpus-per-task=8
#SBATCH --time=02:00:00
#SBATCH --output=/scratch/%u/logs/%x-%A_%a.out

LRS=(1e-5 2e-5 5e-5 1e-4)
RANKS=(8 16 32)
lr=${LRS[$(( SLURM_ARRAY_TASK_ID % 4 ))]}
rank=${RANKS[$(( SLURM_ARRAY_TASK_ID / 4 ))]}

echo "task=$SLURM_ARRAY_TASK_ID lr=$lr rank=$rank"
srun python finetune.py --lr "$lr" --lora-rank "$rank" \
     --run-name "sweep-$SLURM_ARRAY_JOB_ID-$SLURM_ARRAY_TASK_ID"

In the log filename, %A is the array job's parent number and %a is the index. Leave either one out and 12 jobs will overwrite each other in the same file.

A long pretraining run can't finish in one go because of the time limit. Chain it together with dependencies.

# Chain five 24-hour segments together. Each segment starts only if the previous one succeeded
prev=""
for i in $(seq 1 5); do
  if [ -z "$prev" ]; then
    prev=$(sbatch --parsable pretrain.sbatch)
  else
    prev=$(sbatch --parsable --dependency="afterok:$prev" pretrain.sbatch)
  fi
  echo "segment $i -> job $prev"
done

afterok only chains onward on a successful exit. If an earlier segment fails, everything after it sits as DependencyNeverSatisfied, so you never lose days without noticing the failure. Conversely, if you want to keep going even after death by preemption, use afterany. Your training script must be written to automatically find and resume from the latest checkpoint for this chain to actually mean anything.

Anatomy of PENDING

The last column of squeue is the reason. That's the starting point for diagnosis.

squeue -u "$USER" -o "%.10i %.12P %.20j %.8T %.10M %.6D %R"
squeue -j 123456 --start          # expected start time
sprio -j 123456 -l                # priority broken down by component
scontrol show job 123456          # full requested resources and reason string
sinfo -R                          # down/drained nodes and why

The most commonly confused pair is Resources versus Priority. Resources means "no slot right now," and it resolves as time passes. Priority means "a slot opened up but it's not your turn yet," requiring your account's usage to drop or the jobs ahead of you to finish. The AssocGrpGRESRunMinutes family never resolves no matter how long you wait, so if you see this reason, go straight to an admin.

Shrinking your request often makes it run much faster. Instead of requesting 8 nodes for 24 hours, requesting 4 nodes for 12 hours twice gets picked up by backfill scheduling and runs sooner. Writing --time close to what you actually need is the precondition for backfill to work. Habitually writing the maximum costs you.

Autopsy of a Dead Job — And Preparing for Preemption

When a job disappears, there are three places to look: sacct, the log file, and the node itself.

sacct -j 123456 --format=JobID,JobName%20,State%20,ExitCode,Elapsed,MaxRSS,ReqTRES%40,NodeList%20
sacct -j 123456 --format=JobID,State,DerivedExitCode,Comment%40
seff 123456        # gives you a summary if slurm-contribs is installed

Host-memory OOM and GPU OOM are completely different events. Fail to make this distinction and you end up tweaking the wrong knob.

That last line comes up surprisingly often. Slurm ends a job as failed if it can't open the log file, but to the user it looks like "nothing happened at all." Create the --output path's directory before submitting.

If you suspect the node itself, check the GPU side. Xid errors in the kernel log and remapped-memory rows are the first-line indicators of hardware failure.

srun -w gpu-node-042 nvidia-smi --query-gpu=index,name,ecc.errors.uncorrected.volatile.total --format=csv
srun -w gpu-node-042 nvidia-smi --query-remapped-rows=gpu_bus_id,remapped_rows.pending,remapped_rows.failure --format=csv
srun -w gpu-node-042 bash -c 'dmesg -T | grep -i xid | tail -20'

Preparing for preemption has to be the default design of a training job. Using a preemptible QoS cuts your wait time a lot, but you can get bumped at any moment in exchange. Slurm sends a signal before terminating, so you checkpoint in that window and requeue yourself.

#SBATCH --signal=B:USR1@180     # send USR1 to the batch shell 180 seconds before termination
#SBATCH --requeue               # allow requeuing

CKPT_DIR=/scratch/$USER/ckpt/$SLURM_JOB_NAME
mkdir -p "$CKPT_DIR"

on_preempt() {
  echo "[$(date -Is)] Received USR1. Requesting a checkpoint save."
  touch "$CKPT_DIR/SAVE_AND_EXIT"     # the training loop checks for this file every step
  wait "$SRUN_PID"                    # wait for the save to finish
  echo "[$(date -Is)] Requeuing."
  scontrol requeue "$SLURM_JOB_ID"
  exit 0
}
trap on_preempt USR1

srun --cpus-per-task="$SLURM_CPUS_PER_TASK" python train.py --ckpt-dir "$CKPT_DIR" &
SRUN_PID=$!
wait "$SRUN_PID"

The B: prefix on --signal matters. Without it, the signal goes to the task instead of the batch shell, and the shell's trap never fires. And srun has to be launched in the background with wait so the trap reacts immediately — launched in the foreground, signal handling gets pushed until after srun exits.

How much lead time to set depends on how long the checkpoint save takes. If writing a distributed checkpoint for a 70B model to a parallel filesystem takes 90 seconds, 180 seconds is cutting it close. Measure it for real once, and set double that. The 25.11-line release announcement mentions a mode added for automatic requeue on node failure, but I haven't confirmed the details, so check the release notes for the version you're actually using.

Slurm or Kubernetes

Both run containers, both allocate GPUs. But the design premise is different. Slurm is a batch scheduler that fairly divides a finite pool of resources across a queue; Kubernetes is an orchestrator that continuously maintains a declared state.

The practical judgment mostly settles as: Slurm if large-scale pretraining and long batch jobs are your main workload; Kubernetes if inference serving and microservices are your main workload. Many organizations need both, so bridging projects have appeared. SchedMD's Slinky is an operator that runs Slurm on top of Kubernetes, and going the other direction, Kueue (v0.19.0 as of 2026-07-22) and Volcano (v1.15.1 as of 2026-07-30) add queueing and gang scheduling to Kubernetes.

If you're adopting new, the deciding factor isn't a feature comparison table, it's who's going to operate it. If an HPC team is already running a Slurm cluster, putting your training workload on it is overwhelmingly the faster path; if a platform team only knows Kubernetes, the cost of newly adopting Slurm outweighs the benefits of batch scheduling.

Closing — The Scheduler Isn't a Queue, It's a Contract

Using Slurm well isn't about knowing a lot of flags, it's about accurately writing the contract between the resources you're requesting and what the cluster can actually give you. Write your needed time honestly and you run sooner via backfill; use a preemptible QoS but prepare with checkpointing and your wait shrinks; know how to read PENDING reasons by code and you never sit waiting three days without knowing why.

And write your training script on the premise that it can die and get resumed at any moment. Half of the LLM Ops post covered next is really about turning that premise into code.