Skip to content
Published on

A GPU Troubleshooting Playbook — Fix the Layers, Then Walk Down

🇰🇷KO🇺🇸EN🇯🇵JA🇨🇳ZH
Split View필사 모드
Share
Authors

Introduction — Fix the Layers, Then Walk Down

The biggest time sink in GPU incident response is poking around in no particular order. SSH into the node, run nvidia-smi, read pod logs, restart the DaemonSet, and an hour is gone.

The trouble is that there are five layers and the symptoms resemble each other. A pod not starting could be a scheduling problem, an out-of-capacity problem, or a dead driver. So fixing an order in advance and walking down it is always faster.

Metric names and configuration were verified against the official documentation and repositories on 2026-08-12. They can differ between releases, so check again against the version you are running.

Layer 1 — The Pod Will Not Schedule

The symptom is Pending. Only two things need checking here: is the node advertising the resource, and is any of it left.

# Read the events to see why scheduling failed
kubectl describe pod <pod-name> | tail -20

# Compare capacity against allocatable
kubectl get nodes -o custom-columns=\
'NODE:.metadata.name,CAP:.status.capacity.nvidia\.com/gpu,ALLOC:.status.allocatable.nvidia\.com/gpu'

# If no node advertises anything, start with the labels
kubectl get nodes -L nvidia.com/gpu.count,nvidia.com/gpu.product,nvidia.com/mig.strategy

The path forks here. If the resource does not exist at all, drop to layer 2. If the resource exists but none is free, that is a capacity question, not an incident. And if capacity reads 8 while allocatable reads 7, that is a strong clue. As the Kubernetes documentation states, when a device is marked unhealthy the kubelet decreases allocatable only and leaves capacity unchanged. Seeing that asymmetry, you can jump straight to layer 5.

If no labels are present at all, GPU Feature Discovery or Node Feature Discovery is not running, which puts the problem on the operator side.

Layer 2 — The Pod Runs but Cannot See the GPU

The container starts but nvidia-smi fails or CUDA cannot find a device. Or container creation fails outright.

The official troubleshooting documentation names a representative error message: no runtime for 'nvidia' is configured. That means the NVIDIA Container Toolkit failed to register a runtime handler with the container engine. It is a toolkit problem, not a driver problem.

# Operator pod status at a glance
kubectl get pods -n gpu-operator

# Toolkit and device plugin logs
kubectl logs -n gpu-operator nvidia-container-toolkit-daemonset-<POD-ID>
kubectl logs -n gpu-operator nvidia-device-plugin-daemonset-<POD-ID>

# Where the validator got stuck
kubectl describe pod -n gpu-operator -l app=nvidia-operator-validator

The habit of reading nvidia-operator-validator first matters. That pod verifies in order whether the preceding stages succeeded, so it tells you fastest where things stopped. Operator pods stuck at the Init stage usually mean the driver DaemonSet is not ready yet, and staring at any other pod will not produce an answer.

Layer 3 — Driver and Toolkit Version Mismatch

Symptoms at this layer are subtle. Something that worked yesterday stops after a node reboot, or fails only on newly added nodes.

The cause is usually one of three: the kernel was updated and the driver module no longer matches, OS versions differ across nodes so the driver image the operator ships does not fit, or drivers are already installed on the host while the operator is also trying to deploy them.

# Driver container logs (kernel module build failures land here)
kubectl logs -n gpu-operator nvidia-driver-daemonset-<POD-ID> -c nvidia-driver-ctr

# Compare OS and kernel versions across nodes
kubectl get nodes -o wide

# Compare the driver and runtime version labels GFD applied
kubectl get nodes -L nvidia.com/cuda.driver-version.full,nvidia.com/cuda.runtime-version.full

The last command is especially useful. Comparing the nvidia.com/cuda.driver-version.full and nvidia.com/cuda.runtime-version.full labels that GPU Feature Discovery applies shows in one line which node is the odd one out.

In an environment where drivers already exist on the host, driver.enabled must be false in the Helm values, and the same goes for toolkit.enabled if the toolkit is already configured. Getting those two wrong produces symptoms that look like layer 3 while the cause is configuration.

Systems with NVSwitch carry one more condition. The official documentation notes that on systems requiring fabric management the validator can fail with a message about the system not yet being initialized, and that nvidia-fabricmanager must be installed alongside the driver.

Layer 4 — Memory Exhaustion and OOM

GPU memory exhaustion presents differently from CPU memory exhaustion. Nothing gets killed by a cgroup; the application throws an allocation failure, or performance quietly collapses.

Separate two things first. Is the container out of system memory, or is the GPU out of framebuffer? The former terminates the pod as OOMKilled and leaves an event; the latter leaves the pod alive while requests fail.

# Check whether the pod was OOMKilled (system memory side)
kubectl get pod <pod-name> -o jsonpath='{.status.containerStatuses[*].lastState.terminated.reason}{"\n"}'

# The GPU side is read from metrics
kubectl -n gpu-operator port-forward svc/nvidia-dcgm-exporter 9400:9400
curl -s localhost:9400/metrics | grep -E 'DCGM_FI_DEV_FB_(USED|FREE)'

As metrics, it looks like this.

# GPUs whose framebuffer usage is at the ceiling
DCGM_FI_DEV_FB_USED / (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE) > 0.95

# Framebuffer held per pod (with Kubernetes mapping enabled)
sum by (namespace, pod) (DCGM_FI_DEV_FB_USED)

# For an inference server, preemption gives the answer
sum by (model_name) (rate(vllm:num_preemptions_total[5m])) > 0

Inference servers such as vLLM reserve GPU memory up front at startup, so high framebuffer usage on its own is normal. The criterion is not usage but whether preemption is occurring. The reservation fraction is set by the vLLM gpu_memory_utilization setting, which is also exposed as a label on the cache configuration info metric.

If several workloads share one card through time-slicing, layer 4 problems are inevitable. As covered earlier, there is no memory isolation between replicas.

Layer 5 — GPUs Disappearing from a Node

The most unpleasant layer. The node had eight cards yesterday and has seven today.

The official troubleshooting documentation describes this situation directly. When the device plugin marks devices unhealthy due to Xid errors, the node advertises fewer GPUs than are physically present, identifiable in the device plugin logs by entries about marking a device as unhealthy.

# Look for unhealthy markings in the device plugin log
kubectl logs -n gpu-operator nvidia-device-plugin-daemonset-<POD-ID> | grep -i unhealthy

# Re-confirm the capacity versus allocatable gap
kubectl describe node <node-name> | sed -n '/Capacity/,/Allocated resources/p'

# Collect diagnostics in bulk
curl -o must-gather.sh -L https://raw.githubusercontent.com/NVIDIA/gpu-operator/main/hack/must-gather.sh
chmod +x must-gather.sh
./must-gather.sh

The metric-side evidence lives in DCGM. Enabled in the default CSV, DCGM_FI_DEV_XID_ERRORS holds the value of the last Xid error encountered. Being a last value, it is weak for history. According to the repository README the exporter also offers DCGM_EXP_XID_ERRORS_TOTAL, which watches that field and emits a separate series per observed xid label, with the value 0 treated as no error and not counted. That field is commented out in the default CSV, so using it requires enabling it yourself.

Metrics for hardware degradation are enabled by default: DCGM_FI_DEV_UNCORRECTABLE_REMAPPED_ROWS, DCGM_FI_DEV_CORRECTABLE_REMAPPED_ROWS, and DCGM_FI_DEV_ROW_REMAP_FAILURE. The last one indicates that remapping of rows has failed, so anything appearing there is past the point where software fixes it. For tracking clock events, it helps to remember that DCGM_EXP_CLOCK_EVENTS_TOTAL watches DCGM_FI_DEV_CLOCKS_EVENT_REASONS and splits by a clock_event label.

Closing — The Order Is the Skill

Compressed to one line each, the five layers are: is the resource advertised, does the container see the device, do the versions match, is there enough memory, is the card alive. Check from the top and most incidents resolve at the second or third command.

One habit is worth adding. Make comparing capacity against allocatable your first command. That single line immediately settles whether this is a layer 5 problem, and that judgment alone halves the diagnostic path.

That closes the series. Getting GPUs onto nodes with the operator, advertising them as resources, sharing them when needed, observing them with metrics, building a promise on top, and walking down in order when it breaks was one full loop. The habit of confirming each name in the documentation rather than guessing it is, in the end, what reduces the hours spent awake at three in the morning.

Try It Yourself

Series

References

Discuss on Twitter • View on GitHub