Chaos and Order

💡 왼쪽 원문을 읽으면서 오른쪽에 따라 써보세요. Tab 키로 힌트를 받을 수 있습니다.

Introduction — Beginning the Postmortem
Part 1 — Symptom: A VM Stuck for 112 Days
Part 2 — The GPU Was Registered Correctly
Part 3 — The Real Cause: A Single Line of kubelet
Part 4 — The Physics of Passthrough: Why It Cannot Go "to Another Node"
Part 5 — Recovery: Label First, Then swap
Closing — Lessons from the Postmortem
References

Introduction — Beginning the Postmortem

In the previous post, the GPU operator I wrote in Rust delivered an unexpected diagnosis: all 4 GPU nodes are NotReady, meaning the entire GPU capability of the cluster is dead. This post is the postmortem record of tracing the cause of that death all the way to the end on a real cluster. The subject is an 8-node homelab (GPU Operator v25.3.0, KubeVirt v1.7.0), and every piece of evidence is measured. If you need the background concepts for this post, it is worth reading the GPU Operator × KubeVirt overview first.

Part 1 — Symptom: A VM Stuck for 112 Days

First, I looked at the state of the KubeVirt VMs. Here is the phase distribution of the VMs on the real cluster:

$ kubectl get vm -A
phase             count note
────────────────  ────  ─────────────────────────────
Running              3
Paused               2
Stopped              1
ErrorUnschedulable   2    ← gpu-fedora, rhel9-gpu-vm  (both are GPU VMs!)

gpu-fedora and rhel9-gpu-vm — exactly the two VMs that require a GPU are ErrorUnschedulable. And for how long? The AGE from kubectl get vm is 112 days. They had been left unscheduled for 112 days. The VMs that do not use a GPU (Running·Paused) are fine. The culprit is somewhere in the GPU path.

The state of the GPU Operator operand pods was not healthy either:

$ kubectl get pods -n gpu-operator  (status histogram)
Running       14
Error          6     ← sandbox-device-plugin, sandbox-validator etc.
Terminating    4

Part 2 — The GPU Was Registered Correctly

An interesting twist: the GPU configuration itself was flawless. Looking at the permittedHostDevices of the KubeVirt CR, the 5090 and the 3090 are registered accurately, down to the PCI ID:

$ kubectl get kubevirt -n kubevirt -o yaml   (permittedHostDevices excerpt)
pciHostDevices:
  - pciVendorSelector: "10de:2b85"   # NVIDIA GB202 = RTX 5090
    resourceName: nvidia.com/GB202_GEFORCE_RTX_5090
    externalResourceProvider: true
  - pciVendorSelector: "10de:2204"   # NVIDIA GA102 = RTX 3090
    resourceName: nvidia.com/GA102_GEFORCE_RTX_3090
    externalResourceProvider: true

Right down to sandboxWorkloads.enabled: true and driver.enabled: false (use the host driver), the stage for handing a GPU to a VM was fully set. And yet the VMs did not come up. Even if the configuration is correct, it is useless without a node — this is what changed the direction of the postmortem.

Part 3 — The Real Cause: A Single Line of kubelet

I went down to the only node where the GPU VMs could be scheduled (omen2, equipped with an RTX 5090) and looked at the kubelet log. The culprit was not a fancy GPU problem but this single line:

$ journalctl -u kubelet -n 5   (omen2)
E0710 15:39:51 run.go:72] "command failed" err="failed to run Kubelet:
    running with swap on is not supported, please disable swap
    or set --fail-swap-on flag to false"
systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE

systemctl is-active kubelet → activating (that is, a crash-loop). Someone had turned on a 32GB swapfile on this node, and by default the Kubernetes kubelet refuses to start when swap is on. Once kubelet dies, the node becomes NotReady, NFD erases the GPU labels, the device-plugin cannot advertise nvidia.com/gpu — and everything stacked on top of it collapsed.

Drawn as a single causal chain, it looks like this:

swap on  →  kubelet refused(crash-loop)  →  node NotReady
         →  NFD labels gone + device-plugin stopped
         →  GPU VM cannot be scheduled(ErrorUnschedulable, 112 days)

Part 4 — The Physics of Passthrough: Why It Cannot Go "to Another Node"

With an ordinary pod, even if one node dies the scheduler moves it to another node. Why can a GPU passthrough VM not be moved? Because it is bound to a physical device. gpu-fedora requires the resource nvidia.com/GB202_GEFORCE_RTX_5090, and that physical 5090 is plugged in only on omen2. If that node dies, the VM has nowhere to go.

This is not a bug but a design — it is the physical embodiment of the sentence I wrote in the GPU Operator × KubeVirt post: "a GPU-passthrough VM cannot be live-migrated; this is a physical law independent of the version." Unlike container GPUs (split with MIG and shared by multiple pods), VM passthrough is tightly coupled to one GPU = one node. It is the shadow of the trade-off that gives up density in exchange for strong isolation.

Part 5 — Recovery: Label First, Then swap

There was one more trap in the recovery. The workload label on omen2 was nvidia.com/gpu.workload.config=vm-passthrough. If you simply revive kubelet in this state, the GPU Operator brings up vfio-manager and rebinds the 5090 to the vfio-pci driver — and then every CUDA/torch job that was running on that node dies instantly (this GPU is the very card I used in the small-models experiment post).

That is why the recovery order matters:

① (before reviving kubelet) switch the label to container to block vfio rebinding
     kubectl label node omen2 nvidia.com/gpu.workload.config=container --overwrite
② turn off swap and make it permanent
     sudo swapoff -a
     sudo sed -i '/swap/s/^/#/' /etc/fstab
③ restart kubelet → node Ready, 5090 exposed to pods as nvidia.com/gpu
     sudo systemctl restart kubelet

The reason for doing ① first is the crux — because driver.enabled: false means the GPU Operator uses the host driver as-is, so with the container label it exposes the GPU to pods while keeping the host's nvidia driver. In other words, torch survives and k8s also sees the GPU. (① has already been applied, while ②③ need sudo and therefore the hand of the node owner.)

Closing — Lessons from the Postmortem

The biggest lesson is clichéd but new every time: a failure in the fancy layer usually comes from the boring layer. GPU passthrough, vfio, permittedHostDevices, PCI IDs were all perfect, but what brought it down was the single line swap on. The second lesson is the physics of passthrough — give a VM a whole GPU and that VM is nailed to that node. If you need density, containers + MIG; if you need isolation, VM passthrough; the price of that choice is "when the node dies, it dies with it." And the starting point of this entire diagnosis was the 20-line operator I wrote in Rust — you see infrastructure only as far as you have touched it yourself.