Chaos and Order

💡 왼쪽 원문을 읽으면서 오른쪽에 따라 써보세요. Tab 키로 힌트를 받을 수 있습니다.

Introduction — GPUs for Containers, GPUs for VMs
Part 1 — GPU Operator: Components, Configuration, Versions
Part 2 — Applying MIG to Only Some GPUs
Part 3 — Manual MIG: Working Directly with nvidia-smi
Part 4 — KubeVirt: Components, Configuration, Versions
Part 5 — Bridging the Two Worlds: GPU Operator × KubeVirt
Closing Thoughts
References

Introduction — GPUs for Containers, GPUs for VMs

GPU infrastructure on Kubernetes is split into two worlds: the one that plugs GPUs into containers (GPU Operator), and the one that runs virtual machines on Kubernetes and hands them a GPU wholesale (KubeVirt). The interesting part is that the two eventually meet in a single stack — the GPU Operator also takes care of GPU provisioning for KubeVirt VMs. This post is a complete rundown of both pillars — components, configuration, and versions — and covers two frequently asked topics with hands-on commands: applying MIG to only some GPUs and operating MIG manually. Reading the GPU Operator installation and MIG basics guide first will make this post twice as easy.

Part 1 — GPU Operator: Components, Configuration, Versions

Components (Operands)

The GPU Operator deploys "everything a GPU node needs" as DaemonSet operands and coordinates them all through a single ClusterPolicy.

Operand                    Role
─────────────────────────  ─────────────────────────────────────
driver                     Loads the NVIDIA driver as a container
container-toolkit          Configures the runtime to expose GPUs to containers
device-plugin              Advertises the nvidia.com/gpu resource to the scheduler
gpu-feature-discovery      Turns GPU model, memory, and MIG state into node labels
dcgm-exporter              Exports utilization, temperature, and power as Prometheus metrics
mig-manager                Node-label-driven MIG partitioning
node-feature-discovery     Hardware detection (dependency component)
validator                  Validation jobs for each stage
(sandbox family)           vfio-manager, vgpu-device-manager,
                           kubevirt-gpu-device-plugin — see Part 5

Configuration — ClusterPolicy Is the Single Source of Truth

Every operand's configuration converges on a single CR: clusterpolicies.nvidia.com/cluster-policy. Helm's --set flags ultimately just fill in fields of this CR. The fields you touch most often in practice:

# Key ClusterPolicy fields (excerpt)
spec:
  driver:
    enabled: true          # false if the driver is already on the node
    version: "580.65.06"   # pin the driver version
  toolkit:
    enabled: true
  mig:
    strategy: mixed        # single | mixed
  migManager:
    enabled: true
    config:
      name: custom-mig-config   # custom MIG profile ConfigMap
    env:
      - name: WITH_REBOOT       # environments that require a reboot, e.g. CSPs
        value: "true"
  devicePlugin:
    config:
      name: time-slicing-config # time-slicing settings also live in a ConfigMap
  sandboxWorkloads:
    enabled: false         # true for KubeVirt integration (Part 5)

Changes are made through a Helm upgrade or kubectl patch clusterpolicies..., and the operator detects them and reconfigures only the affected operands.

Versioning Scheme

The GPU Operator uses year-based versions (vYY.N.patch) — as of this writing the newest line is v26.x (e.g. v26.3.3), with the v25.x, v24.9, and v23.9 lines before it. Each release documents the exact bundle of supported driver versions and the supported Kubernetes range, so checking the compatibility matrix before an upgrade is a must. The principle to remember: the operator version manages the driver version, so if you want to bump only the driver, the proper lever is driver.version in the ClusterPolicy.

Part 2 — Applying MIG to Only Some GPUs

On an 8-GPU node, "slice only GPU 0 into small pieces for inference and keep the other seven whole for training" is a very common requirement. The answer is the mixed strategy plus a custom MIG config.

apiVersion: v1
kind: ConfigMap
metadata:
  name: custom-mig-config
  namespace: gpu-operator
data:
  config.yaml: |
    version: v1
    mig-configs:
      # MIG on GPU 0 only; GPUs 1-7 stay as they are
      partial-mig:
        - devices: [0]
          mig-enabled: true
          mig-devices:
            "1g.10gb": 4
            "3g.40gb": 1
        - devices: [1, 2, 3, 4, 5, 6, 7]
          mig-enabled: false

# ① Point the ClusterPolicy at this ConfigMap (skip if already configured)
kubectl patch clusterpolicies.nvidia.com/cluster-policy --type='json' \
  -p='[{"op":"replace","path":"/spec/migManager/config/name","value":"custom-mig-config"}]'

# ② Make sure the strategy is mixed — profiles vary per GPU, so single cannot express this
kubectl patch clusterpolicies.nvidia.com/cluster-policy --type='json' \
  -p='[{"op":"replace","path":"/spec/mig/strategy","value":"mixed"}]'

# ③ Apply the profile to the node
kubectl label nodes <node> nvidia.com/mig.config=partial-mig --overwrite

# ④ Result: resources are advertised like this
#   nvidia.com/gpu: 7                ← the 7 GPUs without MIG
#   nvidia.com/mig-1g.10gb: 4        ← slices carved out of GPU 0
#   nvidia.com/mig-3g.40gb: 1

Pods then request either nvidia.com/gpu (a whole GPU) or nvidia.com/mig-1g.10gb (a slice), depending on their purpose. Two core rules — whenever profiles differ between GPUs, the strategy must be mixed, and if you want the behavior of GPUs missing from a devices list to be explicit, always add a mig-enabled: false entry for them (do not leave it to implicit state).

Part 3 — Manual MIG: Working Directly with nvidia-smi

On bare-metal servers outside Kubernetes, or in environments that do not use mig-manager, you partition directly with nvidia-smi mig. This is exactly what the GPU Operator does under the hood, so doing it by hand once also sharpens your operator-debugging skills.

# ① Enable MIG mode (GPU 0 only) — drain processes from the target GPU first
sudo nvidia-smi -i 0 -mig 1
# If "Pending" appears, a GPU reset or reboot is required:
sudo nvidia-smi --gpu-reset -i 0

# ② Check which GPU instance (GI) profiles can be created
nvidia-smi mig -lgip
#  e.g. H100 80GB → 1g.10gb(ID 19), 2g.20gb(14), 3g.40gb(9), 7g.80gb(0) ...

# ③ Create GIs — one 3g.40gb plus four 1g.10gb; -C also creates the compute instances (CIs)
sudo nvidia-smi mig -i 0 -cgi 9,19,19,19,19 -C

# ④ Verify
nvidia-smi mig -lgi        # list the created GIs
nvidia-smi -L              # list MIG device UUIDs (MIG-xxxx format)

# ⑤ Run on a specific instance — target it by UUID
CUDA_VISIBLE_DEVICES=MIG-<uuid> python infer.py

# ⑥ Tear down — delete CIs, then GIs, then disable MIG mode
sudo nvidia-smi mig -i 0 -dci && sudo nvidia-smi mig -i 0 -dgi
sudo nvidia-smi -i 0 -mig 0

Concepts to know: MIG is a two-tier structure in which a GI (GPU Instance — a hardware partition of memory and SMs) contains CIs (Compute Instances — a further subdivision of the SMs). In most cases one CI per GI (the -C flag) is enough; splitting CIs further is for the special case where workloads share the same memory partition and divide only the compute. Also, the profile IDs in step ② differ per GPU model, so always check with -lgip first.

Part 4 — KubeVirt: Components, Configuration, Versions

Why VMs on Kubernetes

More workloads resist containerization than you might think — legacy systems that need kernel modules, Windows guests, multi-tenant GPU environments that demand strong isolation. KubeVirt makes VMs first-class citizens (CRDs) of Kubernetes, letting you handle pods and VMs with the same cluster, the same network, and the same kubectl.

The Four Components

Component        Form            Role
───────────────  ──────────────  ─────────────────────────────────────
virt-operator    Deployment      Installs and upgrades KubeVirt itself
virt-controller  Deployment      Cluster-level VM lifecycle (creates VMIs, etc.)
virt-handler     DaemonSet       Node-level daemon — syncs VMI specs with libvirt domains
virt-launcher    Pod (1 per VM)  The pod wrapping the VM — its libvirtd runs QEMU/KVM

The flow goes like this: a user creates a VirtualMachine CR → virt-controller creates a VirtualMachineInstance (VMI) and a virt-launcher pod → virt-handler on that node translates the VMI spec into a libvirt domain → libvirtd inside virt-launcher starts the QEMU/KVM process. The VM is literally a process inside a pod, so Kubernetes networking and scheduling apply as-is. Importing disk images is handled by a separate project, CDI (Containerized Data Importer), through the DataVolume CRD.

Configuration — the KubeVirt CR and featureGates

KubeVirt's global configuration also converges on a single CR (the KubeVirt resource in kubevirt.io/v1). The key settings needed for GPU passthrough:

apiVersion: kubevirt.io/v1
kind: KubeVirt
metadata:
  name: kubevirt
  namespace: kubevirt
spec:
  configuration:
    developerConfiguration:
      featureGates:
        - GPU                 # allow assigning GPUs to VMs
        - HostDevices
    permittedHostDevices:     # which host devices may be handed to VMs
      pciHostDevices:
        - pciVendorSelector: "10de:2331"     # NVIDIA H100 (vendor:device)
          resourceName: "nvidia.com/GH100_H100_PCIE"
          externalResourceProvider: true     # advertised by the GPU Operator plugin

Version Highlights

KubeVirt declared production readiness with v1.0 in 2023 and has continued the v1.x line with quarterly minor releases (v1.1, v1.2, ...). It is a CNCF incubating project, and each release specifies the range of Kubernetes versions it supports (generally the three most recent). The changes you actually feel are the maturing instancetype/preference API (declaring VM sizes like t-shirt sizes), live resource adjustment such as memory hotplug and CPU hotplug, and ever more robust live migration. A VM with a passed-through GPU is physically bound to its node, so live migration does not work for it — remember that as a law of physics, independent of any version.

Part 5 — Bridging the Two Worlds: GPU Operator × KubeVirt

Install the GPU Operator with sandboxWorkloads.enabled=true and it also takes charge of GPU provisioning for VMs.

helm install --wait gpu-operator \
  -n gpu-operator --create-namespace \
  nvidia/gpu-operator --version=v26.3.3 \
  --set sandboxWorkloads.enabled=true

The node label nvidia.com/gpu.workload.config then becomes the switch:

Value            Node purpose             Operands deployed
──────────────  ───────────────────────  ─────────────────────────────
container        Regular GPU containers   driver, device-plugin, ... (the Part 1 set)
vm-passthrough   Whole GPU to a VM        vfio-manager (binds vfio-pci),
                                          kubevirt-gpu-device-plugin
vm-vgpu          vGPU slices to a VM      vgpu-manager, vgpu-device-manager

On a passthrough node, vfio-manager binds the GPU to the vfio-pci driver (instead of the host driver), and kubevirt-gpu-device-plugin advertises that device as a resource like nvidia.com/GH100_H100_PCIE. The VM spec then requests it like this:

# VirtualMachine spec excerpt
spec:
  template:
    spec:
      domain:
        devices:
          gpus:
            - deviceName: nvidia.com/GH100_H100_PCIE
              name: gpu1

To sum up — within a single cluster, just by switching labels, some nodes can serve container GPUs, some VM passthrough, and some vGPU. If you need strong isolation (security, driver freedom), go with VM passthrough; if you need density, go with containers plus MIG; vGPU sits in between.

Closing Thoughts

The GPU Operator commands its fleet of operands through a single ClusterPolicy, KubeVirt brings VMs into the world of pods with four components, and the two meet at sandboxWorkloads and the workload.config label. Partial MIG is the mixed strategy plus a devices list; manual MIG is nvidia-smi mig -cgi/-dgi. Once you know what the operator is actually automating for you, you also know where to look when something breaks. You can build up your Kubernetes fundamentals with the Kubernetes Playground and the Kubestronaut Quiz.