Skip to content
Published on

Mastering the NVIDIA GPU Operator — From Install and Deployment to MIG Partitioning

🇰🇷KO🇺🇸EN🇯🇵JA
Split View필사 모드
Share
Authors

Introduction — Escaping GPU-Node Setup Hell

Anyone who has manually attached a GPU node to a Kubernetes cluster knows the drill: install the NVIDIA driver matched to the kernel version, install the container toolkit, adjust the runtime config, deploy the device plugin, attach a monitoring exporter — and if the version compatibility of even one of these five layers slips, pods cannot see the GPU. With ten nodes you do this ten times, and when a kernel update lands you start over.

The NVIDIA GPU Operator automates this whole stack with the Kubernetes operator pattern. The administrator installs one Helm chart, declares the desired state, and the operator deploys the needed components to each node as containers and keeps reconciling them. This article covers everything from installation and verification to the killer feature of A100/H100-class GPUs: MIG (Multi-Instance GPU) partitioning — with real commands. If you want to warm up on Kubernetes basics first, try this site's Kubernetes playground.

What the GPU Operator Deploys

Installing the operator lays down the following operands as DaemonSets in the gpu-operator namespace. Each corresponds to one job you used to do by hand.

  • NVIDIA Driver (containerized) — loads the kernel module from a container instead of installing the driver on the host. Kernel-update handling becomes dramatically easier.
  • NVIDIA Container Toolkit — lets the container runtime expose GPUs to pods.
  • Device Plugin — advertises the nvidia.com/gpu resource to the Kubernetes scheduler.
  • GPU Feature Discovery (GFD) — publishes GPU model/memory/MIG state as node labels usable in node selectors.
  • DCGM Exporter — exports GPU utilization, memory, temperature, and power as Prometheus metrics.
  • MIG Manager — the protagonist of this article. Watches a node label and applies MIG partitions to the hardware.
  • Validator — runs jobs verifying that each stage actually works.

Installation — One Helm Chart

Three prerequisites: nodes with NVIDIA GPUs, a supported container runtime (containerd etc.), and Node Feature Discovery (installed with the operator by default).

# 1) Add the NVIDIA Helm repository
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

# 2) Install (check the docs for the latest version — v26.3.3 at time of writing)
helm install --wait gpu-operator \
  -n gpu-operator --create-namespace \
  nvidia/gpu-operator \
  --version=v26.3.3

Two common variants depending on your environment:

# If the driver is already installed on the node (DGX OS, manual install, etc.)
helm install --wait gpu-operator \
  -n gpu-operator --create-namespace \
  nvidia/gpu-operator \
  --version=v26.3.3 \
  --set driver.enabled=false

# If the container toolkit is also preinstalled, add:
#   --set toolkit.enabled=false

Check installation status:

kubectl get pods -n gpu-operator
# You are healthy when driver-daemonset, container-toolkit, device-plugin,
# dcgm-exporter, gpu-feature-discovery, operator-validator are Running/Completed

Verification — Can a Pod See the GPU?

The most reliable check is a test pod that requests a GPU.

apiVersion: v1
kind: Pod
metadata:
  name: gpu-smoke-test
spec:
  restartPolicy: Never
  containers:
    - name: nvidia-smi
      image: nvidia/cuda:12.4.1-base-ubuntu22.04
      command: ['nvidia-smi']
      resources:
        limits:
          nvidia.com/gpu: 1

kubectl apply -f gpu-smoke-test.yaml
kubectl logs gpu-smoke-test
# Success when nvidia-smi prints your GPU model

Also worth checking the node labels GFD attached — useful for writing node selectors:

kubectl get node <node-name> -o jsonpath='{.metadata.labels}' | jq . | grep nvidia
# nvidia.com/gpu.product, nvidia.com/gpu.memory, nvidia.com/gpu.count, etc.

MIG — Slicing One GPU at the Hardware Level

Datacenter GPUs like the A100/H100 are too big as a single unit. When one small inference workload occupies an entire 80 GB GPU, the rest of the capacity idles. MIG (Multi-Instance GPU), supported since the Ampere generation, splits one GPU into up to seven hardware-isolated instances.

The key phrase is "hardware-isolated." Each MIG instance is physically allocated its own share of SMs (compute units), L2 cache, and memory bandwidth. That is the decisive difference from time-slicing — with time-slicing, pods take turns on the GPU, so one pod's burst becomes another pod's latency; with MIG, your performance holds no matter what the neighboring instance is doing. This is why MIG is preferred for multi-tenant clusters, inference serving, and per-team GPU allocation.

Profile names follow the format <GI count>g.<memory>gb. One H100 80GB, for example, can typically be split like this:

Profile      Max instances    Rough use case
1g.10gb   x7                 7 small inference services
2g.20gb   x3                 3 medium inference/experiments
3g.40gb   x2                 2 mid-to-large training runs
7g.80gb   x1                 the whole GPU, unpartitioned
(Mixes are possible too: e.g. one 3g.40gb + one 2g.20gb + two 1g.10gb)

On an A100 40GB the same principle yields 1g.5gbx7, 2g.10gbx3, 3g.20gbx2, and so on. Exact combination rules differ per GPU model — treat the profile tables in the NVIDIA MIG User Guide as the source of truth.

MIG Strategy — single vs. mixed

To use MIG with the GPU Operator, you first choose a strategy at install time. This choice changes how resources are advertised to Kubernetes.

  • single — every GPU on the node runs MIG, all with the same profile. Resources keep the familiar nvidia.com/gpu name; only the count grows (e.g., one H100 → nvidia.com/gpu: 7). No pod-spec changes needed — the easiest adoption path.
  • mixed — different profiles (even non-MIG GPUs) coexist on a node. Resources are advertised per profile (e.g., nvidia.com/mig-1g.10gb, nvidia.com/mig-3g.40gb). More flexible, but pods must name a profile.
# Install with the single strategy
helm install --wait gpu-operator \
  -n gpu-operator --create-namespace \
  nvidia/gpu-operator \
  --version=v26.3.3 \
  --set mig.strategy=single

# In cloud environments (GKE/EKS etc.) that require a reboot, add:
#   --set migManager.env[0].name=WITH_REBOOT \
#   --set-string migManager.env[0].value=true

To change only the strategy on an existing cluster, patch the ClusterPolicy:

kubectl patch clusterpolicies.nvidia.com/cluster-policy \
  --type='json' \
  -p='[{"op":"replace", "path":"/spec/mig/strategy", "value":"mixed"}]'

Applying MIG — With a Single Node Label

MIG Manager's mechanism is elegantly simple: it watches the nvidia.com/mig.config node label, and when the label changes, it applies that profile to the hardware.

# Split an H100 node into seven 1g.10gb instances
kubectl label nodes <node-name> nvidia.com/mig.config=all-1g.10gb --overwrite

Built-in profiles include all-disabled (turn MIG off), all-1g.10gb, all-2g.20gb, all-3g.40gb, and all-balanced (a balanced mix of sizes). Track the application via labels:

kubectl get node <node-name> -o jsonpath='{.metadata.labels}' | jq . | grep mig
# nvidia.com/mig.config.state transitions
#   pending -> (rebooting, if needed) -> success

After success, check resources to see the result:

kubectl describe node <node-name> | grep -A5 'Allocatable'
# single strategy: nvidia.com/gpu: 7  (gpu.product label shows MIG-1g.10gb)
# mixed  strategy: per-profile resources like nvidia.com/mig-1g.10gb: 7

Under the mixed strategy, pods request a specific profile like this:

resources:
  limits:
    nvidia.com/mig-1g.10gb: 1

Custom Profiles — Mixing Sizes Within One Card

When the built-ins are not enough (say, "GPU 0 gets a 3g+1g+1g mix and the rest keep MIG off"), define your own via a ConfigMap.

apiVersion: v1
kind: ConfigMap
metadata:
  name: custom-mig-config
  namespace: gpu-operator
data:
  config.yaml: |
    version: v1
    mig-configs:
      all-disabled:
        - devices: all
          mig-enabled: false
      inference-mix:
        - devices: [0]
          mig-enabled: true
          mig-devices:
            "1g.10gb": 5
            "2g.20gb": 1
        - devices: [1, 2, 3]
          mig-enabled: false

# Point MIG Manager at this ConfigMap via a ClusterPolicy patch
kubectl patch clusterpolicies.nvidia.com/cluster-policy \
  --type='json' \
  -p='[{"op":"replace", "path":"/spec/migManager/config/name", "value":"custom-mig-config"}]'

# Apply the custom profile to a node
kubectl label nodes <node-name> nvidia.com/mig.config=inference-mix --overwrite

Note: since v26.3.0, MIG Manager queries the node's hardware via NVML at startup and generates the list of possible profiles at runtime, writing a per-node ConfigMap. When a new GPU model ships, you no longer wait for an updated built-in profile list.

Operational Cautions — Non-Incidents If You Know Them in Advance

  • Reconfiguration is destructive. Before changing a profile, MIG Manager stops every pod on the affected GPUs — including the device plugin, DCGM, and GFD. On nodes running user workloads, cordon + drain first is the safe operating procedure.
  • Clouds may require a reboot. Some CSP environments need a node reboot to toggle MIG mode — the WITH_REBOOT setting above is the countermeasure. Passing through rebooting in mig.config.state is the normal flow.
  • A pod cannot span instances. One MIG instance is one pod's ceiling. If a large training job needing the 7g profile may arrive, plan a mixed strategy or a separate node pool. NVLink P2P between instances is also unavailable.
  • If the state sticks at pending, read the MIG Manager pod logs first. Common causes: a label typo (nonexistent profile name), a profile combination the GPU does not support, or a pod still holding the GPU.
  • Monitor with DCGM. The DCGM exporter emits per-MIG-instance metrics, so Prometheus shows per-instance utilization and memory as-is. Finding "sliced but idle" instances is the second half of MIG operations.
  • Choosing vs. time-slicing: if you need isolation and predictability, MIG; if your goal is oversubscription (packing more pods somehow) and you can tolerate interference, time-slicing. The two features serve different purposes.

Closing

The GPU Operator turned the repetitive labor of "GPU node setup" into one declarative resource, and MIG Manager lets you handle the utilization problem of very expensive GPUs with a single label line on top of it. In summary: install with Helm, choose a strategy (single/mixed), partition with the nvidia.com/mig.config label, confirm with mig.config.state, observe with DCGM — those five sentences are the whole story. What remains is finding the profile that fits your workloads, and your monitoring data will tell you that.

To shore up Linux and Kubernetes fundamentals alongside, try the Linux terminal and the Kubestronaut quiz.

References

Discuss on Twitter • View on GitHub