Skip to content
←

필사 모드: NVIDIA GPU Operator — The Six Pieces You Used to Install by Hand

English
0%
정확도 0%
💡 왼쪽 원문을 읽으면서 오른쪽에 따라 써보세요. Tab 키로 힌트를 받을 수 있습니다.

Introduction — If You Built a GPU Node by Hand

For CPU-only workloads, adding a Kubernetes node is boring in the best way. You attach the node, the kubelet registers, pods land. Add a GPU and that simplicity evaporates. The card is seated, the kubelet looks healthy, and yet pods sit in Pending forever while the word GPU appears nowhere in the node allocatable.

The reason is plain: Kubernetes does not know what a GPU is. Making one visible as a schedulable resource takes at least six pieces on every node, and those pieces have to agree on versions. The GPU Operator does not make those pieces disappear. It moves them under one controller. Understanding that difference before you start is what separates a ten-minute incident from a two-day one.

Metric names and configuration were verified against the official documentation and repositories on 2026-08-12. They can differ between releases, so check again against the version you are running.

What You Used to Install by Hand

Without the Operator, preparing a GPU node runs roughly like this. Each step depends on the one before it, so the order is not negotiable.

First you install the NVIDIA driver, including the kernel modules and the userspace libraries, on the node. Bump the kernel and you rebuild or reinstall. Next comes the NVIDIA Container Toolkit, which registers an nvidia runtime handler with containerd or CRI-O. Without it, a perfectly healthy driver still leaves the device nodes invisible inside containers.

Then the device plugin. It registers with the kubelet and advertises an extended resource named nvidia.com/gpu on the node. Only at this point does the scheduler treat a GPU as a resource at all. To then target the right nodes you need labels describing which cards are present and how many, which is GPU Feature Discovery, sitting on top of the general-purpose labeler, Node Feature Discovery. Finally, to get temperature, power, and error telemetry out, you need DCGM and an exporter in front of it.

Multiply six pieces by your node count, then multiply again by kernel updates and driver upgrades, and it is obvious why this ended up wrapped in an operator.

What the GPU Operator Bundles

The official overview lists the managed components as the NVIDIA drivers, the Kubernetes device plugin for GPUs, the NVIDIA Container Toolkit, GPU Feature Discovery for automatic node labeling, NVIDIA DCGM for monitoring, the NVIDIA MIG Manager for Kubernetes, the Validator, the NVIDIA DCGM Exporter, and the NVIDIA Driver Manager for Kubernetes. Optional pieces layer on top: the KubeVirt GPU Device Plugin, the vGPU Device Manager, the GDS Driver, the Kata Manager, and the GDRCopy Driver.

The desired state for all of it converges on one custom resource, clusterpolicies.nvidia.com. A practical rule falls out of that: edit a DaemonSet directly and the operator reverts you. The place to make changes is always the ClusterPolicy or the Helm values.

The chart defaults lay out each switch plainly. These values come from deployments/gpu-operator/values.yaml in the repository.

# deployments/gpu-operator/values.yaml (excerpt, verified 2026-08-12)
mig:
  strategy: single

driver:
  enabled: true
  kernelModuleType: 'auto'
  usePrecompiled: false

toolkit:
  enabled: true
  image: container-toolkit
  installDir: '/usr/local/nvidia'

devicePlugin:
  enabled: true
  image: k8s-device-plugin

dcgm:
  # disabled by default to use embedded nv-hostengine by exporter
  enabled: false

dcgmExporter:
  enabled: true
  enablePodLabels: false
  enablePodUID: false
  serviceMonitor:
    enabled: true
    interval: 15s

gfd:
  enabled: true

migManager:
  enabled: true

dcgm.enabled defaulting to false is worth pausing on. The comment states the reason outright: the exporter uses an embedded nv-hostengine. Running a separate DCGM DaemonSet is the option you reach for when you want a remote hostengine instead.

Less conspicuous in the list, but the piece you meet most often during incidents, is the Validator. It appears in the chart as a validator section and runs in the cluster as a pod named nvidia-operator-validator. Its job is to verify, in order, that the preceding stages actually succeeded: that the driver came up, that the toolkit registered with the runtime, that the device plugin is advertising the resource. If any one of those fails, the components behind it stall at startup. So when a GPU-related pod is stuck initializing, the cause usually lives in an earlier stage rather than in that pod itself. Knowing only this one fact shortens diagnosis considerably.

What Installation Looks Like

The official getting-started page reduces installation to two Helm commands.

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia && helm repo update

helm install --wait --generate-name \
    -n gpu-operator --create-namespace \
    nvidia/gpu-operator \
    --version=v26.3.3

If drivers are already on the nodes or the container toolkit is already configured, turn those pieces off at install time.

helm install --wait --generate-name \
    -n gpu-operator --create-namespace \
    nvidia/gpu-operator \
    --version=v26.3.3 \
    --set driver.enabled=false \
    --set toolkit.enabled=false

Verification after install means the pod list, the ClusterPolicy status, and the actual allocatable.

kubectl get pods -n gpu-operator
kubectl get clusterpolicy

# Confirm the node advertises GPUs as a resource
kubectl get nodes -o json \
  | jq '.items[] | select(.status.allocatable["nvidia.com/gpu"] != null)
        | {node: .metadata.name, gpu: .status.allocatable["nvidia.com/gpu"]}'

Finally, run one real workload through it. The sample from the official docs is fine as is.

apiVersion: v1
kind: Pod
metadata:
  name: cuda-vectoradd
spec:
  restartPolicy: OnFailure
  containers:
    - name: cuda-vectoradd
      image: 'nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0-ubuntu22.04'
      resources:
        limits:
          nvidia.com/gpu: 1

What to Check Before You Install

Of the prerequisites the docs list, four are the ones that actually trip people up.

First, worker nodes need to run the same OS version, or have drivers pre-installed if they do not. The operator ships drivers as containers, so images are split by kernel and OS combination.

Second, nodes must be configured with a container engine such as CRI-O or containerd. The toolkit needs something to register a runtime handler with.

Third, if your cluster uses Pod Security Admission to restrict pod behavior, you must label the operator namespace so the enforcement policy is privileged. The driver container touches kernel modules, so there is no way around it.

Fourth, Node Feature Discovery is required. The chart deploys it by default, but if NFD already exists in your cluster you need to avoid installing it twice. Conflicts here produce symptoms that look nothing like a GPU problem, which is why they take so long to find.

Closing — The Operator Relocates the Install, It Does Not Remove It

Adopting the GPU Operator does not make driver versions or kernel compatibility go away. What goes away is a human SSHing into every node to repeat the same work. What appears in its place is a new layer. When something breaks now, you read pod logs in the gpu-operator namespace before you read the node.

So this series walks the layers in order. The next post opens up the one piece that actually governs scheduling: the device plugin. Where the name nvidia.com/gpu comes from, and why you cannot split that number into a fraction, is the subject.

Try It Yourself

  • K8s Lab — edit manifests directly and watch how resource requests change scheduling.
  • kubectl Command Finder — look up the verification commands above by situation.
  • Kubestronaut Quiz — test your grip on operators and custom resources.

Series

References

현재 단락 (1/86)

For CPU-only workloads, adding a Kubernetes node is boring in the best way. You attach the node, the...

작성 글자: 0자원문 글자: 7,584자작성 단락: 0/86