Skip to content
Published on

Device Plugins and GPU Scheduling — Where nvidia.com/gpu Comes From

🇰🇷KO🇺🇸EN🇯🇵JA🇨🇳ZH
Split View필사 모드
Share
Authors

Introduction — Where nvidia.com/gpu Comes From

Run kubectl describe node against a GPU node and you see a line reading nvidia.com/gpu with a number beside it. At first glance Kubernetes appears to have recognized the GPU. The truth is the opposite. There is no concept of a GPU anywhere in Kubernetes core. That line exists only because a process running on the node told the kubelet, in effect, that it has eight of a thing by that name.

That process is the device plugin. This post walks its registration path and the scheduling constraints that follow from it. When a GPU pod sits in Pending, this is where you find out which way to look.

Metric names and configuration were verified against the official documentation and repositories on 2026-08-12. They can differ between releases, so check again against the version you are running.

What a Device Plugin Does

The Kubernetes documentation spells out the gRPC service a device plugin must implement. The interface is five methods.

service DevicePlugin {
      rpc GetDevicePluginOptions(Empty) returns (DevicePluginOptions) {}
      rpc ListAndWatch(Empty) returns (stream ListAndWatchResponse) {}
      rpc Allocate(AllocateRequest) returns (AllocateResponse) {}
      rpc GetPreferredAllocation(PreferredAllocationRequest) returns (PreferredAllocationResponse) {}
      rpc PreStartContainer(PreStartContainerRequest) returns (PreStartContainerResponse) {}
}

Of these, GetPreferredAllocation and PreStartContainer are not required. The kubelet calls GetDevicePluginOptions first to find out which optional functions exist.

Registration happens over a Unix socket. The plugin connects to /var/lib/kubelet/device-plugins/kubelet.sock to register itself, and opens its own gRPC server on a socket under the same directory. The documentation is explicit that this path is hardcoded and is not affected by the kubelet --root-dir or other configuration. Which means: if that file is missing, or the plugin pod fails to mount that directory, registration never happens at all.

Resource names must follow the vendor-domain/resourcetype scheme, which is why an NVIDIA GPU becomes nvidia.com/gpu. The vendor domain in the name is a rule, not a convention.

Plugin behavior is tuned by a handful of flags. Among those documented in the repository, a few are worth knowing. --mig-strategy corresponds to the environment variable MIG_STRATEGY and defaults to none. --fail-on-init-error defaults to true, so the plugin dies outright when initialization fails, and that behavior matters: dying and restarting in a loop is far easier to diagnose than quietly staying alive while advertising nothing. Beyond those, --device-list-strategy governs how the device list reaches the container and defaults to envvar, while --device-id-strategy governs the identifier form and defaults to uuid.

Devices have exactly two states: Healthy and Unhealthy. When the plugin marks a device unhealthy in its ListAndWatch response, the kubelet decreases the node allocatable for that resource. What the documentation makes clear next matters: the capacity count does not change. That asymmetry is diagnostically useful. Capacity 8 with allocatable 7 means one card has died and the plugin already knows it.

The Rules for requests and limits

This is where people coming from CPU habits get it wrong first. The Kubernetes rules are three lines.

  • You can specify limits and omit requests. Kubernetes uses the limit value as the request.
  • You can specify both, but the two values must be equal.
  • You cannot specify requests without specifying limits.

In other words, a GPU is only ever requested in a shape equivalent to Guaranteed. The familiar CPU strategy of a low request and a high limit does not apply.

apiVersion: v1
kind: Pod
metadata:
  name: gpu-single
spec:
  restartPolicy: OnFailure
  containers:
    - name: worker
      image: 'nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0-ubuntu22.04'
      resources:
        limits:
          nvidia.com/gpu: 1

Why a GPU Cannot Be Shared the Way a CPU Is

The Kubernetes documentation states flatly that GPUs are not overcommittable, not shareable, and not fractional. The reason spans two layers.

The first is the resource model layer. Extended resources are integer-valued. There is no equivalent to the CPU millicore defined at all. So nvidia.com/gpu: 0.5 is not a syntax error so much as a value the scheduler cannot interpret.

The second is the hardware and driver layer. CPU time can be forcibly divided per process by the kernel scheduler, and memory can be capped by cgroups. A GPU has neither of those by default. Put two containers on the same card and VRAM goes to whoever grabs it first, while kernel launches shove each other around. If one side exhausts memory, the other simply fails. Splitting the resource number while isolation is absent means the scheduler makes a promise the hardware cannot keep.

So GPU sharing is solved by separate technology rather than by scheduler configuration. Time-slicing divides time, MIG divides hardware, and their isolation guarantees are not remotely comparable. That is exactly the subject of the next post.

Targeting GPUs by Node Label

In a cluster with mixed card types, a resource count is not enough. You use the labels GPU Feature Discovery applies to the node. Among the labels documented in the repository, the ones that see regular use are nvidia.com/gpu.product, nvidia.com/gpu.count, nvidia.com/gpu.memory, nvidia.com/gpu.family, nvidia.com/gpu.machine, nvidia.com/cuda.driver-version.full, nvidia.com/cuda.runtime-version.full, nvidia.com/gpu.compute.major, nvidia.com/gpu.compute.minor, nvidia.com/mig.capable, nvidia.com/gpu.sharing-strategy, and nvidia.com/gfd.timestamp.

# Inspect labels and resources on GPU nodes
kubectl get nodes -L nvidia.com/gpu.product,nvidia.com/gpu.count,nvidia.com/gpu.memory

kubectl describe node <node-name> | sed -n '/Capacity/,/System Info/p'

# Check whether the plugin registered, from the kubelet socket directory
kubectl debug node/<node-name> -it --image=busybox -- ls /host/var/lib/kubelet/device-plugins/

Using those labels for scheduling is an ordinary nodeSelector.

apiVersion: v1
kind: Pod
metadata:
  name: gpu-on-specific-card
spec:
  restartPolicy: OnFailure
  nodeSelector:
    nvidia.com/gpu.product: Tesla-T4
  containers:
    - name: worker
      image: 'nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0-ubuntu22.04'
      resources:
        limits:
          nvidia.com/gpu: 1

The exact label value string differs per card. Rather than guessing, reading the real value with the kubectl get nodes -L above is always faster.

Closing — Pending Almost Always Has One of Three Causes

When a GPU pod will not start, the cause is nearly always in one of three places. The node advertises no such resource at all (the plugin never registered), it advertises the resource but none are free (other pods hold them), or the label constraints match no node. All three separate within seconds using kubectl describe node and kubectl describe pod.

To summarize: the device plugin creates the existence of the resource, the extended resource rules pin it to whole integers, and GFD labels let you choose which one. The desire to have several workloads share one card is not solved at any of those three layers, which is why separate technology exists. The next post compares the two options.

Try It Yourself

  • Kubernetes Playground — change resource requests and watch the scheduling outcome shift.
  • kubectl Command Finder — look up the inspection commands above by situation.
  • K8s Lab — write nodeSelector and resource limits yourself to build intuition.

Series

References

Discuss on Twitter • View on GitHub