Skip to content
Published on

DCGM Exporter — GPU Utilization Is Not What You Think It Is

🇰🇷KO🇺🇸EN🇯🇵JA🇨🇳ZH
Split View필사 모드
Share
Authors

Introduction — How DCGM and the Exporter Relate

Ask most people to build a GPU dashboard and they start in the same place. Deploy DCGM Exporter, import the default dashboard, put GPU utilization on the big gauge. When the gauge reads 95 percent, everyone is satisfied. The problem is that the number does not measure what people expect it to.

Start with the structure: DCGM is what actually reads values off the GPU, and the exporter is a thin layer translating those values into Prometheus exposition format. That is also why dcgm.enabled is false in the GPU Operator chart defaults. As the comment says outright, the exporter uses an embedded nv-hostengine, so no separate DCGM DaemonSet is needed.

Metric names and configuration were verified against the official documentation and repositories on 2026-08-12. They can differ between releases, so check again against the version you are running.

What Is Enabled by Default

What gets exposed is decided by etc/default-counters.csv in the repository, the only authoritative source on this subject, whose format is three columns: the DCGM field name, the Prometheus metric type, and a help string. Look at real output first.

kubectl -n gpu-operator port-forward svc/nvidia-dcgm-exporter 9400:9400

curl -s localhost:9400/metrics | grep -E '^DCGM_FI_(DEV|PROF)' | head -20

The output shape shown in the repository README looks like this.

DCGM_FI_DEV_SM_CLOCK{gpu="0", UUID="GPU-604ac76c-d9cf-fef3-62e9-d92044ab6e52",container="",namespace="",pod=""} 139
DCGM_FI_DEV_MEM_CLOCK{gpu="0", UUID="GPU-604ac76c-d9cf-fef3-62e9-d92044ab6e52",container="",namespace="",pod=""} 405

Grouping the fields that are not commented out in the default CSV, meaning the ones actually enabled, by character gives this. Parentheses hold the CSV help string.

  • Clocks: DCGM_FI_DEV_SM_CLOCK (SM clock frequency (in MHz)), DCGM_FI_DEV_MEM_CLOCK
  • Temperature and power: DCGM_FI_DEV_GPU_TEMP, DCGM_FI_DEV_MEMORY_TEMP, DCGM_FI_DEV_POWER_USAGE (Power draw (in W)), DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION (counter, since boot, in mJ)
  • Memory: DCGM_FI_DEV_FB_USED, DCGM_FI_DEV_FB_FREE, DCGM_FI_DEV_FB_RESERVED — all in MiB
  • Utilization: DCGM_FI_DEV_GPU_UTIL, DCGM_FI_DEV_MEM_COPY_UTIL, DCGM_FI_DEV_ENC_UTIL, DCGM_FI_DEV_DEC_UTIL
  • Errors: DCGM_FI_DEV_XID_ERRORS (Value of the last XID error encountered), DCGM_FI_DEV_PCIE_REPLAY_COUNTER
  • Row remapping: DCGM_FI_DEV_UNCORRECTABLE_REMAPPED_ROWS, DCGM_FI_DEV_CORRECTABLE_REMAPPED_ROWS, DCGM_FI_DEV_ROW_REMAP_FAILURE
  • Profiling: DCGM_FI_PROF_GR_ENGINE_ACTIVE, DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, DCGM_FI_PROF_DRAM_ACTIVE, DCGM_FI_PROF_PCIE_TX_BYTES, DCGM_FI_PROF_PCIE_RX_BYTES

One important fact already falls out. DCGM_FI_PROF_SM_ACTIVE and DCGM_FI_PROF_SM_OCCUPANCY are not on that list. Both exist in the CSV but are commented out. Which means with the default configuration, the metrics this post is about to argue for are not collected at all.

The Trap in the Word Utilization

The CSV help for DCGM_FI_DEV_GPU_UTIL is one line: GPU utilization (in %). From the name and that description you would read it as some percentage of the GPU's capability being consumed. Yet the section comment in the CSV quietly hints otherwise, noting that the sample period varies depending on the product.

NVIDIA's NVML API documentation defines the same-named quantity far more bluntly. The gpu field of nvmlUtilization_t is the percent of time over the past sample period during which one or more kernels was executing on the GPU. Take that sentence apart and what it omits becomes visible. How many kernels, how many SMs they occupy, whether tensor cores are touched, how much memory bandwidth is moved — none of it is in there. What is measured is time alone: was any kernel resident or not.

So this happens. Suppose a pod runs a single-block kernel continuously. On a card with 108 SMs, that kernel uses exactly one. The other 107 sit idle. But a kernel is always executing, so the utilization gauge reads 100 percent. The dashboard is green and the card is barely one percent used.

The metric the DCGM documentation offers to close that gap is DCGM_FI_PROF_SM_ACTIVE, field ID 1002, defined as the fraction of time at least one warp was active on a multiprocessor, averaged over all multiprocessors. The worked example the same document attaches nails the character of the metric. If a GPU has N SMs, a kernel using N blocks that runs over the entire interval yields an activity of 1. A kernel using N/5 blocks over the entire interval yields 0.2. And a kernel using N blocks that runs over one fifth of the interval with the SMs otherwise idle also yields 0.2.

Return to the earlier example and the answer appears. Utilization is 100 percent while DCGM_FI_PROF_SM_ACTIVE sits near 0.01. Both numbers describe the same situation, and the second one is telling the truth.

What to Look at Alongside It

The DCGM documentation even attaches interpretation guidance to DCGM_FI_PROF_SM_ACTIVE: a value of 0.8 or greater is necessary but not sufficient for effective use of the GPU, and a value less than 0.5 likely indicates ineffective usage. The word necessary is the key. SMs being busy and SMs doing useful work are different claims.

So look at four things together, at minimum.

Use DCGM_FI_PROF_SM_ACTIVE (1002) to see how widely the work spreads. DCGM_FI_PROF_SM_OCCUPANCY (1003), defined as the fraction of resident warps on a multiprocessor relative to the maximum number of concurrent warps it supports, tells you how full each SM is. Interpret it carefully: as the documentation warns directly, higher occupancy does not necessarily indicate better GPU usage. It does for memory-bandwidth-limited workloads, but not necessarily for compute-limited ones.

DCGM_FI_PROF_PIPE_TENSOR_ACTIVE (1004), the fraction of cycles the tensor pipe was active, tells you whether the tensor cores are actually turning, and DCGM_FI_PROF_DRAM_ACTIVE (1005), the fraction of cycles where data was sent to or received from device memory, tells you whether memory is the bottleneck. On an LLM inference server, the former near the floor means the matrix math is not reaching tensor cores, while a high latter with a low former during decode-heavy stretches is the normal picture.

As queries, they look like this. Prometheus expressions use braces in label selectors, so they must live inside code blocks.

# Mean SM activity per node
avg by (hostname) (DCGM_FI_PROF_SM_ACTIVE)

# Find GPUs with high utilization but empty SMs
DCGM_FI_DEV_GPU_UTIL > 90 and on(gpu, UUID) DCGM_FI_PROF_SM_ACTIVE < 0.3

# Framebuffer usage ratio
DCGM_FI_DEV_FB_USED / (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE)

# Grafana variables use the dollar sign, so keep them inside code blocks only
avg by (gpu) (rate(DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION{hostname=~"$node"}[$__rate_interval]))

One more thing. Hardware constraints mean only certain groups of profiling metrics can be read together, so DCGM applies automatic multiplexing by statistically sampling the requested metrics. The documentation therefore warns that collection at higher frequencies will result in zeroes returned as DCGM attempts to group metrics together. If unexplained zeroes appear on your dashboard, this is worth remembering.

Configuring Custom Metrics

There are three ways to change the default CSV.

First, point at a CSV file directly. The flag is -f or --collectors, the environment variable is DCGM_EXPORTER_COLLECTORS, and the default is /etc/dcgm-exporter/default-counters.csv.

Second, use a YAML configuration file, via --config-file or DCGM_EXPORTER_CONFIG_FILE. The structure the README shows is this.

version: 1
metrics:
  file: /etc/dcgm-exporter/default-counters.csv
collection:
  interval: 30s

You can also list fields inline, in which case each entry uses the keys name, prometheusType, and help. For Kubernetes the README instructs you to mount custom metric ConfigMaps as files and set metrics.file to the mounted path. YAML is read only at startup, so edits require a restart.

Third, if you run the GPU Operator, specify it through chart values. The dcgmExporter.config block takes name, create, and data, and the chart comments state an important constraint: when pointing at an existing ConfigMap it must exist in the same namespace as the release, and the metrics are expected to be listed under a key called dcgm-metrics.csv.

# helm values example
dcgmExporter:
  config:
    name: custom-dcgm-exporter-metrics
    create: true
    data: |-
      DCGM_FI_DEV_GPU_UTIL,      gauge, GPU utilization (in %).
      DCGM_FI_DEV_FB_USED,       gauge, Framebuffer memory used (in MiB).
      DCGM_FI_PROF_SM_ACTIVE,    gauge, The ratio of cycles an SM has at least 1 warp assigned.
      DCGM_FI_PROF_SM_OCCUPANCY, gauge, The ratio of number of warps resident on an SM.
      DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, gauge, Ratio of cycles the tensor (HMMA) pipe is active.
      DCGM_FI_PROF_DRAM_ACTIVE,  gauge, Ratio of cycles the device memory interface is active sending or receiving data.

The collection interval is -c or --collect-interval, environment variable DCGM_EXPORTER_INTERVAL, in milliseconds, defaulting to 30000. The listen address is -a or DCGM_EXPORTER_LISTEN, defaulting to port 9400. The chart enables a ServiceMonitor by default at a 15 second scrape interval, which sits out of step with the 30 second collection interval, so you will scrape the same value twice in places.

Attaching Pod and Container Labels

Out of the box, the series carry only GPU identity labels. The labels confirmed in the source are gpu, UUID, pci_bus_id, device, modelName, and hostname, with GPU_I_PROFILE and GPU_I_ID added for MIG instances. None of that tells you which team is eating the card.

The flag that turns on Kubernetes mapping is --kubernetes or -k, environment variable DCGM_EXPORTER_KUBERNETES, default false. Enabling it adds the pod, namespace, and container labels. For the pod UID, enable --kubernetes-enable-pod-uid to get a pod_uid label; for pod labels, enable --kubernetes-enable-pod-labels, which prefixes the label keys with pod_label_.

Ingesting pod labels wholesale explodes cardinality, so filter with --kubernetes-pod-label-allowlist-regex, environment variable DCGM_EXPORTER_KUBERNETES_POD_LABEL_ALLOWLIST_REGEX. Leave it empty and every label is included. In the chart it surfaces like this.

dcgmExporter:
  enablePodLabels: true
  enablePodUID: true
  podLabelAllowlistRegex:
    - '^app$'
    - '^team$'

The chart comments append a warning. Turning on either one makes the operator create a cluster-scoped ClusterRole and binding granting get, list, and watch on pods. That is a permission widening, so flip the switch knowingly.

Three additions. The identifier used to match GPUs to pods is chosen with --kubernetes-gpu-id-type, whose values are uid and device-name, defaulting to uid. Capturing virtual GPU metrics from time-slicing or MPS needs --kubernetes-virtual-gpus, which adds a vgpu label. And enabling -o or DCGM_EXPORTER_USE_OLD_NAMESPACE switches the label names to the older form: pod_name, pod_namespace, container_name. That is a common reason queries in an inherited dashboard do not match.

Once pod labels are attached, questions like these become answerable.

# Framebuffer usage summed per namespace
sum by (namespace) (DCGM_FI_DEV_FB_USED)

# SM activity per pod (works only after adding SM_ACTIVE to the CSV)
avg by (namespace, pod) (DCGM_FI_PROF_SM_ACTIVE)

# Pods holding a GPU while barely using the SMs
avg by (namespace, pod) (DCGM_FI_PROF_SM_ACTIVE) < 0.2
  and on(namespace, pod) (avg by (namespace, pod) (DCGM_FI_DEV_FB_USED) > 1024)

Closing — Do Not Trust the Name, Read the Definition

Two things are worth actually enforcing. First, never put a lone utilization gauge on a dashboard. That number says whether a kernel was resident, not how much of the card is being used. It takes SM activity and framebuffer usage beside it before the picture is complete.

Second, put the fields you need into the CSV yourself. Not knowing that DCGM_FI_PROF_SM_ACTIVE and DCGM_FI_PROF_SM_OCCUPANCY are commented out in the default CSV means that however well you write the query, you stare at an empty graph because the data was never collected.

The next post climbs one layer up, to application metrics. The series vLLM exposes answer questions GPU metrics cannot: how many requests are waiting right now, and how long until the first token.

Try It Yourself

Series

References

Discuss on Twitter • View on GitHub