Split View: 세 줄짜리 설정 파일이 GPU 4장을 일주일 동안 죽였다 — containerd 드롭인 병합의 진실

A Three-Line Config File Killed Four GPUs for a Week — How containerd Drop-in Merging Really Works

• Opening — the file is there, the config is not
• First misdiagnosis — "the main config overrides the drop-in"
• Second misdiagnosis — "version = 2 is missing"
• The experiment — take the drop-ins away one at a time
• The rule — merging is not field by field
• Why nobody noticed for a week
• The fix — same result no matter how many times it runs
• Aftershock — containerd is alive but the node is dead
• Verification — with a pod, not a number
• So what should you actually do
• Summary
• 🧠 Comprehension Check Quiz
• References

Opening — the file is there, the config is not

In the previous post I brought up the GPU Operator and confirmed that its configuration lands not in config.toml but in a drop-in file, conf.d/99-nvidia.toml. I closed that post like this.

This design is deliberate. Since it never mixes with the configuration the host already had, removing the Operator only means deleting the drop-in file, and the host goes back to its original state.

That is correct. One condition was missing, though. It holds only when that drop-in is the only one.

A week later, all four GPU nodes stopped advertising GPUs.

$ kubectl get nodes -o custom-columns="NODE:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu"
NODE     GPU
nuc1     0
nuc2     0
omen     0
omen2    0

Lab pods could not start. And yet the file looks perfectly fine.

$ ssh omen 'grep -c "runtimes.nvidia" /etc/containerd/conf.d/99-nvidia.toml'
6

All three runtimes — nvidia, nvidia-cdi, nvidia-legacy — are spelled out correctly, down to BinaryName. The file is there.

First misdiagnosis — "the main config overrides the drop-in"

I looked at the loaded configuration.

$ ssh omen 'containerd config dump | grep -n "runtimes"'
111:      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
113:        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
128:          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]

Only runc. And line 111 caught my eye. The main config.toml was defining the runtimes table directly.

A plausible hypothesis fell into place immediately. TOML merging happens at the table level, and since the main file already holds that table, the drop-in's copy of the same table gets thrown away. So I told people to put the nvidia block directly into the main file.

Wrong. And that instruction took a node down.

containerd: failed to load TOML: /etc/containerd/config.toml: (300, 2): duplicated tables
Job for containerd.service failed because the control process exited with error code.

I had handed out a command that appends with >>, and if you paste it twice the same table gets declared twice. TOML treats that as a syntax error, and containerd refuses to start. The whole node went down.

The first lesson arrived before the diagnosis did. A command you hand someone to run on a production node has to produce the same result when it runs twice. People paste things twice.

Second misdiagnosis — "version = 2 is missing"

Even after clearing out the duplicate, nvidia still would not show up. Going back through conf.d, a second file caught my attention.

$ ssh omen 'ls /etc/containerd/conf.d/'
99-nvidia.toml
zz-labhub-registry.toml

zz-labhub-registry.toml is three lines long. I added it a week earlier to point at a certificate directory, because the private registry serves plain HTTP.

# LabHub: trust the Harbor (HTTP) registry
[plugins."io.containerd.grpc.v1.cri".registry]
  config_path = "/etc/containerd/certs.d"

What stood out was the missing version = 2 line. There is a well-known story that when the version marker is absent from a containerd config, it gets treated as v1 and plugin keys are interpreted differently. A second hypothesis formed, and this time I measured before applying it.

$ # dump using a copy of the zz file with version = 2 added
$ containerd --config /tmp/t2.toml config dump 2>/dev/null | grep -c "runtimes.nvidia"
0

Wrong again.

Both hypotheses were plausible and both were wrong. That is where I changed my approach. I stopped reading config files and reasoning about them, and decided to remove them one at a time and watch whether the result changed.

The experiment — take the drop-ins away one at a time

containerd --config FILE config dump lets you feed containerd an arbitrary config file without touching the running process. Which means you can measure on a production node while changing nothing.

$ for f in 99-nvidia zz-labhub-registry; do
    printf 'version = 2\nimports = ["/etc/containerd/conf.d/%s.toml"]\n' $f > /tmp/t.toml
    echo -n "import $f only → "
    containerd --config /tmp/t.toml config dump 2>/dev/null | grep -c "runtimes.nvidia"
  done
import 99-nvidia only → 6
import zz-labhub-registry only → 0

$ # both
$ printf 'version = 2\nimports = ["/etc/containerd/conf.d/*.toml"]\n' > /tmp/t3.toml
$ containerd --config /tmp/t3.toml config dump 2>/dev/null | grep -c "runtimes.nvidia"
0

The third line is the answer.

Load 99-nvidia.toml alone and the nvidia runtime shows up six times. Load the three-line registry file alongside it and you get 0. A file that does not say a single word about runtimes wiped out three runtimes.

The rule — merging is not field by field

containerd's imports merging is wholesale per-plugin replacement.

If a drop-in touches any part of a plugin's configuration, that plugin's entire configuration is replaced with the contents of that file. Anything the drop-in does not spell out does not fall back to the value from the earlier file — it becomes the default.

Drop-ins are read in name order. So in the end, the last file that mentions a plugin takes all of it.

zz- sorts after 99-. Which is how these three lines replaced the entire CRI plugin configuration.

load order         CRI plugin config
─────────────────  ────────────────────────────────────
config.toml        runc, sandbox_image, cgroup settings …
99-nvidia.toml     runc + nvidia ×3, certs.d …          ← replaces the above wholesale
zz-registry.toml   config_path, nothing else            ← replaces it wholesale again
─────────────────  ────────────────────────────────────
final              config_path + everything else default

Put the two files side by side and merge them in your head and you count four runtimes. What actually got loaded was one.

Why nobody noticed for a week

The genuinely frightening part of this failure is not the merge rule. It is that there are no symptoms.

Here is the config dump from the broken state again.

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"

sandbox_image = "registry.k8s.io/pause:3.8"

runc is there. sandbox_image looks plausible. The configuration looks alive.

All of it is defaults. Not one of the values we wrote survived, but the defaults look close enough to what we would have written that nothing stands out. pause:3.8 is exactly what kubeadm uses, and runc is the default runtime anyway.

And the runtime handler is looked up only when a container is created. GPU pods that were already running keep running fine even after the configuration disappears. So the breakage is not visible right away — it all detonates at once the moment you reboot a node or the pods get recreated.

Those two things combined bought a week.

The fix — same result no matter how many times it runs

Once you know the cause the fix is simple. Move zz-labhub-registry.toml out of conf.d. The config_path value that file was holding is already in 99-nvidia.toml.

This time, though, I did not hand out a one-line command. I had already taken a node down that way. I wrote a script instead, and gave it two properties.

First, it does not append. It strips out everything that is there and then writes exactly one copy back. Zero previous appends, one, or two — the result is the same.

def strip_nvidia(lines):
    """Strip out the nvidia table and all of its sub-tables."""
    out, cur, dropped = [], "", 0
    for line in lines:
        h = header_of(line)
        if h is not None:
            cur = h
        if cur == NV or cur.startswith(NV + "."):
            dropped += 1
            continue
        out.append(line)
    return out, dropped

Second, it touches the live file last. It builds the candidate configuration as a temporary file, has containerd read that file to confirm nvidia and certs.d actually come through, and only then swaps it in. If verification fails, the original is not changed by a single character.

rc, out, err = dump(candidate)          # containerd --config <candidate> config dump
if rc != 0 or out.count("runtimes.nvidia") == 0:
    print("The candidate config does not pick up the nvidia runtime. Leaving the original alone.")
    return 1
if "/etc/containerd/certs.d" not in out:
    print("The registry config would disappear. Leaving the original alone.")
    return 1
# from here on we actually change things

The second check is the important one. It deletes the file only after confirming that some other file carries the configuration the doomed file was holding. Delete without that check and I would have revived the GPUs and cut off image pulls.

Aftershock — containerd is alive but the node is dead

I applied it to all four nodes and restarted. Three came straight back, and one stayed NotReady.

$ ssh omen2 'systemctl is-active containerd'
active

$ kubectl describe node omen2 | grep -A1 "Ready "
  Ready   False   KubeletNotReady   container runtime is down

containerd is active but kubelet says the runtime is dead. The logs have the answer.

level=warning msg="failed to load plugin io.containerd.grpc.v1.cri"
  error="failed to create CRI service: failed to create cni conf monitor for default:
         failed to create fsnotify watcher: too many open files"
level=info msg="containerd successfully booted in 0.417476s"

The process came up, and only the CRI plugin failed to load. The default fs.inotify.max_user_instances of 128 had been exhausted. Look at systemctl status alone and the state appears healthy.

$ sudo sh -c 'printf "fs.inotify.max_user_instances = 8192\nfs.inotify.max_user_watches = 1048576\n" \
    > /etc/sysctl.d/99-inotify.conf && sysctl -p /etc/sysctl.d/99-inotify.conf'
$ sudo systemctl restart containerd

All five nodes were on the default of 128 with no config file at all. omen2 just blew up first; the rest were only lucky. On a Kubernetes node you are better off raising this ahead of time.

Verification — with a pod, not a number

$ kubectl get nodes -o custom-columns="NODE:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu"
NODE     GPU
nuc1     1
nuc2     1
omen     1
omen2    1

Do not stop here. There is a state where the advertisement works but pods still will not start. Actually run one.

$ kubectl -n gpu-operator logs gputest
GPU 0: NVIDIA GeForce RTX 4070 Laptop GPU (UUID: GPU-53be3556-a942-531c-a9a5-20e92af45279)

The pod schedules, the sandbox is created with the nvidia runtime, and nvidia-smi inside the container sees the card. You are only done once you get this far.

So what should you actually do

One. Do not draw conclusions from reading files. What cat shows you is what somebody intended to write; what config dump shows you is what actually got loaded. Most GPU configuration incidents come from those two differing while only the former gets looked at.

Two. Even when reading the dump, tell your own values apart from defaults. That is why this incident stayed hidden for a week. The only way to tell them apart is to remove things one at a time and see whether the result changes.

Three. Look at the name before you drop one more file into conf.d. If that file sorts last and mentions any plugin, it becomes responsible for that plugin's entire configuration. Write only three lines and everything else becomes a default.

Four. Commands you hand out for production nodes have to be the same when run twice. People paste things twice. I handed out a command that could not survive that, and it took a node down.

Five. For irreversible changes, touch the live target last. Build a candidate, verify it, and swap only if verification passes. This ordering makes the script a few lines longer and drives the cost of being wrong to zero.

Summary

The lesson that sticks longest is about the diagnostic method. Both wrong diagnoses were hypotheses built by reading config files, and what gave me the answer was an experiment that removed things one at a time and measured the result. Confidence gained by reading is not verification.

🧠 Comprehension Check Quiz

References

• containerd — configuration file docs
• containerd — imports and drop-ins
• NVIDIA Container Toolkit — architecture overview
• NVIDIA GPU Operator official docs
• Kubernetes docs — Runtime Class
• Linux inotify manual