Split View: Rust로 쿠버네티스 GPU 오퍼레이터 만들기 — kube-rs로 실제 클러스터를 진단하다

Building a Kubernetes GPU Operator in Rust — Diagnosing a Real Cluster with kube-rs

• Introduction — An Operator in Rust, Not Go
• Part 1 — The CRD as a Rust Struct
• Part 2 — The Reconcile Loop: Scan Nodes and Write Status
• Part 3 — Two Traps I Actually Stepped On
  • Trap 1 — server-side apply's "invalid object type"
  • Trap 2 — Writing Status Calls Itself Again
• Part 4 — The Operator's Diagnosis: My GPU Fleet Was Dead
• Part 5 — Build and Runtime Environment
• Closing
• References

Introduction — An Operator in Rust, Not Go

Kubernetes operators are usually built in Go (controller-runtime). But as kube-rs has matured, you can now build them seriously in Rust too — type safety, a small binary, and above all the peace of mind the compiler gives you. This post is not theory but measurement: against a live 8-node homelab cluster (k8s v1.32.5), I wrote and launched an operator in Rust that investigates GPU status, and I record the real results the operator reported and the real traps I stepped on along the way.

One key design choice: the operator runs outside the cluster (out-of-cluster). Because kube::Client::try_default() reads ~/.kube/config, during development and debugging I attach to the API server with a local binary instead of baking a container image — this is the standard kube-rs development workflow.

Part 1 — The CRD as a Rust Struct

The heart of an operator is its custom resource. kube-rs turns a struct into a CRD with a single #[derive(CustomResource)].

#[derive(CustomResource, Deserialize, Serialize, Clone, Debug, JsonSchema)]
#[kube(
    group = "homelab.youngju.dev",
    version = "v1",
    kind = "GpuInventory",
    plural = "gpuinventories",
    shortname = "gpuinv",
    status = "GpuInventoryStatus"
)]
pub struct GpuInventorySpec {
    #[serde(default)]
    pub note: String,
}

#[derive(Deserialize, Serialize, Clone, Debug, JsonSchema, Default)]
pub struct GpuInventoryStatus {
    pub observed_at: String,
    pub total_gpu_nodes: i32,
    pub ready_gpu_nodes: i32,
    pub total_gpus: i32,
    pub nodes: Vec<GpuNodeEntry>,
}

This derive generates a method called GpuInventory::crd(), and at startup the operator applies it to the API server to install its own CRD by itself. No separate YAML is needed.

Part 2 — The Reconcile Loop: Scan Nodes and Write Status

The body of the controller is the reconcile function. Here it sweeps through every node, extracts the NVIDIA NFD labels (nvidia.com/gpu.product, gpu.count, gpu.memory) and the Ready status, and records them in the CR's status.

async fn reconcile_inventory(obj: Arc<GpuInventory>, ctx: Arc<Ctx>) -> Result<Action, Error> {
    let (status, _cm) = scan_cluster(&ctx.client).await?;   // iterate over all nodes
    let api: Api<GpuInventory> = Api::all(ctx.client.clone());
    let patch = serde_json::json!({
        "apiVersion": "homelab.youngju.dev/v1",   // ← these two lines were the trap (Part 3)
        "kind": "GpuInventory",
        "status": status
    });
    api.patch_status(&obj.name_any(), &PatchParams::apply(FIELD_MANAGER).force(),
                     &Patch::Apply(&patch)).await?;
    Ok(Action::requeue(Duration::from_secs(30)))
}

The operator launches two controllers in a single binary — (1) the loop above that reconciles the GpuInventory CR, and (2) a second loop that updates a ConfigMap named gpu-node-status on every node event. This is the archetype of an operator where "multiple controllers coexist in one process".

Part 3 — Two Traps I Actually Stepped On

This is the real meat of the post. At first it did not work.

Trap 1 — server-side apply's "invalid object type"

On the first run the reconcile scanned the nodes fine (the log shows gpu_nodes=4), but the moment it wrote the status, this error blew up:

WARN reconciled GpuInventory cr=cluster gpu_nodes=4 ready=0 gpus=1
WARN inventory reconcile failed: kube error: ApiError:
     invalid object type: /, Kind=: BadRequest (code: 400)

The cause: when writing the status subresource via server-side apply (Patch::Apply), the patch body must include apiVersion and kind. I had sent only { "status": status }, and the server threw a 400 asking "what type is this?" (/, Kind=). Adding those two lines (apiVersion, kind) from the Part 2 code fixed it right away. Go's controller-runtime fills these in automatically, but when you apply raw JSON in kube-rs you have to add them yourself.

Trap 2 — Writing Status Calls Itself Again

Once I fixed that, this time the reconcile ran away a dozen-plus times per second.

INFO reconciled GpuInventory cr=cluster ... (15:54:06.484)
INFO reconciled GpuInventory cr=cluster ... (15:54:06.525)
INFO reconciled GpuInventory cr=cluster ... (15:54:06.628)
INFO reconciled GpuInventory cr=cluster ... (15:54:06.728)   ← 100ms-interval runaway

The cause: the reconcile writes observed_at (the current time) fresh into the status every time, and that status write mutates the CR, which produces a watch event, which in turn calls reconcile again — a self-referential loop. Since the timestamp changes every time, it never stops. It is the trap operator beginners hit most often. The textbook solution is to "write the status only when meaningful data has actually changed" (compare against the current status and skip if identical). You will pick up this intuition quickly by running controller patterns by hand in the Kubernetes Playground.

Part 4 — The Operator's Diagnosis: My GPU Fleet Was Dead

Once I fixed both traps, the operator cleanly recorded the cluster's GPU status into the CR. But that result turned out to be an unexpected incident report.

$ kubectl get gpuinventory cluster -o yaml   (status excerpt)
observed_at     : 2026-07-10T15:54:10Z
total_gpu_nodes : 4
ready_gpu_nodes : 0        ← 0 Ready out of 4 GPU nodes
total_gpus      : 1
  nuc1   ready=false  NVIDIA-GeForce-RTX-4070-Laptop-GPU-SHARED  x1 mem=8188  vm-passthrough
  nuc2   ready=false  <unlabeled>                                x0           vm-passthrough
  omen   ready=false  <unlabeled>                                x0           vm-passthrough
  omen2  ready=false  <unlabeled>                                x0           container

The story reads in three layers. First, all four GPU nodes are NotReady — the cluster's entire GPU capability is offline. Second, the NotReady nodes have had their NFD labels wiped, so they show up as <unlabeled> (only the living nuc1 is caught as an RTX 4070). Third, only omen2 is container while the rest are vm-passthrough, splitting the workload label — this is the real-world instance of that label switch I described in the GPU Operator × KubeVirt post.

In other words, the operator I built discovered my cluster's failure on its own. This is the real value of writing an operator yourself — being told "GPU nodes abnormal" by someone else's dashboard versus watching a 20-line reconcile loop you wrote record that fact into the status are two different depths of understanding. (The cause was a kubelet swap problem, which I will cover in a separate post.)

Part 5 — Build and Runtime Environment

Here is the measured environment, recorded honestly.

Crates      kube 0.96 (client+runtime+derive+rustls-tls), k8s-openapi 0.23 (v1_31)
Build       cargo 1.97, first release build 16.7s (with dependencies), incremental 2.6s
Run         out-of-cluster, connect to the API (192.168.219.111:6443) via ~/.kube/config
Target      k8s v1.32.5, 8 nodes (cri-dockerd), GPU Operator v25.3.0

For TLS I chose rustls-tls over openssl — because it builds anywhere without a system openssl dependency (which is especially convenient when deploying the operator across multiple architectures).

Closing

Having written an operator in Rust, I found that kube-rs gives you a mental model strikingly similar to controller-runtime — CRD derivation, a reconcile function, requeue. What differs is that the compiler enforces the types, and that you have to handle low-level things like server-side apply yourself (Trap 1). What you get in exchange is the peace of mind that "if it runs, it is mostly correct," and what you give up is a few conveniences. Above all — when you write an operator yourself, it hands the truth of your infrastructure back to you. That my GPU fleet was dead, too, was first revealed to me by the reconcile loop I wrote. In the next post I will dig into the cause of that death (KubeVirt, GPU passthrough, kubelet) through the actual cluster state.

References

• kube-rs — Official Site and Docs
• kube-rs Controller/runtime Guide
• Kubernetes Server-Side Apply (the background behind invalid object type)
• NVIDIA GPU Operator Docs
• Node Feature Discovery (source of the nvidia.com/gpu.* labels)