필사 모드: Building a Kubernetes GPU Operator in Rust — Diagnosing a Real Cluster with kube-rs
English- Introduction — An Operator in Rust, Not Go
- Part 1 — The CRD as a Rust Struct
- Part 2 — The Reconcile Loop: Scan Nodes and Write Status
- Part 3 — Two Traps I Actually Stepped On
- Part 4 — The Operator's Diagnosis: My GPU Fleet Was Dead
- Part 5 — Build and Runtime Environment
- Closing
- References
Introduction — An Operator in Rust, Not Go
Kubernetes operators are usually built in Go (controller-runtime). But as kube-rs has matured, you can now build them seriously in Rust too — type safety, a small binary, and above all the peace of mind the compiler gives you. This post is not theory but measurement: against a live 8-node homelab cluster (k8s v1.32.5), I wrote and launched an operator in Rust that investigates GPU status, and I record the real results the operator reported and the real traps I stepped on along the way.
One key design choice: the operator runs outside the cluster (out-of-cluster). Because kube::Client::try_default() reads ~/.kube/config, during development and debugging I attach to the API server with a local binary instead of baking a container image — this is the standard kube-rs development workflow.
Part 1 — The CRD as a Rust Struct
The heart of an operator is its custom resource. kube-rs turns a struct into a CRD with a single #[derive(CustomResource)].
#[derive(CustomResource, Deserialize, Serialize, Clone, Debug, JsonSchema)]
#[kube(
group = "homelab.youngju.dev",
version = "v1",
kind = "GpuInventory",
plural = "gpuinventories",
shortname = "gpuinv",
status = "GpuInventoryStatus"
)]
pub struct GpuInventorySpec {
#[serde(default)]
pub note: String,
}
#[derive(Deserialize, Serialize, Clone, Debug, JsonSchema, Default)]
pub struct GpuInventoryStatus {
pub observed_at: String,
pub total_gpu_nodes: i32,
pub ready_gpu_nodes: i32,
pub total_gpus: i32,
pub nodes: Vec<GpuNodeEntry>,
}
This derive generates a method called GpuInventory::crd(), and at startup the operator applies it to the API server to install its own CRD by itself. No separate YAML is needed.
Part 2 — The Reconcile Loop: Scan Nodes and Write Status
The body of the controller is the reconcile function. Here it sweeps through every node, extracts the NVIDIA NFD labels (nvidia.com/gpu.product, gpu.count, gpu.memory) and the Ready status, and records them in the CR's status.
async fn reconcile_inventory(obj: Arc<GpuInventory>, ctx: Arc<Ctx>) -> Result<Action, Error> {
let (status, _cm) = scan_cluster(&ctx.client).await?; // iterate over all nodes
let api: Api<GpuInventory> = Api::all(ctx.client.clone());
let patch = serde_json::json!({
"apiVersion": "homelab.youngju.dev/v1", // ← these two lines were the trap (Part 3)
"kind": "GpuInventory",
"status": status
});
api.patch_status(&obj.name_any(), &PatchParams::apply(FIELD_MANAGER).force(),
&Patch::Apply(&patch)).await?;
Ok(Action::requeue(Duration::from_secs(30)))
}
The operator launches two controllers in a single binary — (1) the loop above that reconciles the GpuInventory CR, and (2) a second loop that updates a ConfigMap named gpu-node-status on every node event. This is the archetype of an operator where "multiple controllers coexist in one process".
Part 3 — Two Traps I Actually Stepped On
This is the real meat of the post. At first it did not work.
Trap 1 — server-side apply's "invalid object type"
On the first run the reconcile scanned the nodes fine (the log shows gpu_nodes=4), but the moment it wrote the status, this error blew up:
WARN reconciled GpuInventory cr=cluster gpu_nodes=4 ready=0 gpus=1
WARN inventory reconcile failed: kube error: ApiError:
invalid object type: /, Kind=: BadRequest (code: 400)
The cause: when writing the status subresource via server-side apply (Patch::Apply), the patch body must include apiVersion and kind. I had sent only { "status": status }, and the server threw a 400 asking "what type is this?" (/, Kind=). Adding those two lines (apiVersion, kind) from the Part 2 code fixed it right away. Go's controller-runtime fills these in automatically, but when you apply raw JSON in kube-rs you have to add them yourself.
Trap 2 — Writing Status Calls Itself Again
Once I fixed that, this time the reconcile ran away a dozen-plus times per second.
INFO reconciled GpuInventory cr=cluster ... (15:54:06.484)
INFO reconciled GpuInventory cr=cluster ... (15:54:06.525)
INFO reconciled GpuInventory cr=cluster ... (15:54:06.628)
INFO reconciled GpuInventory cr=cluster ... (15:54:06.728) ← 100ms-interval runaway
The cause: the reconcile writes observed_at (the current time) fresh into the status every time, and that status write mutates the CR, which produces a watch event, which in turn calls reconcile again — a self-referential loop. Since the timestamp changes every time, it never stops. It is the trap operator beginners hit most often. The textbook solution is to "write the status only when meaningful data has actually changed" (compare against the current status and skip if identical). You will pick up this intuition quickly by running controller patterns by hand in the Kubernetes Playground.
Part 4 — The Operator's Diagnosis: My GPU Fleet Was Dead
Once I fixed both traps, the operator cleanly recorded the cluster's GPU status into the CR. But that result turned out to be an unexpected incident report.
$ kubectl get gpuinventory cluster -o yaml (status excerpt)
observed_at : 2026-07-10T15:54:10Z
total_gpu_nodes : 4
ready_gpu_nodes : 0 ← 0 Ready out of 4 GPU nodes
total_gpus : 1
nuc1 ready=false NVIDIA-GeForce-RTX-4070-Laptop-GPU-SHARED x1 mem=8188 vm-passthrough
nuc2 ready=false <unlabeled> x0 vm-passthrough
omen ready=false <unlabeled> x0 vm-passthrough
omen2 ready=false <unlabeled> x0 container
The story reads in three layers. First, all four GPU nodes are NotReady — the cluster's entire GPU capability is offline. Second, the NotReady nodes have had their NFD labels wiped, so they show up as <unlabeled> (only the living nuc1 is caught as an RTX 4070). Third, only omen2 is container while the rest are vm-passthrough, splitting the workload label — this is the real-world instance of that label switch I described in the GPU Operator × KubeVirt post.
In other words, the operator I built discovered my cluster's failure on its own. This is the real value of writing an operator yourself — being told "GPU nodes abnormal" by someone else's dashboard versus watching a 20-line reconcile loop you wrote record that fact into the status are two different depths of understanding. (The cause was a kubelet swap problem, which I will cover in a separate post.)
Part 5 — Build and Runtime Environment
Here is the measured environment, recorded honestly.
Crates kube 0.96 (client+runtime+derive+rustls-tls), k8s-openapi 0.23 (v1_31)
Build cargo 1.97, first release build 16.7s (with dependencies), incremental 2.6s
Run out-of-cluster, connect to the API (192.168.219.111:6443) via ~/.kube/config
Target k8s v1.32.5, 8 nodes (cri-dockerd), GPU Operator v25.3.0
For TLS I chose rustls-tls over openssl — because it builds anywhere without a system openssl dependency (which is especially convenient when deploying the operator across multiple architectures).
Closing
Having written an operator in Rust, I found that kube-rs gives you a mental model strikingly similar to controller-runtime — CRD derivation, a reconcile function, requeue. What differs is that the compiler enforces the types, and that you have to handle low-level things like server-side apply yourself (Trap 1). What you get in exchange is the peace of mind that "if it runs, it is mostly correct," and what you give up is a few conveniences. Above all — when you write an operator yourself, it hands the truth of your infrastructure back to you. That my GPU fleet was dead, too, was first revealed to me by the reconcile loop I wrote. In the next post I will dig into the cause of that death (KubeVirt, GPU passthrough, kubelet) through the actual cluster state.
References
현재 단락 (1/71)
Kubernetes operators are usually built in Go (controller-runtime). But as [kube-rs](https://kube.rs/...