💡 왼쪽 원문을 읽으면서 오른쪽에 따라 써보세요. Tab 키로 힌트를 받을 수 있습니다.

원문 렌더가 준비되기 전까지 텍스트 가이드로 표시합니다.

Introduction

In the previous article we built a working Operator with Kubebuilder. There, the reconcile function was a simple skeleton: "build the desired state and converge idempotently." But when you actually run an Operator in production, you realize that the subtle behavior of the reconcile loop governs stability and performance.

This article digs deep into the internals of the reconcile loop. It covers how a request travels from informer through the workqueue to reconcile, how to control retries with Result, how to implement idempotency robustly, how to avoid status conflicts, and how to tune concurrency and caching. This is based on controller-runtime v0.24.x.

The Reconcile Request Flow: informer to workqueue to reconcile

Understanding the path by which a reconcile function gets called is the starting point for everything. controller-runtime builds the following pipeline.

API Server

| watch (change stream)

+-----------+ +-------------+ +------------+

| sync) | | dedup) | +-----+------+

+-----------+ +-------------+ |

^ | on retry

| read local cache | requeue

+----------------------------------------+

The role of each stage is as follows.

- **Informer**: Watches the API Server and syncs objects into a local cache. When a change occurs, it fires an event. Thanks to this cache, reconcile can read objects without hitting the API Server every time.

- **WorkQueue**: Takes events and enqueues "keys of objects to reconcile" (namespace/name). The queue deduplicates (the same object enqueued multiple times is processed once) and controls the rate.

- **Reconciler**: Pulls a key from the queue and calls the reconcile function. Reconcile brings the object for that key to its desired state.

The key insight is that **what is passed to reconcile is not the object itself but only the object's key.** Reconcile must re-read the latest object by that key. This connects deeply to idempotency. What the event was (create/modify/delete) does not matter; only "what is the desired state now and what is the actual state" matters.

Controlling Retries with Result and requeue

The reconcile function returns `(Result, error)`. These two values determine what happens next.

return ctrl.Result{}, nil

-> Success. Not re-enqueued (waits until the next watch event)

return ctrl.Result{}, err

-> Error. The workqueue automatically retries with exponential backoff

return ctrl.Result{Requeue: true}, nil

-> Not an error, but request reprocessing (immediate re-enqueue)

return ctrl.Result{RequeueAfter: 30 * time.Second}, nil

-> Reconcile again after 30 seconds (useful for periodic checks)

This distinction matters because treating every "not yet" situation as an error floods the logs with errors. For example, if an external dependency is not ready yet, that is not an error but a "let's look again a bit later" situation. In such cases, use `RequeueAfter` instead of an error.

Wrong pattern:

if dependency_not_ready:

return Result{}, fmt.Errorf("not ready yet") # error-log explosion

Correct pattern:

if dependency_not_ready:

return Result{RequeueAfter: 10s}, nil # quietly check again

`RequeueAfter` is also useful for periodically checking "changes that watch alone does not catch." For example, time-dependent conditions like certificate expiry are re-checked at a fixed interval via RequeueAfter.

Idempotency Implementation Patterns

Idempotency is an absolute principle of reconcile. Even if reconcile is called dozens of times on the same object, the result must be the same. Let us look at two key patterns for implementing this robustly.

1. Server-side apply

The traditional "get then create if missing, update if present" pattern works, but conflicts arise easily when multiple controllers touch the same object. **Server-side apply (SSA)** is an approach where you declare "for these fields, the value I want is this," and the API Server merges field by field.

Advantages of server-side apply:

- The API Server tracks field ownership

- Updates only the fields I manage, preserving fields others manage

- Reduces the "read current state first and compare" boilerplate

- Idempotency is naturally guaranteed

With SSA, reconcile takes the form of "declaratively apply the whole desired object," making the code simpler and robust to concurrent modification.

2. Clarifying ownership with owner references

As emphasized in the previous article, attach an owner reference to child resources the controller creates. This gives two benefits. First, children are cleaned up automatically when the parent is deleted (garbage collection). Second, the controller manages only "what it owns," so the reconcile scope is clear. Without owner references, the controller must track what to clean up itself, which complicates finalizer logic.

Status Updates and Conditions

Treat spec and status separately

By Kubernetes API conventions, **spec is what the user/controller wants**, and **status is what the controller observed**. These two use different update paths. When the status subresource is enabled, status must be updated only with `Status().Update()`. Trying to write spec and status together in a single Update causes conflicts or being ignored.

Wrong:

obj.Spec.X = ...

obj.Status.Y = ...

Update(obj) # if the status subresource is on, status is ignored

Correct:

obj.Status.Y = ...

Status().Update(obj) # update status via a separate path

Standardized state with conditions

For status, using a **conditions** array is recommended over a simple phase string. A condition is a standard structure with `type`, `status` (True/False/Unknown), `reason`, `message`, and `lastTransitionTime`. For example, with types like `Ready`, `Progressing`, and `Degraded`, observability tools and users can interpret state in a consistent way.

Example conditions:

- type: Ready, status: "True", reason: AllReplicasReady

- type: Progressing, status: "False", reason: Stable

Conditions accumulate and update, and a condition of the same type updates lastTransitionTime only when its status changes. Following this convention forms the foundation for Capability Level 4 (Deep Insights).

Event Filters: predicate

By default, the controller reconciles on every change to its watched targets. But not every change is meaningful. For example, reconciling on an event where only status changed, or where only fields you do not care about changed, is wasteful. A **predicate** filters which events trigger reconcile.

Commonly used predicates:

- GenerationChangedPredicate

: Processes only when metadata.generation changed (= spec change only).

Useful for preventing self-triggered reconciles from status changes.

- LabelChangedPredicate / AnnotationChangedPredicate

: Interested only in specific metadata changes.

- Custom predicate

: Filter on arbitrary conditions.

`GenerationChangedPredicate` is especially important. When reconcile updates status, that itself creates a new watch event that can call reconcile again. Since generation increases only when spec changes, using this predicate prevents an unnecessary reconcile storm from status updates. (However, if your controller needs periodic checks via RequeueAfter, take care that this predicate does not block those checks.)

Rate Limiting and Concurrency

MaxConcurrentReconciles

By default, the controller runs only one reconcile at a time. If throughput is insufficient, you can raise concurrency.

Controller option:

MaxConcurrentReconciles: 5

-> reconcile runs in parallel across up to 5 workers

Note that the workqueue **never gives the same object key to two workers at once.** That is, reconciles for the same object are serialized, so even with higher concurrency there is no race condition for the same object. Different objects are processed in parallel, which raises throughput.

The rate limiter

The workqueue controls retry speed through a rate limiter. The default combines exponential backoff with an overall throughput limit. An object whose errors repeat is retried at progressively longer intervals, so one object's failure does not starve the whole queue. If your controller calls a downstream external API that could be overwhelmed, you can adjust the rate limiter to protect it.

Cache and Client: get from cache vs. API

controller-runtime's default client **reads from the cache and writes to the API Server.** Failing to understand this distinction leads to subtle bugs.

| Operation | Default behavior | Caveat |

| --- | --- | --- |

| Get/List | Reads from the informer cache | The cache may lag slightly (eventually consistent) |

| Create/Update/Delete | Directly to the API Server | Reflected immediately, but the cache updates a moment later |

Here is a common pitfall. If you Get an object right after you Create it, the cache may not have updated yet and return "not found." Therefore reconcile should be written idempotently on the premise that "the next reconcile will see it naturally," not "wait until what I just made appears in the cache." In rare cases where you truly need the latest value, you can use a direct-read client that bypasses the cache, but for performance the default is cache reads.

Observability: metrics and logging

A production Operator must expose its own state.

- **Metrics**: controller-runtime exposes Prometheus metrics by default, such as reconcile count, processing time, queue depth, and error count. A continuously growing queue depth signals insufficient throughput, and long reconcile times signal that external calls are the bottleneck.

- **Structured logging**: The logger obtained from `log.FromContext(ctx)` automatically includes context like the object key. Leaving key-value structured logs makes it easy to trace a specific object's reconcile flow.

- **Events**: Publish important reconciliation results as Kubernetes Events so they show up in `kubectl describe`.

As of 2026, the standard is to protect the metrics endpoint with controller-runtime's authentication/authorization middleware (WithAuthenticationAndAuthorization) without a separate sidecar.

Common Bugs

Here are bugs that recur in the reconcile loop.

| Bug | Cause | Fix |

| --- | --- | --- |

| Infinite reconcile | Status update creates a new event, self-triggering | Apply GenerationChangedPredicate, remove unnecessary status writes |

| Status conflict | Status().Update on a stale object from cache | Re-Get the latest object then update, or use SSA |

| "already exists" error | Non-idempotent create | Branch after get, or server-side apply |

| Just-created object not visible | Cache lag | Idempotent design, leave it to the next reconcile |

| One object's failure delays everything | Insufficient rate limiter/concurrency | Adjust MaxConcurrentReconciles |

| External API overwhelmed | Excessive requeue | Adjust the interval with RequeueAfter |

In particular, **infinite reconcile** and **status conflict** are rites of passage almost every Operator developer goes through once. Both stem from "how you handle status," so carefully designing the status path is key.

Performance Tuning Checklist

When performance problems arise, check in the following order.

1. **Look at metrics first**: Check queue depth, reconcile time, and error rate to pinpoint where the bottleneck is.

2. **Reduce unnecessary reconciles**: Filter meaningless events with predicates. In particular, block status self-triggering.

3. **Raise concurrency**: If throughput is insufficient, raise MaxConcurrentReconciles. The same object is serialized, so it is safe.

4. **Minimize external calls**: If you call a slow external API on every reconcile, that is the bottleneck. Cache it or lower the frequency with RequeueAfter.

5. **Leverage the cache**: Read from the cache, and use direct reads only when truly necessary.

6. **Minimize status writes**: Update status only when something actually changed, reducing unnecessary events and conflicts.

Expanding Watch Targets: What Triggers Reconcile

When reconcile is called is determined by what the controller watches. controller-runtime provides several ways to declare watch targets.

For(&MyKind{})

: The primary resource. Changes to this kind directly trigger reconcile.

Owns(&appsv1.Deployment{})

: A child resource I own (owner reference).

When this resource changes, it triggers reconcile of its owner (the primary resource).

Watches(&otherKind{}, handler)

: Watch an arbitrary resource that is not in an ownership relation.

A handler maps "which primary resource's reconcile this change should call."

`For` and `Owns` cover most controllers. But sometimes you must react to changes in resources that are not in an ownership relation. For example, if you must reconcile all CRs referencing a ConfigMap when that ConfigMap changes, you use `Watches` with a mapping function.

How the mapping function works

The mapping function (EnqueueRequestsFromMapFunc) takes "the changed object" as input and returns "a list of keys of primary resources to reconcile." That is, one external change can enqueue reconciles for several primary resources.

ConfigMap "shared-config" changes

-> mapping function called

-> returns the list of CRs referencing this ConfigMap: [cr-a, cr-b, cr-c]

-> three reconcile requests enter the queue

This pattern is powerful but requires care. If the mapping function returns too many primary resources, a single change can cause a reconcile storm. Keep the mapping function lightweight and precise.

Validating Reconcile with Tests

The subtle behavior of reconcile is most reliably caught with tests. Let us look at the typical flow of an envtest-based test.

test scenario:

1. create a CR

2. poll for a while, checking that the expected child resources appeared

3. change the CR's spec

4. check that the child resources updated to the new values

5. change a child resource by hand

6. check that reconcile reverts it to the desired value (self-healing check)

7. delete the CR

8. check that finalizer/owner reference cleanup works

The key is the **polling (eventually pattern).** Since reconcile runs asynchronously, a test must check whether the expected state is reached "within a moment," not "right now." Assert immediately without accounting for cache lag and async processing, and the test becomes flaky.

Unit tests vs. integration tests

| Type | Tool | What is verified |

| --- | --- | --- |

| Unit | fake client | Pure logic (builder functions, etc.) |

| Integration | envtest | API Server-level behavior (validation, status, GC) |

Pure functions (e.g., a builder that creates the desired Deployment) are verified quickly with a fake client or plain unit tests. Behaviors where the API Server is involved — the status subresource, CRD validation, owner reference garbage collection — must be verified with envtest to be realistic. Mixing the two appropriately is a good testing strategy.

The Worker Queue Internals and Backoff Behavior in Detail

Earlier we introduced the workqueue briefly, but knowing how backoff actually works helps a lot with debugging. controller-runtime's default rate limiter combines two mechanisms.

default rate limiter = per-item exponential backoff + overall bucket limit

per-item exponential backoff:

each time the same key fails, the wait doubles

e.g.: 5ms -> 10ms -> 20ms -> 40ms -> ... -> a max ceiling

overall bucket limit (token bucket):

a global cap on items processable per second

prevents a runaway queue from paralyzing the whole system

These two mechanisms combine so that even if a particular object keeps failing, only that object is retried more and more sparsely, while overall throughput stays stable. Importantly, when reconcile succeeds (returns without error), the backoff counter for that key is **reset**. So after a transient failure followed by success, the next failure again starts from a short wait.

The difference between backoff and RequeueAfter

Here is a point that is easy to confuse. Backoff from returning an error and RequeueAfter are different mechanisms.

| --- | --- | --- | --- |

Error backoff is "something went wrong, so retry more and more slowly," while RequeueAfter is "everything is fine, but check again after a set time." Confuse the two and treat a normal situation as an error, and logs get polluted and backoff accumulates, hurting responsiveness.

A Real-World Scenario: Designing a Staged Reconcile

A complex Operator's reconcile is usually split into multiple stages. Rather than trying to finish everything at once, splitting each stage idempotently is good design. Take a database cluster Operator as an example.

reconcile(cluster):

1. ensure finalizer (add if missing)

2. if deletionTimestamp present -> branch to cleanup

3. ensure Secret (password)

4. ensure ConfigMap (config)

5. ensure StatefulSet

6. ensure Service

7. check whether bootstrap is complete

if not, return RequeueAfter(10s)

8. elect/confirm primary

9. update status.conditions

return Result{}

Each "ensure" stage is idempotent. If it already exists, compare and update if needed; if not, create. The key is that **the stages are ordered.** You cannot make a StatefulSet without a Secret, so a stage proceeds only after the previous one finishes. When you hit a stage that is not yet ready, you express "continue a bit later" with RequeueAfter rather than an error.

Stage branching and early return

The advantage of this pattern is that reconcile always starts from the same entry point and re-judges "how far along it currently is" every time. Even if the controller dies and comes back mid-way, the next reconcile runs from the beginning again, skipping already-finished stages and continuing from the unfinished one. This is exactly why reconcile does not need to hold "how far it got" state in memory. The real state is always in the cluster (the observed state).

Example of early return:

if bootstrap_incomplete:

return Result{RequeueAfter: 10s}, nil # end here, continue next time

everything below runs only when bootstrap is done

Designed this way, even if the reconcile function looks long, each part is independent and idempotent, making it easy to reason about and test.

A Reconcile Mental Model

Let us compress everything covered so far into a single mental model — the questions you should always bring to mind when writing or debugging reconcile.

When reconcile is invoked, ask yourself:

1. "What is this object's desired state right now?" (desired)

2. "What is the actual cluster state?" (observed)

3. "What is the difference between them?" (diff)

4. "How do I close that gap idempotently?" (action)

5. "Is there a part I could not close yet?" (requeue decision)

6. "How do I reflect what I observed into status?" (status)

These six questions are the skeleton of reconcile. What the event was (create/modify/delete) is deliberately not asked. Reconcile always starts from "the truth of this moment." Internalize this mental model, and you will naturally know where to look when reading new Operator code or tracking a bug.

The thought flow when debugging

When reconcile misbehaves, usually one of the six steps above is broken.

| Symptom | Step to suspect |

| --- | --- |

| Nothing happens | desired computation or watch setup |

| Keeps repeating the same thing | idempotency (action) or status self-trigger |

| The gap is not closed | diff comparison logic or RBAC permissions |

| Status does not match | status update path or cache lag |

This habit of narrowing straight from symptom to suspected step turns vague debugging into systematic tracing.

Leader Election and High Availability

In production you run the controller manager with multiple replicas for availability. But if two replicas reconcile the same object at once, conflicts arise. **Leader election** prevents this.

leader election:

among multiple manager replicas, only one becomes the "leader" and runs reconcile

the leader periodically renews its leadership via a Lease object

if the leader dies, the Lease expires and another replica becomes leader

controller-runtime can enable leader election with a single option. When enabled, normally only one replica actually works while the rest stand by. If the leader disappears due to failure, another replica quickly takes over and continues reconcile. This lets you run the controller without a single point of failure.

Note that even with leader election on, the idempotency of reconcile itself remains important. There is an edge case where the same object can be processed twice at the moment of a leader handover. Idempotency is the foundation of every safeguard.

A Production Operations Checklist

Checking the following before putting the reconcile loop into production prevents common incidents.

[ ] Is reconcile idempotent? (is repeated execution on the same input safe)

[ ] Is an owner reference set on all child resources?

[ ] If it handles external resources, is there a finalizer?

[ ] Is status updated only via Status().Update?

[ ] Is there a predicate to block status self-triggering?

[ ] Are errors and "not yet" handled distinctly? (use RequeueAfter)

[ ] Can you observe behavior with metrics and logging?

[ ] Is multi-replica safety ensured with leader election?

[ ] Do external API calls have timeouts and retry limits?

[ ] Does RBAC follow the principle of least privilege?

This checklist is also a summary of all the concepts covered earlier. Idempotency, owner references, finalizers, the status path, predicates, error handling, observability, high availability, security — a robust Operator is the result of attending to all of these without gaps.

Conclusion

The reconcile loop looks like a simple function, but behind it informer, workqueue, rate limiter, and cache mesh together precisely. To build a robust Operator you must understand this pipeline, write reconcile idempotently, handle the status path carefully, reduce unnecessary work with predicates, and observe behavior with metrics.

The core principle in one sentence: **"Reconcile does not ask what the event was. It merely compares the current desired state with the actual state and converges idempotently."** Hold to this mindset and you will naturally avoid most infinite loops, status conflicts, and performance pitfalls.

References

- [controller-runtime (pkg.go.dev)](https://pkg.go.dev/sigs.k8s.io/controller-runtime)

- [controller-runtime predicate package](https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/predicate)

- [Kubebuilder Book — How reconcile works](https://book.kubebuilder.io/cronjob-tutorial/controller-implementation.html)

- [Kubernetes API conventions (spec/status)](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md)

- [Server-side apply (Kubernetes official docs)](https://kubernetes.io/docs/reference/using-api/server-side-apply/)

- [Operator pattern (Kubernetes official docs)](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/)

- [Operator SDK documentation](https://sdk.operatorframework.io/)

- [kubernetes-sigs/controller-runtime (GitHub)](https://github.com/kubernetes-sigs/controller-runtime)

- [Operator Capability Levels](https://sdk.operatorframework.io/docs/overview/operator-capabilities/)