Skip to content
Published on

CRD Design and Versioning — Schema, Validation, and Conversion Webhooks

🇰🇷KO🇺🇸EN🇯🇵JA
Split View필사 모드
Authors

Introduction — A CRD Is an API Contract, Not Code

When we build our first operator, we tend to focus on the controller logic: how to write the reconcile loop, how to call external systems. Over time, though, what trips us up is rarely the controller. It is the CRD itself.

The reason is simple. A CRD is the schema for the YAML your users write by hand, and once it is deployed and objects start accumulating in the cluster, that schema effectively becomes a permanent public API contract. Controller code can be refactored and redeployed at any time, but the thousands of custom resources already stored in etcd and the manifests users have committed to their GitOps repositories cannot be changed casually.

That is why CRD design deserves the same weight as designing a library API. Get one field wrong and it will follow you through v1alpha1, v1beta1, and v1, accumulating compatibility burden for years. This article, using the 2026 tooling stack (Kubebuilder, Kubernetes 1.36 / Go 1.26, controller-runtime v0.24.x, controller-tools v0.21.x), covers how to design and version CRDs that survive from day one.

Here is what we will cover.

  • OpenAPI v3 schema design principles and CEL-based validation
  • Multi-version strategy from v1alpha1 to v1, and the storage version
  • Hub-and-Spoke conversion webhook implementation
  • Backward compatibility, deprecation, and field evolution strategy
  • additionalPrinterColumns and subresources (status/scale)
  • Performance considerations for large-scale CRDs
  • Declarative API design best practices, migration, and pitfalls

CRD Schema Design Principles

OpenAPI v3 Structural Schema

Since Kubernetes 1.16, every CRD requires a structural schema. This means every field must have its type declared in OpenAPI v3, and you must not abuse escape hatches like x-kubernetes-preserve-unknown-fields that allow arbitrary key-value pairs. A structural schema is what makes server-side apply, field pruning, and the CEL validation discussed below actually work.

With Kubebuilder you annotate your Go types with markers, and the schema is generated automatically. Here are the type definitions for a hypothetical database operator.

package v1beta1

import (
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// DatabaseSpec is the desired state declared by the user.
type DatabaseSpec struct {
	// Engine is the database engine to use.
	// It cannot be changed after creation.
	// +kubebuilder:validation:Enum=postgres;mysql;mariadb
	// +kubebuilder:validation:Required
	Engine string `json:"engine"`

	// Version is the engine version.
	// +kubebuilder:validation:Required
	// +kubebuilder:validation:Pattern=`^[0-9]+\.[0-9]+$`
	Version string `json:"version"`

	// Replicas is the number of replicas. Defaults to 1.
	// +kubebuilder:validation:Minimum=1
	// +kubebuilder:validation:Maximum=9
	// +kubebuilder:default=1
	Replicas int32 `json:"replicas,omitempty"`

	// StorageGB is the per-instance storage size in GB.
	// +kubebuilder:validation:Minimum=10
	// +kubebuilder:default=20
	StorageGB int32 `json:"storageGB,omitempty"`

	// BackupPolicy is an optional backup configuration.
	// +optional
	BackupPolicy *BackupPolicy `json:"backupPolicy,omitempty"`
}

type BackupPolicy struct {
	// Schedule is the backup interval in cron format.
	// +kubebuilder:validation:Required
	Schedule string `json:"schedule"`

	// RetentionDays is how many days to keep backups.
	// +kubebuilder:validation:Minimum=1
	// +kubebuilder:default=7
	RetentionDays int32 `json:"retentionDays,omitempty"`
}

Notice that each marker is translated into the OpenAPI schema of the generated CRD. +kubebuilder:default=1 becomes default: 1 in the schema, and the API server fills it in automatically during admission.

The Meaning of required, default, and optional

You must distinguish three things clearly.

CategoryMarkerBehaviorWhen to use
Required+kubebuilder:validation:RequiredRejected if emptyCore fields with no meaningful default
Default+kubebuilder:default=NAuto-filled if emptyFields with a sensible default
Optional+optionalAllowed empty, never filledGenuinely optional features

Do not make a field Required if it could have a default. You add boilerplate that users must repeat every time, and it becomes hard to make the field optional later. Conversely, core identifiers (like engine above) should have no default. An explicit rejection is better than the wrong resource being created from a bad default.

Immutable Fields and CEL Validation

A field like engine must not change after creation. Switching an engine from postgres to mysql is no different from creating a new resource. In the past you had to enforce this immutability in a validating webhook, but now you can declare it inside the schema with CEL (Common Expression Language) validation (x-kubernetes-validations).

// +kubebuilder:validation:XValidation:rule="self == oldSelf",message="engine is immutable"
Engine string `json:"engine"`

CEL goes beyond simple immutability and can express relationships between fields. Here are compound rules applied to the whole type.

// +kubebuilder:validation:XValidation:rule="self.engine != 'mariadb' || self.replicas == 1",message="mariadb supports only a single replica"
// +kubebuilder:validation:XValidation:rule="!has(self.backupPolicy) || self.replicas >= 1",message="at least 1 replica is required to enable backups"
type DatabaseSpec struct {
	// ... fields ...
}

CEL validation runs inside the API server without a webhook, so there is no separate server to operate and no network hop. As of 2026, almost all synchronous validation logic can be moved into CEL, and validating webhooks are best reserved for cases that require querying an external system.

The generated CRD schema looks like this.

openAPIV3Schema:
  type: object
  properties:
    spec:
      type: object
      required:
        - engine
        - version
      properties:
        engine:
          type: string
          enum: [postgres, mysql, mariadb]
          x-kubernetes-validations:
            - rule: "self == oldSelf"
              message: "engine is immutable"
        replicas:
          type: integer
          minimum: 1
          maximum: 9
          default: 1
      x-kubernetes-validations:
        - rule: "self.engine != 'mariadb' || self.replicas == 1"
          message: "mariadb supports only a single replica"

Multi-Version Strategy

Why You Need Multiple Versions

APIs evolve. In v1alpha1 you realize "this field should really have been an object," in v1beta1 you restructure it, and in v1 you stabilize. The Kubernetes API conventions provide clear stages for this evolution.

VersionMeaningCompatibility promiseRecommended use
v1alpha1ExperimentalMay break anytime, removed without noticeGathering early feedback
v1beta1BetaReasonable deprecation window guaranteedProduction trial adoption
v1StableStrong backward compatibility promiseGeneral availability

There is one core rule. All versions of a single CRD must convert losslessly to one another. Only one representation (the storage version) is stored in etcd, and requests for other versions go through conversion.

Storage Version and Served Version

In a CRD each version carries two flags.

  • served: should the API accept requests for this version
  • storage: should this version be stored in etcd (exactly one is true)

Here is part of a CRD that serves three versions while storing as v1beta1.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: databases.example.com
spec:
  group: example.com
  names:
    kind: Database
    plural: databases
  scope: Namespaced
  versions:
    - name: v1alpha1
      served: true
      storage: false
      schema:
        openAPIV3Schema: { }
    - name: v1beta1
      served: true
      storage: true
      schema:
        openAPIV3Schema: { }
    - name: v1
      served: false
      storage: false
      schema:
        openAPIV3Schema: { }

The flow of requests and storage looks like this.

User does GET as v1alpha1
        │
        ▼
  API server reads from etcd as v1beta1 (storage)
        │
        ▼
  conversion webhook: v1beta1 → v1alpha1
        │
        ▼
  v1alpha1 response to the user


User does CREATE as v1
        │
        ▼
  conversion webhook: v1 → v1beta1 (storage)
        │
        ▼
  stored in etcd as v1beta1

Version Promotion Flow

The standard procedure for introducing a new version and retiring an old one is as follows.

Step 1: Add v1beta1 (served=true, storage=false)
        storage is still v1alpha1
        │
        ▼
Step 2: Switch storage to v1beta1 (storage=true)
        re-store existing objects with storage-version-migrator
        │
        ▼
Step 3: Set v1alpha1 to served=false
        no longer accepts new requests
        │
        ▼
Step 4: Remove v1alpha1 from CRD status.storedVersions
        only after all objects are re-stored as v1beta1
        │
        ▼
Step 5: Delete the v1alpha1 schema definition

The most common mistake here is skipping the re-storage (storage migration) in Step 2. If you only change the storage version without rewriting existing objects, etcd still holds objects stored in the old version. Later, when you delete the old version schema, those objects become unreadable.

Conversion Webhook Implementation

Conversion Strategy: Hub-and-Spoke

With N versions, converting between every pair requires N×(N-1) conversion functions. controller-runtime offers the Hub-and-Spoke pattern to reduce this. You designate one version as the Hub, and the others (Spokes) only implement conversion to and from the Hub. Conversion between any two versions then always goes through the Hub.

   v1alpha1 (Spoke)        v1 (Spoke)
        │                     │
        │  ConvertTo/From     │  ConvertTo/From
        ▼                     ▼
        └────► v1beta1 (Hub) ◄┘

The Hub is usually the storage version. Routing conversion through storage reduces the extra conversion cost.

Defining the Hub Version

The Hub only needs to implement a single marker interface.

package v1beta1

import "sigs.k8s.io/controller-runtime/pkg/conversion"

// Hub marks this version as the conversion hub.
func (*Database) Hub() {}

// Ensure interface satisfaction at compile time.
var _ conversion.Hub = (*Database)(nil)

ConvertTo / ConvertFrom on the Spoke Version

The Spoke version (v1alpha1) implements conversion to the Hub and from the Hub.

package v1alpha1

import (
	"sigs.k8s.io/controller-runtime/pkg/conversion"
	v1beta1 "example.com/api/v1beta1"
)

// ConvertTo converts v1alpha1 to the Hub (v1beta1).
func (src *Database) ConvertTo(dstRaw conversion.Hub) error {
	dst := dstRaw.(*v1beta1.Database)

	// Copy metadata as-is.
	dst.ObjectMeta = src.ObjectMeta

	// Fields that stay the same.
	dst.Spec.Engine = src.Spec.Engine
	dst.Spec.Version = src.Spec.Version
	dst.Spec.Replicas = src.Spec.Replicas

	// Assume v1alpha1 storageGB (int) is identical in v1beta1.
	dst.Spec.StorageGB = src.Spec.StorageGB

	// Field absent in v1alpha1: fill with a sensible default.
	if src.Spec.Backup != "" {
		dst.Spec.BackupPolicy = &v1beta1.BackupPolicy{
			Schedule:      src.Spec.Backup,
			RetentionDays: 7,
		}
	}

	dst.Status.Phase = src.Status.Phase
	return nil
}

// ConvertFrom converts the Hub (v1beta1) to v1alpha1.
func (dst *Database) ConvertFrom(srcRaw conversion.Hub) error {
	src := srcRaw.(*v1beta1.Database)

	dst.ObjectMeta = src.ObjectMeta
	dst.Spec.Engine = src.Spec.Engine
	dst.Spec.Version = src.Spec.Version
	dst.Spec.Replicas = src.Spec.Replicas
	dst.Spec.StorageGB = src.Spec.StorageGB

	// Collapse the structured backup of v1beta1 into the simple string of v1alpha1.
	if src.Spec.BackupPolicy != nil {
		dst.Spec.Backup = src.Spec.BackupPolicy.Schedule
	}

	dst.Status.Phase = src.Status.Phase
	return nil
}

There are two important insights here. First, downconversion (Hub to old version) can involve unavoidable information loss. In the example above, the BackupPolicy.RetentionDays of v1beta1 disappears on the way down to v1alpha1. To prevent such loss, a common technique is a round-trip pattern that preserves the lost information in an annotation. Second, all conversions must be round-trip consistent. Going from A to B and back to A must yield the original, and controller-runtime provides fuzz test utilities to verify this.

Registering the Conversion Webhook

You declare the webhook conversion strategy in the CRD. Kubebuilder generates most of this for you.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: databases.example.com
spec:
  conversion:
    strategy: Webhook
    webhook:
      conversionReviewVersions:
        - v1
      clientConfig:
        service:
          namespace: db-operator-system
          name: db-operator-webhook
          path: /convert
          port: 443
        caBundle: <injected by cert-manager>

In the manager code you set up the conversion webhook for each type.

func (r *Database) SetupWebhookWithManager(mgr ctrl.Manager) error {
	return ctrl.NewWebhookManagedBy(mgr).
		For(r).
		Complete()
}

The caBundle is usually injected by cert-manager's CA Injector. If the webhook goes down, conversion fails and every request that depends on it (even an old-version GET) is blocked, so webhook availability is tied directly to control-plane availability.

Backward Compatibility and Deprecation

Safe Changes and Dangerous Changes

Schema changes range from harmless to destructive.

ChangeSafe?Note
Add optional fieldSafeNo effect on existing objects
Add defaultMostly safeTake care not to change existing object meaning
Add enum valueSafeOld clients simply do not know it
Remove fieldDangerousNew version only, preserve via conversion
Make field requiredDangerousMay invalidate existing objects
Change typeVery dangerousNew version plus conversion required
Remove enum valueDangerousObjects using that value break

The golden rule is this. Never make a destructive change within the same version. If you need to remove a field or change a type, always create a new API version and bridge it with a conversion webhook.

Deprecation Signals

When retiring an old version, you must warn users ahead of time. You can mark a CRD version as deprecated.

versions:
  - name: v1alpha1
    served: true
    storage: false
    deprecated: true
    deprecationWarning: "example.com/v1alpha1 Database is deprecated. Use v1beta1."

This emits a warning when users work with that version via kubectl, giving them time to migrate.

additionalPrinterColumns and Subresources

Printer Columns

You can define the columns shown when you run kubectl get databases. Expose the information operators care about most.

// +kubebuilder:printcolumn:name="Engine",type=string,JSONPath=`.spec.engine`
// +kubebuilder:printcolumn:name="Replicas",type=integer,JSONPath=`.spec.replicas`
// +kubebuilder:printcolumn:name="Phase",type=string,JSONPath=`.status.phase`
// +kubebuilder:printcolumn:name="Age",type=date,JSONPath=`.metadata.creationTimestamp`
type Database struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`
	Spec   DatabaseSpec   `json:"spec,omitempty"`
	Status DatabaseStatus `json:"status,omitempty"`
}

Status Subresource

The +kubebuilder:subresource:status marker turns status into a separate subresource. This is not just a convenience; it carries important semantics.

  • Users modify only spec, and the controller modifies only status. The two never conflict.
  • Status updates do not bump metadata.generation, so the controller can distinguish a spec change (generation increment) from its own status writes.
// +kubebuilder:subresource:status
type Database struct { /* ... */ }

type DatabaseStatus struct {
	// Phase is a human-readable summary status.
	Phase string `json:"phase,omitempty"`

	// Conditions is the standard list of status conditions.
	// +optional
	// +patchMergeKey=type
	// +patchStrategy=merge
	Conditions []metav1.Condition `json:"conditions,omitempty"`

	// ObservedGeneration is the last spec generation processed.
	ObservedGeneration int64 `json:"observedGeneration,omitempty"`
}

Scale Subresource

If your resource has replicas, you can expose a scale subresource so that kubectl scale and the HPA work.

// +kubebuilder:subresource:scale:specpath=.spec.replicas,statuspath=.status.replicas,selectorpath=.status.labelSelector
type Database struct { /* ... */ }

With this, the standard kubectl scale database/mydb --replicas=3 works directly, and a HorizontalPodAutoscaler can target this CRD.

Performance Considerations for Large CRDs

When a CRD scales to thousands or tens of thousands of objects, design decisions affect performance directly.

  • Keep objects small. The etcd object size limit is 1.5MB, but in practice you should be far below that. Do not inline large data (e.g., entire config files) in the CR; reference a ConfigMap or external store instead.
  • Do not accumulate logs or events in status. An ever-growing array in status means the whole object is rewritten on every update. Use fixed-size structures like Conditions.
  • Avoid unnecessary watches and indexes. The controller caches every object in memory (informer), so many large objects grow memory linearly. Narrow the watch scope with label/field selectors.
  • Be conscious of conversion webhook cost. During storage migration or bulk list, every object passes through the webhook. Keep conversion functions lightweight and free of external calls.
  • Keep printer column computation simple. Many high-priority columns or complex JSONPath expressions slow down list response serialization.

API Design Best Practices

It Must Be Declarative

The spec should describe "what you want (desired state)," not "what to do (a command)." For example, an imperative field like spec.restart: true is an antipattern. Once executed it becomes meaningless, and there is no way to track who turned it off and when. Have users declare intent instead.

Bad:   spec.restartNow: true        (command, one-shot)
Good:  spec.version: "16.2"         (intent, reconciled by controller)
Good:  spec.paused: true            (state, persistent meaning)

Keep the API Surface Small

Resist the temptation to expose every option from the start. Any field you add you must support forever. Ask whether the field is truly needed and whether a sensible default could replace it. A small API is easier to learn, easier to evolve, and harder to break.

Clear Separation of spec and status

spec is the user's input and status is the controller's output. Never mix the two. The moment a controller writes a value into spec, the user's GitOps repository and the cluster's actual state begin to drift.

Standardize Conditions

Express state using the metav1.Condition standard. With type, status, reason, message, and lastTransitionTime, this structure is a common language that kubectl, dashboards, and other controllers all understand.

Migration

Introducing a new version into an already-running CRD and moving storage requires a careful sequence. Here is a practical procedure using the storage version migrator.

# 1. Check the currently stored versions.
kubectl get crd databases.example.com -o jsonpath='{.status.storedVersions}'

# 2. Deploy the new version with served=true (storage is still the old version).
kubectl apply -f crd-with-new-version.yaml

# 3. Switch storage to the new version.
kubectl patch crd databases.example.com --type=merge \
  -p '{"spec":{"versions":[{"name":"v1beta1","storage":true}]}}'

# 4. Re-store all objects in the new storage version (trigger with a no-op patch).
kubectl get databases.example.com --all-namespaces -o name | \
  xargs -I{} kubectl patch {} --type=merge -p '{}'

# 5. Verify the old version is gone from status.storedVersions.
kubectl get crd databases.example.com -o jsonpath='{.status.storedVersions}'

# 6. Set the old version to served=false, then remove its schema.

In production, rather than running Step 4 with a direct patch, it is safer to use the Kubernetes storage-version-migrator controller or an equivalent tool. Patching tens of thousands of objects at once overloads the API server.

A Catalog of Pitfalls

Finally, here are the pitfalls encountered repeatedly in the field.

  • Forgetting storage migration. If you only change the storage version without re-storing existing objects, the moment you delete the old schema those objects become unreadable. Do not delete an old version schema until status.storedVersions is a single value.
  • Breaking lossless conversion. Deploying conversion without round-trip tests silently drops data on downconversion. Put controller-runtime's fuzz-based round-trip tests into CI.
  • Mutating the same version destructively. Changing a field type in v1beta1 and redeploying means existing objects fail to decode or lose data to pruning. Destructive changes always go into a new version.
  • Leaving the webhook as a single point of failure. If the conversion webhook dies, even an old-version GET fails. Run two or more replicas and configure a PodDisruptionBudget and proper readiness probe.
  • Imperative field pitfall. One-shot commands like spec.action or spec.restartNow clash with the reconcile model. Express everything as desired state.
  • Mixing spec into status pitfall. If the controller modifies spec, you get GitOps drift. The controller writes only status.
  • Giant object pitfall. Inlining large data in the CR hurts both informer memory and etcd. Pull it out into a reference.
  • Defaulting semantics change pitfall. Changing the default of an already-deployed field changes the meaning of existing objects that did not specify the value. Remember that a default change is effectively a behavior change.

Closing

CRD design looks at first like an appendix to the controller, but over time it becomes the hardest part of the system to change. The good starting points are clear: put validation inside the schema with structural schema and CEL, plan version evolution from the start, keep conversion simple with Hub-and-Spoke, and protect every conversion with round-trip tests.

Above all, treat your CRD as an API contract, not as code. A field exposed to users once is hard to take back. Start small, design declaratively, and pave a path for evolution in advance — then your operator can grow alongside its users without breaking, even years later.

References

Discuss on Twitter • View on GitHub