etcd and Kubernetes: API Server Integration Analysis

The Kubernetes API Server uses etcd as its sole data store. This post analyzes how the API Server utilizes etcd, including key structure, Watch Cache, sizing, and encryption.

1. kube-apiserver etcd Usage

1.1 Storage Backend

kube-apiserver stores all Kubernetes resources using the etcd v3 API through a storage interface supporting Create, Delete, Get, List, Watch, and GuaranteedUpdate operations.

1.2 Codec (Serialization/Deserialization)

Kubernetes resources are serialized before storage in etcd:

• protobuf: Default format (efficient binary)
• JSON: Alternative format (easier debugging)
• CBOR: Experimental support

1.3 Transformer (Encryption)

Transformers convert data before/after etcd storage: Identity (none), AES-CBC, AES-GCM, KMS, or Secretbox encryption.

2. etcd Key Hierarchy

2.1 /registry/ Prefix

All Kubernetes resources are stored under the /registry/ prefix:

/registry/pods/default/my-pod
/registry/services/specs/default/kubernetes
/registry/deployments/default/my-deployment
/registry/secrets/default/my-secret
/registry/namespaces/default
/registry/nodes/node-1

2.2 Key Structure Patterns

Namespaced resources: /registry/RESOURCE_TYPE/NAMESPACE/NAME Cluster resources: /registry/RESOURCE_TYPE/NAME

2.3 Resource Encoding

Data is stored as protobuf-encoded binary. Decoding reveals PodSpec, PodStatus, and other structures.

3. Watch Cache

3.1 Why Watch Cache is Needed

Without Watch Cache, all Watch requests would go directly to etcd causing excessive load, wasted bandwidth, and limited scalability.

3.2 Cacher Structure

type Cacher struct {
    storage     storage.Interface    // actual etcd backend
    watchCache  *watchCache          // cached event history
    reflector   *cache.Reflector     // etcd watch reflector
    watchers    indexedWatchers       // registered watcher list
}

3.3 How It Works

1. Reflector: Sets up etcd Watch for each resource type at API Server startup
2. Watch Cache: Caches events from etcd in a ring buffer
3. Client Watch: Client Watch requests are served from cache

etcd --[watch]--> Reflector --[events]--> Watch Cache --[events]--> Client Watchers

3.4 List Request Processing

List requests are also served from Watch Cache when the cache is current, avoiding etcd access.

4. etcd Sizing

4.1 Object Count Based Sizing

4.2 Request Rate Considerations

Factors affecting etcd load: Pod creation/deletion frequency, ConfigMap/Secret updates, Watch count, Custom Resource count and size.

5. Encryption at Rest

5.1 EncryptionConfiguration

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: BASE64_ENCODED_KEY
      - identity: {}

5.2 Provider Priority

The first provider in the array is used for writes; all providers are used for reads. This enables key rotation:

1. Add new key at first position
2. Restart kube-apiserver
3. Rewrite all Secrets (encrypted with new key)
4. Remove old key

5.3 KMS Provider (v2)

KMS v2 provides more efficient key management: local DEK generation/caching, remote KEK management, no API server restart for rotation, minimal overhead.

5.4 Verifying Encryption Status

etcdctl get /registry/secrets/default/my-secret | hexdump -C | head
# Encrypted: 'k8s:enc:aescbc:v1:key1' prefix visible
# Unencrypted: 'k8s' protobuf prefix visible

6. API Server etcd Connection Configuration

6.1 Key Flags

kube-apiserver \
  --etcd-servers=https://etcd1:2379,https://etcd2:2379,https://etcd3:2379 \
  --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt \
  --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt \
  --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key \
  --etcd-compaction-interval=5m \
  --encryption-provider-config=/etc/kubernetes/encryption-config.yaml

7. Summary

The integration between Kubernetes API Server and etcd is central to cluster operations. Efficient caching through Watch Cache, understanding key structure, configuring Encryption at Rest, and proper etcd sizing are essential for stable cluster operations. Continuous monitoring of etcd health and regular backup/maintenance are critical.