Chaos and Order

💡 왼쪽 원문을 읽으면서 오른쪽에 따라 써보세요. Tab 키로 힌트를 받을 수 있습니다.

Introduction — What It Means to Hand Postgres Over to an Operator
Part 1 — Installation: A Single Manifest
Part 2 — Building a 3-Instance Cluster
Part 3 — Verifying Replication
Part 4 — Killing the Primary: 23.1-Second Failover
Part 5 — Honest Numbers and Pitfalls
Closing
References

Introduction — What It Means to Hand Postgres Over to an Operator

Running a stateful database on Kubernetes was long treated almost as a taboo. CloudNativePG (CNPG) is an operator that breaks that taboo head-on — declare Postgres's high availability, failover, backups, and rolling upgrades with a single CRD, and the operator handles primary election and replica management for you. This article is not an introduction but a measurement: I installed CNPG on a real 8-node cluster, brought up a 3-instance Postgres, and actually killed the primary to measure how many seconds failover takes. It is the counterpart to the Rust GPU operator piece where I wrote an operator myself — this is the story of the "using a well-built operator" side.

Part 1 — Installation: A Single Manifest

Installing CNPG is minimal. Apply a single release manifest and you're done.

kubectl apply --server-side -f \
  https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.30/releases/cnpg-1.30.0.yaml

This one line installs everything — several CRDs (clusters, poolers, scheduledbackups, publications, subscriptions, and more), the controller Deployment in the cnpg-system namespace, RBAC, and webhooks. The controller took about 20 seconds to come up:

$ kubectl -n cnpg-system rollout status deploy/cnpg-controller-manager
deployment "cnpg-controller-manager" successfully rolled out

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
cnpg-controller-manager   1/1     1            1           16s

Part 2 — Building a 3-Instance Cluster

A Postgres cluster is declared with a Cluster CR. I set it to 1 primary + 2 replicas, i.e., 3 instances.

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: pg-test
  namespace: cnpg-test
spec:
  instances: 3
  storage:
    size: 1Gi
    storageClass: nfs-client     # NFS in the homelab — the pitfall is in Part 5
  bootstrap:
    initdb:
      database: appdb
      owner: appuser

Once applied, the operator begins bootstrapping. I watched the state transitions in real time:

Setting up primary
→ Waiting for the instances to become active
→ ready=1  Creating a new replica       ← replica cloning starts once the primary is up
→ ready=2  Creating a new replica
→ ready=3  Cluster in healthy state      ← 3/3 healthy in about 2 minutes

The three instances were automatically spread across different nodes (cubi02, cubi03, cubi04) — the operator uses anti-affinity to keep them from piling onto a single node. CNPG also creates three connection services for you:

Service        Role
────────────  ─────────────────────────────
pg-test-rw     read/write → always routed to the current primary
pg-test-ro     read-only → load-balanced across replicas
pg-test-r      any instance (reads)

The -rw service is the key — if the application looks at just this one name, then even when the primary changes due to failover, it automatically connects to the new primary.

Part 3 — Verifying Replication

I inserted 1000 rows into the primary and checked whether all three instances matched.

$ INSERT 1000 rows into the primary (pg-test-1)
INSERT 0 1000

$ Row count per instance
  pg-test-1 (primary): 1000 rows
  pg-test-2 (replica): 1000 rows      ← replicated
  pg-test-3 (replica): 1000 rows      ← replicated

Streaming replication synchronized all three nodes instantly. Now for the real experiment.

Part 4 — Killing the Primary: 23.1-Second Failover

The thing I was most curious about — if the primary suddenly disappears, how many seconds until recovery? I instantly killed the primary pod with --grace-period=0 --force, then measured the time by polling at 0.5-second intervals until a new primary appeared.

=== FAILOVER TEST: killing primary pg-test-1 ===
deleted at t0; polling for new primary...
=== NEW PRIMARY: pg-test-2 (was pg-test-1) ===
failover time: 23.1 s

  rows after failover: 1000        ← zero data loss

In 23.1 seconds, pg-test-2 was promoted to the new primary, and all 1000 rows were still there. And the pg-test-1 that had died is not thrown away — the operator automatically brings it back and re-enrolls it as a replica:

=== Final roles after self-healing ===
  pg-test-1: replica     ← died and came back, demoted to replica
  pg-test-2: primary     ← the newly promoted primary
  pg-test-3: replica

$ Additional writes to the new primary → OK, 1500 rows total

The new primary immediately accepted writes (1000→1500 rows), and the cluster returned to 3/3 healthy. Zero human intervention. This is what an operator is worth.

Part 5 — Honest Numbers and Pitfalls

In keeping with the principle that this blog writes only what has been verified, I leave the limitations of this experiment intact, too.

Is 23 seconds fast? It depends on the situation. CNPG's failover time is the sum of detecting the primary's death (the health-check interval), promoting a replica, and updating the -rw service endpoint. In production, node failures are more common than pod deletions, and in that case the node-detection time (node-monitor-grace-period, etc.) is added on, which can make it longer. Conversely, with tuning it gets shorter. The "23 seconds" is a measured value for this homelab and this configuration, not a universal constant.
The NFS storage pitfall. I used nfs-client (NFS provisioner) storage, but putting Postgres on NFS is not recommended in production — because of fsync guarantees and file-locking issues. It ran fine as a homelab test, but for a real service you should use local SSD or block storage (Ceph RBD, etc.).
Synchronous vs. asynchronous replication. This test used the default (asynchronous) replication. With asynchronous replication, in theory a tiny number of unreplicated transactions right before the primary's death can be lost. If you need zero loss, you have to turn on CNPG's synchronous replication (minSyncReplicas), and in exchange write latency increases.

Closing

CNPG refutes, with real measurements, the conventional wisdom that "databases on Kubernetes are dangerous" — a single manifest brings up a 3-node HA Postgres, it recovers on its own within 23 seconds even when you kill the primary outright, and the dead node comes back as a replica. Of course the real homework of storage, replication mode, and failover tuning remains, but that is not a question of "can you hand a DB to an operator" — it is a question of "how do you hand it over well." Next, I plan to verify backups (ScheduledBackup) and point-in-time recovery (PITR) the same way, by killing things.