Split View: CloudNativePG 딥다이브 — PostgreSQL 을 쿠버네티스 안에서 누가 돌보는가

CloudNativePG deep dive — who looks after PostgreSQL inside Kubernetes

PostgreSQL does not look after itself

PostgreSQL is an excellent database, but putting it in a container does not make it create its own replicas, promote a standby when the primary dies, take a backup every day, or rewind to three o'clock yesterday afternoon. Until now a person did that.

CloudNativePG (CNPG) is the operator that turns that person's job into something a Kubernetes controller reconciles continuously. This post explains what CNPG does and how, using the values of a cluster that is actually running in production.

operator   cloudnative-pg 1.30.0  +  plugin-barman-cloud v0.14.0
cluster    labhub-db-prod — PostgreSQL 18.4, 2 instances, 5Gi volume (NFS)
state      primary = labhub-db-prod-1, standby = labhub-db-prod-2, timeline 3

Two places where the design is unusual

It does not use a StatefulSet

Common sense says stateful things in Kubernetes run as StatefulSets. CNPG does not. The operator creates and deletes every pod itself.

The reason is that database instances are not interchangeable. A StatefulSet assumes "from ordinal 0 upward, all from the same template". A database needs decisions like "recreate only instance 2 on a fresh volume so it re-replicates from scratch" or "touch the primary last". To make those decisions you must be able to handle pods individually.

PID 1 in the pod is not postgres

Process 1 inside a CNPG pod is the instance manager. It starts postgres as a child, answers the readiness and liveness probes, receives promotion commands, ships WAL to the archive, and reloads configuration when it changes.

Two things follow. The database keeps running if the operator dies (the manager lives inside the pod). And "is this instance really alive" is judged by Kubernetes from PostgreSQL's actual response, not from a TCP port.

Role by role

A cluster is one CR

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata: { name: labhub-db-prod }
spec:
  instances: 2
  imageName: ghcr.io/cloudnative-pg/postgresql:18.4-system-trixie
  storage: { size: 5Gi, storageClass: nfs-synology }
  primaryUpdateStrategy: unsupervised
  primaryUpdateMethod: restart
  postgresql:
    parameters: { wal_level: logical, archive_timeout: 5min }
  replicationSlots: { highAvailability: { enabled: true } }
  plugins:
    - name: barman-cloud.cloudnative-pg.io
      isWALArchiver: true

This one object produces everything below — pods, services, secrets, certificates, replication, backups. There is also only one way to change anything: edit this CR, and the operator computes the difference and moves only as much as needed.

Replication and roles

One instance is the primary; the rest are standbys following it through streaming replication. This is what the primary reports right now:

application_name | state     | sync_state
labhub-db-prod-2 | streaming | async

async means the primary does not wait for the standby's acknowledgement before confirming a commit. It is fast, but any transaction that has not yet reached the standby at the moment the primary dies is lost. This cluster's lag is effectively zero — because the load is small today, not because anything guarantees it.

Replication slot HA is enabled, so if a standby drops off for a while the primary does not delete the WAL that standby has not yet received. Without it, a few minutes of network trouble would mean rebuilding the standby from scratch.

Three services

Applications do not connect to pod names. They connect to one of three services the operator creates.

On failover the operator only changes the endpoints of -rw. The application sees a dropped connection and reconnects. The fact that no primary address is written in the app's configuration is where the freedom to move and replace the database begins.

Failover, switchover, and the timeline

When the primary dies the operator picks the most advanced standby and promotes it. PostgreSQL then increments the timeline — a mark that history has forked. This cluster is at timelineID: 3; the two times its primary has changed so far are recorded in that number.

A planned change is a switchover. kubectl cnpg promote labhub-db-prod labhub-db-prod-2 names the standby; the operator drains the old primary, brings up the new one, and turns the old one into a standby. Only the roles change, with no data loss.

Rolling updates

When the image or a restart-requiring setting changes, the operator replaces the standbys first and touches the primary last. Two values decide how that last step goes.

• primaryUpdateStrategy: unsupervised — proceed without human approval (supervised stops in front of the primary and waits).
• primaryUpdateMethod: restart — restart the primary in place (switchover promotes a standby first to shorten the primary's downtime).

A minor upgrade (18.4 → 18.5) is just a change of imageName, run through this procedure.

Configuration lives in the CR; reload vs restart is the operator's call

Values in postgresql.parameters are written to postgresql.conf by the operator, which knows whether each one takes effect on reload or needs a restart and handles it accordingly. This cluster touches two:

• wal_level: logical — so logical replication and change-data capture are possible.
• archive_timeout: 5min — the WAL archiving interval from the next section.

Backups — two kinds must be combined before you can restore

This is the most misunderstood part. "We took a backup" is only half a sentence.

WAL archiving. PostgreSQL writes every change to the WAL (write-ahead log) first. Whenever a WAL segment fills up or archive_timeout (5 minutes) elapses, CNPG ships that segment to object storage. So changes from the last five minutes may not be in the archive yet — that is this cluster's RPO.

Base backup. A copy of the entire data directory. A ScheduledBackup takes one every day at 03:30 UTC; the last three days are all completed.

A restore combines the two. Unpack the base backup, then replay the WAL after it up to the moment you want. That is how "three o'clock yesterday afternoon" becomes reachable (PITR), and the earliest reachable moment is recorded in the status as firstRecoverabilityPoint.

This cluster does its backups not through the built-in barmanObjectStore but through a CNPG-I plugin (plugin-barman-cloud). That is why spec.backup is empty and plugins[].isWALArchiver: true stands in its place. Moving backup logic out of the operator core is the direction recent CNPG releases have taken.

Other features

• Declarative roles and databases — managed.roles and the Database CR manage users and databases as YAML.
• Certificates — the operator issues and renews server and client TLS certificates.
• Pooler — one Pooler CR puts PgBouncer in front. For when connections grow into the hundreds.
• Metrics — with monitoring.enablePodMonitor: true Prometheus scrapes them.
• Replica clusters — a standby in another cluster (another datacenter) for DR.
• Volume snapshot backups, hibernation (keep the volumes, remove the pods), online storage resize.

What the real cluster shows, and the traps

From here on this is not documentation but experience.

A status field is frozen. Backups run daily, yet status.lastSuccessfulBackup stopped at August 23. After the move to the plugin, the Cluster's status fields and the cnpg_collector_* metrics stop updating. Alert on those and you get false alarms. Judge by the Backup objects' phase and the barman_cloud_* metrics instead.

The first backup may not be restorable. A backup taken right after archiving was switched on showed completed, yet restoring it failed because the WAL position it started at (beginWal) was not in the archive. Rotate WAL once with pg_switch_wal(), take the backup again, and actually restore it. A backup is confirmed only by "we brought it back", never by "it was taken".

Async replication with two instances. As shown above, async can lose the last few transactions on failover. Synchronous replication fixes that, but with only two instances it creates the opposite risk — writes stop the moment the standby dies. Grow to three first, then switch it on.

A database on NFS. The volume is NFS (5Gi). Data is 7.8MB today so capacity is not the issue, but WAL fsync on NFS is slower than on local disk and the CNPG documentation recommends local storage. It is the first thing to move when load grows.

Metrics are off. enablePodMonitor is disabled, so Prometheus has no database metrics. Turn it on to see connections, replication lag and WAL archive failures.

In one line

CNPG's role is to make a controller — not a person — always able to answer "who is the primary right now, what address is in front of it, and can we go back to yesterday." This cluster has all three. What remains is regular restore drills, three instances with synchronous replication, and metrics switched on.