cnpg: enable preferred sync replication on 2-instance clusters #15

2026-06-13T10:10:35Z

k.renner commented

2026-06-13 10:10:35 +00:00

Summary

Enables one synchronous standby on ak-pg and pprl-pg (both 2-instance clusters) using CNPG's modern spec.postgresql.synchronous block with dataDurability: preferred. This eliminates the dominant cause of slow switchover: the new primary chasing missing WAL after the old one terminated mid-stream.

What slow today looks like

Today's kubelet-restart on k8s-2 (ak-pg) and k8s-3 (pprl-pg) each triggered a CNPG-managed switchover that took ~5 minutes from probe-failure to client reconnection. From the ak-pg-2 instance-manager logs:

07:57:00  pg_ctl promote -w called
07:59:01  postgres: "received promote request" + "redo done"   (+2m)
08:01:03  postgres: "selected new timeline ID: 4"              (+2m)
08:02:03  postgres: history file restored from archive
08:02:04  database system ready to accept connections

The two-minute gap from pg_ctl promote to "received promote request" is the standby cycling through wal_retrieve_retry_interval + barman-cloud-wal-restore probes against garage-synology S3 looking for WAL the old primary never archived before it died. Sync replication eliminates this entirely.

Change

spec:
  postgresql:
    parameters:
      shared_buffers: "256MB"
      max_connections: "400"
    synchronous:
      method: any
      number: 1
      dataDurability: preferred

File	Cluster
`kubernetes/seed/authentik/cluster.yaml`	ak-pg
`kubernetes/seed/paperless/cluster.yaml`	pprl-pg

Why `dataDurability: preferred`

Setting	Standby healthy	Standby down
`required`	Sync — instant switchover	Writes block until standby returns
`preferred`	Sync — instant switchover	Falls back to async — writes continue

preferred keeps fast switchover when the cluster is healthy without sacrificing write availability when a standby fails. The trade-off versus current async config is ~1–5 ms added commit latency on healthy LAN+Ceph — well under noise for authentik and paperless workloads.

What this is NOT

Not touched: im-pg (immich) and gf-pg (grafana). Both are 1-instance; sync replication requires ≥2.
Not changed: any other postgres parameters, resources, or behavior. Just the synchronous: block.

Rollout

On merge, CNPG applies the new config to the live Cluster CRs. Each cluster gets synchronous_standby_names = "ANY 1 (...)" set; postgres reloads config (no restart). Standby starts acking commits within seconds.

Test plan

Merge
kubectl -n authentik get cluster ak-pg -o yaml | yq .spec.postgresql.synchronous shows the new block
kubectl -n authentik exec ak-pg-2 -c postgres -- psql -t -c "SHOW synchronous_standby_names" returns ANY 1 ("ak-pg-1") (or similar)
Repeat for paperless
Validate by triggering a switchover (e.g. kubectl cnpg promote ak-pg ak-pg-1 after merge); time-to-ready should drop from ~5min to <30s

🤖 Generated with Claude Code

## Summary Enables one synchronous standby on `ak-pg` and `pprl-pg` (both 2-instance clusters) using CNPG's modern `spec.postgresql.synchronous` block with `dataDurability: preferred`. This eliminates the dominant cause of slow switchover: the new primary chasing missing WAL after the old one terminated mid-stream. ## What slow today looks like Today's kubelet-restart on k8s-2 (ak-pg) and k8s-3 (pprl-pg) each triggered a CNPG-managed switchover that took ~5 minutes from probe-failure to client reconnection. From the ak-pg-2 instance-manager logs: ``` 07:57:00 pg_ctl promote -w called 07:59:01 postgres: "received promote request" + "redo done" (+2m) 08:01:03 postgres: "selected new timeline ID: 4" (+2m) 08:02:03 postgres: history file restored from archive 08:02:04 database system ready to accept connections ``` The two-minute gap from `pg_ctl promote` to "received promote request" is the standby cycling through `wal_retrieve_retry_interval` + `barman-cloud-wal-restore` probes against garage-synology S3 looking for WAL the old primary never archived before it died. Sync replication eliminates this entirely. ## Change ```yaml spec: postgresql: parameters: shared_buffers: "256MB" max_connections: "400" synchronous: method: any number: 1 dataDurability: preferred ``` | File | Cluster | |---|---| | `kubernetes/seed/authentik/cluster.yaml` | ak-pg | | `kubernetes/seed/paperless/cluster.yaml` | pprl-pg | ## Why `dataDurability: preferred` | Setting | Standby healthy | Standby down | |---|---|---| | `required` | Sync — instant switchover | **Writes block** until standby returns | | `preferred` | Sync — instant switchover | Falls back to async — writes continue | `preferred` keeps fast switchover when the cluster is healthy without sacrificing write availability when a standby fails. The trade-off versus current async config is ~1–5 ms added commit latency on healthy LAN+Ceph — well under noise for authentik and paperless workloads. ## What this is NOT - **Not** touched: `im-pg` (immich) and `gf-pg` (grafana). Both are 1-instance; sync replication requires ≥2. - **Not** changed: any other postgres parameters, resources, or behavior. Just the `synchronous:` block. ## Rollout On merge, CNPG applies the new config to the live Cluster CRs. Each cluster gets `synchronous_standby_names = "ANY 1 (...)"` set; postgres reloads config (no restart). Standby starts acking commits within seconds. ## Test plan - [ ] Merge - [ ] `kubectl -n authentik get cluster ak-pg -o yaml | yq .spec.postgresql.synchronous` shows the new block - [ ] `kubectl -n authentik exec ak-pg-2 -c postgres -- psql -t -c "SHOW synchronous_standby_names"` returns `ANY 1 ("ak-pg-1")` (or similar) - [ ] Repeat for paperless - [ ] Validate by triggering a switchover (e.g. `kubectl cnpg promote ak-pg ak-pg-1` after merge); time-to-ready should drop from ~5min to <30s 🤖 Generated with [Claude Code](https://claude.com/claude-code)

k.renner added 1 commit 2026-06-13 10:10:35 +00:00

cnpg: enable preferred synchronous replication on 2-instance clusters 1017e23829

Today's kubelet-restart-induced switchovers on ak-pg and pprl-pg each
took ~5 minutes from probe-failure to authentik/paperless reconnecting,
dominated by ~4 minutes of postgres-internal promote work. Most of
that time is the new primary chasing missing WAL via barman-cloud (the
old primary terminated mid-stream, queued WAL had to be replayed from
the archive) and the standby waiting through wal_retrieve_retry_interval
cycles before noticing the promote trigger.

With one synchronous standby (`synchronous.number: 1`), the standby is
always at the same LSN as the primary at commit time, so on promote
there is no missing WAL to chase and the standby is immediately ready
to accept connections. CNPG switchovers in this mode complete in single
digits of seconds.

`dataDurability: preferred` (CNPG v1.20+ modern API, replaces the
legacy minSyncReplicas/maxSyncReplicas fields) makes sync best-effort:
the primary stops waiting for the standby if it becomes unavailable,
so a standby outage does not block writes. The trade-off versus async
is ~1-5 ms of added commit latency on healthy LAN+Ceph, which is well
under the noise floor for authentik and paperless workloads.

Single-instance clusters (im-pg, gf-pg) are not touched - sync
replication requires >= 2 instances.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

k.renner merged commit c047d6b870 into master

2026-06-13 10:12:32 +00:00

k.renner referenced this issue from a commit

2026-06-13 10:12:33 +00:00

Merge pull request 'cnpg: enable preferred sync replication on 2-instance clusters' (#15) from fix/cnpg-sync-replication into master

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#15