claude-botandClaude Opus 5.5
a17c4e8161
A CNPG cluster restored from barman archived back into the location it
recovered from. plugin-barman-cloud's restore job checks the archive target
(plugins[] ObjectStore + serverName) and stops with "Expected empty archive"
when it holds WAL, which after a disaster it always does. It has no
exception for target == recovery source (internal/cnpgi/restore/restore.go,
v0.15). Restore rehearsal run 28 hit it on wp-pg; the other six never got
that far.
The check stays. Instead, archive and source become two ObjectStores whose
paths are chosen per incarnation:
- kubernetes/cnpg-object-stores/: all 14 ObjectStores (<x>-store archives,
<x>-store-source is read by bootstrap.recovery), in a Kustomization of their
own with postBuild.substituteFrom: cnpg-paths. Nothing else lives there:
substitution blanks every ${...}, which is why it is not on the app
Kustomizations (home-assistant's scripts, the Grafana dashboards). The
ConfigMap is not optional; without it the paths would render empty.
woodpecker's objectstore.yaml moves there as wp-pg.yaml.
- bootstrap.sh writes cnpg-paths in the flux phase, before Flux starts, once
per incarnation (an existing one is kept):
CNPG_<X>_SOURCE the prefix among <x>, <x>-g<14 digits> and the store's
cnpg.renner.uno/source-prefixes whose <x>/base/ holds
the newest completed base backup (backup.info
status=DONE; barman uploads it as STARTED when a
backup begins, barman src/barman/cloud.py
coordinate_backup)
CNPG_<X>_ARCHIVE <x>-g<UTC timestamp>, new and so empty
Every ScheduledBackup has immediate: true, so a new incarnation has a base
backup in its own prefix minutes after it starts, and a later recovery
picks it. S3 access is curl --aws-sigv4 with the restore key (already in
bootstrap.sh's secret list), so the CI image needs no new tool; the
OpenBao restore reads the same S3 endpoint and region.
- The six clusters that recovered from their archive store now name
<x>-store-source in externalClusters. serverName stays the cluster name
everywhere. The app Kustomizations depend on cnpg-object-stores, which
depends on infrastructure-cnpg-barman.
Verified against the bucket, read-only, 2026-10-16: the restore key lists
renner-uno-k8s-cnpg; the listing parser on real responses, top level and an
encoded nested prefix; for wp-pg it picks wp-pg-woodpecker (newest
20261016T024000) over wp-pg (20260818T023502); leftover test serverNames
hold no base backups and match no candidate pattern.
Production rollout, one time:
1. Create flux-system/cnpg-paths with the current prefixes as both source and
archive (ak-pg, ha-pg, im-pg, mx-pg, pprl-pg, tr-pg; wp-pg-woodpecker), so
the running clusters keep archiving where they do.
2. The app Kustomizations that give up their ObjectStores (authentik,
home-assistant, immich, matrix, paperless, trmnl, woodpecker) must run
with prune: false (the stack's first PR) when this merges;
cnpg-object-stores adopts the objects. All 14 also carry prune: disabled.
3. Whether changing externalClusters on a live Cluster restarts its pods is
not verified; check on one cluster first.
Older generations are not cleaned up: retention applies only to the current
archive prefix.
RECOVERY.md describes the new layout: where the current prefixes are
(cnpg-paths), the per-incarnation backup paths for the precondition check,
archiving to the new prefix after promotion, and adding a CNPG cluster to a
running cluster (its keys go into the existing cnpg-paths by hand). The
manual restore drill reads <x>-store, the current archive. The app
kustomization comments name cnpg-object-stores instead of
infrastructure-cnpg-barman. write_cnpg_paths checks for yq first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
88 lines
3.7 KiB
YAML
88 lines
3.7 KiB
YAML
apiVersion: postgresql.cnpg.io/v1
|
|
kind: Cluster
|
|
metadata:
|
|
name: tr-pg
|
|
namespace: trmnl
|
|
spec:
|
|
instances: 2
|
|
smartShutdownTimeout: 10
|
|
stopDelay: 60
|
|
# PG 18, NOT the repo-standard 16. Terminus ships a committed
|
|
# config/db/structure.sql containing `SET transaction_timeout = 0;` — a GUC
|
|
# that does not exist before PostgreSQL 17, so schema load against a 16
|
|
# server dies with `unrecognized configuration parameter`. Upstream's
|
|
# compose.yml develops against 18.4, so 18 is the tested target; 17 is the
|
|
# bare minimum. Bare major tag matches the other clusters (minor security
|
|
# bumps flow automatically, major upgrades stay an explicit decision).
|
|
imageName: ghcr.io/cloudnative-pg/postgresql:18
|
|
enableSuperuserAccess: true
|
|
storage:
|
|
storageClass: ceph-block
|
|
size: 5Gi
|
|
plugins:
|
|
- name: barman-cloud.cloudnative-pg.io
|
|
isWALArchiver: true
|
|
parameters:
|
|
barmanObjectName: tr-pg-store
|
|
# Honoured only when PGDATA is absent: a cold rebuild restores from the
|
|
# barman backup. No initdb fallback; an empty object store hangs bootstrap.
|
|
bootstrap:
|
|
recovery:
|
|
source: tr-pg-store
|
|
database: terminus
|
|
owner: terminus
|
|
externalClusters:
|
|
- name: tr-pg-store
|
|
plugin:
|
|
name: barman-cloud.cloudnative-pg.io
|
|
parameters:
|
|
barmanObjectName: tr-pg-store-source
|
|
serverName: tr-pg
|
|
postgresql:
|
|
parameters:
|
|
shared_buffers: "128MB"
|
|
max_connections: "50"
|
|
# zstd, not "on": the boolean spelling resolves to pglz (2016-era, weaker
|
|
# and slower than the lz4/zstd this build also accepts). Compresses the
|
|
# full-page images that dominate WAL volume after each checkpoint.
|
|
wal_compression: "zstd"
|
|
# A WAL segment is a fixed 16 MiB file, and archive_timeout forces a switch
|
|
# on a timer -- the segment is then written and archived at its full size
|
|
# no matter how little real WAL it holds. CNPG defaults to 5min, i.e. up
|
|
# to 288 segments/day = 4.6 GB/day that is largely padding.
|
|
#
|
|
# Verified on gf-pg first (the staged rollout): 211 -> 41 MB/hour, an 81%
|
|
# cut, with archived segments falling from ~12/hour to ~1.5/hour in step.
|
|
# Two independent signals moving together is what confirms the volume was
|
|
# padding rather than data.
|
|
#
|
|
# The cost is RPO: up to 30min of WAL is unrecoverable if this cluster and
|
|
# its volumes are lost outright.
|
|
archive_timeout: "30min"
|
|
# Write-reduction tuning. These databases are largely idle, and the
|
|
# Postgres defaults favour throughput over storage that wears out.
|
|
#
|
|
# checkpoint_timeout is the lever. The 5min default pushes every dirty
|
|
# page to disk 288 times a day, so a small database whose same few pages
|
|
# are re-dirtied constantly (sessions, tokens, job state) gets rewritten
|
|
# over and over. Measured on ak-pg: 25.8M checkpoint buffers flushed
|
|
# (~206 GB) against a 1046 MB database -- the whole thing roughly twice a
|
|
# day. At 15min those pages coalesce and are written a third as often.
|
|
#
|
|
# max_wal_size must rise with it, or checkpoints trigger on size instead
|
|
# and the timeout change does nothing. It also bounds crash recovery to
|
|
# at most 2GB of WAL replay.
|
|
#
|
|
# NOT changed: synchronous_commit stays at the default `on`. It is off for
|
|
# ha-pg because a state recorder is telemetry; these hold credentials,
|
|
# documents and job state, where losing the last few hundred ms of commits
|
|
# on an unclean shutdown is not an acceptable trade.
|
|
checkpoint_timeout: "15min"
|
|
max_wal_size: "2GB"
|
|
resources:
|
|
requests:
|
|
cpu: 100m
|
|
memory: 256Mi
|
|
limits:
|
|
memory: 512Mi
|