claude-botandClaude Opus 5.5
a17c4e8161
A CNPG cluster restored from barman archived back into the location it
recovered from. plugin-barman-cloud's restore job checks the archive target
(plugins[] ObjectStore + serverName) and stops with "Expected empty archive"
when it holds WAL, which after a disaster it always does. It has no
exception for target == recovery source (internal/cnpgi/restore/restore.go,
v0.15). Restore rehearsal run 28 hit it on wp-pg; the other six never got
that far.
The check stays. Instead, archive and source become two ObjectStores whose
paths are chosen per incarnation:
- kubernetes/cnpg-object-stores/: all 14 ObjectStores (<x>-store archives,
<x>-store-source is read by bootstrap.recovery), in a Kustomization of their
own with postBuild.substituteFrom: cnpg-paths. Nothing else lives there:
substitution blanks every ${...}, which is why it is not on the app
Kustomizations (home-assistant's scripts, the Grafana dashboards). The
ConfigMap is not optional; without it the paths would render empty.
woodpecker's objectstore.yaml moves there as wp-pg.yaml.
- bootstrap.sh writes cnpg-paths in the flux phase, before Flux starts, once
per incarnation (an existing one is kept):
CNPG_<X>_SOURCE the prefix among <x>, <x>-g<14 digits> and the store's
cnpg.renner.uno/source-prefixes whose <x>/base/ holds
the newest completed base backup (backup.info
status=DONE; barman uploads it as STARTED when a
backup begins, barman src/barman/cloud.py
coordinate_backup)
CNPG_<X>_ARCHIVE <x>-g<UTC timestamp>, new and so empty
Every ScheduledBackup has immediate: true, so a new incarnation has a base
backup in its own prefix minutes after it starts, and a later recovery
picks it. S3 access is curl --aws-sigv4 with the restore key (already in
bootstrap.sh's secret list), so the CI image needs no new tool; the
OpenBao restore reads the same S3 endpoint and region.
- The six clusters that recovered from their archive store now name
<x>-store-source in externalClusters. serverName stays the cluster name
everywhere. The app Kustomizations depend on cnpg-object-stores, which
depends on infrastructure-cnpg-barman.
Verified against the bucket, read-only, 2026-10-16: the restore key lists
renner-uno-k8s-cnpg; the listing parser on real responses, top level and an
encoded nested prefix; for wp-pg it picks wp-pg-woodpecker (newest
20261016T024000) over wp-pg (20260818T023502); leftover test serverNames
hold no base backups and match no candidate pattern.
Production rollout, one time:
1. Create flux-system/cnpg-paths with the current prefixes as both source and
archive (ak-pg, ha-pg, im-pg, mx-pg, pprl-pg, tr-pg; wp-pg-woodpecker), so
the running clusters keep archiving where they do.
2. The app Kustomizations that give up their ObjectStores (authentik,
home-assistant, immich, matrix, paperless, trmnl, woodpecker) must run
with prune: false (the stack's first PR) when this merges;
cnpg-object-stores adopts the objects. All 14 also carry prune: disabled.
3. Whether changing externalClusters on a live Cluster restarts its pods is
not verified; check on one cluster first.
Older generations are not cleaned up: retention applies only to the current
archive prefix.
RECOVERY.md describes the new layout: where the current prefixes are
(cnpg-paths), the per-incarnation backup paths for the precondition check,
archiving to the new prefix after promotion, and adding a CNPG cluster to a
running cluster (its keys go into the existing cnpg-paths by hand). The
manual restore drill reads <x>-store, the current archive. The app
kustomization comments name cnpg-object-stores instead of
infrastructure-cnpg-barman. write_cnpg_paths checks for yq first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
14 lines
386 B
YAML
14 lines
386 B
YAML
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
resources:
|
|
- externalsecret.yaml
|
|
- synapse.yaml
|
|
- mas.yaml
|
|
- element.yaml
|
|
- ingress.yaml
|
|
# mx-pg CNPG Postgres + offsite Barman backup. The media store and signing
|
|
# key are on synapse-data, backed up by Velero (matrix-daily).
|
|
- cluster.yaml
|
|
- scheduled-backup.yaml
|
|
- hetzner-creds-externalsecret.yaml
|