claude-botandClaude Opus 5.5
a17c4e8161
A CNPG cluster restored from barman archived back into the location it
recovered from. plugin-barman-cloud's restore job checks the archive target
(plugins[] ObjectStore + serverName) and stops with "Expected empty archive"
when it holds WAL, which after a disaster it always does. It has no
exception for target == recovery source (internal/cnpgi/restore/restore.go,
v0.15). Restore rehearsal run 28 hit it on wp-pg; the other six never got
that far.
The check stays. Instead, archive and source become two ObjectStores whose
paths are chosen per incarnation:
- kubernetes/cnpg-object-stores/: all 14 ObjectStores (<x>-store archives,
<x>-store-source is read by bootstrap.recovery), in a Kustomization of their
own with postBuild.substituteFrom: cnpg-paths. Nothing else lives there:
substitution blanks every ${...}, which is why it is not on the app
Kustomizations (home-assistant's scripts, the Grafana dashboards). The
ConfigMap is not optional; without it the paths would render empty.
woodpecker's objectstore.yaml moves there as wp-pg.yaml.
- bootstrap.sh writes cnpg-paths in the flux phase, before Flux starts, once
per incarnation (an existing one is kept):
CNPG_<X>_SOURCE the prefix among <x>, <x>-g<14 digits> and the store's
cnpg.renner.uno/source-prefixes whose <x>/base/ holds
the newest completed base backup (backup.info
status=DONE; barman uploads it as STARTED when a
backup begins, barman src/barman/cloud.py
coordinate_backup)
CNPG_<X>_ARCHIVE <x>-g<UTC timestamp>, new and so empty
Every ScheduledBackup has immediate: true, so a new incarnation has a base
backup in its own prefix minutes after it starts, and a later recovery
picks it. S3 access is curl --aws-sigv4 with the restore key (already in
bootstrap.sh's secret list), so the CI image needs no new tool; the
OpenBao restore reads the same S3 endpoint and region.
- The six clusters that recovered from their archive store now name
<x>-store-source in externalClusters. serverName stays the cluster name
everywhere. The app Kustomizations depend on cnpg-object-stores, which
depends on infrastructure-cnpg-barman.
Verified against the bucket, read-only, 2026-10-16: the restore key lists
renner-uno-k8s-cnpg; the listing parser on real responses, top level and an
encoded nested prefix; for wp-pg it picks wp-pg-woodpecker (newest
20261016T024000) over wp-pg (20260818T023502); leftover test serverNames
hold no base backups and match no candidate pattern.
Production rollout, one time:
1. Create flux-system/cnpg-paths with the current prefixes as both source and
archive (ak-pg, ha-pg, im-pg, mx-pg, pprl-pg, tr-pg; wp-pg-woodpecker), so
the running clusters keep archiving where they do.
2. The app Kustomizations that give up their ObjectStores (authentik,
home-assistant, immich, matrix, paperless, trmnl, woodpecker) must run
with prune: false (the stack's first PR) when this merges;
cnpg-object-stores adopts the objects. All 14 also carry prune: disabled.
3. Whether changing externalClusters on a live Cluster restarts its pods is
not verified; check on one cluster first.
Older generations are not cleaned up: retention applies only to the current
archive prefix.
RECOVERY.md describes the new layout: where the current prefixes are
(cnpg-paths), the per-incarnation backup paths for the precondition check,
archiving to the new prefix after promotion, and adding a CNPG cluster to a
running cluster (its keys go into the existing cnpg-paths by hand). The
manual restore drill reads <x>-store, the current archive. The app
kustomization comments name cnpg-object-stores instead of
infrastructure-cnpg-barman. write_cnpg_paths checks for yq first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
57 lines
1.4 KiB
YAML
57 lines
1.4 KiB
YAML
---
|
|
apiVersion: barmancloud.cnpg.io/v1
|
|
kind: ObjectStore
|
|
metadata:
|
|
name: wp-pg-store
|
|
namespace: woodpecker
|
|
annotations:
|
|
kustomize.toolkit.fluxcd.io/prune: disabled
|
|
spec:
|
|
configuration:
|
|
destinationPath: ${CNPG_WP_PG_ARCHIVE}
|
|
endpointURL: https://fsn1.your-objectstorage.com
|
|
s3Credentials:
|
|
accessKeyId:
|
|
name: hetzner-s3-creds
|
|
key: ACCESS_KEY_ID
|
|
secretAccessKey:
|
|
name: hetzner-s3-creds
|
|
key: ACCESS_SECRET_KEY
|
|
region:
|
|
name: hetzner-s3-creds
|
|
key: REGION
|
|
wal:
|
|
compression: gzip
|
|
data:
|
|
compression: gzip
|
|
retentionPolicy: "30d"
|
|
---
|
|
apiVersion: barmancloud.cnpg.io/v1
|
|
kind: ObjectStore
|
|
metadata:
|
|
name: wp-pg-store-source
|
|
namespace: woodpecker
|
|
annotations:
|
|
kustomize.toolkit.fluxcd.io/prune: disabled
|
|
cnpg.renner.uno/server-name: wp-pg
|
|
# wp-pg's archive since its move out of `apps`.
|
|
cnpg.renner.uno/source-prefixes: wp-pg-woodpecker
|
|
spec:
|
|
configuration:
|
|
destinationPath: ${CNPG_WP_PG_SOURCE}
|
|
endpointURL: https://fsn1.your-objectstorage.com
|
|
s3Credentials:
|
|
accessKeyId:
|
|
name: hetzner-s3-creds
|
|
key: ACCESS_KEY_ID
|
|
secretAccessKey:
|
|
name: hetzner-s3-creds
|
|
key: ACCESS_SECRET_KEY
|
|
region:
|
|
name: hetzner-s3-creds
|
|
key: REGION
|
|
wal:
|
|
compression: gzip
|
|
data:
|
|
compression: gzip
|