konairius/homelab
10
0
Fork 0
Code Issues 8 Pull Requests 2 Packages Projects Releases 1 Wiki Activity

feat(cnpg): every cluster incarnation archives to a new prefix; recovery reads the newest
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
Details
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
Details
review-bot reviewed: comment
Details

Browse Source 
A CNPG cluster restored from barman archived back into the location it
recovered from. plugin-barman-cloud's restore job checks the archive target
(plugins[] ObjectStore + serverName) and stops with "Expected empty archive"
when it holds WAL, which after a disaster it always does. It has no
exception for target == recovery source (internal/cnpgi/restore/restore.go,
v0.15). Restore rehearsal run 28 hit it on wp-pg; the other six never got
that far.

The check stays. Instead, archive and source become two ObjectStores whose
paths are chosen per incarnation:

- kubernetes/cnpg-object-stores/: all 14 ObjectStores (<x>-store archives,
  <x>-store-source is read by bootstrap.recovery), in a Kustomization of their
  own with postBuild.substituteFrom: cnpg-paths. Nothing else lives there:
  substitution blanks every ${...}, which is why it is not on the app
  Kustomizations (home-assistant's scripts, the Grafana dashboards). The
  ConfigMap is not optional; without it the paths would render empty.
  woodpecker's objectstore.yaml moves there as wp-pg.yaml.
- bootstrap.sh writes cnpg-paths in the flux phase, before Flux starts, once
  per incarnation (an existing one is kept):
    CNPG_<X>_SOURCE   the prefix among <x>, <x>-g<14 digits> and the store's
                      cnpg.renner.uno/source-prefixes whose <x>/base/ holds
                      the newest completed base backup (backup.info
                      status=DONE; barman uploads it as STARTED when a
                      backup begins, barman src/barman/cloud.py
                      coordinate_backup)
    CNPG_<X>_ARCHIVE  <x>-g<UTC timestamp>, new and so empty
  Every ScheduledBackup has immediate: true, so a new incarnation has a base
  backup in its own prefix minutes after it starts, and a later recovery
  picks it. S3 access is curl --aws-sigv4 with the restore key (already in
  bootstrap.sh's secret list), so the CI image needs no new tool; the
  OpenBao restore reads the same S3 endpoint and region.
- The six clusters that recovered from their archive store now name
  <x>-store-source in externalClusters. serverName stays the cluster name
  everywhere. The app Kustomizations depend on cnpg-object-stores, which
  depends on infrastructure-cnpg-barman.

Verified against the bucket, read-only, 2026-10-16: the restore key lists
renner-uno-k8s-cnpg; the listing parser on real responses, top level and an
encoded nested prefix; for wp-pg it picks wp-pg-woodpecker (newest
20261016T024000) over wp-pg (20260818T023502); leftover test serverNames
hold no base backups and match no candidate pattern.

Production rollout, one time:
1. Create flux-system/cnpg-paths with the current prefixes as both source and
   archive (ak-pg, ha-pg, im-pg, mx-pg, pprl-pg, tr-pg; wp-pg-woodpecker), so
   the running clusters keep archiving where they do.
2. The app Kustomizations that give up their ObjectStores (authentik,
   home-assistant, immich, matrix, paperless, trmnl, woodpecker) must run
   with prune: false (the stack's first PR) when this merges;
   cnpg-object-stores adopts the objects. All 14 also carry prune: disabled.
3. Whether changing externalClusters on a live Cluster restarts its pods is
   not verified; check on one cluster first.

Older generations are not cleaned up: retention applies only to the current
archive prefix.

RECOVERY.md describes the new layout: where the current prefixes are
(cnpg-paths), the per-incarnation backup paths for the precondition check,
archiving to the new prefix after promotion, and adding a CNPG cluster to a
running cluster (its keys go into the existing cnpg-paths by hand). The
manual restore drill reads <x>-store, the current archive. The app
kustomization comments name cnpg-object-stores instead of
infrastructure-cnpg-barman. write_cnpg_paths checks for yq first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
claude-bot
2026-10-17 22:18:14 +02:00
co-authored by Claude Opus 5.5
parent 93cb53130c
commit a17c4e8161
37 changed files with 587 additions and 289 deletions
Download Patch File Download Diff File Expand all files Collapse all files
kubernetes/RECOVERY.md
+34 -16
View File
@@ -6,18 +6,31 @@ below (like `wp-pg`) keeps `bootstrap.recovery` in git: a cold rebuild restores
each from its backup with no manual step. This document covers the
precondition, watching and verifying that restore, and the reasoning behind it.
| Cluster | Namespace | ObjectStore | barman path (`destinationPath`) | app DB / owner |
|-----------|----------------|-----------------|--------------------------------------|---------------------------------|
| `ak-pg` | authentik | `ak-pg-store` | `s3://renner-uno-k8s-cnpg/ak-pg` | `authentik`/`authentik` |
| `im-pg` | immich | `im-pg-store` | `s3://renner-uno-k8s-cnpg/im-pg` | `immich`/`immich` |
| `pprl-pg` | paperless | `pprl-pg-store` | `s3://renner-uno-k8s-cnpg/pprl-pg` | `paperless`/`paperless` |
| `ha-pg` | home-assistant | `ha-pg-store` | `s3://renner-uno-k8s-cnpg/ha-pg` | `homeassistant`/`homeassistant` |
| `mx-pg` | matrix | `mx-pg-store` | `s3://renner-uno-k8s-cnpg/mx-pg` | `synapse`/`synapse` |
| `tr-pg` | trmnl | `tr-pg-store` | `s3://renner-uno-k8s-cnpg/tr-pg` | `terminus`/`terminus` |
| Cluster | Namespace | app DB / owner |
|-----------|----------------|---------------------------------|
| `ak-pg` | authentik | `authentik`/`authentik` |
| `im-pg` | immich | `immich`/`immich` |
| `pprl-pg` | paperless | `paperless`/`paperless` |
| `ha-pg` | home-assistant | `homeassistant`/`homeassistant` |
| `mx-pg` | matrix | `synapse`/`synapse` |
| `tr-pg` | trmnl | `terminus`/`terminus` |
| `wp-pg` | woodpecker | `woodpecker`/`woodpecker` |
The barman **`serverName`** is the last path segment of `destinationPath`, i.e.
it equals the cluster name. `gf-pg` (monitoring) has no ObjectStore and stays
on `initdb`.
Each cluster `<x>` has two ObjectStores in `kubernetes/cnpg-object-stores/`:
`<x>-store`, where it archives WAL and takes base backups, and
`<x>-store-source`, which `bootstrap.recovery` reads. Their paths come from the
`flux-system/cnpg-paths` ConfigMap, which `bootstrap.sh` writes once per
cluster incarnation: the source is the prefix in `s3://renner-uno-k8s-cnpg`
whose `<x>/base/` holds the newest base backup, the archive a new prefix
`<x>-g<timestamp>`. The barman **`serverName`** is the cluster name in both.
`gf-pg` (monitoring) has no ObjectStore and stays on `initdb`.
The current prefixes: `kubectl -n flux-system get cm cnpg-paths -o yaml`
(`CNPG_<X>_SOURCE`, `CNPG_<X>_ARCHIVE`). `bootstrap.sh` never rewrites an
existing `cnpg-paths`. Adding a CNPG cluster to a running cluster therefore
means adding its two keys to that ConfigMap by hand, with the archive prefix
as the source as well; without them `cnpg-object-stores` fails to build and
every app Kustomization behind it stalls.
---
@@ -52,12 +65,16 @@ Example uses `ak-pg`; substitute the row from the table above for the others.
### 1. Precondition — confirm a backup actually exists
Never start a recovery you can't finish. Check the object store has a base
backup and recent WAL for the cluster's `serverName` before touching anything:
backup and recent WAL for the cluster's `serverName` before touching anything.
Backups live under `<prefix>/<serverName>/base/`, one prefix per incarnation
(`ak-pg` and `ak-pg-g<timestamp>`):
```sh
# From a machine with the Hetzner S3 creds (see hetzner-creds-externalsecret):
aws --endpoint-url https://fsn1.your-objectstorage.com \
s3 ls s3://renner-uno-k8s-cnpg/ak-pg/base/
s3 ls s3://renner-uno-k8s-cnpg/ | grep ak-pg
aws --endpoint-url https://fsn1.your-objectstorage.com \
s3 ls s3://renner-uno-k8s-cnpg/<prefix>/ak-pg/base/
```
If that is empty, **stop** — recovery will hang forever (no initdb fallback).
@@ -73,9 +90,10 @@ drives the restore.
brings back *all* databases and roles, not just the app DB. The
`database`/`owner` fields only tell CNPG which app user the `-app` service
should target after promotion.
- After the restored primary is promoted, CNPG bumps the PostgreSQL timeline
and `spec.plugins[].isWALArchiver: true` resumes archiving to the same
`serverName` path without clobbering the old WAL.
- After the restored primary is promoted, `spec.plugins[].isWALArchiver: true`
archives to `<x>-store`, the new incarnation's own empty prefix; the
prefix it recovered from is left untouched. Its first base backup there is
taken right away (`immediate: true` on the ScheduledBackup).
```sh
kubectl -n authentik get cluster ak-pg -w
kubernetes/apps/authentik/cluster.yaml
+1 -1
View File
@@ -47,7 +47,7 @@ spec:
plugin:
name: barman-cloud.cloudnative-pg.io
parameters:
barmanObjectName: ak-pg-store
barmanObjectName: ak-pg-store-source
serverName: ak-pg
postgresql:
parameters:
kubernetes/apps/authentik/kustomization.yaml
+2 -3
View File
@@ -11,10 +11,9 @@ resources:
- ldap-outpost-cnp.yaml
# ak-pg CNPG Postgres backend (moved here from the retired seed-authentik KS;
# each carries prune: disabled so ownership transferred without a teardown).
# authentik has no data PVCs — all state lives in ak-pg. This KS gains the
# infrastructure-cnpg-barman dep + CNPG Cluster healthCheckExprs (see
# authentik has no data PVCs — all state lives in ak-pg. This KS depends on
# cnpg-object-stores and health-checks the CNPG Cluster (see
# clusters/homelab/authentik.yaml).
- cluster.yaml
- objectstore.yaml
- scheduled-backup.yaml
- hetzner-creds-externalsecret.yaml
kubernetes/apps/authentik/objectstore.yaml
-33
View File
@@ -1,33 +0,0 @@
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: ak-pg-store
namespace: authentik
annotations:
# Prune-disabled — a standing guard kept from the seed->authentik-KS handoff (backup target).
kustomize.toolkit.fluxcd.io/prune: disabled
spec:
configuration:
# Offsite Hetzner Object Storage (fsn1) — replaces Garage on nas-1.
# No SSE: Hetzner supports only SSE-C, unusable by barman-cloud
# (plugin #646), so at-rest encryption is unavailable; transit is TLS.
destinationPath: s3://renner-uno-k8s-cnpg/ak-pg
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
# Region is required (REGION=fsn1) — without it boto3's HeadBucket
# (barman-cloud-check-wal-archive during bootstrap.recovery) 400s.
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
retentionPolicy: "30d"
kubernetes/apps/home-assistant/ha-pg-cluster.yaml
+1 -1
View File
@@ -116,7 +116,7 @@ spec:
plugin:
name: barman-cloud.cloudnative-pg.io
parameters:
barmanObjectName: ha-pg-store
barmanObjectName: ha-pg-store-source
serverName: ha-pg
postgresql:
parameters:
kubernetes/apps/home-assistant/ha-pg-objectstore.yaml
-35
View File
@@ -1,35 +0,0 @@
---
# Barman object store for ha-pg: base backups + continuous WAL archiving to
# Hetzner Object Storage, same target bucket as the other CNPG clusters.
#
# Its own prefix. Two clusters archiving WAL to one destinationPath interleave
# their streams and corrupt both — see the note in
# kubernetes/apps/woodpecker/objectstore.yaml, which learned that the hard way.
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: ha-pg-store
namespace: home-assistant
spec:
configuration:
destinationPath: s3://renner-uno-k8s-cnpg/ha-pg
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
# Matches the 30d retention the other CNPG clusters use and the
# renner-uno-k8s-cnpg bucket's noncurrent-version expiry in
# opentofu/object_storage.tf, so old S3 versions don't outlive the backups
# that superseded them.
retentionPolicy: "30d"
kubernetes/apps/home-assistant/kustomization.yaml
-1
View File
@@ -42,7 +42,6 @@ resources:
# Live: recorder.db_url in the homeassistant repo points here, and the
# long-term statistics carry their full history from 2024.
- ha-pg-hetzner-creds-externalsecret.yaml
- ha-pg-objectstore.yaml
- ha-pg-cluster.yaml
- ha-pg-scheduled-backup.yaml
# MQTT credential management (REN-802): the account controller manages the
kubernetes/apps/immich/cluster.yaml
+1 -1
View File
@@ -43,7 +43,7 @@ spec:
plugin:
name: barman-cloud.cloudnative-pg.io
parameters:
barmanObjectName: im-pg-store
barmanObjectName: im-pg-store-source
serverName: im-pg
postgresql:
# vchord only. The image also ships pgvecto.rs (vectors.so), but its
kubernetes/apps/immich/kustomization.yaml
+2 -3
View File
@@ -8,10 +8,9 @@ resources:
- ingress.yaml
# im-pg CNPG Postgres + library/ml-cache PVCs (moved here from the retired
# seed-immich KS; each carries prune: disabled so ownership transferred without
# a teardown). This KS gains the infrastructure-cnpg-barman dep + CNPG Cluster
# healthCheckExprs (see clusters/homelab/immich.yaml).
# a teardown). This KS depends on cnpg-object-stores and health-checks the
# CNPG Cluster (see clusters/homelab/immich.yaml).
- cluster.yaml
- objectstore.yaml
- scheduled-backup.yaml
- hetzner-creds-externalsecret.yaml
- pvcs.yaml
kubernetes/apps/immich/objectstore.yaml
-33
View File
@@ -1,33 +0,0 @@
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: im-pg-store
namespace: immich
annotations:
# Prune-disabled — a standing guard kept from the seed->immich-KS handoff (backup target).
kustomize.toolkit.fluxcd.io/prune: disabled
spec:
configuration:
# Offsite Hetzner Object Storage (fsn1) — replaces Garage on nas-1.
# No SSE: Hetzner supports only SSE-C, unusable by barman-cloud
# (plugin #646), so at-rest encryption is unavailable; transit is TLS.
destinationPath: s3://renner-uno-k8s-cnpg/im-pg
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
# Region is required (REGION=fsn1) — without it boto3's HeadBucket
# (barman-cloud-check-wal-archive during bootstrap.recovery) 400s.
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
retentionPolicy: "30d"
kubernetes/apps/matrix/cluster.yaml
+1 -1
View File
@@ -31,7 +31,7 @@ spec:
plugin:
name: barman-cloud.cloudnative-pg.io
parameters:
barmanObjectName: mx-pg-store
barmanObjectName: mx-pg-store-source
serverName: mx-pg
postgresql:
parameters:
kubernetes/apps/matrix/kustomization.yaml
-1
View File
@@ -9,6 +9,5 @@ resources:
# mx-pg CNPG Postgres + offsite Barman backup. The media store and signing
# key are on synapse-data, backed up by Velero (matrix-daily).
- cluster.yaml
- objectstore.yaml
- scheduled-backup.yaml
- hetzner-creds-externalsecret.yaml
kubernetes/apps/matrix/objectstore.yaml
-30
View File
@@ -1,30 +0,0 @@
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: mx-pg-store
namespace: matrix
spec:
configuration:
# Offsite Hetzner Object Storage (fsn1), same bucket as the other CNPG
# clusters. No SSE: Hetzner supports only SSE-C, unusable by barman-cloud
# (plugin #646), so at-rest encryption is unavailable; transit is TLS.
destinationPath: s3://renner-uno-k8s-cnpg/mx-pg
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
# Region is required (REGION=fsn1) — without it boto3's HeadBucket
# (barman-cloud-check-wal-archive during bootstrap.recovery) 400s.
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
retentionPolicy: "30d"
kubernetes/apps/paperless/cluster.yaml
+1 -1
View File
@@ -39,7 +39,7 @@ spec:
plugin:
name: barman-cloud.cloudnative-pg.io
parameters:
barmanObjectName: pprl-pg-store
barmanObjectName: pprl-pg-store-source
serverName: pprl-pg
postgresql:
parameters:
kubernetes/apps/paperless/kustomization.yaml
+2 -3
View File
@@ -14,10 +14,9 @@ resources:
- network-policy.yaml
# pprl-pg CNPG Postgres + data/media/consume/redis PVCs (moved here from the
# retired seed-paperless KS; each carries prune: disabled so ownership
# transferred without a teardown). This KS gains the infrastructure-cnpg-barman
# dep + CNPG Cluster healthCheckExprs (see clusters/homelab/paperless.yaml).
# transferred without a teardown). This KS depends on cnpg-object-stores and
# health-checks the CNPG Cluster (see clusters/homelab/paperless.yaml).
- cluster.yaml
- objectstore.yaml
- scheduled-backup.yaml
- hetzner-creds-externalsecret.yaml
- pvcs.yaml
kubernetes/apps/paperless/objectstore.yaml
-35
View File
@@ -1,35 +0,0 @@
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: pprl-pg-store
namespace: paperless
annotations:
# Prune-disabled — a standing guard kept from the seed->paperless-KS handoff (backup target).
kustomize.toolkit.fluxcd.io/prune: disabled
spec:
configuration:
# Offsite Hetzner Object Storage (fsn1) — replaces local Garage on
# nas-1 so paperless backups survive nas-1 being powered off. barman
# uses boto3 default (auto) addressing; Hetzner (Ceph RGW) accepts it.
destinationPath: s3://renner-uno-k8s-cnpg/pprl-pg
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
# No `encryption: AES256` (SSE-S3): Hetzner Object Storage supports
# only SSE-C, which the barman-cloud plugin can't use (cnpg
# plugin-barman-cloud #646). At-rest encryption is therefore not
# available here; transit is TLS and the bucket is private.
wal:
compression: gzip
data:
compression: gzip
retentionPolicy: "30d"
kubernetes/apps/trmnl/cluster.yaml
+1 -1
View File
@@ -36,7 +36,7 @@ spec:
plugin:
name: barman-cloud.cloudnative-pg.io
parameters:
barmanObjectName: tr-pg-store
barmanObjectName: tr-pg-store-source
serverName: tr-pg
postgresql:
parameters:
kubernetes/apps/trmnl/kustomization.yaml
-1
View File
@@ -18,6 +18,5 @@ resources:
# tr-pg CNPG Postgres + offsite Barman backup. Note this cluster runs
# PostgreSQL 18, not the repo-standard 16 — see the comment in cluster.yaml.
- cluster.yaml
- objectstore.yaml
- scheduled-backup.yaml
- hetzner-creds-externalsecret.yaml
kubernetes/apps/trmnl/objectstore.yaml
-30
View File
@@ -1,30 +0,0 @@
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: tr-pg-store
namespace: trmnl
spec:
configuration:
# Offsite Hetzner Object Storage (fsn1), same bucket as the other CNPG
# clusters. No SSE: Hetzner supports only SSE-C, unusable by barman-cloud
# (plugin #646), so at-rest encryption is unavailable; transit is TLS.
destinationPath: s3://renner-uno-k8s-cnpg/tr-pg
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
# Region is required (REGION=fsn1) — without it boto3's HeadBucket
# (barman-cloud-check-wal-archive during bootstrap.recovery) 400s.
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
retentionPolicy: "30d"
kubernetes/apps/woodpecker/kustomization.yaml
+2 -3
View File
@@ -31,9 +31,8 @@ resources:
# not here: it must exist before the `infrastructure-crds` KS applies the
# namespace's CiliumNetworkPolicies, and this KS runs after both of those.
#
# This KS depends on infrastructure-cnpg-barman for the barman plugin behind
# the ObjectStore and ScheduledBackup.
# This KS depends on cnpg-object-stores for the ObjectStores its Cluster and
# ScheduledBackup name.
- hetzner-creds-externalsecret.yaml
- objectstore.yaml
- cluster.yaml
- scheduled-backup.yaml
kubernetes/bootstrap.sh
+125 -2
View File
@@ -383,6 +383,125 @@ read_secret() {
cat "$OPENBAO_SECRETS_DIR/$name"
}
# Hetzner Object Storage, where the offsite backups live. Read with the restore
# key (s3-access, s3-secret).
S3_ENDPOINT="fsn1.your-objectstorage.com"
S3_REGION="fsn1"
# One request, signed by curl with the restore key. The body goes to stdout;
# anything but a 2xx fails, with S3's error document on stderr.
s3_request() {
local method=$1 path=$2 body code
body=$(mktemp)
if ! code=$(curl -sS -o "$body" -w '%{http_code}' -X "$method" \
--aws-sigv4 "aws:amz:$S3_REGION:s3" \
--user "$(read_secret s3-access):$(read_secret s3-secret)" \
"https://$S3_ENDPOINT/$path"); then
rm -f "$body"
return 1
fi
if [[ "$code" != 2?? ]]; then
echo "$method $path: HTTP $code" >&2
cat "$body" >&2
echo >&2
rm -f "$body"
return 1
fi
cat "$body"
rm -f "$body"
}
# The "directories" directly under PREFIX in BUCKET, one per line, without
# PREFIX and the trailing slash.
s3_dirs() {
local bucket=$1 prefix=$2 xml
xml=$(s3_request GET "$bucket?list-type=2&delimiter=%2F&prefix=${prefix//\//%2F}") || return 1
[[ "$xml" == *"<IsTruncated>false</IsTruncated>"* ]] \
|| { echo "the listing of $bucket/$prefix is truncated or not a listing" >&2; return 1; }
# awk, not grep: no match is not an error.
awk -v p="$prefix" '{
while (match($0, /<CommonPrefixes><Prefix>[^<]*<\/Prefix>/)) {
d = substr($0, RSTART + 24, RLENGTH - 33)
$0 = substr($0, RSTART + RLENGTH)
sub(/\/$/, "", d)
print substr(d, length(p) + 1)
}
}' <<<"$xml"
}
# The bucket every CNPG cluster recovers from (cnpg-object-stores/).
CNPG_BUCKET="renner-uno-k8s-cnpg"
CNPG_STORES_DIR="$REPO_ROOT/kubernetes/cnpg-object-stores"
# cnpg-paths, the ConfigMap cnpg-object-stores takes its paths from. Written
# once per cluster incarnation: an existing one belongs to clusters already
# archiving under its paths and is kept.
#
# For each source ObjectStore in cnpg-object-stores/, cluster <x>:
# CNPG_<X>_SOURCE the candidate prefix whose <x>/base/ holds the newest
# completed base backup. Candidates: <x>, <x>-g<14 digits> and the
# store's cnpg.renner.uno/source-prefixes.
# CNPG_<X>_ARCHIVE <x>-g<now> in ARCHIVE_BUCKET: new, so empty, so the
# plugin's empty-archive check passes rather than being
# skipped.
write_cnpg_paths() {
local archive_bucket=$1
local gen dirs name extra var p listing newest best best_at id info
local -a literals=()
echo "Writing the CNPG backup paths (cnpg-paths)..."
command -v yq >/dev/null || fail "yq not found (reads $CNPG_STORES_DIR)"
if kubectl -n flux-system get configmap cnpg-paths -o name >/dev/null 2>"$OPENBAO_SECRETS_DIR/.err"; then
echo " already present, kept"
return 0
fi
grep -q NotFound "$OPENBAO_SECRETS_DIR/.err" \
|| { cat "$OPENBAO_SECRETS_DIR/.err" >&2; fail "could not read the cnpg-paths ConfigMap"; }
gen=g$(date -u +%Y%m%d%H%M%S)
dirs=$(s3_dirs "$CNPG_BUCKET" "") || fail "could not list $CNPG_BUCKET"
while IFS=$'\t' read -r name extra; do
best=''
best_at=''
for p in $dirs; do
[[ "$p" == "$name" || "$p" =~ ^${name}-g[0-9]{14}$ || " $extra " == *" $p "* ]] || continue
listing=$(s3_dirs "$CNPG_BUCKET" "$p/$name/base/") \
|| fail "could not list $CNPG_BUCKET/$p/$name/base/"
# Newest completed backup: barman uploads backup.info with
# status=STARTED when a backup begins and rewrites it with DONE or
# FAILED when it ends.
newest=''
for id in $(sort -r <<<"$listing"); do
[[ "$id" > "$best_at" ]] || break
info=$(s3_request GET "$CNPG_BUCKET/$p/$name/base/$id/backup.info") \
|| fail "could not read $CNPG_BUCKET/$p/$name/base/$id/backup.info"
if grep -qx 'status=DONE' <<<"$info"; then
newest=$id
break
fi
echo " $name: skipping $p/$name/base/$id ($(grep '^status=' <<<"$info" || echo 'no status'))"
done
if [[ -n "$newest" ]]; then
best=$p
best_at=$newest
fi
done
[[ -n "$best" ]] \
|| fail "no completed base backup of $name in $CNPG_BUCKET (looked under $name, $name-g<timestamp>${extra:+, $extra})"
var=CNPG_$(tr 'a-z-' 'A-Z_' <<<"$name")
literals+=("--from-literal=${var}_SOURCE=s3://$CNPG_BUCKET/$best"
"--from-literal=${var}_ARCHIVE=s3://$archive_bucket/$name-$gen")
echo " $name: recovers from $CNPG_BUCKET/$best (base backup $best_at), archives to $archive_bucket/$name-$gen"
done < <(yq -r 'select(.kind == "ObjectStore" and .metadata.annotations["cnpg.renner.uno/server-name"] != null)
| [.metadata.annotations["cnpg.renner.uno/server-name"],
(.metadata.annotations["cnpg.renner.uno/source-prefixes"] // "")] | @tsv' \
"$CNPG_STORES_DIR"/*.yaml)
[[ ${#literals[@]} -gt 0 ]] \
|| fail "no source ObjectStore (cnpg.renner.uno/server-name) in $CNPG_STORES_DIR"
kubectl -n flux-system create configmap cnpg-paths "${literals[@]}" >/dev/null \
|| fail "could not create the cnpg-paths ConfigMap"
ok
}
# Provision a throwaway cluster on Hetzner Cloud, then run the Talos phase
# against it unchanged — same shape as the vmtest phase below, and for the same
# reason: a rehearsal that runs different code proves less than one that does
@@ -1547,8 +1666,8 @@ bao_exec() {
# OpenBao — so Velero cannot be the thing that restores OpenBao.
OPENBAO_KOPIA_BUCKET="renner-uno-k8s-velero"
OPENBAO_KOPIA_PREFIX="kopia/openbao/"
OPENBAO_S3_ENDPOINT="fsn1.your-objectstorage.com"
OPENBAO_S3_REGION="fsn1"
OPENBAO_S3_ENDPOINT="$S3_ENDPOINT"
OPENBAO_S3_REGION="$S3_REGION"
# Velero's hardcoded backup repository password (pkg/repository/keys/keys.go).
OPENBAO_KOPIA_PASSWORD="static-passw0rd"
@@ -1895,6 +2014,10 @@ if phase_runs flux; then
fi
unset seal_key existing
# Before Flux: cnpg-object-stores fails without it rather than render empty
# paths, and the paths must exist before any Cluster bootstraps.
write_cnpg_paths "$CNPG_BUCKET"
if [[ "$FLUX_SOURCE" == "github" ]]; then
step 0 "Bootstrapping FluxCD from the GitHub break-glass mirror..."
echo -e "${YELLOW}BREAK-GLASS: Flux source is github.com/${GITHUB_OWNER}/${GITHUB_REPO}, NOT in-cluster Gitea.${NC}"
kubernetes/clusters/homelab/authentik.yaml
+1 -1
View File
@@ -24,7 +24,7 @@ spec:
- name: infrastructure
- name: infrastructure-crds
- name: infrastructure-ceph-cluster
- name: infrastructure-cnpg-barman
- name: cnpg-object-stores # the ObjectStores its Cluster names
- name: gitea # server, LDAP outpost and gate-sync images are in Gitea's registry
healthCheckExprs:
- apiVersion: postgresql.cnpg.io/v1
kubernetes/clusters/homelab/cnpg-object-stores.yaml
+27
View File
@@ -0,0 +1,27 @@
# The CNPG ObjectStores. Their paths come from the cnpg-paths ConfigMap, which
# bootstrap.sh writes once per cluster incarnation and never replaces:
# CNPG_<X>_SOURCE the bucket prefix holding <x>'s newest base backup
# CNPG_<X>_ARCHIVE a new, empty prefix <x>-g<timestamp>
# Not optional: without it every path renders empty.
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: cnpg-object-stores
namespace: flux-system
spec:
interval: 10m
retryInterval: 1m
timeout: 5m
sourceRef:
kind: GitRepository
name: flux-system
path: ./kubernetes/cnpg-object-stores
prune: true
wait: true
dependsOn:
- name: infrastructure
- name: infrastructure-cnpg-barman
postBuild:
substituteFrom:
- kind: ConfigMap
name: cnpg-paths
kubernetes/clusters/homelab/home-assistant.yaml
+1 -13
View File
@@ -23,19 +23,7 @@ spec:
- name: infrastructure
- name: infrastructure-crds
- name: infrastructure-ceph-cluster
# ha-pg-objectstore.yaml declares an ObjectStore, whose CRD comes from the
# CNPG Barman plugin in infrastructure-cnpg-barman. Without this edge the
# kind is unknown when this path is first applied, and a server-side
# dry-run rejects the ENTIRE set over the one object it cannot recognise:
#
# ObjectStore/home-assistant/ha-pg-store dry-run failed:
# no matches for kind "ObjectStore" in version ...
#
# Every other ObjectStore user — authentik, immich, paperless, trmnl,
# woodpecker — already declares it. This was the only one that did not, and
# it took speech-to-phrase, zone-configurator and matter-server down with
# it, since all three wait on home-assistant.
- name: infrastructure-cnpg-barman
- name: cnpg-object-stores # the ObjectStores its Cluster names
- name: gitea # mqtt-account-controller's image is in Gitea's registry
- name: influxdb # HA's InfluxDB integration writes there
---
kubernetes/clusters/homelab/immich.yaml
+1 -1
View File
@@ -24,7 +24,7 @@ spec:
- name: infrastructure
- name: infrastructure-crds
- name: infrastructure-ceph-cluster
- name: infrastructure-cnpg-barman
- name: cnpg-object-stores # the ObjectStores its Cluster names
healthCheckExprs:
- apiVersion: postgresql.cnpg.io/v1
kind: Cluster
kubernetes/clusters/homelab/matrix.yaml
+1 -1
View File
@@ -21,7 +21,7 @@ spec:
- name: infrastructure
- name: infrastructure-crds
- name: infrastructure-ceph-cluster
- name: infrastructure-cnpg-barman
- name: cnpg-object-stores # the ObjectStores its Cluster names
healthCheckExprs:
- apiVersion: postgresql.cnpg.io/v1
kind: Cluster
kubernetes/clusters/homelab/paperless.yaml
+1 -1
View File
@@ -24,7 +24,7 @@ spec:
- name: infrastructure
- name: infrastructure-crds
- name: infrastructure-ceph-cluster
- name: infrastructure-cnpg-barman
- name: cnpg-object-stores # the ObjectStores its Cluster names
- name: gitea # ftp-scanner (vsftpd) image is in Gitea's registry
healthCheckExprs:
- apiVersion: postgresql.cnpg.io/v1
kubernetes/clusters/homelab/trmnl.yaml
+1 -1
View File
@@ -21,7 +21,7 @@ spec:
- name: infrastructure
- name: infrastructure-crds
- name: infrastructure-ceph-cluster
- name: infrastructure-cnpg-barman
- name: cnpg-object-stores # the ObjectStores its Cluster names
- name: home-assistant # the renderer reads HA's API
healthCheckExprs:
- apiVersion: postgresql.cnpg.io/v1
kubernetes/clusters/homelab/woodpecker.yaml
+1 -1
View File
@@ -24,7 +24,7 @@ spec:
- name: infrastructure
- name: infrastructure-crds
- name: infrastructure-ceph-cluster
- name: infrastructure-cnpg-barman
- name: cnpg-object-stores # the ObjectStores its Cluster names
- name: gitea # its forge and login provider
healthCheckExprs:
- apiVersion: postgresql.cnpg.io/v1
kubernetes/cnpg-object-stores/ak-pg.yaml
+54
View File
@@ -0,0 +1,54 @@
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: ak-pg-store
namespace: authentik
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled
spec:
configuration:
destinationPath: ${CNPG_AK_PG_ARCHIVE}
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
retentionPolicy: "30d"
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: ak-pg-store-source
namespace: authentik
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled
cnpg.renner.uno/server-name: ak-pg
spec:
configuration:
destinationPath: ${CNPG_AK_PG_SOURCE}
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
kubernetes/cnpg-object-stores/ha-pg.yaml
+54
View File
@@ -0,0 +1,54 @@
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: ha-pg-store
namespace: home-assistant
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled
spec:
configuration:
destinationPath: ${CNPG_HA_PG_ARCHIVE}
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
retentionPolicy: "30d"
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: ha-pg-store-source
namespace: home-assistant
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled
cnpg.renner.uno/server-name: ha-pg
spec:
configuration:
destinationPath: ${CNPG_HA_PG_SOURCE}
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
kubernetes/cnpg-object-stores/im-pg.yaml
+54
View File
@@ -0,0 +1,54 @@
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: im-pg-store
namespace: immich
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled
spec:
configuration:
destinationPath: ${CNPG_IM_PG_ARCHIVE}
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
retentionPolicy: "30d"
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: im-pg-store-source
namespace: immich
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled
cnpg.renner.uno/server-name: im-pg
spec:
configuration:
destinationPath: ${CNPG_IM_PG_SOURCE}
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
kubernetes/cnpg-object-stores/kustomization.yaml
+24
View File
@@ -0,0 +1,24 @@
# Every CNPG cluster's two ObjectStores: <x>-store, where it archives WAL and
# takes base backups, and <x>-store-source, which its bootstrap.recovery reads.
#
# Their destinationPaths are Flux variables from the cnpg-paths ConfigMap
# (clusters/homelab/cnpg-object-stores.yaml). Nothing else may live in this
# directory: postBuild substitution blanks every ${...} it renders.
#
# The source store carries cnpg.renner.uno/server-name, the cluster whose
# backups bootstrap.sh looks for, and optionally
# cnpg.renner.uno/source-prefixes, bucket prefixes outside the <name> and
# <name>-g<timestamp> pattern that may still hold them.
#
# No SSE: Hetzner supports only SSE-C, unusable by barman-cloud (plugin #646),
# so at-rest encryption is unavailable; transit is TLS.
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ak-pg.yaml
- ha-pg.yaml
- im-pg.yaml
- mx-pg.yaml
- pprl-pg.yaml
- tr-pg.yaml
- wp-pg.yaml
kubernetes/cnpg-object-stores/mx-pg.yaml
+54
View File
@@ -0,0 +1,54 @@
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: mx-pg-store
namespace: matrix
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled
spec:
configuration:
destinationPath: ${CNPG_MX_PG_ARCHIVE}
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
retentionPolicy: "30d"
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: mx-pg-store-source
namespace: matrix
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled
cnpg.renner.uno/server-name: mx-pg
spec:
configuration:
destinationPath: ${CNPG_MX_PG_SOURCE}
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
kubernetes/cnpg-object-stores/pprl-pg.yaml
+54
View File
@@ -0,0 +1,54 @@
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: pprl-pg-store
namespace: paperless
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled
spec:
configuration:
destinationPath: ${CNPG_PPRL_PG_ARCHIVE}
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
retentionPolicy: "30d"
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: pprl-pg-store-source
namespace: paperless
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled
cnpg.renner.uno/server-name: pprl-pg
spec:
configuration:
destinationPath: ${CNPG_PPRL_PG_SOURCE}
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
kubernetes/cnpg-object-stores/tr-pg.yaml
+54
View File
@@ -0,0 +1,54 @@
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: tr-pg-store
namespace: trmnl
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled
spec:
configuration:
destinationPath: ${CNPG_TR_PG_ARCHIVE}
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
retentionPolicy: "30d"
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: tr-pg-store-source
namespace: trmnl
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled
cnpg.renner.uno/server-name: tr-pg
spec:
configuration:
destinationPath: ${CNPG_TR_PG_SOURCE}
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
kubernetes/apps/woodpecker/objectstore.yaml → kubernetes/cnpg-object-stores/wp-pg.yaml
+32 -35
View File
@@ -1,46 +1,14 @@
---
# RECOVERY SOURCE — read-only. The prefix wp-pg archives to (wp-pg-store below).
#
# serverName on the consuming externalCluster must be `wp-pg`, the name of the
# cluster that produced these backups.
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: wp-pg-store-source
namespace: woodpecker
spec:
configuration:
destinationPath: s3://renner-uno-k8s-cnpg/wp-pg-woodpecker
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
# No retentionPolicy: wp-pg-store owns the lifecycle of this prefix.
---
# ARCHIVING TARGET.
#
# Not s3://renner-uno-k8s-cnpg/wp-pg: the retired `apps/wp-pg` archived there,
# and two clusters archiving WAL to one destinationPath interleave their streams
# and corrupt both.
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: wp-pg-store
namespace: woodpecker
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled
spec:
configuration:
destinationPath: s3://renner-uno-k8s-cnpg/wp-pg-woodpecker
destinationPath: ${CNPG_WP_PG_ARCHIVE}
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
@@ -57,3 +25,32 @@ spec:
data:
compression: gzip
retentionPolicy: "30d"
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: wp-pg-store-source
namespace: woodpecker
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled
cnpg.renner.uno/server-name: wp-pg
# wp-pg's archive since its move out of `apps`.
cnpg.renner.uno/source-prefixes: wp-pg-woodpecker
spec:
configuration:
destinationPath: ${CNPG_WP_PG_SOURCE}
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip