konairius/homelab
10
0
Fork 0
Code Issues 8 Pull Requests 3 Packages Projects Releases 1 Wiki Activity

feat(cnpg): every cluster incarnation archives to a new prefix; recovery reads the newest

Browse Source 
A CNPG cluster restored from barman archived back into the location it
recovered from. plugin-barman-cloud's restore job checks the archive target
(plugins[] ObjectStore + serverName) and stops with "Expected empty archive"
when it holds WAL, which after a disaster it always does. It has no
exception for target == recovery source (internal/cnpgi/restore/restore.go,
v0.15). Run 28 hit it on wp-pg; the other six never got that far.

The check stays. Instead, archive and source become two ObjectStores whose
paths are chosen per incarnation:

- kubernetes/cnpg-object-stores/: all 14 ObjectStores (<x>-store archives,
  <x>-store-source is read by bootstrap.recovery), in a Kustomization of their
  own with postBuild.substituteFrom: cnpg-paths. Nothing else lives there:
  substitution blanks every ${...}, which is why it is not on the app
  Kustomizations (home-assistant's scripts, the Grafana dashboards). The
  ConfigMap is not optional; without it the paths would render empty.
- bootstrap.sh writes cnpg-paths before Flux starts, once per incarnation
  (an existing one is kept):
    CNPG_<X>_SOURCE   the prefix among <x>, <x>-g<14 digits> and the store's
                      cnpg.renner.uno/source-prefixes whose <x>/base/ holds
                      the newest base backup
    CNPG_<X>_ARCHIVE  <x>-g<UTC timestamp>, new and so empty
  Every ScheduledBackup has immediate: true, so a new incarnation has a base
  backup in its own prefix minutes after it starts, and a later recovery
  picks it. S3 access is curl --aws-sigv4 with the restore key (already in
  bootstrap.sh's secret list), so the CI image needs no new tool.
- The restore test archives to renner-uno-k8s-cnpg-rtest, created by the
  flux phase and emptied and deleted by hcloudclean. A test incarnation with a
  base backup in the production bucket would be its newest, and the next real
  recovery would read the test's data.
- The six clusters that recovered from their archive store now name
  <x>-store-source in externalClusters. serverName stays the cluster name
  everywhere. The app Kustomizations depend on cnpg-object-stores, which
  depends on infrastructure-cnpg-barman.

Verified against the bucket, read-only, 2026-10-16:
- the restore key lists renner-uno-k8s-cnpg and can create and delete a
  bucket (renner-uno-k8s-cnpg-rtest created and deleted again);
- the listing parser on real responses, top level and an encoded nested
  prefix; for wp-pg it picks wp-pg-woodpecker (newest 20261016T024000) over
  wp-pg (20260818T023502);
- leftover test serverNames (ak-pg-rtest, im-pg-rtest, ha-pg-restore-test,
  pprleval-pg) hold no base backups and match no candidate pattern.
shellcheck clean; all seven apps and cnpg-object-stores build; the render
holds exactly the 14 expected variables.

Production rollout, one time:
1. Create flux-system/cnpg-paths with the current prefixes as both source and
   archive (ak-pg, ha-pg, im-pg, mx-pg, pprl-pg, tr-pg; wp-pg-woodpecker), so
   the running clusters keep archiving where they do.
2. Move the ObjectStores in two phases (add in cnpg-object-stores, then drop
   from the apps). All 14 now carry prune: disabled; before this only ak, im
   and pprl did.
3. Whether changing externalClusters on a live Cluster restarts its pods is
   not verified; check on one cluster first.

Older generations are not cleaned up: retention applies only to the current
archive prefix. A cleanup script is planned separately.

RECOVERY.md: the table describes the new layout; the manual restore drill
reads <x>-store-source. That drill wrote 3 WAL files to ak-pg/ak-pg-rtest on
2026-07-25 despite isWALArchiver: false, so its "only ever reads" claim
does not hold; not changed here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
claude-bot
2026-10-16 21:27:21 +02:00
co-authored by Claude Opus 5.5
parent 43dfa6158c
commit cffb7e027a
37 changed files with 600 additions and 277 deletions
Download Patch File Download Diff File Expand all files Collapse all files
kubernetes/RECOVERY.md
+18 -12
View File
@@ -6,18 +6,24 @@ below (like `wp-pg`) keeps `bootstrap.recovery` in git: a cold rebuild restores
each from its backup with no manual step. This document covers the
precondition, watching and verifying that restore, and the reasoning behind it.
| Cluster | Namespace | ObjectStore | barman path (`destinationPath`) | app DB / owner |
|-----------|----------------|-----------------|--------------------------------------|---------------------------------|
| `ak-pg` | authentik | `ak-pg-store` | `s3://renner-uno-k8s-cnpg/ak-pg` | `authentik`/`authentik` |
| `im-pg` | immich | `im-pg-store` | `s3://renner-uno-k8s-cnpg/im-pg` | `immich`/`immich` |
| `pprl-pg` | paperless | `pprl-pg-store` | `s3://renner-uno-k8s-cnpg/pprl-pg` | `paperless`/`paperless` |
| `ha-pg` | home-assistant | `ha-pg-store` | `s3://renner-uno-k8s-cnpg/ha-pg` | `homeassistant`/`homeassistant` |
| `mx-pg` | matrix | `mx-pg-store` | `s3://renner-uno-k8s-cnpg/mx-pg` | `synapse`/`synapse` |
| `tr-pg` | trmnl | `tr-pg-store` | `s3://renner-uno-k8s-cnpg/tr-pg` | `terminus`/`terminus` |
| Cluster | Namespace | app DB / owner |
|-----------|----------------|---------------------------------|
| `ak-pg` | authentik | `authentik`/`authentik` |
| `im-pg` | immich | `immich`/`immich` |
| `pprl-pg` | paperless | `paperless`/`paperless` |
| `ha-pg` | home-assistant | `homeassistant`/`homeassistant` |
| `mx-pg` | matrix | `synapse`/`synapse` |
| `tr-pg` | trmnl | `terminus`/`terminus` |
| `wp-pg` | woodpecker | `woodpecker`/`woodpecker` |
The barman **`serverName`** is the last path segment of `destinationPath`, i.e.
it equals the cluster name. `gf-pg` (monitoring) has no ObjectStore and stays
on `initdb`.
Each cluster `<x>` has two ObjectStores in `kubernetes/cnpg-object-stores/`:
`<x>-store`, where it archives WAL and takes base backups, and
`<x>-store-source`, which `bootstrap.recovery` reads. Their paths come from the
`flux-system/cnpg-paths` ConfigMap, which `bootstrap.sh` writes once per
cluster incarnation: the source is the prefix in `s3://renner-uno-k8s-cnpg`
whose `<x>/base/` holds the newest base backup, the archive a new prefix
`<x>-g<timestamp>`. The barman **`serverName`** is the cluster name in both.
`gf-pg` (monitoring) has no ObjectStore and stays on `initdb`.
---
@@ -129,7 +135,7 @@ cluster in the *same* namespace (so it can reference the existing namespaced
restore, but never archiving, so it only ever *reads* the backup path.
```sh
NS=authentik ; SRC=ak-pg ; STORE=ak-pg-store ; DB=authentik
NS=authentik ; SRC=ak-pg ; STORE=ak-pg-store-source ; DB=authentik
kubectl apply -f - <<EOF
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
kubernetes/apps/authentik/cluster.yaml
+1 -1
View File
@@ -47,7 +47,7 @@ spec:
plugin:
name: barman-cloud.cloudnative-pg.io
parameters:
barmanObjectName: ak-pg-store
barmanObjectName: ak-pg-store-source
serverName: ak-pg
postgresql:
parameters:
kubernetes/apps/authentik/kustomization.yaml
-1
View File
@@ -15,6 +15,5 @@ resources:
# infrastructure-cnpg-barman dep + CNPG Cluster healthCheckExprs (see
# clusters/homelab/authentik.yaml).
- cluster.yaml
- objectstore.yaml
- scheduled-backup.yaml
- hetzner-creds-externalsecret.yaml
kubernetes/apps/authentik/objectstore.yaml
-33
View File
@@ -1,33 +0,0 @@
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: ak-pg-store
namespace: authentik
annotations:
# Prune-disabled — a standing guard kept from the seed->authentik-KS handoff (backup target).
kustomize.toolkit.fluxcd.io/prune: disabled
spec:
configuration:
# Offsite Hetzner Object Storage (fsn1) — replaces Garage on nas-1.
# No SSE: Hetzner supports only SSE-C, unusable by barman-cloud
# (plugin #646), so at-rest encryption is unavailable; transit is TLS.
destinationPath: s3://renner-uno-k8s-cnpg/ak-pg
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
# Region is required (REGION=fsn1) — without it boto3's HeadBucket
# (barman-cloud-check-wal-archive during bootstrap.recovery) 400s.
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
retentionPolicy: "30d"
kubernetes/apps/home-assistant/ha-pg-cluster.yaml
+1 -1
View File
@@ -116,7 +116,7 @@ spec:
plugin:
name: barman-cloud.cloudnative-pg.io
parameters:
barmanObjectName: ha-pg-store
barmanObjectName: ha-pg-store-source
serverName: ha-pg
postgresql:
parameters:
kubernetes/apps/home-assistant/ha-pg-objectstore.yaml
-35
View File
@@ -1,35 +0,0 @@
---
# Barman object store for ha-pg: base backups + continuous WAL archiving to
# Hetzner Object Storage, same target bucket as the other CNPG clusters.
#
# Its own prefix. Two clusters archiving WAL to one destinationPath interleave
# their streams and corrupt both — see the note in
# kubernetes/apps/woodpecker/objectstore.yaml, which learned that the hard way.
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: ha-pg-store
namespace: home-assistant
spec:
configuration:
destinationPath: s3://renner-uno-k8s-cnpg/ha-pg
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
# Matches the 30d retention the other CNPG clusters use and the
# renner-uno-k8s-cnpg bucket's noncurrent-version expiry in
# opentofu/object_storage.tf, so old S3 versions don't outlive the backups
# that superseded them.
retentionPolicy: "30d"
kubernetes/apps/home-assistant/kustomization.yaml
-1
View File
@@ -57,7 +57,6 @@ resources:
# Live: recorder.db_url in the homeassistant repo points here, and the
# long-term statistics carry their full history from 2024.
- ha-pg-hetzner-creds-externalsecret.yaml
- ha-pg-objectstore.yaml
- ha-pg-cluster.yaml
- ha-pg-scheduled-backup.yaml
# MQTT credential management (REN-802): the account controller manages the
kubernetes/apps/immich/cluster.yaml
+1 -1
View File
@@ -43,7 +43,7 @@ spec:
plugin:
name: barman-cloud.cloudnative-pg.io
parameters:
barmanObjectName: im-pg-store
barmanObjectName: im-pg-store-source
serverName: im-pg
postgresql:
# vchord only. The image also ships pgvecto.rs (vectors.so), but its
kubernetes/apps/immich/kustomization.yaml
-1
View File
@@ -11,7 +11,6 @@ resources:
# a teardown). This KS gains the infrastructure-cnpg-barman dep + CNPG Cluster
# healthCheckExprs (see clusters/homelab/immich.yaml).
- cluster.yaml
- objectstore.yaml
- scheduled-backup.yaml
- hetzner-creds-externalsecret.yaml
- pvcs.yaml
kubernetes/apps/immich/objectstore.yaml
-33
View File
@@ -1,33 +0,0 @@
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: im-pg-store
namespace: immich
annotations:
# Prune-disabled — a standing guard kept from the seed->immich-KS handoff (backup target).
kustomize.toolkit.fluxcd.io/prune: disabled
spec:
configuration:
# Offsite Hetzner Object Storage (fsn1) — replaces Garage on nas-1.
# No SSE: Hetzner supports only SSE-C, unusable by barman-cloud
# (plugin #646), so at-rest encryption is unavailable; transit is TLS.
destinationPath: s3://renner-uno-k8s-cnpg/im-pg
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
# Region is required (REGION=fsn1) — without it boto3's HeadBucket
# (barman-cloud-check-wal-archive during bootstrap.recovery) 400s.
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
retentionPolicy: "30d"
kubernetes/apps/matrix/cluster.yaml
+1 -1
View File
@@ -31,7 +31,7 @@ spec:
plugin:
name: barman-cloud.cloudnative-pg.io
parameters:
barmanObjectName: mx-pg-store
barmanObjectName: mx-pg-store-source
serverName: mx-pg
postgresql:
parameters:
kubernetes/apps/matrix/kustomization.yaml
-1
View File
@@ -9,6 +9,5 @@ resources:
# mx-pg CNPG Postgres + offsite Barman backup. The media store and signing
# key are on synapse-data, backed up by Velero (matrix-daily).
- cluster.yaml
- objectstore.yaml
- scheduled-backup.yaml
- hetzner-creds-externalsecret.yaml
kubernetes/apps/matrix/objectstore.yaml
-30
View File
@@ -1,30 +0,0 @@
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: mx-pg-store
namespace: matrix
spec:
configuration:
# Offsite Hetzner Object Storage (fsn1), same bucket as the other CNPG
# clusters. No SSE: Hetzner supports only SSE-C, unusable by barman-cloud
# (plugin #646), so at-rest encryption is unavailable; transit is TLS.
destinationPath: s3://renner-uno-k8s-cnpg/mx-pg
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
# Region is required (REGION=fsn1) — without it boto3's HeadBucket
# (barman-cloud-check-wal-archive during bootstrap.recovery) 400s.
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
retentionPolicy: "30d"
kubernetes/apps/paperless/cluster.yaml
+1 -1
View File
@@ -39,7 +39,7 @@ spec:
plugin:
name: barman-cloud.cloudnative-pg.io
parameters:
barmanObjectName: pprl-pg-store
barmanObjectName: pprl-pg-store-source
serverName: pprl-pg
postgresql:
parameters:
kubernetes/apps/paperless/kustomization.yaml
-1
View File
@@ -17,7 +17,6 @@ resources:
# transferred without a teardown). This KS gains the infrastructure-cnpg-barman
# dep + CNPG Cluster healthCheckExprs (see clusters/homelab/paperless.yaml).
- cluster.yaml
- objectstore.yaml
- scheduled-backup.yaml
- hetzner-creds-externalsecret.yaml
- pvcs.yaml
kubernetes/apps/paperless/objectstore.yaml
-35
View File
@@ -1,35 +0,0 @@
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: pprl-pg-store
namespace: paperless
annotations:
# Prune-disabled — a standing guard kept from the seed->paperless-KS handoff (backup target).
kustomize.toolkit.fluxcd.io/prune: disabled
spec:
configuration:
# Offsite Hetzner Object Storage (fsn1) — replaces local Garage on
# nas-1 so paperless backups survive nas-1 being powered off. barman
# uses boto3 default (auto) addressing; Hetzner (Ceph RGW) accepts it.
destinationPath: s3://renner-uno-k8s-cnpg/pprl-pg
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
# No `encryption: AES256` (SSE-S3): Hetzner Object Storage supports
# only SSE-C, which the barman-cloud plugin can't use (cnpg
# plugin-barman-cloud #646). At-rest encryption is therefore not
# available here; transit is TLS and the bucket is private.
wal:
compression: gzip
data:
compression: gzip
retentionPolicy: "30d"
kubernetes/apps/trmnl/cluster.yaml
+1 -1
View File
@@ -36,7 +36,7 @@ spec:
plugin:
name: barman-cloud.cloudnative-pg.io
parameters:
barmanObjectName: tr-pg-store
barmanObjectName: tr-pg-store-source
serverName: tr-pg
postgresql:
parameters:
kubernetes/apps/trmnl/kustomization.yaml
-1
View File
@@ -18,6 +18,5 @@ resources:
# tr-pg CNPG Postgres + offsite Barman backup. Note this cluster runs
# PostgreSQL 18, not the repo-standard 16 — see the comment in cluster.yaml.
- cluster.yaml
- objectstore.yaml
- scheduled-backup.yaml
- hetzner-creds-externalsecret.yaml
kubernetes/apps/trmnl/objectstore.yaml
-30
View File
@@ -1,30 +0,0 @@
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: tr-pg-store
namespace: trmnl
spec:
configuration:
# Offsite Hetzner Object Storage (fsn1), same bucket as the other CNPG
# clusters. No SSE: Hetzner supports only SSE-C, unusable by barman-cloud
# (plugin #646), so at-rest encryption is unavailable; transit is TLS.
destinationPath: s3://renner-uno-k8s-cnpg/tr-pg
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
# Region is required (REGION=fsn1) — without it boto3's HeadBucket
# (barman-cloud-check-wal-archive during bootstrap.recovery) 400s.
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
retentionPolicy: "30d"
kubernetes/apps/woodpecker/kustomization.yaml
-1
View File
@@ -34,6 +34,5 @@ resources:
# This KS depends on infrastructure-cnpg-barman for the barman plugin behind
# the ObjectStore and ScheduledBackup.
- hetzner-creds-externalsecret.yaml
- objectstore.yaml
- cluster.yaml
- scheduled-backup.yaml
kubernetes/bootstrap.sh
+162 -2
View File
@@ -394,6 +394,155 @@ read_secret() {
cat "$OPENBAO_SECRETS_DIR/$name"
}
# Hetzner Object Storage, where the offsite backups live. Read with the restore
# key (s3-access, s3-secret).
S3_ENDPOINT="fsn1.your-objectstorage.com"
S3_REGION="fsn1"
# One request, signed by curl with the restore key. The body goes to stdout;
# anything but a 2xx fails, with S3's error document on stderr.
s3_request() {
local method=$1 path=$2 body code
body=$(mktemp)
if ! code=$(curl -sS -o "$body" -w '%{http_code}' -X "$method" \
--aws-sigv4 "aws:amz:$S3_REGION:s3" \
--user "$(read_secret s3-access):$(read_secret s3-secret)" \
"https://$S3_ENDPOINT/$path"); then
rm -f "$body"
return 1
fi
if [[ "$code" != 2?? ]]; then
echo "$method $path: HTTP $code" >&2
cat "$body" >&2
echo >&2
rm -f "$body"
return 1
fi
cat "$body"
rm -f "$body"
}
# The "directories" directly under PREFIX in BUCKET, one per line, without
# PREFIX and the trailing slash.
s3_dirs() {
local bucket=$1 prefix=$2 xml
xml=$(s3_request GET "$bucket?list-type=2&delimiter=%2F&prefix=${prefix//\//%2F}") || return 1
[[ "$xml" == *"<IsTruncated>false</IsTruncated>"* ]] \
|| { echo "the listing of $bucket/$prefix is truncated or not a listing" >&2; return 1; }
# awk, not grep: no match is not an error.
awk -v p="$prefix" '{
while (match($0, /<CommonPrefixes><Prefix>[^<]*<\/Prefix>/)) {
d = substr($0, RSTART + 24, RLENGTH - 33)
$0 = substr($0, RSTART + RLENGTH)
sub(/\/$/, "", d)
print substr(d, length(p) + 1)
}
}' <<<"$xml"
}
# The bucket every CNPG cluster recovers from (cnpg-object-stores/). The
# restore test archives to CNPG_TEST_BUCKET instead: a test incarnation with a
# base backup in the production bucket would be the newest there, and the next
# real recovery would read the test's data.
CNPG_BUCKET="renner-uno-k8s-cnpg"
CNPG_TEST_BUCKET="renner-uno-k8s-cnpg-rtest"
CNPG_STORES_DIR="$REPO_ROOT/kubernetes/cnpg-object-stores"
# cnpg-paths, the ConfigMap cnpg-object-stores takes its paths from. Written
# once per cluster incarnation: an existing one belongs to clusters already
# archiving under its paths and is kept.
#
# For each source ObjectStore in cnpg-object-stores/, cluster <x>:
# CNPG_<X>_SOURCE the candidate prefix whose <x>/base/ holds the newest
# base backup. Candidates: <x>, <x>-g<14 digits> and the
# store's cnpg.renner.uno/source-prefixes.
# CNPG_<X>_ARCHIVE <x>-g<now> in ARCHIVE_BUCKET: new, so empty, so the
# plugin's empty-archive check passes rather than being
# skipped.
write_cnpg_paths() {
local archive_bucket=$1
local gen dirs name extra var p listing newest best best_at
local -a literals=()
echo "Writing the CNPG backup paths (cnpg-paths)..."
if kubectl -n flux-system get configmap cnpg-paths -o name >/dev/null 2>"$OPENBAO_SECRETS_DIR/.err"; then
echo " already present, kept"
return 0
fi
grep -q NotFound "$OPENBAO_SECRETS_DIR/.err" \
|| { cat "$OPENBAO_SECRETS_DIR/.err" >&2; fail "could not read the cnpg-paths ConfigMap"; }
gen=g$(date -u +%Y%m%d%H%M%S)
dirs=$(s3_dirs "$CNPG_BUCKET" "") || fail "could not list $CNPG_BUCKET"
while IFS=$'\t' read -r name extra; do
best=''
best_at=''
for p in $dirs; do
[[ "$p" == "$name" || "$p" =~ ^${name}-g[0-9]{14}$ || " $extra " == *" $p "* ]] || continue
listing=$(s3_dirs "$CNPG_BUCKET" "$p/$name/base/") \
|| fail "could not list $CNPG_BUCKET/$p/$name/base/"
newest=$(sort <<<"$listing" | tail -1)
if [[ -n "$newest" && "$newest" > "$best_at" ]]; then
best=$p
best_at=$newest
fi
done
[[ -n "$best" ]] \
|| fail "no base backup of $name in $CNPG_BUCKET (looked under $name, $name-g<timestamp>${extra:+, $extra})"
var=CNPG_$(tr 'a-z-' 'A-Z_' <<<"$name")
literals+=("--from-literal=${var}_SOURCE=s3://$CNPG_BUCKET/$best"
"--from-literal=${var}_ARCHIVE=s3://$archive_bucket/$name-$gen")
echo " $name: recovers from $CNPG_BUCKET/$best (base backup $best_at), archives to $archive_bucket/$name-$gen"
done < <(yq -r 'select(.kind == "ObjectStore" and .metadata.annotations["cnpg.renner.uno/server-name"] != null)
| [.metadata.annotations["cnpg.renner.uno/server-name"],
(.metadata.annotations["cnpg.renner.uno/source-prefixes"] // "")] | @tsv' \
"$CNPG_STORES_DIR"/*.yaml)
[[ ${#literals[@]} -gt 0 ]] \
|| fail "no source ObjectStore (cnpg.renner.uno/server-name) in $CNPG_STORES_DIR"
kubectl -n flux-system create configmap cnpg-paths "${literals[@]}" >/dev/null \
|| fail "could not create the cnpg-paths ConfigMap"
ok
}
# The restore test's archive bucket. Created when missing; on a rerun it
# already exists, which S3 reports as an error.
create_cnpg_test_bucket() {
local out
out=$(s3_request PUT "$CNPG_TEST_BUCKET" 2>&1) && return 0
[[ "$out" == *BucketAlreadyOwnedByYou* ]] && return 0
echo "$out" >&2
fail "could not create $CNPG_TEST_BUCKET"
}
# Empties and deletes the restore test's archive bucket. Each pass deletes the
# first page of keys and lists again, so no continuation token is needed.
delete_cnpg_test_bucket() {
local xml key
while :; do
if ! xml=$(s3_request GET "$CNPG_TEST_BUCKET?list-type=2" 2>&1); then
[[ "$xml" == *NoSuchBucket* ]] && return 0
echo "$xml" >&2
echo " WARNING: $CNPG_TEST_BUCKET not emptied — check the console" >&2
return 0
fi
while read -r key; do
[[ -n "$key" ]] || continue
s3_request DELETE "$CNPG_TEST_BUCKET/$key" >/dev/null \
|| { echo " WARNING: $CNPG_TEST_BUCKET/$key not deleted — check the console" >&2; return 0; }
done < <(awk '{
while (match($0, /<Key>[^<]*<\/Key>/)) {
print substr($0, RSTART + 5, RLENGTH - 11)
$0 = substr($0, RSTART + RLENGTH)
}
}' <<<"$xml")
[[ "$xml" == *"<IsTruncated>true</IsTruncated>"* ]] || break
done
if s3_request DELETE "$CNPG_TEST_BUCKET" >/dev/null; then
echo " deleted bucket $CNPG_TEST_BUCKET"
else
echo " WARNING: bucket $CNPG_TEST_BUCKET not deleted — check the console" >&2
fi
}
# Provision a throwaway cluster on Hetzner Cloud, then run the Talos phase
# against it unchanged — same shape as the vmtest phase below, and for the same
# reason: a rehearsal that runs different code proves less than one that does
@@ -776,6 +925,8 @@ if [[ "$PHASE" == "hcloudclean" ]]; then
require_secrets hcloud-token
HCLOUD_TOKEN=$(read_secret hcloud-token); export HCLOUD_TOKEN
cleanup_hcloudtest
require_secrets s3-access s3-secret
delete_cnpg_test_bucket
# The restore test's read-only deploy keys on the GitHub mirror.
if command -v gh >/dev/null; then
restore_test_repo
@@ -1834,8 +1985,8 @@ bao_exec() {
# OpenBao — so Velero cannot be the thing that restores OpenBao.
OPENBAO_KOPIA_BUCKET="renner-uno-k8s-velero"
OPENBAO_KOPIA_PREFIX="kopia/openbao/"
OPENBAO_S3_ENDPOINT="fsn1.your-objectstorage.com"
OPENBAO_S3_REGION="fsn1"
OPENBAO_S3_ENDPOINT="$S3_ENDPOINT"
OPENBAO_S3_REGION="$S3_REGION"
# Velero's hardcoded backup repository password (pkg/repository/keys/keys.go).
OPENBAO_KOPIA_PASSWORD="static-passw0rd"
@@ -2208,6 +2359,15 @@ if phase_runs flux; then
fi
unset seal_key existing
# Before Flux: cnpg-object-stores fails without it rather than render empty
# paths, and the paths must exist before any Cluster bootstraps.
if [[ "$FLUX_SOURCE" == "restore-test" ]]; then
create_cnpg_test_bucket
write_cnpg_paths "$CNPG_TEST_BUCKET"
else
write_cnpg_paths "$CNPG_BUCKET"
fi
if [[ "$FLUX_SOURCE" == "restore-test" ]]; then
# The production tree through kubernetes/restore-test/flux-root.yaml,
# whose patches are the test's whole deviation from production. Nothing
kubernetes/clusters/homelab/authentik.yaml
+1 -1
View File
@@ -24,7 +24,7 @@ spec:
- name: infrastructure
- name: infrastructure-crds
- name: infrastructure-ceph-cluster
- name: infrastructure-cnpg-barman
- name: cnpg-object-stores # the ObjectStores its Cluster names
- name: gitea # server, LDAP outpost and gate-sync images are in Gitea's registry
healthCheckExprs:
- apiVersion: postgresql.cnpg.io/v1
kubernetes/clusters/homelab/cnpg-object-stores.yaml
+27
View File
@@ -0,0 +1,27 @@
# The CNPG ObjectStores. Their paths come from the cnpg-paths ConfigMap, which
# bootstrap.sh writes once per cluster incarnation and never replaces:
# CNPG_<X>_SOURCE the bucket prefix holding <x>'s newest base backup
# CNPG_<X>_ARCHIVE a new, empty prefix <x>-g<timestamp>
# Not optional: without it every path renders empty.
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: cnpg-object-stores
namespace: flux-system
spec:
interval: 10m
retryInterval: 1m
timeout: 5m
sourceRef:
kind: GitRepository
name: flux-system
path: ./kubernetes/cnpg-object-stores
prune: true
wait: true
dependsOn:
- name: infrastructure
- name: infrastructure-cnpg-barman
postBuild:
substituteFrom:
- kind: ConfigMap
name: cnpg-paths
kubernetes/clusters/homelab/home-assistant.yaml
+1 -13
View File
@@ -27,19 +27,7 @@ spec:
- name: infrastructure
- name: infrastructure-crds
- name: infrastructure-ceph-cluster
# ha-pg-objectstore.yaml declares an ObjectStore, whose CRD comes from the
# CNPG Barman plugin in infrastructure-cnpg-barman. Without this edge the
# kind is unknown when this path is first applied, and a server-side
# dry-run rejects the ENTIRE set over the one object it cannot recognise:
#
# ObjectStore/home-assistant/ha-pg-store dry-run failed:
# no matches for kind "ObjectStore" in version ...
#
# Every other ObjectStore user — authentik, immich, paperless, trmnl,
# woodpecker — already declares it. This was the only one that did not, and
# it took speech-to-phrase, zone-configurator and matter-server down with
# it, since all three wait on home-assistant.
- name: infrastructure-cnpg-barman
- name: cnpg-object-stores # the ObjectStores its Cluster names
- name: gitea # shelly-manager, mqtt-account-controller images are in Gitea's registry
- name: influxdb # HA's InfluxDB integration writes there
---
kubernetes/clusters/homelab/immich.yaml
+1 -1
View File
@@ -24,7 +24,7 @@ spec:
- name: infrastructure
- name: infrastructure-crds
- name: infrastructure-ceph-cluster
- name: infrastructure-cnpg-barman
- name: cnpg-object-stores # the ObjectStores its Cluster names
healthCheckExprs:
- apiVersion: postgresql.cnpg.io/v1
kind: Cluster
kubernetes/clusters/homelab/matrix.yaml
+1 -1
View File
@@ -21,7 +21,7 @@ spec:
- name: infrastructure
- name: infrastructure-crds
- name: infrastructure-ceph-cluster
- name: infrastructure-cnpg-barman
- name: cnpg-object-stores # the ObjectStores its Cluster names
healthCheckExprs:
- apiVersion: postgresql.cnpg.io/v1
kind: Cluster
kubernetes/clusters/homelab/paperless.yaml
+1 -1
View File
@@ -24,7 +24,7 @@ spec:
- name: infrastructure
- name: infrastructure-crds
- name: infrastructure-ceph-cluster
- name: infrastructure-cnpg-barman
- name: cnpg-object-stores # the ObjectStores its Cluster names
- name: gitea # ftp-scanner (vsftpd) image is in Gitea's registry
healthCheckExprs:
- apiVersion: postgresql.cnpg.io/v1
kubernetes/clusters/homelab/trmnl.yaml
+1 -1
View File
@@ -21,7 +21,7 @@ spec:
- name: infrastructure
- name: infrastructure-crds
- name: infrastructure-ceph-cluster
- name: infrastructure-cnpg-barman
- name: cnpg-object-stores # the ObjectStores its Cluster names
- name: home-assistant # the renderer reads HA's API
healthCheckExprs:
- apiVersion: postgresql.cnpg.io/v1
kubernetes/clusters/homelab/woodpecker.yaml
+1 -1
View File
@@ -24,7 +24,7 @@ spec:
- name: infrastructure
- name: infrastructure-crds
- name: infrastructure-ceph-cluster
- name: infrastructure-cnpg-barman
- name: cnpg-object-stores # the ObjectStores its Cluster names
- name: gitea # its forge and login provider
healthCheckExprs:
- apiVersion: postgresql.cnpg.io/v1
kubernetes/cnpg-object-stores/ak-pg.yaml
+54
View File
@@ -0,0 +1,54 @@
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: ak-pg-store
namespace: authentik
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled
spec:
configuration:
destinationPath: ${CNPG_AK_PG_ARCHIVE}
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
retentionPolicy: "30d"
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: ak-pg-store-source
namespace: authentik
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled
cnpg.renner.uno/server-name: ak-pg
spec:
configuration:
destinationPath: ${CNPG_AK_PG_SOURCE}
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
kubernetes/cnpg-object-stores/ha-pg.yaml
+54
View File
@@ -0,0 +1,54 @@
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: ha-pg-store
namespace: home-assistant
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled
spec:
configuration:
destinationPath: ${CNPG_HA_PG_ARCHIVE}
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
retentionPolicy: "30d"
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: ha-pg-store-source
namespace: home-assistant
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled
cnpg.renner.uno/server-name: ha-pg
spec:
configuration:
destinationPath: ${CNPG_HA_PG_SOURCE}
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
kubernetes/cnpg-object-stores/im-pg.yaml
+54
View File
@@ -0,0 +1,54 @@
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: im-pg-store
namespace: immich
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled
spec:
configuration:
destinationPath: ${CNPG_IM_PG_ARCHIVE}
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
retentionPolicy: "30d"
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: im-pg-store-source
namespace: immich
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled
cnpg.renner.uno/server-name: im-pg
spec:
configuration:
destinationPath: ${CNPG_IM_PG_SOURCE}
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
kubernetes/cnpg-object-stores/kustomization.yaml
+24
View File
@@ -0,0 +1,24 @@
# Every CNPG cluster's two ObjectStores: <x>-store, where it archives WAL and
# takes base backups, and <x>-store-source, which its bootstrap.recovery reads.
#
# Their destinationPaths are Flux variables from the cnpg-paths ConfigMap
# (clusters/homelab/cnpg-object-stores.yaml). Nothing else may live in this
# directory: postBuild substitution blanks every ${...} it renders.
#
# The source store carries cnpg.renner.uno/server-name, the cluster whose
# backups bootstrap.sh looks for, and optionally
# cnpg.renner.uno/source-prefixes, bucket prefixes outside the <name> and
# <name>-g<timestamp> pattern that may still hold them.
#
# No SSE: Hetzner supports only SSE-C, unusable by barman-cloud (plugin #646),
# so at-rest encryption is unavailable; transit is TLS.
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ak-pg.yaml
- ha-pg.yaml
- im-pg.yaml
- mx-pg.yaml
- pprl-pg.yaml
- tr-pg.yaml
- wp-pg.yaml
kubernetes/cnpg-object-stores/mx-pg.yaml
+54
View File
@@ -0,0 +1,54 @@
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: mx-pg-store
namespace: matrix
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled
spec:
configuration:
destinationPath: ${CNPG_MX_PG_ARCHIVE}
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
retentionPolicy: "30d"
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: mx-pg-store-source
namespace: matrix
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled
cnpg.renner.uno/server-name: mx-pg
spec:
configuration:
destinationPath: ${CNPG_MX_PG_SOURCE}
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
kubernetes/cnpg-object-stores/pprl-pg.yaml
+54
View File
@@ -0,0 +1,54 @@
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: pprl-pg-store
namespace: paperless
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled
spec:
configuration:
destinationPath: ${CNPG_PPRL_PG_ARCHIVE}
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
retentionPolicy: "30d"
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: pprl-pg-store-source
namespace: paperless
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled
cnpg.renner.uno/server-name: pprl-pg
spec:
configuration:
destinationPath: ${CNPG_PPRL_PG_SOURCE}
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
kubernetes/cnpg-object-stores/tr-pg.yaml
+54
View File
@@ -0,0 +1,54 @@
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: tr-pg-store
namespace: trmnl
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled
spec:
configuration:
destinationPath: ${CNPG_TR_PG_ARCHIVE}
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
retentionPolicy: "30d"
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: tr-pg-store-source
namespace: trmnl
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled
cnpg.renner.uno/server-name: tr-pg
spec:
configuration:
destinationPath: ${CNPG_TR_PG_SOURCE}
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
kubernetes/apps/woodpecker/objectstore.yaml → kubernetes/cnpg-object-stores/wp-pg.yaml
+32 -35
View File
@@ -1,46 +1,14 @@
---
# RECOVERY SOURCE — read-only. The prefix wp-pg archives to (wp-pg-store below).
#
# serverName on the consuming externalCluster must be `wp-pg`, the name of the
# cluster that produced these backups.
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: wp-pg-store-source
namespace: woodpecker
spec:
configuration:
destinationPath: s3://renner-uno-k8s-cnpg/wp-pg-woodpecker
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
# No retentionPolicy: wp-pg-store owns the lifecycle of this prefix.
---
# ARCHIVING TARGET.
#
# Not s3://renner-uno-k8s-cnpg/wp-pg: the retired `apps/wp-pg` archived there,
# and two clusters archiving WAL to one destinationPath interleave their streams
# and corrupt both.
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: wp-pg-store
namespace: woodpecker
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled
spec:
configuration:
destinationPath: s3://renner-uno-k8s-cnpg/wp-pg-woodpecker
destinationPath: ${CNPG_WP_PG_ARCHIVE}
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
@@ -57,3 +25,32 @@ spec:
data:
compression: gzip
retentionPolicy: "30d"
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: wp-pg-store-source
namespace: woodpecker
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled
cnpg.renner.uno/server-name: wp-pg
# wp-pg's archive since its move out of `apps`.
cnpg.renner.uno/source-prefixes: wp-pg-woodpecker
spec:
configuration:
destinationPath: ${CNPG_WP_PG_SOURCE}
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip