diff --git a/kubernetes/seed/RECOVERY.md b/kubernetes/seed/RECOVERY.md index 45ac6b2a..67729aaa 100644 --- a/kubernetes/seed/RECOVERY.md +++ b/kubernetes/seed/RECOVERY.md @@ -172,8 +172,9 @@ PVC restores but **explicitly does not touch CNPG** (barman-cloud is a separate code path). Run this drill periodically to prove the CNPG backups are actually restorable, **without touching the live clusters**. It restores into a throwaway cluster in the *same* namespace (so it can reference the existing namespaced -`ObjectStore` and creds) and never enables WAL archiving, so it only ever -*reads* the backup path. +`ObjectStore` and creds) and declares the barman plugin with +`isWALArchiver: false` — loaded so it can *read* WAL to replay during the +restore, but never archiving, so it only ever *reads* the backup path. ```sh NS=authentik ; SRC=ak-pg ; STORE=ak-pg-store ; DB=authentik @@ -189,7 +190,18 @@ spec: storage: storageClass: ceph-block size: 10Gi - # NO spec.plugins / isWALArchiver here — read-only restore, never archives. + # The barman plugin MUST be declared so the instance manager can load it to + # replay WAL during the restore. Omitting it (as an earlier version of this + # runbook said to do) fails at instance start with "Unknown plugin: + # barman-cloud.cloudnative-pg.io" and the cluster never becomes healthy. + # isWALArchiver: false keeps it read-only — loaded for restore, but it never + # archives, so it only ever READS the backup path (verified 2026-07-25 on im-pg). + plugins: + - name: barman-cloud.cloudnative-pg.io + enabled: true + isWALArchiver: false + parameters: + barmanObjectName: ${STORE} bootstrap: recovery: source: ${STORE} @@ -206,12 +218,22 @@ EOF kubectl -n ${NS} wait --for=jsonpath='{.status.phase}'='Cluster in healthy state' \ cluster/${SRC}-rtest --timeout=20m -kubectl -n ${NS} exec ${SRC}-rtest-1 -c postgres -- \ - psql -U postgres -d ${DB} -c "select count(*) from pg_stat_user_tables;" +kubectl -n ${NS} exec ${SRC}-rtest-1 -c postgres -- psql -U postgres -d ${DB} \ + -c "select count(*) as user_tables from pg_stat_user_tables;" \ + -c "select pg_size_pretty(pg_database_size('${DB}')) as db_size;" +# NOTE: pg_stat_user_tables row estimates (n_live_tup) read 0 immediately after +# a restore — the stats collector hasn't run yet, so that view is NOT a data +# check. Confirm real data with pg_database_size above, or count(*) on a known +# table (e.g. im-pg: `select count(*) from asset;` returned 33316 on 2026-07-25). -# Tear down — deletes the throwaway cluster and its PVCs (ceph-block is -# Retain, so also reclaim the released PV afterwards). +# Tear down. ceph-block is reclaimPolicy: Retain, so deleting the cluster leaves +# a Released PV whose backing RBD image would otherwise leak (issue #262). +# Capture the PV name FIRST, then delete the cluster, then flip the released PV +# to Delete so the CSI driver reclaims the RBD image: +PV=$(kubectl -n ${NS} get pvc ${SRC}-rtest-1 -o jsonpath='{.spec.volumeName}') kubectl -n ${NS} delete cluster ${SRC}-rtest +kubectl patch pv "$PV" --type=merge -p '{"spec":{"persistentVolumeReclaimPolicy":"Delete"}}' +kubectl delete pv "$PV" --ignore-not-found ``` For `im-pg` use `imageName: ghcr.io/corentingiraud/cnpg-pgvector-vectorchord:16-migration`