kubernetes/seed/RECOVERY.md

@@ -172,8 +172,9 @@ PVC restores but **explicitly does not touch CNPG** (barman-cloud is a separate
 code path). Run this drill periodically to prove the CNPG backups are actually
 restorable, **without touching the live clusters**. It restores into a throwaway
 cluster in the *same* namespace (so it can reference the existing namespaced
-`ObjectStore` and creds) and never enables WAL archiving, so it only ever
-*reads* the backup path.
+`ObjectStore` and creds) and declares the barman plugin with
+`isWALArchiver: false` — loaded so it can *read* WAL to replay during the
+restore, but never archiving, so it only ever *reads* the backup path.
 
 ```sh
 NS=authentik ; SRC=ak-pg ; STORE=ak-pg-store ; DB=authentik
@@ -189,7 +190,18 @@ spec:
   storage:
     storageClass: ceph-block
     size: 10Gi
-  # NO spec.plugins / isWALArchiver here — read-only restore, never archives.
+  # The barman plugin MUST be declared so the instance manager can load it to
+  # replay WAL during the restore. Omitting it (as an earlier version of this
+  # runbook said to do) fails at instance start with "Unknown plugin:
+  # barman-cloud.cloudnative-pg.io" and the cluster never becomes healthy.
+  # isWALArchiver: false keeps it read-only — loaded for restore, but it never
+  # archives, so it only ever READS the backup path (verified 2026-07-25 on im-pg).
+  plugins:
+    - name: barman-cloud.cloudnative-pg.io
+      enabled: true
+      isWALArchiver: false
+      parameters:
+        barmanObjectName: ${STORE}
   bootstrap:
     recovery:
       source: ${STORE}
@@ -206,12 +218,22 @@ EOF
 
 kubectl -n ${NS} wait --for=jsonpath='{.status.phase}'='Cluster in healthy state' \
   cluster/${SRC}-rtest --timeout=20m
-kubectl -n ${NS} exec ${SRC}-rtest-1 -c postgres -- \
-  psql -U postgres -d ${DB} -c "select count(*) from pg_stat_user_tables;"
+kubectl -n ${NS} exec ${SRC}-rtest-1 -c postgres -- psql -U postgres -d ${DB} \
+  -c "select count(*) as user_tables from pg_stat_user_tables;" \
+  -c "select pg_size_pretty(pg_database_size('${DB}')) as db_size;"
+# NOTE: pg_stat_user_tables row estimates (n_live_tup) read 0 immediately after
+# a restore — the stats collector hasn't run yet, so that view is NOT a data
+# check. Confirm real data with pg_database_size above, or count(*) on a known
+# table (e.g. im-pg: `select count(*) from asset;` returned 33316 on 2026-07-25).
 
-# Tear down — deletes the throwaway cluster and its PVCs (ceph-block is
-# Retain, so also reclaim the released PV afterwards).
+# Tear down. ceph-block is reclaimPolicy: Retain, so deleting the cluster leaves
+# a Released PV whose backing RBD image would otherwise leak (issue #262).
+# Capture the PV name FIRST, then delete the cluster, then flip the released PV
+# to Delete so the CSI driver reclaims the RBD image:
+PV=$(kubectl -n ${NS} get pvc ${SRC}-rtest-1 -o jsonpath='{.spec.volumeName}')
 kubectl -n ${NS} delete cluster ${SRC}-rtest
+kubectl patch pv "$PV" --type=merge -p '{"spec":{"persistentVolumeReclaimPolicy":"Delete"}}'
+kubectl delete pv "$PV" --ignore-not-found
 ```
 
 For `im-pg` use `imageName: ghcr.io/corentingiraud/cnpg-pgvector-vectorchord:16-migration`