docs(recovery): fix CNPG restore-test drill for the plugin-based barman #404

2026-07-25T07:12:08Z

claude-bot commented

2026-07-25 07:12:08 +00:00

Found by actually running the RECOVERY.md restore-test against im-pg (before the immich seed→apps fold). The drill as written is broken for the plugin-based barman this homelab now uses — a real cold-rebuild DR following it would have failed.

The bug

The drill said to omit spec.plugins ("NO spec.plugins / isWALArchiver here"). But with barmancloud.cnpg.io ObjectStore + the barman-cloud.cloudnative-pg.io plugin, the instance manager needs the plugin declared to replay WAL during restore. Without it the pod dies at startup:

error: while getting plugin connection: Unknown plugin: barman-cloud.cloudnative-pg.io

and the cluster never leaves cannot proceed to reconciliation due to an error while interacting with plugins.

Verified fix (im-pg, 2026-07-25)

Declaring the plugin with isWALArchiver: false (loaded for restore, never archives → read-only against the backup path) let it restore cleanly: 33,316 rows in asset, 584 MB DB. Backup confirmed restorable.

Changes

Restore-test manifest: add the spec.plugins block (isWALArchiver: false); fix the intro prose.
Verification: note that pg_stat_user_tables n_live_tup is 0 right after restore (stats not gathered) — use pg_database_size / a real table count instead.
Teardown: replace the vague "reclaim the released PV afterwards" with the explicit patch-to-Delete recipe (the vague version leaks the Retain PV's RBD image — issue #262).

The in-place cold-rebuild section (keeps the live cluster's spec.plugins isWALArchiver: true) was already correct — unchanged.

Found by actually **running** the RECOVERY.md restore-test against `im-pg` (before the immich seed→apps fold). The drill as written is broken for the plugin-based barman this homelab now uses — a real cold-rebuild DR following it would have failed. ### The bug The drill said to **omit** `spec.plugins` ("NO spec.plugins / isWALArchiver here"). But with `barmancloud.cnpg.io` ObjectStore + the `barman-cloud.cloudnative-pg.io` plugin, the instance manager needs the plugin **declared** to replay WAL during restore. Without it the pod dies at startup: ``` error: while getting plugin connection: Unknown plugin: barman-cloud.cloudnative-pg.io ``` and the cluster never leaves `cannot proceed to reconciliation due to an error while interacting with plugins`. ### Verified fix (im-pg, 2026-07-25) Declaring the plugin with `isWALArchiver: false` (loaded for restore, never archives → read-only against the backup path) let it restore cleanly: **33,316 rows in `asset`, 584 MB DB**. Backup confirmed restorable. ### Changes - Restore-test manifest: add the `spec.plugins` block (`isWALArchiver: false`); fix the intro prose. - Verification: note that `pg_stat_user_tables` `n_live_tup` is 0 right after restore (stats not gathered) — use `pg_database_size` / a real table count instead. - Teardown: replace the vague "reclaim the released PV afterwards" with the explicit patch-to-Delete recipe (the vague version leaks the Retain PV's RBD image — issue #262). The in-place cold-rebuild section (keeps the live cluster's `spec.plugins isWALArchiver: true`) was already correct — unchanged.

claude-bot added 1 commit 2026-07-25 07:12:08 +00:00

docs(recovery): fix CNPG restore-test drill for the plugin-based barman

ci/woodpecker/pr/pr-review Pipeline was successful

Details

6274dd66fb

The restore-test drill in RECOVERY.md told you to omit spec.plugins from the
throwaway cluster ('NO spec.plugins / isWALArchiver here'). With the plugin-based
barman this homelab now uses (barmancloud.cnpg.io ObjectStore + the
barman-cloud.cloudnative-pg.io plugin), that is invalid: the instance manager
can't load the plugin it needs to replay WAL during restore and dies at startup
with 'Unknown plugin: barman-cloud.cloudnative-pg.io' — the cluster never becomes
healthy. A real cold-rebuild DR following this drill would have failed the same
way. Found + fixed by running the drill against im-pg on 2026-07-25 (restore
succeeded once the plugin was declared; 33316 rows in 'asset', 584 MB DB).

- Declare the barman plugin in spec.plugins with isWALArchiver: false — loaded so
  it can READ WAL for the restore, but never archives (stays read-only against
  the backup path). Update the intro prose to match.
- Verification: pg_stat_user_tables n_live_tup reads 0 right after a restore
  (stats not gathered), so that view is not a data check — add pg_database_size
  and note counting a real table instead.
- Teardown: the old 'reclaim the released PV afterwards' was too vague and leaks
  the Retain PV's RBD image (issue #262). Give the explicit recipe: capture the
  PV name, delete the cluster, then patch the released PV to Delete so the CSI
  reclaims the RBD image.

The in-place cold-rebuild section (which keeps the live cluster's existing
spec.plugins isWALArchiver: true) was already correct and is unchanged.

Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

review-bot reviewed

2026-07-25 07:13:34 +00:00

review-bot left a comment

🤖 Automated review — model claude-haiku-4-5-20251001 (complexity: trivial), commit 6274dd66fba0c197ec39c053f0286656ab15f7c1.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

This PR fixes critical errors in the CNPG restore-test procedure that would have caused actual cold-rebuild DR to fail. The original runbook incorrectly omitted the barman plugin declaration; restores would hang with "Unknown plugin" errors. The fix declares the plugin with isWALArchiver: false (loaded for restore, read-only against backups), improves verification to check real data rather than unreliable stats, and replaces vague teardown instructions with explicit PV cleanup that prevents resource leaks. All changes verified by actual testing on 2026-07-25.

🔴 Blocking

None.

🟡 Suggestions

None. The detailed comment block (lines 193–198) is lengthy, but appropriate and necessary for a procedural runbook where operators need to understand why the plugin must be present to avoid repeating the original mistake.

🟢 Nits

None.

Tests

This is documentation only, but the author verified the procedure by running it against im-pg with real data (33,316 rows in asset table, 584 MB DB). The fix makes the restore-test procedure actually restorable — it now matches the requirements of the barman-cloud plugin already in use in this homelab.

Recommendation

Looks reasonable — no blocking issues; comments above are optional.

🤖 **Automated review** — model `claude-haiku-4-5-20251001` (complexity: trivial), commit `6274dd66fba0c197ec39c053f0286656ab15f7c1`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary This PR fixes critical errors in the CNPG restore-test procedure that would have caused actual cold-rebuild DR to fail. The original runbook incorrectly omitted the barman plugin declaration; restores would hang with "Unknown plugin" errors. The fix declares the plugin with `isWALArchiver: false` (loaded for restore, read-only against backups), improves verification to check real data rather than unreliable stats, and replaces vague teardown instructions with explicit PV cleanup that prevents resource leaks. All changes verified by actual testing on 2026-07-25. ## 🔴 Blocking None. ## 🟡 Suggestions None. The detailed comment block (lines 193–198) is lengthy, but appropriate and necessary for a procedural runbook where operators need to understand *why* the plugin must be present to avoid repeating the original mistake. ## 🟢 Nits None. ## Tests This is documentation only, but the author verified the procedure by running it against `im-pg` with real data (33,316 rows in `asset` table, 584 MB DB). The fix makes the restore-test procedure actually restorable — it now matches the requirements of the barman-cloud plugin already in use in this homelab. ## Recommendation Looks reasonable — no blocking issues; comments above are optional.

k.renner merged commit d5c7cea403 into master

2026-07-25 08:42:19 +00:00

k.renner referenced this issue from a commit

2026-07-25 08:42:20 +00:00

Merge pull request 'docs(recovery): fix CNPG restore-test drill for the plugin-based barman' (#404) from fix-recovery-md-cnpg-restore-test into master

claude-bot referenced this pull request

2026-07-25 08:46:21 +00:00

refactor(immich): fold seed-immich into the immich KS (stage 2) #405

claude-bot referenced this issue from a commit

2026-07-25 09:12:49 +00:00

chore(authentik): pre-stage prune:disabled on the ak-pg CNPG objects (stage 1)

claude-bot referenced this pull request