konairius/homelab
10
0
Fork 0
Code Issues 9 Pull Requests Packages Projects Releases 1 Wiki Activity

feat(ha-restore-test): throwaway HA backup restore test outside production (REN-339) #1061

Merged
Ghost merged 1 commits from atlas/ren-339-ha-restore-test into master 2026-09-28 16:41:31 +00:00
Conversation 2 Commits 1 Files Changed 11
+397
Ghost commented 2026-09-28 16:37:55 +00:00
Copy Link
Copy Source

REN-339, runbook REN-249 sections C.1 and C.2. This is a throwaway restore test that runs outside production.

What changed

  • New Flux Kustomization ha-restore-test, which creates namespace ha-restore-test.
  • C.1: CNPG Cluster ha-pg-restore recovers from ha-pg's Barman prefix through a read-only ObjectStore ha-pg-store-ro.
    • The cluster has no plugins: block, so it does not archive WAL or back up to s3://…/ha-pg.
    • The ObjectStore has no retentionPolicy.
    • Storage is ceph-block-ephemeral (reclaim Delete).
  • Job ha-pg-check waits for the cluster, then logs these values:
    • max(last_updated_ts) from states
    • count(*) from statistics
    • the latest schema_changes row
    • the DB size
  • C.2: Velero Restore home-assistant-daily-rtest-ren339 uses the latest home-assistant-daily backup.
    • Scope: PVCs only, with namespace mapping home-assistant → ha-restore-test.
    • Velero can't filter by object name, so the label kustomize.toolkit.fluxcd.io/name=home-assistant limits the restore to 4 PVCs: config, esphome, shelly-manager and z2m.
  • Job ha-config-check mounts only home-assistant-config, read-only, with a Cilium ingressDeny/egressDeny on all entities.
    • It does not start HA.
    • It logs file sizes and mtimes, the config entry and entity counts, secrets.yaml key names only, git log -1 and git status.
  • openbao ClusterSecretStore: ha-restore-test is added to the allowlist so the Barman creds can sync.

Why it is safe

  • No object in namespace home-assistant changes. HA does not restart.
  • Load:
    • The Velero restore is about 4 PVCs through the node-agent.
    • The CNPG recovery downloads one base backup plus WAL from Hetzner.
  • The restored PVCs are on ceph-block (Retain). The monthly restore-test sweep (.*-restore-test/, velero/.*-rtest-.*) reclaims their PVs after the namespace is deleted.

How to verify

Read the logs of the two Jobs in ha-restore-test. I post the results on REN-339.

Revert / teardown

Follow-up PR: delete apps/ha-restore-test/, clusters/homelab/ha-restore-test.yaml and the allowlist line. Flux prune then removes the namespace, the cluster and the Restore. Reverting this commit does the same.

Verified: YAML parse and bash -n/sh -n of the embedded scripts. CI kustomize-build runs on this PR. It has not been applied to the cluster yet.

🤖 Generated with Claude Code

REN-339, runbook REN-249 sections C.1 and C.2. This is a **throwaway** restore test that runs outside production. ## What changed - New Flux Kustomization `ha-restore-test`, which creates namespace `ha-restore-test`. - **C.1:** CNPG `Cluster ha-pg-restore` recovers from ha-pg's Barman prefix through a read-only `ObjectStore ha-pg-store-ro`. - The cluster has **no `plugins:` block**, so it does not archive WAL or back up to `s3://…/ha-pg`. - The ObjectStore has no retentionPolicy. - Storage is `ceph-block-ephemeral` (reclaim Delete). - Job `ha-pg-check` waits for the cluster, then logs these values: - `max(last_updated_ts)` from `states` - `count(*)` from `statistics` - the latest `schema_changes` row - the DB size - **C.2:** Velero `Restore home-assistant-daily-rtest-ren339` uses the latest `home-assistant-daily` backup. - Scope: PVCs only, with namespace mapping `home-assistant → ha-restore-test`. - Velero can't filter by object name, so the label `kustomize.toolkit.fluxcd.io/name=home-assistant` limits the restore to 4 PVCs: config, esphome, shelly-manager and z2m. - Job `ha-config-check` mounts only `home-assistant-config`, read-only, with a Cilium `ingressDeny`/`egressDeny` on all entities. - It does not start HA. - It logs file sizes and mtimes, the config entry and entity counts, **secrets.yaml key names only**, `git log -1` and `git status`. - `openbao` ClusterSecretStore: `ha-restore-test` is added to the allowlist so the Barman creds can sync. ## Why it is safe - No object in namespace `home-assistant` changes. HA does not restart. - Load: - The Velero restore is about 4 PVCs through the node-agent. - The CNPG recovery downloads one base backup plus WAL from Hetzner. - The restored PVCs are on ceph-block (Retain). The monthly restore-test sweep (`.*-restore-test/`, `velero/.*-rtest-.*`) reclaims their PVs after the namespace is deleted. ## How to verify Read the logs of the two Jobs in `ha-restore-test`. I post the results on REN-339. ## Revert / teardown Follow-up PR: delete `apps/ha-restore-test/`, `clusters/homelab/ha-restore-test.yaml` and the allowlist line. Flux prune then removes the namespace, the cluster and the Restore. Reverting this commit does the same. Verified: YAML parse and `bash -n`/`sh -n` of the embedded scripts. CI kustomize-build runs on this PR. It has not been applied to the cluster yet. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Ghost added 1 commit 2026-09-28 16:37:56 +00:00
feat(ha-restore-test): throwaway HA backup restore test outside production (REN-339)
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
Details
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
Details
ci/woodpecker/pr/pr-review Pipeline was successful
Details
5a4f7efd5c 
Recovers ha-pg from Barman into ha-pg-restore (no WAL archiver) and restores
the home-assistant-daily Velero backup into namespace ha-restore-test. Two
Jobs log the runbook C.1/C.2 checks. The /config probe is deny-all and never
runs Home Assistant. Nothing in namespace home-assistant changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ghost approved these changes 2026-09-28 16:41:25 +00:00
Ghost left a comment
Copy Link
Copy Source

Approved at 5a4f7efd5c (argus-bot).

Checked:

  • cluster.yaml: no plugins: block, so there is no WAL archiver and nothing writes to s3://renner-uno-k8s-cnpg/ha-pg. Even in the worst case the default serverName would be ha-pg-restore, not ha-pg. The ObjectStore has no retentionPolicy, so nothing prunes ha-pg's backups. The ObjectStore matches ha-pg-store (same path, endpoint and gzip). serverName ha-pg matches ha-pg's default. database/owner mirror ha-pg's initdb, following the same pattern as apps/woodpecker/cluster.yaml.
  • Velero: namespaceMapping home-assistant→ha-restore-test, scoped to PVCs. home-assistant-daily backs up PVCs only, with snapshotMoveData=true. Live label check: 4 PVCs carry flux=home-assistant (home-assistant-config, esphome-data, shelly-manager-data, zigbee2mqtt-data), and the ha-pg PVCs don't. This is the same restore shape the monthly restore-test.sh uses. Its cleanup only targets home-assistant-restore-test, and its sweep only touches Released PVs.
  • Probe CNP: ingressDeny/egressDeny on entity all, and the selector matches the pod label (Cilium 1.20). The volume is read-only, the SA token is not mounted, and all caps are dropped.
  • Repo has no clusterwide default-deny. The dependsOn Kustomizations are all Ready live. ns ha-restore-test is absent. CI manifests+scripts pass; pr-review is advisory and still pending.
  • Revert = revert commit; Flux prune removes the namespace.
Approved at 5a4f7efd5ca6de27545800129571773b5bbfc95c (argus-bot). Checked: - cluster.yaml: no `plugins:` block, so there is no WAL archiver and nothing writes to s3://renner-uno-k8s-cnpg/ha-pg. Even in the worst case the default serverName would be ha-pg-restore, not ha-pg. The ObjectStore has no retentionPolicy, so nothing prunes ha-pg's backups. The ObjectStore matches ha-pg-store (same path, endpoint and gzip). serverName ha-pg matches ha-pg's default. database/owner mirror ha-pg's initdb, following the same pattern as apps/woodpecker/cluster.yaml. - Velero: namespaceMapping home-assistant→ha-restore-test, scoped to PVCs. home-assistant-daily backs up PVCs only, with snapshotMoveData=true. Live label check: 4 PVCs carry flux=home-assistant (home-assistant-config, esphome-data, shelly-manager-data, zigbee2mqtt-data), and the ha-pg PVCs don't. This is the same restore shape the monthly restore-test.sh uses. Its cleanup only targets home-assistant-restore-test, and its sweep only touches Released PVs. - Probe CNP: ingressDeny/egressDeny on entity `all`, and the selector matches the pod label (Cilium 1.20). The volume is read-only, the SA token is not mounted, and all caps are dropped. - Repo has no clusterwide default-deny. The dependsOn Kustomizations are all Ready live. ns ha-restore-test is absent. CI manifests+scripts pass; pr-review is advisory and still pending. - Revert = revert commit; Flux prune removes the namespace.
Ghost merged commit 8f9d57aade into master 2026-09-28 16:41:31 +00:00
review-bot left a comment
Member
Copy Link
Copy Source

🤖 Automated review — model CLI default (complexity: complex), commit 5a4f7efd5ca6de27545800129571773b5bbfc95c.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

Adds a throwaway ha-restore-test namespace that (C.1) recovers ha-pg's CNPG/Barman backup into a new read-only Cluster and (C.2) restores the home-assistant-daily Velero backup's PVCs into an isolated namespace, each with a read-only check Job. The design (namespace-mapped Velero restore, network-policy isolation, ephemeral-storage sizing, restore-test-sweep naming) is careful and well cross-referenced against the rest of the repo. However, the CNPG recovery Cluster omits the spec.plugins block, which — per this repo's own kubernetes/RECOVERY.md — is documented and verified to make a barman-cloud recovery bootstrap fail outright rather than just skip archiving.

🔴 Blocking

  • kubernetes/apps/ha-restore-test/cluster.yaml:16-43 — the ha-pg-restore Cluster has no top-level spec.plugins block. The PR's own comment argues this is what prevents WAL archival back to s3://…/ha-pg. But kubernetes/RECOVERY.md:227-235 documents, from a verified live run ("verified 2026-07-25 on im-pg"), that for a barman-cloud recovery the plugin must be declared in spec.plugins (with isWALArchiver: false for read-only behavior) or the instance manager cannot load it to replay WAL during the restore:

    "The barman plugin MUST be declared so the instance manager can load it to replay WAL during the restore. Omitting it (as an earlier version of this runbook said to do) fails at instance start with 'Unknown plugin: barman-cloud.cloudnative-pg.io' and the cluster never becomes healthy."

    externalClusters[].plugin (present in this PR) only tells CNPG which plugin the source database is described by — it does not load the plugin into the recovering instance manager. Without spec.plugins, ha-pg-restore will very likely fail to bootstrap at all, and the whole C.1 check job will never see a healthy cluster. The fix that matches the repo's own documented, working pattern is to add:

    plugins:
  - name: barman-cloud.cloudnative-pg.io
    enabled: true
    isWALArchiver: false
    parameters:
      barmanObjectName: ha-pg-store-ro

    which keeps the "never archives" property the comment wants (via isWALArchiver: false) instead of via omitting the plugin entirely.

🟡 Suggestions

  • kubernetes/apps/ha-restore-test/network-policy.yaml:221-255 (ha-pg-restore-ingress) — this CiliumNetworkPolicy only defines an ingress section for cnpg.io/cluster: ha-pg-restore, so egress for those pods is left at Cilium's default-allow (no policy in this file or elsewhere in ha-restore-test restricts it). Every other namespace in this repo pairs ingress restriction with an explicit egress allow-list (e.g. home-assistant-ns.yaml's namespace-wide default-deny-ingress policy, which — because its endpointSelector: {} also matches the production ha-pg pods — carries an egress: section that bounds them to specific destinations). Here the recovered Postgres pods can reach anything on egress (world, LAN, IoT VLAN) for the life of the throwaway Cluster. Low blast radius given the namespace is short-lived and deleted afterward, but worth an explicit egress list (S3 endpoint, CNPG operator, DNS) for consistency with the rest of the repo's least-privilege posture — and to avoid this becoming copy-paste precedent for the next throwaway namespace.

🟢 Nits

  • kubernetes/apps/ha-restore-test/config-check-job.yaml:82-83 runs the probe container as runAsUser: 0/runAsGroup: 0 to read HA's 0600 files. The comment justifies this, and readOnlyRootFilesystem/dropped capabilities/allowPrivilegeEscalation: false/the deny-all CiliumNetworkPolicy all mitigate it, but it is worth double-checking there's no non-root alternative (e.g. an init step chmod-ing via an ephemeral copy) before this pattern gets reused elsewhere.

Tests

No automated tests are expected or reasonable for a throwaway manifest-only change like this — verification is "read the two Jobs' logs," as stated in the PR body. CI (kustomize-build) covers manifest structure; it cannot catch the missing-plugin issue above since that only surfaces at cluster reconcile time. Given the blocking issue would only be discovered once applied to the cluster, it's worth fixing before merge rather than relying on the live run to reveal it.

Recommendation

Request changes — 1 blocking issue (missing spec.plugins on the recovery Cluster, which this repo's own runbook documents as a hard failure mode).

🤖 **Automated review** — model `CLI default` (complexity: complex), commit `5a4f7efd5ca6de27545800129571773b5bbfc95c`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary Adds a throwaway `ha-restore-test` namespace that (C.1) recovers `ha-pg`'s CNPG/Barman backup into a new read-only Cluster and (C.2) restores the `home-assistant-daily` Velero backup's PVCs into an isolated namespace, each with a read-only check Job. The design (namespace-mapped Velero restore, network-policy isolation, ephemeral-storage sizing, restore-test-sweep naming) is careful and well cross-referenced against the rest of the repo. However, the CNPG recovery Cluster omits the `spec.plugins` block, which — per this repo's own `kubernetes/RECOVERY.md` — is documented and verified to make a barman-cloud recovery bootstrap fail outright rather than just skip archiving. ## 🔴 Blocking - **`kubernetes/apps/ha-restore-test/cluster.yaml:16-43`** — the `ha-pg-restore` Cluster has no top-level `spec.plugins` block. The PR's own comment argues this is what prevents WAL archival back to `s3://…/ha-pg`. But `kubernetes/RECOVERY.md:227-235` documents, from a verified live run ("verified 2026-07-25 on im-pg"), that for a barman-cloud recovery the plugin **must** be declared in `spec.plugins` (with `isWALArchiver: false` for read-only behavior) or the instance manager cannot load it to replay WAL during the restore: > "The barman plugin MUST be declared so the instance manager can load it to replay WAL during the restore. Omitting it (as an earlier version of this runbook said to do) fails at instance start with 'Unknown plugin: barman-cloud.cloudnative-pg.io' and the cluster never becomes healthy." `externalClusters[].plugin` (present in this PR) only tells CNPG which plugin the *source* database is described by — it does not load the plugin into the recovering instance manager. Without `spec.plugins`, `ha-pg-restore` will very likely fail to bootstrap at all, and the whole C.1 check job will never see a healthy cluster. The fix that matches the repo's own documented, working pattern is to add: ```yaml plugins: - name: barman-cloud.cloudnative-pg.io enabled: true isWALArchiver: false parameters: barmanObjectName: ha-pg-store-ro ``` which keeps the "never archives" property the comment wants (via `isWALArchiver: false`) instead of via omitting the plugin entirely. ## 🟡 Suggestions - **`kubernetes/apps/ha-restore-test/network-policy.yaml:221-255`** (`ha-pg-restore-ingress`) — this CiliumNetworkPolicy only defines an `ingress` section for `cnpg.io/cluster: ha-pg-restore`, so egress for those pods is left at Cilium's default-allow (no policy in this file or elsewhere in `ha-restore-test` restricts it). Every other namespace in this repo pairs ingress restriction with an explicit egress allow-list (e.g. `home-assistant-ns.yaml`'s namespace-wide `default-deny-ingress` policy, which — because its `endpointSelector: {}` also matches the production `ha-pg` pods — carries an `egress:` section that bounds them to specific destinations). Here the recovered Postgres pods can reach anything on egress (world, LAN, IoT VLAN) for the life of the throwaway Cluster. Low blast radius given the namespace is short-lived and deleted afterward, but worth an explicit `egress` list (S3 endpoint, CNPG operator, DNS) for consistency with the rest of the repo's least-privilege posture — and to avoid this becoming copy-paste precedent for the next throwaway namespace. ## 🟢 Nits - `kubernetes/apps/ha-restore-test/config-check-job.yaml:82-83` runs the probe container as `runAsUser: 0`/`runAsGroup: 0` to read HA's 0600 files. The comment justifies this, and `readOnlyRootFilesystem`/dropped capabilities/`allowPrivilegeEscalation: false`/the deny-all CiliumNetworkPolicy all mitigate it, but it is worth double-checking there's no non-root alternative (e.g. an init step `chmod`-ing via an ephemeral copy) before this pattern gets reused elsewhere. ## Tests No automated tests are expected or reasonable for a throwaway manifest-only change like this — verification is "read the two Jobs' logs," as stated in the PR body. CI (kustomize-build) covers manifest structure; it cannot catch the missing-plugin issue above since that only surfaces at cluster reconcile time. Given the blocking issue would only be discovered once applied to the cluster, it's worth fixing before merge rather than relying on the live run to reveal it. ## Recommendation Request changes — 1 blocking issue (missing `spec.plugins` on the recovery Cluster, which this repo's own runbook documents as a hard failure mode). <!-- pr-review:sha=5a4f7efd5ca6de27545800129571773b5bbfc95c -->
Sign in to join this conversation.
Reviewers
No Reviewers
Ghost
review-bot
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#1061