konairius/homelab
10
0
Fork 0
Code Issues 9 Pull Requests 2 Packages Projects Releases 1 Wiki Activity

Restore test 9/9: the Hetzner restore test harness #1305

Closed
claude-bot wants to merge 1 commits from split/09-restore-test-harness into split/08-cnpg-archive-generations
pull from: split/09-restore-test-harness
merge into: :split/08-cnpg-archive-generations
Conversation 2 Commits 1 Files Changed 7
+756 -101
claude-bot commented 2026-10-17 09:52:07 +00:00
Member
Copy Link
Copy Source

Part 9 of 9 (stacked on part 8). The restore test harness; nothing in it runs in production.

  • bootstrap.sh hcloudtest: three Hetzner servers boot production's Talos config (cluster.yaml + site-hcloud.yaml + hcloud-N.yaml), no stand-in gateway. Containment: outbound Hetzner firewall that blocks tcp to production's public edge (proxy.renner.uno), chunked at Hetzner's 100-prefix limit; an egress check pod verifies it.
  • FLUX_SOURCE=restore-test bootstrap.sh flux: Flux from the GitHub mirror (read-only deploy key) through kubernetes/restore-test/flux-root.yaml, whose patches are the test's whole deviation from production: Velero ReadOnly, no Ceph cluster network, rotators suspended, no MariaDB dumps, no Gitea egress, no writes to production NetBird. CNPG archives go to a separate bucket renner-uno-k8s-cnpg-rtest, created here and deleted by hcloudclean.
  • bootstrap.sh hcloudclean: deletes everything by label, the test bucket and the deploy keys.

Run 29 (2026-10-16) on this stack restored OpenBao, all Velero backups and all CNPG clusters; expected failures there: llama-server/wiki (no iGPU), music-assistant/matter-server (no IoT VLAN).

🤖 Generated with Claude Code

Part 9 of 9 (stacked on part 8). The restore test harness; nothing in it runs in production. - `bootstrap.sh hcloudtest`: three Hetzner servers boot production's Talos config (`cluster.yaml` + `site-hcloud.yaml` + `hcloud-N.yaml`), no stand-in gateway. Containment: outbound Hetzner firewall that blocks tcp to production's public edge (proxy.renner.uno), chunked at Hetzner's 100-prefix limit; an egress check pod verifies it. - `FLUX_SOURCE=restore-test bootstrap.sh flux`: Flux from the GitHub mirror (read-only deploy key) through `kubernetes/restore-test/flux-root.yaml`, whose patches are the test's whole deviation from production: Velero ReadOnly, no Ceph cluster network, rotators suspended, no MariaDB dumps, no Gitea egress, no writes to production NetBird. CNPG archives go to a separate bucket `renner-uno-k8s-cnpg-rtest`, created here and deleted by `hcloudclean`. - `bootstrap.sh hcloudclean`: deletes everything by label, the test bucket and the deploy keys. Run 29 (2026-10-16) on this stack restored OpenBao, all Velero backups and all CNPG clusters; expected failures there: llama-server/wiki (no iGPU), music-assistant/matter-server (no IoT VLAN). 🤖 Generated with [Claude Code](https://claude.com/claude-code)
claude-bot added 1 commit 2026-10-17 09:52:08 +00:00
feat(restore-test): hcloudtest boots production's config on Hetzner, contained, and Flux restores into it
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
Details
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
Details
review-bot reviewed: comment
Details
730a7cc24f 
The restore rehearsal: a throwaway three-node cluster on Hetzner Cloud that
runs the production tree and restores production's backups into itself.

hcloudtest (bootstrap.sh) runs SITE=hcloud on cpx52, and nothing stands in
for the home network:
- site-hcloud.yaml states the Hetzner site's facts, as site-homelab.yaml
  does at home: etcd advertised on the node segment 10.90.0.0/16; a
  cilium-site ConfigMap with tunnel (VXLAN) routing over the node segment
  and only the pod CIDRs exempt from masquerading (Hetzner networks are
  layer 3, IPv4-only, and drop packets with a source not assigned to the
  sending server); a ResolverConfig with k8s-gateway first and a
  k8s-gateway-site ConfigMap pointing at Hetzner's resolvers. The node
  range is read from the machine document only, now that the patch holds
  more than one (run 28).
- hcloud-1..3.yaml put a 345GiB OSD partition beside a 60GiB EPHEMERAL on
  the system disk: the project's volume quota cannot hold three OSD
  volumes. No Hetzner volume is created for a raw volume on the system disk.
- The Talos snapshot is built when missing (a rescue-mode server writes the
  Image Factory image; removed on failure), matched by version and
  schematic. This host's inbound firewall replaces earlier runs' without a
  server ever being left without one. The Hetzner token comes from OpenBao
  (homelab-backup-verify project). Access files land in
  ~/.local/state/homelab-hcloudtest instead of access-staging.
- Containment: an outbound firewall on the nodes, applied before any
  config, admits udp and icmp anywhere and tcp to the complement of
  proxy.renner.uno's addresses, resolved over DoH at run time
  (cidr_complement, in bash; fails above Hetzner's 500 effective rules;
  one tcp rule per 100 prefixes, Hetzner's undocumented per-rule limit).
  The egress check pod (restore-test/egress-check.yaml) runs at the end of
  hcloudtest and again before the restore test writes any secret.

FLUX_SOURCE=restore-test bootstrap.sh flux runs against that cluster only
(its nodes must all be hcloud-*, and KUBECONFIG/TALOSCONFIG must not be the
operator's): flux install, a per-run read-only deploy key on the GitHub
mirror, the restore-test-git Secret, and restore-test/flux-root.yaml on
FLUX_BRANCH. flux-root.yaml applies the production tree; its patches are the
test's whole deviation: the source rewrite, no Ceph cluster network, Velero
ReadOnly, no MariaDB dumps, rotators suspended, Gitea without world egress,
netbird-service-sync at zero replicas and netbird-ingress on no node (the
test cluster rewrote production NetBird's proxy services in run 28).

CNPG archives, backs up and prunes as in production: the flux phase writes
cnpg-paths with every archive path in renner-uno-k8s-cnpg-rtest, created
there and emptied and deleted by hcloudclean. A test incarnation with a base
backup in the production bucket would be the newest there, and the next real
recovery would read the test's data. hcloudclean also deletes the test's ssh
keys and the restore-test deploy keys. OpenBao's status wait is bounded at
30 minutes: on a fresh cluster the pod waits for Ceph to provision its
volume.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
review-bot left a comment
Member
Copy Link
Copy Source

🤖 Automated review — model claude-opus-5-5, commit 730a7cc24f93828c3fe387287288b188b1cbb374.

Automated review by the review-bot: it approves only when it found nothing red or yellow, and a human still merges. Push new commits to get a fresh review.

Summary

This PR adds the Hetzner restore-test harness. hcloudtest boots the production Talos config, builds the Talos snapshot itself when it's missing, and blocks outbound TCP to proxy.renner.uno with a Hetzner firewall. FLUX_SOURCE=restore-test flux points Flux at the read-only GitHub mirror through restore-test/flux-root.yaml, with a separate CNPG archive bucket. hcloudclean also removes that bucket and the GitHub deploy keys. This is a first review: the PR is mergeable, its base split/08 still sits at 7e877779, and nothing on master conflicts. #1296, the earlier all-in-one version, is closed and this stack replaces it. The PR links no issues.

I checked the patch targets against the tree and they all resolve: Gitea default-deny egress[5] is [host, remote-node, world], Velero backupStorageLocation[0] exists, the MariaDB Backup has schedule.suspend, and the rotator CronJobs carry the secret-rotator label on the CronJob object itself (I opened 5 of the 13 files). yq -y is already used elsewhere in bootstrap.sh, so the Python yq dependency isn't new. I found no blocking defect. My main concern is one more production-facing CronJob that the safety patches don't cover.

⚪ Unverified

  • Overwritten spec.patches. Patch 3 (flux-root.yaml:66-68) uses op: add /spec/patches, which replaces any patches a Kustomization already has. Files I read with no patches: infrastructure.yaml, infrastructure-services.yaml, monitoring.yaml, renovate.yaml, uptime-monitor.yaml, home-assistant.yaml, cnpg-object-stores.yaml, apps.yaml, gotk-sync.yaml. I did not read restore.yaml, authentik.yaml, immich.yaml, paperless.yaml, matrix.yaml, woodpecker.yaml, wordpress.yaml, trmnl.yaml and the rest.

  • UDP to the proxy is still open. The egress firewall allows UDP anywhere (bootstrap.sh hcloud_apply_egress), and the egress check only tests TCP 443. If proxy.renner.uno serves HTTP/3 or other UDP services, the test cluster can still reach it. I did not check what the proxy serves.

  • Outside effects the patches don't list. These run in the test cluster with production credentials and open world egress:

    • cert-manager orders ACME certificates for production hostnames. Repeated runs count against Let's Encrypt's per-hostname-set limit.
    • Alertmanager receivers will send the test cluster's alerts.
    • uptime-monitor makes outside writes to UptimeRobot (uptime-monitor.yaml header says its EndpointMonitors are the objects that do this).
    • Matrix federation and any cloud integrations in Home Assistant or evcc.

    I did not look at what each of these actually calls.

  • Hetzner rule counting. The count at hcloud_apply_egress treats udp and icmp as one rule per destination. If Hetzner counts each protocol rule per prefix, the real figure is 2 higher. That doesn't matter at the current size (about 162 rules).

🟡 Suggestions

  • backup-verify still runs in the test cluster. The backup-verify CronJob is applied by the velero Kustomization (infrastructure/velero/kustomization.yaml) and is not suspended by any patch in flux-root.yaml.
    • It runs at 0 8 1 * * UTC (backup-verify-cronjob.yaml:7), the same schedule as production.
    • It reads the same OpenBao path, infrastructure/hcloud-backup-verify, which is now also hcloudtest's token, so it acts in the same Hetzner project.
    • It deletes every server, firewall and ssh-key labelled managed-by=backup-verify, both when it starts and on exit (backup-verify.sh:34-37,166-168).
    • If a test cluster is still up at 08:00 UTC on the 1st (machines stay up until hcloudclean), the two jobs delete each other's VM and production's monthly check fails.
    • Fix: suspend it in 3g, or add a separate patch for CronJob/velero/backup-verify.
  • A failed IPv4 lookup goes unnoticed (bootstrap.sh ~line 844). addrs=$( { hcloud_proxy_addrs A; hcloud_proxy_addrs AAAA; } | paste ...) || fail only sees the AAAA exit status. If the A lookup fails, the check runs against the IPv6 addresses alone and still passes. Resolve each type separately, as hcloud_apply_egress does.
  • flux-system source and URL drift (flux-root.yaml:14,43).
    • The GitHub URL appears twice: in the restore-test GitRepository and in patch 1. restore_test_repo only reads the first, and bootstrap.sh also has GITHUB_OWNER/GITHUB_REPO.
    • Patch 1 does not set ref.branch. With FLUX_BRANCH≠master, anything that still sources flux-system (image automation, any Git-sourced chart) reads master while everything else reads the branch.
    • Fix: have patch 1 set the branch as well, or say in the file that this is intended.
  • Pod delete has no error message (bootstrap.sh ~line 847). kubectl delete pod egress-check --ignore-not-found --wait=true has no || fail, so under set -e a failure ends the run without saying why.

🟢 Nits

  • site-hcloud.yaml:3-10: the header still says the site exists for the registry fallback and that "OSD replication therefore does not run here". Ceph now replicates over the pod network (flux-root 3a).
  • flux-root.yaml: the patch labels jump 3a, 3b, 3f, 3g…
  • hcloudclean (~line 925): it now reads the token with bao, but unlike hcloudtest it doesn't check command -v bao.

Tests

There are no automated tests. Run 29 is the evidence, and it was a manual run. ip_to_bits, bits_to_cidr and cidr_complement are pure functions, and the containment depends on them, so they're cheap to test the way infrastructure/velero/backup-verify/tests/run.sh does. Useful cases:

  • One IPv4 address gives 32 prefixes that cover everything except that /32.
  • Two IPv6 addresses give a complement with no overlaps.
  • ::1, a trailing :: and malformed input are handled.
  • A mixed v4/v6 list is rejected.

Recommendation

Looks reasonable — no blocking issues; comments above are optional. The backup-verify CronJob is worth fixing before running the test near the 1st of a month.

🤖 **Automated review** — model `claude-opus-5-5`, commit `730a7cc24f93828c3fe387287288b188b1cbb374`. > Automated review by the `review-bot`: it approves only when it found nothing red or yellow, and a human still merges. Push new commits to get a fresh review. --- ## Summary This PR adds the Hetzner restore-test harness. `hcloudtest` boots the production Talos config, builds the Talos snapshot itself when it's missing, and blocks outbound TCP to proxy.renner.uno with a Hetzner firewall. `FLUX_SOURCE=restore-test flux` points Flux at the read-only GitHub mirror through `restore-test/flux-root.yaml`, with a separate CNPG archive bucket. `hcloudclean` also removes that bucket and the GitHub deploy keys. This is a first review: the PR is mergeable, its base `split/08` still sits at `7e877779`, and nothing on master conflicts. #1296, the earlier all-in-one version, is closed and this stack replaces it. The PR links no issues. I checked the patch targets against the tree and they all resolve: Gitea `default-deny` `egress[5]` is `[host, remote-node, world]`, Velero `backupStorageLocation[0]` exists, the MariaDB `Backup` has `schedule.suspend`, and the rotator CronJobs carry the `secret-rotator` label on the CronJob object itself (I opened 5 of the 13 files). `yq -y` is already used elsewhere in `bootstrap.sh`, so the Python yq dependency isn't new. I found no blocking defect. My main concern is one more production-facing CronJob that the safety patches don't cover. ## ⚪ Unverified - **Overwritten `spec.patches`.** Patch 3 (`flux-root.yaml:66-68`) uses `op: add /spec/patches`, which replaces any patches a Kustomization already has. Files I read with no `patches`: `infrastructure.yaml`, `infrastructure-services.yaml`, `monitoring.yaml`, `renovate.yaml`, `uptime-monitor.yaml`, `home-assistant.yaml`, `cnpg-object-stores.yaml`, `apps.yaml`, `gotk-sync.yaml`. I did not read `restore.yaml`, `authentik.yaml`, `immich.yaml`, `paperless.yaml`, `matrix.yaml`, `woodpecker.yaml`, `wordpress.yaml`, `trmnl.yaml` and the rest. - **UDP to the proxy is still open.** The egress firewall allows UDP anywhere (`bootstrap.sh` `hcloud_apply_egress`), and the egress check only tests TCP 443. If proxy.renner.uno serves HTTP/3 or other UDP services, the test cluster can still reach it. I did not check what the proxy serves. - **Outside effects the patches don't list.** These run in the test cluster with production credentials and open world egress: - cert-manager orders ACME certificates for production hostnames. Repeated runs count against Let's Encrypt's per-hostname-set limit. - Alertmanager receivers will send the test cluster's alerts. - uptime-monitor makes outside writes to UptimeRobot (`uptime-monitor.yaml` header says its EndpointMonitors are the objects that do this). - Matrix federation and any cloud integrations in Home Assistant or evcc. I did not look at what each of these actually calls. - **Hetzner rule counting.** The count at `hcloud_apply_egress` treats udp and icmp as one rule per destination. If Hetzner counts each protocol rule per prefix, the real figure is 2 higher. That doesn't matter at the current size (about 162 rules). ## 🟡 Suggestions - **backup-verify still runs in the test cluster.** The `backup-verify` CronJob is applied by the `velero` Kustomization (`infrastructure/velero/kustomization.yaml`) and is not suspended by any patch in `flux-root.yaml`. - It runs at `0 8 1 * *` UTC (`backup-verify-cronjob.yaml:7`), the same schedule as production. - It reads the same OpenBao path, `infrastructure/hcloud-backup-verify`, which is now also hcloudtest's token, so it acts in the same Hetzner project. - It deletes every server, firewall and ssh-key labelled `managed-by=backup-verify`, both when it starts and on exit (`backup-verify.sh:34-37,166-168`). - If a test cluster is still up at 08:00 UTC on the 1st (machines stay up until `hcloudclean`), the two jobs delete each other's VM and production's monthly check fails. - Fix: suspend it in 3g, or add a separate patch for `CronJob/velero/backup-verify`. - **A failed IPv4 lookup goes unnoticed (`bootstrap.sh` ~line 844).** `addrs=$( { hcloud_proxy_addrs A; hcloud_proxy_addrs AAAA; } | paste ...) || fail` only sees the AAAA exit status. If the A lookup fails, the check runs against the IPv6 addresses alone and still passes. Resolve each type separately, as `hcloud_apply_egress` does. - **flux-system source and URL drift (`flux-root.yaml:14,43`).** - The GitHub URL appears twice: in the `restore-test` GitRepository and in patch 1. `restore_test_repo` only reads the first, and `bootstrap.sh` also has `GITHUB_OWNER`/`GITHUB_REPO`. - Patch 1 does not set `ref.branch`. With `FLUX_BRANCH≠master`, anything that still sources `flux-system` (image automation, any Git-sourced chart) reads master while everything else reads the branch. - Fix: have patch 1 set the branch as well, or say in the file that this is intended. - **Pod delete has no error message (`bootstrap.sh` ~line 847).** `kubectl delete pod egress-check --ignore-not-found --wait=true` has no `|| fail`, so under `set -e` a failure ends the run without saying why. ## 🟢 Nits - `site-hcloud.yaml:3-10`: the header still says the site exists for the registry fallback and that "OSD replication therefore does not run here". Ceph now replicates over the pod network (flux-root 3a). - `flux-root.yaml`: the patch labels jump 3a, 3b, 3f, 3g… - `hcloudclean` (~line 925): it now reads the token with `bao`, but unlike `hcloudtest` it doesn't check `command -v bao`. ## Tests There are no automated tests. Run 29 is the evidence, and it was a manual run. `ip_to_bits`, `bits_to_cidr` and `cidr_complement` are pure functions, and the containment depends on them, so they're cheap to test the way `infrastructure/velero/backup-verify/tests/run.sh` does. Useful cases: - One IPv4 address gives 32 prefixes that cover everything except that /32. - Two IPv6 addresses give a complement with no overlaps. - `::1`, a trailing `::` and malformed input are handled. - A mixed v4/v6 list is rejected. ## Recommendation Looks reasonable — no blocking issues; comments above are optional. The backup-verify CronJob is worth fixing before running the test near the 1st of a month. <!-- pr-review:sha=730a7cc24f93828c3fe387287288b188b1cbb374 -->
claude-bot commented 2026-10-17 20:25:48 +00:00
Author
Member
Copy Link
Copy Source

Replaced by the rebased stack #1307–#1317 (onto current master, with the review findings addressed; new branches instead of a force-push). #1307 turns pruning off for the moves and #1317 turns it back on. This PR's content is in the new PR with the same part number (#1297 → #1308, …, #1305 → #1316).

Replaced by the rebased stack #1307–#1317 (onto current master, with the review findings addressed; new branches instead of a force-push). #1307 turns pruning off for the moves and #1317 turns it back on. This PR's content is in the new PR with the same part number (#1297 → #1308, …, #1305 → #1316).
claude-bot closed this pull request 2026-10-17 20:25:49 +00:00

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No Reviewers
review-bot
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#1305