claude-botandClaude Opus 5.5
c862082d0e
The restore rehearsal: a throwaway three-node cluster on Hetzner Cloud that runs the production tree and restores production's backups into itself. hcloudtest (bootstrap.sh) runs SITE=hcloud on cpx52, and nothing stands in for the home network: - site-hcloud.yaml states the Hetzner site's facts, as site-homelab.yaml does at home: etcd advertised on the node segment 10.90.0.0/16; a cilium-site ConfigMap with tunnel (VXLAN) routing over the node segment and only the pod CIDRs exempt from masquerading (Hetzner networks are layer 3, IPv4-only, and drop packets with a source not assigned to the sending server); a ResolverConfig with k8s-gateway first and a k8s-gateway-site ConfigMap pointing at Hetzner's resolvers. The node range is read from the machine document only, now that the patch holds more than one (run 28). - hcloud-1..3.yaml put a 345GiB OSD partition beside a 60GiB EPHEMERAL on the system disk: the project's volume quota cannot hold three OSD volumes. No Hetzner volume is created for a raw volume on the system disk. - The Talos snapshot is built when missing (a rescue-mode server writes the Image Factory image; removed on failure), matched by version and schematic. This host's inbound firewall replaces earlier runs' without a server ever being left without one. The Hetzner token comes from OpenBao (homelab-backup-verify project). Access files land in ~/.local/state/homelab-hcloudtest instead of access-staging. - Containment: an outbound firewall on the nodes, applied before any config, admits udp and icmp anywhere and tcp to the complement of proxy.renner.uno's addresses, resolved over DoH at run time (cidr_complement, in bash; fails above Hetzner's 500 effective rules; one tcp rule per 100 prefixes, Hetzner's undocumented per-rule limit). The egress check pod (restore-test/egress-check.yaml) runs at the end of hcloudtest and again before the restore test writes any secret. FLUX_SOURCE=restore-test bootstrap.sh flux runs against that cluster only (its nodes must all be hcloud-*, and KUBECONFIG/TALOSCONFIG must not be the operator's): flux install, a per-run read-only deploy key on the GitHub mirror, the restore-test-git Secret, and restore-test/flux-root.yaml on FLUX_BRANCH. flux-root.yaml applies the production tree; its patches are the test's whole deviation: the source rewrite, no Ceph cluster network, Velero ReadOnly, no MariaDB dumps, rotators suspended, Gitea without world egress, netbird-service-sync at zero replicas and netbird-ingress on no node (the test cluster rewrote production NetBird's proxy services in run 28), and backup-verify suspended: it reads the same Hetzner token as hcloudtest and would delete production's monthly verify VM (both label managed-by=backup-verify). The patches are labelled 3a to 3h without gaps. The egress check resolves A and AAAA separately and fails on either error; before, a failed A lookup went unnoticed and the check ran on IPv6 alone. Removing a previous egress-check pod fails with a message. hcloudclean checks for bao like hcloudtest does. site-hcloud.yaml's header says what the site is for now and that Ceph replicates over the pod network there. CNPG archives, backs up and prunes as in production: the flux phase writes cnpg-paths with every archive path in renner-uno-k8s-cnpg-rtest, created there and emptied and deleted by hcloudclean. A test incarnation with a base backup in the production bucket would be the newest there, and the next real recovery would read the test's data. hcloudclean also deletes the test's ssh keys and the restore-test deploy keys. OpenBao's status wait is bounded at 30 minutes: on a fresh cluster the pod waits for Ceph to provision its volume. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>