claude-botandClaude Opus 5.5
c862082d0e
The restore rehearsal: a throwaway three-node cluster on Hetzner Cloud that runs the production tree and restores production's backups into itself. hcloudtest (bootstrap.sh) runs SITE=hcloud on cpx52, and nothing stands in for the home network: - site-hcloud.yaml states the Hetzner site's facts, as site-homelab.yaml does at home: etcd advertised on the node segment 10.90.0.0/16; a cilium-site ConfigMap with tunnel (VXLAN) routing over the node segment and only the pod CIDRs exempt from masquerading (Hetzner networks are layer 3, IPv4-only, and drop packets with a source not assigned to the sending server); a ResolverConfig with k8s-gateway first and a k8s-gateway-site ConfigMap pointing at Hetzner's resolvers. The node range is read from the machine document only, now that the patch holds more than one (run 28). - hcloud-1..3.yaml put a 345GiB OSD partition beside a 60GiB EPHEMERAL on the system disk: the project's volume quota cannot hold three OSD volumes. No Hetzner volume is created for a raw volume on the system disk. - The Talos snapshot is built when missing (a rescue-mode server writes the Image Factory image; removed on failure), matched by version and schematic. This host's inbound firewall replaces earlier runs' without a server ever being left without one. The Hetzner token comes from OpenBao (homelab-backup-verify project). Access files land in ~/.local/state/homelab-hcloudtest instead of access-staging. - Containment: an outbound firewall on the nodes, applied before any config, admits udp and icmp anywhere and tcp to the complement of proxy.renner.uno's addresses, resolved over DoH at run time (cidr_complement, in bash; fails above Hetzner's 500 effective rules; one tcp rule per 100 prefixes, Hetzner's undocumented per-rule limit). The egress check pod (restore-test/egress-check.yaml) runs at the end of hcloudtest and again before the restore test writes any secret. FLUX_SOURCE=restore-test bootstrap.sh flux runs against that cluster only (its nodes must all be hcloud-*, and KUBECONFIG/TALOSCONFIG must not be the operator's): flux install, a per-run read-only deploy key on the GitHub mirror, the restore-test-git Secret, and restore-test/flux-root.yaml on FLUX_BRANCH. flux-root.yaml applies the production tree; its patches are the test's whole deviation: the source rewrite, no Ceph cluster network, Velero ReadOnly, no MariaDB dumps, rotators suspended, Gitea without world egress, netbird-service-sync at zero replicas and netbird-ingress on no node (the test cluster rewrote production NetBird's proxy services in run 28), and backup-verify suspended: it reads the same Hetzner token as hcloudtest and would delete production's monthly verify VM (both label managed-by=backup-verify). The patches are labelled 3a to 3h without gaps. The egress check resolves A and AAAA separately and fails on either error; before, a failed A lookup went unnoticed and the check ran on IPv6 alone. Removing a previous egress-check pod fails with a message. hcloudclean checks for bao like hcloudtest does. site-hcloud.yaml's header says what the site is for now and that Ceph replicates over the pod network there. CNPG archives, backs up and prunes as in production: the flux phase writes cnpg-paths with every archive path in renner-uno-k8s-cnpg-rtest, created there and emptied and deleted by hcloudclean. A test incarnation with a base backup in the production bucket would be the newest there, and the next real recovery would read the test's data. hcloudclean also deletes the test's ssh keys and the restore-test deploy keys. OpenBao's status wait is bounded at 30 minutes: on a fresh cluster the pod waits for Ceph to provision its volume. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
156 lines
5.7 KiB
YAML
156 lines
5.7 KiB
YAML
# Flux entry point of the off-site restore test (bootstrap.sh hcloudtest, then
|
|
# FLUX_SOURCE=restore-test bootstrap.sh flux). Not reconciled by production.
|
|
#
|
|
# Applies the production tree unchanged. spec.patches below is the complete
|
|
# list of differences between the test cluster and production; each is marked
|
|
# source, hardware or safety.
|
|
apiVersion: source.toolkit.fluxcd.io/v1
|
|
kind: GitRepository
|
|
metadata:
|
|
name: restore-test
|
|
namespace: flux-system
|
|
spec:
|
|
interval: 10m
|
|
url: ssh://git@github.com/konairius/Homelab.git
|
|
ref:
|
|
branch: master
|
|
# Read-only deploy key, created per run by bootstrap.sh.
|
|
secretRef:
|
|
name: restore-test-git
|
|
---
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: restore-test
|
|
namespace: flux-system
|
|
spec:
|
|
interval: 10m
|
|
path: ./kubernetes/clusters/homelab
|
|
prune: true
|
|
sourceRef:
|
|
kind: GitRepository
|
|
name: restore-test
|
|
patches:
|
|
# 1. Source: the GitHub mirror, read-only. In-cluster Gitea is restored by
|
|
# this test, not available to it.
|
|
- target:
|
|
kind: GitRepository
|
|
name: flux-system
|
|
namespace: flux-system
|
|
patch: |
|
|
- op: replace
|
|
path: /spec/url
|
|
value: ssh://git@github.com/konairius/Homelab.git
|
|
- op: replace
|
|
path: /spec/secretRef/name
|
|
value: restore-test-git
|
|
# 2. Source: the tree's own root sync is suspended; this Kustomization
|
|
# replaces it.
|
|
- target:
|
|
kind: Kustomization
|
|
name: flux-system
|
|
namespace: flux-system
|
|
patch: |
|
|
- op: add
|
|
path: /spec/suspend
|
|
value: true
|
|
# 3. Source: every Kustomization of the tree reads restore-test (the branch
|
|
# bootstrap.sh sets) and carries the patches below; a target a
|
|
# Kustomization does not contain matches nothing.
|
|
- target:
|
|
kind: Kustomization
|
|
patch: |
|
|
- op: replace
|
|
path: /spec/sourceRef/name
|
|
value: restore-test
|
|
- op: add
|
|
path: /spec/patches
|
|
value:
|
|
# 3a. Hardware: no Ceph cluster network. Hetzner networks are
|
|
# layer 3 and a macvlan child does not cross servers; Ceph
|
|
# replicates over the public (pod) network.
|
|
- target:
|
|
kind: CephCluster
|
|
name: rook-ceph
|
|
patch: |
|
|
- op: remove
|
|
path: /spec/network
|
|
# 3b. Safety: Velero ReadOnly blocks backups, deletion, GC and
|
|
# repository maintenance; restores and backup sync still work.
|
|
- target:
|
|
kind: HelmRelease
|
|
name: velero
|
|
patch: |
|
|
- op: add
|
|
path: /spec/values/configuration/backupStorageLocation/0/accessMode
|
|
value: ReadOnly
|
|
# CNPG needs no patch: bootstrap.sh points every cluster's archive
|
|
# at the test bucket (cnpg-paths), and the source stores carry no
|
|
# retentionPolicy, so WAL, base backups and retention all act on
|
|
# the test bucket only.
|
|
# 3c. Safety: no MariaDB logical dumps.
|
|
- target:
|
|
group: k8s.mariadb.com
|
|
kind: Backup
|
|
patch: |
|
|
- op: replace
|
|
path: /spec/schedule/suspend
|
|
value: true
|
|
# 3d. Safety: credential rotators would roll production's tokens
|
|
# at their issuers.
|
|
- target:
|
|
group: batch
|
|
kind: CronJob
|
|
labelSelector: app.kubernetes.io/component=secret-rotator
|
|
patch: |
|
|
- op: add
|
|
path: /spec/suspend
|
|
value: true
|
|
# 3e. Safety: Gitea gets no egress beyond the cluster. Its restored
|
|
# push mirror force-pushes (--mirror) over the GitHub mirror.
|
|
- target:
|
|
kind: CiliumNetworkPolicy
|
|
name: default-deny
|
|
namespace: gitea
|
|
patch: |
|
|
- op: test
|
|
path: /spec/egress/5/toEntities
|
|
value: [host, remote-node, world]
|
|
- op: replace
|
|
path: /spec/egress/5/toEntities
|
|
value: [host, remote-node]
|
|
# 3f. Safety: no writes to production NetBird. netbird-service-sync
|
|
# rewrites the public proxy services at netbird.renner.uno to
|
|
# this cluster's Traefik, fighting production's own controller.
|
|
- target:
|
|
kind: Deployment
|
|
name: netbird-service-sync
|
|
namespace: netbird-service-sync
|
|
patch: |
|
|
- op: replace
|
|
path: /spec/replicas
|
|
value: 0
|
|
# 3g. Safety: no routing peers in the production mesh. They would
|
|
# announce this cluster's service and LoadBalancer ranges, the
|
|
# same ranges production announces.
|
|
- target:
|
|
kind: DaemonSet
|
|
name: netbird-ingress
|
|
namespace: ingress
|
|
patch: |
|
|
- op: add
|
|
path: /spec/template/spec/nodeSelector
|
|
value:
|
|
restore-test.renner.uno/never: "true"
|
|
# 3h. Safety: no second backup-verify. It runs with production's
|
|
# Hetzner token in the same project and deletes every VM
|
|
# labelled managed-by=backup-verify, production's run included.
|
|
- target:
|
|
kind: CronJob
|
|
name: backup-verify
|
|
namespace: velero
|
|
patch: |
|
|
- op: replace
|
|
path: /spec/suspend
|
|
value: true
|