konairius/homelab
10
0
Fork 0
Code Issues 9 Pull Requests 2 Packages Projects Releases 1 Wiki Activity

Restore test fixes: site facts, self-restore, CNPG archive generations, dependency graph #1296

Closed
claude-bot wants to merge 43 commits from worktree-restore-test-real into main
pull from: worktree-restore-test-real
merge into: :main
Conversation 1 Commits 43 Files Changed 151
+2596 -574
claude-bot commented 2026-10-17 08:29:29 +00:00
Member
Copy Link
Copy Source

The changes from the Hetzner restore test (runs 23–29): the production configuration boots a cluster on new hardware and restores itself, unchanged, and the dependency graph says what the services actually need.

Do not merge in one step. Several objects move between pruned Kustomizations (pruning the Velero HelmRelease out of infrastructure would uninstall Velero), and some changes need state in production first. The rollout order is below.

What changes

Site facts in the Talos site patches

  • cilium-site and k8s-gateway-site ConfigMaps as Talos inline manifests per site; the HelmReleases read them via valuesFrom (pod routing and native-routing CIDRs, k8s-gateway upstream).
  • etcd advertisedSubnets and nodeIP subnets per site; IPv4 first in validSubnets.
  • Nodes resolve through k8s-gateway at every site; bootstrap.sh installs it right after the CNI.
  • Node labels for hardware: hardware.renner.uno/intel-gpu, net.renner.uno/iot-vlan; bird runs only where the site sets net.renner.uno/pod-table=47.

Restore from backup, without manual steps

  • The cluster restores its PVCs itself: one restore-<schedule> Kustomization per Velero Schedule, Gitea first; each app depends on its own restore.
  • CNPG: every cluster incarnation archives to a new prefix <x>-g<timestamp>; recovery reads the prefix with the newest base backup. bootstrap.sh writes flux-system/cnpg-paths before Flux starts; the 14 ObjectStores live in kubernetes/cnpg-object-stores with Flux substitution. The plugin's empty-archive check stays on.
  • woodpecker recovers from the prefix it archives to (also merged as #1295).
  • Rook: the ceph-csi-drivers chart Rook requires; Gitea PVC without a pinned volumeName.

Dependency graph (scripts/flux-graph.sh draws it)

  • Every app Kustomization waits for its workloads (wait: true); otbr suspended.
  • Edges: in-house images → gitea (authentik, paperless, home-assistant, review-bot, wiki, infrastructure-inhouse); gitea-mcp, woodpecker, renovate → gitea; evcc, trmnl → home-assistant; home-assistant, monitoring → influxdb.
  • infrastructure is Ready only once OpenBao is unsealed.
  • crowdsec, external-dns-unifi, netbird-ingress and velero get Kustomizations of their own after infrastructure-crds, each with its ExternalSecrets; Velero Schedules follow velero.
  • bird and netbird-service-sync move to infrastructure-inhouse (waits for Gitea).
  • shelly-manager and the mDNS relay move to home-assistant-iot, so HA does not wait on the IoT VLAN.
  • authentik's server ServiceMonitor and the radius metrics Service: the chart/operator ones are used, no hand-written copies.
  • home-assistant timeout 30m (was 4h, sized for a retired seed step).

Restore test harness (bootstrap.sh hcloudtest, FLUX_SOURCE=restore-test, hcloudclean)

  • Hetzner cluster from cluster.yaml + site-hcloud.yaml + hcloud-N.yaml, contained by an egress firewall; Flux from the GitHub mirror through kubernetes/restore-test/flux-root.yaml, whose patches are the test's deviations (Velero ReadOnly, rotators suspended, no NetBird writes, no Gitea egress).
  • The test's CNPG archives go to its own bucket renner-uno-k8s-cnpg-rtest, created and deleted by the harness.

Rollout order (production)

  1. Talos first (one node at a time, drain, wait for Ceph): apply site-homelab.yaml and k8s-{1,2,3}.yaml. Check that the nodes carry hardware.renner.uno/intel-gpu, net.renner.uno/iot-vlan and net.renner.uno/pod-table=47, and that the cilium-site and k8s-gateway-site ConfigMaps exist in kube-system (Talos creates inline manifests only on a config change; talosctl upgrade-k8s if needed).
  2. Create flux-system/cnpg-paths by hand with the current prefixes as both source and archive, so the running clusters keep archiving where they do: CNPG_<X>_SOURCE/_ARCHIVE = s3://renner-uno-k8s-cnpg/<x>-pg for ak, ha, im, mx, pprl, tr, and .../wp-pg-woodpecker for wp.
  3. Two-phase moves, each: add the new Kustomization with the objects first, let it adopt them, then drop them from the old one.
    • infrastructure → crowdsec, external-dns-unifi, netbird-ingress, velero (+ ExternalSecrets from infrastructure-crds), velero-schedules
    • infrastructure → infrastructure-inhouse (bird, netbird-service-sync)
    • apps → cnpg-object-stores (all ObjectStores now carry prune: disabled)
    • home-assistant → home-assistant-iot
  4. Check before merging the restore Kustomizations: what a restore-<schedule> Restore does in a running production cluster whose PVCs already exist. Not verified in production.
  5. Not verified: whether changing externalClusters on a live CNPG Cluster restarts its pods. Check on one cluster first.

Evidence

Restore test run 29 (2026-10-16) on this branch: cluster, OpenBao restore, all Velero restores and all CNPG recoveries succeeded (the empty-archive check passed against the new prefixes; woodpecker, which failed in run 28, recovered). Failures found there are fixed on the branch: the home-assistant ↔ home-assistant-iot Secret cycle, the radius Service blocking monitoring-rules, Secret ordering of the infrastructure services. Expected on Hetzner: llama-server/wiki (no iGPU), music-assistant/matter-server (no IoT VLAN). Not yet tested end to end: the home-assistant chain (HA, evcc, trmnl, zone-configurator, speech-to-phrase), which run 29 did not reach.

🤖 Generated with Claude Code

The changes from the Hetzner restore test (runs 23–29): the production configuration boots a cluster on new hardware and restores itself, unchanged, and the dependency graph says what the services actually need. **Do not merge in one step.** Several objects move between pruned Kustomizations (pruning the Velero HelmRelease out of `infrastructure` would uninstall Velero), and some changes need state in production first. The rollout order is below. ## What changes **Site facts in the Talos site patches** - `cilium-site` and `k8s-gateway-site` ConfigMaps as Talos inline manifests per site; the HelmReleases read them via `valuesFrom` (pod routing and native-routing CIDRs, k8s-gateway upstream). - etcd `advertisedSubnets` and nodeIP subnets per site; IPv4 first in `validSubnets`. - Nodes resolve through k8s-gateway at every site; bootstrap.sh installs it right after the CNI. - Node labels for hardware: `hardware.renner.uno/intel-gpu`, `net.renner.uno/iot-vlan`; bird runs only where the site sets `net.renner.uno/pod-table=47`. **Restore from backup, without manual steps** - The cluster restores its PVCs itself: one `restore-<schedule>` Kustomization per Velero Schedule, Gitea first; each app depends on its own restore. - CNPG: every cluster incarnation archives to a new prefix `<x>-g<timestamp>`; recovery reads the prefix with the newest base backup. bootstrap.sh writes `flux-system/cnpg-paths` before Flux starts; the 14 ObjectStores live in `kubernetes/cnpg-object-stores` with Flux substitution. The plugin's empty-archive check stays on. - woodpecker recovers from the prefix it archives to (also merged as #1295). - Rook: the ceph-csi-drivers chart Rook requires; Gitea PVC without a pinned volumeName. **Dependency graph** (`scripts/flux-graph.sh` draws it) - Every app Kustomization waits for its workloads (`wait: true`); otbr suspended. - Edges: in-house images → gitea (authentik, paperless, home-assistant, review-bot, wiki, infrastructure-inhouse); gitea-mcp, woodpecker, renovate → gitea; evcc, trmnl → home-assistant; home-assistant, monitoring → influxdb. - infrastructure is Ready only once OpenBao is unsealed. - crowdsec, external-dns-unifi, netbird-ingress and velero get Kustomizations of their own after infrastructure-crds, each with its ExternalSecrets; Velero Schedules follow velero. - bird and netbird-service-sync move to `infrastructure-inhouse` (waits for Gitea). - shelly-manager and the mDNS relay move to `home-assistant-iot`, so HA does not wait on the IoT VLAN. - authentik's server ServiceMonitor and the radius metrics Service: the chart/operator ones are used, no hand-written copies. - home-assistant timeout 30m (was 4h, sized for a retired seed step). **Restore test harness** (`bootstrap.sh hcloudtest`, `FLUX_SOURCE=restore-test`, `hcloudclean`) - Hetzner cluster from `cluster.yaml` + `site-hcloud.yaml` + `hcloud-N.yaml`, contained by an egress firewall; Flux from the GitHub mirror through `kubernetes/restore-test/flux-root.yaml`, whose patches are the test's deviations (Velero ReadOnly, rotators suspended, no NetBird writes, no Gitea egress). - The test's CNPG archives go to its own bucket `renner-uno-k8s-cnpg-rtest`, created and deleted by the harness. ## Rollout order (production) 1. **Talos first** (one node at a time, drain, wait for Ceph): apply `site-homelab.yaml` and `k8s-{1,2,3}.yaml`. Check that the nodes carry `hardware.renner.uno/intel-gpu`, `net.renner.uno/iot-vlan` and `net.renner.uno/pod-table=47`, and that the `cilium-site` and `k8s-gateway-site` ConfigMaps exist in kube-system (Talos creates inline manifests only on a config change; `talosctl upgrade-k8s` if needed). 2. **Create `flux-system/cnpg-paths` by hand** with the current prefixes as both source and archive, so the running clusters keep archiving where they do: `CNPG_<X>_SOURCE`/`_ARCHIVE` = `s3://renner-uno-k8s-cnpg/<x>-pg` for ak, ha, im, mx, pprl, tr, and `.../wp-pg-woodpecker` for wp. 3. **Two-phase moves**, each: add the new Kustomization with the objects first, let it adopt them, then drop them from the old one. - `infrastructure` → `crowdsec`, `external-dns-unifi`, `netbird-ingress`, `velero` (+ ExternalSecrets from `infrastructure-crds`), `velero-schedules` - `infrastructure` → `infrastructure-inhouse` (bird, netbird-service-sync) - apps → `cnpg-object-stores` (all ObjectStores now carry `prune: disabled`) - `home-assistant` → `home-assistant-iot` 4. **Check before merging the restore Kustomizations:** what a `restore-<schedule>` Restore does in a running production cluster whose PVCs already exist. Not verified in production. 5. Not verified: whether changing `externalClusters` on a live CNPG Cluster restarts its pods. Check on one cluster first. ## Evidence Restore test run 29 (2026-10-16) on this branch: cluster, OpenBao restore, all Velero restores and all CNPG recoveries succeeded (the empty-archive check passed against the new prefixes; woodpecker, which failed in run 28, recovered). Failures found there are fixed on the branch: the home-assistant ↔ home-assistant-iot Secret cycle, the radius Service blocking monitoring-rules, Secret ordering of the infrastructure services. Expected on Hetzner: llama-server/wiki (no iGPU), music-assistant/matter-server (no IoT VLAN). Not yet tested end to end: the home-assistant chain (HA, evcc, trmnl, zone-configurator, speech-to-phrase), which run 29 did not reach. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
claude-bot added 43 commits 2026-10-17 08:29:29 +00:00
fix(talos): IPv4 first in nodeIP.validSubnets, as the live nodes run 5927e4bc0e 
Talos keeps the validSubnets order (siderolabs/net FilterIPs) and the first
entry becomes the primary node IP. Git had fd02:46::/64 first since
be643533 while k8s-1 runs with 10.2.46.0/24 first. A cluster built from git
gets an IPv6 primary and kube-apiserver exits with
'service IP family "10.96.0.0/12" must match public address family
"fd02:46::11"' — reproduced on a Hetzner rebuild rehearsal 2026-10-15.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix(gitea): no pinned volumeName on data-gitea-0 fb7c12a92a 
The claim named its production PV, which does not exist on a rebuilt
cluster: the claim stayed Pending there, and a restored claim bound to a
new volume made the gitea Kustomization fail on the immutable field.

volumeName leaves git and the gitea Kustomization ignores
/spec/volumeName on this claim. kustomize-controller is the field's
only manager on the live claim, so its ignore rule adopts the live value
instead of clearing it; size growth still applies.

To verify after merge: the gitea Kustomization reconciles Ready and the
live claim still names pvc-e7c56db2-481d-40cb-b94a-b264b7b4b5be.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat(velero): the cluster restores its app volumes itself dcc034be92 
One Velero Restore per Schedule (kubernetes/restore/restores.yaml),
applied by Flux, restoring the schedule's newest completed backup. Every
app Kustomization with backed-up volumes depends on the new `restore`
Kustomization, so on a fresh cluster no app starts on an empty volume.
Restores the capability 952e6150 removed without an owner decision.

Ordering and failure handling, from Velero release-1.17 source:
- A Restore whose schedule has no synced backup yet goes FailedValidation
  for good (restore_controller.go). `restore` therefore depends on
  `velero-backups-synced`, which is Ready once the BackupStorageLocation
  has completed a sync (status.lastSyncedTime).
- After that sync, FailedValidation with "No backups found for schedule"
  means there is nothing to restore and counts as done; any other
  FailedValidation, Failed or PartiallyFailed fails the Kustomization.
- A claim that already exists is skipped (csi pvc_action: "PVC already
  exists"), and the Restore completes with a warning. On the running
  cluster the Restores touch nothing.
- Velero deletes a Restore with the backup it came from (30-day TTL);
  Flux recreates it, and it again skips the existing claims.
- All StorageClasses bind Immediate, so the data mover does not wait for
  a consuming pod.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix(rook): install the ceph-csi-drivers chart Rook requires 99af3d6efc 
Since Rook 1.20 the CSI Driver and OperatorConfig resources come from the
ceph-csi-drivers chart (rook Documentation/Helm-Charts/csi-drivers-chart.md:
"Rook requires the installation of the Ceph-CSI Driver chart"). The chart
was never installed here: the live cluster's Drivers are the ones the
operator migrated once during the 1.19->1.20 upgrade (2026-05-09), and
PR #12 only rendered the ServiceAccounts and RBAC. A fresh cluster
therefore has no CSI driver and provisions no volume: found by the
Hetzner restore test, where every PVC stayed Pending ("Waiting for a
volume to be created either by the external provisioner
'rook-ceph.rbd.csi.ceph.com'").

Values: Rook's recommended values for 1.21.0 plus the live shape:
2 controller replicas with Recreate, the workload-class=pinned
toleration on the node plugins, a snapshotter for RBD (Velero's CSI
snapshots of ceph-block) and group snapshots for CephFS.

The chart's ServiceAccounts are named per driver
(rook-ceph-rbd-csi-ceph-com-ctrlplugin-sa, ...), so they do not collide
with the static ones; the static ServiceAccounts and RBAC stay until the
live CSI pods run on the chart's, and go in a second change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix(velero): restore Gitea first, the rest after it 33077cd1bc 
Gitea is the registry for the in-house images. With one shared restore gate it waited for every volume, Immich's included, while everything pulling from it looped on image pull errors. restore-gitea restores only Gitea and the gitea Kustomization depends on it alone; the other Restores depend on it, so Velero moves Gitea's data first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix(talos): etcd subnet in the site patches; hcloud OSD on the system disk af7c48e29f 
cluster.yaml named VLAN 46 as etcd's advertised subnet, a fact of the home
network. It moves unchanged to site-homelab.yaml (the rendered production
config is byte-identical); site-hcloud.yaml advertises etcd on the node
segment bootstrap.sh provisions, 10.90.0.0/16.

cpx52 carries 480 GB local disk and the project's volume quota cannot hold
three OSD volumes: EPHEMERAL is capped at 60GiB and osd-data takes at least
345GiB of the system disk (ported from 92c7ee73, 8d42d89c).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat(bootstrap): hcloudtest boots production's config, contained by egress firewall bb6874f59c 
hcloudtest runs SITE=hcloud on cpx52 with the OSD on the system disk, and
nothing stands in for the home network. Ported from worktree-hcloud-wg-site:
the Talos snapshot builder (removes its server on failure), this host's
inbound firewall replacing earlier runs', the Hetzner token from OpenBao, the
FLUX_SOURCE=restore-test phase (flux install, per-run read-only deploy key on
the GitHub mirror, restore-test-git Secret, flux-root.yaml on FLUX_BRANCH),
its safety prechecks, access files under ~/.local/state/homelab-hcloudtest
instead of access-staging, hcloudclean deleting ssh keys and restore-test
deploy keys, and the 30-minute OpenBao status wait.

Containment: an outbound firewall on the nodes, applied before any config,
admits udp and icmp anywhere and tcp to the complement of proxy.renner.uno's
addresses, resolved over DoH at run time (cidr_complement, in bash; fails
above Hetzner's 500 effective rules). The egress check pod runs at the end
of hcloudtest and again before the restore test writes any secret.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat(restore-test): Flux root with the test's deviations, and the egress check b550e84fcc 
flux-root.yaml applies the production tree from the GitHub mirror; its
patches are the source rewrite, no Ceph cluster network (Hetzner networks
are layer 3), and the safety set: Velero ReadOnly, no CNPG WAL archiving,
retention or base backups, no MariaDB dumps, rotators suspended, Gitea
without world egress.

egress-check.yaml asserts from a pod: public DNS, github.com https and ssh,
ghcr.io and object storage reachable; every proxy.renner.uno address closed
on 443. stderr is reported, not discarded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix(bootstrap): create Cilium's CNI subdirectory and install Multus on a cold cluster f58e06fc76 
Cilium's cni.confPath is /etc/cni/net.d/multus, a directory Multus creates
(#832). On an empty cluster neither exists: /etc/cni is read-only to the
kubelet on Talos, so the agent's DirectoryOrCreate mount cannot make it,
and the install waits forever. Without Multus the kubelet reports "cni
plugin not initialized" and nothing schedules, Flux included.

A hostNetwork pod per node creates the directory (there is no pod network
before Cilium), and Multus is applied from the manifests Flux owns right
after Cilium. Found in the Hetzner restore rehearsal, where both steps ran
successfully in runs 22 and 23; this is the production bootstrap path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix(bootstrap): split the hcloudtest tcp egress rule at 100 prefixes 4922d8777e 
Hetzner rejects a firewall rule with more than 100 destination prefixes
("rules[2].destination_ips: value required to be smaller"; the limit is
not in the API spec or the FAQ — measured on an unattached firewall: 100
accepted, 101 rejected). Run 24 failed on the single rule carrying all
245 complement prefixes. The tcp allow-list is now one rule per 100.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat(cilium): pod routing is a site fact, from the site patch 16b6aee04d 
routingMode, autoDirectNodeRoutes and nodePort.directRoutingDevice describe
the site's network, not the cluster: at home pod traffic crosses nodes
natively on pod0 (VLAN 48, one L2 segment). On Hetzner Cloud that cannot
work: networks are layer 3, route only server addresses, and carry IPv4
only (docs.hetzner.com networks FAQ), so cross-node pod traffic had no
path (run 25: every DNS query to a CoreDNS pod on another node timed out).

Each Talos site patch now carries a cilium-site ConfigMap as an inline
manifest with its part of Cilium's values; the HelmRelease reads it via
valuesFrom and bootstrap.sh merges it the same way (site values first,
HelmRelease values on top, as Flux merges them):
- homelab: native, autoDirectNodeRoutes, directRoutingDevice pod0
- hcloud: tunnel (VXLAN) over the node segment
- qemu: native on the provisioner's single L2 bridge

Production renders unchanged: helm template of cilium 1.20.3 with master's
values vs. homelab site values + the new values gives identical manifests
(random TLS secrets excluded). talosctl validate passes for k8s-1 (metal)
and hcloud-1 (cloud).

Rollout order for production: the ConfigMap must exist before this
HelmRelease reconciles (valuesFrom is not optional). Talos creates inline
manifests on a config change but never updates them; changes go through
talosctl upgrade-k8s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat(cilium): native-routing CIDRs are a site fact too ed8241384d 
ipv4/ipv6NativeRoutingCIDR exempt traffic into those ranges from
masquerading (Cilium: handed to the kernel "without applying any SNAT").
At home that is the point: 10.0.0.0/8 and fd00::/8 let LAN services see
real pod IPs, routed back by the UDM via BGP. On Hetzner the same range
covers the node segment 10.90.0.0/16, whose gateway drops packets with a
source not assigned to the sending server (uRPF, Hetzner networks FAQ).

Run 26: a pod on hcloud-2 reached only its own node's kube-apiserver
(10.90.0.3:6443 -> 401); 10.90.0.2 and 10.90.0.4 timed out, the pod's SYNs
went to-stack with the pod address. Flux controllers on hcloud-2 crash-
looped and flux install failed.

The CIDRs move into each site's cilium-site values: homelab and qemu keep
10.0.0.0/8 and fd00::/8, hcloud exempts only the pod CIDRs. Production
still renders identically (helm template, TLS secrets excluded); talosctl
validate passes for k8s-1 and hcloud-1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix(flux): infrastructure is Ready only once OpenBao is unsealed a5c5d0e79d 
The infrastructure Kustomization health-checked the openbao HelmRelease,
which is Ready as soon as the chart is installed, sealed or not. On a cold
cluster infrastructure-crds then applied the openbao ClusterSecretStore
while OpenBao was unreachable and then sealed ("503 Vault is sealed").
External Secrets backs the store off exponentially, so every
ExternalSecret stayed "ClusterSecretStore not ready" for minutes after
OpenBao was restored and unsealed; Velero's pod could not mount
cloud-credentials-hetzner meanwhile (restore rehearsal run 27; ~15 min in
run 22).

The openbao StatefulSet is now a health check too. Its readiness probe is
`bao status`, which fails while sealed, and podManagementPolicy is
OrderedReady, so 3/3 ready means every replica is unsealed. No cycle:
OpenBao's PVC comes from infrastructure-ceph-cluster, which does not
depend on infrastructure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat(dns): the nodes resolve through k8s-gateway at every site, from the first pod on 4a4390ef46 
Node-level image pulls of gitea.renner.uno/* go through the node's resolver,
not the cluster's. Only site-homelab.yaml pointed it at k8s-gateway (PR
#1292); on Hetzner the nodes asked Hetzner's resolvers, got the public
proxy (blocked by the test firewall) and fell through to ghcr, which only
mirrors konairius/* - bird, netbird-service-sync and review-bot stayed in
ImagePullBackOff (restore rehearsal run 27) while the restored Gitea had
them.

PR #1292 also left a cold-start gap (run 23): until k8s-gateway exists the
node's first resolver does not answer, every lookup waits out the 2 s read
timeout, CoreDNS gives up first and pods resolve nothing - Flux included.

- Each site patch states its node resolver list (ResolverConfig: k8s-gateway
  first, then the site's resolver: UDM at home, Hetzner's on hcloud) and
  k8s-gateway's upstream as a k8s-gateway-site ConfigMap (deployment
  dnsPolicy/dnsConfig), read by the HelmRelease via valuesFrom. The Corefile
  forwards to the pod's resolv.conf, never to the node's resolver (loop).
- bootstrap.sh installs k8s-gateway right after the nodes are Ready, from
  the HelmRelease under the site values, as Flux merges them; Flux adopts
  the release. k8s-gateway answers before anything else needs DNS.

helm template (chart 4.0.1) renders forward . /etc/resolv.conf with the
site's nameservers in dnsConfig for homelab and hcloud. talosctl validate
passes for k8s-1 and hcloud-1.

Rollout order for production: the k8s-gateway-site ConfigMap must exist
before this HelmRelease reconciles (valuesFrom is not optional).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix(bootstrap): read the hcloud node range from the machine document only d25366a80c 
site-hcloud.yaml now holds a ResolverConfig document too; the unselected
query printed "10.90.0.0/16" and "null" and hcloud network create
rejected the range (run 28).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix(flux): apps with in-house images depend on gitea abc2a1b0a5 
Their images are pulled from Gitea's registry. On a cold cluster they
started before the restored Gitea served it and sat in ImagePullBackOff,
retried on the kubelet's back-off (restore rehearsal runs 27, 28). They now
wait for the gitea Kustomization: apps (ical-proxy), authentik (server,
LDAP outpost, gate-sync), home-assistant (shelly-manager,
mqtt-account-controller), paperless (ftp-scanner), review-bot, trmnl
(renderer), wiki.

No cycle: gitea depends only on restore-gitea and the infrastructure
layers. The infrastructure Kustomization itself also runs in-house images
(bird, netbird-service-sync, secret-rotator, backup-verify, cni-plugins);
it cannot depend on gitea, which depends on it, and is left for a
separate change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix(flux): in-house-image infrastructure waits for Gitea; fallback images don't 36b42677d6 
bird and netbird-service-sync run images from Gitea's registry with no
copy on ghcr.io. In the infrastructure Kustomization they started long
before the restored Gitea served them and sat in ImagePullBackOff (runs 27,
28). infrastructure cannot depend on gitea, which depends on it, so both
move to a new Kustomization, infrastructure-inhouse, that depends on
infrastructure, infrastructure-crds and gitea. Their namespace stays in
infrastructure.

Images published to ghcr.io too have a fallback in the nodes' registry
mirror and need no ordering: apps (ical-proxy: GitHub CI pushes
ghcr.io/konairius/ical-proxy main-<sha>) and trmnl (alpine-jq) lose the
gitea edge from the previous commit; multus cni-plugins stays in
infrastructure. wiki keeps it: no ghcr copy of konairius/wiki was found.

The secret-rotator and backup-verify CronJobs also use Gitea images but
only run on schedule; they are left in place.

Production rollout must be two-phase (moving objects between pruned
Kustomizations): first add infrastructure-inhouse with both paths while
infrastructure still lists them, then remove them from infrastructure.
Applied as one change, infrastructure prunes bird before the new
Kustomization re-creates it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix(flux): monitoring depends on influxdb 37352808c6 
Grafana mounts the grafana-influxdb-token Secret, which a Job in the
influxdb app writes (apps/influxdb/grafana-token-job.yaml). On a cold
cluster monitoring started first; Grafana sat in CreateContainerConfigError
and the monitoring Kustomization held a kustomize-controller worker on its
health check (restore rehearsal run 28). No cycle: influxdb does not use
monitoring's CRDs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix(bird): runs only where the site routes pods into table 47 4ebc00e57a 
bird fills routing table 47, which exists only where the site patch routes
pod traffic into it (site-homelab.yaml RoutingRuleConfig), and renders its
config from node annotations that only that site and the k8s-N patches set
(pod0, egress gateway). On Hetzner the template rendered <no value> and the
init container crash-looped ("syntax error, unexpected '<'", run 28).

site-homelab.yaml labels its nodes net.renner.uno/pod-table=47 and the
DaemonSet selects on it; on a site without table 47 it has no pods.

Production rollout order: the label must be on the nodes (talosctl patch,
no reboot) before the DaemonSet change reconciles, or bird is removed from
every node and pod traffic falls through to the main table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix(flux): evcc depends on home-assistant 0ac74b7b5b 
evcc reads its MQTT client certificate from mqtt-evcc-client-tls
(apps/evcc/deployment.yaml:48-59), which the mqtt-account-controller in
the home-assistant Kustomization delivers into the evcc namespace
(apps/home-assistant/mqtt-accounts.yaml:40-43), and connects to
mosquitto.home-assistant. On a cold cluster evcc started first and sat in
CreateContainerConfigError (restore rehearsal runs 27, 28). No cycle:
home-assistant does not reach evcc.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix(velero): one restore Kustomization per schedule; each app waits for its own a02affc872 
Every app with backed-up volumes depended on one restore Kustomization that
was Ready only after all twelve Velero restores finished. authentik waited
for immich's photo library, home-assistant for paperless; in restore
rehearsal run 28 that was 45 minutes, during which the restore
Kustomization also held one of kustomize-controller's four workers.

kubernetes/restore/<schedule>/ now holds one Restore each, applied by a
restore-<schedule> Kustomization (clusters/homelab/restore.yaml) with the
same health check, after restore-gitea so Gitea's data still goes first.
Each app depends on its own: apps, audiobookshelf (libation through it),
authentik, home-assistant (matter-server, otbr, whisper too, same
namespace and schedule), immich, influxdb, matrix, music-assistant,
navidrome, paperless, trmnl, wordpress. Timeouts follow the data: immich
6h, audiobookshelf 3h, the rest 1h.

Not on production yet: the restore Kustomizations exist only on this
branch, so no Restore objects move between pruned Kustomizations there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix(restore-test): no writes to production NetBird from the test cluster f3ba845e52 
netbird-service-sync in the test cluster reconciled production NetBird
(netbird.renner.uno) with the PAT from the restored OpenBao: it rewrote the
public proxy services to the test cluster's Traefik ClusterIP, and
production's own controller re-created 14 and updated 5 of them every
minute from 09:53 to 10:39 UTC during run 28. The netbird-ingress routing
peers likewise joined the production mesh, announcing the same service and
LoadBalancer ranges as production.

Two safety patches in flux-root.yaml: netbird-service-sync runs with zero
replicas (3i), and netbird-ingress gets a nodeSelector no node carries (3j).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix(mqtt-account-controller): start without the HA client certificate e9ca0c4e90 
The controller mounted mqtt-home-assistant-tls as a required volume, and is
the only creator of that Secret (it applies the cert-manager Certificates of
the register's cert accounts). On a new cluster - Velero restores volumes,
not Secrets - its pod could never start, and mosquitto, Home Assistant,
Zigbee2MQTT and evcc stayed in ContainerCreating behind the Secrets it would
have created.

Optional now. Verified in konairius/mqtt-account-controller main: the cert
dir is read only by the HA cert sync (02:00-05:00 window, errors logged);
the broker is dialled lazily; Secret.Read treats a missing Secret as empty;
Reconcile runs reconcileCerts even when the broker step fails. So the
controller starts, creates the Certificates, the consumers start, and the
next loop reconciles the broker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat(gpu): iGPU workloads run only where a node declares the iGPU 7186e4d035 
llama-server, llama-embed, wiki-embed (gpu.intel.com/i915) and the wiki
behind them are non-essential and bound to the Intel iGPU. On a site
without one (Hetzner, restore rehearsal runs 27/28) their pods stayed
Pending, llama-server failed its health check and the Kustomizations held
kustomize-controller workers; the GPU plugin and exporter DaemonSets sat
on every node with nothing to serve.

The node patches k8s-1..3 label their machines
hardware.renner.uno/intel-gpu=true. The GPU plugin, the exporter, the
three llama Deployments and the wiki select on it, and llama-server and
wiki are no longer waited on. Elsewhere the DaemonSets have no pods and the
Deployments stay unscheduled without blocking anything.

Production rollout order: the label must be on the nodes (talosctl patch,
no reboot) before these manifests reconcile, or the GPU plugin and the
llama/wiki pods are removed or left Pending.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat(talos): hcloud nodes carry ens19 as a dummy link ec8f496d95 
The v30iot NetworkAttachmentDefinitions of matter-server, music-assistant,
otbr, shelly-manager and the mdns relay name master: ens19. On the hcloud
site no such link existed, so macvlan creation failed outright. Each hcloud
node now declares ens19 as a DummyLinkConfig: no address, nothing behind it.

The NADs lease by DHCP, and nothing answers on a dummy link, so the pods get
their macvlan but the dhcp IPAM step still times out.

Validated: talosctl gen config (cluster + site-hcloud + hcloud-1) then
talosctl validate --mode cloud --strict.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix(authentik): its ServiceMonitor lives in monitoring-rules 26976414e4 
The authentik chart renders server/servicemonitor.yaml whenever
server.metrics.serviceMonitor.enabled is set, without checking
.Capabilities for monitoring.coreos.com (verified in chart 2026.8.3). The
authentik Kustomization does not depend on monitoring, so on an empty cluster
the Helm install fails until kube-prometheus-stack has installed the CRDs.

The chart's ServiceMonitor is switched off and declared standalone in
monitoring-rules, as cilium and velero already do. The metrics Service is
gated only on server.metrics.enabled and stays. The selector matches the live
Service authentik-server-metrics (port metrics, 9300).

The new name, authentik-server-metrics, differs from the chart's
authentik-server, so the two never contend for one object while production
rolls over.

Swept every manifest for other chart-rendered monitoring CRs outside the
monitoring Kustomization: cilium and velero already off, blackbox-exporter
renders none (no targets, selfMonitor off).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat(scripts): flux-graph.sh draws the Kustomization dependency graph 09bf50731d 
Reads every kustomize.toolkit.fluxcd.io/v1 Kustomization in
kubernetes/clusters/homelab and emits Graphviz DOT, one edge per dependsOn
entry, pointing from the dependency to its dependent. Each node carries its
wait mode and timeout. Infrastructure is green, the Velero restore chain
blue, wait: false dashed.

With no argument or a .dot path it writes the full graph; with a .svg or .png
path it renders the transitive reduction (tred), which drops edges implied by
a longer path.

Generated rather than checked in, so it cannot drift from the manifests.
Works with either yq (mikefarah's, as in the CI images, or the Python
wrapper): yq only converts YAML to JSON, jq builds the DOT.

Verified: the DOT matches the earlier hand-run yq output byte for byte;
.png and .svg render; shellcheck clean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix(flux): every app Kustomization waits for its workloads; otbr suspended 813822462a 
With wait: false and no healthChecks, Ready means only that the manifests
were applied: dependents are released before the app runs, and a broken app
reports green. kustomize-controller's checkHealth returns before any check in
that case (len(Spec.HealthChecks) == 0 && !Spec.Wait), so healthCheckExprs
are not evaluated either.

- home-assistant: wait: true. The reason given (the USB meter, the canary)
  no longer holds; HA reads the meter over MQTT. evcc, zone-configurator and
  speech-to-phrase depend on it, and evcc needs mqtt-evcc-client-tls from it.
- llama-server, wiki: wait: true, reverting the wait: false of 7186e4d0. On a
  site without the iGPU they fail rather than report Ready.
- wordpress: wait: true. It set no wait, so its MariaDB healthCheckExprs
  were never evaluated.
- otbr: suspended. It is not in use.

infrastructure keeps wait: false with its explicit healthChecks, which is a
health gate on the operators, not the absence of one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix(scripts): flux-graph.sh draws what Flux actually checks abf549bb9a 
A Kustomization with healthChecks and no wait is health-checked (as
infrastructure is), and one with neither has no check at all, whatever its
healthCheckExprs say. Suspended Kustomizations are filled grey and labelled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix(flux): gitea-mcp depends on gitea 0486ea2d99 
gitea-mcp is a front end to Gitea's API (-H https://gitea.renner.uno) and
does nothing without it. Its image is on docker.gitea.com, which is why the
in-house-image sweep did not add this edge. /healthz reports the process
only, so without the edge the Kustomization went Ready with no Gitea behind
it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix(flux): woodpecker depends on gitea 7d9dadccba 
Gitea is woodpecker's forge and its OAuth login provider
(WOODPECKER_GITEA_URL https://gitea.renner.uno); the server has no function
without it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix(flux): trmnl depends on home-assistant, home-assistant on influxdb cae4d83f1b 
- trmnl: the renderer reads HA's API
  (http://home-assistant-bgp.home-assistant.svc.cluster.local:8123).
- home-assistant: its InfluxDB integration writes to influxdb
  (influxdb_token in HA's secrets.yaml).

No cycle: influxdb depends only on its restore and infrastructure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix(woodpecker): recover wp-pg from the prefix it archives to 43dfa6158c 
wp-pg-store-source, the recovery source of wp-pg's bootstrap.recovery,
pointed at s3://renner-uno-k8s-cnpg/wp-pg: the prefix of the retired
apps/wp-pg. Since the namespace move wp-pg archives to wp-pg-woodpecker
(wp-pg-store). A cold rebuild would therefore have restored woodpecker as of
2026-08-18 and reported success.

Bucket, listed read-only 2026-10-16:
- wp-pg/wp-pg/base: newest 20260818T023502
- wp-pg-woodpecker/wp-pg/base: 20260916T024000 .. 20261016T024000 (daily),
  wals/ through timeline 6 (00000006.history from 2026-09-23)

The source now reads wp-pg-woodpecker; serverName stays wp-pg, the name the
live cluster archives under. bootstrap.recovery is honoured only when PGDATA
is absent, so the running cluster is unaffected. The old prefix wp-pg is no
longer referenced and keeps its objects; nothing applies retention to it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat(cnpg): every cluster incarnation archives to a new prefix; recovery reads the newest cffb7e027a 
A CNPG cluster restored from barman archived back into the location it
recovered from. plugin-barman-cloud's restore job checks the archive target
(plugins[] ObjectStore + serverName) and stops with "Expected empty archive"
when it holds WAL, which after a disaster it always does. It has no
exception for target == recovery source (internal/cnpgi/restore/restore.go,
v0.15). Run 28 hit it on wp-pg; the other six never got that far.

The check stays. Instead, archive and source become two ObjectStores whose
paths are chosen per incarnation:

- kubernetes/cnpg-object-stores/: all 14 ObjectStores (<x>-store archives,
  <x>-store-source is read by bootstrap.recovery), in a Kustomization of their
  own with postBuild.substituteFrom: cnpg-paths. Nothing else lives there:
  substitution blanks every ${...}, which is why it is not on the app
  Kustomizations (home-assistant's scripts, the Grafana dashboards). The
  ConfigMap is not optional; without it the paths would render empty.
- bootstrap.sh writes cnpg-paths before Flux starts, once per incarnation
  (an existing one is kept):
    CNPG_<X>_SOURCE   the prefix among <x>, <x>-g<14 digits> and the store's
                      cnpg.renner.uno/source-prefixes whose <x>/base/ holds
                      the newest base backup
    CNPG_<X>_ARCHIVE  <x>-g<UTC timestamp>, new and so empty
  Every ScheduledBackup has immediate: true, so a new incarnation has a base
  backup in its own prefix minutes after it starts, and a later recovery
  picks it. S3 access is curl --aws-sigv4 with the restore key (already in
  bootstrap.sh's secret list), so the CI image needs no new tool.
- The restore test archives to renner-uno-k8s-cnpg-rtest, created by the
  flux phase and emptied and deleted by hcloudclean. A test incarnation with a
  base backup in the production bucket would be its newest, and the next real
  recovery would read the test's data.
- The six clusters that recovered from their archive store now name
  <x>-store-source in externalClusters. serverName stays the cluster name
  everywhere. The app Kustomizations depend on cnpg-object-stores, which
  depends on infrastructure-cnpg-barman.

Verified against the bucket, read-only, 2026-10-16:
- the restore key lists renner-uno-k8s-cnpg and can create and delete a
  bucket (renner-uno-k8s-cnpg-rtest created and deleted again);
- the listing parser on real responses, top level and an encoded nested
  prefix; for wp-pg it picks wp-pg-woodpecker (newest 20261016T024000) over
  wp-pg (20260818T023502);
- leftover test serverNames (ak-pg-rtest, im-pg-rtest, ha-pg-restore-test,
  pprleval-pg) hold no base backups and match no candidate pattern.
shellcheck clean; all seven apps and cnpg-object-stores build; the render
holds exactly the 14 expected variables.

Production rollout, one time:
1. Create flux-system/cnpg-paths with the current prefixes as both source and
   archive (ak-pg, ha-pg, im-pg, mx-pg, pprl-pg, tr-pg; wp-pg-woodpecker), so
   the running clusters keep archiving where they do.
2. Move the ObjectStores in two phases (add in cnpg-object-stores, then drop
   from the apps). All 14 now carry prune: disabled; before this only ak, im
   and pprl did.
3. Whether changing externalClusters on a live Cluster restarts its pods is
   not verified; check on one cluster first.

Older generations are not cleaned up: retention applies only to the current
archive prefix. A cleanup script is planned separately.

RECOVERY.md: the table describes the new layout; the manual restore drill
reads <x>-store-source. That drill wrote 3 WAL files to ak-pg/ak-pg-rtest on
2026-07-25 despite isWALArchiver: false, so its "only ever reads" claim
does not hold; not changed here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat(iot): v30iot workloads run only where a node declares the IoT VLAN; HA no longer waits on them 420bd35e7f 
Five workloads attach a macvlan on ens19 (v30iot) with DHCP IPAM:
shelly-manager and the mDNS relay (inside the home-assistant Kustomization),
matter-server, music-assistant and otbr. On a node without that VLAN the CNI
add fails and the pod never starts.

- k8s-1..3 carry net.renner.uno/iot-vlan: "true" beside the GPU label; each
  of them declares ens19 in its VLANConfig. The five Deployments select on
  it. The comments they replace described the retired k8s-edge taint.
- shelly-manager and the mDNS relay move to apps/home-assistant-iot, with a
  Kustomization of their own (depends on restore-home-assistant, gitea,
  home-assistant and the infrastructure Kustomizations). home-assistant's
  wait: true no longer covers them, so HA and everything waiting on it
  (evcc, trmnl, zone-configurator, speech-to-phrase) are Ready without the
  VLAN, and a broken IoT bridge no longer holds them back in production
  either.

On a site without the label the IoT Kustomizations fail at their timeout,
naming the Pending pods; nothing else depends on them.

Verified: home-assistant, home-assistant-iot, matter-server,
music-assistant, otbr, authentik and monitoring-rules build; the
home-assistant render has no IoT attachment left; every IoT workload carries
the selector; k8s-1 with site-homelab passes talosctl validate --strict.

Production rollout:
1. Label the nodes first (talosctl patch the nodeLabels), or the five pods go
   Pending.
2. The move between Kustomizations is two-phase (add under
   home-assistant-iot, then drop from home-assistant). shelly-manager's PVC
   already carries prune: disabled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Revert "feat(talos): hcloud nodes carry ens19 as a dummy link" 4befc7c492 
This reverts commit ec8f496d. The v30iot workloads now select
net.renner.uno/iot-vlan, which no hcloud node carries, so nothing on that
site attaches to ens19 any more.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix(home-assistant): 30m timeout, not 4h 9dc3f5b4a0 
The 4h was sized for a seed initContainer that waited up to 3h for an HAOS
backup to appear in Garage, then downloaded and staged it. That step no
longer exists: Velero restores the config PVC in restore-home-assistant
before this Kustomization runs, and Garage is retired. The HA container's
startupProbe allows 10 minutes.

The slowest part left is ha-pg recovering from barman: its newest base backup
(ha-pg/ha-pg/base/20261016T025000) is 1.7 GiB compressed, then WAL replay and
the second instance's clone. 30m, as the other CNPG apps have; the next
restore test measures it.

With wait: true, the 4h also held one of kustomize-controller's workers that
long on a failed cold start, and with it evcc, trmnl, zone-configurator and
speech-to-phrase.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix(flux): renovate depends on gitea 12a0f8c000 
Renovate's platform is Gitea: endpoint https://git.renner.uno/api/v1 in
apps/renovate/configmap.yaml, the same instance as gitea.renner.uno. It has
no function without it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix(flux): services that need OpenBao secrets apply after the secret store 9b488b037e 
crowdsec, external-dns-unifi, netbird-ingress and velero were in the
infrastructure Kustomization, while the ExternalSecrets that create their
Secrets were in infrastructure-crds, which depends on infrastructure. On a
cold start each was therefore installed before its Secret could exist: the
pods sat in CreateContainerConfigError until OpenBao was unsealed and ESO
synced. For the HelmReleases (crowdsec, external-dns-unifi, velero) Helm's
wait timed out meanwhile; with remediation.retries: 3 they give up after
four attempts and stay failed even once the Secret appears. Velero failing
that way stops every restore.

| workload | Secrets |
|---|---|
| crowdsec | crowdsec-bouncer-key (crowdsec, ingress), crowdsec-edge-bouncer-key |
| external-dns-unifi | external-dns-unifi-secret |
| netbird-ingress | netbird-ingress-setup-key |
| velero (+ backup-verify CronJob) | cloud-credentials, cloud-credentials-hetzner, hcloud-backup-verify |

Each now has a Kustomization of its own (clusters/homelab/
infrastructure-services.yaml), depending on infrastructure and
infrastructure-crds, with its ExternalSecrets beside it. The openbao
ClusterSecretStore in infrastructure-crds turns Ready only once OpenBao is
unsealed (a5c5d0e7), so the Secret exists before the consumer installs. One
Kustomization each so that a broken service holds back only itself.

Velero's Schedules need the velero.io CRDs from its HelmRelease, so they move
from post-crds to infrastructure/velero-schedules with a Kustomization after
velero. velero-backups-synced depends on velero (the BackupStorageLocation
and its credentials) instead of infrastructure-crds.

post-crds keeps cloudflare-api-token (consumed by its own ClusterIssuer) and
flux-git. The directories themselves stay where they are, so paths such as
the CI test infrastructure/velero/backup-verify/tests/run.sh are unchanged.

Verified: infrastructure, post-crds and the six service paths build;
infrastructure renders no ExternalSecret.

Production rollout: moving objects out of the pruned infrastructure and
infrastructure-crds Kustomizations is two-phase (add the new Kustomizations
first, then drop the old entries). Pruning the velero HelmRelease would
uninstall Velero.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix(restore-test): CNPG archives, backs up and prunes as in production 0933ccdf7b 
Patches 3c (isWALArchiver: false), 3d (no retentionPolicy) and 3e
(ScheduledBackups suspended) kept the test cluster from writing into the
production CNPG bucket. Since cffb7e02 it cannot: bootstrap.sh writes
cnpg-paths with every archive path in renner-uno-k8s-cnpg-rtest, the
cnpg-object-stores Kustomization fails rather than render without it, and
the source stores, the only ones in the production bucket, carry no
retentionPolicy.

The patches now only stopped the test from exercising what a real recovery
does next: archiving WAL into a new prefix and taking the immediate base
backup there. Removing them leaves three fewer deviations from production.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix(monitoring-rules): drop the radius metrics Service; authentik's operator creates it afdeecc161 
authentik-outpost-servicemonitors.yaml hand-wrote a headless Service
ak-outpost-local-radius-metrics (9300/TCP) and a ServiceMonitor for it,
because the radius outpost controller used not to expose its metrics port.
authentik 2026.8.3 does: the operator creates that same Service
(http-metrics 9300, managed-by goauthentik.io, service-type metrics) and its
own ServiceMonitor ak-outpost-local-radius selecting exactly those labels.

Restore test run 29 (2026-10-16): on a cold cluster the operator created the
Service first, with a ClusterIP. clusterIP is immutable, so the dry-run of
the hand-written clusterIP: None copy failed and took the whole
monitoring-rules Kustomization with it:

  Service/authentik/ak-outpost-local-radius-metrics dry-run failed (Invalid):
  spec.clusterIPs[0]: Invalid value: ["None"]: may not change once set

In production Flux had created it first; the operator adopted it, and Flux's
label managed-by: homelab now keeps the operator's ServiceMonitor from
matching. Removing the file lets the operator own both; when Flux prunes the
Service, the operator recreates it with its own labels.

No alert rule refers to the radius job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix(home-assistant): shelly-manager-auth stays with mqtt-account-controller 20b0621e2e 
420bd35e moved shelly-manager-auth-externalsecret.yaml to home-assistant-iot
with the rest of shelly-manager. mqtt-account-controller, which stays in
home-assistant, mounts that Secret as well (volume shelly-manager-key, item
api-key, not optional). That made a cycle:

- mqtt-account-controller does not start without shelly-manager-auth;
- it creates the MQTT client certificates HA, mosquitto and zigbee2mqtt
  mount (mqtt-home-assistant-tls, mqtt-mosquitto-probe-tls,
  mqtt-zigbee2mqtt-bridge-tls), so they do not start either;
- home-assistant therefore never turns Ready, and home-assistant-iot, which
  would create shelly-manager-auth, depends on it.

Restore test run 29 (2026-10-16) sat in it: every other Secret in the
namespace existed, mqtt-account-controller reported
  MountVolume.SetUp failed for volume "shelly-manager-key" :
  secret "shelly-manager-auth" not found
and evcc, trmnl, zone-configurator, speech-to-phrase and home-assistant-iot
waited behind it.

The ExternalSecret goes back to home-assistant; shelly-manager in
home-assistant-iot runs after it and finds the Secret. Every Secret the
home-assistant render mounts now has its producer in the same Kustomization
(mqtt-dynsec-seed is optional).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merge master into worktree-restore-test-real
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
Details
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
Details
review-bot review failed: context deadline exceeded
9c35f95fb3 
Conflict in kubernetes/apps/woodpecker/objectstore.yaml: master (#1295)
pointed wp-pg-store-source at wp-pg-woodpecker; this branch moved both
woodpecker ObjectStores to kubernetes/cnpg-object-stores/wp-pg.yaml, whose
source path comes from cnpg-paths and covers wp-pg-woodpecker through
cnpg.renner.uno/source-prefixes. Resolved by keeping the deletion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
claude-bot changed title from WIP: restore test fixes — site facts, self-restore, CNPG archive generations, dependency graph to Restore test fixes: site facts, self-restore, CNPG archive generations, dependency graph 2026-10-17 08:32:08 +00:00
claude-bot commented 2026-10-17 09:52:23 +00:00
Author
Member
Copy Link
Copy Source

Too large to review (the review bot runs out of context). Split into 9 stacked PRs, each based on the previous one, to merge in order: #1297, #1298, #1299, #1300, #1301, #1302, #1303, #1304, #1305. Together their tree is identical to this branch (9c35f95f).

Too large to review (the review bot runs out of context). Split into 9 stacked PRs, each based on the previous one, to merge in order: #1297, #1298, #1299, #1300, #1301, #1302, #1303, #1304, #1305. Together their tree is identical to this branch (9c35f95f).
claude-bot closed this pull request 2026-10-17 09:52:24 +00:00

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#1296