Root cause of #393, confirmed. Three things combine:
1. The public renner.uno zone has a *.renner.uno CNAME to proxy.renner.uno
that matches at EVERY depth, so renner.uno never returns NXDOMAIN:
definitely-not-real-xyz123.iot.renner.uno -> proxy.renner.uno -> 45.157.177.26
Any name that fails locally resolves publicly to the WAN proxy instead of
failing honestly.
2. TTL asymmetry. The correct answer is a dnsmasq DHCP-lease record with
TTL 5 (verified: `dig @10.2.50.1 shellyplusi4-....iot.renner.uno` returns
`aa` + TTL 5). The wrong wildcard answer has TTL 300 — 60x more durable.
So one 5-second failure pins a wrong answer for 5 minutes. That is exactly
the "correct at 14:10, wildcard-only at 15:45" report in the issue.
3. The UDM's dnsmasq refuses connections in bursts — 330 "connection refused"
errors in the last 3000 k8s-gateway log lines, on BOTH 10.2.40.1 and
fd00::1.
Note the issue's premise was wrong: there is no "pinning job". The device
records are ephemeral DHCP-lease records in dnsmasq, not IaC. external-dns-unifi
is also NOT the culprit despite running --policy=sync: it writes the UDM's
static-DNS list, not the lease table, and --registry=txt confines deletion to
records carrying its own TXT.
Fix: raise the cache floor and serve through upstream outages.
success ... 60 clamps the 5s TTL up to a 60s floor
serve_stale 1h immediate keeps answering while the upstream refuses
This also fixes the zigbee2mqtt crash-loop, which has the same cause — z2m
exits on `getaddrinfo EAI_AGAIN slzb-zigbee.renner.uno` with no retry, and the
k8s-gateway log shows that exact name failing against both UDM addresses.
Validated before committing, because CoreDNS is cluster-critical and the
`template` plugin previously broke cluster DNS here (7a09bad5):
- The repo's full Corefile runs under the real coredns:v1.13.2 image (parses;
only the expected out-of-cluster kubernetes-plugin error).
- The repo's k8s-gateway values render via `helm template` (rc=0) and the
resulting Corefile starts under the real k8s_gateway:1.7.0 image
(CoreDNS 1.14.2).
- The cache parameters MUST stay on one line: a multi-line value breaks the
chart's ConfigMap render with "could not find expected ':'". Confirmed by
hitting it.
- The forward block is deliberately untouched — its options require newlines
and cannot be expressed through the chart's single-line parameters.
- The `template` plugin is NOT reintroduced.
Known not fixed: AAAA for lease-backed names is deterministically shadowed
(dnsmasq has an A from the lease but no AAAA, so AAAA forwards upstream and
hits the wildcard every time). That needs a NODATA response for those names
and is a separate, tightly-scoped change — filing separately.
Refs #393
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There are zero ceph_* series in Prometheus, and there have been for all 79
days of the cluster's life:
count of ceph_ metrics in Prometheus: 0
rook-ceph active scrape targets: 0
Not a Rook misconfiguration — `monitoring.enabled: true` is set and both
ServiceMonitors exist. They are simply never selected:
Prometheus serviceMonitorSelector : matchLabels {release: kube-prometheus-stack}
rook-ceph-mgr / rook-ceph-exporter: labels {team: rook}
The Rook operator creates those ServiceMonitors at runtime with its own
hardcoded label, so no static manifest or Kustomize patch in this repo can
relabel them.
Cluster-wide this was dropping 12 of 30 ServiceMonitors:
rook-ceph/rook-ceph-mgr, rook-ceph/rook-ceph-exporter (team: rook)
authentik/ak-outpost-* x10 (goauthentik.io/...)
Consequences: the entire storage layer had no telemetry and no history — so
Ceph trend analysis is impossible for anything before this lands, and any Ceph
PrometheusRule added today would have silently had nothing to fire on.
This is the ServiceMonitor twin of the trap already documented for rules in
monitoring-rules/prometheusrule-velero.yaml ("without this label the rule is
created but never loaded into Prometheus").
ruleSelector is deliberately NOT loosened — this repo's PrometheusRules all
carry the release label on purpose, and keeping that selector strict stops a
stray third-party rule from firing here.
Blast radius: 12 new scrape targets. The 10 authentik outposts come along with
the 2 rook ones; they are legitimate targets, but they are a side effect of
this change rather than its goal.
Found while investigating the Ceph BLUESTORE_SLOW_OP_ALERT warning, which
itself turned out to be a benign self-limiting event.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The two v6 addresses in hairpin_endpoints were attributed to the opposite
proxy members. DNS is the operational truth, verified independently:
getent ahosts proxy-1.renner.uno -> 2a03:4000:28:f6e:a4c7:59ff:fe1e:9a29
(v6-only member, single AAAA)
getent ahosts proxy-2.renner.uno -> 2a03:4000:4b:c1e:4474:43ff:fedf:8f85
45.157.177.26 (dual-stack member)
Purely a labelling fix with zero functional impact: both entries DNAT to the
same target (fd99::86), so the hairpin behaved identically before and after.
Slots are untouched — the file's "never reuse a freed slot" rule applies to
the keys, not the values.
Also trims the now-stale NOTE in probes-blackbox.yaml that existed only to warn
readers away from the wrong labels; the two files now agree.
Surfaced while triaging #392/#420, both of which turned out to be already
fixed. `tofu validate` passes. (Note: `tofu fmt` reports pre-existing
alignment drift further down this file, at the jsonencode block around line
108 — present on master before this change and deliberately left alone.)
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The velero e2e check manufactured phantom PartiallyFailed Backups that then
polluted backup-health triage. Sequence, from Loki for e2e-1785031248:
02:01:05 Backup + PVC created (DataUpload -> phase Prepared)
... data path never starts (node-agent slot held elsewhere)
~02:06 the 60x5s = 300s wait expires -> check FAIL, but does NOT abort
-> script deletes ns velero-e2e and creates the Restore
02:07:03 restore error "no DataUpload result cm found" -> PartiallyFailed
02:10:08 script deletes Backup+Restore -> "Cancel du under phase Prepared
because it is being deleted" -> Backup lands PartiallyFailed
Both 2026-07-26 occurrences (02:00:55 and 12:04:05) fall inside a running
restore-test sweep. node-agent-config.yaml sets no loadConcurrency, so Velero's
default of one data path per node applies and the sweep's data-mover pods hold
the slot — a 1 GiB e2e DataUpload then sits in Prepared well past 5 minutes.
The backup was never actually faulty; the test cancelled it.
Three changes:
- Wait 120x15s (30 min) instead of 60x5s (5 min), so a queued DataUpload has
time to run once the slot frees.
- Guard the destructive half (namespace teardown + restore roundtrip) behind
whether the backup check actually passed. Tearing down while the upload is
in flight is what cancels it.
- Do not delete the Backup on the failure path either — that is the same race
relocated to cleanup. The Backup carries ttl: 2h0m0s, so Velero's GC reaps
it regardless and leaving it costs nothing.
Cleanup of the restore, namespace and released PVs stays unconditional, so
nothing leaks. Verified with `bash -n`.
Found while triaging #427; the restore-test failure there was unrelated and
is already fixed.
Refs #427
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The alert triage agents could reach Prometheus and Alertmanager through the
apiserver service proxy, but not Loki:
kubectl auth can-i get "services/loki:3100" --subresource=proxy -n monitoring \
--as=system:serviceaccount:monitoring:ci-diagnostics -> no
(same check for kube-prometheus-stack-prometheus:9090 -> yes)
That gap has a sharp edge. A KubeJobFailed-class alert can only be cleared by
DELETING the failed Job — the rule's own annotation instructs this — and the
deletion destroys the pod and its logs. So the agents would inspect a failed
Job, find no live pod, declare the root cause "unconfirmable", and then delete
the last copy of the evidence.
The evidence was never actually gone: Alloy ships all pod logs to Loki with
retention_period: 2160h (90d). Issue #380 declared velero/restore-test-29740440
"probably unconfirmable" and its full failure output was in Loki the whole time
(FAIL: 3 — apps restore phase + two PVC parity checks).
Adds `loki` / `loki:3100` to the existing service-proxy Role (read-only GET,
same bare + :port-qualified pattern the other entries use, verified against the
live Service which exposes http-metrics:3100), and teaches both agent prompts
the query recipe so "the pod is gone" stops being accepted as a dead end.
No new privilege class: the agents already hold pods/log cluster-wide. This
mainly extends their reach to *deleted* pods.
Refs #427, #380
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The `>=1.0.0 <2.0.0` ceiling had reached its practical maximum: chart 1.3.0
(app v1.3.0) is the last 1.x ever published, so the bound was holding the
cluster on a dead line while upstream had moved on to app v1.4.19.
Verified before bumping:
- The only breaking change at the 1.x->2.0.0 boundary is the relocation of
`deployment.image.*` to top-level `image.*`. This HelmRelease sets
`values: {}`, so neither key is in play and the bump is a drop-in.
- `helm template stakater/reloader --version 2.2.14` renders cleanly with
default values (rc=0, image v1.4.19).
- ClusterRole rules are byte-identical between 1.3.0 and 2.2.14 — the
cluster-wide privileges of this restart-authority controller do not change.
2.x additionally ships a namespace-scoped Role (configmaps
list/get/watch/create/update) used for leader election. Additive only.
- `kubectl kustomize kubernetes/infrastructure` renders 199 resources with the
HelmRelease resolving to the new constraint.
Floored at 2.2.0 rather than 2.0.0 to skip the early 2.0/2.1 line outright.
Resolves to 2.2.14 today.
Blast radius on upgrade: reloader itself rolls; the 7 annotated workloads
(apps/external-dns, apps/woodpecker-agent, apps/woodpecker-server,
home-assistant/home-assistant, home-assistant/shelly-manager,
monitoring/pve-exporter, paperless/paperless) are not restarted by a
controller upgrade alone.
Refs #426
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Follow-up to #416 (ceph-block-ephemeral) for restore-test.sh's own PV leak
(incident 2026-05-17, ~2.7 TB of orphan CephFS subvolumes).
The intent was to retire purge_released_pvs entirely. That is not safely
possible, and the reason is worth recording. Three categories of throwaway
PVC exist per iteration; only one is ours to pick a StorageClass for:
1. source snapshot-probe PVCs (rt-src-*) — this script creates them.
FIXED HERE: the CephFS branch of sc_for_snapshot_probe returned
`ceph-filesystem` (Retain), so every probe leaked a Released PV. It
now returns `ceph-filesystem-ephemeral` (Delete). The two classes are
identical apart from reclaimPolicy — same provisioner, parameters and
mountOptions — so the snapshot restores into it unchanged.
The ceph-block branch needed no change and must not get one: it
already uses ceph-block-snapshot-readonly, which is Delete, and whose
`noload` mountOption is load-bearing (skips ext4 journal replay on the
RO snapshot mount). ceph-block-ephemeral lacks noload, so pointing the
probe at it would reintroduce the dirty-journal mount failure.
2. destination PVCs in *-restore-test namespaces — Velero creates them
from the backed-up PVC spec, so the class is whatever production uses
(Retain). The only remap knob is the change-storage-class
RestoreItemAction ConfigMap, which Velero applies to EVERY restore in
the installation with no per-restore scoping. Using it would strip
Retain from a real disaster-recovery restore. Rejected.
3. Velero data-mover staging PVCs in velero/ (*-rtest-*) — derived from
the target PVC. The node-agent restorePVC config supports only
`ignoreDelayBinding`; unlike backupPVC there is no storageClass field
on Velero v1.18.
So 2 and 3 keep purge_released_pvs load-bearing; it stays, now documented.
Also fixes a real residual leak the ephemeral class does not address:
cleanup_service's purge is scoped to ${RUN_ID}, which is right during a run
but means a run that dies before cleanup (activeDeadlineSeconds, OOM, node
eviction, killed Job) strands its PVs forever — no later run's pattern ever
matches an older RUN_ID. Adds an unscoped sweep at run start. Safe because
concurrencyPolicy Forbid guarantees no other restore-test Job is active, and
it only ever touches Released PVs whose claimRef matches restore-test-only
naming patterns.
Verified against the live cluster: the sweep pattern matches zero PVs today,
and in particular does not match the three intentional Retain safety PVs
(crowdsec/crowdsec-db-pvc, openbao/data-openbao-0,
home-assistant/mosquitto-data). Categories 2 and 3 were confirmed to
provision on Retain classes by observing an in-flight run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- No TOFU: the CronJob writes a pinned known_hosts (ed25519 key captured
via ssh-keyscan 2026-07-26 and cross-checked against the workstation's
trusted entry) and uses StrictHostKeyChecking=yes. A reprovisioned VPS
makes the job fail loudly — desired.
- The dump now contains ONLY store.db: /opt/netbird/config is fully
regenerable from the role's templates + vault, and including it put
the plaintext TURN secret into every nightly tarball. Restore is
role-run (rebuilds config) + untar into /opt/netbird/management/.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The restart could in principle deconfigure the interface and cut the
SSH connection Ansible rides on (dhcpcd default without persistent).
Verified live: persistent is in fact already set on both members (line
19), so the risk was theoretical here — but 'dhcpcd -n' (reload config
+ rebind, verified supported) achieves the same effect with no
deconfigure window at all, and the role now asserts persistent
explicitly instead of relying on it being there.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Addresses the automated review on #418.
Closes the review's open question about whether the -prune/-path/paren
grouping behaves the same in the probe pod's own busybox build. Rather
than leaving it as a "watch the first real run" note, it was tested
directly: an alpine:3.21 pod (BusyBox v1.37.0) with the same
securityContext and read-only mount as count_files_in_pvc's probe,
against the live audiobookshelf-metadata volume.
unfiltered: 2339
pruned streams + tmp: 775 == 2339 - 1531 - 33
Two robustness properties also confirmed in that image:
- pruning a path that does not exist on the volume is a no-op with
exit 0, so a stale entry degrades gracefully instead of zeroing the
count and faking a catastrophic-loss FAIL;
- each prune subtracts exactly its own subtree (tmp alone -> 2306).
Also renames the local `expr` to `find_expr` so it cannot be misread as
the expr(1) command by someone skimming for invocations. No behaviour
change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- The blocking catch: the openbao ClusterSecretStore's namespace
conditions did not include the new netbird-backup namespace, so the
ExternalSecret could never sync even after key provisioning. Added
(sorted, with consumer comment).
- rm -f /backup/*.part up front — aborted-run leftovers are outside the
retention glob.
- storageClassName: ceph-block explicit, matching the repo's other
single-PVC manifests.
Keeping the key on root with restrict,command= rather than a dedicated
VPS user: the forced command is the actual privilege boundary (the key
can only ever produce a dump), and a separate user would need read
access to root-owned live SQLite + config either via group surgery on
/opt/netbird or sudo rules — more moving parts for the same effective
capability.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The review flagged the v6 target as 'proxy-2 per unifi_nat_hairpin.tf'.
The tf label keys are the swapped ones: DNS (dig) and the live hosts
(the v6-only member with the nameserver-less dhcpcd lease answers as
proxy-1.renner.uno = 2a03:…:28:f6e…; the dual-stack member answers as
proxy-2.renner.uno = 45.157.177.26 + 2a03:…:4b:c1e…) both agree with
the probe's original attribution, as does the netbird-proxy role's
'proxy-1 is v6-only' doc. Keep DNS attribution, document the tf
mislabel (tracked for a separate relabel — renaming those tf map keys
churns NAT rule resources, so it doesn't belong in this PR), and:
- add proxy-2's v6 address as a third target (previously its v6 path
was unprobed; verified live from a pod, returns 200)
- pin job="blackbox-http" on NetbirdControllerUnreachable for
consistency with the other rules
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- insertbefore: BOF so nohook stays global if an interface stanza ever
appears (none exist today, verified live on both members).
- dhcpcd restart is now an inline when-changed task instead of a
handler: nohook must take effect before a lease renewal can re-clobber
the pinned file, and meta: flush_handlers would also fire the
proxy-recreate handler before the compose file exists on a fresh host.
Unit name verified as plain dhcpcd.service on both members.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 2026-07-26 restore-test run failed 1 of 40 checks:
audiobookshelf: PVC audiobookshelf-metadata file count parity (±10%)
-- source=2339 restored=803
This was a false positive. The backup was complete: backup
audiobookshelf-daily-20260725034518 is phase=Completed with no errors or
warnings, and all four of its DataUploads finished with bytesDone ==
totalBytes.
The parity check compares a backup taken up to ~24 h ago against the live
volume *now*, counting files with an unfiltered `find`. Between the backup
(07-25 03:45) and the source probe (07-26 02:12), one Audiobookshelf
playback session wrote 1531 HLS transcode files (~153 MiB) into
metadata/streams/. Excluding that directory: 2339 - 1531 = 808 persistent
files vs 803 restored, a 0.6% delta — comfortably inside the tolerance.
The next day's metadata DataUpload was +157 MB, independently confirming
the stream segments were the only thing that changed.
Fixes:
1. Add volatile_paths_for(), a per-"<namespace>/<pvc>" list of transient
subtrees pruned from BOTH the source and restored counts. Only
streams/ and tmp/ are excluded for audiobookshelf-metadata; cache/ is
left counted deliberately, since it is stable enough not to threaten
the tolerance and dropping it would cost real coverage. Every entry is
a blind spot, so the list is kept minimal. Volumes with no entry keep
the exact previous behaviour.
2. Move the CronJob from Sunday 02:00 to Sunday 08:00 UTC. The old slot
was picked as "well before Monday's daily backups", which missed that
the daily schedules also fire on Sunday — a ~5 h sweep starting at
02:00 sat on top of the whole 03:15-07:15 backup window and starved it
of node-agent DataUpload concurrency. audiobookshelf-daily took 2 h
18 m (07-19) and 2 h 33 m (07-26) against a ~1 min baseline, with the
books DataUpload not starting until 06:18. Both Sundays; no other day
shows it.
The pruning find expression was verified against the live
audiobookshelf-metadata volume (busybox find, same as the alpine probe
image): it returns 775, matching 2339 - 1531 - 33.
Refs #380.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- e2e-test.sh: note that purge_released_pvs '^default/e2e-pvc$' is now a
one-time safety net (e2e-pvc uses ceph-block-ephemeral, so it normally
no-ops) rather than dead weight.
- RECOVERY.md: explain why Delete reclaim is safe *here* specifically — this is
the throwaway restore-test, never a real DR target (a real cold-rebuild uses
the live cluster's Retain ceph-block storage).
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes the last big SPOF-assessment item: the management SQLite store
(peers, setup keys, groups, routes) had zero backup coverage — losing
the Hetzner VPS meant re-enrolling the entire mesh. The VPS cannot
reach any PBS (both PBS names resolve to the public proxy, ports
closed), so the cluster pulls instead of the VPS pushing:
- VPS (netbird-server role): a dump script takes a CONSISTENT snapshot
via sqlite3 .backup (online-backup API, safe against the live writer)
plus /opt/netbird/config and streams a tar.gz to stdout. A dedicated
ed25519 key is authorized for root restricted to exactly that script
(restrict,command=…) — the puller can run nothing else.
- Cluster (kubernetes/infrastructure/netbird-backup): nightly CronJob
(02:45) ssh-pulls the dump onto a 5Gi Ceph PVC, gzip-verifies before
renaming, keeps 14 dumps; ExternalSecret syncs the private key from
OpenBao (homelab/infrastructure/netbird-backup, one-time manual
provisioning like the rotator bootstraps); restricted-profile pod.
- Offsite: new Velero Schedule netbird-backup-daily (07:45, PVC-only,
snapshotMoveData, 30 d TTL) ships the PVC to Hetzner Object Storage.
- Alerts: NetbirdVpsBackupStale (critical, >26 h without success) and
NetbirdVpsBackupNeverRan (warning — catches the unprovisioned-secret
state so the backup cannot die silently).
Restore: untar over /opt/netbird/{management,config} after the role
rebuilds the stack.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Root cause of the daily RBD-image leak (found while reclaiming 15 orphaned
images on 2026-07-26): throwaway block PVCs used the default ceph-block
StorageClass, which is reclaimPolicy: Retain. On PVC deletion the PV goes
Released (image retained), and the ad-hoc reclaim (patch reclaimPolicy=Delete ->
delete pv) races the ceph-csi provisioner — if it acts on the stale Retain
policy it removes the PV but orphans the RBD image. Same class of bug as #262,
which was already fixed for CephFS via ceph-filesystem-ephemeral but never for
RBD.
- Add ceph-block-ephemeral (reclaimPolicy: Delete), the RBD mirror of the
existing ceph-filesystem-ephemeral, in rook-ceph-cluster/storageclass.yaml.
- e2e-test.sh: point the throwaway default/e2e-pvc at it (this was the ~1
image/day source via the daily e2e crons).
- RECOVERY.md: point the CNPG restore-test drill's throwaway cluster at it and
simplify the teardown — with Delete reclaim, deleting the cluster removes the
PVC -> PV -> RBD image automatically, so the manual PV-reclaim dance is gone.
Residual (separate): the velero restore-test (Sundays) restores PVCs with the
source's storageClass (ceph-block, Retain) and can't be repointed the same way;
it still relies on its purge_released_pvs helper.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes the vantage-point gap that let the 2026-07-24 controller outage
run ~20 h unnoticed: every existing probe of the public names resolves
in-cluster (CoreDNS/k8s_gateway) or gets hairpinned, so nothing measured
the actual WAN path.
- blackbox module http_2xx_wan_edge: probes the edge by proxy IP
literal with SNI/Host pinned to home.renner.uno. The Server/Ingress
VLANs are deliberately absent from the UniFi hairpin, so IP-literal
probes from the in-cluster exporter genuinely leave via the WAN and
come back through the netcup edge (verified live from a pod: v4
returns 200; v6 currently fails against proxy-1 — real breakage,
fixed separately by the resolv.conf PR).
- New probes: WAN edge per member/family (proxy-2 v4, proxy-1 v6),
https://netbird.renner.uno (controller), relay :9000/health on both
netcup members, tcp :33080 relay listeners.
- New PrometheusRule netbird-availability: controller unreachable
(critical 10m), whole edge down from WAN (critical 5m), single edge
member down (warning 15m), relay member down (warning 15m).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
dhcpcd's resolv.conf hook overwrites the systemd-resolved stub file that
/etc/resolv.conf symlinks to. On v6-only proxy-1 the lease carries no
resolver options, leaving the file with zero nameserver lines; host
tools keep working via nss-resolve, but the host-network proxy container
copies that file at create time and Go's resolver falls back to
localhost:53. Result: proxy-1's reverse-proxy has been unable to reach
management since the host-network switch — no cert mapping, TLS
'unknown domain "home.renner.uno"' for v6 edge clients that resolve to
its address. Bridge mode had masked the broken file via Docker's
embedded DNS; proxy-2 only survived because its v4 lease happened to
write usable nameservers into the same clobbered file.
Fix on both members: nohook resolv.conf for dhcpcd (+ restart handler so
it takes effect immediately) and a static /etc/resolv.conf pointing at
the resolved stub (127.0.0.53, healthy netcup upstreams on both hosts),
then force-recreate the proxy so it picks up the fixed file.
Found by the WAN-path edge probe work: the v4 probe returned 200 while
the v6 probe failed TLS against proxy-1.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The seed->apps ownership handoffs are complete, so the prune: disabled comments
on the folded CNPG DB objects no longer read 'ahead of the ... handoff' (future
tense). Normalize the 17 comments across authentik/immich/paperless/woodpecker/
influxdb to standing-guard/past tense. Comment-only; no manifest changes.
Closes the tense-normalization item flagged on the immich fold review (#405).
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Review follow-up: note in the compose template why the relay is not
covered by the proxy watchdog (fallback component, client-side relay
failover, restart: unless-stopped) and record the :9000/health endpoint
(verified on 0.73.2) for future active monitoring.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Second/third NetBird relay (netbirdio/relay, host network, ws :33080,
embedded STUN :3478/udp) on proxy-1/proxy-2, sharing the existing relay
auth secret, and registered in management.json Relay.Addresses and
Stuns. Clients pick a relay at random and fail over automatically, so
relayed connections and — via the added STUN — new P2P candidate
discovery now survive a controller-VPS (Hetzner) outage. Closes
mitigation option 2 from the 2026-07-25 SPOF assessment.
Plaintext rel:// is deliberate: :443/:80 on these hosts belong to the
reverse-proxy and redirect (no HTTP-01/ALPN slot), and a zone-wide
DNS-01 token on an internet-facing host is a worse trade. Peer payload
stays WireGuard-encrypted end to end; exposure is relay framing
metadata and credential sniff-and-reuse bounded by the 24h TTL.
proxy-1 is v6-only — v4-only clients that draw it fail over to the
dual-stack members.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The last app in the seed->apps migration. Folds authentik's ak-pg CNPG backend
into the existing dedicated authentik KS and, since authentik was the final
holdout, fully retires kubernetes/seed/.
- git mv seed/authentik/{cluster,objectstore,scheduled-backup,barman-creds-
externalsecret,hetzner-creds-externalsecret}.yaml -> apps/authentik/, add to
apps/authentik/kustomization.yaml (authentik has no data PVCs — all state is
in ak-pg).
- authentik KS: drop 'dependsOn: seed-authentik', add
'dependsOn: infrastructure-cnpg-barman', add the CNPG Cluster healthCheckExprs.
- Delete clusters/homelab/seed-authentik.yaml and the now-empty seed/authentik/.
Seed-layer retirement cleanup (kubernetes/seed/ is now empty):
- git mv seed/RECOVERY.md -> kubernetes/RECOVERY.md (sibling to the MIGRATION-*
docs); fix its stale 'flux reconcile kustomization seed-authentik' example to
'authentik'.
- e2e-test.sh: drop the last seed-* readiness check (seed-authentik).
- bootstrap.sh: rewrite the obsolete 'per-app seed-<app> KSes' gating comment to
reflect that app KSes now own their CNPG DBs directly, and de-'seed' the
wait-description labels (the one-shot rsync/pg_restore seed Jobs are long gone).
- opentofu/object_storage.tf: fix the CNPG ObjectStore path comment
(kubernetes/seed/* -> kubernetes/apps/*).
Safe ownership transfer (nothing pruned): all 5 seed/authentik objects carry
prune: disabled (from stage 1 #407), so deleting seed-authentik skips them in GC
and the authentik KS adopts them in place. No CNPG restart, no app restart.
Restore-test passed first: ak-pg backup restored cleanly (25 users, 368 tables,
1225 MB) using the corrected RECOVERY.md recipe, with a clean no-leak teardown.
Verified: kubectl kustomize apps/authentik builds with Cluster/ak-pg + exactly
5 prune-protected; bash -n bootstrap.sh + e2e-test.sh OK; kubernetes/seed/ fully
removed; no stale seed file-path references remain.
Left intentionally: barman/hetzner ExternalSecrets still read OpenBao keys under
seed/authentik/... (vault paths, not repo paths).
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The host-network switch (#406) broke both proxy members: the
reverse-proxy runs as a non-root user, and on the host netns the default
ip_unprivileged_port_start=1024 denies binding :443 — the container
crash-loops with 'listen tcp :443: bind: permission denied'. In bridge
mode Docker set the sysctl to 0 inside the container netns, which is why
the published-port setup never hit this.
Set net.ipv4.ip_unprivileged_port_start=443 persistently on the proxy
hosts (governs v6 binds too, despite the name). The crash-looping
container self-heals within its restart backoff once the sysctl lands.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Stage 1 of folding seed-authentik into the existing dedicated authentik KS — the
last app in the seed-layer retirement, and the highest blast radius (ak-pg backs
the SSO identity provider). Adds prune: disabled, in place, to the 5 CNPG DB
objects under seed/authentik/ so the ownership handoff can never prune them:
- Cluster/ak-pg (HA, 2 instances — pruning deletes the identity DB's data PVCs)
- ObjectStore/ak-pg-store, ScheduledBackup/ak-pg-daily
- ExternalSecret/barman-cloud-creds, ExternalSecret/hetzner-s3-creds
authentik has no data PVCs in seed/ (all state is in ak-pg). Metadata-only; no
CNPG restart. ak-pg has 30 completed barman backups (last ~6.7h) as a backstop,
and a restore-test is running against it now (corrected #404 recipe) before the
stage 2 fold.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Every connection of the k8s netbird-ingress peers relays through the
single controller-VPS relay (verified 2026-07-25: 5/5 connected peers
Relayed, zero P2P) — which is why the 2026-07-24 Hetzner DC outage took
the public edge down even though the reverse-proxy cluster runs at a
different provider. Three structural blockers, two fixed here:
- netcup proxy members: run the reverse-proxy container on the host
network instead of bridge+443:443. Behind the bridge the embedded
client's WireGuard/ICE UDP sat on a random Docker-NAT66 port that the
pods' egress policy could never match; on the host network its ICE
host candidates are the un-NATed public addresses, reachable
outbound-only from the pods (no hole punching). proxy-2 serves
v4+v6, proxy-1 v6-only — pods have working egress on both families
(verified). :443 binds as before; the hosts run no firewall
(verified), so exposure is unchanged.
- netbird-ingress-egress CNP: allow pod-to-pod between the ingress
peers (previously absent entirely — even same-cluster peers relayed
via Hetzner) and any-port UDP-capable egress to the three proxy
addresses (the embedded client's port is 51820 by default but not
contractual).
The controller relay remains available as designed fallback. Merging
auto-triggers the deploy-netbird-proxy Woodpecker pipeline (serial: 1,
one member at a time) and Flux applies the CNP.
Verify after rollout: netbird status -d in a netbird-ingress pod should
show 'Connection type: P2P' for the proxy peers and the sibling ingress
peers.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
k.renner
claude-botandClaude Opus 5
issue-fix-bot
security-bot