Compare commits

..
Author SHA1 Message Date
claude-botandClaude Opus 5.5 164cddc862 docs: per-site base domain and service CIDR for the restore test (design + plan)
The restore-test site gets its own base domain (rtest.renner.uno), service
CIDR, NetBird identity and ACME server as site values in the Talos site
patches, and the features it lacks (iGPU, IoT VLAN, UniFi gateway) are
suspended there. Production renders unchanged. Approved by the owner
2026-10-18.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 16:24:39 +02:00
k.renner c782dca48c Merge pull request 'chore: point the wiki and Renovate at konairius/notes and konairius/renovate-config' (#1323) from claude/notes-renovate-konairius into master
Reviewed-on: k.renner/homelab#1323
2026-10-18 14:15:49 +00:00
claude-botandClaude Opus 5.5 8e930e17b1 chore: point the wiki and Renovate at konairius/notes and konairius/renovate-config
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
review-bot reviewed: comment
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
Both repos moved from k.renner to the konairius org on 2026-10-18. The old
paths still resolve through Gitea's redirect (the preset API read of
k.renner/renovate-config returns 200), so nothing is broken; this removes the
dependency on the redirect.

- wiki: Wiki__NoteStore__Gitea__Repository = konairius/notes.
- renovate.json extends local>konairius/renovate-config; the runner comments follow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 16:13:09 +02:00
k.renner 3661dedd19 Merge pull request 'chore(paperless): pull vsftpd from konairius/vsftpd' (#1322) from claude/vsftpd-konairius-image into master
ci/woodpecker/cron/alert-monitor Pipeline failed
Reviewed-on: k.renner/homelab#1322
Reviewed-by: Review Bot <review-bot@renner.uno>
2026-10-18 14:09:08 +00:00
claude-botandClaude Opus 5.5 27fc56278f chore(paperless): pull vsftpd from konairius/vsftpd
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
review-bot reviewed: approve
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
The vsftpd repo moved to the konairius org and its pipeline now publishes as
registry-bot to gitea.renner.uno/konairius/vsftpd (konairius/vsftpd#3,
pipeline #12 pushed build.1792310564, same tags as before). The ImageRepository,
the ftp-scanner image and the renovate ignore list follow. The new path is
anonymously pullable like the old one, so no pull secret is needed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 10:03:55 +02:00
k.renner 98d2533c42 Merge pull request 'fix(openbao): give the registry-bot token read:repository for its pull check' (#1321) from claude/registry-bot-read-scope into master
ci/woodpecker/cron/e2e Pipeline failed
ci/woodpecker/cron/security-scan Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline failed
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#1321
2026-10-18 07:48:09 +00:00
claude-botandClaude Opus 5.5 65a22139f6 fix(openbao): give the registry-bot token read:repository for its pull check
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
review-bot reviewed: comment
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
The first run of registry-bot-secret-rotate minted the token but failed
verification: "failed verification (pull=false push=true)". The rotator
checks a new token by git-pulling the private konairius/vsftpd, and a
write:package-only token cannot read code even though the publishers team
grants code read: token scopes cap the account's rights. read:repository is
read-only and is what the pull needs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 09:44:40 +02:00
k.renner 47e6f493d0 Merge pull request 'feat(openbao): rotate a registry-bot token for konairius image publishing' (#1320) from claude/registry-bot into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#1320
2026-10-18 07:37:09 +00:00
claude-botandClaude Opus 5.5 0898fa495a feat(openbao): rotate a registry-bot token for konairius image publishing
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
review-bot reviewed: comment
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
registry-bot is a dedicated Gitea account in the konairius team "publishers"
(packages write, code read). Every konairius release pipeline publishes its
image to gitea.renner.uno/konairius/* with one shared token instead of a
per-repo k.renner PAT.

- CronJob openbao/registry-bot-secret-rotate (quarterly): secret-rotator
  gitea-token mints a write:package PAT, verifies it can pull the private
  konairius/vsftpd, stores it at infrastructure/registry-bot#token; then
  woodpecker-secret upserts the konairius org secret registry_bot_token with
  events push,tag,release,manual (no pull_request).
- OpenBao policy rotator-registry-bot. Already applied with its role, and the
  token path seeded with a placeholder, on 2026-10-18.
- Netpol halves: gitea-allow-registry-bot-rotator, woodpecker-allow-rotators,
  rotator-registry-bot-egress.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 09:32:54 +02:00
security-bot 5eeee9018f chore(image-automation): bump gitea.renner.uno/gitea_admin/review-bot:2026-10-18-0706
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-10-18 07:15:06 +00:00
k.renner ac33b0a9f2 Merge pull request 'chore(ci): bump ci-claude image to 2.1.302' (#1318) from ci-image/claude-2.1.302 into master
ci/woodpecker/push/build-ci-images Pipeline was successful
Reviewed-on: k.renner/homelab#1318
Reviewed-by: Review Bot <review-bot@renner.uno>
2026-10-18 07:06:23 +00:00
k.renner 7090f130f8 Merge branch 'master' into ci-image/claude-2.1.302
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
review-bot reviewed: approve
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
2026-10-18 06:56:43 +00:00
security-bot 462f56ab3c chore(image-automation): bump gitea.renner.uno/gitea_admin/review-bot:2026-10-18-0612
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-10-18 06:15:06 +00:00
k.renner e36e3713a7 Merge pull request 'fix(review-bot): approve with Gitea's APPROVED event' (#1319) from claude/review-bot-approved into master
ci/woodpecker/push/build-ci-images Pipeline was successful
Reviewed-on: k.renner/homelab#1319
2026-10-18 06:12:38 +00:00
claude-botandClaude Opus 5.5 04f1f1d639 fix(review-bot): approve with Gitea's APPROVED event
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
review-bot reviewed: approve
ci/woodpecker/pr/review-bot-test Pipeline was successful
The bot posted approvals with event "APPROVE". Gitea knows APPROVED,
REQUEST_CHANGES and COMMENT and stores any other value as a pending review
that only its author sees (routers/api/v1/repo/pull_review.go,
preparePullReviewType, default branch). Every approval since cdba8b62 is such
a pending draft: homelab #1270, #1278, #1287, #1318, homeassistant #385.

The bot also counted its own pending review as "reviewed at this head", so
those PRs were never reviewed again. A pending review no longer counts; the
sweep reviews them again, and Gitea submits the existing pending review with
the new event.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 08:09:47 +02:00
claude-bot 4cba950ef2 chore(ci): bump ci-claude image to 2.1.302
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
review-bot reviewed: approve
2026-10-18 00:01:26 +00:00
k.renner 438d74ae49 Merge pull request 'fix(flux): prune again in the Kustomizations that handed objects over (restore-test stack 10/10)' (#1317) from split/10-prune-restore into master
ci/woodpecker/cron/ci-claude-refresh Pipeline was successful
ci/woodpecker/cron/e2e Pipeline failed
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#1317
2026-10-17 21:24:55 +00:00
k.renner c820bf632d Merge pull request 'feat(restore-test): hcloudtest boots production's config on Hetzner, contained (restore-test stack 9/10)' (#1316) from split2/09-restore-test-harness into master
Reviewed-on: k.renner/homelab#1316
2026-10-17 21:23:26 +00:00
k.renner ee9f2ca933 Merge pull request 'feat(cnpg): every cluster incarnation archives to a new prefix; recovery reads the newest (restore-test stack 8/10)' (#1315) from split2/08-cnpg-archive-generations into master
Reviewed-on: k.renner/homelab#1315
2026-10-17 21:19:02 +00:00
k.renner 13f9f83f3a Merge pull request 'feat(velero): the cluster restores its app volumes itself, Gitea first (restore-test stack 7/10)' (#1314) from split2/07-velero-self-restore into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#1314
2026-10-17 21:11:42 +00:00
k.renner a83a179851 Merge pull request 'feat: iGPU and IoT-VLAN workloads run only where a node declares the hardware (restore-test stack 6/10)' (#1313) from split2/06-hardware-gating into master
Reviewed-on: k.renner/homelab#1313
2026-10-17 21:06:25 +00:00
k.renner 201acba44c Merge pull request 'fix(flux): services that need OpenBao secrets apply after the secret store (restore-test stack 5/10)' (#1312) from split2/05-infra-services into master
Reviewed-on: k.renner/homelab#1312
2026-10-17 21:01:06 +00:00
k.renner 5c93cc355e Merge pull request 'fix(flux): in-house-image infrastructure waits for Gitea; bird only where table 47 exists (restore-test stack 4/10)' (#1311) from split2/04-infra-inhouse into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#1311
2026-10-17 20:57:43 +00:00
k.renner 398126c2d3 Merge pull request 'fix(flux): each Kustomization depends on what its workloads need, and waits for them (restore-test stack 3/10)' (#1310) from split2/03-dependency-graph into master
Reviewed-on: k.renner/homelab#1310
2026-10-17 20:52:52 +00:00
k.renner dee454bd2e Merge pull request 'feat(talos): per-site facts in the site patches; bootstrap reads them (restore-test stack 2/10)' (#1309) from split2/02-site-facts into master
Reviewed-on: k.renner/homelab#1309
2026-10-17 20:49:56 +00:00
k.renner 17b10b11ae Merge pull request 'fix: ceph-csi-drivers chart, Gitea claim without volumeName, optional HA MQTT cert (restore-test stack 1/10)' (#1308) from split2/01-small-fixes into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#1308
2026-10-17 20:33:35 +00:00
k.renner c501cbd7b3 Merge pull request 'fix(flux): no pruning in the Kustomizations that hand objects to new owners (restore-test stack 0/10)' (#1307) from split/00-prune-handover into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#1307
2026-10-17 20:28:06 +00:00
claude-botandClaude Opus 5.5 fb953c0ed5 fix(flux): prune again in the Kustomizations that handed objects over
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
review-bot reviewed: comment
The stack's first PR turned pruning off in infrastructure,
infrastructure-crds, home-assistant, authentik, immich, matrix, paperless,
trmnl and woodpecker, so that the moved objects were dropped from their
inventories instead of deleted. Merged after every move has reconciled,
this turns it back on: the moved objects are no longer in the old
Kustomizations' inventories by then, so garbage collection has nothing of
theirs to delete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 22:20:02 +02:00
claude-botandClaude Opus 5.5 c862082d0e feat(restore-test): hcloudtest boots production's config on Hetzner, contained, and Flux restores into it
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
review-bot reviewed: comment
The restore rehearsal: a throwaway three-node cluster on Hetzner Cloud that
runs the production tree and restores production's backups into itself.

hcloudtest (bootstrap.sh) runs SITE=hcloud on cpx52, and nothing stands in
for the home network:
- site-hcloud.yaml states the Hetzner site's facts, as site-homelab.yaml
  does at home: etcd advertised on the node segment 10.90.0.0/16; a
  cilium-site ConfigMap with tunnel (VXLAN) routing over the node segment
  and only the pod CIDRs exempt from masquerading (Hetzner networks are
  layer 3, IPv4-only, and drop packets with a source not assigned to the
  sending server); a ResolverConfig with k8s-gateway first and a
  k8s-gateway-site ConfigMap pointing at Hetzner's resolvers. The node
  range is read from the machine document only, now that the patch holds
  more than one (run 28).
- hcloud-1..3.yaml put a 345GiB OSD partition beside a 60GiB EPHEMERAL on
  the system disk: the project's volume quota cannot hold three OSD
  volumes. No Hetzner volume is created for a raw volume on the system disk.
- The Talos snapshot is built when missing (a rescue-mode server writes the
  Image Factory image; removed on failure), matched by version and
  schematic. This host's inbound firewall replaces earlier runs' without a
  server ever being left without one. The Hetzner token comes from OpenBao
  (homelab-backup-verify project). Access files land in
  ~/.local/state/homelab-hcloudtest instead of access-staging.
- Containment: an outbound firewall on the nodes, applied before any
  config, admits udp and icmp anywhere and tcp to the complement of
  proxy.renner.uno's addresses, resolved over DoH at run time
  (cidr_complement, in bash; fails above Hetzner's 500 effective rules;
  one tcp rule per 100 prefixes, Hetzner's undocumented per-rule limit).
  The egress check pod (restore-test/egress-check.yaml) runs at the end of
  hcloudtest and again before the restore test writes any secret.

FLUX_SOURCE=restore-test bootstrap.sh flux runs against that cluster only
(its nodes must all be hcloud-*, and KUBECONFIG/TALOSCONFIG must not be the
operator's): flux install, a per-run read-only deploy key on the GitHub
mirror, the restore-test-git Secret, and restore-test/flux-root.yaml on
FLUX_BRANCH. flux-root.yaml applies the production tree; its patches are the
test's whole deviation: the source rewrite, no Ceph cluster network, Velero
ReadOnly, no MariaDB dumps, rotators suspended, Gitea without world egress,
netbird-service-sync at zero replicas and netbird-ingress on no node (the
test cluster rewrote production NetBird's proxy services in run 28), and
backup-verify suspended: it reads the same Hetzner token as hcloudtest and
would delete production's monthly verify VM (both label
managed-by=backup-verify). The patches are labelled 3a to 3h without gaps.

The egress check resolves A and AAAA separately and fails on either error;
before, a failed A lookup went unnoticed and the check ran on IPv6 alone.
Removing a previous egress-check pod fails with a message. hcloudclean checks
for bao like hcloudtest does. site-hcloud.yaml's header says what the site
is for now and that Ceph replicates over the pod network there.

CNPG archives, backs up and prunes as in production: the flux phase writes
cnpg-paths with every archive path in renner-uno-k8s-cnpg-rtest, created
there and emptied and deleted by hcloudclean. A test incarnation with a base
backup in the production bucket would be the newest there, and the next real
recovery would read the test's data. hcloudclean also deletes the test's ssh
keys and the restore-test deploy keys. OpenBao's status wait is bounded at
30 minutes: on a fresh cluster the pod waits for Ceph to provision its
volume.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 22:19:40 +02:00
claude-botandClaude Opus 5.5 a17c4e8161 feat(cnpg): every cluster incarnation archives to a new prefix; recovery reads the newest
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
review-bot reviewed: comment
A CNPG cluster restored from barman archived back into the location it
recovered from. plugin-barman-cloud's restore job checks the archive target
(plugins[] ObjectStore + serverName) and stops with "Expected empty archive"
when it holds WAL, which after a disaster it always does. It has no
exception for target == recovery source (internal/cnpgi/restore/restore.go,
v0.15). Restore rehearsal run 28 hit it on wp-pg; the other six never got
that far.

The check stays. Instead, archive and source become two ObjectStores whose
paths are chosen per incarnation:

- kubernetes/cnpg-object-stores/: all 14 ObjectStores (<x>-store archives,
  <x>-store-source is read by bootstrap.recovery), in a Kustomization of their
  own with postBuild.substituteFrom: cnpg-paths. Nothing else lives there:
  substitution blanks every ${...}, which is why it is not on the app
  Kustomizations (home-assistant's scripts, the Grafana dashboards). The
  ConfigMap is not optional; without it the paths would render empty.
  woodpecker's objectstore.yaml moves there as wp-pg.yaml.
- bootstrap.sh writes cnpg-paths in the flux phase, before Flux starts, once
  per incarnation (an existing one is kept):
    CNPG_<X>_SOURCE   the prefix among <x>, <x>-g<14 digits> and the store's
                      cnpg.renner.uno/source-prefixes whose <x>/base/ holds
                      the newest completed base backup (backup.info
                      status=DONE; barman uploads it as STARTED when a
                      backup begins, barman src/barman/cloud.py
                      coordinate_backup)
    CNPG_<X>_ARCHIVE  <x>-g<UTC timestamp>, new and so empty
  Every ScheduledBackup has immediate: true, so a new incarnation has a base
  backup in its own prefix minutes after it starts, and a later recovery
  picks it. S3 access is curl --aws-sigv4 with the restore key (already in
  bootstrap.sh's secret list), so the CI image needs no new tool; the
  OpenBao restore reads the same S3 endpoint and region.
- The six clusters that recovered from their archive store now name
  <x>-store-source in externalClusters. serverName stays the cluster name
  everywhere. The app Kustomizations depend on cnpg-object-stores, which
  depends on infrastructure-cnpg-barman.

Verified against the bucket, read-only, 2026-10-16: the restore key lists
renner-uno-k8s-cnpg; the listing parser on real responses, top level and an
encoded nested prefix; for wp-pg it picks wp-pg-woodpecker (newest
20261016T024000) over wp-pg (20260818T023502); leftover test serverNames
hold no base backups and match no candidate pattern.

Production rollout, one time:
1. Create flux-system/cnpg-paths with the current prefixes as both source and
   archive (ak-pg, ha-pg, im-pg, mx-pg, pprl-pg, tr-pg; wp-pg-woodpecker), so
   the running clusters keep archiving where they do.
2. The app Kustomizations that give up their ObjectStores (authentik,
   home-assistant, immich, matrix, paperless, trmnl, woodpecker) must run
   with prune: false (the stack's first PR) when this merges;
   cnpg-object-stores adopts the objects. All 14 also carry prune: disabled.
3. Whether changing externalClusters on a live Cluster restarts its pods is
   not verified; check on one cluster first.

Older generations are not cleaned up: retention applies only to the current
archive prefix.

RECOVERY.md describes the new layout: where the current prefixes are
(cnpg-paths), the per-incarnation backup paths for the precondition check,
archiving to the new prefix after promotion, and adding a CNPG cluster to a
running cluster (its keys go into the existing cnpg-paths by hand). The
manual restore drill reads <x>-store, the current archive. The app
kustomization comments name cnpg-object-stores instead of
infrastructure-cnpg-barman. write_cnpg_paths checks for yq first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 22:18:14 +02:00
claude-botandClaude Opus 5.5 93cb53130c feat(velero): the cluster restores its app volumes itself, Gitea first
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
review-bot reviewed: comment
One Velero Restore per Schedule (kubernetes/restore/<schedule>/), applied
by Flux through a restore-<schedule> Kustomization
(clusters/homelab/restore.yaml), restoring the schedule's newest completed
backup. Every app Kustomization with backed-up volumes depends on its own
restore Kustomization, so on a fresh cluster no app starts on an empty
volume, and no app waits for another app's data. This reverses the
decision on #200 (option B, 952e6150): that removed a restore Job that was
never wired in, while the seed-* rsync Kustomizations were still the
bootstrap path. Both are gone; restoring from Velero on a fresh cluster is
now wanted.

Gitea goes first: it is the registry for the in-house images. The gitea
Kustomization depends on restore-gitea alone; every other restore
Kustomization depends on restore-gitea, so Velero moves Gitea's data
first. Each app depends on its own: apps, audiobookshelf (libation through
it), authentik, home-assistant (matter-server, home-assistant-iot, otbr,
whisper too, same namespace and schedule), immich, influxdb, matrix,
music-assistant, navidrome, paperless, trmnl, wordpress. Timeouts follow the
data: immich 6h, audiobookshelf 3h, the rest 1h. One shared gate made
authentik wait for immich's photo library (45 minutes in restore rehearsal
run 28) and held a kustomize-controller worker all that time.

Ordering and failure handling, from Velero release-1.17 source:
- A Restore whose schedule has no synced backup yet goes FailedValidation
  for good (restore_controller.go). The restore Kustomizations therefore
  depend on velero-backups-synced, which is Ready once the
  BackupStorageLocation has completed a sync (status.lastSyncedTime), and
  which depends on the velero Kustomization (the BackupStorageLocation and
  its credentials).
- After that sync, FailedValidation with "No backups found for schedule"
  means there is nothing to restore and counts as done; any other
  FailedValidation, Failed or PartiallyFailed fails the Kustomization.
- A claim that already exists is skipped (csi pvc_action: "PVC already
  exists"), and the Restore completes with a warning. On the running
  cluster the Restores touch nothing.
- Velero deletes a Restore with the backup it came from (30-day TTL);
  Flux recreates it, and it again skips the existing claims.
- All StorageClasses bind Immediate, so the data mover does not wait for
  a consuming pod.

The velero-schedules header points at the restores instead of the manual
runbook, and bootstrap.sh's wait-apps comment says the volumes come from
Velero. wait-apps waits for gitea, then apps, up to an hour each (the
restore-gitea timeout), since both now include restores.
RUNBOOK-cold-rebuild.md Step 5 describes the restores and how to retry a
failed one: a Restore never retries, and while it is failed every
Kustomization depending on its restore Kustomization stops applying.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 22:15:55 +02:00
claude-botandClaude Opus 5.5 24e969abb0 docs(gpu): the intel-gpu-plugin comment describes its node label
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
review-bot reviewed: comment
It still said the DaemonSet has no nodeSelector and runs on every node.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 22:15:06 +02:00
claude-botandClaude Opus 5.5 7f1d7ac777 feat(iot): v30iot workloads run only where a node declares the IoT VLAN; HA no longer waits on them
Five workloads attach a macvlan on ens19 (v30iot) with DHCP IPAM:
shelly-manager and the mDNS relay (inside the home-assistant Kustomization),
matter-server, music-assistant and otbr. On a node without that VLAN the CNI
add fails and the pod never starts.

- k8s-1..3 carry net.renner.uno/iot-vlan: "true"; each of them declares
  ens19 in its VLANConfig. The five Deployments select on it. The comments
  they replace described the retired k8s-edge taint.
- shelly-manager and the mDNS relay move to apps/home-assistant-iot, with a
  Kustomization of their own (depends on gitea, home-assistant and the
  infrastructure Kustomizations). home-assistant's wait: true no longer
  covers them, so HA and everything waiting on it (evcc, trmnl,
  zone-configurator, speech-to-phrase) are Ready without the VLAN, and a
  broken IoT bridge no longer holds them back in production either.
- shelly-manager-auth stays in apps/home-assistant: mqtt-account-controller
  mounts it too, and that Kustomization comes first. Moved with
  shelly-manager, mqtt-account-controller would wait for a Secret that only
  the later Kustomization creates.
- The probes-blackbox and authentik blueprint comments name the new paths.

On a site without the label the IoT Kustomizations fail at their timeout,
naming the Pending pods; nothing else depends on them.

Production rollout:
1. Label the nodes first (talosctl patch the nodeLabels), or the five pods go
   Pending.
2. home-assistant must run with prune: false (the stack's first PR) when
   this merges, so it drops shelly-manager and the mDNS relay from its
   inventory without deleting them and home-assistant-iot adopts them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 22:15:03 +02:00
claude-botandClaude Opus 5.5 5e115cde11 feat(gpu): iGPU workloads run only where a node declares the iGPU
llama-server, llama-embed, wiki-embed (gpu.intel.com/i915) and the wiki
behind them are bound to the Intel iGPU. On a site without one (Hetzner,
restore rehearsal runs 27/28) their pods stayed Pending, and the GPU plugin
and exporter DaemonSets sat on every node with nothing to serve.

The node patches k8s-1..3 label their machines
hardware.renner.uno/intel-gpu=true. The GPU plugin, the exporter, the
three llama Deployments and the wiki now select on it. Elsewhere the
DaemonSets have no pods and the Deployments stay unscheduled.

Production rollout order: the label must be on the nodes (talosctl patch,
no reboot) before these manifests reconcile, or the GPU plugin and the
llama/wiki pods are removed or left Pending.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 22:14:47 +02:00
claude-botandClaude Opus 5.5 ac44925baf fix(flux): services that need OpenBao secrets apply after the secret store
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
review-bot reviewed: comment
crowdsec, external-dns-unifi, netbird-ingress and velero were in the
infrastructure Kustomization, while the ExternalSecrets that create their
Secrets were in infrastructure-crds, which depends on infrastructure. On a
cold start each was therefore installed before its Secret could exist: the
pods sat in CreateContainerConfigError until OpenBao was unsealed and ESO
synced. For the HelmReleases (crowdsec, external-dns-unifi, velero) Helm's
wait timed out meanwhile; with remediation.retries: 3 they give up after
four attempts and stay failed even once the Secret appears. Velero failing
that way stops every restore.

| workload | Secrets |
|---|---|
| crowdsec | crowdsec-bouncer-key (crowdsec, ingress), crowdsec-edge-bouncer-key |
| external-dns-unifi | external-dns-unifi-secret |
| netbird-ingress | netbird-ingress-setup-key |
| velero (+ backup-verify CronJob) | cloud-credentials, cloud-credentials-hetzner, hcloud-backup-verify |

Each now has a Kustomization of its own (clusters/homelab/
infrastructure-services.yaml), depending on infrastructure and
infrastructure-crds, with its ExternalSecrets beside it. The openbao
ClusterSecretStore in infrastructure-crds turns Ready only once OpenBao is
unsealed, so the Secret exists before the consumer installs. One
Kustomization each so that a broken service holds back only itself.

Velero's Schedules need the velero.io CRDs from its HelmRelease, so they move
from post-crds to infrastructure/velero-schedules with a Kustomization after
velero.

post-crds keeps cloudflare-api-token (consumed by its own ClusterIssuer) and
flux-git. The directories themselves stay where they are, so paths such as
the CI test infrastructure/velero/backup-verify/tests/run.sh are unchanged.

netbird-ingress gets 10m to roll its DaemonSet over three nodes
(maxUnavailable 1, minReadySeconds 45). The crowdsec comments name the
Kustomization it now runs in, and flux-graph.sh colours the new nodes.

Production rollout: infrastructure and infrastructure-crds must run with
prune: false (the stack's first PR) when this merges. Pruned, the velero
HelmRelease would be uninstalled, and the moved ExternalSecrets would take
their Secrets with them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 22:14:41 +02:00
claude-botandClaude Opus 5.5 59ebc125fa fix(flux): in-house-image infrastructure waits for Gitea; bird runs only where table 47 exists
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
review-bot reviewed: comment
bird and netbird-service-sync run images from Gitea's registry with no
copy on ghcr.io. In the infrastructure Kustomization they started long
before the restored Gitea served them and sat in ImagePullBackOff (restore
rehearsal runs 27, 28). infrastructure cannot depend on gitea, which
depends on it, so both move to a new Kustomization, infrastructure-inhouse,
that depends on infrastructure, infrastructure-crds and gitea. Their
namespace stays in infrastructure. e2e-test.sh checks the new Kustomization
for Ready.

Images published to ghcr.io too have a fallback in the nodes' registry
mirror and need no ordering; multus cni-plugins stays in infrastructure.
The secret-rotator and backup-verify CronJobs also use Gitea images but
only run on schedule; they are left in place.

bird fills routing table 47, which exists only where the site patch routes
pod traffic into it, and renders its config from node annotations only that
site sets. On a site without it the template rendered <no value> and the
init container crash-looped (run 28). The DaemonSet now selects on
net.renner.uno/pod-table=47, which site-homelab.yaml puts on its nodes.

Production rollout:
1. The pod-table label must be on the nodes (talosctl patch, no reboot)
   before the DaemonSet change reconciles, or bird is removed from every
   node and pod traffic falls through to the main table.
2. infrastructure must run with prune: false (the stack's first PR) when
   this merges. It then drops bird and netbird-service-sync from its
   inventory without deleting them, and infrastructure-inhouse adopts
   them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 22:13:48 +02:00
claude-botandClaude Opus 5.5 30a2d37fdc feat(scripts): flux-graph.sh draws the Kustomization dependency graph
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
review-bot reviewed: comment
Reads every kustomize.toolkit.fluxcd.io/v1 Kustomization in
kubernetes/clusters/homelab and emits Graphviz DOT, one edge per dependsOn
entry, pointing from the dependency to its dependent. Each node carries its
wait mode and timeout, drawn as Flux actually checks it: a Kustomization
with healthChecks and no wait is health-checked (as infrastructure is), one
with neither has no check at all, whatever its healthCheckExprs say.
Infrastructure is green, the Velero restore chain blue, wait: false dashed,
suspended Kustomizations grey.

With no argument or a .dot path it writes the full graph; with a .svg or .png
path it renders the transitive reduction (tred). Generated rather than
checked in, so it cannot drift from the manifests. Works with either yq
(mikefarah's or the Python wrapper): yq only converts YAML to JSON, jq builds
the DOT.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 22:13:24 +02:00
claude-botandClaude Opus 5.5 469abc9305 fix(authentik): its ServiceMonitor lives in monitoring-rules; the radius one is the operator's
The authentik chart renders server/servicemonitor.yaml whenever
server.metrics.serviceMonitor.enabled is set, without checking
.Capabilities for monitoring.coreos.com (verified in chart 2026.8.3). The
authentik Kustomization does not depend on monitoring, so on an empty
cluster the Helm install fails until kube-prometheus-stack has installed the
CRDs. The chart's ServiceMonitor is switched off and declared standalone in
monitoring-rules, as cilium and velero already do. The selector matches the
live Service authentik-server-metrics (port metrics, 9300); the new name
differs from the chart's authentik-server, so the two never contend for one
object while production rolls over.

authentik-outpost-servicemonitors.yaml hand-wrote a headless Service
ak-outpost-local-radius-metrics and a ServiceMonitor for it, because the
radius outpost controller used not to expose its metrics port. authentik
2026.8.3 does: the operator creates that Service (http-metrics 9300) and its
own ServiceMonitor. On a cold cluster the operator created the Service first,
with a ClusterIP; clusterIP is immutable, so the dry-run of the hand-written
clusterIP: None copy failed and took the whole monitoring-rules
Kustomization with it (restore test run 29). The file goes; the operator
owns both. No alert rule refers to the radius job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 22:13:24 +02:00
claude-botandClaude Opus 5.5 6aa8e54346 fix(flux): each Kustomization depends on what its workloads need, and waits for them
On a cold cluster (restore rehearsal runs 22-28) apps started before the
things they need and sat in ImagePullBackOff or CreateContainerConfigError,
while their Kustomizations reported Ready or held kustomize-controller
workers. The dependency graph now states those needs:

- infrastructure health-checks the openbao StatefulSet, not only the
  HelmRelease (Ready as soon as the chart is installed, sealed or not). Its
  readiness probe is `bao status`, which fails while sealed, so 3/3 ready
  means every replica is unsealed; infrastructure-crds no longer applies the
  ClusterSecretStore against a sealed OpenBao, which External Secrets then
  backs off for minutes.
- gitea edges: apps whose images live only in Gitea's registry (authentik,
  home-assistant, paperless, review-bot, wiki) and apps that are front ends
  to Gitea (gitea-mcp, woodpecker, renovate). Images also published to
  ghcr.io have a fallback in the nodes' registry mirror and get no edge
  (apps' ical-proxy, trmnl's alpine-jq).
- monitoring depends on influxdb: Grafana mounts grafana-influxdb-token,
  which a Job in the influxdb app writes.
- evcc depends on home-assistant: mqtt-account-controller there delivers
  mqtt-evcc-client-tls, and mosquitto is its broker.
- trmnl depends on home-assistant (the renderer reads HA's API);
  home-assistant depends on influxdb (its InfluxDB integration writes there).
- home-assistant and wordpress wait for their workloads. With wait: false
  and no healthChecks, Ready means only that the manifests applied, and
  healthCheckExprs are never evaluated (kustomize-controller's checkHealth
  returns first). otbr is suspended: it is not in use.
- home-assistant's timeout is 30m, not 4h. The 4h was sized for a seed
  initContainer that waited up to 3h for an HAOS backup; the slowest part
  left is ha-pg recovering from barman. With wait: true the 4h would also
  hold a kustomize-controller worker that long on a failed cold start.

No cycles: gitea depends only on the infrastructure layers; influxdb does
not use monitoring's CRDs; home-assistant does not reach evcc or trmnl.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 22:13:24 +02:00
claude-botandClaude Opus 5.5 c7b504a3f5 fix(bootstrap): k8s-gateway values with either yq; cni-confdir step re-runs
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
review-bot reviewed: comment
- The k8s-gateway values were read with `yq -y`, a flag only the Python yq
  has. The vmtest image installs Alpine's `yq`, which is yq-go (mikefarah),
  so the talos phase would fail there. Without the flag both print a
  document helm accepts, as the Cilium values line already relies on.
- A cni-confdir pod left from an earlier run (Failed or Succeeded) made
  `kubectl apply` a no-op and the wait time out. The step deletes it first.
- The Cilium HelmRelease comment named itself the only place Cilium is
  configured; the site patch's cilium-site ConfigMap is the other.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 22:13:13 +02:00
claude-botandClaude Opus 5.5 10bb7c7893 feat(talos): site facts live in the site patches, and what reads them follows
What differs between sites moves out of cluster-wide config into the Talos
site patches (site-homelab.yaml, site-qemu.yaml), and the consumers read it
from there:

- nodeIP.validSubnets lists IPv4 first, as the live nodes run. Talos keeps
  the order (siderolabs/net FilterIPs) and the first entry becomes the
  primary node IP; with fd02:46::/64 first, kube-apiserver exits with
  'service IP family "10.96.0.0/12" must match public address family'.
- etcd.advertisedSubnets (VLAN 46, a fact of the home network) moves
  unchanged from cluster.yaml to site-homelab.yaml; the rendered production
  config is byte-identical.
- Pod routing (routingMode, autoDirectNodeRoutes, directRoutingDevice) and
  the native-routing CIDRs are a cilium-site ConfigMap per site patch, read
  by the Cilium HelmRelease via valuesFrom. bootstrap.sh merges it the same
  way: site values first, HelmRelease values on top. Production renders
  unchanged (helm template, TLS secrets excluded).
- Each site patch states its node resolver list (ResolverConfig: k8s-gateway
  first, then the site's resolver) and k8s-gateway's upstream as a
  k8s-gateway-site ConfigMap, read by the k8s-gateway HelmRelease via
  valuesFrom. bootstrap.sh installs k8s-gateway right after the nodes are
  Ready, from the HelmRelease under the site values, and Flux adopts it:
  until k8s-gateway answers, every lookup waits out the node's 2 s read
  timeout and pods resolve nothing, Flux included.
- bootstrap.sh creates Cilium's cni.confPath subdirectory with a
  hostNetwork pod per node (/etc/cni is read-only to the kubelet on Talos)
  and applies Multus right after Cilium; without it the kubelet reports
  "cni plugin not initialized" and nothing schedules on a cold cluster.
- site-homelab.yaml labels its nodes net.renner.uno/pod-table=47 (table 47
  exists only where the site routes pod traffic into it); k8s-1..3 label
  hardware.renner.uno/intel-gpu and net.renner.uno/iot-vlan, the hardware
  each machine has. The workloads select on these labels in later changes.

Rollout order for production: the cilium-site and k8s-gateway-site
ConfigMaps must exist before the HelmReleases reconcile (valuesFrom is not
optional). Talos creates inline manifests on a config change but never
updates them; changes go through talosctl upgrade-k8s. The node labels go
on with talosctl patch, no reboot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 22:12:17 +02:00
claude-botandClaude Opus 5.5 03c6013190 docs(rook): CSI ServiceAccount comments describe the ceph-csi-drivers release
review-bot reviewed: comment
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
serviceaccounts.yaml and the rook-ceph operator HelmRelease still said the
ceph-csi-drivers chart is not installed and the SAs and RBAC are only
rendered statically.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 22:12:12 +02:00
claude-botandClaude Opus 5.5 0aef8dce4e fix(mqtt-account-controller): start without the HA client certificate
The controller mounted mqtt-home-assistant-tls as a required volume, and is
the only creator of that Secret (it applies the cert-manager Certificates of
the register's cert accounts). On a new cluster - Velero restores volumes,
not Secrets - its pod could never start, and mosquitto, Home Assistant,
Zigbee2MQTT and evcc stayed in ContainerCreating behind the Secrets it would
have created.

Optional now. Verified in konairius/mqtt-account-controller main: the cert
dir is read only by the HA cert sync (02:00-05:00 window, errors logged);
the broker is dialled lazily; Secret.Read treats a missing Secret as empty;
Reconcile runs reconcileCerts even when the broker step fails. So the
controller starts, creates the Certificates, the consumers start, and the
next loop reconciles the broker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 22:10:33 +02:00
claude-botandClaude Opus 5.5 bb85e00f2f fix(gitea): no pinned volumeName on data-gitea-0
The claim named its production PV, which does not exist on a rebuilt
cluster: the claim stayed Pending there, and a restored claim bound to a
new volume made the gitea Kustomization fail on the immutable field.

volumeName leaves git and the gitea Kustomization ignores
/spec/volumeName on this claim. kustomize-controller is the field's
only manager on the live claim, so its ignore rule adopts the live value
instead of clearing it; size growth still applies.

To verify after merge: the gitea Kustomization reconciles Ready and the
live claim still names pvc-e7c56db2-481d-40cb-b94a-b264b7b4b5be.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 22:10:33 +02:00
claude-botandClaude Opus 5.5 9b598fab38 fix(rook): install the ceph-csi-drivers chart Rook requires
Since Rook 1.20 the CSI Driver and OperatorConfig resources come from the
ceph-csi-drivers chart (rook Documentation/Helm-Charts/csi-drivers-chart.md:
"Rook requires the installation of the Ceph-CSI Driver chart"). The chart
was never installed here: the live cluster's Drivers are the ones the
operator migrated once during the 1.19->1.20 upgrade (2026-05-09), and
PR #12 only rendered the ServiceAccounts and RBAC. A fresh cluster
therefore has no CSI driver and provisions no volume: found by the
Hetzner restore test, where every PVC stayed Pending ("Waiting for a
volume to be created either by the external provisioner
'rook-ceph.rbd.csi.ceph.com'").

Values: Rook's recommended values for 1.21.0 plus the live shape:
2 controller replicas with Recreate, the workload-class=pinned
toleration on the node plugins, a snapshotter for RBD (Velero's CSI
snapshots of ceph-block) and group snapshots for CephFS.

The chart's ServiceAccounts are named per driver
(rook-ceph-rbd-csi-ceph-com-ctrlplugin-sa, ...), so they do not collide
with the static ones; the static ServiceAccounts and RBAC stay until the
live CSI pods run on the chart's, and go in a second change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 22:10:33 +02:00
claude-botandClaude Opus 5.5 04b3f81e00 fix(flux): no pruning in the Kustomizations that hand objects to new owners
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
review-bot reviewed: comment
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
The restore-test stack moves objects between Kustomizations:
- bird and netbird-service-sync to infrastructure-inhouse;
- crowdsec, external-dns-unifi, netbird-ingress, velero and the Velero
  Schedules, plus their ExternalSecrets, to their own Kustomizations;
- shelly-manager and the mDNS relay to home-assistant-iot;
- the CNPG ObjectStores to cnpg-object-stores.

Flux's garbage collection deletes an object dropped from a Kustomization's
source if its live owner labels (kustomize.toolkit.fluxcd.io/name and
/namespace) still name that Kustomization. Those labels belong to whichever
Kustomization applied the object last. Two phases (add to the new owner,
then remove from the old one) therefore do not guarantee anything: if the
old owner applied last before phase 2, it prunes the object
(kustomize-controller v1.9.6 deleteObjects, fluxcd/pkg/ssa v0.76.2
manager_delete.go).

With prune: false the giving Kustomizations drop the moved objects from
their inventory without deleting them, and the new owners adopt them. This
covers infrastructure, infrastructure-crds, home-assistant, authentik,
immich, matrix, paperless, trmnl and woodpecker. The last PR of the stack
switches prune back on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 22:10:22 +02:00
security-bot c45da43575 chore(image-automation): bump ghcr.io/music-assistant/server:2.10.7
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-10-17 13:43:00 +00:00
security-bot b5df790876 chore(image-automation): bump gitea.renner.uno/gitea_admin/review-bot:2026-10-17-1253
ci/woodpecker/cron/alert-monitor Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
Updates managed by image-automation-controller.
2026-10-17 13:02:09 +00:00
k.renner 30a082864e Merge pull request 'feat(review-bot): stream Mistral, add an Anthropic backend and a TypeSafe model selector' (#1306) from claude/review-bot-backends into master
ci/woodpecker/push/build-ci-images Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#1306
2026-10-17 12:53:08 +00:00
claude-botandClaude Opus 5.5 7749636a8c fix(review-bot): read all API keys from apps/review-bot
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/review-bot-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
The Mistral, Anthropic and TypeSafe keys come from the bot's own OpenBao path
apps/review-bot (minstral-api-key, antropic-api-key, typesafe-api-key) instead
of the wiki's keys. All keys and both tokens are in review-bot-secrets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 14:45:51 +02:00
claude-botandClaude Opus 5.5 16d957dff1 feat(review-bot): stream Mistral, add an Anthropic backend and a TypeSafe model selector
review-bot queued
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/review-bot-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
Mistral cut every POST /conversations off with HTTP 502 exactly ten minutes
after it started (about 30 times in the last 7 days), and every retry restarted
the whole review, so a review needing more than ten minutes could never finish.

- Mistral: the conversation now streams (stream=true, SSE). A stream that
  breaks off after the conversation started is looked up in the stored history
  (store=true, GET /conversations/{id}/history) before it is asked again. A
  broken stream is retryable like 429/502/503/504; a
  conversation.response.error event is not. One agent per model, named
  <AGENT_NAME>-<model>.
- Anthropic: new backend on the Messages API (Go SDK, streaming, adaptive
  thinking, effort high). Anthropic connects to the Gitea MCP server
  (gitea-mcp.renner.uno/mcp) with a read-only token; the toolset is in
  allowlist mode with only the read tools enabled. Paused turns are resumed,
  refusals use the server-side fallback on Opus/Sonnet 5.5.
- Models: MODELS lists backend:model = description, the first is the default.
  A backend whose keys are missing is switched off with a warning instead of
  stopping the bot.
- Selector: SELECTOR=typesafe asks TypeSafe (Jev, Choice) which model reviews
  each PR, from the title, description, size and changed files. A failure or
  an unusable choice falls back to the default model.
- Secrets: the Anthropic and TypeSafe keys reuse the wiki keys
  (apps/ai/wiki-anthropic, apps/ai/wiki-typesafe), like the Mistral key does.
  The MCP token is a second review-bot PAT scoped read:repository,read:issue,
  read:user, minted by a new rotate-mint-mcp step into
  infrastructure/review-bot#mcp_token (seeded with a placeholder). Each
  optional key has its own ExternalSecret and optional env ref.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 14:08:22 +02:00
security-bot 0111139a33 chore(image-automation): bump ghcr.io/home-assistant/home-assistant:2026.10.2
ci/woodpecker/cron/e2e Pipeline failed
ci/woodpecker/cron/security-scan Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-10-16 20:49:13 +00:00
k.renner f15331a19e Merge pull request 'fix(woodpecker): recover wp-pg from the prefix it archives to' (#1295) from fix/wp-pg-recovery-source into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#1295
2026-10-16 19:25:16 +00:00
claude-botandClaude Opus 5.5 b989f3eea7 fix(woodpecker): recover wp-pg from the prefix it archives to
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
wp-pg-store-source, the recovery source of wp-pg's bootstrap.recovery,
pointed at s3://renner-uno-k8s-cnpg/wp-pg: the prefix of the retired
apps/wp-pg. Since the namespace move wp-pg archives to wp-pg-woodpecker
(wp-pg-store). A cold rebuild would therefore have restored woodpecker as of
2026-08-18 and reported success.

Bucket, listed read-only 2026-10-16:
- wp-pg/wp-pg/base: newest 20260818T023502
- wp-pg-woodpecker/wp-pg/base: 20260916T024000 .. 20261016T024000 (daily),
  wals/ through timeline 6 (00000006.history from 2026-09-23)

The source now reads wp-pg-woodpecker; serverName stays wp-pg, the name the
live cluster archives under. bootstrap.recovery is honoured only when PGDATA
is absent, so the running cluster is unaffected. The old prefix wp-pg is no
longer referenced and keeps its objects; nothing applies retention to it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 21:13:36 +02:00
security-bot e733bd6f7b chore(image-automation): bump docker.gitea.com/gitea-mcp-server:1.9.0
ci/woodpecker/cron/e2e Pipeline failed
ci/woodpecker/cron/security-scan Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-10-16 09:33:08 +00:00
k.renner a742ef4e3c Merge pull request 'chore(paperclip): remove Paperclip and everything that only served it' (#1291) from chore/remove-paperclip into master
ci/woodpecker/push/build-ci-images Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#1291
2026-10-16 06:46:56 +00:00
security-bot 68c07963fa chore(image-automation): bump ghcr.io/paperless-ngx/paperless-ngx:3.3.2
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-10-16 06:42:45 +00:00
security-bot d15f3c526c chore(image-automation): bump gitea.renner.uno/gitea_admin/review-bot:2026-10-16-0620
ci/woodpecker/cron/alert-monitor Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
Updates managed by image-automation-controller.
2026-10-16 06:23:52 +00:00
k.renner 48bb5a9074 Merge pull request 'feat(review-bot): report queued/running/failed per PR (commit status + /status API)' (#1293) from feat/review-bot-status into master
ci/woodpecker/push/build-ci-images Pipeline was successful
Reviewed-on: k.renner/homelab#1293
2026-10-16 06:20:25 +00:00
claude-botandClaude Opus 5.5 9113cf270c feat(review-bot): serve /status on review-bot.renner.uno (LAN-only)
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/review-bot-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
Ingress routes only the /status prefix; /hook and /metrics stay
cluster-internal. No public label, so it is not on the NetBird proxy.
The new CNP lets Traefik reach the pod on 8080.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 08:12:54 +02:00
claude-botandClaude Opus 5.5 5ccb3ceaa6 feat(review-bot): report queued/running/retrying/failed per PR
Every phase of a pull request's review is now visible in two places:

- a commit status "review-bot" on the reviewed head: pending while
  queued, running or waiting to retry; success once reviewed (with the
  verdict) or skipped; error with the reason when the review failed;
  warning when new commits superseded the reviewed head. The statuses
  are posted one at a time in order, so a later phase never lands before
  an earlier one, and never block the queue.
- GET /status and GET /status/{owner}/{repo}/{n}: the in-memory state
  of every PR seen since start, with the queue position while queued.
  Finished entries are dropped after a day.

Until now a failed review posted nothing, so a PR whose review failed
looked the same as one still waiting.

Also gofmt report_test.go, which the CI gofmt check flags on master.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 08:12:42 +02:00
k.renner 4579fcd54b Merge pull request 'fix(talos): host DNS asks k8s-gateway first' (#1292) from node-dns-via-k8s-gateway into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#1292
2026-10-16 06:04:48 +00:00
claude-botandClaude Opus 5.5 afe009637a fix(talos): host DNS asks k8s-gateway first
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
containerd pulls in-house images from gitea.renner.uno in the host
network namespace, through Talos host DNS. Its only upstream was the
UDM (from DHCP), so every node-level pull of an in-cluster registry
depended on an external DNS.

A ResolverConfig document lists k8s-gateway (10.96.0.11) first and the
UDM second. It replaces the DHCP nameservers. Talos host DNS tries
upstreams in order and moves on after an error or SERVFAIL/REFUSED
(internal/pkg/dns/dns.go); a silent upstream costs the 2 s proxy read
timeout (coredns plugin/pkg/proxy), inside the 4.5 s per-query
deadline, so a cold node still resolves through the UDM until Cilium
and k8s-gateway are up.

Dry run on k8s-1: applies without reboot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 07:51:19 +02:00
k.renner 39f8279e27 Merge pull request 'fix(k8s-gateway): port values to the chart 4.x servers schema' (#1289) from worktree-k8s-gateway-chart-4 into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#1289
2026-10-16 05:33:44 +00:00
claude-botandClaude Opus 5.5 f13069f3c7 fix(k8s-gateway): port values to the chart 4.x servers schema
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
Chart 4.0.0 (installed 2026-10-08 via the open semver range) replaced
domain/watchedResources/fallthrough/extraZonePlugins with a servers:
list. The old keys were ignored and the chart default ran instead:
k8s_gateway example.com watching Service/HTTPRoute, forward to
/etc/resolv.conf, Service type LoadBalancer. No renner.uno Ingress name
has been answered in-cluster since.

The new values render the same Corefile as the last 3.x release
(helm rev 7): .:1053, k8s_gateway renner.uno on Ingress with
fallthrough renner.uno., cache with serve_stale, forward to the UDM,
Service ClusterIP 10.96.0.11 port 53, runAsUser 1000. Verified with
helm template against chart 4.0.1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 07:25:07 +02:00
security-bot 104c5edf67 chore(image-automation): bump ghcr.io/usetrmnl/trmnl-ha-amd64:0.11.1
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-10-15 23:43:55 +00:00
209 changed files with 5494 additions and 968 deletions
+1 -1
View File
@@ -101,7 +101,7 @@ steps:
bash scripts/ci/make-diag-kubeconfig.sh
- name: alert-triage-agent
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.296
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.302
environment:
CLAUDE_CODE_OAUTH_TOKEN:
from_secret: claude_code_oauth_token
+4 -4
View File
@@ -35,25 +35,25 @@ when:
steps:
- name: model-selection-tests
depends_on: []
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.296
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.302
commands:
- sh scripts/ci/test-model-selection.sh
- name: pr-review-mistral-tests
depends_on: []
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.296
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.302
commands:
- sh scripts/ci/test-pr-review-mistral.sh
- name: promtool-rules-guard-tests
depends_on: []
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.296
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.302
commands:
- sh scripts/ci/test-promtool-rules.sh
- name: yaml-citations-guard-tests
depends_on: []
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.296
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.302
commands:
- sh scripts/ci/test-yaml-citations.sh
+1 -1
View File
@@ -26,7 +26,7 @@ steps:
# fetch-models.sh response mapping. No cluster, no secrets, no network — the
# ci-claude image just provides sh + jq. Keeps the flag-injection regression
# test from rotting (it exercises the exact injection shapes).
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.296
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.302
commands:
- sh scripts/ci/test-model-selection.sh
+1 -1
View File
@@ -62,7 +62,7 @@ steps:
- bash scripts/ci/make-diag-kubeconfig.sh
- name: issue-fix-agent
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.296
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.302
environment:
CLAUDE_CODE_OAUTH_TOKEN:
from_secret: claude_code_oauth_token
+1 -1
View File
@@ -70,7 +70,7 @@ steps:
# this step is skipped — otherwise we'd fork-bomb PRs.
when:
- branch: master
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.296
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.302
environment:
CLAUDE_CODE_OAUTH_TOKEN:
from_secret: claude_code_oauth_token
+1 -1
View File
@@ -30,7 +30,7 @@ when:
steps:
- name: update-audit-agent
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.296
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.302
environment:
CLAUDE_CODE_OAUTH_TOKEN:
from_secret: claude_code_oauth_token
@@ -0,0 +1,162 @@
# Site-specific base domain and service CIDR Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** The Hetzner restore-test site runs production's tree under its own base domain, service CIDR, NetBird identity and ACME server, with the features it lacks suspended, so it can be reached through the NetBird proxy without touching production.
**Architecture:** Each Talos site patch carries one more inline manifest, ConfigMap `flux-system/site-settings` (plus the `flux-system` Namespace), holding the site's values as `SITE_*` variables. Flux reads it through `postBuild.substituteFrom`: the root sync substitutes it into the Kustomization objects (for `spec.suspend`), and every Kustomization whose manifests carry a site value substitutes it into its manifests. Production's values reproduce today's literals, so its rendered output is byte-identical; a render check enforces that per PR.
**Tech Stack:** Flux v2 (kustomize-controller v1.9.5, flux CLI ≥ v2.8 for `flux envsubst --strict`), kustomize via `kubectl kustomize`, Talos inline manifests, bash, yq (both flavours must work), Woodpecker CI.
**Spec:** `docs/superpowers/specs/2026-10-18-site-base-domain-design.md`
## Global Constraints
- Only what must differ, differs. Every `SITE_*` variable exists because of a collision named in the spec.
- Production (`site-homelab`) renders byte-identical to `origin/master` for every changed path, except the two intended pod-spec additions named in Task 3 (Gitea `GITEA__server__ROOT_URL`, gate-sync `COOKIE_DOMAIN`), each set to the value already in effect.
- Site values live in the Talos site patches (`kubernetes/talos/patches/site-{homelab,hcloud,qemu}.yaml`). No test-only patch in `restore-test/flux-root.yaml` may introduce a site difference.
- Never force-push; stacked PRs, each based on the previous; never self-merge.
- Moving objects between pruned Kustomizations is not part of this plan; if a task finds it needs one, stop and use the #1307 prune handover.
- Talos changes reach production as incremental `talosctl patch machineconfig` (never the whole repo file), `--dry-run` output filtered so no machine-config secret is printed, k8s-1 first, then k8s-2, k8s-3.
- Matrix stays hardcoded (`server_name: matrix.renner.uno`); out of scope.
- Comments state facts, not history. No prose in config files.
Site values (exact):
| Variable | homelab | hcloud | qemu |
|---|---|---|---|
| `SITE_DOMAIN` | `renner.uno` | `rtest.renner.uno` | `renner.uno` |
| `SITE_ACME_SERVER` | `https://acme-v02.api.letsencrypt.org/directory` | `https://acme-staging-v02.api.letsencrypt.org/directory` | `https://acme-v02.api.letsencrypt.org/directory` |
| `SITE_SERVICE_CIDR` | `10.96.0.0/12` | `10.112.0.0/12` | `10.96.0.0/12` |
| `SITE_DNS_GATEWAY_IP` | `10.96.0.11` | `10.112.0.11` | `10.96.0.11` |
| `SITE_NETBIRD_NETWORK` | `k8s public ingress` | `k8s public ingress rtest` | `k8s public ingress` |
| `SITE_NETBIRD_ROUTER_GROUP` | `k8s-ingress-router` | `k8s-ingress-router-rtest` | `k8s-ingress-router` |
| `SITE_NETBIRD_INGRESS_KEY` | `infrastructure/netbird-ingress` | `infrastructure/netbird-ingress-rtest` | `infrastructure/netbird-ingress` |
| `SITE_SUSPEND_IGPU` | `false` | `true` | `true` |
| `SITE_SUSPEND_IOT_VLAN` | `false` | `true` | `true` |
| `SITE_SUSPEND_UNIFI` | `false` | `true` | `true` |
`10.112.0.0/12` overlaps nothing in the repo (checked: Ceph SAN `10.99.0.0/29` sits inside `10.96.0.0/12`, Hetzner nodes use `10.90.0.0/16`, NetBird `100.64.0.0/10`).
## Review Focus
1. A manifest in a newly substituting Kustomization contains unrelated `${...}` (shell scripts, Grafana dashboards, proxy configs, CEL) — Flux would blank it silently. Expected: `flux envsubst --strict` in the render check fails on it; the object carries `kustomize.toolkit.fluxcd.io/substitute: disabled` or the `$` is escaped as `$${...}`. Test: Task 1's `test-site-render.sh` case `strict_fails_on_unknown_var`.
2. A suspended Kustomization is a `dependsOn` target of one that is not suspended on the same site — the dependent waits forever. Expected: nothing unsuspended depends on a suspended one. Test: Task 2 case `no_dependents_of_suspended`.
3. A served name left hardcoded on the hcloud site (an Ingress host, redirect URI, probe target) — the test publishes or calls a production name. Expected: the hcloud render contains `renner.uno` only as `rtest.renner.uno`, in the allow-list (label/annotation keys, image references, the Git source, email addresses, external hosts, Matrix). Test: Task 3 case `hcloud_has_no_production_names`.
4. The restore-test root (`restore-test/flux-root.yaml`) lacks the substitution the production root has, so suspends and domains render empty on the test. Expected: both roots substitute `site-settings`. Test: Task 2 case `restore_root_substitutes`.
5. The test's NetBird peers gain access to production through NetBird's "All" group policies. Expected: verified against the live NetBird policies before run 31 (Task 6 Step 2).
---
### Task 1: Render check for site values
**Files:**
- Create: `scripts/ci/site-render.sh`
- Create: `scripts/ci/test-site-render.sh`
- Create: `scripts/ci/testdata/site-render/` (fixtures)
- Modify: `.woodpecker/ci-manifests-test.yml` (new step `site-render`)
**Interfaces:**
- Produces: `scripts/ci/site-render.sh <site> <path>` — prints `kubectl kustomize <path>` piped through `flux envsubst --strict` with the `SITE_*` values read from the `site-settings` inline manifest of `kubernetes/talos/patches/site-<site>.yaml`; exit non-zero on an undefined variable. With `--root`, renders `kubernetes/clusters/homelab` the way the root sync does.
- Produces: `scripts/ci/site-render.sh --changed <site> <base-ref>` — for every Flux Kustomization path whose object has `postBuild.substituteFrom` naming `site-settings`, diff the render at HEAD against the plain render at `<base-ref>`; exit 1 on a difference, printing it.
- [ ] **Step 1: Write the failing tests** in `scripts/ci/test-site-render.sh` (same style as `scripts/ci/test-yaml-citations.sh`), with fixtures under `scripts/ci/testdata/site-render/`:
- `renders_site_value`: fixture kustomization with `host: app.${SITE_DOMAIN}` and a fixture site patch with `SITE_DOMAIN: example.test` → output contains `host: app.example.test`.
- `strict_fails_on_unknown_var`: fixture containing `${FOO}` → exit status non-zero, stderr names `FOO`.
- `escaped_dollar_survives`: fixture containing `$${FOO}` → output contains `${FOO}`.
- `disabled_object_untouched`: object annotated `kustomize.toolkit.fluxcd.io/substitute: disabled` containing `${FOO}` → output keeps `${FOO}`, exit 0.
- [ ] **Step 2: Run** `bash scripts/ci/test-site-render.sh` — expected FAIL (`site-render.sh` missing).
- [ ] **Step 3: Implement `site-render.sh`.** Read the ConfigMap from the site patch with the same yq expression bootstrap.sh uses for `cilium-site` (`select(.cluster) | .cluster.inlineManifests[] | select(.name == "site-settings") | .contents`), no `-y`; export its `data` keys; honour the `substitute: disabled` annotation by rendering those objects untouched (split the stream with yq, substitute the rest). Never `2>/dev/null`.
- [ ] **Step 4: Run** `bash scripts/ci/test-site-render.sh` — expected PASS; `shellcheck -S warning scripts/ci/site-render.sh scripts/ci/test-site-render.sh` clean.
- [ ] **Step 5: Add CI step** `site-render` to `.woodpecker/ci-manifests-test.yml` running the test script, then `site-render.sh --changed homelab "$CI_COMMIT_TARGET_BRANCH"` on pull requests. Image must carry flux and yq (check `ci/images/ci-k8s`; add flux there if missing, in this task).
- [ ] **Step 6: Commit** `ci(manifests): render check for per-site values`.
### Task 2: site-settings ConfigMap, root substitution, feature suspension
**Files:**
- Modify: `kubernetes/talos/patches/site-homelab.yaml`, `site-hcloud.yaml`, `site-qemu.yaml` (new inline manifest `site-settings`: Namespace `flux-system` + ConfigMap with all `SITE_*` keys from the table)
- Modify: `kubernetes/clusters/homelab/flux-system/kustomization.yaml` (patch adding `postBuild.substituteFrom: [{kind: ConfigMap, name: site-settings}]` to Kustomization `flux-system`; bootstrap keeps this file)
- Modify: `kubernetes/restore-test/flux-root.yaml` (same `postBuild` on Kustomization `restore-test`)
- Modify: `kubernetes/clusters/homelab/llama-server.yaml`, `wiki.yaml` (`suspend: ${SITE_SUSPEND_IGPU}`), `home-assistant.yaml` (`matter-server`, `home-assistant-iot`: `${SITE_SUSPEND_IOT_VLAN}`), `music-assistant.yaml` (`${SITE_SUSPEND_IOT_VLAN}`), `infrastructure-services.yaml` (`external-dns-unifi`: `${SITE_SUSPEND_UNIFI}`)
- Test: `scripts/ci/test-site-render.sh`
**Interfaces:**
- Consumes: `site-render.sh` (Task 1).
- Produces: ConfigMap `flux-system/site-settings` with the keys in the table; root substitution available to every Kustomization object in `kubernetes/clusters/homelab`.
- [ ] **Step 1: Write the failing tests:**
- `root_suspends_per_site`: `site-render.sh --root hcloud` shows `suspend: true` on the six Kustomizations listed above; `--root homelab` shows `suspend: false` on them and `otbr` unchanged (`true`).
- `restore_root_substitutes`: `kubernetes/restore-test/flux-root.yaml`'s `restore-test` Kustomization has `postBuild.substituteFrom` naming `site-settings` (yq query, exit 0).
- `no_dependents_of_suspended`: for each site, no Kustomization with `suspend: false` lists a suspended one in `dependsOn`.
- `homelab_root_unchanged`: `--root homelab` render equals the plain render at `origin/master` except `suspend: false` on the six objects.
- [ ] **Step 2: Run tests** — expected FAIL.
- [ ] **Step 3: Add the inline manifest and the suspend fields** listed under Files. Production values make `suspend: false` explicit; Flux treats it like the absent field.
- [ ] **Step 4: Run tests** — expected PASS; `site-render.sh --root homelab` exits 0 (no undefined variable anywhere in `kubernetes/clusters/homelab`).
- [ ] **Step 5: Commit** `feat(flux): site-settings from the Talos site patch; features a site lacks are suspended`.
- [ ] **Step 6: Production rollout before merge** (PR body): incremental Talos patch adding only the `site-settings` inline manifest, dry-run filtered, k8s-1 then k8s-2, k8s-3; verify `kubectl -n flux-system get cm site-settings -o yaml` equals the homelab column.
### Task 3: Base domain and ACME server per site
**Files (served names only; the spec's inventory lists them, about 120 files):**
- Modify: every Ingress/TLS host, cert-manager `dnsZones` and `dnsNames`, external-dns `domainFilters` and hostname annotations, k8s-gateway zone (`infrastructure/k8s-gateway/helmrelease.yaml:56,61`), CoreDNS zone (`infrastructure/coredns-config/configmap.yaml:28`), app URLs, authentik blueprint redirect URIs, `authentik_host_browser`, brand domain, LDAP names, monitoring probes, gatus, uptime-monitor — `renner.uno` → `${SITE_DOMAIN}`.
- Modify: `infrastructure/post-crds/clusterissuer.yaml` `${ACME_SERVER}` → `${SITE_ACME_SERVER}`; `infrastructure/cluster-settings/cluster-settings.yaml` drops `ACME_SERVER`; `clusters/homelab/infrastructure.yaml` `infrastructure-crds` adds `site-settings` to `substituteFrom`.
- Modify: `apps/gitea/statefulset.yaml` add `GITEA__server__ROOT_URL: https://gitea.${SITE_DOMAIN}`.
- Modify: `apps/authentik/gate-sync.yaml` add `COOKIE_DOMAIN: ${SITE_DOMAIN}`.
- Modify: each `kubernetes/clusters/homelab/*.yaml` Kustomization whose path now contains `${SITE_DOMAIN}` — add `postBuild.substituteFrom: [{kind: ConfigMap, name: site-settings}]`.
- Modify: objects in those paths with unrelated `${...}` (list in the spec's review and `git grep -l '\${[A-Za-z_]'`: home-assistant deployment/proxy.conf/config-sync.sh/zigbee2mqtt scripts, home-assistant-iot shelly proxy conf, otbr, monitoring dashboards and helmrelease-prometheus, multus cni-plugins-installer, openbao snapshot-cronjob, rook toolbox, image-automation lidarr, velero backup-verify) — annotate `kustomize.toolkit.fluxcd.io/substitute: disabled` where the object needs no site value, else escape as `$${...}`.
- Unchanged by rule: label/annotation keys (`dns.renner.uno/*`, `auth.renner.uno/*`, `net.renner.uno/*`, `cnpg.renner.uno/*`, …), image references, `gotk-sync.yaml` Git URL, email addresses, names of hosts outside the cluster (udm, nas-1/2, pve, jupiter/saturn/neptune, `netbird.renner.uno`, `slzb-*`, `*.iot.renner.uno`, `edge-1`), Matrix.
**Interfaces:**
- Consumes: `site-settings` keys `SITE_DOMAIN`, `SITE_ACME_SERVER` (Task 2); `site-render.sh` (Task 1).
- [ ] **Step 1: Write the failing test** `hcloud_has_no_production_names` in `test-site-render.sh`: render every substituting path with `hcloud`; every `renner.uno` occurrence must be `rtest.renner.uno` or match the allow-list regex file `scripts/ci/testdata/site-render/allowed-production-names.txt` (one regex per line, each with a comment naming why it stays).
- [ ] **Step 2: Run** — expected FAIL with the list of leftover names.
- [ ] **Step 3: Replace** per the Files list, in reviewable commits by area: infrastructure (DNS, cert-manager, ACME), authentik, apps, monitoring.
- [ ] **Step 4: Run** `test-site-render.sh` — PASS; `site-render.sh --changed homelab origin/master` — exits 0 except the two intended env additions, which it prints; confirm their values equal production's (`gitea.renner.uno`, `renner.uno`).
- [ ] **Step 5: Commit** per area, `feat(site): <area> reads the base domain from site-settings`.
### Task 4: Service CIDR per site
**Files:**
- Modify: `kubernetes/talos/patches/cluster.yaml:80-81` (remove `serviceSubnets`), `site-homelab.yaml`, `site-qemu.yaml` (`serviceSubnets: [10.96.0.0/12]`), `site-hcloud.yaml` (`[10.112.0.0/12]`, `ResolverConfig` nameserver `10.112.0.11` at :78)
- Modify: `infrastructure/k8s-gateway/helmrelease.yaml:104` `clusterIP: ${SITE_DNS_GATEWAY_IP}`; `infrastructure/coredns-config/configmap.yaml:35` `forward . ${SITE_DNS_GATEWAY_IP}`; `infrastructure-inhouse/netbird-service-sync/deployment.yaml:99` `${SITE_SERVICE_CIDR}`; their Kustomizations get `site-settings` in `substituteFrom` if Task 3 did not add it.
- Modify: `kubernetes/bootstrap.sh` k8s-gateway install (~1459): pass the site's `SITE_DNS_GATEWAY_IP` as the Service clusterIP (read from `site-settings` with the same yq expression as Task 1).
**Interfaces:**
- Consumes: `SITE_SERVICE_CIDR`, `SITE_DNS_GATEWAY_IP`.
- [ ] **Step 1: Failing tests:** `service_cidr_per_site` — `yq` over the merged Talos patches for each site yields the table's CIDR exactly once; `hcloud` render of k8s-gateway has `clusterIP: 10.112.0.11` and CoreDNS forwards there.
- [ ] **Step 2: Run** — FAIL.
- [ ] **Step 3: Implement** per Files.
- [ ] **Step 4: Run** — PASS; homelab render unchanged; `shellcheck -S warning kubernetes/bootstrap.sh`.
- [ ] **Step 5: Commit** `feat(site): service CIDR and DNS gateway address per site`.
- [ ] **Step 6: Production rollout:** none on the nodes — production's `serviceSubnets` value is unchanged, only moved between patch files; the live machine config already holds it.
### Task 5: NetBird identity per site
**Files:**
- Modify: `infrastructure-inhouse/netbird-service-sync/deployment.yaml` env `NETWORK_NAME: ${SITE_NETBIRD_NETWORK}`, `ROUTER_GROUP_NAME: ${SITE_NETBIRD_ROUTER_GROUP}`.
- Modify: `infrastructure/netbird-ingress/externalsecret.yaml` `key: ${SITE_NETBIRD_INGRESS_KEY}`; `clusters/homelab/infrastructure-services.yaml` `netbird-ingress` gets `site-settings` substitution.
- Create: `infrastructure/openbao/rotation/netbird-ingress-rtest.yaml` — a second CronJob like `netbird-ingress.yaml`, `BAO_PATH: homelab/infrastructure/netbird-ingress-rtest`, `NB_AUTO_GROUPS: <id of group k8s-ingress-router-rtest>`; add it to `infrastructure/openbao/rotation/kustomization.yaml`; extend the `rotator-netbird` OpenBao policy (`infrastructure/openbao/policies/`) to write that path.
**Interfaces:**
- Consumes: `SITE_NETBIRD_*` (Task 2).
- Produces: OpenBao key `homelab/infrastructure/netbird-ingress-rtest` holding a setup key whose only auto-group is `k8s-ingress-router-rtest`.
- [ ] **Step 1: Owner step (blocking):** the owner creates NetBird group `k8s-ingress-router-rtest` with no policies granting it access to production resources and gives its ID. Do not create it with credentials read from OpenBao.
- [ ] **Step 2: Failing tests:** hcloud render of netbird-service-sync has `NETWORK_NAME=k8s public ingress rtest`, `ROUTER_GROUP_NAME=k8s-ingress-router-rtest`; the netbird-ingress ExternalSecret extracts `infrastructure/netbird-ingress-rtest`; homelab render unchanged.
- [ ] **Step 3: Implement** per Files.
- [ ] **Step 4: Run tests** — PASS.
- [ ] **Step 5: Commit** `feat(site): NetBird network, router group and setup key per site`.
- [ ] **Step 6: Production rollout after merge:** create one Job from the new CronJob (`kubectl -n openbao create job --from=cronjob/netbird-secret-rotate-rtest …`), confirm the OpenBao key exists (metadata only, never the value), so the next raft snapshot carries it into the test.
### Task 6: Restore test with the new site (run 30, then run 31)
**Files:**
- Modify: `kubernetes/restore-test/flux-root.yaml` — run 31 only: remove patches 3f and 3g; renumber.
- [ ] **Step 0: Before run 30:** list every object carrying the label `external-dns/cloudflare=true` (the Cloudflare instance's `labelFilter`) and the names it would publish on hcloud. With `--default-targets` set to a `*.cfargotunnel.com` name and `--force-default-targets` (`apps/external-dns/helmrelease.yaml`), each such record would shadow the public wildcard with an unreachable target; if any exists, stop and bring it to the owner.
- [ ] **Step 1: Run 30** with patches 3f/3g still in place, hands-off, stall detector armed. Pass criteria: no undefined-variable errors; the six features show suspended; every other Kustomization Ready; all names on the test are `*.rtest.renner.uno`; certificates come from LE staging; production unaffected (production probes and `flux get ks` on production stay green); stale production URLs in restored data fail from the test (a pod on the test reaching `https://ci.renner.uno` and `https://auth.renner.uno` times out at the egress firewall). Record results in the wiki.
- [ ] **Step 2: Before run 31:** owner and builder read the live NetBird policies for group "All" and `k8s-ingress-router-rtest`; the test's peers must reach nothing of production. Check whether the NetBird proxy accepts services under `rtest.renner.uno` without a separately registered custom domain; if registration is needed, it is the owner's step.
- [ ] **Step 3: Remove patches 3f/3g**, commit `fix(restore-test): NetBird sync and routing peers run on the test site`, PR.
- [ ] **Step 4: Run 31** hands-off. Production checks every 5 min (NetBird service list for production hosts unchanged, `netbird-service-sync` log in production, public probes). Stop and tear down on the first production deviation. Pass criteria additionally: the owner opens `https://<app>.rtest.renner.uno` from the internet and logs in with the password, and an app behind the passkey step-up (`passkey-stepup.yaml`: e.g. Gitea) lets them register a new passkey on the test host after password login.
@@ -0,0 +1,227 @@
# Site-specific base domain and address ranges for the restore test
Date: 2026-10-18. Status: design, awaiting the owner's approval.
## Intent
The Hetzner restore test (`bootstrap.sh hcloudtest` + `FLUX_SOURCE=restore-test bootstrap.sh flux`)
boots production's configuration on throwaway machines and restores production's data into it.
Today it has to run with parts of production switched off (`kubernetes/restore-test/flux-root.yaml`
patches 3f and 3g, external-dns left unresolved), because it uses production's names and would
overwrite production's NetBird services, routing and DNS. In run 29 production ingress broke during
the test and the cause was never found.
The fix is to give the test cluster its own identity where, and only where, it collides with
production: a subdomain as its base domain and its own address ranges. Then the test can run with
everything production runs, including the internet path through the NetBird proxy, and the owner can
open the restored services from the internet to check that they actually work.
## Goals
- The base domain is a site fact. Production (`site-homelab`) keeps `renner.uno`. The restore-test
site (`site-hcloud`) uses a subdomain of `renner.uno` (working name `rtest.renner.uno`; the builder
confirms the final label with the owner only if it matters).
- The address range that collides is a site fact: the service CIDR, and every fixed address derived
from it. Pod CIDR and LoadBalancer pool are not routed into NetBird for the test site (see 3), so
they collide nowhere and stay identical.
- The NetBird identity of the cluster is per site, so production and test live side by side in the
same NetBird account.
- The test cluster is reachable from the internet through the same NetBird proxy as production, under
`*.<test domain>` names, and works like production there.
- With these in place, the restore-test patches that exist only because of name or range collisions
are removed (3f netbird-service-sync at zero replicas, 3g netbird-ingress on no node), and the
Cloudflare external-dns runs on the test site. external-dns-unifi does not, and neither do the
features the Hetzner site lacks: they are suspended there (see Decided).
## Non-goals
- Making production serve several domains.
- Matrix (Synapse, MAS, Element): out of scope (owner, 2026-10-18). Synapse refuses to start if
`server_name` changes (upstream `synapse/storage/databases/main/__init__.py`), so it stays
`matrix.renner.uno`; Matrix may be broken or unreachable on the test site.
- Passkeys on the test site: authentik's WebAuthn RP ID is the request host
(`authentik/stages/authenticator_webauthn/utils.py:6-11`, version 2026.8.3, not configurable), so
passkeys registered on `auth.renner.uno` do not work on the test host. The owner logs in with
the password; passkeys are not a design criterion (owner, 2026-10-18).
- Rewriting production URLs inside restored data (see Constraints, "Stale production URLs").
## Anti-goals (a failure even if it technically works)
- Any difference between the sites that is not needed. "Only things that must differ should differ"
(owner). Each per-site value must be justified by a collision.
- A harness or script that changes the test nodes or network to make the test pass. The test must
boot production's configuration; per-site facts live in the site and node files
(`site-homelab` describes the local cluster, `site-hcloud` the Hetzner cluster, node specifics in
their own files — owner's rule).
- The test cluster writing anything to production: production's NetBird services, routes, peers or
groups; production's DNS names; production's Woodpecker, Gitea or other services.
- The cluster depending on external DNS for its own services (owner's design rule).
- Any change in production's behaviour. For production, the change is a pure move of values into
the site files; rendered manifests stay identical.
## What has to differ per site, and why
### 1. Base domain
Collision: the test's Ingress hosts, NetBird proxy services (keyed by domain) and DNS records would be
production's.
Where the domain is written (inventory at origin/master, 2026-10-18, about 160 files contain
`renner.uno`; only the served names below change per site):
- Ingress and TLS hosts: about 34 files under `kubernetes/apps` and `kubernetes/infrastructure`
(e.g. `apps/gitea/ingress.yaml:19,29`), including nested names `*.home.renner.uno`.
- cert-manager: `infrastructure/post-crds/clusterissuer.yaml` (`dnsZones: renner.uno` at 19, 39, 66;
the ACME email at 8, 28, 55 is an address, not a served name); Certificates with `dnsNames`
(`apps/authentik/ldap-outpost-cert.yaml:28`, `apps/home-assistant/mosquitto-certificate.yaml:12`,
`apps/monitoring/helmrelease-alloy-syslog.yaml:12`).
- DNS: both external-dns instances (`apps/external-dns/helmrelease.yaml:51` Cloudflare;
`infrastructure/external-dns-unifi/helmrelease.yaml:84`), the `external-dns.kubernetes.io/hostname`
annotations (6 files), k8s-gateway zone (`infrastructure/k8s-gateway/helmrelease.yaml:56,61`),
CoreDNS zone (`infrastructure/coredns-config/configmap.yaml:28`).
- Application config with URLs: Gitea, Woodpecker, Immich, Paperless, WordPress, TRMNL, gitea-mcp,
review-bot, wiki, renovate, Grafana and the monitoring probes, gatus, uptime-monitor.
- authentik blueprints (`apps/authentik/blueprints-configmap.yaml`): OAuth redirect URIs (~813-951),
LDAP name and `tls_server_name`, `authentik_host_browser`, brand domain (1170).
- authentik gate-sync: `AUTHENTIK_HOST_BROWSER` (`apps/authentik/gate-sync.yaml:127`) and
`COOKIE_DOMAIN`, which is unset and defaults to `renner.uno` in the controller (`controller.py:57`);
it must be set explicitly.
- Gitea `ROOT_URL` lives in `app.ini` on the PVC (restored data), not in git; git must set it
(`GITEA__server__ROOT_URL`) so the test serves its own URL.
Stays the same on every site (not served names): label and annotation keys such as
`dns.renner.uno/public`, `auth.renner.uno/*`, `net.renner.uno/*`, `cnpg.renner.uno/*`; image
references and the registry mirror (`gitea.renner.uno` as a pull source); Flux's Git source;
email addresses; comments and docs. Names of things outside the cluster (UDM, NAS, PVE hosts,
IoT devices, `netbird.renner.uno`) stay too.
### 2. Service CIDR
Collision: the NetBird proxy reaches a cluster through a route to its service CIDR and connects to
Traefik's ClusterIP inside it (`ci/images/netbird-service-sync/controller.py`, `SERVICE_CIDR`,
`service_body()`). Two clusters with the same CIDR give the proxy two routes to the same range; it
cannot tell the clusters apart.
Currently shared, to become site facts:
- `serviceSubnets: 10.96.0.0/12` in `kubernetes/talos/patches/cluster.yaml:80-81`.
- Fixed service addresses: k8s-gateway `10.96.0.11` (`infrastructure/k8s-gateway/helmrelease.yaml:104`),
CoreDNS forward to it (`infrastructure/coredns-config/configmap.yaml:35`), the Talos
`ResolverConfig` in the site patches, kube-dns `10.96.0.10`.
- `netbird-service-sync` `SERVICE_CIDR` env in its Deployment.
Stay identical (no collision, because the test site routes nothing else into NetBird): pod CIDR
`10.244.0.0/16`, LoadBalancer pool `10.2.99.0/24` / `fd99::/64`
(`infrastructure/post-crds/cilium-bgp.yaml:22-25`) and the VIPs in it (Traefik `10.2.99.86`, Home
Assistant `10.2.99.87`). The builder confirms nothing else routes these between the sites; if
something does, that is a collision and the range becomes a site fact too.
Production keeps its current value. Its service CIDR cannot change on a running cluster anyway.
### 3. NetBird identity
Collision (verified in `ci/images/netbird-service-sync/controller.py`): the controller finds its
network by name (`NETWORK_NAME`, default `k8s public ingress`, line 44) and its resource by the
service CIDR; it treats every TLS service targeting that resource as its own and deletes those no
local Ingress carries (`is_managed` line 212, delete line 257). A second cluster with the same name
would delete production's services. `netbird-ingress` peers enrol with a setup key whose auto-group
is `k8s-ingress-router`, the router group of production's network; test peers would become routers
for production's traffic.
Per site: `NETWORK_NAME`, `ROUTER_GROUP_NAME` (and the policy name derived from the network name), the
`netbird-ingress` setup key and its auto-group.
Decided (owner, 2026-10-18): the test site gets its own setup key that only allows access to where it
is needed. The secret rotator creates it (production's key is minted by
`infrastructure/openbao/rotation/netbird-ingress.yaml`, `NB_AUTO_GROUPS`). The test's peers get no
production group, no production policy, no mesh routes and no mesh DNS: only what the proxy path to
the test's own services needs. Production's routes and nameserver groups for `k8s-ingress-router` are
set up by hand in NetBird, not in the repo; the test's peers must not be in that group. Peers join
NetBird's "All" group automatically; the builder checks what "All" grants in production and makes
sure the test's peers gain no access to production through it.
## Constraints
- Production config is the spec. Per-site values are expressed in the site files (`site-homelab.yaml`,
`site-hcloud.yaml`, `site-qemu.yaml`), the way `cilium-site` and `k8s-gateway-site` already are
(inline ConfigMaps read via `valuesFrom`, Talos-created). The builder chooses the mechanism by which
Flux-applied manifests read the site's domain and ranges (e.g. postBuild substitution from a
site ConfigMap). Known trap: postBuild substitution blanks every undefined `${var}`, and some app
manifests contain unrelated `${...}` syntax (`infrastructure/cluster-settings/cluster-settings.yaml:6-9`).
- For production the change is a no-op: rendered manifests and live objects stay identical. Prove it
(e.g. render before/after) per PR.
- Changes reach production as reviewable, stacked PRs. Moving objects between pruned Kustomizations
needs the prune handover used in #1307/#1317. Never force-push.
- The test may create its own entries in production's NetBird account and Cloudflare zone, limited to
its own names (owner, 2026-10-18). It must not modify or delete production's.
- Egress containment stays: the test's Hetzner firewall blocks TCP to production's public edge
(`proxy.renner.uno`, `bootstrap.sh` `hcloud_apply_egress`). NetBird management, signal and relay run
on `netbird.renner.uno`, a different address, and stay reachable. Inbound traffic arrives through the
NetBird tunnel. The builder must check that the test's routing peer works with TCP to the proxy
hosts blocked (the proxy hosts' extra relays on TCP 33080 are then unreachable; UDP passes).
- Stale production URLs in restored data stay as they are: Gitea's Woodpecker webhooks
(`https://ci.renner.uno/api/hook?access_token=…`, valid tokens), Home Assistant's `.storage`
(`external_url`, integrations), Immich's OIDC issuer in OpenBao `apps/immich`, WordPress post
content. They must not reach production: the test resolves its own names in-cluster
(k8s-gateway), and production names resolve publicly to the proxy, which the egress firewall
blocks. The builder verifies this holds for each of these.
- Let's Encrypt: the test site uses Let's Encrypt staging (owner, 2026-10-18: "staging certs are
fine"), so test runs never count against the `renner.uno` limits production's renewals share (about
40 Certificates per run). The ACME server therefore becomes a site fact; today `ACME_SERVER` is in
the Flux-applied `infrastructure/cluster-settings/cluster-settings.yaml`, substituted into the
ClusterIssuer. The owner accepts browser warnings when inspecting the test.
- To verify, not assumed: what the Cloudflare external-dns instance would write for the test names.
It runs `policy: upsert-only` with `--default-targets` set to a `*.cfargotunnel.com` name and
`--force-default-targets` (`apps/external-dns/helmrelease.yaml`); the public wildcard `*.renner.uno`
already resolves every test name to the proxy. A record it creates must not shadow the wildcard
with an unreachable target.
- Passkey step-up: some apps require a passkey step-up after login (blueprint `passkey-stepup.yaml`:
Home Assistant, NetBird, Audiobookshelf, Immich, Gitea, Grafana). On the test site the owner would
register a new passkey after password login; the builder checks the flow allows that.
## Decided (owner)
- Subdomain of `renner.uno` as the test base domain.
- Reachable from the internet through the NetBird proxy, like production; 1:1 with production
otherwise.
- The test may create its own NetBird and Cloudflare entries, limited to its names.
- Address ranges are site facts in the site files.
- Only what must differ, differs.
- Matrix out of scope; passkeys not a criterion, password login on the test.
- The test's NetBird peers use their own rotator-minted setup key with access only where needed.
- Features a site does not have are not applied there at all: their Kustomizations are suspended on
that site, so Flux does not try them and `flux get ks` shows them as suspended instead of failing
at a timeout ("noise", owner). The site file declares what the site has, like its other site facts:
the Intel iGPU, the IoT VLAN, and the UniFi gateway. Talos creates these as a ConfigMap (the way
`cilium-site` works), and Flux's root sync reads it to set `spec.suspend` on the affected
Kustomizations. Affected today: llama-server and wiki (iGPU), home-assistant-iot, matter-server and
music-assistant (IoT VLAN), external-dns-unifi (UniFi gateway; it does not run on the test
cluster). otbr is suspended on every site already. The builder confirms the list against the
current node-label gating (`hardware.renner.uno/intel-gpu`, `net.renner.uno/iot-vlan`, #1313); the
node labels stay, they say which nodes of a site carry the hardware.
- Let's Encrypt staging on the test site.
## Calls made in the design (not the owner's; check these)
- Only the service CIDR becomes a site range; pod CIDR and LoadBalancer pool stay identical, because
with the restricted setup key nothing routes them between the sites.
- The run-29 production ingress break was never explained. Run 30 turns on the parts that touch
production's public path, so the plan enables them in stages and checks production after each:
first `netbird-service-sync` for one pass (`ONCE`) against the test's own network, then the
routing peers, and stops on the first sign of production impact.
- The builder checks whether the NetBird proxy accepts services under the test subdomain without a
separately registered custom domain (production services are custom domains `<app>.renner.uno`,
wiki `[[netbird-reverse-proxy-custom-domains]]`). If a one-time registration is needed, it is the
owner's step to approve.
## Left to the builder
- The exact test domain label and the test's service CIDR (must not overlap production's, the LAN, or
NetBird's ranges).
- The substitution / site-value mechanism and how to keep it safe against unrelated `${...}`,
including on the root sync that sets `spec.suspend` (production must render unchanged: every
feature present, nothing suspended that is not suspended today).
- How the test's NetBird setup key and router group are created, within the constraints.
- How the Let's Encrypt limit is respected.
- Order and split of PRs, and the restore-test run that proves it (run 30), including inspecting the
restored services through the internet.
+34 -16
View File
@@ -6,18 +6,31 @@ below (like `wp-pg`) keeps `bootstrap.recovery` in git: a cold rebuild restores
each from its backup with no manual step. This document covers the
precondition, watching and verifying that restore, and the reasoning behind it.
| Cluster | Namespace | ObjectStore | barman path (`destinationPath`) | app DB / owner |
|-----------|----------------|-----------------|--------------------------------------|---------------------------------|
| `ak-pg` | authentik | `ak-pg-store` | `s3://renner-uno-k8s-cnpg/ak-pg` | `authentik`/`authentik` |
| `im-pg` | immich | `im-pg-store` | `s3://renner-uno-k8s-cnpg/im-pg` | `immich`/`immich` |
| `pprl-pg` | paperless | `pprl-pg-store` | `s3://renner-uno-k8s-cnpg/pprl-pg` | `paperless`/`paperless` |
| `ha-pg` | home-assistant | `ha-pg-store` | `s3://renner-uno-k8s-cnpg/ha-pg` | `homeassistant`/`homeassistant` |
| `mx-pg` | matrix | `mx-pg-store` | `s3://renner-uno-k8s-cnpg/mx-pg` | `synapse`/`synapse` |
| `tr-pg` | trmnl | `tr-pg-store` | `s3://renner-uno-k8s-cnpg/tr-pg` | `terminus`/`terminus` |
| Cluster | Namespace | app DB / owner |
|-----------|----------------|---------------------------------|
| `ak-pg` | authentik | `authentik`/`authentik` |
| `im-pg` | immich | `immich`/`immich` |
| `pprl-pg` | paperless | `paperless`/`paperless` |
| `ha-pg` | home-assistant | `homeassistant`/`homeassistant` |
| `mx-pg` | matrix | `synapse`/`synapse` |
| `tr-pg` | trmnl | `terminus`/`terminus` |
| `wp-pg` | woodpecker | `woodpecker`/`woodpecker` |
The barman **`serverName`** is the last path segment of `destinationPath`, i.e.
it equals the cluster name. `gf-pg` (monitoring) has no ObjectStore and stays
on `initdb`.
Each cluster `<x>` has two ObjectStores in `kubernetes/cnpg-object-stores/`:
`<x>-store`, where it archives WAL and takes base backups, and
`<x>-store-source`, which `bootstrap.recovery` reads. Their paths come from the
`flux-system/cnpg-paths` ConfigMap, which `bootstrap.sh` writes once per
cluster incarnation: the source is the prefix in `s3://renner-uno-k8s-cnpg`
whose `<x>/base/` holds the newest base backup, the archive a new prefix
`<x>-g<timestamp>`. The barman **`serverName`** is the cluster name in both.
`gf-pg` (monitoring) has no ObjectStore and stays on `initdb`.
The current prefixes: `kubectl -n flux-system get cm cnpg-paths -o yaml`
(`CNPG_<X>_SOURCE`, `CNPG_<X>_ARCHIVE`). `bootstrap.sh` never rewrites an
existing `cnpg-paths`. Adding a CNPG cluster to a running cluster therefore
means adding its two keys to that ConfigMap by hand, with the archive prefix
as the source as well; without them `cnpg-object-stores` fails to build and
every app Kustomization behind it stalls.
---
@@ -52,12 +65,16 @@ Example uses `ak-pg`; substitute the row from the table above for the others.
### 1. Precondition — confirm a backup actually exists
Never start a recovery you can't finish. Check the object store has a base
backup and recent WAL for the cluster's `serverName` before touching anything:
backup and recent WAL for the cluster's `serverName` before touching anything.
Backups live under `<prefix>/<serverName>/base/`, one prefix per incarnation
(`ak-pg` and `ak-pg-g<timestamp>`):
```sh
# From a machine with the Hetzner S3 creds (see hetzner-creds-externalsecret):
aws --endpoint-url https://fsn1.your-objectstorage.com \
s3 ls s3://renner-uno-k8s-cnpg/ak-pg/base/
s3 ls s3://renner-uno-k8s-cnpg/ | grep ak-pg
aws --endpoint-url https://fsn1.your-objectstorage.com \
s3 ls s3://renner-uno-k8s-cnpg/<prefix>/ak-pg/base/
```
If that is empty, **stop** — recovery will hang forever (no initdb fallback).
@@ -73,9 +90,10 @@ drives the restore.
brings back *all* databases and roles, not just the app DB. The
`database`/`owner` fields only tell CNPG which app user the `-app` service
should target after promotion.
- After the restored primary is promoted, CNPG bumps the PostgreSQL timeline
and `spec.plugins[].isWALArchiver: true` resumes archiving to the same
`serverName` path without clobbering the old WAL.
- After the restored primary is promoted, `spec.plugins[].isWALArchiver: true`
archives to `<x>-store`, the new incarnation's own empty prefix; the
prefix it recovered from is left untouched. Its first base backup there is
taken right away (`immediate: true` on the ScheduledBackup).
```sh
kubectl -n authentik get cluster ak-pg -w
+19 -4
View File
@@ -365,10 +365,25 @@ kubernetes/bootstrap.sh wait-apps
```
Waits per-app (apps, authentik, monitoring, paperless, home-assistant, immich,
etc., each absorbing its seed Job) and then waits for all cert-manager
Certificates to issue. Stateful apps restore from their seed KSes / backups
(CNPG DBs, migrating PVCs). Long tails are expected (immich library seed rsync
~26 min). End-to-end health check: `bash kubernetes/e2e-test.sh`.
etc.) and then waits for all cert-manager Certificates to issue. The volumes
come back by themselves: each Velero Schedule has a `restore-<schedule>`
Kustomization (`clusters/homelab/restore.yaml`) that applies a Velero Restore
of the schedule's newest backup, `restore-gitea` first, and every app waits
for its own. CNPG databases recover from barman (`RECOVERY.md`). Long tails
are expected (immich's library). End-to-end health check:
`bash kubernetes/e2e-test.sh`.
A Restore runs once and is never retried. One that ends `PartiallyFailed`,
`Failed` or `FailedValidation` (other than "No backups found for schedule")
keeps its `restore-*` Kustomization not Ready, and every Kustomization that
depends on it stops applying, until the Restore is gone. The Restore of
`restore-<name>` is `velero/<name>`. Read its result, fix the cause, then
delete it; Flux applies it again against the newest backup:
```sh
velero restore describe <name> --details
kubectl -n velero delete restore <name>
```
## Step 6 — switch the Flux source back to Gitea
@@ -131,7 +131,7 @@ data:
state: absent
# shelly-manager.home.renner.uno is gated by authentik-gate-sync
# (kubernetes/apps/home-assistant/shelly-manager-ingress.yaml), which
# (kubernetes/apps/home-assistant-iot/shelly-manager-ingress.yaml), which
# admits this group. Members are managed in Authentik.
- model: authentik_core.group
identifiers:
+1 -1
View File
@@ -47,7 +47,7 @@ spec:
plugin:
name: barman-cloud.cloudnative-pg.io
parameters:
barmanObjectName: ak-pg-store
barmanObjectName: ak-pg-store-source
serverName: ak-pg
postgresql:
parameters:
+4 -3
View File
@@ -152,10 +152,11 @@ spec:
targetCPUUtilizationPercentage: 70
metrics:
enabled: true
# The chart renders it without checking for monitoring.coreos.com, so
# the install would wait on monitoring. Declared in
# monitoring-rules/authentik-servicemonitor.yaml instead.
serviceMonitor:
enabled: true
labels:
release: kube-prometheus-stack
enabled: false
ingress:
enabled: true
ingressClassName: traefik
+2 -3
View File
@@ -11,10 +11,9 @@ resources:
- ldap-outpost-cnp.yaml
# ak-pg CNPG Postgres backend (moved here from the retired seed-authentik KS;
# each carries prune: disabled so ownership transferred without a teardown).
# authentik has no data PVCs — all state lives in ak-pg. This KS gains the
# infrastructure-cnpg-barman dep + CNPG Cluster healthCheckExprs (see
# authentik has no data PVCs — all state lives in ak-pg. This KS depends on
# cnpg-object-stores and health-checks the CNPG Cluster (see
# clusters/homelab/authentik.yaml).
- cluster.yaml
- objectstore.yaml
- scheduled-backup.yaml
- hetzner-creds-externalsecret.yaml
@@ -1,33 +0,0 @@
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: ak-pg-store
namespace: authentik
annotations:
# Prune-disabled — a standing guard kept from the seed->authentik-KS handoff (backup target).
kustomize.toolkit.fluxcd.io/prune: disabled
spec:
configuration:
# Offsite Hetzner Object Storage (fsn1) — replaces Garage on nas-1.
# No SSE: Hetzner supports only SSE-C, unusable by barman-cloud
# (plugin #646), so at-rest encryption is unavailable; transit is TLS.
destinationPath: s3://renner-uno-k8s-cnpg/ak-pg
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
# Region is required (REGION=fsn1) — without it boto3's HeadBucket
# (barman-cloud-check-wal-archive during bootstrap.recovery) 400s.
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
retentionPolicy: "30d"
+1 -1
View File
@@ -23,7 +23,7 @@ spec:
type: RuntimeDefault
containers:
- name: gitea-mcp
image: docker.gitea.com/gitea-mcp-server:1.8.1 # {"$imagepolicy": "flux-system:gitea-mcp"}
image: docker.gitea.com/gitea-mcp-server:1.9.0 # {"$imagepolicy": "flux-system:gitea-mcp"}
# Image has no ENTRYPOINT (binary is in CMD), so args alone would replace it.
command:
- /app/gitea-mcp
+22
View File
@@ -50,6 +50,28 @@ spec:
- port: "3000"
protocol: TCP
---
# Secret rotation: the registry-bot rotator mints registry-bot's PAT. Egress half:
# rotator-registry-bot-egress in network-policies/openbao-ns.yaml.
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: gitea-allow-registry-bot-rotator
namespace: gitea
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: gitea
ingress:
- fromEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": openbao
app.kubernetes.io/component: secret-rotator
app.kubernetes.io/name: registry-bot
toPorts:
- ports:
- port: "3000"
protocol: TCP
---
# Package retention CronJob (package-retention.yaml, REN-1203). It lists pods
# to find the running image tags. The namespace baseline does not open the API
# server: `kube-apiserver` is its own entity in Cilium; port is the post-DNAT
+2 -5
View File
@@ -10,10 +10,8 @@
# - Name, namespace, accessModes, storageClassName and volumeMode must match
# the live claim exactly: they are immutable, a mismatch fails the whole
# gitea Kustomization (no change to the live claim).
# - volumeName must stay: kustomize-controller has owned spec.volumeName
# since the namespace move pre-created this claim (eb23d3ac, #878).
# Without it, server-side apply tries to clear the field and the API server
# rejects the whole claim as an immutable-spec change.
# - No volumeName: the gitea Kustomization ignores /spec/volumeName on this
# claim, so the bound volume is kept and a fresh cluster binds a new one.
# - Grow only. A smaller value is rejected by the API server.
# - prune: disabled, so removing this file from git never deletes the claim
# and its data. To hand ownership back, delete the file; the claim stays.
@@ -33,7 +31,6 @@ spec:
accessModes: ["ReadWriteOnce"]
storageClassName: ceph-block
volumeMode: Filesystem
volumeName: pvc-e7c56db2-481d-40cb-b94a-b264b7b4b5be
resources:
requests:
storage: 60Gi
@@ -0,0 +1,35 @@
# The home-assistant workloads attached to the IoT VLAN (v30iot). Separate from
# ../home-assistant so that HA and what depends on it do not wait on a VLAN
# the node may not have; the pods run only where it does
# (net.renner.uno/iot-vlan).
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- shelly-manager-pvc.yaml
- shelly-manager-network-attachment.yaml
- shelly-manager-firmware-rbac.yaml
- shelly-ble-proxy-key-externalsecret.yaml
# shelly-manager-auth comes from ../home-assistant: mqtt-account-controller
# mounts it too, and that Kustomization comes first.
- shelly-manager-llat-externalsecret.yaml
- shelly-manager-deployment.yaml
- shelly-manager-service.yaml
- shelly-manager-ingress.yaml
# mDNS relay — carries IoT-VLAN announcements into HA's zeroconf listener as
# unicast, which is the only direction the CNI will move them. Without it HA
# never learns that a device's DHCP lease moved. See the header comment in
# mdns-relay-deployment.yaml.
- mdns-relay-network-attachment.yaml
- mdns-relay-service.yaml
- mdns-relay-deployment.yaml
configMapGenerator:
# shelly-manager api-proxy sidecar config (REN-935). Hash suffix disabled
# (name stays stable), so editing shelly-manager-api-proxy.conf does not
# auto-roll the pod — restart manually with
# `kubectl rollout restart deploy/shelly-manager -n home-assistant`.
- name: shelly-manager-api-proxy
namespace: home-assistant
files:
- api.conf.template=shelly-manager-api-proxy.conf
generatorOptions:
disableNameSuffixHash: true
@@ -47,6 +47,9 @@ spec:
homelab.renner.uno/restarted-for: "2026-10-06 cni-plugins v1.9.1 attach check"
k8s.v1.cni.cncf.io/networks: mdns-relay-iot
spec:
# Runs where the node patch declares the IoT VLAN (ens19).
nodeSelector:
net.renner.uno/iot-vlan: "true"
enableServiceLinks: false
securityContext:
runAsNonRoot: true
@@ -47,10 +47,10 @@ spec:
spec:
# eth0 (Cilium-managed pod network) is still the primary — it
# carries the ingress, mosquitto LB return path, etc. The macvlan
# above is additive. No explicit edge exclusion is needed: the
# `workload-class=pinned:NoSchedule` taint on k8s-edge already
# keeps this pod off there (we never tolerate it), and the NAD's
# `master: ens19` (stable predictable name for the v30iot NIC) is present on every k8s-{1,2,3} worker.
# above is additive.
# Runs where the node patch declares the IoT VLAN (ens19).
nodeSelector:
net.renner.uno/iot-vlan: "true"
enableServiceLinks: false
# Grants shelly-manager the in-cluster identity used by the firmware
# pipeline to create esphome flash Jobs and stream their logs. RBAC is
@@ -95,7 +95,7 @@ spec:
memory: 32Mi
containers:
- name: home-assistant
image: ghcr.io/home-assistant/home-assistant:2026.10.1 # {"$imagepolicy": "flux-system:home-assistant"}
image: ghcr.io/home-assistant/home-assistant:2026.10.2 # {"$imagepolicy": "flux-system:home-assistant"}
env:
- name: TZ
value: Europe/Berlin
@@ -116,7 +116,7 @@ spec:
plugin:
name: barman-cloud.cloudnative-pg.io
parameters:
barmanObjectName: ha-pg-store
barmanObjectName: ha-pg-store-source
serverName: ha-pg
postgresql:
parameters:
@@ -1,35 +0,0 @@
---
# Barman object store for ha-pg: base backups + continuous WAL archiving to
# Hetzner Object Storage, same target bucket as the other CNPG clusters.
#
# Its own prefix. Two clusters archiving WAL to one destinationPath interleave
# their streams and corrupt both — see the note in
# kubernetes/apps/woodpecker/objectstore.yaml, which learned that the hard way.
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: ha-pg-store
namespace: home-assistant
spec:
configuration:
destinationPath: s3://renner-uno-k8s-cnpg/ha-pg
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
# Matches the 30d retention the other CNPG clusters use and the
# renner-uno-k8s-cnpg bucket's noncurrent-version expiry in
# opentofu/object_storage.tf, so old S3 versions don't outlive the backups
# that superseded them.
retentionPolicy: "30d"
@@ -21,26 +21,11 @@ resources:
- mosquitto-iot-service.yaml
- mosquitto-deployment.yaml
- mosquitto-evcc-credentials-externalsecret.yaml
- shelly-manager-pvc.yaml
- shelly-manager-network-attachment.yaml
- shelly-manager-firmware-rbac.yaml
- shelly-ble-proxy-key-externalsecret.yaml
- shelly-manager-auth-externalsecret.yaml
- shelly-manager-llat-externalsecret.yaml
- shelly-manager-deployment.yaml
- shelly-manager-service.yaml
- shelly-manager-ingress.yaml
# shelly-manager and the mDNS relay sit on the IoT VLAN: ../home-assistant-iot.
- esphome-pvc.yaml
- esphome-deployment.yaml
- esphome-service.yaml
- esphome-ingress.yaml
# mDNS relay — carries IoT-VLAN announcements into HA's zeroconf listener as
# unicast, which is the only direction the CNI will move them. Without it HA
# never learns that a device's DHCP lease moved. See the header comment in
# mdns-relay-deployment.yaml.
- mdns-relay-network-attachment.yaml
- mdns-relay-service.yaml
- mdns-relay-deployment.yaml
# Zigbee2MQTT — ZHA replacement bridge. Ships scaled to 0; the cutover
# runbook (zigbee2mqtt-migration-runbook.md) scales it up. The MQTT
# `zigbee2mqtt` broker user is provisioned by mosquitto-bootstrap.yaml
@@ -57,7 +42,6 @@ resources:
# Live: recorder.db_url in the homeassistant repo points here, and the
# long-term statistics carry their full history from 2024.
- ha-pg-hetzner-creds-externalsecret.yaml
- ha-pg-objectstore.yaml
- ha-pg-cluster.yaml
- ha-pg-scheduled-backup.yaml
# MQTT credential management (REN-802): the account controller manages the
@@ -66,6 +50,8 @@ resources:
# Register: mqtt-accounts.yaml (configMapGenerator).
- mosquitto-dynsec-test.yaml
- mqtt-account-controller.yaml
# Also mounted by shelly-manager (../home-assistant-iot), which runs after this.
- shelly-manager-auth-externalsecret.yaml
configMapGenerator:
# nginx config for the reverse-proxy sidecar. Hash suffix disabled so
# the cm name stays stable; editing proxy.conf does not auto-roll the
@@ -116,13 +102,5 @@ configMapGenerator:
namespace: home-assistant
files:
- mqtt-accounts.yaml
# shelly-manager api-proxy sidecar config (REN-935). Hash suffix disabled
# (name stays stable), so editing shelly-manager-api-proxy.conf does not
# auto-roll the pod — restart manually with
# `kubectl rollout restart deploy/shelly-manager -n home-assistant`.
- name: shelly-manager-api-proxy
namespace: home-assistant
files:
- api.conf.template=shelly-manager-api-proxy.conf
generatorOptions:
disableNameSuffixHash: true
@@ -305,10 +305,13 @@ spec:
- key: api-key
path: api-key
# Client certificate HA gets (cert-manager renews it in place; not a
# subPath mount, so the next loop sees the new file).
# subPath mount, so the next loop sees the new file). Optional: this
# controller creates its Certificate, so on a new cluster the Secret
# appears only after the controller has started.
- name: ha-mqtt-tls
secret:
secretName: mqtt-home-assistant-tls
optional: true
# HA long-lived token of the service user `MQTT Cert Sync`
# (system-admin; config flows need admin). REN-1268.
- name: ha-token
+1 -1
View File
@@ -43,7 +43,7 @@ spec:
plugin:
name: barman-cloud.cloudnative-pg.io
parameters:
barmanObjectName: im-pg-store
barmanObjectName: im-pg-store-source
serverName: im-pg
postgresql:
# vchord only. The image also ships pgvecto.rs (vectors.so), but its
+2 -3
View File
@@ -8,10 +8,9 @@ resources:
- ingress.yaml
# im-pg CNPG Postgres + library/ml-cache PVCs (moved here from the retired
# seed-immich KS; each carries prune: disabled so ownership transferred without
# a teardown). This KS gains the infrastructure-cnpg-barman dep + CNPG Cluster
# healthCheckExprs (see clusters/homelab/immich.yaml).
# a teardown). This KS depends on cnpg-object-stores and health-checks the
# CNPG Cluster (see clusters/homelab/immich.yaml).
- cluster.yaml
- objectstore.yaml
- scheduled-backup.yaml
- hetzner-creds-externalsecret.yaml
- pvcs.yaml
-33
View File
@@ -1,33 +0,0 @@
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: im-pg-store
namespace: immich
annotations:
# Prune-disabled — a standing guard kept from the seed->immich-KS handoff (backup target).
kustomize.toolkit.fluxcd.io/prune: disabled
spec:
configuration:
# Offsite Hetzner Object Storage (fsn1) — replaces Garage on nas-1.
# No SSE: Hetzner supports only SSE-C, unusable by barman-cloud
# (plugin #646), so at-rest encryption is unavailable; transit is TLS.
destinationPath: s3://renner-uno-k8s-cnpg/im-pg
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
# Region is required (REGION=fsn1) — without it boto3's HeadBucket
# (barman-cloud-check-wal-archive during bootstrap.recovery) 400s.
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
retentionPolicy: "30d"
@@ -44,6 +44,9 @@ spec:
labels:
app.kubernetes.io/name: llama-server
spec:
# Runs only where the node patch declares the Intel iGPU.
nodeSelector:
hardware.renner.uno/intel-gpu: "true"
securityContext:
runAsNonRoot: true
runAsUser: 1000
+3
View File
@@ -34,6 +34,9 @@ spec:
labels:
app.kubernetes.io/name: llama-embed
spec:
# Runs only where the node patch declares the Intel iGPU.
nodeSelector:
hardware.renner.uno/intel-gpu: "true"
securityContext:
runAsNonRoot: true
runAsUser: 1000
@@ -23,6 +23,9 @@ spec:
labels:
app.kubernetes.io/name: wiki-embed
spec:
# Runs only where the node patch declares the Intel iGPU.
nodeSelector:
hardware.renner.uno/intel-gpu: "true"
securityContext:
runAsNonRoot: true
runAsUser: 1000
+1 -1
View File
@@ -31,7 +31,7 @@ spec:
plugin:
name: barman-cloud.cloudnative-pg.io
parameters:
barmanObjectName: mx-pg-store
barmanObjectName: mx-pg-store-source
serverName: mx-pg
postgresql:
parameters:
@@ -9,6 +9,5 @@ resources:
# mx-pg CNPG Postgres + offsite Barman backup. The media store and signing
# key are on synapse-data, backed up by Velero (matrix-daily).
- cluster.yaml
- objectstore.yaml
- scheduled-backup.yaml
- hetzner-creds-externalsecret.yaml
-30
View File
@@ -1,30 +0,0 @@
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: mx-pg-store
namespace: matrix
spec:
configuration:
# Offsite Hetzner Object Storage (fsn1), same bucket as the other CNPG
# clusters. No SSE: Hetzner supports only SSE-C, unusable by barman-cloud
# (plugin #646), so at-rest encryption is unavailable; transit is TLS.
destinationPath: s3://renner-uno-k8s-cnpg/mx-pg
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
# Region is required (REGION=fsn1) — without it boto3's HeadBucket
# (barman-cloud-check-wal-archive during bootstrap.recovery) 400s.
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
retentionPolicy: "30d"
@@ -32,11 +32,9 @@ spec:
# OTBR via OTBR's ClusterIP Service). The macvlan attachment
# above is additive, not a replacement.
#
# No nodeSelector/toleration: this pod only needs v30iot (present
# on all nodes), not the edge USB devices. Dropping the
# workload-class=pinned toleration lets the edge taint keep it off
# the (tiny) edge box, reserving that node for the USB-bound
# home-assistant + zwave-js pods.
# Runs where the node patch declares the IoT VLAN (ens19).
nodeSelector:
net.renner.uno/iot-vlan: "true"
containers:
- name: matter-server
# Use the HAOS-published image from `ghcr.io/matter-js/...`,
@@ -1,63 +0,0 @@
---
# Only radius needs a hand-written ServiceMonitor. The goauthentik.io operator
# creates one per outpost and Prometheus discovers those directly, because
# serviceMonitorSelector here is empty (see monitoring/helmrelease-prometheus.yaml).
# Standalone copies for ldap and paperless-gate used to live here as well; they
# duplicated the operator's and were removed.
#
# Radius requires extra plumbing. The radius binary listens on :9300 just
# like ldap/proxy (hardcoded in authentik/lib/default.yml), but authentik's
# Python operator (`providers/radius/controllers/kubernetes.py`) omits the
# `DeploymentPort(9300, "http-metrics", "tcp", 9300)` entry from its
# `deployment_ports` list — that's an upstream bug; the ldap controller has
# it, radius does not. Result: the operator-managed `ak-outpost-local-radius`
# Service exposes only UDP 1812. Add a sibling Service here that selects the
# same pod labels and exposes 9300/TCP, then a ServiceMonitor for it.
#
# The operator's own `ak-outpost-local-radius` ServiceMonitor cannot cover
# this: it selects on `app.kubernetes.io/managed-by: goauthentik.io` together
# with `goauthentik.io/service-type: metrics`, and no Service carries both —
# the operator's has the first, the one below has the second.
apiVersion: v1
kind: Service
metadata:
name: ak-outpost-local-radius-metrics
namespace: authentik
labels:
app.kubernetes.io/managed-by: homelab
goauthentik.io/outpost-name: local-radius
goauthentik.io/outpost-type: radius
spec:
type: ClusterIP
clusterIP: None
selector:
goauthentik.io/outpost-name: local-radius
goauthentik.io/outpost-type: radius
ports:
- name: http-metrics
port: 9300
targetPort: 9300
protocol: TCP
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: authentik-outpost-radius
namespace: authentik
labels:
release: kube-prometheus-stack
spec:
namespaceSelector:
matchNames:
- authentik
selector:
# Pinned to the Service above. Without managed-by this would also match the
# operator's Service and scrape the same endpoint twice.
matchLabels:
app.kubernetes.io/managed-by: homelab
goauthentik.io/outpost-name: local-radius
goauthentik.io/outpost-type: radius
endpoints:
- port: http-metrics
path: /metrics
interval: 30s
@@ -0,0 +1,24 @@
---
# authentik server metrics. The chart's metrics Service stays; only its
# ServiceMonitor is declared here (see apps/authentik/helmrelease.yaml).
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: authentik-server-metrics
namespace: authentik
labels:
release: kube-prometheus-stack
spec:
namespaceSelector:
matchNames:
- authentik
selector:
matchLabels:
app.kubernetes.io/name: authentik
app.kubernetes.io/instance: authentik
app.kubernetes.io/component: server-metrics
endpoints:
- port: metrics
path: /metrics
interval: 30s
scrapeTimeout: 3s
@@ -49,7 +49,7 @@ resources:
# cannot live beside the Velero HelmRelease.
- velero-podmonitor.yaml
- velero-servicemonitor.yaml
- authentik-outpost-servicemonitors.yaml
- authentik-servicemonitor.yaml
# Rook's exporter/mgr monitors: the operator aborts its node-daemon reconcile
# when it cannot create these, which held the CephCluster in Progressing and
# deadlocked the whole apps layer. See the file for the cycle.
@@ -269,9 +269,9 @@ spec:
# per the app's own test (shelly-manager repo,
# AuthenticationTests.Health_IsUnauthenticated); it is also the pod's
# kubelet liveness/readiness path
# (../home-assistant/shelly-manager-deployment.yaml). AllowedHosts is
# (../home-assistant-iot/shelly-manager-deployment.yaml). AllowedHosts is
# "*", so the cluster-local Host header is accepted. Service :80 ->
# container :8080 (../home-assistant/shelly-manager-service.yaml).
# container :8080 (../home-assistant-iot/shelly-manager-service.yaml).
#
# NO CiliumNetworkPolicy change: the home-assistant default-deny
# admits the whole monitoring namespace on every port
@@ -25,6 +25,8 @@ spec:
- key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule
nodeSelector:
hardware.renner.uno/intel-gpu: "true"
securityContext:
seccompProfile:
type: RuntimeDefault
@@ -33,12 +33,9 @@ spec:
k8s.v1.cni.cncf.io/networks: music-assistant-iot
spec:
enableServiceLinks: false
# No nodeSelector/toleration: MA only needs a node on v30iot (all
# of them) for audio-sync L2, not the edge USB devices. It's the
# heaviest of the formerly edge-pinned pods (2Gi limit), so
# dropping the workload-class=pinned toleration — which lets the
# edge taint keep it off the tiny edge box — is the main capacity
# win for this step.
# Runs where the node patch declares the IoT VLAN (ens19).
nodeSelector:
net.renner.uno/iot-vlan: "true"
initContainers:
# MA advertises a stored publish_ip to players (Sendspin / Snapcast /
# SlimProto) and binds its stream + Sendspin servers to a stored
@@ -55,7 +52,7 @@ spec:
# stdlib only; Flux keeps the tag in lockstep via the shared image
# policy) — no extra dependency.
- name: reconcile-publish-ip
image: ghcr.io/music-assistant/server:2.10.6 # {"$imagepolicy": "flux-system:music-assistant"}
image: ghcr.io/music-assistant/server:2.10.7 # {"$imagepolicy": "flux-system:music-assistant"}
command: ["/app/venv/bin/python", "/scripts/reconcile.py"]
volumeMounts:
- name: data
@@ -65,7 +62,7 @@ spec:
readOnly: true
containers:
- name: music-assistant
image: ghcr.io/music-assistant/server:2.10.6 # {"$imagepolicy": "flux-system:music-assistant"}
image: ghcr.io/music-assistant/server:2.10.7 # {"$imagepolicy": "flux-system:music-assistant"}
env:
- name: LOG_LEVEL
value: info
+4 -5
View File
@@ -30,11 +30,10 @@ spec:
# the IoT VLAN for mDNS/SRP advertisements and RA listening.
k8s.v1.cni.cncf.io/networks: otbr-iot
spec:
# No nodeSelector/toleration: OTBR only needs v30iot L2 presence
# (macvlan master: ens19, present on every node), not the edge USB
# devices, and its Thread RCP is reached over the network
# (slzb-thread.renner.uno). Dropping the workload-class=pinned
# toleration lets the edge taint keep it off the tiny edge box.
# Runs where the node patch declares the IoT VLAN (ens19). Its Thread
# RCP is reached over the network (slzb-thread.renner.uno).
nodeSelector:
net.renner.uno/iot-vlan: "true"
containers:
# The Thread RCP is a network-attached radio (Serial-over-IP), so the
# spinel transport needs a TCP-to-PTY bridge. Two obvious approaches
+1 -1
View File
@@ -39,7 +39,7 @@ spec:
plugin:
name: barman-cloud.cloudnative-pg.io
parameters:
barmanObjectName: pprl-pg-store
barmanObjectName: pprl-pg-store-source
serverName: pprl-pg
postgresql:
parameters:
+1 -1
View File
@@ -43,7 +43,7 @@ spec:
mountPath: /consume
containers:
- name: vsftpd
image: gitea.renner.uno/k.renner/vsftpd:build.1789630783 # {"$imagepolicy": "flux-system:vsftpd"}
image: gitea.renner.uno/konairius/vsftpd:build.1792310564 # {"$imagepolicy": "flux-system:vsftpd"}
ports:
- name: ftp-ctrl
containerPort: 21
+2 -3
View File
@@ -14,10 +14,9 @@ resources:
- network-policy.yaml
# pprl-pg CNPG Postgres + data/media/consume/redis PVCs (moved here from the
# retired seed-paperless KS; each carries prune: disabled so ownership
# transferred without a teardown). This KS gains the infrastructure-cnpg-barman
# dep + CNPG Cluster healthCheckExprs (see clusters/homelab/paperless.yaml).
# transferred without a teardown). This KS depends on cnpg-object-stores and
# health-checks the CNPG Cluster (see clusters/homelab/paperless.yaml).
- cluster.yaml
- objectstore.yaml
- scheduled-backup.yaml
- hetzner-creds-externalsecret.yaml
- pvcs.yaml
@@ -1,35 +0,0 @@
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: pprl-pg-store
namespace: paperless
annotations:
# Prune-disabled — a standing guard kept from the seed->paperless-KS handoff (backup target).
kustomize.toolkit.fluxcd.io/prune: disabled
spec:
configuration:
# Offsite Hetzner Object Storage (fsn1) — replaces local Garage on
# nas-1 so paperless backups survive nas-1 being powered off. barman
# uses boto3 default (auto) addressing; Hetzner (Ceph RGW) accepts it.
destinationPath: s3://renner-uno-k8s-cnpg/pprl-pg
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
# No `encryption: AES256` (SSE-S3): Hetzner Object Storage supports
# only SSE-C, which the barman-cloud plugin can't use (cnpg
# plugin-barman-cloud #646). At-rest encryption is therefore not
# available here; transit is TLS and the bucket is private.
wal:
compression: gzip
data:
compression: gzip
retentionPolicy: "30d"
+1 -1
View File
@@ -182,7 +182,7 @@ spec:
drop: ["ALL"]
containers:
- name: paperless
image: ghcr.io/paperless-ngx/paperless-ngx:3.3.1 # {"$imagepolicy": "flux-system:paperless-ngx"}
image: ghcr.io/paperless-ngx/paperless-ngx:3.3.2 # {"$imagepolicy": "flux-system:paperless-ngx"}
ports:
- name: http
containerPort: 8000
+1 -1
View File
@@ -4,7 +4,7 @@
# This is the configuration of the RUNNER only: where Gitea is, and which
# repositories to process. What Renovate does INSIDE a repository — schedule,
# grouping, commit titles, digest pinning — lives in the shared preset repository
# `k.renner/renovate-config`, which each repository's renovate.json extends. Change
# `konairius/renovate-config`, which each repository's renovate.json extends. Change
# update policy there, not here.
#
# autodiscover + requireConfig=required + onboarding=false
+1 -1
View File
@@ -2,7 +2,7 @@
# Renovate runner (REN-216).
#
# RUNS DAILY, BUT OPENS ROUTINE PULL REQUESTS ON MONDAY ONLY. The weekly rhythm
# is enforced by the `schedule` in the shared preset (k.renner/renovate-config),
# is enforced by the `schedule` in the shared preset (konairius/renovate-config),
# not by this cron line. The daily run exists for one reason: a security update
# (`vulnerabilityAlerts`, which the preset leaves unscheduled) must not wait up
# to six days. On Tuesday to Sunday a run finds nothing in schedule, opens and
+25 -3
View File
@@ -32,7 +32,7 @@ spec:
type: RuntimeDefault
containers:
- name: review-bot
image: gitea.renner.uno/gitea_admin/review-bot:2026-10-15-0721 # {"$imagepolicy": "flux-system:review-bot"}
image: gitea.renner.uno/gitea_admin/review-bot:2026-10-18-0706 # {"$imagepolicy": "flux-system:review-bot"}
env:
- name: REPOS
value: k.renner/homelab,k.renner/homeassistant,k.renner/authentik-samba-bridge,konairius/shelly-manager
@@ -40,10 +40,32 @@ spec:
value: https://gitea.renner.uno
- name: AGENT_NAME
value: pr-review
- name: AGENT_MODEL
value: mistral-large-4-0
- name: CONNECTOR_ID
value: 01a12adf-cdc3-7562-b32b-b1fa96be244c
# The first model is the default; a backend without its keys is off.
- name: MODELS
value: |
mistral:mistral-large-4-0 = Mistral Large 4. Solid on routine changes of small to medium size.
anthropic:claude-haiku-5-5 = Claude Haiku 5.5. Cheapest and fastest; only for trivial changes such as documentation, comments or version bumps.
anthropic:claude-sonnet-5-5 = Claude Sonnet 5.5. Fast and capable; good for most code and configuration changes.
anthropic:claude-opus-5-5 = Claude Opus 5.5. The strongest reviewer; for large or cross-cutting changes and anything touching security, secrets, network policy or data.
- name: SELECTOR
value: typesafe
- name: ANTHROPIC_API_KEY
valueFrom:
secretKeyRef:
name: review-bot-secrets
key: ANTHROPIC_API_KEY
- name: MCP_TOKEN
valueFrom:
secretKeyRef:
name: review-bot-secrets
key: MCP_TOKEN
- name: TYPESAFE_API_KEY
valueFrom:
secretKeyRef:
name: review-bot-secrets
key: TYPESAFE_API_KEY
- name: WEBHOOK_SECRET
valueFrom:
secretKeyRef:
+16 -3
View File
@@ -17,10 +17,23 @@ spec:
property: webhook-secret
- secretKey: MISTRAL_API_KEY
remoteRef:
key: apps/wiki
property: minstral-token
# The review-bot token, rotated quarterly by the review-bot-secret-rotate CronJob.
key: apps/review-bot
property: minstral-api-key
- secretKey: ANTHROPIC_API_KEY
remoteRef:
key: apps/review-bot
property: antropic-api-key
- secretKey: TYPESAFE_API_KEY
remoteRef:
key: apps/review-bot
property: typesafe-api-key
# The review-bot tokens, rotated quarterly by the review-bot-secret-rotate CronJob; MCP_TOKEN is
# the read-only one for the Gitea MCP server.
- secretKey: REVIEW_TOKEN
remoteRef:
key: infrastructure/review-bot
property: token
- secretKey: MCP_TOKEN
remoteRef:
key: infrastructure/review-bot
property: mcp_token
+27
View File
@@ -0,0 +1,27 @@
# review-bot.renner.uno — LAN-only (no dns.renner.uno/public label); only the read-only /status API is routed.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: review-bot
namespace: review-bot
labels:
app.kubernetes.io/name: review-bot
annotations:
cert-manager.io/cluster-issuer: letsencrypt-cluster
spec:
ingressClassName: traefik
tls:
- hosts:
- review-bot.renner.uno
secretName: review-bot-tls
rules:
- host: review-bot.renner.uno
http:
paths:
- path: /status
pathType: Prefix
backend:
service:
name: review-bot
port:
number: 8080
@@ -4,3 +4,4 @@ resources:
- deployment.yaml
- service.yaml
- externalsecret.yaml
- ingress.yaml
+1 -1
View File
@@ -36,7 +36,7 @@ spec:
plugin:
name: barman-cloud.cloudnative-pg.io
parameters:
barmanObjectName: tr-pg-store
barmanObjectName: tr-pg-store-source
serverName: tr-pg
postgresql:
parameters:
-1
View File
@@ -18,6 +18,5 @@ resources:
# tr-pg CNPG Postgres + offsite Barman backup. Note this cluster runs
# PostgreSQL 18, not the repo-standard 16 — see the comment in cluster.yaml.
- cluster.yaml
- objectstore.yaml
- scheduled-backup.yaml
- hetzner-creds-externalsecret.yaml
-30
View File
@@ -1,30 +0,0 @@
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: tr-pg-store
namespace: trmnl
spec:
configuration:
# Offsite Hetzner Object Storage (fsn1), same bucket as the other CNPG
# clusters. No SSE: Hetzner supports only SSE-C, unusable by barman-cloud
# (plugin #646), so at-rest encryption is unavailable; transit is TLS.
destinationPath: s3://renner-uno-k8s-cnpg/tr-pg
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
# Region is required (REGION=fsn1) — without it boto3's HeadBucket
# (barman-cloud-check-wal-archive during bootstrap.recovery) 400s.
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
retentionPolicy: "30d"
+1 -1
View File
@@ -239,7 +239,7 @@ spec:
# the panel after one lands. ARCH-SUFFIXED image: upstream publishes
# no multi-arch manifest.
- name: renderer
image: ghcr.io/usetrmnl/trmnl-ha-amd64:0.11.0 # {"$imagepolicy": "flux-system:trmnl-renderer"}
image: ghcr.io/usetrmnl/trmnl-ha-amd64:0.11.1 # {"$imagepolicy": "flux-system:trmnl-renderer"}
ports:
- name: http
containerPort: 10000
+5 -1
View File
@@ -39,6 +39,10 @@ spec:
labels:
app.kubernetes.io/name: wiki
spec:
# Its embedding backend (wiki-embed) needs the Intel iGPU; without it
# the wiki has nothing to serve, so it runs where the iGPU is declared.
nodeSelector:
hardware.renner.uno/intel-gpu: "true"
automountServiceAccountToken: false
securityContext:
runAsNonRoot: true
@@ -63,7 +67,7 @@ spec:
- name: Wiki__NoteStore__Gitea__BaseUrl
value: https://gitea.renner.uno
- name: Wiki__NoteStore__Gitea__Repository
value: k.renner/notes
value: konairius/notes
- name: Wiki__NoteStore__Gitea__Branch
value: main
- name: Wiki__Embeddings__Models__bge-m3__Url
@@ -31,9 +31,8 @@ resources:
# not here: it must exist before the `infrastructure-crds` KS applies the
# namespace's CiliumNetworkPolicies, and this KS runs after both of those.
#
# This KS depends on infrastructure-cnpg-barman for the barman plugin behind
# the ObjectStore and ScheduledBackup.
# This KS depends on cnpg-object-stores for the ObjectStores its Cluster and
# ScheduledBackup name.
- hetzner-creds-externalsecret.yaml
- objectstore.yaml
- cluster.yaml
- scheduled-backup.yaml
@@ -33,6 +33,10 @@ spec:
"k8s:io.kubernetes.pod.namespace": openbao
app.kubernetes.io/component: secret-rotator
app.kubernetes.io/name: claude-oauth
- matchLabels:
"k8s:io.kubernetes.pod.namespace": openbao
app.kubernetes.io/component: secret-rotator
app.kubernetes.io/name: registry-bot
toPorts:
- ports:
- port: "8000"
@@ -1,66 +0,0 @@
---
# RECOVERY SOURCE — read-only. The prefix the retired `apps/wp-pg` archived to.
#
# Kept after the migration on purpose. Nothing writes to it, and the backups
# under it age out on their own retention, but while they last this is a second,
# independent restore point that predates the move — worth having until the new
# prefix has a meaningful backup history of its own.
#
# serverName on the consuming externalCluster must be `wp-pg`, the name of the
# cluster that produced these backups.
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: wp-pg-store-source
namespace: woodpecker
spec:
configuration:
destinationPath: s3://renner-uno-k8s-cnpg/wp-pg
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
# No retentionPolicy: this store is a read source during the migration, and
# the old cluster in `apps` owns the lifecycle of the data underneath it.
---
# ARCHIVING TARGET — a DIFFERENT prefix from the source, deliberately.
#
# Two clusters archiving WAL to one destinationPath interleave their streams and
# corrupt both. That mattered during the parallel-run window; it still matters
# now, because the source prefix above is retained and must stay frozen for the
# independent restore point it provides.
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: wp-pg-store
namespace: woodpecker
spec:
configuration:
destinationPath: s3://renner-uno-k8s-cnpg/wp-pg-woodpecker
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
retentionPolicy: "30d"
+724 -96
View File
File diff suppressed because it is too large Load Diff
+1
View File
@@ -17,6 +17,7 @@ spec:
# woodpecker migrated) — secrets come from OpenBao via ESO
# (MIGRATION-sops-openbao.md).
dependsOn:
- name: restore-apps
- name: infrastructure
- name: infrastructure-crds
- name: infrastructure-ceph-cluster
@@ -22,6 +22,7 @@ spec:
secretRef:
name: sops-age
dependsOn:
- name: restore-audiobookshelf
- name: infrastructure
- name: infrastructure-crds
- name: infrastructure-ceph-cluster
+3 -1
View File
@@ -20,10 +20,12 @@ spec:
# from OpenBao via ESO (MIGRATION-sops-openbao.md). This was the LAST
# sops consumer; phase 4 (age-key retirement) may proceed after soak.
dependsOn:
- name: restore-authentik
- name: infrastructure
- name: infrastructure-crds
- name: infrastructure-ceph-cluster
- name: infrastructure-cnpg-barman
- name: cnpg-object-stores # the ObjectStores its Cluster names
- name: gitea # server, LDAP outpost and gate-sync images are in Gitea's registry
healthCheckExprs:
- apiVersion: postgresql.cnpg.io/v1
kind: Cluster
@@ -0,0 +1,27 @@
# The CNPG ObjectStores. Their paths come from the cnpg-paths ConfigMap, which
# bootstrap.sh writes once per cluster incarnation and never replaces:
# CNPG_<X>_SOURCE the bucket prefix holding <x>'s newest base backup
# CNPG_<X>_ARCHIVE a new, empty prefix <x>-g<timestamp>
# Not optional: without it every path renders empty.
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: cnpg-object-stores
namespace: flux-system
spec:
interval: 10m
retryInterval: 1m
timeout: 5m
sourceRef:
kind: GitRepository
name: flux-system
path: ./kubernetes/cnpg-object-stores
prune: true
wait: true
dependsOn:
- name: infrastructure
- name: infrastructure-cnpg-barman
postBuild:
substituteFrom:
- kind: ConfigMap
name: cnpg-paths
+2
View File
@@ -19,3 +19,5 @@ spec:
- name: infrastructure
- name: infrastructure-crds
- name: infrastructure-ceph-cluster
# mqtt-account-controller there delivers mqtt-evcc-client-tls; mosquitto is the broker.
- name: home-assistant
@@ -16,3 +16,4 @@ spec:
dependsOn:
- name: infrastructure
- name: infrastructure-crds
- name: gitea # serves Gitea's API
+7
View File
@@ -14,7 +14,14 @@ spec:
path: ./kubernetes/apps/gitea
prune: true
wait: true
# The claim's bound volume differs per cluster (apps/gitea/pvc.yaml).
ignore:
- paths: ["/spec/volumeName"]
target:
kind: PersistentVolumeClaim
name: data-gitea-0
dependsOn:
- name: restore-gitea
- name: infrastructure
- name: infrastructure-crds
- name: infrastructure-ceph-cluster
+39 -25
View File
@@ -4,43 +4,28 @@ metadata:
name: home-assistant
namespace: flux-system
spec:
# Seed Job collapsed into the HA pod's `seed` initContainer so the
# backup tarball + .HA_RESTORE manifest staging happens in the same
# pod as HA Core (no Multi-Attach race against the RWO Ceph-RBD
# PVC if production grows to multiple edge nodes). HA Core still
# does the actual restore from the staged manifest on its boot.
# Timeout has to absorb the worst-case 3h wait for the HAOS backup
# to appear in Garage plus the in-pod download + HA's native restore.
# The slowest part on a cold start is ha-pg recovering from barman (base
# backup ~1.7 GiB compressed, then WAL replay); the config PVC is restored
# beforehand by restore-home-assistant.
interval: 10m
retryInterval: 1m
timeout: 4h
timeout: 30m
sourceRef:
kind: GitRepository
name: flux-system
path: ./kubernetes/apps/home-assistant
prune: true
# Same TBD-USB caveat for the stromzaehler hostPath. Manifests apply;
# HA pod may CrashLoop until the USB is present — acceptable during canary.
wait: false
wait: true
# decryption block dropped 2026-07-13: path is sops-free — secrets come
# from OpenBao via ESO (MIGRATION-sops-openbao.md).
dependsOn:
- name: restore-home-assistant
- name: infrastructure
- name: infrastructure-crds
- name: infrastructure-ceph-cluster
# ha-pg-objectstore.yaml declares an ObjectStore, whose CRD comes from the
# CNPG Barman plugin in infrastructure-cnpg-barman. Without this edge the
# kind is unknown when this path is first applied, and a server-side
# dry-run rejects the ENTIRE set over the one object it cannot recognise:
#
# ObjectStore/home-assistant/ha-pg-store dry-run failed:
# no matches for kind "ObjectStore" in version ...
#
# Every other ObjectStore user — authentik, immich, paperless, trmnl,
# woodpecker — already declares it. This was the only one that did not, and
# it took speech-to-phrase, zone-configurator and matter-server down with
# it, since all three wait on home-assistant.
- name: infrastructure-cnpg-barman
- name: cnpg-object-stores # the ObjectStores its Cluster names
- name: gitea # mqtt-account-controller's image is in Gitea's registry
- name: influxdb # HA's InfluxDB integration writes there
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
@@ -58,10 +43,35 @@ spec:
prune: true
wait: true
dependsOn:
- name: restore-home-assistant
- name: infrastructure
- name: infrastructure-crds
- name: infrastructure-ceph-cluster
---
# shelly-manager and the mDNS relay: home-assistant's IoT-VLAN workloads.
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: home-assistant-iot
namespace: flux-system
spec:
interval: 10m
retryInterval: 1m
timeout: 10m
sourceRef:
kind: GitRepository
name: flux-system
path: ./kubernetes/apps/home-assistant-iot
prune: true
wait: true
dependsOn:
- name: restore-home-assistant # shelly-manager's PVC
- name: infrastructure
- name: infrastructure-crds
- name: infrastructure-ceph-cluster
- name: gitea # shelly-manager's image is in Gitea's registry
- name: home-assistant # both serve HA: its API, its zeroconf listener
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
@@ -76,10 +86,13 @@ spec:
name: flux-system
path: ./kubernetes/apps/otbr
prune: true
wait: false
# Not in use.
suspend: true
wait: true
# decryption block dropped 2026-07-13: path is sops-free — secrets come
# from OpenBao via ESO (MIGRATION-sops-openbao.md).
dependsOn:
- name: restore-home-assistant
- name: infrastructure
- name: infrastructure-crds
---
@@ -120,6 +133,7 @@ spec:
# First boot downloads the Whisper model (a few hundred MB) — give it room.
wait: true
dependsOn:
- name: restore-home-assistant
- name: infrastructure
- name: infrastructure-crds
- name: infrastructure-ceph-cluster
+2 -1
View File
@@ -20,10 +20,11 @@ spec:
# decryption block dropped 2026-07-13: path is sops-free — secrets come
# from OpenBao via ESO (MIGRATION-sops-openbao.md).
dependsOn:
- name: restore-immich
- name: infrastructure
- name: infrastructure-crds
- name: infrastructure-ceph-cluster
- name: infrastructure-cnpg-barman
- name: cnpg-object-stores # the ObjectStores its Cluster names
healthCheckExprs:
- apiVersion: postgresql.cnpg.io/v1
kind: Cluster
@@ -25,6 +25,7 @@ spec:
# boilerplate from the monolithic seed KS — InfluxDB has no CNPG cluster, only
# a chart-provisioned ceph-block PVC, so ceph is the only storage dep needed.
dependsOn:
- name: restore-influxdb
- name: infrastructure
- name: infrastructure-crds
- name: infrastructure-ceph-cluster
@@ -0,0 +1,19 @@
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: infrastructure-inhouse
namespace: flux-system
spec:
interval: 10m
retryInterval: 2m
timeout: 10m
sourceRef:
kind: GitRepository
name: flux-system
path: ./kubernetes/infrastructure-inhouse
prune: true
wait: true
dependsOn:
- name: infrastructure
- name: infrastructure-crds
- name: gitea # its images are in Gitea's registry
@@ -0,0 +1,98 @@
# Infrastructure services that consume Secrets from OpenBao. Each carries its
# ExternalSecrets and depends on infrastructure-crds, where the openbao
# ClusterSecretStore turns Ready only once OpenBao is unsealed, so the
# HelmRelease never installs before its Secret exists. One Kustomization each:
# a broken service holds back nothing but itself (velero: the restore chain).
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: crowdsec
namespace: flux-system
spec:
interval: 10m
retryInterval: 1m
timeout: 10m
sourceRef:
kind: GitRepository
name: flux-system
path: ./kubernetes/infrastructure/crowdsec
prune: true
wait: true
dependsOn:
- name: infrastructure
- name: infrastructure-crds
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: external-dns-unifi
namespace: flux-system
spec:
interval: 10m
retryInterval: 1m
timeout: 5m
sourceRef:
kind: GitRepository
name: flux-system
path: ./kubernetes/infrastructure/external-dns-unifi
prune: true
wait: true
dependsOn:
- name: infrastructure
- name: infrastructure-crds
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: netbird-ingress
namespace: flux-system
spec:
interval: 10m
retryInterval: 1m
timeout: 10m
sourceRef:
kind: GitRepository
name: flux-system
path: ./kubernetes/infrastructure/netbird-ingress
prune: true
wait: true
dependsOn:
- name: infrastructure
- name: infrastructure-crds
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: velero
namespace: flux-system
spec:
interval: 10m
retryInterval: 1m
timeout: 15m
sourceRef:
kind: GitRepository
name: flux-system
path: ./kubernetes/infrastructure/velero
prune: true
wait: true
dependsOn:
- name: infrastructure
- name: infrastructure-crds
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: velero-schedules
namespace: flux-system
spec:
interval: 10m
retryInterval: 1m
timeout: 5m
sourceRef:
kind: GitRepository
name: flux-system
path: ./kubernetes/infrastructure/velero-schedules
prune: true
wait: true
dependsOn:
- name: velero # the velero.io CRDs
@@ -40,6 +40,13 @@ spec:
kind: HelmRelease
name: openbao # the ESO backend ExternalSecrets resolve against
namespace: openbao
# Ready only when every replica is unsealed (readiness probe: bao status).
# The HelmRelease is Ready as soon as the chart is installed; the
# ClusterSecretStore in infrastructure-crds must not come up before.
- apiVersion: apps/v1
kind: StatefulSet
name: openbao
namespace: openbao
- apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
name: rook-ceph-operator # storage — apps mount PVCs
+2 -1
View File
@@ -17,10 +17,11 @@ spec:
prune: true
wait: true
dependsOn:
- name: restore-matrix
- name: infrastructure
- name: infrastructure-crds
- name: infrastructure-ceph-cluster
- name: infrastructure-cnpg-barman
- name: cnpg-object-stores # the ObjectStores its Cluster names
healthCheckExprs:
- apiVersion: postgresql.cnpg.io/v1
kind: Cluster
@@ -46,6 +46,7 @@ spec:
# from OpenBao via ESO (MIGRATION-sops-openbao.md).
dependsOn:
- name: grafana-db
- name: influxdb # its Job writes the grafana-influxdb-token Secret Grafana mounts
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
@@ -22,6 +22,7 @@ spec:
secretRef:
name: sops-age
dependsOn:
- name: restore-music-assistant
- name: infrastructure
- name: infrastructure-crds
- name: infrastructure-ceph-cluster
@@ -14,6 +14,7 @@ spec:
prune: true
wait: true
dependsOn:
- name: restore-navidrome
- name: infrastructure
- name: infrastructure-crds
- name: infrastructure-ceph-cluster
+3 -1
View File
@@ -20,10 +20,12 @@ spec:
# decryption block dropped 2026-07-13: path is sops-free — secrets come
# from OpenBao via ESO (MIGRATION-sops-openbao.md).
dependsOn:
- name: restore-paperless
- name: infrastructure
- name: infrastructure-crds
- name: infrastructure-ceph-cluster
- name: infrastructure-cnpg-barman
- name: cnpg-object-stores # the ObjectStores its Cluster names
- name: gitea # ftp-scanner (vsftpd) image is in Gitea's registry
healthCheckExprs:
- apiVersion: postgresql.cnpg.io/v1
kind: Cluster
@@ -35,3 +35,4 @@ spec:
# The `openbao` ClusterSecretStore, including the `renovate` entry in its
# conditions.namespaces allowlist.
- name: infrastructure-crds
- name: gitea # the platform it runs against (git.renner.uno)
+396
View File
@@ -0,0 +1,396 @@
# Ready once Velero has completed a sync of the backups in the bucket. A
# Restore created before that finds no backup and fails for good.
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: velero-backups-synced
namespace: flux-system
spec:
interval: 10m
timeout: 30m
sourceRef:
kind: GitRepository
name: flux-system
path: ./kubernetes/restore/gate
prune: true
dependsOn:
- name: velero # the BackupStorageLocation and its credentials
healthChecks:
- apiVersion: velero.io/v1
kind: BackupStorageLocation
name: default
namespace: velero
healthCheckExprs:
- apiVersion: velero.io/v1
kind: BackupStorageLocation
current: has(status.lastSyncedTime)
---
# One Kustomization per Velero Schedule (infrastructure/velero-schedules/schedules.yaml),
# each restoring that schedule's newest backup. An app depends on its own, so
# it waits for its data and for nothing else. Gitea's data goes first: it is
# the registry for the in-house images; the others depend on restore-gitea.
# openbao-daily has none: bootstrap.sh restores OpenBao from its raft snapshot.
# Every one carries the same health check: a schedule without any backup has
# nothing to restore (FailedValidation "No backups found for schedule").
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: restore-gitea
namespace: flux-system
spec:
interval: 10m
timeout: 1h
sourceRef:
kind: GitRepository
name: flux-system
path: ./kubernetes/restore/gitea
prune: true
wait: true
dependsOn:
- name: velero-backups-synced
- name: infrastructure-ceph-cluster
healthCheckExprs:
- apiVersion: velero.io/v1
kind: Restore
failed: >-
has(status.phase) && (status.phase in ['Failed', 'PartiallyFailed'] ||
(status.phase == 'FailedValidation' && !(has(status.validationErrors) &&
status.validationErrors.exists(e, e.startsWith('No backups found for schedule')))))
current: >-
has(status.phase) && (status.phase == 'Completed' || status.phase == 'FailedValidation')
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: restore-apps
namespace: flux-system
spec:
interval: 10m
timeout: 1h
sourceRef:
kind: GitRepository
name: flux-system
path: ./kubernetes/restore/apps
prune: true
wait: true
dependsOn:
- name: restore-gitea
- name: velero-backups-synced
- name: infrastructure-ceph-cluster
healthCheckExprs:
- apiVersion: velero.io/v1
kind: Restore
failed: >-
has(status.phase) && (status.phase in ['Failed', 'PartiallyFailed'] ||
(status.phase == 'FailedValidation' && !(has(status.validationErrors) &&
status.validationErrors.exists(e, e.startsWith('No backups found for schedule')))))
current: >-
has(status.phase) && (status.phase == 'Completed' || status.phase == 'FailedValidation')
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: restore-audiobookshelf
namespace: flux-system
spec:
interval: 10m
timeout: 3h
sourceRef:
kind: GitRepository
name: flux-system
path: ./kubernetes/restore/audiobookshelf
prune: true
wait: true
dependsOn:
- name: restore-gitea
- name: velero-backups-synced
- name: infrastructure-ceph-cluster
healthCheckExprs:
- apiVersion: velero.io/v1
kind: Restore
failed: >-
has(status.phase) && (status.phase in ['Failed', 'PartiallyFailed'] ||
(status.phase == 'FailedValidation' && !(has(status.validationErrors) &&
status.validationErrors.exists(e, e.startsWith('No backups found for schedule')))))
current: >-
has(status.phase) && (status.phase == 'Completed' || status.phase == 'FailedValidation')
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: restore-authentik
namespace: flux-system
spec:
interval: 10m
timeout: 1h
sourceRef:
kind: GitRepository
name: flux-system
path: ./kubernetes/restore/authentik
prune: true
wait: true
dependsOn:
- name: restore-gitea
- name: velero-backups-synced
- name: infrastructure-ceph-cluster
healthCheckExprs:
- apiVersion: velero.io/v1
kind: Restore
failed: >-
has(status.phase) && (status.phase in ['Failed', 'PartiallyFailed'] ||
(status.phase == 'FailedValidation' && !(has(status.validationErrors) &&
status.validationErrors.exists(e, e.startsWith('No backups found for schedule')))))
current: >-
has(status.phase) && (status.phase == 'Completed' || status.phase == 'FailedValidation')
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: restore-home-assistant
namespace: flux-system
spec:
interval: 10m
timeout: 1h
sourceRef:
kind: GitRepository
name: flux-system
path: ./kubernetes/restore/home-assistant
prune: true
wait: true
dependsOn:
- name: restore-gitea
- name: velero-backups-synced
- name: infrastructure-ceph-cluster
healthCheckExprs:
- apiVersion: velero.io/v1
kind: Restore
failed: >-
has(status.phase) && (status.phase in ['Failed', 'PartiallyFailed'] ||
(status.phase == 'FailedValidation' && !(has(status.validationErrors) &&
status.validationErrors.exists(e, e.startsWith('No backups found for schedule')))))
current: >-
has(status.phase) && (status.phase == 'Completed' || status.phase == 'FailedValidation')
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: restore-immich
namespace: flux-system
spec:
interval: 10m
timeout: 6h
sourceRef:
kind: GitRepository
name: flux-system
path: ./kubernetes/restore/immich
prune: true
wait: true
dependsOn:
- name: restore-gitea
- name: velero-backups-synced
- name: infrastructure-ceph-cluster
healthCheckExprs:
- apiVersion: velero.io/v1
kind: Restore
failed: >-
has(status.phase) && (status.phase in ['Failed', 'PartiallyFailed'] ||
(status.phase == 'FailedValidation' && !(has(status.validationErrors) &&
status.validationErrors.exists(e, e.startsWith('No backups found for schedule')))))
current: >-
has(status.phase) && (status.phase == 'Completed' || status.phase == 'FailedValidation')
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: restore-influxdb
namespace: flux-system
spec:
interval: 10m
timeout: 1h
sourceRef:
kind: GitRepository
name: flux-system
path: ./kubernetes/restore/influxdb
prune: true
wait: true
dependsOn:
- name: restore-gitea
- name: velero-backups-synced
- name: infrastructure-ceph-cluster
healthCheckExprs:
- apiVersion: velero.io/v1
kind: Restore
failed: >-
has(status.phase) && (status.phase in ['Failed', 'PartiallyFailed'] ||
(status.phase == 'FailedValidation' && !(has(status.validationErrors) &&
status.validationErrors.exists(e, e.startsWith('No backups found for schedule')))))
current: >-
has(status.phase) && (status.phase == 'Completed' || status.phase == 'FailedValidation')
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: restore-matrix
namespace: flux-system
spec:
interval: 10m
timeout: 1h
sourceRef:
kind: GitRepository
name: flux-system
path: ./kubernetes/restore/matrix
prune: true
wait: true
dependsOn:
- name: restore-gitea
- name: velero-backups-synced
- name: infrastructure-ceph-cluster
healthCheckExprs:
- apiVersion: velero.io/v1
kind: Restore
failed: >-
has(status.phase) && (status.phase in ['Failed', 'PartiallyFailed'] ||
(status.phase == 'FailedValidation' && !(has(status.validationErrors) &&
status.validationErrors.exists(e, e.startsWith('No backups found for schedule')))))
current: >-
has(status.phase) && (status.phase == 'Completed' || status.phase == 'FailedValidation')
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: restore-music-assistant
namespace: flux-system
spec:
interval: 10m
timeout: 1h
sourceRef:
kind: GitRepository
name: flux-system
path: ./kubernetes/restore/music-assistant
prune: true
wait: true
dependsOn:
- name: restore-gitea
- name: velero-backups-synced
- name: infrastructure-ceph-cluster
healthCheckExprs:
- apiVersion: velero.io/v1
kind: Restore
failed: >-
has(status.phase) && (status.phase in ['Failed', 'PartiallyFailed'] ||
(status.phase == 'FailedValidation' && !(has(status.validationErrors) &&
status.validationErrors.exists(e, e.startsWith('No backups found for schedule')))))
current: >-
has(status.phase) && (status.phase == 'Completed' || status.phase == 'FailedValidation')
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: restore-navidrome
namespace: flux-system
spec:
interval: 10m
timeout: 1h
sourceRef:
kind: GitRepository
name: flux-system
path: ./kubernetes/restore/navidrome
prune: true
wait: true
dependsOn:
- name: restore-gitea
- name: velero-backups-synced
- name: infrastructure-ceph-cluster
healthCheckExprs:
- apiVersion: velero.io/v1
kind: Restore
failed: >-
has(status.phase) && (status.phase in ['Failed', 'PartiallyFailed'] ||
(status.phase == 'FailedValidation' && !(has(status.validationErrors) &&
status.validationErrors.exists(e, e.startsWith('No backups found for schedule')))))
current: >-
has(status.phase) && (status.phase == 'Completed' || status.phase == 'FailedValidation')
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: restore-paperless
namespace: flux-system
spec:
interval: 10m
timeout: 1h
sourceRef:
kind: GitRepository
name: flux-system
path: ./kubernetes/restore/paperless
prune: true
wait: true
dependsOn:
- name: restore-gitea
- name: velero-backups-synced
- name: infrastructure-ceph-cluster
healthCheckExprs:
- apiVersion: velero.io/v1
kind: Restore
failed: >-
has(status.phase) && (status.phase in ['Failed', 'PartiallyFailed'] ||
(status.phase == 'FailedValidation' && !(has(status.validationErrors) &&
status.validationErrors.exists(e, e.startsWith('No backups found for schedule')))))
current: >-
has(status.phase) && (status.phase == 'Completed' || status.phase == 'FailedValidation')
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: restore-trmnl
namespace: flux-system
spec:
interval: 10m
timeout: 1h
sourceRef:
kind: GitRepository
name: flux-system
path: ./kubernetes/restore/trmnl
prune: true
wait: true
dependsOn:
- name: restore-gitea
- name: velero-backups-synced
- name: infrastructure-ceph-cluster
healthCheckExprs:
- apiVersion: velero.io/v1
kind: Restore
failed: >-
has(status.phase) && (status.phase in ['Failed', 'PartiallyFailed'] ||
(status.phase == 'FailedValidation' && !(has(status.validationErrors) &&
status.validationErrors.exists(e, e.startsWith('No backups found for schedule')))))
current: >-
has(status.phase) && (status.phase == 'Completed' || status.phase == 'FailedValidation')
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: restore-wordpress
namespace: flux-system
spec:
interval: 10m
timeout: 1h
sourceRef:
kind: GitRepository
name: flux-system
path: ./kubernetes/restore/wordpress
prune: true
wait: true
dependsOn:
- name: restore-gitea
- name: velero-backups-synced
- name: infrastructure-ceph-cluster
healthCheckExprs:
- apiVersion: velero.io/v1
kind: Restore
failed: >-
has(status.phase) && (status.phase in ['Failed', 'PartiallyFailed'] ||
(status.phase == 'FailedValidation' && !(has(status.validationErrors) &&
status.validationErrors.exists(e, e.startsWith('No backups found for schedule')))))
current: >-
has(status.phase) && (status.phase == 'Completed' || status.phase == 'FailedValidation')
@@ -16,3 +16,4 @@ spec:
dependsOn:
- name: infrastructure
- name: infrastructure-crds
- name: gitea # its image is in Gitea's registry
+3 -1
View File
@@ -17,10 +17,12 @@ spec:
prune: true
wait: true
dependsOn:
- name: restore-trmnl
- name: infrastructure
- name: infrastructure-crds
- name: infrastructure-ceph-cluster
- name: infrastructure-cnpg-barman
- name: cnpg-object-stores # the ObjectStores its Cluster names
- name: home-assistant # the renderer reads HA's API
healthCheckExprs:
- apiVersion: postgresql.cnpg.io/v1
kind: Cluster
+1
View File
@@ -20,3 +20,4 @@ spec:
- name: infrastructure-ceph-cluster
# wiki-embed (the embedding backend) lives in the llama-server app.
- name: llama-server
- name: gitea # its image is in Gitea's registry
+2 -1
View File
@@ -24,7 +24,8 @@ spec:
- name: infrastructure
- name: infrastructure-crds
- name: infrastructure-ceph-cluster
- name: infrastructure-cnpg-barman
- name: cnpg-object-stores # the ObjectStores its Cluster names
- name: gitea # its forge and login provider
healthCheckExprs:
- apiVersion: postgresql.cnpg.io/v1
kind: Cluster
@@ -12,7 +12,9 @@ spec:
name: flux-system
path: ./kubernetes/apps/wordpress
prune: true
wait: true
dependsOn:
- name: restore-wordpress
# infrastructure gates on the mariadb-operator HelmRelease being Ready,
# so the MariaDB CRD exists and has a controller before this applies.
- name: infrastructure
+54
View File
@@ -0,0 +1,54 @@
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: ak-pg-store
namespace: authentik
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled
spec:
configuration:
destinationPath: ${CNPG_AK_PG_ARCHIVE}
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
retentionPolicy: "30d"
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: ak-pg-store-source
namespace: authentik
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled
cnpg.renner.uno/server-name: ak-pg
spec:
configuration:
destinationPath: ${CNPG_AK_PG_SOURCE}
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
+54
View File
@@ -0,0 +1,54 @@
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: ha-pg-store
namespace: home-assistant
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled
spec:
configuration:
destinationPath: ${CNPG_HA_PG_ARCHIVE}
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
retentionPolicy: "30d"
---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: ha-pg-store-source
namespace: home-assistant
annotations:
kustomize.toolkit.fluxcd.io/prune: disabled
cnpg.renner.uno/server-name: ha-pg
spec:
configuration:
destinationPath: ${CNPG_HA_PG_SOURCE}
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip

Some files were not shown because too many files have changed in this diff Show More