Compare commits

...
Author SHA1 Message Date
claude-botandClaude Opus 5.5 be643533f6 feat(talos): node IP and etcd on VLAN 46 (#576 step 3)
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Records the live state after moving all three nodes:
- site-homelab.yaml: kubelet nodeIP.validSubnets fd02:46::/64 and
  10.2.46.0/24 (was 10.2.40.0/24 and the legacy fd40::/64).
- cluster.yaml: etcd advertisedSubnets 10.2.46.0/24.
- mgmt0 comments: VLAN 40 is world-to-node only now.

Applied per node (k8s-3, k8s-2, k8s-1): drain, stage the config (delete
the old lists, set the new ones, one staged patch call), etcd
forfeit-leadership and leave, reboot. Each node rejoined etcd with its
VLAN 46 peer URL; EPHEMERAL was not wiped.

Refs #576

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 14:43:17 +02:00
k.renner 311b4639a6 Merge pull request 'feat(bird): put node0 in the pod routing table (#576 step 2)' (#919) from feat/bird-attached-node0 into master
ci/woodpecker/cron/e2e Pipeline failed
ci/woodpecker/cron/security-scan Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#919
2026-09-23 11:31:32 +00:00
k.renner a95efdef33 Merge pull request 'feat(talos): node network on VLAN 46, etcd pinned to VLAN 40 (#576 step 1)' (#918) from feat/node-vlan-46-link into master
Reviewed-on: k.renner/homelab#918
2026-09-23 11:31:23 +00:00
claude-botandClaude Opus 5.5 aea4be734c feat(bird): put node0 in the pod routing table (#576 step 2)
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Pod-sourced traffic is routed from table 47. Once the node IP moves to
VLAN 46 (step 3), pods reach node addresses there: the kubernetes
ClusterIP (socket LB rewrites it to the apiserver's advertise address),
hostNetwork pod IPs, and replies to kubelet probes. Without the attached
node0 route in table 47 those flows take the default to the UDM and every
in-cluster API client breaks. Taking it from the interface, like the
other attached networks, keeps the table in one owner.

Refs #576

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 13:28:57 +02:00
claude-botandClaude Opus 5.5 07dd6097ec feat(talos): node network on VLAN 46, etcd pinned to VLAN 40 (#576 step 1)
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
- node0: VLANConfig on the uplink trunk, VLAN 46, static fd02:46::1x/64
  and 10.2.46.1x/24, accept_ra 0. Closed like pod0. Unused until the node
  IP moves there (step 3).
- cluster.etcd.advertisedSubnets pinned to 10.2.40.0/24 so a reboot
  cannot pick a node0 address: Talos never updates an existing member's
  peer URLs.

Applied live with talosctl patch --mode=no-reboot, k8s-3 first. Node IPs
unchanged, etcd 3/3 healthy, node0 reachable node to node over v6 and v4.

Refs #576

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 13:25:12 +02:00
k.renner 60361d4c8a Merge pull request 'chore(talos): drop the #832 stage labels' (#917) from chore/cni-drop-stage-labels into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#917
2026-09-23 10:46:35 +00:00
claude-botandClaude Opus 5.5 a8cef5de1b chore(talos): drop the #832 stage labels
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Nothing selects cni.renner.uno/cilium-uninstall or
cni.renner.uno/multus-sole-writer since stage B (#915, #916). Removed
live from all three nodes with a $patch: delete strategic merge patch
(--mode=no-reboot); Talos dropped them from the Kubernetes nodes and
left the other labels in place.

Closes #832

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 12:40:10 +02:00
k.renner 505802dad5 Merge pull request 'feat(cilium): write the conflist where only Multus reads it (stage B2 of #832)' (#916) from feat/cni-sole-writer-converge-b2 into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#916
2026-09-23 10:38:00 +00:00
claude-botandClaude Opus 5.5 ba66b604d4 feat(cilium): write the conflist where only Multus reads it (stage B2 of #832)
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
cni.confPath: /etc/cni/net.d/multus mounts that host directory as the
agent's /host/etc/cni/net.d, so write-cni-conf-when-ready (unchanged,
.../05-cilium.conflist) lands in /etc/cni/net.d/multus/05-cilium.conflist,
the file the per-node CiliumNodeConfigs already produced on all three
nodes. The agent finds it unchanged and does not rewrite it.

updateStrategy maxUnavailable: 1 (chart default 2) so the roll takes one
node's datapath at a time.

bootstrap.sh renders the chart from these values, so a cold rebuild
lands directly in this layout.

Refs #832

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 11:55:12 +02:00
k.renner 4fbc49fc8c Merge pull request 'feat(cni): converge Multus as sole CNI writer, part 1 (stage B1 of #832)' (#915) from feat/cni-sole-writer-converge-b1 into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#915
2026-09-23 09:53:24 +00:00
claude-botandClaude Opus 5.5 f493c9fe52 feat(cni): converge Multus as sole CNI writer, part 1 (stage B1 of #832)
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
All three nodes run the sole-writer layout. Converge the Multus side and
drop the per-node Cilium overrides:
- kube-multus-ds-subdir becomes the only Multus DaemonSet, no node
  affinity; kube-multus-ds (0 pods) and its ConfigMap are pruned.
- /etc/cni/net.d/multus is mounted DirectoryOrCreate so Multus waits for
  the Cilium conflist instead of exiting on a missing directory.
- CiliumNodeConfigs removed. They only act when a cilium pod is created;
  stage B2 moves the same path into the HelmRelease (cni.confPath) and
  rolls the agents.

Refs #832

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 11:43:48 +02:00
k.renner 8a3c66eb2e Merge pull request 'chore(talos): record #832 stage labels on k8s-1' (#914) from chore/cni-stage-k8s-1 into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#914
2026-09-23 09:31:07 +00:00
claude-botandClaude Opus 5.5 9928999da4 chore(talos): record #832 stage labels on k8s-1
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
k8s-1 carries cni.renner.uno/cilium-uninstall and
cni.renner.uno/multus-sole-writer, applied live with talosctl patch.
Verified on the node: /etc/cni/net.d holds only 00-multus.conf at the
top level, and after a reboot with immediate uncordon the OSD came up
with net1 (10.99.0.1) and no restarts. All three nodes are converted.

Refs #832

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 11:15:45 +02:00
k.renner b3def28f7c Merge pull request 'fix(whereabouts): let the reconciler respect cordons' (#913) from fix/whereabouts-reconciler-cordon into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#913
2026-09-23 09:04:29 +00:00
claude-botandClaude Opus 5.5 93dbcdb2fb fix(whereabouts): let the reconciler respect cordons
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The Deployment tolerated every NoSchedule taint (operator: Exists, no
key), including node.kubernetes.io/unschedulable. During the #832 node
reboots its ReplicaSet kept placing replacement pods on the cordoned,
shutting-down node; the kubelet rejected each with NodeShutdown, about
200 failed pods per reboot. Tolerate only the control-plane taint the
comment names.

Refs #832

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 10:56:50 +02:00
k.renner 47bf16cfac Merge pull request 'chore(talos): record #832 stage labels on k8s-2' (#912) from chore/cni-stage-k8s-2 into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#912
2026-09-23 08:45:30 +00:00
claude-botandClaude Opus 5.5 4664de8f97 chore(talos): record #832 stage labels on k8s-2
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
k8s-2 carries cni.renner.uno/cilium-uninstall and
cni.renner.uno/multus-sole-writer, applied live with talosctl patch.
Verified on the node: /etc/cni/net.d holds only 00-multus.conf at the
top level, and after a reboot with immediate uncordon the OSD came up
with net1 (10.99.0.2) and no restarts.

Refs #832

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 10:44:17 +02:00
k.renner fb52e4be88 Merge pull request 'chore(talos): record #832 stage labels on k8s-3' (#911) from chore/cni-stage-k8s-3 into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#911
2026-09-23 08:28:40 +00:00
claude-botandClaude Opus 5.5 f11a71c33f chore(talos): record #832 stage labels on k8s-3
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
k8s-3 carries cni.renner.uno/cilium-uninstall and
cni.renner.uno/multus-sole-writer, applied live with talosctl patch.
Verified on the node: /etc/cni/net.d holds only 00-multus.conf at the
top level, Cilium writes multus/05-cilium.conflist, and after a reboot
with immediate uncordon the OSD came up with net1 (10.99.0.3) and no
restarts.

Refs #832

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 10:25:53 +02:00
k.renner 29326eead3 Merge pull request 'feat(cni): per-node switch for Multus as sole CNI writer (stage A of #832)' (#910) from feat/cni-staged-sole-writer into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#910
2026-09-23 07:52:22 +00:00
claude-botandClaude Opus 5.5 eb09b1b2a3 feat(cni): per-node switch for Multus as sole CNI writer (stage A of #832)
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Nothing changes until a node is labelled:
- post-crds/cilium-nodeconfig.yaml: CiliumNodeConfig cni-uninstall
  (label cni.renner.uno/cilium-uninstall) arms the agent's preStop
  conflist cleanup; multus-sole-writer (label
  cni.renner.uno/multus-sole-writer) moves write-cni-conf-when-ready to
  /host/etc/cni/net.d/multus/05-cilium.conflist, a subdirectory libcni
  does not read.
- multus/multus-subdir.yaml: kube-multus-ds-subdir with
  multusAutoconfigDir /host/etc/cni/net.d/multus, scheduled only on
  labelled nodes; kube-multus-ds gets the inverse affinity.

The kube-multus-ds template change rolls Multus node by node once on
merge.

Refs #832

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 09:47:16 +02:00
k.renner dd3a1f1e90 Merge pull request 'feat(netbird-service-sync): run the controller in its own namespace (stage 2 of #876)' (#909) from feat/netbird-service-sync-phase2 into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#909
2026-09-23 06:55:25 +00:00
claude-botandClaude Opus 5.5 98a0462137 feat(netbird-service-sync): run the controller in its own namespace (stage 2 of #876)
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
- kubernetes/infrastructure/netbird-proxy-services -> netbird-service-sync:
  SA, ClusterRole(Binding) and Deployment renamed, in namespace
  netbird-service-sync, image gitea_admin/netbird-service-sync pinned to
  the master build digest, BAO_ROLE=netbird-service-sync.
- Old policy doc netbird-proxy-services.hcl removed; Traefik comment
  follows the name.

The infrastructure KS owns both old and new objects, so prune and apply
happen in one pass. OpenBao policy and role netbird-service-sync are
already written.

Closes #876

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 08:41:20 +02:00
k.renner 9aef2e0669 Merge pull request 'feat(netbird-service-sync): prepare rename and own namespace (stage 1 of #876)' (#908) from feat/netbird-service-sync-phase1 into master
ci/woodpecker/push/build-ci-images Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#908
2026-09-23 06:35:14 +00:00
claude-botandClaude Opus 5.5 fd7597ddc2 chore(netbird-service-sync): drop stray __pycache__
ci/woodpecker/push/build-ci-images Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 08:30:28 +02:00
claude-botandClaude Opus 5.5 10b0c1b853 feat(netbird-service-sync): prepare rename and own namespace (stage 1 of #876)
ci/woodpecker/push/build-ci-images Pipeline was canceled
Additive; the running controller is untouched:
- ci/images/netbird-proxy-services -> ci/images/netbird-service-sync,
  build step and image name renamed, so the merge builds
  gitea_admin/netbird-service-sync.
- Namespace netbird-service-sync with a baseline policy: probes in;
  out only to kube-dns, kube-apiserver, OpenBao :8200 and world :443
  (NetBird management API).
- openbao-ns admits netbird-service-sync on :8200, like wiki-index.
- policies/netbird-service-sync.hcl documents the new policy and role
  (applied by hand, as before).

Refs #876

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 08:30:15 +02:00
k.renner 674169cb08 Merge pull request 'chore(gitea): drop the apps-namespace grants for Gitea (stage 3 of #878)' (#907) from chore/gitea-namespace-phase3 into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#907
2026-09-23 06:20:17 +00:00
claude-botandClaude Opus 5.5 0c6174b7a6 chore(gitea): drop the apps-namespace grants for Gitea (stage 3 of #878)
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Gitea runs in the gitea namespace since #906. Remove what only served it
in apps:
- openbao rotator egress (flux-git, review-bot): apps/gitea target.
- ai-ns wiki-index ingress: apps/gitea source.
- apps default-deny egress to wiki-index :8080 (gitea's webhook; now in
  gitea-ns.yaml).
- Velero schedule comments.

Refs #878

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 08:09:30 +02:00
k.renner 76888c3771 Merge pull request 'feat(gitea): move Gitea into the gitea namespace (stage 2 of #878)' (#906) from feat/gitea-namespace-phase2 into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#906
2026-09-23 06:05:33 +00:00
claude-botandClaude Opus 5.5 eb23d3ac6e feat(gitea): move Gitea into the gitea namespace (stage 2 of #878)
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
- kubernetes/apps/gitea: every object in namespace gitea; applied by a
  new gitea Flux Kustomization (clusters/homelab/gitea.yaml) instead of
  apps. gitea-allow-metrics-scrape dropped: the gitea-ns baseline
  already admits monitoring.
- Rotators (flux-git, review-bot) and the Prometheus gitea job use
  gitea.gitea.svc.
- bootstrap.sh waits for the gitea KS.

Data: data-gitea-0 is not copied. The PV (reclaimPolicy Retain) is
rebound to a PVC gitea/data-gitea-0 pre-created with volumeName, which
the new StatefulSet adopts. gitea-tls is pre-issued in the gitea
namespace so HTTPS has no certificate gap.

Refs #878

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 07:44:26 +02:00
k.renner c1e3f3fb98 Merge pull request 'feat(gitea): prepare the gitea namespace (stage 1 of #878)' (#905) from feat/gitea-namespace-phase1 into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#905
2026-09-23 05:35:18 +00:00
claude-botandClaude Opus 5.5 036399095b feat(gitea): prepare the gitea namespace (stage 1 of #878)
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Additive only, nothing moves yet:
- Namespace gitea, declared in infrastructure like woodpecker.
- network-policies/gitea-ns.yaml: baseline (same-ns, monitoring, host,
  ICMP; egress to traefik, kube-dns, wiki-index, world) and
  gitea-allow-traefik for :3000 and :2222.
- openbao rotator egress (flux-git, review-bot) and wiki-index ingress
  also admit gitea in the gitea namespace; the apps entries stay until
  stage 3.
- Velero schedule gitea-daily (07:45) so data-gitea-0 stays backed up
  after the move; apps-daily comment updated (Woodpecker left apps).

Refs #878

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 07:30:36 +02:00
k.renner 0bac0051b2 Merge pull request 'chore(ci): bump ci-claude image to 2.1.280' (#904) from ci-image/claude-2.1.280 into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#904
2026-09-23 05:16:03 +00:00
claude-bot 41a65b65a3 chore(ci): bump ci-claude image to 2.1.280
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
2026-09-23 05:13:16 +00:00
k.renner a530f7c8e6 Merge pull request 'fix(ci-claude-refresh): bump divergent pins instead of aborting' (#903) from fix/ci-claude-refresh-mixed-pins into master
ci/woodpecker/push/ci-claude-refresh Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#903
2026-09-23 05:12:40 +00:00
claude-botandClaude Opus 5.5 8fff4b4012 fix(ci-claude-refresh): bump divergent pins instead of aborting
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
decide required every consumer to pin the same ci-claude tag and aborted
otherwise (pipeline 10497: [2.1.250 2.1.278]), which needed a hand fix
before the latest version could roll out. Record all distinct tags, bump
whenever any is off latest, and rewrite every version-pinned tag to the
new one with a validated x.y.z version.

Merging this is itself a push touching the pipeline file, so the refresh
runs, builds the latest image, and opens the bump PR.

Refs #889

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 07:06:56 +02:00
k.renner 0ada3df260 Merge pull request 'fix(ci-claude-refresh): strip CR/LF from the bump token before use' (#901) from fix/ci-claude-bump-token-newline into master
ci/woodpecker/push/ci-claude-refresh Pipeline failed
Reviewed-on: k.renner/homelab#901
2026-09-23 05:03:19 +00:00
claude-botandClaude Opus 5.5 2144554cfc fix(ci-claude-refresh): strip CR/LF from the bump token before use
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Pipeline 10186 failed in bump-pr with 'url contains a newline in its
username component': ci_claude_bump_bot_token carries a trailing newline,
so the push URL (and the PR API auth header) broke. Strip CR/LF once at
the top of the step and fail fast if the token ends up empty. This makes
the step independent of how the secret value was written.

Closes #889

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 06:59:15 +02:00
k.renner dba810b87b Merge pull request 'fix(paperless): raise the memory limit to 4Gi' (#900) from worktree-fix-paperless-memory-limit into master
ci/woodpecker/cron/e2e Pipeline failed
ci/woodpecker/cron/security-scan Pipeline was successful
ci/woodpecker/cron/wiki-maintenance Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#900
2026-09-21 19:29:08 +00:00
claude-botandClaude Fable 5.1 f8aa65362b fix(paperless): cap OCR at 5 threads per worker
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Unset, paperless derives cpu_count / TASK_WORKERS, i.e. 20 parallel
ocrmypdf workers on k8s-1. Memory during OCR scales with that count, so
the limit alone is not enough headroom for large scans.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-21 21:26:35 +02:00
claude-botandClaude Fable 5.1 0ce0391bb1 fix(paperless): raise the memory limit to 4Gi
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
An 18 MB scan from the ADS-1700W has kept the paperless container in an
OOM crash loop since 2026-09-21 10:21: ocrmypdf runs with 20 workers
(paperless default, cpu_count / TASK_WORKERS on a 20-core node) and blows
the 2Gi limit ~40s into OCR. The consume task is re-queued on every
restart, so the same file kills every attempt. 117 restarts so far, web
UI down as a side effect since worker and webserver share the container.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-21 21:24:36 +02:00
security-bot 4611ea3578 chore(image-automation): bump redis:8.10.2-alpine
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-09-21 18:52:58 +00:00
k.renner 63701fbebb Merge pull request 'ci(wiki-maintenance): verify the PR merged instead of trusting the merge status' (#895) from fix/wiki-maintenance-merge-verify into master
ci/woodpecker/manual/wiki-maintenance Pipeline was successful
ci/woodpecker/cron/e2e Pipeline failed
ci/woodpecker/cron/security-scan Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#895
2026-09-21 08:49:10 +00:00
k.renner e8a047eb14 Merge pull request 'ci(pick-model): newest among comparable candidates, policy in code' (#894) from feat/pick-model-newest-among-comparable into master
Reviewed-on: k.renner/homelab#894
2026-09-21 08:48:58 +00:00
claude-botandClaude Fable 5.1 0279210595 ci(wiki-maintenance): verify the PR merged instead of trusting the merge status
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Run 10254: POST .../merge with delete_branch_after_merge returned
HTTP 500 although PR #2 on k.renner/notes was merged, so the step
failed and left the branch behind. The merge call no longer asks Gitea
to delete the branch; the step reads the PR back and requires
merged=true, then deletes the branch as a separate best-effort call.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-21 10:06:23 +02:00
claude-botandClaude Fable 5.1 25f09e9b52 ci(pick-model): newest among comparable candidates, policy in code
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
First live pick after #893 chose claude-sonnet-4-5 at 0.23 over
claude-sonnet-4-6: same-family models split the probability for a
mid-weight task, and "prefer newer" is a policy, not a judgment. New
scripts/ci/choose-model.sh applies it to the distribution: among
candidates within CHOOSE_MODEL_MARGIN (0.7) of the best probability the
most recently released wins; a clear best stays; NONE yields nothing.
The picker logs the top probabilities. Four policy tests added.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-21 09:59:40 +02:00
k.renner 82f2825d3f Merge pull request 'ci(pick-model): choose the model with one TypeSafe Choice, claude as fallback' (#893) from feat/pick-model-typesafe into master
ci/woodpecker/manual/wiki-maintenance Pipeline failed
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#893
2026-09-21 07:56:11 +00:00
claude-botandClaude Fable 5.1 0fec080b4d ci(pick-model): choose the model with one TypeSafe Choice, claude as fallback
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The picker ran a full claude -p in plan mode per selection and parsed
JSON out of prose. With TYPESAFE_API_KEY set it now sends the task and
the catalog rows as state and asks one Choice over the model ids, each
with its catalog row as criterion plus NONE; the id is validated against
the pristine catalog as before. No key or a failed request falls through
to the claude picker unchanged.

Measured on the two-model fixture (Opus 5 / Haiku 4.5): issue-fix and
security-fix -> Opus 0.99, alert triage -> Opus 0.62, wiki audit ->
Haiku 0.94, update audit -> Haiku 0.46, distill -> Haiku 0.63; about a
second per pick. Dry-run test added to test-model-selection.sh.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-21 09:42:39 +02:00
k.renner 7424c6aefd Merge pull request 'ci(wiki-maintenance): rebase onto main before pushing, retry three times' (#892) from fix/wiki-maintenance-rebase-push into master
ci/woodpecker/manual/wiki-maintenance Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#892
2026-09-21 06:41:40 +00:00
claude-botandClaude Fable 5.1 4e3fb88309 ci(wiki-maintenance): push a branch and merge it through Gitea
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Instead of rebasing onto a moving main: the run's changes go to
wiki-maintenance/<date>-<run>, a PR is opened and merged via the API
(Do: merge, branch deleted afterwards). A merge conflict leaves the PR
open for a human and fails the run. Fields checked against the
instance's swagger.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-21 08:40:47 +02:00
claude-botandClaude Fable 5.1 c8c5ce9015 ci(wiki-maintenance): rebase onto main before pushing, retry three times
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Run 10241: distill and audit succeeded (5 sources, 8 new + 6 updated
notes), the push was rejected non-fast-forward because the workstation
pushed the vault while the agents ran. The push step now fetches,
rebases and retries; a real conflict still fails the run.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-21 08:36:58 +02:00
k.renner 293f51b914 Merge pull request 'ci(wiki-maintenance): strip CR/LF from secrets before use' (#891) from fix/wiki-maintenance-strip-secrets into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
ci/woodpecker/manual/wiki-maintenance Pipeline failed
Reviewed-on: k.renner/homelab#891
2026-09-21 06:22:44 +00:00
claude-botandClaude Fable 5.1 70287db150 ci(wiki-maintenance): allow hand-started runs via the manual event
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
A deployment-event run (10239) errored before its first step: the
typesafe_api_key secret is scoped to cron and manual only. Manual
pipelines (woodpecker pipeline create) are the way to start this by
hand; the deployment trigger is dropped.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-21 08:20:36 +02:00
claude-botandClaude Fable 5.1 efedba1ff0 ci(wiki-maintenance): strip CR/LF from secrets before use
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was canceled
First cron run (10220) failed in checkout-notes with "could not read
Username": the wiki_bot_token secret had been set from a file with a
trailing newline, which git carried into the extraheader. The secret
is rewritten without it; the pipeline now strips CR/LF from the token
and the index credential regardless of how they were entered.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-21 08:19:12 +02:00
security-bot f35a2d5985 chore(image-automation): bump ghcr.io/paperless-ngx/paperless-ngx:3.2.1
ci/woodpecker/cron/wiki-maintenance Pipeline failed
ci/woodpecker/cron/e2e Pipeline failed
ci/woodpecker/cron/update-audit Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-09-20 21:51:31 +00:00
k.renner 19b641a2b6 Merge pull request 'ci(wiki-maintenance): run on ci-claude 2.1.278' (#890) from fix/wiki-maintenance-image-2.1.278 into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#890
2026-09-20 20:46:19 +00:00
claude-botandClaude Fable 5.1 e422e6a13d ci(wiki-maintenance): run on ci-claude 2.1.278
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The refresh triggered by #888 built 2.1.278 (the first ci-claude with
python3) rather than rebuilding 2.1.250, and its bump-pr step failed on
a newline in the push token secret (#889), so the consumer bump never
came. This pipeline needs python for wiki_ask.py, so it moves ahead of
the general bump.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-20 22:45:02 +02:00
k.renner f4d590c40d Merge pull request 'feat(ci): nightly wiki maintenance for the Notes vault as a Woodpecker cron' (#888) from feat/wiki-maintenance-ci into master
ci/woodpecker/push/build-ci-images Pipeline was successful
ci/woodpecker/push/ci-claude-refresh Pipeline failed
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#888
2026-09-20 20:41:58 +00:00
claude-botandClaude Fable 5.1 791cb6e524 feat(ci): nightly wiki maintenance for the Notes vault as a Woodpecker cron
ci/woodpecker/push/build-ci-images Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Replaces the workstation's wiki-maintenance.timer. The pipeline clones
k.renner/notes, runs wiki-distill over raw/ and wiki-audit over wiki/
as two separate `claude --agent` runs (no background tasks in print
mode, which killed the workstation job at the 600 s ceiling), then
commits and pushes as Claude Bot; the push webhook refreshes wiki-index.

The agent definitions now live in the vault (.claude/agents/), so CI
and workstation share them. The agents and their retrieval scripts
address the vault as ~/Notes; each agent step links $HOME/Notes to the
checkout. ci-claude gains python3 and py3-yaml for wiki_ask.py.

Secrets on the repo: claude_code_oauth_token (existing), typesafe_api_key,
wiki_bot_token (claude-bot, write:repository), wiki_index_basic_auth.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-20 22:37:59 +02:00
k.renner 3749df798e Merge pull request 'ci(wiki-index): per-commit image tags, deployment pinned to one' (#886) from fix/wiki-index-commit-tags into master
ci/woodpecker/push/build-ci-images Pipeline was successful
ci/woodpecker/cron/e2e Pipeline failed
ci/woodpecker/cron/security-scan Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
Reviewed-on: k.renner/homelab#886
2026-09-20 09:00:25 +00:00
claude-botandClaude Fable 5.1 50b4cbeb2b fix(wiki-index): pin the deployment to the per-commit tag
ci/woodpecker/pr/ci-manifests-test Pipeline failed
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
wiki-index:2026-09-20-4b529cd1 is the build of the previous commit,
i.e. master's code with the paging fix and /ready. Manifest-only
commit, so it does not trigger another build.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-20 10:55:57 +02:00
claude-botandClaude Fable 5.1 4b529cd15c ci(wiki-index): also tag the image per commit
ci/woodpecker/push/build-ci-images Pipeline was successful
Three builds on 2026-09-20 (two branch pushes, the merge) rewrote
wiki-index:2026-09-20 in an order the digest pin could not follow: the
pinned digest ran the paging-only build without /ready and crash-looped
on its startup probe. A <date>-<sha8> tag is immutable per commit and
lets the deployment pin by tag without a registry lookup.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-20 10:55:38 +02:00
k.renner 53a9348254 Merge pull request 'fix(wiki-index): pin the image to the #882 build' (#885) from fix/wiki-index-pin-digest-2 into master
Reviewed-on: k.renner/homelab#885
2026-09-20 08:49:45 +00:00
claude-botandClaude Fable 5.1 ce06d4e7da fix(wiki-index): pin the image to the #882 build
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Same dated tag again; the pod holds the #879 build. Pinning the new
digest (paging fix, /ready readiness) rolls it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-20 10:49:13 +02:00
k.renner e716632c5c Merge pull request 'fix(wiki-index): stop tree paging on the short last page' (#882) from fix/wiki-index-tree-paging into master
ci/woodpecker/push/build-ci-images Pipeline was successful
Reviewed-on: k.renner/homelab#882
2026-09-20 08:46:35 +00:00
claude-botandClaude Fable 5.1 9736656fed fix(wiki-index): liveness on /health, readiness on /ready
ci/woodpecker/push/build-ci-images Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
/health answered 503 until an index existed and fed startup and
liveness, so a live process without an index counted as dead and was
restarted every 15 min, and a later failed sync would have pulled a pod
with a good index out of the Service. /health is now always 200
(process serves); /ready is 200 once an index exists and stays so
across failed syncs, which keep the last good index. Probes follow:
startup 60 s on /health, readiness on /ready, liveness on /health.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-20 10:45:50 +02:00
claude-botandClaude Fable 5.1 d4e5029ebd fix(wiki-index): stop tree paging on the short last page
ci/woodpecker/push/build-ci-images Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was canceled
The notes repo tree has 3192 entries. Gitea reports truncated=true on
every page including the last one (page 4, 192 entries) and answers
"tree": null one page further, which the sync hit:
"sync failed: 'NoneType' object is not iterable". Paging now stops on a
short page or when total_count is reached, and a null tree is empty.
Unit test with the observed page shapes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-20 10:44:10 +02:00
k.renner 0fb5ff3fb6 Merge pull request 'fix(wiki-index): pin the image to the rebuilt digest' (#881) from fix/wiki-index-pin-digest into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#881
2026-09-20 08:40:41 +00:00
claude-botandClaude Fable 5.1 58e4c25729 fix(wiki-index): pin the image to the rebuilt digest
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
#879 rebuilt wiki-index under the same dated tag 2026-09-20; the node
still holds the first build (sha256:085438d8) and IfNotPresent would
not re-pull. Pinning tag@digest rolls the exact new image.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-20 10:38:45 +02:00
k.renner a3ca6c6065 Merge pull request 'fix(wiki-index): strip OpenBao values, retry a failed sync after 60 s' (#879) from fix/wiki-index-strip-credentials into master
ci/woodpecker/push/build-ci-images Pipeline was successful
Reviewed-on: k.renner/homelab#879
2026-09-20 08:36:03 +00:00
claude-botandClaude Fable 5.1 816f161591 fix(wiki-index): strip OpenBao values, retry a failed sync after 60 s
ci/woodpecker/push/build-ci-images Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
`bao kv put key=@file` stores the file's trailing newline; the first
sync after #875 failed with "Invalid header value b'token ...\n'" and a
webhook signed with the newline-free secret would never have verified.
Credentials are stripped on read. A failed sync now retries after 60 s
instead of waiting the full RESYNC_SECONDS.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-20 10:29:43 +02:00
k.renner 5f86bdde64 Merge pull request 'fix(netpol): let gitea deliver the notes push webhook to wiki-index' (#877) from fix/gitea-webhook-egress-wiki-index into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#877
2026-09-20 08:24:40 +00:00
claude-botandClaude Fable 5.1 814c5b88fa fix(netpol): let gitea deliver the notes push webhook to wiki-index
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The apps namespace is egress default-deny; gitea's webhook to
wiki-index.ai.svc.cluster.local:8080 was dropped (a test delivery
produced no request at the service). Namespace-wide egress rule scoped
to the wiki-index pod on 8080, the mosquitto pattern; wiki-index's own
ingress policy already admits only the gitea pod from apps.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-20 10:21:18 +02:00
k.renner 0420be0a21 Merge pull request 'fix(openbao): admit wiki-index to OpenBao on 8200' (#875) from fix/wiki-index-openbao-ingress into master
Reviewed-on: k.renner/homelab#875
2026-09-20 08:18:02 +00:00
claude-botandClaude Fable 5.1 7da72b34cf fix(openbao): admit wiki-index to OpenBao on 8200
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The openbao namespace policy admits only its own pods, Traefik and
External Secrets. wiki-index (ns ai) logs in through the Kubernetes auth
role on every sync and timed out on the first attempt:
"POST .../v1/auth/kubernetes/login: timed out". Pod-scoped ingress rule
for it, mirroring the external-secrets entry.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-20 10:14:47 +02:00
k.renner c36c290b13 Merge pull request 'feat(ci): wiki-index image — embedding index over the Notes repo' (#873) from feat/wiki-index-image into master
ci/woodpecker/push/build-ci-images Pipeline failed
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#873
2026-09-20 08:11:32 +00:00
k.renner 8fb7430fb7 Merge pull request 'feat(ai): deploy wiki-index, the embedding index service for the Notes repo' (#874) from feat/wiki-index-deploy into master
Reviewed-on: k.renner/homelab#874
2026-09-20 08:11:21 +00:00
claude-botandClaude Fable 5.1 d82b96a751 fix(wiki-index): notes repository branch is main
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-20 10:09:43 +02:00
claude-botandClaude Fable 5.1 463cfa7bde feat(ai): deploy wiki-index, the embedding index service for the Notes repo
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Deployment + Service + PVC + ServiceAccount in namespace ai, LAN-only
Ingress wiki-index.renner.uno behind Traefik BasicAuth (ExternalSecret
from OpenBao, same shape as wiki-embed), Flux Kustomization depending on
llama-server (wiki-embed is the embedding backend), pod-scoped
CiliumNetworkPolicy admitting Traefik and Gitea (push webhook) on 8080
and allowing egress to OpenBao 8200, and the OpenBao policy for the
service's Kubernetes auth role.

Image: gitea.renner.uno/gitea_admin/wiki-index:2026-09-20, built by the
step added in the previous PR; the tag is the build date.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-20 08:47:26 +02:00
claude-botandClaude Fable 5.1 48ba19064a feat(ci): wiki-index image — embedding index over the Notes repo
ci/woodpecker/push/build-ci-images Pipeline failed
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Runtime image for a wiki-index service in namespace ai (manifests follow
once the image exists): standard-library Python that mirrors the notes
repository's wiki/ from Gitea into a bge-m3 index and serves POST /rank
to ~/Notes/bin/wiki_rank.py.

Sync fetches the branch head and tree via the Gitea API, embeds only notes
whose blob sha changed (through the in-cluster wiki-embed), drops notes
that left the tree, and persists the index on a PVC. It runs at start, on
every push webhook (HMAC-SHA256 verified) and every RESYNC_SECONDS. Gitea
token and webhook secret come from OpenBao through the Kubernetes auth
role on every use. Five unit tests cover parsing, incremental sync,
persistence, ranking and error handling.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-20 08:42:03 +02:00
k.renner 761bf9ba1e Merge pull request 'fix(eso): allow namespace ai on the openbao ClusterSecretStore' (#872) from fix/eso-allow-ai into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#872
2026-09-20 05:48:21 +00:00
claude-botandClaude Fable 5.1 7b446194b3 fix(eso): allow namespace ai on the openbao ClusterSecretStore
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The wiki-embed-basic-auth ExternalSecret from #871 never synced:
"using cluster store openbao is not allowed from namespace ai: denied by
spec.condition". The store's namespace allowlist predates the ai
namespace. Adding it, sorted, as the manifest's own maintenance note
prescribes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-20 07:47:43 +02:00
k.renner 9eaafeb92f Merge pull request 'feat(ai): dedicated wiki-embed endpoint for the Notes ranker' (#871) from feat/wiki-embed into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#871
2026-09-20 05:38:26 +00:00
claude-botandClaude Fable 5.1 67d506e1ac feat(ai): dedicated wiki-embed endpoint for the Notes ranker
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Second bge-m3 llama.cpp Deployment in ns ai so wiki queries never queue
behind Paperless on llama-embed. Shares the llama-server-models PVC, so
the weights are not downloaded again.

Exposed LAN-only as wiki-embed.renner.uno (no dns.renner.uno/public
label) for bin/wiki_rank.py on the workstation and the nightly wiki jobs.
The client is a script, and the Authentik gate is browser-only, so the
Ingress uses Traefik BasicAuth backed by an ExternalSecret, the same
shape as loki.renner.uno. A pod-scoped CiliumNetworkPolicy admits Traefik
to wiki-embed only; llama-server stays reachable solely via its gate.

Measured on 40 past wiki-synth queries: embeddings picking 150 candidates
plus one TypeSafe Choice puts a cited note in the top 5 for 95% of
queries at a sixth of the cost of ranking all 1700 notes with TypeSafe.

Needs before first use: OpenBao key apps/ai/wiki-embed-basic-auth with
field `users` (htpasswd line), and the same credential as user:password
in ~/.config/wiki-embed/basic-auth on the workstation.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-19 23:23:30 +02:00
security-bot 932d07edbe chore(image-automation): bump ghcr.io/paperless-ngx/paperless-ngx:3.2.0
ci/woodpecker/cron/security-scan Pipeline was successful
ci/woodpecker/cron/e2e Pipeline failed
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-09-19 02:51:13 +00:00
security-bot 1a1205110d chore(image-automation): bump ghcr.io/home-assistant/home-assistant:2026.9.3
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-09-18 12:53:31 +00:00
security-bot aed827ae5b chore(image-automation): bump ghcr.io/music-assistant/server:2.10.4
ci/woodpecker/cron/e2e Pipeline failed
ci/woodpecker/cron/security-scan Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-09-18 09:51:06 +00:00
k.renner 3abcec5af7 Merge pull request 'feat(heating-can): read the One Base per-room telemetry over CAN' (#867) from feat/onebase-room-telemetry into master
ci/woodpecker/cron/e2e Pipeline failed
ci/woodpecker/cron/security-scan Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#867
2026-09-17 11:17:30 +00:00
claude-botandClaude Opus 5 b14bf0cc5e feat(heating-can): read the One Base per-room telemetry over CAN
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The Einzelraumsteuerung (8 radiator valves, 9 climate sensors, 9 underfloor
actuators, 3 floor thermostats) runs on Viessmann's One Base ZigBee system.
Its room and device registries and its live per-device values sit on ECU 0x6c3
(BACKENDGATEWAY) and are readable in plaintext over the E3 bus -- room
temperature, the active target temperature, humidity, valve opening, demand,
battery and RSSI. That replaces the ViCare cloud room sensors with a local
source.

0x6c3 had never been enumerated: only 0x680, 0x6a1 and 0x68c ever had a full
scan. A full read of 0x6c3 returns 155 decodable datapoints plus 35 that exist
but fail the stock codec.

Those 35 are the reason for the overlay. open3e declares
ZigBee<N>DeviceCurrentValues as 57 bytes; a PAIRED slot answers with 68 -- the
known 57 plus the parent device serial and three trailing bytes. udsoncan reads
only the declared length and then treats the next two bytes, the first half of
that serial, as a foreign data identifier, so every paired slot fails with
"Server returned values for data identifier 0x.... that was not requested". The
bogus identifier is stable per device and differs between devices because the
serial does, which is why this previously read as an encrypted channel. It is
not encrypted.

The overlay re-declares the 29 paired slots at 68 bytes and inherits upstream's
sub-type list, so the per-device-type switch -- including
DeviceTemperatureSetpoint -- comes from upstream rather than being copied.
Empty slots answer with 57, decode against the stock codec and are not polled.

Only CurrentValues are polled. The registries are static configuration and are
read on demand.

Verified against payloads captured from 0x6c3 on 2026-09-17, decoded with
open3e 0.7.5 (the commit pinned in the heating-can image): all 29 slots are 68
bytes, and a radiator valve yields ActualTemperature 19.3 °C, ValveOpening 0 %,
DeviceTemperatureSetpoint 8.0 °C, battery 38 %, RSSI -69 dBm. The setpoint
field was confirmed by changing one room in the ViCare app three times and
re-reading: 8.0 -> 15.0 -> 13.0.

Read-only. No write path to the per-room setpoint is known; writes to the
CurrentValues DID were attempted and had no effect.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 13:13:49 +02:00
k.renner 58444c9862 Merge pull request 'fix(external-dns): let Prometheus reach the metrics port' (#866) from fix/external-dns-scrape-policy into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#866
2026-09-17 08:35:13 +00:00
claude-botandClaude Opus 5 5f3aecb381 fix(external-dns): let Prometheus reach the metrics port
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The ServiceMonitor added in #865 discovers both external-dns targets and
then times out on them: the apps namespace is default-deny-ingress, so the
scrape never arrives.

  lastError: Get "http://10.244.2.218:7979/metrics": context deadline exceeded
  health: down

Same shape as gitea-allow-metrics-scrape, and one policy covers both
releases — the chart gives them the same app.kubernetes.io/name and they
share the namespace, even though external-dns-unifi is declared under
infrastructure/.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 10:29:53 +02:00
k.renner 4ad47b3925 Merge pull request 'feat(monitoring): watch DNS publication — external-dns metrics plus resolution probes' (#865) from feat/dns-monitoring into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#865
2026-09-17 08:26:04 +00:00
claude-botandClaude Opus 5 2131c7b5f5 feat(monitoring): probe that the LAN names still resolve
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
external-dns metrics say what the publisher intends. This says what a client
actually gets, and catches causes the publisher cannot see — a record removed
on the UDM directly, for one.

Two findings shaped it.

The probe asks the UDM directly rather than the cluster resolver. k8s-gateway
caches these names with `serve_stale 1h immediate`, so a deleted record would
keep answering from cache for an hour before the probe noticed.

And rcode alone is not a usable signal here. Measured against 10.2.40.1:

  does-not-exist-xyz.renner.uno            -> NOERROR, CNAME proxy.renner.uno
  does-not-exist-xyz.paperless.renner.uno  -> no answer

The public *.renner.uno wildcard matches one label deep, which is exactly
where ha, mqtt, ldap and syslog live: delete their records and the UDM still
answers NOERROR, pointing at the WAN proxy. A probe checking only the rcode
would have stayed green through the outage for four of the six names. So the
modules require an A record in the answer and reject a CNAME, which separates
a real record from the wildcard without encoding any address.

One module per name because the dns prober reads query_name from the module
and takes the resolver as its target; both families are probed per name and
the alert wants both to fail, so the UDM's bursty connection refusals do not
read as a missing record. That is also why the window is 30m rather than the
10m the HTTP probes use.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 10:20:04 +02:00
claude-botandClaude Opus 5 d35c06d21d feat(monitoring): scrape external-dns and alert when it stops publishing
Neither external-dns release was scraped: both expose /metrics on :7979 and
no ServiceMonitor existed, so external_dns_* series were absent from
Prometheus entirely. The process that publishes every LAN name in the
cluster had no instrumentation at all.

That is how 2026-09-12 went unnoticed for five days. external-dns v0.22.0
changed the default annotation prefix, the service source went empty, and
policy=sync deleted six records. source_endpoints_total fell 48 -> 36 and
controller_verified_records 24 -> 18 per family; nothing was watching either.

ExternalDNSSourceEndpointsDropped is the rule for that shape: a sustained
drop below 90% of the 24h peak. The threshold is chosen against the observed
inventory — one retired Ingress is 2 endpoints of ~48 and stays quiet, a
whole class of names does not. A deliberate mass removal fires it once and
clears itself after 24h, which is the right trade for a signal that
otherwise only speaks when records are already gone.

The rest of the group covers the ways the drop rule itself can go blind:
source and registry errors, the instance being down, and an absent() dead
man mirroring BlackboxProbesAbsent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 10:15:18 +02:00
k.renner 1bbff9fca4 Merge pull request 'fix(dns): migrate external-dns hostname annotations off the alpha prefix' (#864) from fix/external-dns-annotation-prefix into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#864
2026-09-17 08:02:34 +00:00
claude-botandClaude Opus 5 9a521568c0 fix(dns): migrate external-dns hostname annotations off the alpha prefix
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
external-dns v0.22.0 changed the default --annotation-prefix from
external-dns.alpha.kubernetes.io/ to external-dns.kubernetes.io/ with no
fallback, and its release notes lead with the consequence: "This change can
delete all your DNS records."

The chart floated onto it at 2026-09-11 23:47:56 (appVersion 0.21.0 ->
0.22.0). Two hours later, at 2026-09-12 01:47:59, the unifi provider logged
`client deleted record` for every name that depended on the annotation:

  ha.renner.uno       mqtt.renner.uno     ldap.renner.uno
  syslog.renner.uno   ftp.paperless.renner.uno
  syncthing.paperless.renner.uno

A, AAAA and the TXT ownership records, and no `client created record` since.
Every one of those is a LoadBalancer Service, whose hostname external-dns
can only learn from the annotation — so the source went silent and
--policy=sync removed the records. Ingress-sourced names take their host
from spec.rules[].host and were untouched, which is exactly the split
observed: source_endpoints_total is 36 with source_errors_total 0.

These six annotations were the only alpha-prefixed external-dns annotations
in the repo, so moving them to the current prefix is the whole fix. Pinning
the prefix back with --annotation-prefix would keep us on a deprecated
default that upstream intends to drop.

Scanning has been broken since: the Brother resolves ftp.paperless.renner.uno
rather than using the address.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 10:00:30 +02:00
security-bot 3b3ed4fbc4 chore(image-automation): bump gitea.renner.uno/k.renner/vsftpd:build.1789630783
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-09-17 07:43:07 +00:00
k.renner c40466a3e6 Merge pull request 'fix(paperless): stop the ftp-scanner probe from crashing vsftpd' (#862) from fix/ftp-scanner-probe into master
ci/woodpecker/cron/issue-fix Pipeline was successful
Reviewed-on: k.renner/homelab#862
2026-09-17 07:39:25 +00:00
k.renner ff0b3434c1 Merge pull request 'fix(libation): persist refreshed Audible tokens back to the PVC' (#861) from fix/libation-accounts-persistence into master
Reviewed-on: k.renner/homelab#861
2026-09-17 07:39:07 +00:00
claude-botandClaude Opus 5 2cfebc2f8b refactor(paperless): drop the duplicated PASV address from ftp-scanner
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The LB address was declared twice: on the Service as io.cilium/lb-ipam-ips,
and again as a literal PASV_ADDRESS in the ConfigMap generator.

The copy was not doing anything. vsftpd listens on a single IPv6 socket, so
an IPv4 client arrives as ::ffff:a.b.c.d and vsftpd answers PASV with
0.0.0.0 regardless of the configured address — the only such session in 30
days of logs got `227 Entering Passive Mode (0,0,0,0,82,109)` while
PASV_ADDRESS=10.2.99.82 was set. IPv6 clients negotiate EPSV, which carries
a port and no address at all; that is what the scanner uses.

Removing the literal falls back to the image default of 0.0.0.0, which is
the reply clients already receive, so behaviour is unchanged and the address
now exists in exactly one place.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 08:56:51 +02:00
claude-botandClaude Opus 5 3481e43032 fix(paperless): stop the ftp-scanner probe from crashing vsftpd
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
vsftpd 3.0.5 segfaults its standalone listener on roughly 0.3% of the
connections it accepts: the parent closes the accepted socket before it
registers the forked child's pid, so a child that exits inside that window
makes the SIGCHLD path look up an unregistered pid, get NULL, and
dereference it in hash_ip(). PID 1 is vsftpd, so the segfault takes the
container with it.

The tcpSocket probes added in 469cdbc2 opened ~8600 connections a day
against that, which turned a latent crash into a restart every ~25 minutes
and left the scanner hitting CrashLoopBackOff windows. Measured on the live
pod: 52 restarts in 21h, one node-side `vsftpd[...]: segfault at 0` per
restart, starting the hour the probes rolled out.

Assert the listening socket from inside the container instead. The probe
still catches the failure it was added for — vsftpd not serving port 21 —
without being a client of the process it watches.

vsftpd 3.0.5 is upstream's last release (2021) and Alpine ships it
unpatched in 3.21 (3.0.5-r2) and edge (3.0.5-r3), so there is no version
bump that fixes this.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 08:52:14 +02:00
security-bot 98475a47e2 chore(image-automation): bump ghcr.io/advplyr/audiobookshelf:2.36.1
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-09-16 22:50:50 +00:00
security-bot 6f20955e3b chore(image-automation): bump ghcr.io/usetrmnl/terminus:0.74.0
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-09-16 19:51:35 +00:00
claude-botandClaude Opus 5 92ac021931 feat(libation): seed the token master key onto the PVC
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
TokenStorageMethod is Encrypted and the container has no OS secret store,
so LibationCli falls back to minting a portable master key at
/config-internal/libation-master.key -- a path that is not a mounted
volume. The key is therefore new on every run and discarded at exit,
which is why the entrypoint logs LAST-RESORT on each pass.

The entrypoint looks for the key at ${LIBATION_CONFIG_DIR}/
libation-master.key, i.e. on the PVC, and copies it into place when
found. The key is now stored in OpenBao alongside the other bootstrap
files, so seed it like them: without it a rebuilt PVC would hold tokens
encrypted under a key that no longer exists.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 18:47:24 +02:00
claude-botandClaude Opus 5 5bbca6a162 fix(libation): persist refreshed Audible tokens back to the PVC
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The image entrypoint stages config into /config-internal, which is not a
mounted volume, and it does so by copy:

    init_config_file() {
      ...
      cp ${FULLPATH} ${LIBATION_CONFIG_INTERNAL}/

Only LibationContext.db is symlinked to the PVC. AccountsSettings.json is
therefore read from the PVC at startup and written back nowhere, so every
token refresh Libation performs is discarded when the container exits. On
the live PVC the file still carried its 2026-05-14 seed timestamp, four
months after deployment, while the DB beside it was current.

Pass args instead of letting the entrypoint run its own loop. The
entrypoint completes its setup and then execs what it is given, so the
staged environment is intact; an EXIT trap copies the refreshed file back
to the PVC whether the run succeeds or fails.

This was found while recovering from Audible refusing content licenses
for this account. Re-registering the account fixed the refusals; without
this change the fresh registration would have been discarded on the next
run exactly like every refresh before it.

Also corrects the init container comment, which claimed the sentinel
preserved token refreshes. It preserves the DB; it never preserved the
tokens.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 18:26:49 +02:00
k.renner edb07a344e Merge pull request 'fix(libation): pin image to 14.2.1 by digest' (#860) from fix/libation-pin-14.2.1 into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#860
2026-09-16 15:51:49 +00:00
claude-botandClaude Opus 5 260f548e67 fix(libation): pin image to 14.2.1 by digest
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The CronJob ran `rmcrackan/libation:latest` with imagePullPolicy Always
but was stuck on 13.7.11 (2026-08-24). Spegel answers containerd's tag
resolution from its peer cache -- a 200 from the mirror is a hit, so
upstream is never consulted and the tag never advances. Verified: the
pod's imageID was sha256:946177d3 (= tag 13.7.11) while Docker Hub's
:latest pointed at sha256:6f9d8ba1 (= 14.2.1), and querying a Spegel
pod's registry port directly returned the stale digest for that tag.

Audible changed device registration in early September; upstream shipped
an experimental fix in 14.1 and the real fix in 14.2, so the stale image
predates both. Since 2026-09-15 three newly purchased titles fail with
"Content license denied" and the library has not grown.

A digest reference is immune to the mirror's tag cache. Note that the
upstream maintainer's remedy also requires removing and re-adding the
Audible account with an interactive login, which this change does not
and cannot perform.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 17:30:37 +02:00
k.renner ef6bc24491 Merge pull request 'feat(ceph): serve reads from the local OSD (#576)' (#859) from feat/ceph-read-affinity into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#859
2026-09-16 12:25:51 +00:00
claude-botandClaude Opus 5 779f557624 feat(ceph): serve reads from the local OSD (#576)
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Measuring VLAN 40 for the traffic-class split turned up a share of the
management segment that is not management: the node's Ceph client
talking to OSDs on the other two nodes, sourced from the node address
on mgmt0.

Ceph sends every client read to the acting primary of the PG, which
with three OSDs is remote two times out of three. Read affinity has the
client read from the replica in its own CRUSH location instead.

The preconditions hold here. replicapool and cephfs/data0 are size 3
over failureDomain host across three nodes, so every object has a copy
on every node. The OSDs are created under zone buckets
(root=default host=k8s-N zone=<name>) and the nodes carry
topology.kubernetes.io/zone, which is in Rook's default
crushLocationLabels. Ceph is v20.2.4 and the kernel 6.18, against a
documented minimum of 5.8.

Writes are unaffected: they always go to the primary, and the
replication behind them already rides the VLAN 49 cluster network.

cephfs-ci is size 2, so a local replica is not guaranteed there and the
client falls back to the primary.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 14:10:04 +02:00
k.renner 1fbefb3c39 Merge pull request 'fix(bird): put the node's own pod CIDR in the table, not a blackhole (#576)' (#858) from fix/bird-local-cidr into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/e2e Pipeline failed
ci/woodpecker/cron/security-scan Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#858
2026-09-16 11:27:03 +00:00
claude-botandClaude Opus 5 f97aa077f0 fix(bird): put the node's own pod CIDR in the table, not a blackhole (#576)
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Rolling out #857 took cross-node pod traffic down, and cluster DNS with it, until it was reverted. The table was incomplete: the node's own pod CIDR was originated as a blackhole so it could be advertised without being installed, and then filtered out of the kernel export.

The rule matches the SOURCE, and a reply from a pod on another node is forwarded traffic with a pod source — so it matches too. Its destination is a pod on this node, the table had no route for that, and it took the default and left for the UDM. The comment claiming such traffic would fall through to main was wrong: nothing falls through once a default matches.

Originating it via cilium_host instead makes it both advertisable and correct in the table, and the export filter that existed only to hide the blackhole goes with it.

Diagnosed by narrowing the rule to a single pod address, where the same path worked because the return packet no longer matched: 404 with src <pod>/32, 000 with src 10.244.0.0/16, 404 again once 10.244.0.0/24 dev cilium_host was added to table 47 by hand. bird -p parses the corrected config inside the image, exit 0.

Refs #576

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 13:25:25 +02:00
k.renner 8aa662e997 Merge pull request 'feat: one pod routing table, decided by longest prefix (#576)' (#857) from feat/single-pod-table into master
Reviewed-on: k.renner/homelab#857
2026-09-16 11:11:53 +00:00
claude-botandClaude Opus 5 45012470c1 feat: one pod routing table, decided by longest prefix (#576)
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline is running
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
The rule stack grew a carve-out layer because table 47 held nothing but a default route: it matched everything, so every cluster-internal destination had to be excluded by a rule naming its prefix. Those exclusions were wrong twice over — too broad, since fd00::/8 and 10.0.0.0/8 also caught every LAN client and sent its reply back over mgmt0, and unnecessary, since the networks in question are directly attached and a connected route already beats a default.

So the table gets the connected routes and decides on its own. bird's direct protocol generates them from the interfaces rather than from a list, alongside the peers' pod CIDRs it already learns and the egress default it now originates. One rule per family remains, on the pod source prefix; 980/981, 990/991 and table 48 are gone.

bird owns the table outright, including the default. That is what keeps the failure benign: if the daemon stops the table empties, the rule finds nothing and falls through to main, so pod traffic returns to VLAN 40 unseparated rather than misrouted. A default declared in the Talos config would have outlived the specific routes and sent pod traffic to the UDM.

The ingress gateway moves from a route in each node patch to one node annotation in the site patch, which is now the only place it is stated.

Verified: bird -p parses the rendered config inside the image, exit 0; the templates render GW6 fd02:47::1 and GW4 10.2.47.1 from the live k8s-1 object; kubectl kustomize builds.

Refs #576

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 13:09:13 +02:00
security-bot 688f84dc29 chore(image-automation): bump everythingsmarthome/everything-presence-mmwave-configurator:2.2.5
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-09-16 09:51:31 +00:00
k.renner 6335954d6c Merge pull request 'docs: describe the traffic-class split as built (#576)' (#856) from docs/traffic-classes into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#856
2026-09-16 09:20:50 +00:00
claude-botandClaude Opus 5 f0701370db docs: describe the traffic-class split as built (#576)
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Replaces the proposal, which was wrong before it was written down. It presented A, B and C as the option space and concluded pod-to-pod was blocked; policy routing was available the whole time and is what moved pod egress and then pod-to-pod. The file recorded a blind spot and gave it the authority of a checked-in document.

Three of four classes are separated now, so this describes the built design rather than proposing one, and the PROPOSAL- name goes with it: which link each class uses, the rule order and why 980/981 match source and destination, where each table comes from, how the bird mesh renders itself from the API without repeating an address, and why the node's own CIDR is originated rather than read back from the kernel.

Management is the remaining class. Its levers are listed, along with the two approaches the owner rejected — with the caveat that the same reasoning which produced that list also called pod-to-pod impossible, so it is a record of decisions and not an analysis to trust.

Failure behaviour is stated because it is what makes the design safe to roll one node at a time: an empty table 48 falls through to main and pod traffic returns to VLAN 40, unseparated rather than broken.

bootstrap.sh's reference follows the rename; its point was always that the QEMU/Hetzner sites use two segments, not the homelab's split.

Refs #576

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 11:18:57 +02:00
k.renner 9131f73d6c Merge pull request 'feat(talos): send pod-to-pod over pod0 via table 48 (#576)' (#855) from feat/podpod-rules into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#855
2026-09-16 09:05:22 +00:00
claude-botandClaude Opus 5 7e19ed1b59 feat(talos): send pod-to-pod over pod0 via table 48 (#576)
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The last step of the traffic-class split. The bird mesh already fills table 48 with one route per peer pod CIDR, next hop the peer's pod0 address; these two rules are what make the kernel look there.

Source AND destination are matched. Source alone would also catch the node's own traffic to a pod, which would then leave with a pod0 source — an address CiliumNode does not list, so Cilium would classify it as world rather than host and policy would drop it. Destination alone would catch traffic from anywhere.

Priority 980/981 sits ahead of the 990/991 pair that sends the cluster and local networks to main, so pod-to-pod wins for that traffic and everything else is unchanged. Traffic to a pod on the same node falls through: table 48 holds only the peers' CIDRs, the lookup misses, and 990/991 send it to main and cilium_host.

Verified on k8s-1 with --mode=try, 200 requests per family from a pod on k8s-1 to a pod on k8s-2: pod0 gained 6957 transmit packets, matching 400 requests, while pod0 receive moved by 45 — the peer is unpatched and still answers over mgmt0, the same asymmetry the earlier measurement showed. Both rules were installed as expected, one per family.

Refs #576

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 11:04:05 +02:00
k.renner 41556cbb16 Merge pull request 'fix(bird): originate the node's own pod CIDR instead of learning it' (#854) from fix/bird-static into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#854
2026-09-16 09:00:04 +00:00
claude-botandClaude Opus 5 377270375f fix(bird): originate the node's own pod CIDR instead of learning it
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
IPv6 populated table 48 on all three nodes; IPv4 was empty. The daemon log gives the reason: KRT: Received route 10.244.0.0/24 with strange next-hop 10.244.0.195. Cilium's IPv4 pod route points at a next hop inside the prefix it describes, which BIRD's kernel protocol refuses. The IPv6 route is a device route with no next hop, so it passed — the asymmetry, not a filter, is why one family worked.

Reading the route back made BIRD's view depend on the shape Cilium happens to install. The prefix is already in the Node object, so a static protocol originates it directly: blackhole, because the route exists only to be advertised, and the sole kernel protocol exporting from these tables takes BGP routes.

That removes the kernel-learning tables, the pipes and the own-CIDR filters entirely — with them the dependence on learn all, on ifname, and on the maximum-length exclusion.

Verified: the templates render OWN_POD4 10.244.0.0/24 and OWN_POD6 fd10:244::/64 from the live k8s-1 object, and bird -p parses the result inside the image with exit 0.

Refs #576

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 10:57:28 +02:00
k.renner df9019d90e Merge pull request 'fix(bird): reference the image by digest, the dated tag resolves stale' (#853) from fix/bird-digest into master
Reviewed-on: k.renner/homelab#853
2026-09-16 08:54:37 +00:00
claude-botandClaude Opus 5 c1ccf228b7 fix(bird): reference the image by digest, the dated tag resolves stale
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The DaemonSet came up with Init:Error, kubectl: not found. The build had pushed sha256:3bc85006 for bird:2026-09-16, but every pod resolved that tag to sha256:75e19214 — the first build of the same day, from before kubectl was added.

imagePullPolicy: Always does not help: it re-pulls, but the tag itself resolves to the stale digest. Deleting the pods changed nothing; three fresh pods on three nodes all resolved to the old digest. A one-off pod pulling the digest directly ran kubectl 1.33.1 and BIRD 2.17.2, so the registry holds the right image and only tag resolution is stale.

The dated tag cannot carry a same-day rebuild, and this image was rebuilt three times today. Referencing the digest makes the deployment deterministic. The cost is that the reference has to be updated whenever the image is rebuilt; the alternative, tagging by commit, does not work for a manifest living in the same commit.

Refs #576

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 10:53:55 +02:00
k.renner 1cd26b74ae Merge pull request 'feat(bird): iBGP mesh over pod0, routes into table 48 (#576)' (#852) from feat/bird-mesh2 into master
ci/woodpecker/push/build-ci-images Pipeline was successful
Reviewed-on: k.renner/homelab#852
2026-09-16 08:50:26 +00:00
claude-botandClaude Opus 5 3a9ea5eefe feat(bird): iBGP mesh over pod0, routes into table 48 (#576)
ci/woodpecker/push/build-ci-images Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Nothing reads table 48 yet. The mesh fills it so the routing rules land on a table that is already correct, and so a mistake here shows as an empty table rather than as broken pod traffic.

No address is written down twice. Each node publishes its pod0 addresses as Talos nodeAnnotations, beside the VLANConfig that assigns them; the init container reads the node list and renders from it. bird.conf holds no address at all, and the pod CIDRs are not restated either — the own-CIDR filter keys on the outgoing interface plus a maximum-length exclusion, which separates the node's pod route from cilium_host's own host address without naming a prefix.

Rendering is go-template-file, not string building: kubectl renders the two templates, a field selector drops the node itself server-side, and in-cluster config removes the token and CA plumbing. kubectl comes from the Alpine package, so there is no version literal to drift out of sync and no download to trust; it only ever reads core/v1 Nodes. jq and curl are gone.

The init container ends with bird -p, so BIRD's own parser gates startup: a node whose annotation is missing renders as <no value> and fails there rather than crashlooping the daemon or forming a mesh with a hole.

learn all rather than learn, since plain learn skips proto kernel. direct is explicit because iBGP defaults to multihop, which would make the gateway recursive. strict bind keeps the listener off mgmt0 and ingress0.

Verified against the live cluster with all three nodes annotated via --mode=try: the templates rendered the expected defines and four peer protocols, and bird -p parsed the result inside the image with exit 0 — which also confirms net.len and the include ordering. kubectl kustomize builds.

Refs #576

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 10:43:42 +02:00
k.renner b914b030cc Merge pull request 'feat(bird): add curl and jq for the peer-list init container' (#849) from feat/bird-image-api into master
ci/woodpecker/push/build-ci-images Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#849
2026-09-16 08:10:50 +00:00
claude-botandClaude Opus 5 bb3987ce1c feat(bird): add curl and jq for the peer-list init container
ci/woodpecker/push/build-ci-images Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The per-node peer list is rendered from the Kubernetes API rather than written into the repo a second time, so the init container needs an HTTP client that can verify against the API server's CA bundle. busybox wget cannot, and the query is not going to run unverified. Using the same image for the init container and the daemon keeps one lifecycle instead of borrowing a CI image at runtime.

Refs #576

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 10:10:09 +02:00
k.renner 480b2b11e4 Merge pull request 'feat(bird): image for the pod-network BGP mesh (1/3) (#576)' (#846) from feat/bird-image into master
ci/woodpecker/push/build-ci-images Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#846
2026-09-16 07:50:27 +00:00
claude-botandClaude Opus 5 3302e2c5cd fix(bird): pin the base image by digest and self-test the binary
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/push/build-ci-images Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
Every other Alpine image under ci/images/ pins alpine:3.21 by index digest, and alpine-jq states the reason in the file: the resolved version drifts between restarts of an otherwise pinned image. The first revision used a bare alpine:3.22 tag. 3.22 is still right — bird2 first appears in 3.22/community — so the tag stays and the index digest is added.

heating-can runs --help checks before publishing so a wrong path fails the build rather than the first pod; bird documents --version, so it gets the same guarantee.

Refs #576

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 09:38:46 +02:00
claude-botandClaude Opus 5 3792485c2f feat(bird): image for the pod-network BGP mesh (1/3) (#576)
ci/woodpecker/push/build-ci-images Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Pod east-west follows the peer's Kubernetes InternalIP on mgmt0, and Cilium offers no way to point it elsewhere — established by test in #838, and directRoutingDevice (#837) does not do it. Policy routing does: a rule on the pod source prefixes into a table whose routes carry pod0 next-hops, proven on k8s-1 by counting 4647 packets onto pod0.

Those routes have to be generated, not written down: PodCIDRs are Kubernetes-assigned and do not survive a rebuild in place. BGP between the nodes over pod0 generates them, with the session source as the next-hop.

BIRD rather than FRR. FRR's docs describe 'set table' only as setting a BGP table identifier, never a kernel table; binding bgpd to an address implies --no_kernel and would suppress the route installation the design exists for; and a gatewayless IPv6 device route has no next-hop for its route-maps to match, so the own-CIDR filter could not be uniform. BIRD documents 'kernel table', 'strict bind' without touching the kernel protocol, and an 'ifname' filter attribute that discriminates the local route from the learned ones in both families.

No maintained standalone BIRD image exists: pierky/bird self-declares as lab-only and amd64-only, calico/bird is a Calico-internal artifact expecting confd, slipmesh/bird is five weeks old, srl-labs and ibhde are lab-scoped or on BIRD 1.6. A custom controller would have needed an image of its own anyway, so the build is common to both routes.

Image only. The DaemonSet is 2/3 and the routing rules 3/3, so the table fills before anything reads it.

Refs #576

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 09:29:17 +02:00
k.renner 1a1e095f76 Merge pull request 'feat(talos): route pod egress over ingress0 (#576)' (#842) from feat/pod-egress-vlan47 into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#842
2026-09-16 05:40:40 +00:00
claude-botandClaude Opus 5 11df090913 feat(talos): route pod egress over ingress0 (#576)
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The default route cannot simply move to ingress0: the node IP, the API VIP and apid answer on mgmt0, so a reply leaving over ingress0 would carry an mgmt0 source onto VLAN 47. Policy routing splits the two instead — pod source prefixes look up table 47, everything the node sends itself keeps main and mgmt0.

Table 47 holds only the two default routes, so destination rules at 990/991 send the cluster and local networks back to main first; without them a pod's packet to a service matches the source rule, finds only the default, and leaves for the UDM. That was measured on k8s-1, where in-cluster DNS stopped resolving until the rules were added.

The v6 gateway is the UDM's configured ULA on the segment, the address BGP already peers with, rather than the RA-learned link-local. accept_ra_defrtr stays 0 on ingress0 so the RA default does not land in main.

Verified on k8s-1 with --mode=try: a pod egressed as 2003:d6:973f:5605:5a47:caff:fe74:7027, the ingress0 GUA from the VLAN 47 prefix, while in-cluster DNS and the Talos and Kubernetes APIs kept working.

Refs #576

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 07:35:14 +02:00
security-bot cfb1406201 chore(image-automation): bump ghcr.io/immich-app/immich-server:v3.2.2
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-09-15 21:06:23 +00:00
security-bot 4190c634ec chore(image-automation): bump ghcr.io/immich-app/immich-machine-learning:v3.2.2
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-09-15 20:21:56 +00:00
security-bot 717c5a5f8a chore(image-automation): bump ghcr.io/immich-app/immich-machine-learning:v3.2.1
ci/woodpecker/cron/e2e Pipeline failed
ci/woodpecker/cron/security-scan Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-09-14 19:09:30 +00:00
security-bot 4fd3c21308 chore(image-automation): bump ghcr.io/immich-app/immich-server:v3.2.1
Updates managed by image-automation-controller.
2026-09-14 19:04:00 +00:00
security-bot 5e79157e5e chore(image-automation): bump rhasspy/wyoming-whisper:3.8.1
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-09-14 15:51:03 +00:00
security-bot 4c36913fee chore(image-automation): bump rhasspy/wyoming-whisper:3.8.0
ci/woodpecker/cron/update-audit Pipeline was successful
ci/woodpecker/cron/e2e Pipeline failed
ci/woodpecker/cron/security-scan Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-09-11 21:50:19 +00:00
security-bot 605690f9ff chore(image-automation): bump ghcr.io/music-assistant/server:2.10.3
Updates managed by image-automation-controller.
2026-09-11 21:50:01 +00:00
security-bot fe6fa9ec4c chore(image-automation): bump ghcr.io/home-assistant/home-assistant:2026.9.2
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-09-11 20:50:44 +00:00
security-bot c954043f90 chore(image-automation): bump gotenberg/gotenberg:8.37.0
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-09-11 15:50:15 +00:00
security-bot 31be9a115a chore(image-automation): bump ghcr.io/immich-app/immich-server:v3.2.0
ci/woodpecker/cron/e2e Pipeline failed
ci/woodpecker/cron/security-scan Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-09-10 16:53:33 +00:00
security-bot deda43ab85 chore(image-automation): bump ghcr.io/immich-app/immich-machine-learning:v3.2.0
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-09-10 16:18:57 +00:00
security-bot 19a464b2bf chore(image-automation): bump ghcr.io/usetrmnl/terminus:0.73.0
ci/woodpecker/cron/e2e Pipeline failed
ci/woodpecker/cron/security-scan Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-09-09 19:49:44 +00:00
security-bot b29174dff2 chore(image-automation): bump woodpeckerci/woodpecker-server:v3.18.1
ci/woodpecker/cron/e2e Pipeline failed
ci/woodpecker/cron/security-scan Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-09-09 00:49:19 +00:00
security-bot 451e5002e9 chore(image-automation): bump woodpeckerci/woodpecker-agent:v3.18.1
Updates managed by image-automation-controller.
2026-09-09 00:49:17 +00:00
k.renner d443f435aa Merge pull request 'feat(cilium): route pod east-west over pod0 (#576)' (#837) from feat/cilium-native-pod0 into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#837
2026-09-08 14:44:36 +00:00
claude-botandClaude Fable 5.1 63a3aebe84 feat(cilium): route pod east-west over pod0 (#576)
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Pod-to-pod traffic currently rides the management VLAN. In tunnel mode
Cilium encapsulates to the peer's Kubernetes InternalIP, so east-west
follows VLAN 40 wherever it goes -- which blocks moving management onto
the 1G NIC without dragging pod traffic with it.

The node IP stays on VLAN 40. Instead this uses the documented device
selection: native routing hands forwarding to the kernel, and
autoDirectNodeRoutes installs the remote PodCIDR routes, which is
permitted because all three nodes share pod0's L2 segment.

  "By default, if a single device was detected or specified via devices
   then Cilium will use that device for direct routing. Otherwise,
   Cilium will use a device with Kubernetes InternalIP or ExternalIP
   set."

directRoutingDevice is therefore set explicitly: mgmt0, ingress0 and
pod0 all carry addresses, so the fallback would pick mgmt0.

Verified beforehand that pod0 is a working path between nodes -- from a
pod on k8s-2, k8s-1's kubelet answers 401 on both 10.2.48.11:10250 and
[fd02:48::11]:10250, so the L2 and both address families work.

ipv4/ipv6NativeRoutingCIDR are already set and become mandatory in this
mode; 10.244.0.0/16 and fd10:244::/48 sit inside them, so pod-to-pod
stays unmasqueraded as before.

Not rolling: dropping encapsulation is a cluster-wide datapath change
and a node that has switched cannot exchange pod traffic with one that
has not. Needs a scheduled window, not a background reconcile.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 16:37:59 +02:00
k.renner ee2d5d1d25 Merge pull request 'feat(cilium): peer BGP over the ingress VLAN (#576)' (#836) from feat/bgp-peer-ingress into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
Reviewed-on: k.renner/homelab#836
2026-09-08 13:22:05 +00:00
claude-botandClaude Fable 5.1 60b4286910 feat(cilium): peer BGP over the ingress VLAN (#576)
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Phase B step 2 of the traffic-class split. The eBGP session source
address is what the UDM installs as the next-hop for every PodCIDR and
LoadBalancer prefix the cluster advertises. Peering over the management
LAN therefore pinned all world->pod return traffic to the 1G mgmt link,
which is the thing #576 exists to fix. Moving the peers onto VLAN 47
puts that traffic on the 10G ingress link instead.

No localAddress or sourceInterface override is needed: the kernel
already picks the ingress0 address for both families.

  ip -6 route get fd02:47::1
  fd02:47::1 dev ingress0 proto kernel src fd02:47::5a47:caff:fe74:7027

Both peer addresses were verified reachable from the cluster before
this change -- 10.2.47.1 and fd02:47::1 each answer HTTP with a 301,
and the UDM's neighbour entry on ingress0 is REACHABLE.

The UDM already accepts sessions from this segment: PR #835 added the
10.2.47.0/24 and fd02:47::/64 listen ranges and is uploaded. Its VLAN 40
ranges stay in place for now and get dropped in step 3, once every
session is confirmed established on 47.

gracefulRestart (120s) covers the peer replacement, so the UDM holds the
learned routes across the flap rather than withdrawing the LB VIPs.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 15:20:13 +02:00
k.renner cc52181352 Merge pull request 'feat(bgp): let the UDM accept sessions from the ingress VLAN too' (#835) from feat/bgp-listen-ingress into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#835
2026-09-08 13:00:53 +00:00
claude-bot 56d55b6228 feat(bgp): let the UDM accept sessions from the ingress VLAN too
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
First step of moving the world-to-pod class onto ingress0 (#576). This
changes nothing about where the cluster peers from — it only widens
what the UDM will accept.

The order matters. Cilium's sessions are outbound to the UDM, and the
UDM accepts them through `bgp listen range`. Moving the cluster to
ingress0 before the UDM listens on 10.2.47.0/24 would drop every
session, and with them every advertised LoadBalancer VIP, so external
ingress would go down until the UDM caught up. Listing both segments
first removes that window entirely: a listen range only accepts a
session, it never opens one, so the 47 ranges sit idle until the
cluster actually moves.

The prefix-lists and route-maps are untouched. They filter which
prefixes the UDM accepts — pod CIDRs and the LB pool — and the cluster
advertises the same prefixes either way. Only the next hop changes.

Uploaded by hand at UniFi UI -> Settings -> Routing -> BGP, as before.
Verify with `vtysh -c "show ip bgp summary"`: the six existing sessions
on 10.2.40.x and fd40:: must still be established afterwards, and no
new ones appear until the cluster side moves.

The 40 ranges come out in a later change, once every session is
established on 47.

Refs #576
2026-09-08 14:57:49 +02:00
k.renner 6416f2f609 Merge pull request 'fix(talos): address ingress0 statically, leave mgmt0 the only DHCP client' (#833) from feat/ingress-static-addressing into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#833
2026-09-08 12:19:59 +00:00
claude-bot 80c2def518 fix(talos): stop leasing ingress0, leave mgmt0 the only DHCP client
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Adding a second DHCP client per node in #831 broke node DNS. Both
clients announce the same hostname, so the UDM registered each node
against VLAN 47 instead of VLAN 40: `k8s-1.srv.renner.uno` and
`k8s-3.srv.renner.uno` stopped resolving, and the reverse for
10.2.47.11 came back as bare `k8s-1`. The lease's resolver also
displaced the management one, leaving 10.2.47.1 ahead of 10.2.40.1 in
the node's DNS servers.

That matters beyond `talosctl -n <fqdn>`: bootstrap.sh resolves node
FQDNs in the talos phase, so a rebuild depends on those records.

The UDM cannot register a network's clients selectively, so the client
goes rather than the record. DHCP is now off on VLAN 47 entirely and
ingress0 declares .11/.12/.13 by node ordinal, matching pod0's
numbering. Its default route is declared with the same losing metric
2048 the lease supplied, so nothing about routing changes: mgmt0 keeps
both default routes until the flip.

IPv4 only. The problem was DHCPv4 announcing a hostname; IPv6 never had
one. The prefix is announced correctly, so ingress0 keeps taking its
IPv6 address by SLAAC exactly as mgmt0 takes its fd40:: address —
derived from the MAC, stable across reboots, nothing to allocate.

accept_ra therefore stays at its default of 2. An IPv6 default router
is a link-local address that can only be learned from the RA, and with
net.ipv6.conf.all.forwarding=1 — which Cilium sets — Linux ignores RAs
below 2 entirely. Only accept_ra_defrtr is set to 0, because mgmt0 owns
the default until the flip; dropping that one line is what later hands
the IPv6 default to ingress0.

pod0 keeps accept_ra 0: it is a closed segment with no router on it, so
there is genuinely nothing to learn.

No reboot needed. Address, route and sysctl changes apply live, and the
whole change is provable per node with `--mode=try`, which reverts
itself.

Refs #576
2026-09-08 14:13:57 +02:00
k.renner 0ca1753418 Merge pull request 'feat(talos): role-named links, and the pod and ingress VLANs' (#831) from feat/k8s-net-split-phase-a into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/e2e Pipeline failed
ci/woodpecker/cron/security-scan Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#831
2026-09-08 10:27:56 +00:00
claude-botandClaude Fable 5.1 84b415a0e0 feat(talos): role-named links, and the pod and ingress VLANs
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
First pass of the traffic-class split (#576). Purely additive: nothing
moves onto the new segments yet.

Links by role, not by PCI topology. A udev rule per node matches the
burned-in MAC and renames at boot, the way ceph-san already did:
enp2s0f0np0 -> uplink, enp87s0 -> mgmt. Shared manifests now name roles
and only each node's own patch knows the hardware. The two comments
elsewhere that named the old links go with them.

Two new VLANs on the uplink, both created on the UDM already:

- pod0, VLAN 48. Node-to-node only; the overlay rides it once the
  kubelet node IP moves. Statically addressed .11/.12/.13 out of
  10.2.48.0/24 and fd02:48::/64, no gateway, no DHCP, accept_ra 0. A
  lease would make the cluster's data path depend on the UDM being up in
  order to form, on a segment whose traffic never reaches the UDM.
- ingress0, VLAN 47. World-to-pod: BGP session, LoadBalancer next hop,
  masquerade source. Publicly routed, so DHCP — but at routeMetric 2048
  and with accept_ra_defrtr 0, because DHCPv4Config can only
  deprioritise a default route, not suppress one, and two defaults at
  1024 tie.

mgmt0 keeps the node IP and both default routes. A mistrunked VLAN 47 or
48 therefore cannot strand a node, and the flip is a later change that
needs no reboot and is provable with `--mode=try`.

Applying needs a rolling reboot: a udev rule cannot rename a link on a
running node (Resource busy). Drain first, one node at a time, and wait
for CephCluster HEALTH_OK rather than node Ready between them.

Refs #576

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 11:43:33 +02:00
k.renner c0fa69321b Merge pull request 'chore(ci): run pipelines on ci-k8s:2026-09-08' (#829) from chore/ci-k8s-2026-09-08 into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#829
2026-09-08 07:42:15 +00:00
claude-botandClaude Fable 5.1 f774ca4bca chore(ci): run pipelines on ci-k8s:2026-09-08
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The tag build-ci-images published from #827 (pipeline 8573): talosctl
v1.13.9, kubectl v1.36.3, flux 2.9.5. Six references moved from
2026-08-10.

Refs #426

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 09:35:49 +02:00
k.renner 30362e23d7 Merge pull request 'feat(flux): upgrade components to v2.9.5' (#828) from feat/flux-2.9.5 into master
ci/woodpecker/cron/issue-fix Pipeline was successful
Reviewed-on: k.renner/homelab#828
2026-09-08 07:33:27 +00:00
k.renner 7efa556397 Merge pull request 'chore(ci): bump CI tool pins to match the deployed cluster' (#827) from feat/ci-tool-pins into master
ci/woodpecker/push/build-ci-images Pipeline was successful
Reviewed-on: k.renner/homelab#827
2026-09-08 07:33:08 +00:00
k.renner 4f422eb7de Merge pull request 'feat(bootstrap): OpenBao restore and bring-up under the static seal' (#826) from feat/openbao-seal-key-bootstrap into master
Reviewed-on: k.renner/homelab#826
2026-09-08 07:32:33 +00:00
claude-botandClaude Fable 5.1 6bc23cd306 feat(flux): upgrade components to v2.9.5
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Regenerated with `flux install --export
--components-extra=image-reflector-controller,image-automation-controller`
from the v2.9.5 CLI; the same component set as before.

Controllers: source v1.8.3 -> v1.9.5, kustomize v1.8.4 -> v1.9.5, helm
v1.5.4 -> v1.6.4, notification v1.8.4 -> v1.9.4, image-reflector v1.1.1
-> v1.2.5, image-automation v1.1.2 -> v1.2.5. CRDs updated, the
allow-webhooks NetworkPolicy now names port 9292.

Prerequisites from the 2.9 release notes, verified before this commit:
the removed image v1beta2 and notification v1beta2 API versions are
neither used in the repo (image.* all v1, Alert/Provider v1beta3) nor
stored in the cluster's CRDs (storedVersions all current), so no
`flux migrate` step is needed. helm-controller 1.6 changes the default
post-render strategy; no HelmRelease here uses postRenderers.
`flux check --pre` passes on Kubernetes 1.36.3 and the manifests pass a
server-side dry run against the live cluster.

Refs #426

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 09:01:22 +02:00
claude-botandClaude Fable 5.1 6b0617b4a7 chore(ci): bump CI tool pins to match the deployed cluster
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/push/build-ci-images Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
From the update audit (#426): the tools CI drives the cluster with
trailed what the cluster runs.

ci/images/k8s/Containerfile
- talosctl v1.12.6 -> v1.13.9, the Talos version on the nodes. An older
  talosctl can misjudge or reject config generated for a newer machine.
- kubectl v1.36.2 -> v1.36.3, the cluster's Kubernetes version.
- flux CLI 2.9.2 -> 2.9.5, the current 2.9.x. Checksums verified against
  the upstream release assets (talos sha256sum.txt, flux checksums.txt)
  and against a local download.

.woodpecker/ci-manifests-test.yml
- woodpecker-cli v3.17.0 -> v3.18.0, the version the server and agent
  already run; the lint step was one minor behind its own server.

Merging triggers build-ci-images (path ci/images/**), which publishes
ci-k8s:<date>. The six pipelines pinned to ci-k8s:2026-08-10 move to
that tag in a follow-up once it exists.

Refs #426

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 08:58:35 +02:00
claude-botandClaude Fable 5.1 193bcb362e fix(openbao): QuorumLost runbook text must not point at the recovery key
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The description told on-call to try the recovery key as a last resort,
while RUNBOOK Step 4 in the same change (and upstream: recovery keys are
"purely an authorization mechanism") says it cannot unseal an
auto-sealed server. Say what actually helps: fix the Secret, restart
the pod. Also drop a duplicate root-token computation in
restore_openbao.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 08:35:01 +02:00
claude-botandClaude Fable 5.1 35bad376b8 feat(bootstrap): OpenBao restore and bring-up under the static seal
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Follow-up to #825 (static-seal auto-unseal). The rebuild path and the
operator docs still assumed shamir with a manual unseal per replica.

bootstrap.sh
- Required secrets: `seal-key` (pass show homelab/openbao/seal-key)
  replaces `unseal-key`. The snapshots are encrypted under the seal key,
  so it is the DR-critical secret now; the former unseal key is the
  recovery key and is not needed for a restore (upstream: recovery keys
  "are purely an authorization mechanism" and cannot unseal a server
  whose auto-seal is broken).
- flux phase creates the `openbao-seal-key` Secret (namespace + Secret,
  before `flux bootstrap`) from the collected key. --from-literal via a
  substitution, because `pass show` appends a newline and OpenBao
  rejects a key that is not exactly 32 bytes of base64 ("unknown
  encoding for AES-256 key" — verified). An existing Secret with the
  same value is left alone; a different value is refused, since the
  data is written under whichever key OpenBao started with.
- restore_openbao: `init -recovery-shares=1 -recovery-threshold=1`,
  root token from the output, wait for the self-unseal (init returns
  with the node sealed; the server's own retry loop unseals it within
  ~5 s), then `snapshot restore` WITHOUT -force: OpenBao verifies the
  snapshot was written under this server's seal key and refuses a
  mismatch by name, which is the one guard against restoring under a
  wrong Secret. -force skipped that check and would leave a server that
  can never unseal. The throwaway `unseal` step and its argv/tty
  discussion are gone — under an auto-seal there is no unseal key.
- unseal_openbao -> bring_up_openbao: waits for spec.replicas pods to
  report sealed=false, walking the OrderedReady chain as before, and on
  timeout prints each still-sealed pod's `bao status` instead of a bare
  "timed out".

Rehearsed locally with two OpenBao servers sharing a static key (file://
form, bao 2.5.5): init on an auto-seal returns an empty unseal_keys_b64,
one recovery key and a root token and the server is unsealed within
seconds; a restore of A's snapshot into fresh B (both with and without
force) re-seals and reloads the root key via the auto seal ~5 s later
("done reloading root key using auto seal"), after which B serves A's
data with A's root token. Restarting B under a different key leaves it
sealed with "failed to decrypt keys from storage: message
authentication failed".

Docs and alerts: RUNBOOK-cold-rebuild Step 4 and MIGRATION-sops-openbao
describe the auto-seal flow and the recovery-key semantics; the
OpenBao alert descriptions and the snapshot CronJob comment no longer
tell the reader to unseal by hand. vmtest-job comment updated.

Not covered: a full VMTEST_FULL=1 rehearsal through the flux phase. The
wiki records none ever reaching past the Flux bootstrap checkpoint, so
that gap predates this change.

Refs #805

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 08:23:08 +02:00
k.renner 6b7736ea58 Merge pull request 'fix(alerts): page the moment OpenBao is below quorum' (#824) from fix/openbao-alerts-immediate into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#824
2026-09-08 06:02:32 +00:00
k.renner 7ff20e5b68 Merge pull request 'feat(openbao): auto-unseal with a static seal' (#825) from feat/openbao-static-seal into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#825
2026-09-08 05:54:54 +00:00
claude-botandClaude Fable 5.1 296accd1cb feat(openbao): auto-unseal with a static seal
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Every restart sealed the replica until a human unsealed it. On
2026-09-07 a rolling talosctl upgrade rebooted all three nodes; the PDB
refused the second eviction while the first replica was sealed, Talos
waited out its 5-minute drain timeout and rebooted anyway, and raft
quorum was gone for 24 minutes — every ExternalSecret and the
Kustomizations behind them with it. A power loss would do the same
with no timeout to blame.

OpenBao 2.6.1 supports a static seal: a 32-byte key the server uses to
unseal itself. The key is generated into pass (homelab/openbao/seal-key)
and created as the openbao-seal-key Secret before this release reconciles;
the chart hands it to the server as BAO_SEAL_KEY from that Secret.

This overturns the principle in bootstrap.sh that the key protecting
OpenBao must not live in the cluster OpenBao secures. Decided by the
owner on 2026-09-08: the alternatives were a transit seal from a second
OpenBao on edge-1, which raft traffic would wear out, or a workstation
unseal script that may not be at hand when needed. The exposure is
bounded by what ESO already does — every synced secret is plaintext in
etcd (encrypted at rest) — and grows only by what lives solely in
OpenBao.

Migration is a live, per-replica procedure with the shamir key becoming
the recovery key; the StatefulSet is OnDelete, so merging this changes
nothing until each pod is restarted in turn. The bootstrap and rebuild
paths (bootstrap.sh, vmtest) follow in a separate change once the
migration is verified.

Refs #805

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 07:52:27 +02:00
claude-botandClaude Fable 5.1 2665e5ebf3 fix(alerts): page the moment OpenBao is below quorum
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
OpenBaoQuorumLost waited for: 5m. Below quorum the secret store is
down — the ClusterSecretStore goes InvalidProviderConfig and every
ExternalSecret and the Kustomizations behind them stall — and a sealed
replica never recovers on its own, so the window only delayed the page.
It fires on the first evaluation now.

OpenBaoUnavailable, the external probe, drops from 5m to 1m: two
consecutive failed probes, enough to skip a single lost probe through
Traefik, not enough to sit on a sealed store.

Both StatefulSet expressions are wrapped in max by (namespace,
statefulset). kube-state-metrics stamps instance/pod on the series, so
a rescheduled exporter starts a new series and Prometheus resets the
for: timer with it — on 2026-09-07 that made OpenBaoReplicaNotReady fire
14 minutes late. Alertmanager routes match alertname/source only, so the
dropped labels change nothing downstream.

Refs #805

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 07:40:12 +02:00
k.renner 0fe512746b Merge pull request 'feat(monitoring): Intel iGPU dashboard' (#823) from feat/intel-gpu-dashboard into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#823
2026-09-08 05:24:35 +00:00
claude-botandClaude Fable 5.1 4e2303f455 feat(monitoring): Intel iGPU dashboard
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Grafana dashboard for the gpumon_* series from intel-gpu-exporter:
current Render/3D busy, GPU power and GT frequency per node as stat
tiles, then one time series per engine (Render/3D, Video, VideoEnhance,
Blitter) with node as the series, GT frequency, GPU and package power,
and memory-controller read/write bandwidth.

Node colours are fixed per node across every panel — blue, orange, aqua
for k8s-1/2/3 — rather than assigned by series order, so a node keeps
its colour whatever the $node filter shows. The trio validates for
colour-vision deficiency and normal vision on both Grafana themes
(worst all-pairs ΔE 9.4 / 20.9). Reads and writes share one axis and
differ by dash, not hue. One axis per panel throughout.

gpumon_clients_count is deliberately absent: per-client attribution
needs hostPID, which the exporter does not have, so it would read 0
forever and imply an idle GPU.

Refs #752

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 07:22:46 +02:00
k.renner 483093a104 Merge pull request 'feat(monitoring): export Intel iGPU utilisation, frequency and power' (#822) from feat/intel-gpu-exporter into master
Reviewed-on: k.renner/homelab#822
2026-09-08 05:18:43 +00:00
claude-botandClaude Fable 5.1 8d348e7f88 fix(monitoring): let intel_gpu_top pick the GPU instead of hard-coding card0
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The -device flag named /dev/dri/card0. That is the right node today on
every host, but nothing guarantees the number: a second GPU or a
different probe order and the exporter would silently watch the wrong
device or none. The plugin's monitoring resource injects every GPU on
the node and intel_gpu_top selects it on its own — verified with a probe
pod on k8s-2, which produced the same gpumon_* series without the flag.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 07:15:04 +02:00
claude-botandClaude Fable 5.1 6f9ff9d624 feat(monitoring): export Intel iGPU utilisation, frequency and power
A DaemonSet running clambin/intel-gpu-exporter (intel_gpu_top behind a
Prometheus endpoint) with a ServiceMonitor, giving gpumon_engine_usage,
gpumon_frequency, gpumon_power, gpumon_imc_* and gpumon_clients_count
per node.

Privilege model, validated with a one-shot pod before this commit:
/dev/dri comes from the device plugin's gpu.intel.com/monitoring
resource (every GPU on the node, no hostPath); the i915 PMU that
intel_gpu_top reads is a system-wide event source and the nodes run
perf_event_paranoid=3, so CAP_PERFMON is the one capability added.
No privileged, no hostPID, no hostNetwork, read-only root filesystem.
Root rather than a fixed uid because the capability has to be effective
without file capabilities on the binary.

End-to-end check on k8s-2 during a 40 s chat completion on llama-server:
Render/3D busy 98 %, actual frequency 1449 MHz, GPU power 18.8 W, while
the pod's CPU stayed near 100m — the "is it really on the GPU" question
this exists to answer.

XPU Manager was ruled out (Arc Pro / data-centre parts only); the
archived onedr0p exporter needs privileged + hostPID.

Fixes #752

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 07:14:24 +02:00
k.renner 7bf65783be Merge pull request 'feat(intel-gpu-plugin): advertise the monitoring resource' (#820) from feat/intel-gpu-monitoring-resource into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#820
2026-09-08 04:54:15 +00:00
k.renner dd9fb01359 Merge pull request 'fix(reloader): restart audiobookshelf and immich-server on ConfigMap change' (#819) from fix/reloader-configmap-annotations into master
Reviewed-on: k.renner/homelab#819
2026-09-08 04:52:32 +00:00
claude-botandClaude Fable 5.1 da8a725b32 feat(intel-gpu-plugin): advertise the monitoring resource
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Enables -enable-monitoring on the device plugin. With 0.36.0 that
advertises gpu.intel.com/monitoring, a single resource per node that
hands a pod every GPU rather than one of the shared slots; it is what a
metrics exporter requests to see the whole iGPU, and it is the only way
to get /dev/dri into a pod without a hostPath.

Nothing consumes it yet. The nodes currently advertise gpu.intel.com/i915
only — the issue assumed the chart default had this on, but the
DaemonSet here is hand-written and did not.

Refs #752

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 06:51:43 +02:00
claude-botandClaude Fable 5.1 6008d59c9f fix(reloader): restart audiobookshelf and immich-server on ConfigMap change
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Both consume their ConfigMap with envFrom, which Kubernetes does not
watch: a changed value applies in git and in the cluster while the
running container keeps the environment it started with. Paperless hit
exactly this in #739 and got the annotation in #741; these two were the
remaining envFrom consumers without one.

ftp-scanner, the third Deployment named in the issue, needs nothing: its
ConfigMap comes from a configMapGenerator that deliberately keeps the
content-hash suffix, so any change renames the ConfigMap, mutates the
pod template and rolls the pod on its own.

Reloader already watches all namespaces (kubernetes/infrastructure/reloader).

Fixes #742

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 06:46:12 +02:00
k.renner 5aa6c0609d Merge pull request 'fix(netbird-proxy-services): pin the digest the master build produced' (#818) from netbird-proxy-protocol-digest into master
ci/woodpecker/cron/e2e Pipeline failed
ci/woodpecker/cron/security-scan Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#818
2026-09-07 11:52:17 +00:00
claude-botandClaude Fable 5.1 0ebe113abb fix(netbird-proxy-services): pin the digest the master build produced
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
#817 pinned the digest from the branch build (pipeline 8452). Merging
#816 rebuilt the same date tag on master (pipeline 8457), Gitea kept
only the new manifest, and the pod has been in ImagePullBackOff since:
"failed to resolve reference ...@sha256:0d266b02...". With the Recreate
strategy no controller is running, so the services are unchanged.

The kaniko step of the master build pushed sha256:caffa308...; that is
the digest to pin. Comment states the rule.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-07 13:47:22 +02:00
k.renner 1723f73b5d Merge pull request 'feat(netbird-proxy-services): roll the PROXY-protocol build (2/2)' (#817) from netbird-proxy-protocol-deploy into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#817
2026-09-07 11:34:52 +00:00
k.renner 8d8df3b635 Merge pull request 'feat(netbird): carry the WAN client address through the passthrough edge (1/2)' (#816) from netbird-proxy-protocol into master
ci/woodpecker/push/build-ci-images Pipeline was successful
Reviewed-on: k.renner/homelab#816
2026-09-07 11:34:43 +00:00
claude-botandClaude Fable 5.1 29c91769a7 feat(netbird-proxy-services): roll the PROXY-protocol build (2/2)
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Pins the controller image by digest. The registry tag is the build
date and pipeline 8452 rebuilt today's tag in place, so the tag alone
cannot carry a new build to the cluster; the digest can.

Merge only after the Traefik change from 1/2 has reconciled: from this
build on every service target sends a PROXY protocol v2 header, and
Traefik drops such connections unless websecure trusts the pod CIDRs.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-07 13:32:06 +02:00
126 changed files with 4237 additions and 2416 deletions
+2 -2
View File
@@ -42,7 +42,7 @@ when:
steps:
- name: fetch-alerts
image: gitea.renner.uno/gitea_admin/ci-k8s:2026-08-10
image: gitea.renner.uno/gitea_admin/ci-k8s:2026-09-08
environment:
SOPS_AGE_KEY:
from_secret: sops_age_key
@@ -101,7 +101,7 @@ steps:
bash scripts/ci/make-diag-kubeconfig.sh
- name: alert-triage-agent
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.250
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.280
environment:
CLAUDE_CODE_OAUTH_TOKEN:
from_secret: claude_code_oauth_token
+63 -6
View File
@@ -350,17 +350,17 @@ steps:
--destination gitea.renner.uno/gitea_admin/heating-can:$TAG \
--destination gitea.renner.uno/gitea_admin/heating-can:latest
# netbird-proxy-services: the controller that reconciles NetBird reverse-proxy
# netbird-service-sync: the controller that reconciles NetBird reverse-proxy
# services (public Ingress hosts -> TLS passthrough -> Traefik ClusterIP).
# Runtime artifact like heating-can; same registry so the audit tracks it.
- name: build-netbird-proxy-services
- name: build-netbird-service-sync
image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
directory: ci/images/netbird-proxy-services
directory: ci/images/netbird-service-sync
depends_on: []
when:
- event: deployment
- event: push
path: ["ci/images/netbird-proxy-services/**", ".woodpecker/build-ci-images.yml"]
path: ["ci/images/netbird-service-sync/**", ".woodpecker/build-ci-images.yml"]
environment:
GITEA_USER: gitea_admin
GITEA_TOKEN:
@@ -374,5 +374,62 @@ steps:
/kaniko/executor \
--context . \
--dockerfile Containerfile \
--destination gitea.renner.uno/gitea_admin/netbird-proxy-services:$TAG \
--destination gitea.renner.uno/gitea_admin/netbird-proxy-services:latest
--destination gitea.renner.uno/gitea_admin/netbird-service-sync:$TAG \
--destination gitea.renner.uno/gitea_admin/netbird-service-sync:latest
# wiki-index: embedding index over the Notes repo's wiki/, fed from Gitea and
# queried by ~/Notes/bin/wiki_rank.py. Runtime artifact like netbird-service-sync.
- name: build-wiki-index
image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
directory: ci/images/wiki-index
depends_on: []
when:
- event: deployment
- event: push
path: ["ci/images/wiki-index/**", ".woodpecker/build-ci-images.yml"]
environment:
GITEA_USER: gitea_admin
GITEA_TOKEN:
from_secret: ci_images_token
commands:
- mkdir -p /kaniko/.docker
- AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
- printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
- TAG=$(date +%Y-%m-%d)
# Also a per-commit tag: several builds a day (branch pushes and the
# merge) all rewrite the dated tag, and the deployment pin raced them
# on 2026-09-20. The commit tag is immutable and needs no digest lookup.
- SHORT=$(printf '%s' "$CI_COMMIT_SHA" | cut -c1-8)
- |
/kaniko/executor \
--context . \
--dockerfile Containerfile \
--destination gitea.renner.uno/gitea_admin/wiki-index:$TAG \
--destination gitea.renner.uno/gitea_admin/wiki-index:$TAG-$SHORT \
--destination gitea.renner.uno/gitea_admin/wiki-index:latest
# bird: BGP daemon for the pod-network mesh over pod0 (#576). Runtime
# artifact like heating-can; same registry so the audit tracks it.
- name: build-bird
image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
directory: ci/images/bird
depends_on: []
when:
- event: deployment
- event: push
path: ["ci/images/bird/**", ".woodpecker/build-ci-images.yml"]
environment:
GITEA_USER: gitea_admin
GITEA_TOKEN:
from_secret: ci_images_token
commands:
- mkdir -p /kaniko/.docker
- AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
- printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
- TAG=$(date +%Y-%m-%d)
- |
/kaniko/executor \
--context . \
--dockerfile Containerfile \
--destination gitea.renner.uno/gitea_admin/bird:$TAG \
--destination gitea.renner.uno/gitea_admin/bird:latest
+17 -21
View File
@@ -14,8 +14,8 @@
# and ci-claude:latest. The Containerfile makes this layer
# byte-reproducible, so rebuilding an unchanged version dedups
# in the registry instead of orphaning a fresh ~260 MB copy.
# 3. bump-pr — (only when the version actually changed) rewrite the 5
# consumer pins to ci-claude:<version> and open a PR. A human
# 3. bump-pr — (only when any pin is off the latest version) rewrite every
# consumer pin to ci-claude:<version> and open a PR. A human
# reviews and merges it; nothing auto-merges.
#
# TAG SCHEME — deliberately version tags (ci-claude:<claude-version>), NOT the dated
@@ -107,31 +107,22 @@ steps:
[0-9]*.[0-9]*.[0-9]*) : ;;
*) echo "npm returned an implausible version '$$NEW' — aborting."; exit 1 ;;
esac
# The tag every consumer currently references. They are kept identical,
# so this must resolve to EXACTLY ONE non-empty tag; anything else
# (no match, or a partial/disagreeing set) aborts rather than risk a
# corrupting sed later. Counting with `grep -c .` (non-empty lines)
# instead of `wc -l` is deliberate: an empty result yields 0 here but
# 1 under wc -l, which would slip past the guard.
CUR=$$(grep -hoE 'ci-claude:[0-9A-Za-z._-]+' $$CONSUMERS | sed 's/^ci-claude://' | sort -u)
NTAGS=$$(printf '%s' "$$CUR" | grep -c . || true)
if [ -z "$$CUR" ] || [ "$$NTAGS" -ne 1 ]; then
echo "Expected exactly one ci-claude tag across the consumers, got [$$CUR]."
echo "Refusing to guess — fix the pins by hand."; exit 1
fi
echo "latest=$$NEW current=$$CUR event=$$CI_PIPELINE_EVENT"
# Distinct tags the consumers currently reference. Divergent pins are
# normal (a pipeline bumped on its own); the bump rewrites all of them.
CUR=$$(grep -hoE 'ci-claude:[0-9][0-9A-Za-z._-]*' $$CONSUMERS | sed 's/^ci-claude://' | sort -u | tr '\n' ' ' | sed 's/ $$//')
[ -n "$$CUR" ] || { echo "No ci-claude pin found in the consumers — aborting."; exit 1; }
echo "latest=$$NEW current=[$$CUR] event=$$CI_PIPELINE_EVENT"
printf '%s' "$$NEW" > "$$CI_WORKSPACE/.ci-refresh/version"
printf '%s' "$$CUR" > "$$CI_WORKSPACE/.ci-refresh/current"
# Build when the version changed, or on a push (Containerfile edit must
# ship even if the version is unchanged). Bump the pins only when the
# version the consumers reference is not the latest.
# Build when any pin is off latest, or on a push (Containerfile edit must
# ship even if the version is unchanged). Bump when any pin is off latest.
if [ "$$CUR" != "$$NEW" ] || [ "$$CI_PIPELINE_EVENT" = "push" ]; then
touch "$$CI_WORKSPACE/.ci-refresh/do-build"; echo "will build $$NEW"
else
echo "already on $$NEW — nothing to build"
fi
if [ "$$CUR" != "$$NEW" ]; then
touch "$$CI_WORKSPACE/.ci-refresh/do-bump"; echo "will open bump PR $$CUR -> $$NEW"
touch "$$CI_WORKSPACE/.ci-refresh/do-bump"; echo "will open bump PR [$$CUR] -> $$NEW"
else
echo "consumers already pin $$NEW — no PR"
fi
@@ -175,6 +166,9 @@ steps:
- |
[ -f "$$CI_WORKSPACE/.ci-refresh/do-bump" ] || { echo "no version change — no PR"; exit 0; }
set -eu
# Secret values can carry a trailing newline; it breaks the push URL and the auth header.
PUSH_TOKEN=$$(printf '%s' "$$PUSH_TOKEN" | tr -d '\r\n')
[ -n "$$PUSH_TOKEN" ] || { echo "PUSH_TOKEN is empty — aborting"; exit 1; }
NEW=$$(cat "$$CI_WORKSPACE/.ci-refresh/version")
CUR=$$(cat "$$CI_WORKSPACE/.ci-refresh/current")
# Both were validated non-empty by the decide step; re-assert before a
@@ -190,9 +184,11 @@ steps:
# is not a maintained list.
CONSUMERS=$$(grep -rlE 'ci-claude:[0-9]' .woodpecker)
[ -n "$$CONSUMERS" ] || { echo "No ci-claude consumers found — refusing to continue."; exit 1; }
# Exact-string swap of the current tag for the new one, consumers only.
# NEW goes into a sed replacement: allow only a plain x.y.z version.
printf '%s' "$$NEW" | grep -qE '^[0-9]+\.[0-9]+\.[0-9]+$$' || { echo "refusing non-x.y.z version '$$NEW'"; exit 1; }
# Every version-pinned tag becomes NEW, whatever it was; consumers only.
for f in $$CONSUMERS; do
sed -i "s#ci-claude:$$CUR#ci-claude:$$NEW#g" "$$f"
sed -i "s#ci-claude:[0-9][0-9A-Za-z._-]*#ci-claude:$$NEW#g" "$$f"
done
# Author is claude-bot (the PAT's account); the commit subject makes
# the automated-bump nature clear.
+3 -3
View File
@@ -60,7 +60,7 @@ steps:
# run that never touched them. Tracked in #461.
- name: kustomize-build
depends_on: [] # see note above `steps:`
image: gitea.renner.uno/gitea_admin/ci-k8s:2026-08-10
image: gitea.renner.uno/gitea_admin/ci-k8s:2026-09-08
commands:
- |
set -eu
@@ -158,7 +158,7 @@ steps:
# whenever it trails the deployed server. If you bump the server, bump here.
- name: woodpecker-lint
depends_on: [] # see note above `steps:`
image: woodpeckerci/woodpecker-cli:v3.17.0-alpine
image: woodpeckerci/woodpecker-cli:v3.18.0-alpine
commands:
- |
set -eu
@@ -246,7 +246,7 @@ steps:
# this step fails permanently on its own documentation.
- name: no-digest-pins
depends_on: [] # see note above `steps:`
image: gitea.renner.uno/gitea_admin/ci-k8s:2026-08-10
image: gitea.renner.uno/gitea_admin/ci-k8s:2026-09-08
commands:
- |
set -eu
+1 -1
View File
@@ -17,6 +17,6 @@ when:
steps:
- name: model-selection-tests
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.250
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.280
commands:
- sh scripts/ci/test-model-selection.sh
+1 -1
View File
@@ -4,7 +4,7 @@ when:
steps:
- name: e2e
image: gitea.renner.uno/gitea_admin/ci-k8s:2026-08-10
image: gitea.renner.uno/gitea_admin/ci-k8s:2026-09-08
environment:
SOPS_AGE_KEY:
from_secret: sops_age_key
+2 -2
View File
@@ -26,12 +26,12 @@ steps:
# fetch-models.sh response mapping. No cluster, no secrets, no network — the
# ci-claude image just provides sh + jq. Keeps the flag-injection regression
# test from rotting (it exercises the exact injection shapes).
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.250
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.280
commands:
- sh scripts/ci/test-model-selection.sh
- name: e2e-test
image: gitea.renner.uno/gitea_admin/ci-k8s:2026-08-10
image: gitea.renner.uno/gitea_admin/ci-k8s:2026-09-08
environment:
SOPS_AGE_KEY:
from_secret: sops_age_key
+2 -2
View File
@@ -49,7 +49,7 @@ steps:
# (the issue-pick logic needs jq, which only the ci-claude image has);
# minting a 1h read-only token per run is cheap. Degrades to
# snapshot-only if the ci-diagnostics ServiceAccount isn't reconciled.
image: gitea.renner.uno/gitea_admin/ci-k8s:2026-08-10
image: gitea.renner.uno/gitea_admin/ci-k8s:2026-09-08
environment:
SOPS_AGE_KEY:
from_secret: sops_age_key
@@ -62,7 +62,7 @@ steps:
- bash scripts/ci/make-diag-kubeconfig.sh
- name: issue-fix-agent
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.250
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.280
environment:
CLAUDE_CODE_OAUTH_TOKEN:
from_secret: claude_code_oauth_token
+1 -1
View File
@@ -65,7 +65,7 @@ when:
steps:
- name: pr-review-agent
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.250
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.280
environment:
CLAUDE_CODE_OAUTH_TOKEN:
from_secret: claude_code_oauth_token
+1 -1
View File
@@ -70,7 +70,7 @@ steps:
# this step is skipped — otherwise we'd fork-bomb PRs.
when:
- branch: master
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.250
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.280
environment:
CLAUDE_CODE_OAUTH_TOKEN:
from_secret: claude_code_oauth_token
+1 -1
View File
@@ -30,7 +30,7 @@ when:
steps:
- name: update-audit-agent
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.250
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.280
environment:
CLAUDE_CODE_OAUTH_TOKEN:
from_secret: claude_code_oauth_token
+140
View File
@@ -0,0 +1,140 @@
# Nightly wiki maintenance for the Notes vault (k.renner/notes), replacing the
# workstation's wiki-maintenance.timer: distill the raw/ backlog into wiki/,
# then audit wiki/. Each agent is its own `claude` run, not a subagent of an
# orchestrator, so nothing depends on background tasks in print mode (the
# workstation job died on their 600 s ceiling). The pipeline commits and pushes
# as Claude Bot; the push webhook then refreshes the wiki-index service.
#
# The agents and their retrieval scripts address the vault as ~/Notes, so each
# agent step links $HOME/Notes to the checkout instead of rewriting them.
when:
- event: cron
cron: wiki-maintenance
# Hand-started runs: `woodpecker pipeline create k.renner/homelab --branch master`.
# The typesafe_api_key secret is scoped to cron and manual, not deployment.
- event: manual
steps:
- name: checkout-notes
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.280
pull: true
environment:
WIKI_BOT_TOKEN:
from_secret: wiki_bot_token
commands:
- |
set -eu
# Token only on the wire, never in .git/config. Secrets set from files carry
# a trailing newline (run 10220: "could not read Username"); strip CR/LF.
TOKEN=$$(printf '%s' "$$WIKI_BOT_TOKEN" | tr -d '\r\n')
git -c http.extraheader="AUTHORIZATION: token $$TOKEN" \
clone --depth 50 --branch main https://gitea.renner.uno/k.renner/notes.git notes
echo "raw backlog: $$(ls notes/raw/*.md | grep -v _index.md | wc -l) file(s)"
- name: distill
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.280
pull: true
environment:
CLAUDE_CODE_OAUTH_TOKEN:
from_secret: claude_code_oauth_token
TYPESAFE_API_KEY:
from_secret: typesafe_api_key
WIKI_INDEX_BASIC_AUTH:
from_secret: wiki_index_basic_auth
commands:
- |
set -eu
ln -sfn "$$PWD/notes" "$$HOME/Notes"
mkdir -p "$$HOME/.config/wiki-index"
printf '%s' "$$WIKI_INDEX_BASIC_AUTH" | tr -d '\r\n' > "$$HOME/.config/wiki-index/basic-auth"
chmod 600 "$$HOME/.config/wiki-index/basic-auth"
"$$HOME/Notes/bin/wiki_rank.py" --probe-index
- |
set -eu
MODEL=$$(sh scripts/ci/pick-model.sh "distill raw source notes into atomic wiki notes: read sources, check the wiki for existing coverage, write or merge notes; careful writing, moderate reasoning")
if [ -n "$$MODEL" ]; then echo "distill: selected model $$MODEL"; else echo "distill: using the CLI default model"; fi
cd notes
cat ../ci/agents/wiki-maintenance/distill.md | claude $${MODEL:+--model $$MODEL} \
--agent wiki-distill \
--permission-mode acceptEdits \
--allowedTools "Read,Edit,Write,Glob,Grep,Bash(/root/Notes/bin/wiki_rank.py:*),Bash(/root/Notes/bin/wiki_ask.py:*),Bash(mv /root/Notes/raw/:*),Bash(mkdir -p /root/Notes/:*)" \
--settings ../ci/agents/wiki-maintenance/settings.json \
-p
- name: audit
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.280
pull: true
environment:
CLAUDE_CODE_OAUTH_TOKEN:
from_secret: claude_code_oauth_token
TYPESAFE_API_KEY:
from_secret: typesafe_api_key
WIKI_INDEX_BASIC_AUTH:
from_secret: wiki_index_basic_auth
commands:
- |
set -eu
ln -sfn "$$PWD/notes" "$$HOME/Notes"
mkdir -p "$$HOME/.config/wiki-index"
printf '%s' "$$WIKI_INDEX_BASIC_AUTH" | tr -d '\r\n' > "$$HOME/.config/wiki-index/basic-auth"
chmod 600 "$$HOME/.config/wiki-index/basic-auth"
- |
set -eu
MODEL=$$(sh scripts/ci/pick-model.sh "structural audit of ~1700 markdown notes: frontmatter completeness, broken links, orphans; mechanical checks, little reasoning")
if [ -n "$$MODEL" ]; then echo "audit: selected model $$MODEL"; else echo "audit: using the CLI default model"; fi
cd notes
cat ../ci/agents/wiki-maintenance/audit.md | claude $${MODEL:+--model $$MODEL} \
--agent wiki-audit \
--permission-mode acceptEdits \
--allowedTools "Read,Write,Glob,Grep,Bash(/root/Notes/bin/wiki_rank.py:*),Bash(/root/Notes/bin/wiki_ask.py:*)" \
--settings ../ci/agents/wiki-maintenance/settings.json \
-p
- name: push-notes
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.280
pull: true
environment:
WIKI_BOT_TOKEN:
from_secret: wiki_bot_token
commands:
- |
set -eu
cd notes
git config user.email "claude-bot@renner.uno"
git config user.name "Claude Bot"
if git diff --quiet && git diff --cached --quiet && [ -z "$$(git ls-files --others --exclude-standard)" ]; then
echo "wiki-maintenance: no changes"
exit 0
fi
# The workstation pushes main on its own (notes-push.path), so main moves
# while the agents run (run 10241: non-fast-forward). The run's work goes
# to a branch and a PR that this step merges through Gitea; a conflict
# leaves the PR open for a human and fails the run.
BRANCH="wiki-maintenance/$$(date -u +%F)-$$CI_PIPELINE_NUMBER"
git checkout -q -b "$$BRANCH"
git add -A
git commit -q -m "wiki maintenance $$(date -u +%F): distill + audit (CI run $$CI_PIPELINE_NUMBER)"
TOKEN=$$(printf '%s' "$$WIKI_BOT_TOKEN" | tr -d '\r\n')
git -c http.extraheader="AUTHORIZATION: token $$TOKEN" push origin "HEAD:$$BRANCH"
API="https://gitea.renner.uno/api/v1/repos/k.renner/notes"
jq -n --arg title "wiki maintenance $$(date -u +%F) (CI run $$CI_PIPELINE_NUMBER)" \
--arg body "Nightly distill + audit. Merged by the pipeline; open only if the merge failed." \
--arg head "$$BRANCH" '{title: $$title, body: $$body, head: $$head, base: "main"}' > /tmp/pr.json
code=$$(curl -s -o /tmp/pr-resp.json -w '%{http_code}' -X POST \
-H "Authorization: token $$TOKEN" -H "Content-Type: application/json" --data @/tmp/pr.json "$$API/pulls")
echo "POST pull -> HTTP $$code"
[ "$$code" -ge 200 ] && [ "$$code" -lt 300 ] || { cat /tmp/pr-resp.json; exit 1; }
NUM=$$(jq -r '.number' /tmp/pr-resp.json)
# Merge without delete_branch_after_merge: run 10254 got HTTP 500 from that
# call although the merge itself had landed. Verify the PR state instead of
# trusting the status code, then drop the branch as a separate, best-effort step.
code=$$(curl -s -o /tmp/merge-resp.json -w '%{http_code}' -X POST \
-H "Authorization: token $$TOKEN" -H "Content-Type: application/json" \
--data '{"Do":"merge"}' "$$API/pulls/$$NUM/merge")
echo "POST merge #$$NUM -> HTTP $$code"
MERGED=$$(curl -s -H "Authorization: token $$TOKEN" "$$API/pulls/$$NUM" | jq -r '.merged')
echo "PR #$$NUM merged: $$MERGED"
[ "$$MERGED" = "true" ] || { cat /tmp/merge-resp.json; echo "PR #$$NUM left open"; exit 1; }
code=$$(curl -s -o /tmp/del-resp.json -w '%{http_code}' -X DELETE -H "Authorization: token $$TOKEN" \
"$$API/branches/$$(printf '%s' "$$BRANCH" | sed 's|/|%2F|g')")
echo "DELETE branch $$BRANCH -> HTTP $$code (best effort)"
+12 -1
View File
@@ -26,8 +26,16 @@ edge_open3e_poll_seconds: 30
# entries when added), so the multi-frame read is cheap at a 30 s poll.
# Records faults/info events only (NormalOperation, LagDeviceDtcReported,
# LegionellaProtectionTemperatureReached) — NOT setpoint or program changes.
# onebase.*: the 29 paired ZigBee slots on the One Base gateway (ECU 0x6c3) --
# 8 radiator valves, 9 climate sensors, 9 underfloor actuators, 3 floor
# thermostats. Per-device room temperature, active setpoint, humidity, valve
# opening/demand, battery and RSSI; the local replacement for the ViCare cloud
# room sensors. Needs the onebase overlay (files/open3e-onebase.py) -- the stock
# codec declares 57 bytes and a paired slot answers with 68.
# Only CurrentValues are polled. The room and device registries (1884.., 2084..)
# are static configuration and are read on demand, not every 30 s.
edge_open3e_datapoints: >-
vitocal.259,vitocal.260,vitocal.268,vitocal.269,vitocal.271,vitocal.274,vitocal.286,vitocal.288,vitocal.318,vitocal.321,vitocal.324,vitocal.335,vitocal.336,vitocal.396,vitocal.402,vitocal.403,vitocal.476,vitocal.477,vitocal.543,vitocal.548,vitocal.565,vitocal.1007,vitocal.1043,vitocal.1190,vitocal.1211,vitocal.1391,vitocal.1416,vitocal.1417,vitocal.2320,vitocal.2346,vitocal.2350,vitocal.2351,vitocal.2352,vitocal.2369,vitocal.2370,vitocal.2486,vitocal.2487,vitocal.2488,vitocal.2543,vitocal.2560,vitocal.2622,vitocal.2624,vitocal.2625,vitocal.2735,vitocal.3016,vx3.266,vx3.378,vx3.379,vx3.380,vx3.535,vx3.1603,vx3.1664,vx3.1690,vx3.1801,vx3.1802,vx3.1810,vx3.1828,vx3.1830,vx3.1577,vx3.1578,vx3.1579,vx3.1580,vx3.1581,vx3.1582,vx3.1834,vx3.1836,vx3.2214,vx3.2240,vx3.3085,vx3.3086,vx3.3087,vx3.3088,vx3.3089,vx3.3090,vcmu.320,vcmu.322,vcmu.325,vcmu.355,vcmu.389,vcmu.391,vcmu.1772,vcmu.1773,vcmu.1775,vcmu.1776,vcmu.2256,vcmu.2259,vcmu.2333,vcmu.2334,vcmu.2569,vcmu.2571,vitocal.2791
vitocal.259,vitocal.260,vitocal.268,vitocal.269,vitocal.271,vitocal.274,vitocal.286,vitocal.288,vitocal.318,vitocal.321,vitocal.324,vitocal.335,vitocal.336,vitocal.396,vitocal.402,vitocal.403,vitocal.476,vitocal.477,vitocal.543,vitocal.548,vitocal.565,vitocal.1007,vitocal.1043,vitocal.1190,vitocal.1211,vitocal.1391,vitocal.1416,vitocal.1417,vitocal.2320,vitocal.2346,vitocal.2350,vitocal.2351,vitocal.2352,vitocal.2369,vitocal.2370,vitocal.2486,vitocal.2487,vitocal.2488,vitocal.2543,vitocal.2560,vitocal.2622,vitocal.2624,vitocal.2625,vitocal.2735,vitocal.3016,vx3.266,vx3.378,vx3.379,vx3.380,vx3.535,vx3.1603,vx3.1664,vx3.1690,vx3.1801,vx3.1802,vx3.1810,vx3.1828,vx3.1830,vx3.1577,vx3.1578,vx3.1579,vx3.1580,vx3.1581,vx3.1582,vx3.1834,vx3.1836,vx3.2214,vx3.2240,vx3.3085,vx3.3086,vx3.3087,vx3.3088,vx3.3089,vx3.3090,vcmu.320,vcmu.322,vcmu.325,vcmu.355,vcmu.389,vcmu.391,vcmu.1772,vcmu.1773,vcmu.1775,vcmu.1776,vcmu.2256,vcmu.2259,vcmu.2333,vcmu.2334,vcmu.2569,vcmu.2571,vitocal.2791,onebase.2086,onebase.2089,onebase.2092,onebase.2095,onebase.2098,onebase.2101,onebase.2104,onebase.2107,onebase.2110,onebase.2113,onebase.2116,onebase.2119,onebase.2122,onebase.2125,onebase.2128,onebase.2131,onebase.2134,onebase.2137,onebase.2140,onebase.2143,onebase.2262,onebase.2265,onebase.2268,onebase.2271,onebase.2274,onebase.2277,onebase.2280,onebase.2283,onebase.2286
# ECU addresses; -dev selects codec set, tx selects the UDS address.
# vcmu (refrigerant-circuit controller) has no dedicated codec set —
# omitting dpList makes open3e fall back to the generic datapoint list,
@@ -38,6 +46,9 @@ edge_open3e_devices:
# files/open3e-vx3plus.py) — die Stock-vx3-Liste filtert sie sonst weg.
vx3: { tx: "0x6a1", dpList: "/config/vx3plus.py" }
vcmu: { tx: "0x68c" }
# One Base ZigBee gateway. Overlay re-declares the paired ZigBee slots at
# their real 68-byte length (see files/open3e-onebase.py).
onebase: { tx: "0x6c3", dpList: "/config/onebase.py" }
edge_e3oncan_gap_seconds: 10
edge_e3oncan_mqtt_format: "{device}_{didNumber:04d}_{didName}"
@@ -0,0 +1,54 @@
# Managed by ansible (roles/heating-can). Datapoint overlay for the One Base
# ZigBee gateway on ECU 0x6c3 (BACKENDGATEWAY).
#
# Upstream declares ZigBee<N>DeviceCurrentValues as 57 bytes. A PAIRED device
# slot on this gateway answers with 68: the 57 known bytes plus the parent
# device serial and three trailing bytes. udsoncan reads only the declared
# length and then treats the next two bytes -- the first half of that serial --
# as a foreign data identifier, so every paired slot fails to decode with
# "Server returned values for data identifier 0x.... that was not requested".
# The bogus identifier differs per device because the serial does.
#
# This overlay re-declares the 29 paired slots at 68 bytes, reusing upstream's
# sub-type list so the per-device-type O3ESwitch (climate sensor / TRV / floor
# thermostat / actuator, incl. DeviceTemperatureSetpoint) is inherited rather
# than copied.
#
# EMPTY slots answer with 57 and decode against the stock codec, so they are
# left alone and are not polled. A slot that is later unpaired therefore starts
# reporting a decode error on its topic instead of a value.
#
# Verified on ECU 0x6c3, 2026-09-17: 29 paired slots, all 68 bytes.
from open3e.Open3Ecodecs import O3EComplexType, RawCodec
from open3e.Open3Edatapoints import dataIdentifiers as _master
# ZigBee<N>DeviceCurrentValues of the paired slots: devices 1-20 run 2086..2143
# in steps of 3, devices 21-29 continue at 2262..2286.
_PAIRED_CURRENT_VALUES = [2086 + 3 * n for n in range(20)] + [
2262 + 3 * n for n in range(9)
]
def _with_parent_serial(did):
stock = _master["dids"][did]
return O3EComplexType(
68,
stock.id,
list(stock.subTypes)
+ [
RawCodec(8, "ParentDeviceSerialnumber"),
RawCodec(2, "Unknown_65_66"),
RawCodec(1, "Unknown_67"),
],
desc=stock.desc,
info=stock.info,
acc=stock.acc,
)
dataIdentifiers = {
"dids": {
**_master["dids"],
**{did: _with_parent_serial(did) for did in _PAIRED_CURRENT_VALUES},
}
}
+7
View File
@@ -129,6 +129,13 @@
mode: "0644"
notify: Restart heating-can
- name: Deploy open3e One Base datapoint overlay (68-byte ZigBee slots)
ansible.builtin.copy:
src: open3e-onebase.py
dest: "{{ heating_can_base_dir }}/onebase.py"
mode: "0644"
notify: Restart heating-can
# --- Compose project ---------------------------------------------------------
- name: Deploy the broker credential env file
ansible.builtin.template:
@@ -17,6 +17,7 @@ services:
volumes:
- ./devices.json:/config/devices.json:ro
- ./vx3plus.py:/config/vx3plus.py:ro
- ./onebase.py:/config/onebase.py:ro
env_file:
# Broker credential, 0600 -- see heating-can.env.j2.
- heating-can.env
+5
View File
@@ -0,0 +1,5 @@
You are running as the scheduled wiki-maintenance job in CI. The Notes vault is checked out at ~/Notes (the repository k.renner/notes).
Run the standard audit pass over ~/Notes/wiki/ and write the dated report to ~/Notes/output/. Do not modify any wiki note. Do not commit or push; the pipeline does that.
End with two lines: notes scanned, and issues found by category.
+7
View File
@@ -0,0 +1,7 @@
You are running as the scheduled wiki-maintenance job in CI. The Notes vault is checked out at ~/Notes (the repository k.renner/notes). Do exactly this, nothing else:
Glob ~/Notes/raw/*.md (top level only, never ~/Notes/raw/_processed/). Distill every file you find into ~/Notes/wiki/ following your process, including the ranker check for every concept. Move each fully processed source to ~/Notes/raw/_processed/. If raw/ has no top-level files besides _index.md, say so and stop.
Do not commit or push; the pipeline does that after you finish. Do not touch anything outside ~/Notes/wiki/ and ~/Notes/raw/.
End with one line per file: which wiki notes it produced or merged into, or "nothing distillable".
+37
View File
@@ -0,0 +1,37 @@
{
"//": "Permission scope for the wiki-maintenance agents (wiki-distill, wiki-audit) in CI.",
"//2": "The step's environment holds a Gitea push token, a TypeSafe key and the wiki-index",
"//3": "client credential, so the Read tool is kept away from /proc (env dumps), /root outside",
"//4": "the vault symlink target, /etc and the checkout's .git/. The vault itself is the work.",
"//5": "Mirrors ci/agents/issue-fix/settings.json; absolute globs because the vault is reached",
"//6": "through the ~/Notes symlink the pipeline creates.",
"permissions": {
"allow": [
"Read(//root/Notes/**)",
"Edit(//root/Notes/wiki/**)",
"Edit(//root/Notes/output/**)",
"Edit(//root/Notes/raw/**)",
"Bash(/root/Notes/bin/wiki_rank.py:*)",
"Bash(/root/Notes/bin/wiki_ask.py:*)",
"Bash(mv /root/Notes/raw/:*)",
"Bash(mkdir -p /root/Notes/:*)",
"Bash(ls:*)",
"Bash(date:*)",
"Bash(wc:*)",
"Bash(head:*)",
"Bash(tail:*)"
],
"deny": [
"Read(//proc/**)",
"Read(//etc/**)",
"Read(//root/.config/**)",
"Read(//root/Notes/.git/**)",
"Read(//root/Notes/_archive/**)",
"Edit(//root/Notes/_archive/**)",
"Bash(rm:*)",
"Bash(git:*)",
"Bash(curl:*)",
"Bash(wget:*)"
]
}
}
+22
View File
@@ -0,0 +1,22 @@
FROM alpine:3.22@sha256:14358309a308569c32bdc37e2e0e9694be33a9d99e68afb0f5ff33cc1f695dce
# bird: BGP daemon for the pod-network mesh over pod0. Distro package, no build
# step — the image is the binary plus its runtime.
#
# 3.22 rather than the 3.21 the other images here pin: bird2 first appears in
# 3.22/community.
#
# Config is read from /etc/bird/bird.conf, not the package default
# /etc/bird.conf, so the mount cannot shadow what the package ships.
#
# kubectl renders the init container's per-node config: go-template-file for
# the templating, a field selector for excluding the node itself, in-cluster
# config for the auth. From the distro, so there is no version literal to drift
# and no download to trust. It is only ever used to read core/v1 Nodes.
RUN apk add --no-cache bird2 kubectl
# Fails the build rather than the first pod if a binary or its path is wrong.
RUN /usr/sbin/bird --version \
&& kubectl version --client > /dev/null
ENTRYPOINT ["/usr/sbin/bird", "-f", "-c", "/etc/bird/bird.conf"]
+3 -1
View File
@@ -13,7 +13,9 @@ RUN apk add --no-cache \
git \
jq \
openssh-client \
openssl
openssl \
python3 \
py3-yaml
# Install Claude Code, then make the resulting layer byte-stable so that a
# rebuild of the SAME version DEDUPS in the Gitea registry instead of orphaning
+5 -5
View File
@@ -1,9 +1,9 @@
FROM alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d
ARG TALOS_VERSION=v1.12.6
ARG TALOS_VERSION=v1.13.9
ARG VELERO_VERSION=v1.16.0
ARG KUBECTL_VERSION=v1.36.2
ARG FLUX_VERSION=2.9.2
ARG KUBECTL_VERSION=v1.36.3
ARG FLUX_VERSION=2.9.5
RUN apk add --no-cache \
aws-cli \
@@ -18,7 +18,7 @@ RUN apk add --no-cache \
RUN curl -sfL -o /usr/local/bin/talosctl \
"https://github.com/siderolabs/talos/releases/download/${TALOS_VERSION}/talosctl-linux-amd64" \
&& echo "4c41a3b10b075292e64b182283d558e9d8e935f43239de1ed4e4b14359593efb /usr/local/bin/talosctl" | sha256sum -c - \
&& echo "7e1d4b7d5846964bdcf63a794e3c8161bb6ef2983d5ace58ea5322f3bf32a27e /usr/local/bin/talosctl" | sha256sum -c - \
&& chmod +x /usr/local/bin/talosctl
RUN curl -sfL -o /usr/local/bin/kubectl \
@@ -31,7 +31,7 @@ RUN curl -sfL -o /usr/local/bin/kubectl \
RUN curl -sfL -o /tmp/flux.tar.gz \
"https://github.com/fluxcd/flux2/releases/download/v${FLUX_VERSION}/flux_${FLUX_VERSION}_linux_amd64.tar.gz" \
&& echo "4092b367fd060097976fb7261601f79420ecf7266328417dfd8dfee27de0e6a3 /tmp/flux.tar.gz" | sha256sum -c - \
&& echo "b853df82adfd7736f580692f9f734473d571606307139f8fd20c2a80dd1ff473 /tmp/flux.tar.gz" | sha256sum -c - \
&& tar -xzf /tmp/flux.tar.gz -C /usr/local/bin flux \
&& chmod +x /usr/local/bin/flux \
&& rm -f /tmp/flux.tar.gz
@@ -1,4 +1,4 @@
# netbird-proxy-services: reconciles NetBird reverse-proxy services (public
# netbird-service-sync: reconciles NetBird reverse-proxy services (public
# domains -> TLS passthrough -> Traefik ClusterIP) from Ingresses carrying the
# public label. Standard library only, so the image is the interpreter plus one
# file; runs unprivileged as its own user.
@@ -1,5 +1,5 @@
#!/usr/bin/env python3
"""netbird-proxy-services: reconcile NetBird reverse-proxy services from Ingresses.
"""netbird-service-sync: reconcile NetBird reverse-proxy services from Ingresses.
Source of truth: every Ingress carrying the public label (dns.renner.uno/public).
For each host on such an Ingress the NetBird management gets one reverse-proxy
@@ -30,11 +30,11 @@ import urllib.error
import urllib.parse
import urllib.request
LOG = logging.getLogger("netbird-proxy-services")
LOG = logging.getLogger("netbird-service-sync")
NB = os.environ.get("NB_MANAGEMENT_URL", "https://netbird.renner.uno").rstrip("/")
BAO_ADDR = os.environ.get("BAO_ADDR", "http://openbao.openbao.svc.cluster.local:8200").rstrip("/")
BAO_ROLE = os.environ.get("BAO_ROLE", "netbird-proxy-services")
BAO_ROLE = os.environ.get("BAO_ROLE", "netbird-service-sync")
BAO_PAT_PATH = os.environ.get("BAO_PAT_PATH", "homelab/data/infrastructure/secret-rotators-bootstrap/netbird")
BAO_PAT_KEY = os.environ.get("BAO_PAT_KEY", "pat")
SERVICE_CIDR = os.environ.get("SERVICE_CIDR", "10.96.0.0/12")
@@ -156,7 +156,7 @@ def ensure_network(nb):
if n["name"] == NETWORK_NAME:
return n["id"]
n = nb.post("networks", {"name": NETWORK_NAME,
"description": "managed by netbird-proxy-services; public ingress via TLS passthrough"})
"description": "managed by netbird-service-sync; public ingress via TLS passthrough"})
LOG.info("created network %s (%s)", NETWORK_NAME, n["id"])
return n["id"]
@@ -166,7 +166,7 @@ def ensure_resource(nb, net):
if r.get("address") == SERVICE_CIDR:
return r["id"]
r = nb.post(f"networks/{net}/resources", {"name": "cluster service CIDR", "address": SERVICE_CIDR,
"description": "managed by netbird-proxy-services",
"description": "managed by netbird-service-sync",
"enabled": True, "groups": []})
LOG.info("created subnet resource %s (%s)", SERVICE_CIDR, r["id"])
return r["id"]
+10
View File
@@ -0,0 +1,10 @@
# wiki-index: embedding index over the Notes repo's wiki/, fed from Gitea,
# queried by ~/Notes/bin/wiki_rank.py. Standard library only; runs unprivileged.
FROM python:3.13-slim
RUN useradd --system --no-create-home --shell /usr/sbin/nologin wikiindex
COPY wiki_index.py /opt/wiki_index.py
ENV PYTHONUNBUFFERED=1
USER wikiindex
ENTRYPOINT ["python3", "/opt/wiki_index.py"]
+178
View File
@@ -0,0 +1,178 @@
#!/usr/bin/env python3
"""Unit tests for wiki_index. Run: python3 -m unittest test_wiki_index"""
import hashlib
import hmac
import json
import tempfile
import unittest
from pathlib import Path
import wiki_index as wi
NOTE = """---
title: 'UDM intercepts DNS'
type: wiki
tags: [dns, unifi]
aliases: [udm dns interception, dig 1.1.1.1 wrong]
---
# UDM intercepts DNS
The UDM answers every UDP/53 query itself.
## Details
More.
"""
class ParseTests(unittest.TestCase):
def test_note_text(self):
title, text = wi.note_text("udm-dns", NOTE)
self.assertEqual(title, "UDM intercepts DNS")
self.assertIn("udm dns UDM intercepts DNS udm dns interception", text)
self.assertIn("dns unifi", text)
self.assertTrue(text.endswith("The UDM answers every UDP/53 query itself."))
self.assertLessEqual(len(text), wi.EMBED_CHARS)
def test_credentials_are_stripped(self):
original = wi.http
wi.http = lambda method, url, body=None, headers=None, timeout=60.0: (
{"auth": {"client_token": "t"}} if url.endswith("/login")
else {"data": {"data": {"gitea_token": "abc\n", "webhook_secret": " s3cret\n"}}})
original_sa = wi.SA_TOKEN
try:
with tempfile.TemporaryDirectory() as d:
wi.SA_TOKEN = Path(d) / "token"
wi.SA_TOKEN.write_text("jwt\n")
self.assertEqual(wi.Credentials().fetch(), {"gitea_token": "abc", "webhook_secret": "s3cret"})
finally:
wi.http, wi.SA_TOKEN = original, original_sa
def test_tree_paging_stops_on_short_last_page(self):
# Gitea: truncated=true on every page including the last, "tree": null past it.
pages = {
1: {"tree": [{"path": f"wiki/n{i}.md", "type": "blob", "sha": f"s{i}"} for i in range(1000)], "truncated": True, "total_count": 1192},
2: {"tree": [{"path": f"wiki/n{i}.md", "type": "blob", "sha": f"s{i}"} for i in range(1000, 1192)] + [{"path": "raw/x.md", "type": "blob", "sha": "r"}, {"path": "wiki/sub/y.md", "type": "blob", "sha": "y"}], "truncated": True, "total_count": 1192},
3: {"tree": None, "truncated": False, "total_count": 1192},
}
calls = []
original = wi.http
def fake_http(method, url, body=None, headers=None, timeout=60.0):
page = int(url.rsplit("page=", 1)[1])
calls.append(page)
return pages[page]
wi.http = fake_http
try:
tree = wi.Gitea("t").tree("abc")
finally:
wi.http = original
self.assertEqual(calls, [1, 2])
self.assertEqual(len(tree), 1192)
self.assertNotIn("raw/x.md", tree)
self.assertNotIn("wiki/sub/y.md", tree)
self.assertEqual(tree["wiki/n1191.md"], "s1191")
def test_signature(self):
body = b'{"ref":"refs/heads/master"}'
sig = hmac.new(b"s3cret", body, hashlib.sha256).hexdigest()
self.assertTrue(wi.verify_signature("s3cret", body, sig))
self.assertFalse(wi.verify_signature("other", body, sig))
self.assertFalse(wi.verify_signature("s3cret", body, ""))
class FakeGitea:
def __init__(self, head, tree, files):
self._head, self._tree, self.files = head, tree, files
self.raw_calls = []
def head(self):
return self._head
def tree(self, commit):
return self._tree
def raw(self, path, ref):
self.raw_calls.append(path)
return self.files[path]
class IndexTests(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
self.path = Path(self.tmp.name) / "index.json"
self.original_embed = wi.embed
self.embed_calls = []
def fake_embed(texts, batch=2):
self.embed_calls.append(list(texts))
return [wi.unit([1.0, float(len(t) % 7)]) for t in texts]
wi.embed = fake_embed
def tearDown(self):
wi.embed = self.original_embed
self.tmp.cleanup()
def test_sync_is_incremental_and_persists(self):
files = {"wiki/udm-dns.md": NOTE, "wiki/ceph.md": "---\ntitle: Ceph\n---\n# Ceph\n\nOSD.\n", "wiki/_index.md": "---\ntitle: Wiki\n---\n"}
tree = {"wiki/udm-dns.md": "aaa", "wiki/ceph.md": "bbb", "wiki/_index.md": "ccc"}
g = FakeGitea("c1", tree, files)
idx = wi.Index(self.path)
s = idx.sync(g)
self.assertEqual((s["changed"], s["removed"], s["notes"]), (2, 0, 2))
self.assertEqual(set(idx.notes), {"udm-dns", "ceph"})
self.assertEqual(idx.notes["udm-dns"]["title"], "UDM intercepts DNS")
self.assertTrue(self.path.exists())
# head unchanged: nothing fetched
s = idx.sync(g)
self.assertEqual(s.get("skipped"), "head unchanged")
self.assertEqual(len(g.raw_calls), 2)
# one note changed, one removed, new head
g2 = FakeGitea("c2", {"wiki/udm-dns.md": "aaa2"}, {"wiki/udm-dns.md": NOTE.replace("itself", "itself!")})
s = idx.sync(g2)
self.assertEqual((s["changed"], s["removed"], s["notes"]), (1, 1, 1))
self.assertEqual(g2.raw_calls, ["wiki/udm-dns.md"])
# restart loads the persisted index
idx2 = wi.Index(self.path)
self.assertEqual(idx2.commit, "c2")
self.assertEqual(set(idx2.notes), {"udm-dns"})
def test_rank_orders_by_cosine(self):
idx = wi.Index(self.path)
idx.notes = {"a": {"sha": "1", "title": "A", "vector": wi.unit([1.0, 0.0])},
"b": {"sha": "2", "title": "B", "vector": wi.unit([0.0, 1.0])}}
wi.embed = lambda texts, batch=2: [wi.unit([0.2, 1.0])]
rows = idx.rank("q", k=5)
self.assertEqual([r["slug"] for r in rows], ["b", "a"])
self.assertGreater(rows[0]["score"], rows[1]["score"])
def test_status_reports_age_and_errors(self):
idx = wi.Index(self.path)
self.assertFalse(idx.status()["ready"])
g = FakeGitea("c1", {"wiki/a.md": "x"}, {"wiki/a.md": "---\ntitle: A\n---\n# A\n\nt\n"})
idx.sync(g)
st = idx.status()
self.assertTrue(st["ready"])
self.assertEqual(st["notes"], 1)
self.assertEqual(st["commit"], "c1")
class Broken(FakeGitea):
def head(self):
raise wi.ApiError("gitea down")
with self.assertRaises(wi.ApiError):
idx.sync(Broken("c1", {}, {}))
self.assertIn("gitea down", idx.status()["last_error"])
self.assertEqual(idx.status()["notes"], 1) # last good index kept
self.assertTrue(idx.status()["ready"]) # and still served
if __name__ == "__main__":
unittest.main()
+408
View File
@@ -0,0 +1,408 @@
#!/usr/bin/env python3
"""wiki-index: embedding index over the Notes repo's wiki/, fed from Gitea.
Source of truth is the notes repository on Gitea. The index holds, per wiki
note, the git blob sha, title and a bge-m3 vector of slug/title/aliases/tags/
first paragraph. A sync fetches the repo tree, embeds every note whose blob sha
changed (via wiki-embed, in-cluster) and drops notes that left the tree; it
runs at start, on every push webhook from Gitea, and every RESYNC_SECONDS.
HTTP:
POST /rank {"query": str, "k": int} -> {"rows": [{"slug","title","score"}], "index": {...}}
GET /health -> 200 always (liveness) with {"ready", "notes", "commit", "synced_at", "age_s", "syncing", "last_error"}
GET /ready -> 200 when an index exists, else 503 (readiness)
POST /webhook Gitea push event, HMAC-SHA256 in X-Gitea-Signature -> {"queued": true}
Credentials (Gitea token, webhook secret) are read from OpenBao through the
Kubernetes auth role on every use; they never land in a Secret or a log line.
Standard library only.
"""
from __future__ import annotations
import hashlib
import hmac
import json
import logging
import os
import re
import sys
import threading
import time
import urllib.error
import urllib.parse
import urllib.request
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path
LOG = logging.getLogger("wiki-index")
GITEA_URL = os.environ.get("GITEA_URL", "https://gitea.renner.uno").rstrip("/")
NOTES_REPO = os.environ.get("NOTES_REPO", "k.renner/notes")
NOTES_BRANCH = os.environ.get("NOTES_BRANCH", "master")
WIKI_DIR = os.environ.get("WIKI_DIR", "wiki").strip("/")
EMBED_URL = os.environ.get("EMBED_URL", "http://wiki-embed.ai.svc.cluster.local:8080").rstrip("/")
EMBED_MODEL = os.environ.get("EMBED_MODEL", "bge-m3")
EMBED_CHARS = int(os.environ.get("EMBED_CHARS", "1500"))
INDEX_PATH = Path(os.environ.get("INDEX_PATH", "/data/index.json"))
RESYNC_SECONDS = int(os.environ.get("RESYNC_SECONDS", "1800"))
LISTEN_PORT = int(os.environ.get("LISTEN_PORT", "8080"))
BAO_ADDR = os.environ.get("BAO_ADDR", "http://openbao.openbao.svc.cluster.local:8200").rstrip("/")
BAO_ROLE = os.environ.get("BAO_ROLE", "wiki-index")
BAO_SECRET_PATH = os.environ.get("BAO_SECRET_PATH", "homelab/data/apps/ai/wiki-index")
SA_TOKEN = Path(os.environ.get("SA_TOKEN", "/var/run/secrets/kubernetes.io/serviceaccount/token"))
class ApiError(Exception):
pass
# ---------------------------------------------------------------- http helpers
def http(method: str, url: str, body=None, headers=None, timeout: float = 60.0):
data = None
hdrs = dict(headers or {})
if body is not None:
data = json.dumps(body).encode("utf-8")
hdrs.setdefault("Content-Type", "application/json")
req = urllib.request.Request(url, data=data, method=method, headers=hdrs)
try:
with urllib.request.urlopen(req, timeout=timeout) as resp:
raw = resp.read()
ctype = resp.headers.get("Content-Type", "")
except urllib.error.HTTPError as e:
raise ApiError(f"{method} {url}: HTTP {e.code}: {e.read().decode('utf-8', 'replace')[:300]}") from e
except urllib.error.URLError as e:
raise ApiError(f"{method} {url}: {e.reason}") from e
if "json" in ctype:
return json.loads(raw.decode("utf-8"))
return raw
# ---------------------------------------------------------------- credentials
class Credentials:
"""Gitea token and webhook secret from OpenBao, fetched per use."""
def __init__(self, static: dict | None = None):
self.static = static # tests / local runs
def fetch(self) -> dict:
if self.static is not None:
return self.static
jwt = SA_TOKEN.read_text(encoding="utf-8").strip()
login = http("POST", f"{BAO_ADDR}/v1/auth/kubernetes/login", {"role": BAO_ROLE, "jwt": jwt})
token = login["auth"]["client_token"]
kv = http("GET", f"{BAO_ADDR}/v1/{BAO_SECRET_PATH}", headers={"X-Vault-Token": token})
data = {k: str(v).strip() for k, v in kv["data"]["data"].items()} # `bao kv put key=@file` keeps the file's newline
for key in ("gitea_token", "webhook_secret"):
if not data.get(key):
raise ApiError(f"OpenBao {BAO_SECRET_PATH} has no key {key}")
return data
# ---------------------------------------------------------------- gitea
class Gitea:
def __init__(self, token: str, base: str = GITEA_URL, repo: str = NOTES_REPO, branch: str = NOTES_BRANCH):
self.base, self.repo, self.branch = base, repo, branch
self.headers = {"Authorization": f"token {token}"}
def api(self, path: str, **params):
q = ("?" + urllib.parse.urlencode(params)) if params else ""
return http("GET", f"{self.base}/api/v1/repos/{self.repo}{path}{q}", headers=self.headers)
def head(self) -> str:
return self.api(f"/branches/{urllib.parse.quote(self.branch, safe='')}")["commit"]["id"]
def tree(self, commit: str) -> dict[str, str]:
"""path -> blob sha for every wiki note at that commit."""
out: dict[str, str] = {}
page, per_page, seen = 1, 1000, 0
while True:
t = self.api(f"/git/trees/{commit}", recursive="true", per_page=per_page, page=page)
entries = t.get("tree") or [] # Gitea answers "tree": null past the last page
for e in entries:
p = e.get("path", "")
if e.get("type") == "blob" and p.startswith(WIKI_DIR + "/") and p.endswith(".md") and "/" not in p[len(WIKI_DIR) + 1:]:
out[p] = e["sha"]
seen += len(entries)
total = int(t.get("total_count") or 0)
if not entries or len(entries) < per_page or (total and seen >= total):
break
page += 1
if page > 50:
raise ApiError("tree paging runaway")
return out
def raw(self, path: str, ref: str) -> str:
data = http("GET", f"{self.base}/api/v1/repos/{self.repo}/raw/{urllib.parse.quote(path)}?ref={urllib.parse.quote(ref, safe='')}",
headers=self.headers)
return data.decode("utf-8") if isinstance(data, bytes) else json.dumps(data)
# ---------------------------------------------------------------- notes
def parse_frontmatter(text: str) -> tuple[dict, str]:
m = re.match(r"^---[ \t]*\n(.*?)\n---[ \t]*(?:\n|$)", text, re.DOTALL)
if not m:
return {}, text
fm: dict = {}
for line in m.group(1).splitlines():
mm = re.match(r"^([A-Za-z_][\w-]*):\s*(.*)$", line)
if mm:
fm[mm.group(1)] = mm.group(2).strip()
return fm, text[m.end():]
def as_list(value: str) -> list[str]:
s = (value or "").strip()
if s.startswith("[") and s.endswith("]"):
s = s[1:-1]
return [p.strip().strip("'\"") for p in s.split(",") if p.strip()]
def first_paragraph(body: str) -> str:
para: list[str] = []
seen_h1 = False
for line in body.splitlines():
if line.startswith("# "):
seen_h1 = True
continue
if not seen_h1:
continue
if line.strip() == "" or line.startswith("#"):
if para:
break
continue
para.append(line.strip())
return " ".join(para)
def note_text(slug: str, text: str) -> tuple[str, str]:
"""(title, embedding text) for one note file."""
fm, body = parse_frontmatter(text)
title = (fm.get("title") or slug).strip().strip("'\"")
parts = [slug.replace("-", " "), title, " ".join(as_list(fm.get("aliases", ""))),
" ".join(as_list(fm.get("tags", ""))), first_paragraph(body)]
return title, " ".join(parts)[:EMBED_CHARS]
# ---------------------------------------------------------------- embeddings
def embed(texts: list[str], batch: int = 2) -> list[list[float]]:
out: list[list[float]] = []
for i in range(0, len(texts), batch):
resp = http("POST", f"{EMBED_URL}/v1/embeddings", {"model": EMBED_MODEL, "input": texts[i:i + batch]}, timeout=120)
rows = sorted(resp["data"], key=lambda d: d["index"])
out.extend(unit(d["embedding"]) for d in rows)
return out
def unit(v: list[float]) -> list[float]:
n = sum(x * x for x in v) ** 0.5 or 1.0
return [round(x / n, 5) for x in v]
# ---------------------------------------------------------------- index
class Index:
def __init__(self, path: Path):
self.path = path
self.lock = threading.Lock()
self.notes: dict[str, dict] = {} # slug -> {sha, title, vector}
self.commit = ""
self.synced_at = 0.0
self.syncing = False
self.last_error = ""
if path.exists():
try:
d = json.loads(path.read_text(encoding="utf-8"))
self.notes, self.commit, self.synced_at = d.get("notes", {}), d.get("commit", ""), d.get("synced_at", 0.0)
LOG.info("loaded index: %d notes at %s", len(self.notes), self.commit[:8])
except (OSError, ValueError) as e:
LOG.warning("index unreadable (%s); starting empty", e)
def save(self) -> None:
self.path.parent.mkdir(parents=True, exist_ok=True)
tmp = self.path.with_suffix(".tmp")
tmp.write_text(json.dumps({"notes": self.notes, "commit": self.commit, "synced_at": self.synced_at}), encoding="utf-8")
tmp.replace(self.path)
def sync(self, gitea: Gitea) -> dict:
"""Bring the index to the branch head; returns a summary."""
with self.lock:
self.syncing = True
try:
head = gitea.head()
if head == self.commit and self.notes:
summary = {"commit": head, "changed": 0, "removed": 0, "notes": len(self.notes), "skipped": "head unchanged"}
self.synced_at = time.time()
self.save()
return summary
tree = gitea.tree(head)
slugs = {p[len(WIKI_DIR) + 1:-3]: (p, sha) for p, sha in tree.items() if not p.endswith("/_index.md")}
todo = [s for s, (p, sha) in slugs.items() if self.notes.get(s, {}).get("sha") != sha]
removed = [s for s in list(self.notes) if s not in slugs]
LOG.info("sync %s: %d notes, %d to embed, %d removed", head[:8], len(slugs), len(todo), len(removed))
fresh: dict[str, dict] = {}
for i in range(0, len(todo), 20):
chunk = todo[i:i + 20]
texts, titles = [], {}
for s in chunk:
title, text = note_text(s, gitea.raw(slugs[s][0], head))
titles[s] = title
texts.append(text)
for s, vec in zip(chunk, embed(texts)):
fresh[s] = {"sha": slugs[s][1], "title": titles[s], "vector": vec}
with self.lock:
for s in removed:
self.notes.pop(s, None)
self.notes.update(fresh)
self.commit = head
self.synced_at = time.time()
self.last_error = ""
self.save()
return {"commit": head, "changed": len(todo), "removed": len(removed), "notes": len(self.notes)}
except Exception as e:
with self.lock:
self.last_error = str(e)[:300]
raise
finally:
with self.lock:
self.syncing = False
def rank(self, query: str, k: int = 60) -> list[dict]:
q = embed([query])[0]
with self.lock:
rows = [(s, n["title"], sum(a * b for a, b in zip(q, n["vector"]))) for s, n in self.notes.items()]
rows.sort(key=lambda r: -r[2])
return [{"slug": s, "title": t, "score": round(sc, 4)} for s, t, sc in rows[:k]]
def status(self) -> dict:
"""`ready` = an index exists and can answer /rank; a failed sync keeps the last good index."""
with self.lock:
return {"ready": bool(self.notes), "notes": len(self.notes), "commit": self.commit,
"synced_at": self.synced_at, "age_s": round(time.time() - self.synced_at) if self.synced_at else None,
"syncing": self.syncing, "last_error": self.last_error or None}
# ---------------------------------------------------------------- syncer
class Syncer(threading.Thread):
"""One worker: syncs on demand (webhook) and every RESYNC_SECONDS."""
def __init__(self, index: Index, creds: Credentials, interval: int):
super().__init__(daemon=True, name="syncer")
self.index, self.creds, self.interval = index, creds, interval
self.wake = threading.Event()
def trigger(self) -> None:
self.wake.set()
RETRY_SECONDS = 60
def run(self) -> None:
while True:
try:
data = self.creds.fetch()
summary = self.index.sync(Gitea(data["gitea_token"]))
LOG.info("sync done: %s", summary)
wait = self.interval
except Exception as e: # keep serving the last good index; retry soon, not next interval
LOG.error("sync failed: %s", e)
wait = self.RETRY_SECONDS
self.wake.wait(wait)
self.wake.clear()
def verify_signature(secret: str, body: bytes, signature: str) -> bool:
digest = hmac.new(secret.encode("utf-8"), body, hashlib.sha256).hexdigest()
return hmac.compare_digest(digest, (signature or "").strip())
# ---------------------------------------------------------------- server
def make_handler(index: Index, syncer: Syncer, creds: Credentials):
class Handler(BaseHTTPRequestHandler):
def log_message(self, fmt, *args):
LOG.info("%s %s", self.address_string(), fmt % args)
def send_json(self, code: int, obj) -> None:
data = json.dumps(obj).encode("utf-8")
self.send_response(code)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(data)))
self.end_headers()
self.wfile.write(data)
def body(self) -> bytes:
n = int(self.headers.get("Content-Length") or 0)
return self.rfile.read(n) if n else b""
def do_GET(self):
if self.path == "/health": # liveness: the process serves
return self.send_json(200, index.status())
if self.path == "/ready": # readiness: an index exists
st = index.status()
return self.send_json(200 if st["ready"] else 503, st)
self.send_json(404, {"error": "not found"})
def do_POST(self):
if self.path == "/rank":
try:
req = json.loads(self.body() or b"{}")
query = str(req.get("query", "")).strip()
k = max(1, min(int(req.get("k", 60)), 500))
if not query:
return self.send_json(400, {"error": "query required"})
rows = index.rank(query, k)
return self.send_json(200, {"rows": rows, "index": index.status()})
except ApiError as e:
return self.send_json(502, {"error": str(e)})
except (ValueError, TypeError) as e:
return self.send_json(400, {"error": str(e)})
if self.path == "/webhook":
raw = self.body()
try:
secret = creds.fetch()["webhook_secret"]
except ApiError as e:
return self.send_json(503, {"error": str(e)})
if not verify_signature(secret, raw, self.headers.get("X-Gitea-Signature", "")):
return self.send_json(403, {"error": "bad signature"})
try:
event = json.loads(raw or b"{}")
except ValueError:
return self.send_json(400, {"error": "bad json"})
ref = event.get("ref", "")
if ref and ref != f"refs/heads/{NOTES_BRANCH}":
return self.send_json(200, {"queued": False, "ref": ref})
syncer.trigger()
return self.send_json(200, {"queued": True})
self.send_json(404, {"error": "not found"})
return Handler
def main() -> int:
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s", stream=sys.stdout)
index = Index(INDEX_PATH)
creds = Credentials()
syncer = Syncer(index, creds, RESYNC_SECONDS)
syncer.start()
server = ThreadingHTTPServer(("", LISTEN_PORT), make_handler(index, syncer, creds))
LOG.info("listening on :%d, repo %s@%s, embed %s", LISTEN_PORT, NOTES_REPO, NOTES_BRANCH, EMBED_URL)
server.serve_forever()
return 0
if __name__ == "__main__":
sys.exit(main())
+22 -16
View File
@@ -50,15 +50,18 @@ pass insert homelab/openbao/root-token
kubectl -n openbao exec -it openbao-0 -- bao operator unseal # paste key
```
**Manual unseal is required after every pod restart** (node drain, upgrade,
crash), **per replica** — the server runs three raft peers (see
`infrastructure/openbao/helmrelease.yaml`), and a sealed one is neither a
voter nor a standby, so it stays out of the cluster until unsealed:
`kubectl -n openbao exec -it openbao-<N> -- bao operator unseal`, same key for
all three. Three peers mean one sealed replica is survivable; two are not.
Accepted for now; auto-unseal needs a second trust root (transit/KMS) and is
explicitly out of scope. Symptom of sealed: ESO stores go NotReady, synced
Secrets keep their last value (they are real k8s Secrets — pods keep running).
**Unseal is automatic since 2026-09-08** (static seal, PR #825, issue #805):
the seal key lives in the `openbao-seal-key` Secret (created by `bootstrap.sh`
from `pass show homelab/openbao/seal-key`) and every restarted replica unseals
itself. The shamir key above became the **recovery key** in the migration; it
is kept in `pass` under its old name, `homelab/openbao/unseal-key`. The server
runs three raft peers (see `infrastructure/openbao/helmrelease.yaml`); a sealed
one is neither a voter nor a standby, so a replica that stays sealed after a
restart — wrong or missing seal-key Secret — costs redundancy, two cost
quorum. The trade-off taken with the static seal: the key that unlocks the
store sits in the same etcd as the Secrets ESO already syncs there in
plaintext. Symptom of sealed: ESO stores go NotReady, synced Secrets keep
their last value (they are real k8s Secrets — pods keep running).
## Phase 2 — configure + ClusterSecretStore (follow-up PR)
@@ -234,16 +237,19 @@ one full reconcile + one OpenBao pod restart/unseal cycle:
- **Backups:** ✅ nightly raft-snapshot CronJob (03:00, keeps 7 on a
dedicated PVC; see infrastructure/openbao/snapshot-cronjob.yaml) + velero
`openbao-daily` Schedule (06:15) ships that PVC offsite, 30 d retention.
A raft snapshot restore needs the matching unseal key — which lives in
A raft snapshot restore needs the matching seal key — which lives in
`pass`, outside the cluster (GPG key backed up externally 2026-07-14).
A failed snapshot Job is the sealed-after-restart alarm. Still owed:
one tested restore before phase 4.
- **Cluster rebuild order:** Flux brings up openbao (sealed, empty) → restore
raft snapshot → unseal (key from `pass`) → ClusterSecretStore Ready → ESO
materializes Secrets → apps start. Until unseal, secret-consuming apps
simply stay Pending on missing Secrets — same as any missing dependency.
- **Break-glass:** `pass` holds unseal key + root token (GPG, offline-capable).
With those two, everything is recoverable from a raft snapshot alone.
- **Cluster rebuild order:** seal-key Secret from `pass` → Flux brings up
openbao (empty) → init → restore raft snapshot → replicas unseal themselves
→ ClusterSecretStore Ready → ESO materializes Secrets → apps start
(`bootstrap.sh flux` does all of it; RUNBOOK-cold-rebuild.md Step 4 by
hand). Until then, secret-consuming apps simply stay Pending on missing
Secrets — same as any missing dependency.
- **Break-glass:** `pass` holds seal key, recovery key + root token (GPG,
offline-capable). With those, everything is recoverable from a raft
snapshot alone.
- **Ansible Vault password:** `ansible/vault-password.sh` reads
`ANSIBLE_VAULT_PASSWORD` from the environment with no offline backup as of
2026-07-14. It gates decryption of several Ansible-Vault files (PBS
+117
View File
@@ -0,0 +1,117 @@
# Traffic classes on the k8s nodes
Issue #576. Three of the four classes are separated; management is not.
## Where each class runs
| class | link | mechanism |
|---|---|---|
| Ceph replication | `ceph-san`, VLAN 49 | the whole link is handed to the OSD pod (Multus `host-device`) |
| world→pod | `ingress0`, VLAN 47 | BGP session sourced there, so the advertised next hop is that address |
| pod→world | `ingress0`, VLAN 47 | routing rule on the pod source prefix → table 47 |
| pod↔pod | `pod0`, VLAN 48 | routing rule on pod source **and** destination → table 48 |
| management | `mgmt0`, VLAN 40 | not started |
Hard constraint, set by the owner: inter-node and pod traffic never traverses
the UDM. Everything above stays on the 10G switch.
## Why policy routing rather than moving the node IP
Cilium reaches a peer's pods at that peer's Kubernetes InternalIP, in native
and tunnel mode alike. `--ipv4-node` is overwritten from the Node object, and
`nodePort.directRoutingDevice` selects the BPF NodePort device, not a next hop.
Both were established by test (#837, #838), not by reading.
So the InternalIP decides where pod traffic goes — unless the lookup happens in
a different table. The rules do that, and nothing about the node's identity
moves. The routing decision runs before Cilium's masquerade hook, so a rule
matching the pod source sees the pod address, and the masquerade then rewrites
it to the egress link's.
## Rule order
Lower priority runs first.
| prio | match | table |
|---|---|---|
| 980 | src pod, dst pod | 48 |
| 981 | src pod, dst pod (v4) | 48 |
| 990 | dst `fd00::/8` | main |
| 991 | dst `10.0.0.0/8` | main |
| 1000 | src pod | 47 |
| 1001 | src pod (v4) | 47 |
980/981 match source **and** destination. Source alone would also catch the
node's own traffic to a pod, which would then leave with a `pod0` source — an
address `CiliumNode.spec.addresses` does not list, so Cilium classifies it as
`world` rather than `host`.
990/991 are Cilium's `ipv4/ipv6NativeRoutingCIDR`, and they are what keeps
cluster-internal traffic out of table 47. Without them a pod's packet to a
service matches the source rule, finds only a default route, and leaves for the
UDM — measured, in-cluster DNS stopped resolving.
Traffic to a pod on the same node falls through on its own: table 48 holds only
the peers' CIDRs, so the lookup misses and 990/991 send it to main.
## Where each table comes from
- **47** — two default routes declared per node in `talos/patches/k8s-*.yaml`.
The IPv6 gateway is the UDM's configured address on the segment, not the
RA-learned link-local; `accept_ra_defrtr` stays `0` on `ingress0` so the RA
default does not land in main.
- **48** — filled by the `bird` DaemonSet: an iBGP mesh over `pod0`, one route
per peer pod CIDR with the peer's `pod0` address as next hop.
## The bird mesh
`kubernetes/infrastructure/bird/`. Each node advertises only its own pod CIDR
and installs what it learns into table 48. Cilium's routes in main are
untouched, which is why the two coexist.
No address is written into this repo twice. Each node publishes its `pod0`
addresses as `nodeAnnotations`, beside the `VLANConfig` that assigns them; the
pod CIDRs come from the Node objects. An init container renders both from the
API with `kubectl -o go-template-file`, excludes the node itself with a field
selector, and ends with `bird -p` so BIRD's own parser gates startup.
The node's own CIDR is **originated**, not read back from the kernel. Reading
it back made BIRD depend on the shape Cilium installs, and the IPv4 route is
rejected outright: its next hop sits inside the prefix it describes, which KRT
refuses with `strange next-hop`. The IPv6 route is a device route and passed,
so the failure was silent on one family only.
## Failure behaviour
Benign at every layer. An empty or absent table 48 makes 980/981 fall through
to main, and pod traffic returns to VLAN 40 — unseparated, not broken. The same
holds if the DaemonSet cannot start, which is the case during a cold rebuild
until Gitea serves its image.
## Management, the remaining class
Not started. The levers differ per service and none of them is the node IP:
| service | can leave the node IP? | lever |
|---|---|---|
| etcd | yes | `cluster.etcd.advertisedSubnets`; needs a reboot per node |
| kube-apiserver endpoint | yes | it is a VIP — `Layer2VIPConfig.link` |
| Talos API | n/a | `apid` answers on every node address; a firewall question |
| kubelet | no | it *is* the node IP |
Two approaches were rejected by the owner and are not to be proposed again:
moving the node IP and etcd to `pod0` (#838), and relabelling VLAN 40 as the
cluster VLAN while moving etcd and the API VIP to a new management VLAN.
Replacing the CNI (#839, Antrea) was shelved.
Note that the reasoning which produced that list also concluded pod↔pod was
blocked, which policy routing disproved. Treat it as a record of decisions, not
as an exhaustive analysis.
## Sources
Cilium v1.20.1 `pkg/datapath/linux/node.go`, `pkg/node/types/node.go`,
`pkg/node/sync/local_node_sync.go`; BIRD 2.17 user manual (kernel protocol,
`learn all`, `strict bind`, `direct`); Talos v1.13 `RoutingRuleConfig` and
`machine.nodeAnnotations`. PRs #831, #833, #835, #836, #837, #842, #852, #854,
#855; closed unmerged: #830, #838, #839.
@@ -1,194 +0,0 @@
# Proposal — separate the four traffic classes onto their own VLANs
Status: draft. The question this started with is **answered** — from the Cilium
source and flag reference, not from a test — and the answer changes the shape
of the proposal rather than confirming it.
Supersedes the framing in issue #576, which described this as "give node
management its own NIC, and Ceph public its own VLAN". That was two of four
classes and one of them turned out already solved.
## The four classes
| class | what it is | where it goes today |
|---|---|---|
| **Ceph replication** | OSD-to-OSD heartbeat, replication, recovery | VLAN 49, its own X710 port — **separated** |
| **cluster-internal** | pod-to-pod east-west, including Ceph client I/O | pod network, VXLAN over `mgmt0` |
| **cluster-external** | ingress to LoadBalancer VIPs, egress to LAN and internet | `mgmt0` |
| **management** | Talos API, etcd, kube-apiserver, kubelet | `mgmt0` |
Three of four share one interface. That is not a design decision — until the
bare-metal conversion there was one NIC to work with.
Ceph client I/O living on the pod network is correct rather than a leftover:
that traffic *is* cluster-internal. Ceph is done.
## Principle
**Each class gets its own VLAN. Which physical interface a VLAN lands on is a
separate, changeable decision.**
Defining classes by VLAN means the separation survives recabling, port failures
and capacity changes, and it does not require one port per class. Defining them
by interface would freeze a capacity decision into the architecture.
## The finding that shapes everything else
The original open question was whether `nodePort.directRoutingDevice` moves the
next-hop of Cilium's direct routes, or only selects an interface. It is
answerable from the source, and the answer is: **only the interface.**
`installDirectRoute` takes the next-hop as a parameter:
```go
func installDirectRoute(log *slog.Logger, prefix netip.Prefix, nodeIP net.IP,
skipUnreachable bool) (routeSpec *netlink.Route, err error)
```
and `nodeUpdate` supplies it as `newNode.GetNodeIP(false)`. `GetNodeIP` returns,
in priority order, `NodeInternalIP` → `NodeExternalIP` → anything else. The
same `newIP4` is what tunnel mode uses as the VXLAN endpoint.
`NodeInternalIP` comes from the Kubernetes Node object. `--ipv4-node` looks like
an override but is not one: `initFromConfig()` calls `SetNodeInternalIP` and
`initFromK8s()` runs **after** it and overwrites the value. In a Kubernetes
cluster there is no path by which Cilium's node IP differs from the kubelet's
`InternalIP`.
`--direct-routing-device` is documented as "Device name used to connect nodes in
direct routing mode" — a device, not an address.
**So cluster-internal traffic is welded to the node IP, and the node IP is the
kubelet InternalIP.** Two consequences:
- The class cannot be moved off whatever VLAN the node IP lives on.
- **Routing mode is irrelevant to this problem.** Tunnel mode sources the VXLAN
from the node IP; native mode uses it as the direct-route next-hop. Native
routing buys nothing here, so the disruptive, non-rolling step this proposal
previously contained is simply gone.
## Consequence: draw the line the other way
If cluster-internal cannot leave the node IP, then separation has to come from
moving *management services* off it — and each one has to be checked
individually, because they do not share a mechanism.
| management service | can it leave the node IP? | lever |
|---|---|---|
| etcd | **yes** | `cluster.etcd.advertisedSubnets` (+ `listenSubnets`, which defaults to it) |
| kube-apiserver endpoint | **yes** | already a VIP — `Layer2VIPConfig` with a `link:` field |
| Talos API | **n/a** | `apid` answers on every node address (measured); a client and firewall choice, not a bind setting |
| kubelet | **no** | it *is* the node IP, by definition |
So four VLANs are reachable, but the boundary does not fall where the class
names suggest: **kubelet stays with cluster-internal.** That is the honest
result, and it is worth stating plainly rather than filing kubelet under
management and hoping.
The remaining classes keep the levers established earlier:
- **cluster-external ingress** — `CiliumBGPNodeConfigOverride.spec.bgpInstances[].peers[].localAddress`.
For eBGP the session source is the next-hop, so pinning the session pins
where return traffic arrives. The session to the UDM has been running for
months.
- **cluster-external egress** — eBPF masquerading follows the output device, so
this is a default-route decision plus `enable-masquerade-to-route-source`
(today `false`).
- **Ceph replication** — Multus macvlan on VLAN 49, done.
## The general escape hatch
Ceph is separated because its OSDs got a second interface and stopped using the
pod network for replication. That is the *only* mechanism that fully detaches a
workload from the node IP, and it generalises: anything that must not ride the
node IP gets its own `NetworkAttachmentDefinition`.
It is also expensive per workload — an attachment, an IPAM pool, a pod spec
change. Worth it for Ceph replication. Not worth it for east-west traffic in
general.
## Recommended mapping
Two ways to arrange this, and they differ only in which VLAN gets relabelled:
**A — move the node IP to a new cluster VLAN.** VLAN 40 keeps etcd, the API VIP
and Talos API. Cleanest match to the class names, but it changes every node's
identity, which is the one address the whole cluster is administered through.
**B (recommended) — leave the node IP on VLAN 40 and move management off it.**
VLAN 40 becomes the cluster VLAN; a new VLAN carries etcd and the API VIP.
Identical end state, and no node ever changes its InternalIP.
Under B:
| VLAN | carries |
|---|---|
| 49 | Ceph replication |
| 40 (relabelled *cluster*) | pod east-west, kubelet |
| new *management* | etcd, kube-apiserver VIP, Talos API by convention |
| new *external* | LoadBalancer ingress, egress default route |
The property the user asked for — management carries only management traffic —
holds under B. The label moves; the property is what matters.
## Migration shape
Every step is rolling and reversible. The routing-mode change is gone.
1. **New VLANs** on the switch, trunked to the X710. Nothing uses them.
2. **Addresses** on the new VLANs via Talos, one node at a time, `--mode=try`.
Still unused; this only proves the L2 works.
3. **etcd** to the management VLAN via `advertisedSubnets`. One node at a time,
waiting for a healthy quorum between each.
4. **API VIP** to the management VLAN via `Layer2VIPConfig.link`. DNS for
`k8s.renner.uno` follows.
5. **cluster-external**: BGP `localAddress`, default route, masquerade flag.
Reversible by reverting the override; failure is visible immediately in
`cilium-dbg shell -- bgp/peers`.
6. Verify VLAN 40 by what is *absent* from it, not by what still works.
Step 3 is the one that deserves care — etcd is the only step where a mistake
costs quorum rather than connectivity.
## What this does not buy
Management currently shares a 10 Gb link and is not contended. Nothing observed
today is slow because of this. The case for the split is that traffic classes
with different failure modes and security postures should not be
indistinguishable — not that something is broken.
## Sources and measurements
Source, Cilium v1.20.1 (the running version):
- `pkg/datapath/linux/node.go` — `installDirectRoute(…, nodeIP net.IP, …)`,
fed from `newIP4 = newNode.GetNodeIP(false)`; `createDirectRouteSpec` rejects
a next-hop that is not directly reachable.
- `pkg/node/types/node.go` — `GetNodeIP` prefers `NodeInternalIP`.
- `pkg/node/sync/local_node_sync.go` — `initFromConfig()` applies
`--ipv4-node`, `initFromK8s()` runs afterwards and overwrites it from the
Kubernetes Node.
- `cilium-agent` flag reference — `--direct-routing-device`: "Device name used
to connect nodes in direct routing mode".
Talos:
- Multihoming guide — `cluster.etcd.advertisedSubnets`; `listenSubnets`
defaults to it. kube-apiserver and Talos API listen addresses are not
covered there.
- `kubernetes/talos/patches/k8s-{1,2,3}.yaml` — `Layer2VIPConfig` for
`10.2.40.60` with `link: mgmt0`.
Measured on the cluster:
- `apid` answered on a node address that is not the InternalIP, so the Talos
API is not bound to one address.
- k8s-1 currently holds one routable v4 address plus the VIP on `mgmt0`; its
InternalIP had changed since the last session, which is why nothing here
names an address as fixed.
- Cilium runs `routing-mode: tunnel` with `enable-masquerade-to-route-source`
unset and no direct-routing device configured.
Not verified: whether the kube-apiserver's `--advertise-address` follows the
node IP independently of the VIP. It does not affect the recommendation, since
clients reach the API through the VIP either way.
+2 -2
View File
@@ -432,8 +432,8 @@ is why it belongs here.
among its devices but not `ens19`, because `ens19` is deliberately
address-less. Cilium therefore owns `mgmt0` and sees VLAN 40 frames already
untagged by the 8021q layer, while VLAN 30 frames only ever meet the filter
on the shared parent `enp2s0f0np0`, still tagged. Any future address-less
VLAN carrying pod traffic will need the same treatment.
on the shared parent `uplink`, still tagged. Any future address-less VLAN
carrying pod traffic will need the same treatment.
The symptom does not look like a VLAN problem. A tc-layer drop reaches an
AF_PACKET sender as ENOBUFS, so the CNI dhcp plugin reports `no buffer space
+60 -34
View File
@@ -67,10 +67,12 @@ These must exist outside the cluster or the rebuild cannot complete:
`flux` phase creates the `sops-age` Secret, and any not-yet-migrated sops
files decrypt with it. (Post-OpenBao-phase-4 this may be retired — see
`MIGRATION-sops-openbao.md`.)
- [ ] **OpenBao unseal key + root token** — in the GPG `pass` store
(`homelab/openbao/unseal-key`, `homelab/openbao/root-token`), and the GPG
private key backed up off-machine. Without the unseal key, no OpenBao
snapshot can be restored and no secret-consuming app can start.
- [ ] **OpenBao seal key, recovery key + root token** — in the GPG `pass`
store (`homelab/openbao/seal-key`, `homelab/openbao/unseal-key` — the
former shamir key, now the recovery key — and `homelab/openbao/root-token`),
and the GPG private key backed up off-machine. The seal key is the one
the snapshots are encrypted under: without it, no OpenBao snapshot can be
restored and no secret-consuming app can start.
- [ ] **OpenBao raft snapshot** reachable — nightly PVC snapshots
(`infrastructure/openbao/snapshot-cronjob.yaml`) plus the Velero
`openbao-daily` offsite copy. In a total-loss rebuild the offsite Velero
@@ -294,42 +296,65 @@ output** — a green step over a still-forming Ceph is a known trap.
> Steady-state (non-rebuild) bootstrap is just `kubernetes/bootstrap.sh flux`
> with `GITEA_TOKEN` set; `FLUX_SOURCE` defaults to `gitea`.
## Step 4 — openbao: restore the snapshot + unseal
## Step 4 — openbao: restore the snapshot
After Step 3, OpenBao comes up **sealed and empty**. Until it is restored and
unsealed, the `ClusterSecretStore` stays NotReady, ESO cannot materialize
Secrets, and every secret-consuming app stays Pending. This step is **manual**
(the unseal key lives in `pass`, off-cluster by design).
`bootstrap.sh flux` does this step itself (`restore_openbao`): it creates the
`openbao-seal-key` Secret from `pass` before Flux brings OpenBao up, initialises
the empty server, restores the newest raft snapshot straight out of the Velero
kopia repository, and waits until every replica has unsealed itself from the
static seal. What follows is the same procedure by hand, for when the script
is not an option.
OpenBao unseals itself from the static seal (`openbao-seal-key` Secret, key
`key`, the 32-byte base64 key from `pass show homelab/openbao/seal-key`). The
snapshots are encrypted under that key, so it must be the same key the backed
up cluster used — a different one restores nothing. Until the snapshot is
restored, the `ClusterSecretStore` stays NotReady, ESO cannot materialize
Secrets, and every secret-consuming app stays Pending.
```sh
# 1. Wait for the OpenBao pods to exist (all three sealed, 0/1 Ready is expected).
kubectl -n openbao get pods
# 1. The seal key, before the pods start (a pod without it never starts).
kubectl create namespace openbao --dry-run=client -o yaml | kubectl apply -f -
kubectl -n openbao create secret generic openbao-seal-key \
--from-literal=key="$(pass show homelab/openbao/seal-key)"
# 2. Restore the most recent raft snapshot. Source it from the snapshot PVC
# 2. Wait for openbao-0 (uninitialised: 0/1 Ready is expected), then
# initialise it. With an auto-seal, init hands out recovery keys and the
# server unseals itself. These keys are throwaway: the restore replaces them.
kubectl -n openbao exec -it openbao-0 -- bao operator init \
-recovery-shares=1 -recovery-threshold=1
# 3. Restore the most recent raft snapshot. Source it from the snapshot PVC
# (infrastructure/openbao/snapshot-cronjob.yaml) or, in a total-loss rebuild,
# from the Velero `openbao-daily` offsite copy. Copy it into the pod, then:
# from the Velero `openbao-daily` offsite copy. Root token from the init
# output. No -force: OpenBao checks the snapshot was written under this
# server's seal key and refuses otherwise ("different autoseal key") — that
# refusal is the signal the Secret is wrong. -force would skip the check and
# leave a server that can never unseal.
kubectl -n openbao cp <snapshot>.snap openbao-0:/tmp/restore.snap
kubectl -n openbao exec -it openbao-0 -- bao operator raft snapshot restore /tmp/restore.snap
kubectl -n openbao exec -it openbao-0 -- \
env BAO_TOKEN=<init root token> bao operator raft snapshot restore /tmp/restore.snap
# 3. Unseal with the key from pass (threshold 1) — EVERY replica, same key.
# A sealed server is neither a raft voter nor a standby, so unsealing only
# openbao-0 leaves the three-peer cluster one node short of quorum.
kubectl -n openbao exec -it openbao-0 -- bao operator unseal # paste homelab/openbao/unseal-key
kubectl -n openbao exec -it openbao-1 -- bao operator unseal # same key
kubectl -n openbao exec -it openbao-2 -- bao operator unseal # same key
# 4. The server seals against the snapshot's keyring and unseals itself from
# the static seal; openbao-1 and openbao-2 follow (OrderedReady). Nothing
# to type. Afterwards the tokens and recovery key from pass apply again.
kubectl -n openbao get pods -w
```
Verify the store recovers before moving on:
```sh
kubectl get clustersecretstore openbao # want: Valid / Ready
kubectl -n <ns> get externalsecret # want: SecretSynced
kubectl -n openbao exec openbao-0 -- bao status # want: Seal Type static, Sealed false
kubectl get clustersecretstore openbao # want: Valid / Ready
kubectl -n <ns> get externalsecret # want: SecretSynced
```
> **Manual unseal is required after every OpenBao pod restart**, not just
> during rebuilds, and for *each* replica that restarts — a rolling node
> drain seals all three in turn. Details + rationale:
> `MIGRATION-sops-openbao.md`.
> A replica that stays **sealed** after a restart could not use the static
> seal — the Secret is missing or does not match the key the data was
> written under. Fix the Secret and restart the pod; `bao operator unseal`
> with the recovery key (`pass show homelab/openbao/unseal-key`) does not
> unseal an auto-sealed server and is only for recovery-mode operations.
> Details + rationale: `MIGRATION-sops-openbao.md`.
## Step 5 — apps: let the rest reconcile
@@ -393,14 +418,15 @@ it: repo → **Settings → Repository → Mirror Settings** in Gitea, or
- **`bootstrap.sh talos` is dead** — it cannot generate a config (#581), and
Step 2 no longer uses it. Its `flux` and `wait-apps` phases are still used in
Steps 3, 5 and 6 and work; do not read this as the whole script being gone.
It has also never had an `openbao` phase — Step 4 is manual because the
unseal key must not live in CI; see issue #187 on whether to wire a
semi-automated OpenBao restore.
- **OpenBao seals on every restart and each replica must be unsealed
individually.** Three raft replicas mean the service survives losing one, but
a sealed server is neither a voter nor a standby — unsealing only one leaves
the cluster short of quorum. Step 4 covers this; it is repeated here because
it is the step most likely to be half-done under pressure.
Its `flux` phase carries the OpenBao restore (Step 4); the seal key and the
other secrets are read from `pass` or streamed in at run time and never live
in CI.
- **OpenBao unseals itself, but only under the seal key the snapshot was
written with.** Three raft replicas mean the service survives losing one; a
sealed server is neither a voter nor a standby. A pod that stays sealed after
a restart has the wrong `openbao-seal-key` Secret — Step 4 covers this; it is
repeated here because it is the step most likely to be half-done under
pressure.
- Ceph's cluster network is a Multus attachment on VLAN 49 with whereabouts
IPAM, not host networking. Nothing in the rebuild needs special handling for
it — the nodes hold no address on that segment — but a rebuilt cluster will
@@ -21,6 +21,9 @@ metadata:
namespace: audiobookshelf
labels:
app.kubernetes.io/name: audiobookshelf
annotations:
# envFrom does not restart the pod when the ConfigMap changes; Reloader does.
configmap.reloader.stakater.com/reload: audiobookshelf-config
spec:
replicas: 1
strategy:
@@ -67,7 +70,7 @@ spec:
mountPath: /ca
containers:
- name: audiobookshelf
image: ghcr.io/advplyr/audiobookshelf:2.36.0 # {"$imagepolicy": "flux-system:audiobookshelf"}
image: ghcr.io/advplyr/audiobookshelf:2.36.1 # {"$imagepolicy": "flux-system:audiobookshelf"}
ports:
- name: http
containerPort: 80
@@ -40,7 +40,7 @@ metadata:
# so this Service produces no public record — LDAP stays
# LAN-internal, which matches the threat model of the cleartext
# bind on 389.
external-dns.alpha.kubernetes.io/hostname: ldap.renner.uno
external-dns.kubernetes.io/hostname: ldap.renner.uno
spec:
type: LoadBalancer
externalTrafficPolicy: Local
@@ -3,3 +3,4 @@ kind: Kustomization
resources:
- externalsecret.yaml
- helmrelease.yaml
- networkpolicy.yaml
@@ -0,0 +1,22 @@
# The apps namespace is default-deny-ingress, so the ServiceMonitor added in
# monitoring-rules/external-dns-servicemonitor.yaml scraped into a timeout.
# The selector matches both releases — the Cloudflare one deployed here and
# external-dns-unifi from infrastructure/ — since they share the chart's
# app.kubernetes.io/name and land in the same namespace.
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: external-dns-allow-metrics-scrape
namespace: apps
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: external-dns
ingress:
- fromEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: monitoring
toPorts:
- ports:
- port: "7979"
protocol: TCP
+1 -1
View File
@@ -2,7 +2,7 @@ apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: gitea
namespace: apps
namespace: gitea
labels:
app.kubernetes.io/name: gitea
annotations:
+1 -1
View File
@@ -2,7 +2,7 @@ apiVersion: traefik.io/v1alpha1
kind: IngressRouteTCP
metadata:
name: gitea-ssh
namespace: apps
namespace: gitea
labels:
app.kubernetes.io/name: gitea
spec:
+6 -26
View File
@@ -1,28 +1,8 @@
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: gitea-allow-metrics-scrape
namespace: apps
spec:
# Prometheus (monitoring ns) scrapes gitea's /metrics on :3000. The apps
# default-deny-ingress only allows same-ns / home-assistant / host, so the
# scrape is dropped (context deadline exceeded). This became necessary only
# after the LXC->in-cluster cutover — the old static target was external, so
# no in-cluster policy applied. Scoped to the gitea pod + :3000.
endpointSelector:
matchLabels:
app.kubernetes.io/name: gitea
ingress:
- fromEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": monitoring
toPorts:
- ports:
- port: "3000"
protocol: TCP
---
# Prometheus scrape and Traefik are admitted by the namespace baseline in
# infrastructure/network-policies/gitea-ns.yaml.
#
# Secret rotation: the flux-git rotator (openbao ns) reaches gitea to mint a new
# flux PAT. It uses the in-cluster Service (gitea.apps.svc:3000) — the public
# flux PAT. It uses the in-cluster Service (gitea.gitea.svc:3000) — the public
# gitea.renner.uno name hairpins off the ingress LB from a pod (same failure as
# pve.renner.uno). Scoped to the gitea pod :3000 and the rotator label; the
# egress half is rotator-gitea-flux-egress in network-policies/openbao-ns.yaml.
@@ -30,7 +10,7 @@ apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: gitea-allow-flux-rotator
namespace: apps
namespace: gitea
spec:
endpointSelector:
matchLabels:
@@ -54,7 +34,7 @@ apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: gitea-allow-review-bot-rotator
namespace: apps
namespace: gitea
spec:
endpointSelector:
matchLabels:
+1 -1
View File
@@ -13,7 +13,7 @@ apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: gitea
namespace: apps
namespace: gitea
labels:
app.kubernetes.io/name: gitea
spec:
+1 -1
View File
@@ -2,7 +2,7 @@ apiVersion: v1
kind: Service
metadata:
name: gitea
namespace: apps
namespace: gitea
labels:
app.kubernetes.io/name: gitea
spec:
+2 -2
View File
@@ -2,7 +2,7 @@ apiVersion: apps/v1
kind: StatefulSet
metadata:
name: gitea
namespace: apps
namespace: gitea
labels:
app.kubernetes.io/name: gitea
spec:
@@ -152,7 +152,7 @@ spec:
# NOTE: volumeClaimTemplates is immutable on an existing StatefulSet,
# so bumping this value cannot be `kubectl apply`-ed in place — Flux
# will reject the whole object. Landing this size change requires a
# one-time `kubectl delete statefulset gitea -n apps --cascade=orphan`
# one-time `kubectl delete statefulset gitea -n gitea --cascade=orphan`
# (pod + PVC keep running) so Flux re-creates the STS from this
# template and re-adopts them. After the cleanup_packages GC reclaims
# the ~10G of orphaned blobs, real usage is ~9G, so 40Gi is headroom.
@@ -18,7 +18,7 @@ metadata:
lb-pool: bgp
annotations:
lbipam.cilium.io/ips: "10.2.99.87,fd99::87"
external-dns.alpha.kubernetes.io/hostname: ha.renner.uno
external-dns.kubernetes.io/hostname: ha.renner.uno
spec:
type: LoadBalancer
externalTrafficPolicy: Local
@@ -95,7 +95,7 @@ spec:
memory: 32Mi
containers:
- name: home-assistant
image: ghcr.io/home-assistant/home-assistant:2026.9.1 # {"$imagepolicy": "flux-system:home-assistant"}
image: ghcr.io/home-assistant/home-assistant:2026.9.3 # {"$imagepolicy": "flux-system:home-assistant"}
env:
- name: TZ
value: Europe/Berlin
@@ -19,7 +19,7 @@ metadata:
# LAN can connect with `mqtt.renner.uno` (matches the broker cert
# SAN) instead of the bare IP. No Cloudflare label — kept off the
# public DNS; UniFi serves a split-DNS answer for LAN clients.
external-dns.alpha.kubernetes.io/hostname: mqtt.renner.uno
external-dns.kubernetes.io/hostname: mqtt.renner.uno
spec:
type: LoadBalancer
externalTrafficPolicy: Local
+1 -1
View File
@@ -32,7 +32,7 @@ spec:
type: RuntimeDefault
containers:
- name: immich-machine-learning
image: ghcr.io/immich-app/immich-machine-learning:v3.1.0 # {"$imagepolicy": "flux-system:immich-machine-learning"}
image: ghcr.io/immich-app/immich-machine-learning:v3.2.2 # {"$imagepolicy": "flux-system:immich-machine-learning"}
ports:
- name: http
containerPort: 3003
+4 -1
View File
@@ -35,6 +35,9 @@ metadata:
namespace: immich
labels:
app.kubernetes.io/name: immich-server
annotations:
# envFrom does not restart the pod when the ConfigMap changes; Reloader does.
configmap.reloader.stakater.com/reload: immich-config
spec:
# Scaled to 0 alongside immich-machine-learning during single-node
# maintenance; see the note there. The CNPG Postgres cluster stays running,
@@ -84,7 +87,7 @@ spec:
mountPath: /ca
containers:
- name: immich-server
image: ghcr.io/immich-app/immich-server:v3.1.0 # {"$imagepolicy": "flux-system:immich-server"}
image: ghcr.io/immich-app/immich-server:v3.2.2 # {"$imagepolicy": "flux-system:immich-server"}
ports:
- name: http
containerPort: 3001
+1 -1
View File
@@ -3,8 +3,8 @@ kind: Kustomization
resources:
- gatus/
- external-dns/
- gitea/
- ical-proxy/
# gitea/ has its own `gitea` Kustomization (clusters/homelab/gitea.yaml, #878).
# woodpecker/ moved to its own dedicated `woodpecker` Kustomization
# (clusters/homelab/woodpecker.yaml) so its wp-pg CNPG cluster's barman
# dependency + Cluster health-gating don't contaminate this shared apps KS.
+27 -4
View File
@@ -42,9 +42,11 @@ spec:
type: RuntimeDefault
initContainers:
# Seed the libation-config PVC from the bootstrap secret on
# first run only — sentinel file gates re-runs so token
# refreshes in AccountsSettings.json and DB updates from
# subsequent liberate passes are preserved.
# first run only — sentinel file gates re-runs so DB updates
# from subsequent liberate passes are preserved. Refreshed
# Audible tokens are preserved by the container args, not by
# this sentinel: the entrypoint copies AccountsSettings.json
# into ephemeral /config-internal rather than symlinking it.
- name: seed-config
image: busybox:1.37.0
command: ["/bin/sh", "-eu", "-c"]
@@ -66,6 +68,11 @@ spec:
install -m 660 /bootstrap/AccountsSettings.json /libation-state/AccountsSettings.json
install -m 660 /bootstrap/Settings.json /libation-state/Settings.json
install -m 660 /bootstrap/LibationContext.db /libation-state/LibationContext.db
# Without this key the seeded tokens cannot be decrypted:
# TokenStorageMethod is Encrypted and the container has no
# OS secret store, so a missing key file makes the
# entrypoint mint a throwaway one on every run.
install -m 660 /bootstrap/libation-master.key /libation-state/libation-master.key
touch /libation-state/.bootstrapped
ls -la /libation-state
volumeMounts:
@@ -85,7 +92,23 @@ spec:
type: RuntimeDefault
containers:
- name: libation
image: docker.io/rmcrackan/libation:latest
# Pinned by digest: Spegel caches mutable tag resolutions, so
# :latest stayed on 13.7.11 despite imagePullPolicy Always.
image: docker.io/rmcrackan/libation:14.2.1@sha256:6f9d8ba19dc3803dd6ba295ab5a7abfb6ff3bc06286eebf4ecd8ccbe43245888
# The entrypoint stages config into ephemeral /config-internal
# by copy (only the DB is symlinked to the PVC), so Audible
# tokens refreshed during a run are lost at container exit.
# Passing args makes the entrypoint set up the environment and
# then exec this instead of its own loop; the trap copies the
# refreshed file back to the PVC on every exit path.
args:
- /bin/sh
- -c
- |
trap 'cp /config-internal/AccountsSettings.json /libation-state/AccountsSettings.json' EXIT
set -e
/libation/LibationCli scan
/libation/LibationCli liberate
env:
- name: LIBATION_CONFIG_DIR
value: /libation-state
@@ -6,3 +6,5 @@ resources:
- embed.yaml
- gate-service.yaml
- ingress.yaml
- wiki-embed.yaml
- wiki-embed-ingress.yaml
@@ -0,0 +1,57 @@
---
# wiki-embed.renner.uno — LAN-only (no dns.renner.uno/public label), consumed by
# bin/wiki_rank.py on the workstation and the nightly wiki jobs. Machine client,
# so Traefik BasicAuth instead of the browser-only Authentik gate; same pattern
# as kubernetes/apps/monitoring/loki-ingress.yaml.
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: wiki-embed-basic-auth
namespace: ai
spec:
basicAuth:
secret: wiki-embed-basic-auth
realm: wiki-embed
---
# OpenBao key apps/ai/wiki-embed-basic-auth, field `users` = htpasswd line.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: wiki-embed-basic-auth
namespace: ai
spec:
refreshInterval: 1h
secretStoreRef:
kind: ClusterSecretStore
name: openbao
target:
name: wiki-embed-basic-auth
dataFrom:
- extract:
key: apps/ai/wiki-embed-basic-auth
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: wiki-embed
namespace: ai
annotations:
cert-manager.io/cluster-issuer: letsencrypt-cluster
traefik.ingress.kubernetes.io/router.middlewares: ai-wiki-embed-basic-auth@kubernetescrd
spec:
ingressClassName: traefik
tls:
- hosts:
- wiki-embed.renner.uno
secretName: wiki-embed-tls
rules:
- host: wiki-embed.renner.uno
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: wiki-embed
port:
number: 8080
@@ -0,0 +1,117 @@
---
# Embedding endpoint for the ~/Notes wiki ranker (bin/wiki_rank.py). Own
# Deployment so wiki traffic never queues behind Paperless on llama-embed.
# Same model and PVC as llama-embed, so no second download.
apiVersion: apps/v1
kind: Deployment
metadata:
name: wiki-embed
namespace: ai
labels:
app.kubernetes.io/name: wiki-embed
spec:
replicas: 1
# Above the 600s startupProbe budget; see embed.yaml.
progressDeadlineSeconds: 900
strategy:
type: Recreate
selector:
matchLabels:
app.kubernetes.io/name: wiki-embed
template:
metadata:
labels:
app.kubernetes.io/name: wiki-embed
spec:
securityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
seccompProfile:
type: RuntimeDefault
containers:
- name: wiki-embed
image: ghcr.io/ggml-org/llama.cpp:server-vulkan-b10731
args:
- --hf-repo
- gpustack/bge-m3-GGUF:Q8_0
- --embeddings
- --alias
- bge-m3
# bge-m3 is trained for CLS pooling; the GGUF default is not.
- --pooling
- cls
- --n-gpu-layers
- "99"
- --threads
- "2"
# wiki_rank.py caps each note at 1500 chars and sends 2 per request.
- --batch-size
- "1536"
- --ubatch-size
- "1536"
- --ctx-size
- "4096"
- --no-webui
env:
- name: LLAMA_CACHE
value: /models
ports:
- name: http
containerPort: 8080
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
# Same sizing as llama-embed: 1.6 GiB measured peak, headroom for slots.
resources:
requests:
cpu: 250m
memory: 2Gi
gpu.intel.com/i915: "1"
limits:
cpu: "2"
memory: 8Gi
gpu.intel.com/i915: "1"
startupProbe:
httpGet:
path: /health
port: http
periodSeconds: 10
failureThreshold: 60
readinessProbe:
httpGet:
path: /health
port: http
periodSeconds: 10
livenessProbe:
httpGet:
path: /health
port: http
periodSeconds: 30
timeoutSeconds: 10
failureThreshold: 6
volumeMounts:
- name: models
mountPath: /models
volumes:
- name: models
persistentVolumeClaim:
claimName: llama-server-models
---
apiVersion: v1
kind: Service
metadata:
name: wiki-embed
namespace: ai
labels:
app.kubernetes.io/name: wiki-embed
spec:
selector:
app.kubernetes.io/name: wiki-embed
ports:
- name: http
port: 8080
targetPort: http
@@ -0,0 +1,25 @@
# Both external-dns releases (Cloudflare and the UniFi webhook) carry
# app.kubernetes.io/name: external-dns and differ only by instance, so one
# monitor covers both. jobLabel keeps them apart as separate job values.
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: external-dns
namespace: monitoring
labels:
app.kubernetes.io/name: external-dns
release: kube-prometheus-stack
spec:
jobLabel: app.kubernetes.io/instance
namespaceSelector:
matchNames:
- apps
selector:
matchLabels:
app.kubernetes.io/name: external-dns
endpoints:
# The controller's own metrics. Port 8080 on the unifi release is the
# webhook sidecar and exports no external_dns_* series.
- port: http
path: /metrics
interval: 60s
@@ -0,0 +1,20 @@
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: intel-gpu-exporter
namespace: monitoring
labels:
app.kubernetes.io/name: intel-gpu-exporter
release: kube-prometheus-stack
spec:
selector:
matchLabels:
app.kubernetes.io/name: intel-gpu-exporter
endpoints:
- port: http-metrics
path: /metrics
interval: 30s
relabelings:
# The series would otherwise carry only the exporter pod's IP.
- sourceLabels: [__meta_kubernetes_pod_node_name]
targetLabel: node
@@ -21,6 +21,14 @@ resources:
# alerted if that stopped — HetznerBackupEndpointDown only probes the S3
# endpoint, which proves reachability, not that anything was written.
- prometheusrule-cnpg-backup.yaml
# DNS publication. Nothing scraped external-dns, so it deleted six LAN
# records on 2026-09-12 and stayed silent for five days.
- prometheusrule-external-dns.yaml
- external-dns-servicemonitor.yaml
# Outcome check for the same failure: the names themselves, asked of the
# UDM. Catches a cause external-dns cannot see, e.g. records removed there.
- prometheusrule-dns-resolution.yaml
- probes-dns.yaml
- alloy-servicemonitor.yaml
- cilium-servicemonitor.yaml
- cnpg-podmonitors.yaml
@@ -39,4 +47,5 @@ resources:
- immich-servicemonitor.yaml
- loki-servicemonitor.yaml
- smartctl-exporter-servicemonitor.yaml
- intel-gpu-exporter-servicemonitor.yaml
- probes-blackbox.yaml
@@ -0,0 +1,143 @@
# DNS probes for the names external-dns publishes to the UDM from
# LoadBalancer Services. One Probe per name because the dns prober's
# query_name is module-scoped; the target is the resolver, queried on both
# families so a single-stack blip does not read as a missing record.
#
# The UDM is queried directly rather than through the cluster resolver:
# k8s-gateway caches these with `serve_stale 1h immediate`, which would keep
# a deleted record green for an hour.
#
# `release` is load-bearing — probeSelector matches on it.
apiVersion: monitoring.coreos.com/v1
kind: Probe
metadata:
name: dns-lan-ha
namespace: monitoring
labels:
release: kube-prometheus-stack
spec:
jobName: blackbox-dns
prober:
url: blackbox-exporter-prometheus-blackbox-exporter.monitoring.svc.cluster.local:9115
module: dns_lan_ha
interval: 60s
scrapeTimeout: 10s
targets:
staticConfig:
static:
- "[fd00::1]:53"
- "10.2.40.1:53"
# instance is the resolver and identical across these Probes, so the
# queried name has to come from a label.
labels:
dns_name: ha.renner.uno
---
apiVersion: monitoring.coreos.com/v1
kind: Probe
metadata:
name: dns-lan-mqtt
namespace: monitoring
labels:
release: kube-prometheus-stack
spec:
jobName: blackbox-dns
prober:
url: blackbox-exporter-prometheus-blackbox-exporter.monitoring.svc.cluster.local:9115
module: dns_lan_mqtt
interval: 60s
scrapeTimeout: 10s
targets:
staticConfig:
static:
- "[fd00::1]:53"
- "10.2.40.1:53"
labels:
dns_name: mqtt.renner.uno
---
apiVersion: monitoring.coreos.com/v1
kind: Probe
metadata:
name: dns-lan-ldap
namespace: monitoring
labels:
release: kube-prometheus-stack
spec:
jobName: blackbox-dns
prober:
url: blackbox-exporter-prometheus-blackbox-exporter.monitoring.svc.cluster.local:9115
module: dns_lan_ldap
interval: 60s
scrapeTimeout: 10s
targets:
staticConfig:
static:
- "[fd00::1]:53"
- "10.2.40.1:53"
labels:
dns_name: ldap.renner.uno
---
apiVersion: monitoring.coreos.com/v1
kind: Probe
metadata:
name: dns-lan-syslog
namespace: monitoring
labels:
release: kube-prometheus-stack
spec:
jobName: blackbox-dns
prober:
url: blackbox-exporter-prometheus-blackbox-exporter.monitoring.svc.cluster.local:9115
module: dns_lan_syslog
interval: 60s
scrapeTimeout: 10s
targets:
staticConfig:
static:
- "[fd00::1]:53"
- "10.2.40.1:53"
labels:
dns_name: syslog.renner.uno
---
apiVersion: monitoring.coreos.com/v1
kind: Probe
metadata:
name: dns-lan-ftp-scanner
namespace: monitoring
labels:
release: kube-prometheus-stack
spec:
jobName: blackbox-dns
prober:
url: blackbox-exporter-prometheus-blackbox-exporter.monitoring.svc.cluster.local:9115
module: dns_lan_ftp_scanner
interval: 60s
scrapeTimeout: 10s
targets:
staticConfig:
static:
- "[fd00::1]:53"
- "10.2.40.1:53"
labels:
dns_name: ftp.paperless.renner.uno
---
apiVersion: monitoring.coreos.com/v1
kind: Probe
metadata:
name: dns-lan-syncthing
namespace: monitoring
labels:
release: kube-prometheus-stack
spec:
jobName: blackbox-dns
prober:
url: blackbox-exporter-prometheus-blackbox-exporter.monitoring.svc.cluster.local:9115
module: dns_lan_syncthing
interval: 60s
scrapeTimeout: 10s
targets:
staticConfig:
static:
- "[fd00::1]:53"
- "10.2.40.1:53"
labels:
dns_name: syncthing.paperless.renner.uno
@@ -34,7 +34,9 @@ spec:
# query parameter, and an exact match would break silently if the
# instance label ever encodes it differently.
expr: probe_success{job="blackbox-http", instance=~"https://openbao\\.renner\\.uno/v1/sys/health.*"} == 0
for: 5m
# Two consecutive failed probes: enough to skip a single lost probe
# through Traefik, not enough to sit on a sealed store.
for: 1m
labels:
severity: critical
annotations:
@@ -0,0 +1,49 @@
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: dns-resolution
namespace: monitoring
labels:
# ruleSelector matches on this label; without it the rule is created but
# never loaded into Prometheus.
release: kube-prometheus-stack
spec:
groups:
- name: dns-resolution.rules
rules:
# min by dns_name: both resolver families are probed, and the record
# is only gone when neither answers. A single-family failure is the
# UDM's dnsmasq refusing connections in bursts (k8s-gateway carries
# the same note), which is why the window is 30m rather than the 10m
# used for HTTP probes.
- alert: LanDnsRecordMissing
expr: min by (dns_name) (probe_success{job="blackbox-dns"}) == 0
for: 30m
labels:
severity: critical
annotations:
summary: "{{ $labels.dns_name }} no longer resolves to its LAN address"
description: >-
The UDM stopped answering for a name external-dns publishes from
a LoadBalancer Service, or answered with the public
*.renner.uno wildcard instead of the record. Clients that use
the name rather than the address lose the service while
everything in the cluster stays healthy — nothing else in the
stack reports this. Check the external-dns alerts first, then
the record on the UDM.
# Dead-man's-switch, mirroring BlackboxProbesAbsent: without probe
# results the rule above cannot fire.
- alert: LanDnsProbesAbsent
expr: absent(probe_success{job="blackbox-dns"})
for: 15m
labels:
severity: critical
annotations:
summary: "No LAN DNS probe results are reaching Prometheus"
description: >-
The blackbox-dns Probes stopped being discovered or the exporter
rejects the modules. A 400 from /probe yields no probe_* series
at all, which looks the same as no Probe — check that every
dns_lan_* module the Probes reference exists in the
blackbox-exporter config.
@@ -0,0 +1,87 @@
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: external-dns
namespace: monitoring
labels:
# ruleSelector matches on this label; without it the rule is created but
# never loaded into Prometheus.
release: kube-prometheus-stack
spec:
groups:
- name: external-dns.rules
rules:
# The signal that was missing on 2026-09-12: the sources stopped
# producing endpoints they had produced before, and policy=sync
# deleted the records. 10% below the 24h peak clears a single
# retired Ingress (2 of ~48) and catches a whole class of names.
- alert: ExternalDNSSourceEndpointsDropped
expr: >-
external_dns_source_endpoints_total
< 0.9 * max_over_time(external_dns_source_endpoints_total[24h])
for: 30m
labels:
severity: critical
annotations:
summary: "{{ $labels.job }} is publishing {{ $value }} endpoints, below its 24h peak"
description: >-
external-dns stopped seeing endpoints it had before. With
policy=sync the matching DNS records are deleted, so names stop
resolving without anything else failing. Compare
external_dns_source_endpoints_total against the last 24h, then
check whether the resources still carry a hostname annotation
the running version understands — a changed annotation prefix
default empties the source silently. Deliberate removals also
fire this and resolve themselves 24h later.
# Source errors mean the Kubernetes side failed; registry errors mean
# the provider did. Either way desired state is computed from partial
# input, which policy=sync acts on.
- alert: ExternalDNSSourceErrors
expr: increase(external_dns_source_errors_total[15m]) > 0
for: 15m
labels:
severity: warning
annotations:
summary: "{{ $labels.job }} is failing to read its sources"
description: >-
external-dns could not list the Kubernetes resources it derives
records from. Check the pod log and its RBAC.
- alert: ExternalDNSRegistryErrors
expr: increase(external_dns_registry_errors_total[15m]) > 0
for: 15m
labels:
severity: warning
annotations:
summary: "{{ $labels.job }} is failing to reach its DNS provider"
description: >-
external-dns could not read or write records at the provider.
For external-dns-unifi that is the UniFi webhook sidecar and the
controller behind it; for external-dns it is the Cloudflare API.
- alert: ExternalDNSDown
expr: up{job=~"external-dns.*"} == 0
for: 15m
labels:
severity: critical
annotations:
summary: "{{ $labels.job }} is not being scraped"
description: >-
The external-dns instance is down or unreachable. Records already
at the provider keep resolving, but nothing reconciles them any
more and every other alert in this group is blind.
# Dead-man's-switch for the group, mirroring BlackboxProbesAbsent: a
# dropped ServiceMonitor makes every rule above evaluate to nothing.
- alert: ExternalDNSMetricsMissing
expr: absent(external_dns_source_endpoints_total)
for: 15m
labels:
severity: critical
annotations:
summary: "No external-dns metrics are reaching Prometheus"
description: >-
Either both external-dns instances are gone or the ServiceMonitor
stopped being discovered. Verify it still carries
release=kube-prometheus-stack.
@@ -28,24 +28,28 @@ spec:
# scheduling would also fire this — but both want a human too, so the
# imprecision costs nothing.
- alert: OpenBaoReplicaNotReady
# A replica that is sealed never recovers on its own, so the
# duration is only there to ignore ordinary rescheduling: a healthy
# pod moving between nodes is Ready again in well under a minute.
expr: kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"} < 3
# A replica unseals itself from the static seal, so a restart is
# Ready again in well under a minute; the duration only ignores that
# ordinary rescheduling. A replica still sealed after it never
# recovers on its own: the seal failed (openbao-seal-key Secret
# missing or wrong).
# max by: kube-state-metrics stamps instance/pod on the series, so a
# rescheduled exporter starts a new series and resets the timer.
expr: max by (namespace, statefulset) (kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"}) < 3
for: 10m
labels:
severity: warning
annotations:
summary: "OpenBao has {{ $value }} of 3 replicas ready"
description: >-
An OpenBao replica has been unavailable for 10 minutes. The usual
cause is a sealed replica after a node drain — every restart
seals it, and it stays out of the raft cluster until unsealed by
hand. The service is still up on the remaining replicas, so
nothing is broken yet, but redundancy is reduced and a second
drain would cost quorum. Check `kubectl -n openbao get pods` and
`bao status` in the not-ready pod; unseal with the key from
`pass` (homelab/openbao/unseal-key).
An OpenBao replica has been unavailable for 10 minutes. A restart
unseals itself from the static seal, so this is not the ordinary
drain case. The service is still up on the remaining replicas,
so nothing is broken yet, but redundancy is reduced and a second
failure would cost quorum. Check `kubectl -n openbao get pods`,
the pod log and `bao status` in the not-ready pod. If it is
sealed, the static seal failed: verify the openbao-seal-key
Secret matches `pass show homelab/openbao/seal-key`.
- alert: OpenBaoQuorumLost
# Raft quorum of a three-node cluster is two. At one ready replica
@@ -53,11 +57,10 @@ spec:
# the ClusterSecretStore goes InvalidProviderConfig and no
# ExternalSecret can refresh.
#
# Shorter window than the warning above because this is an outage,
# not a degradation — but not zero, so a rolling restart that
# briefly dips does not page.
expr: kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"} < 2
for: 5m
# No window: below quorum the secret store is down and every
# ExternalSecret stalls with it, so waiting only delays the page.
expr: max by (namespace, statefulset) (kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"}) < 2
for: 0m
labels:
severity: critical
annotations:
@@ -67,9 +70,11 @@ spec:
quorum and the secret store is unavailable. ExternalSecrets stop
refreshing; existing Secrets keep their last values, so failures
surface only when something rotates or a new secret is needed.
Unseal the sealed replicas (key in `pass`,
homelab/openbao/unseal-key) — each one individually, since a
sealed server cannot be a standby.
Replicas unseal themselves from the static seal; one that stays
sealed could not. Check its log and the openbao-seal-key Secret
against `pass show homelab/openbao/seal-key`, fix the Secret and
restart the pod. The recovery key cannot unseal an auto-sealed
server. Procedure: kubernetes/RUNBOOK-cold-rebuild.md, Step 4.
- alert: OpenBaoMetricsMissing
# Guards the two alerts above. Both are threshold comparisons, so if
@@ -0,0 +1,368 @@
{
"title": "Intel iGPU",
"uid": "intel-gpu",
"tags": ["gpu", "intel", "i915"],
"schemaVersion": 39,
"editable": true,
"timezone": "",
"refresh": "1m",
"time": { "from": "now-6h", "to": "now" },
"templating": {
"list": [
{
"name": "node",
"label": "Node",
"type": "query",
"datasource": { "type": "prometheus", "uid": "prometheus" },
"query": {
"query": "label_values(gpumon_power, node)",
"refId": "PrometheusVariableQueryEditor-VariableQuery"
},
"definition": "label_values(gpumon_power, node)",
"includeAll": true,
"multi": true,
"refresh": 2,
"sort": 1,
"current": { "selected": true, "text": ["All"], "value": ["$__all"] }
}
]
},
"panels": [
{
"type": "stat",
"title": "Render/3D busy now",
"gridPos": { "h": 4, "w": 8, "x": 0, "y": 0 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "percent",
"decimals": 0,
"min": 0,
"max": 100,
"color": { "mode": "fixed", "fixedColor": "text" }
},
"overrides": []
},
"options": {
"colorMode": "none",
"graphMode": "none",
"justifyMode": "center",
"orientation": "horizontal",
"textMode": "value_and_name",
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }
},
"targets": [
{
"expr": "gpumon_engine_usage{attrib=\"busy\", engine=\"Render/3D\", node=~\"$node\"}",
"legendFormat": "{{node}}",
"refId": "A"
}
]
},
{
"type": "stat",
"title": "GPU power now",
"gridPos": { "h": 4, "w": 8, "x": 8, "y": 0 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "watt",
"decimals": 1,
"color": { "mode": "fixed", "fixedColor": "text" }
},
"overrides": []
},
"options": {
"colorMode": "none",
"graphMode": "none",
"justifyMode": "center",
"orientation": "horizontal",
"textMode": "value_and_name",
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }
},
"targets": [
{
"expr": "gpumon_power{type=\"gpu\", node=~\"$node\"}",
"legendFormat": "{{node}}",
"refId": "A"
}
]
},
{
"type": "stat",
"title": "GT frequency now",
"gridPos": { "h": 4, "w": 8, "x": 16, "y": 0 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "suffix: MHz",
"decimals": 0,
"color": { "mode": "fixed", "fixedColor": "text" }
},
"overrides": []
},
"options": {
"colorMode": "none",
"graphMode": "none",
"justifyMode": "center",
"orientation": "horizontal",
"textMode": "value_and_name",
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }
},
"targets": [
{
"expr": "gpumon_frequency{type=\"actual\", node=~\"$node\"}",
"legendFormat": "{{node}}",
"refId": "A"
}
]
},
{
"type": "timeseries",
"title": "Render/3D busy",
"description": "Compute and 3D. llama.cpp's Vulkan backend runs here; 0 % during an inference means the model is on the CPU.",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 4 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "percent",
"min": 0,
"max": 100,
"custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
},
"overrides": [
{ "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] }
]
},
"options": {
"legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"expr": "gpumon_engine_usage{attrib=\"busy\", engine=\"Render/3D\", node=~\"$node\"}",
"legendFormat": "{{node}}",
"refId": "A"
}
]
},
{
"type": "timeseries",
"title": "Video busy",
"description": "Fixed-function decode/encode (QuickSync).",
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 4 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "percent",
"min": 0,
"max": 100,
"custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
},
"overrides": [
{ "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] }
]
},
"options": {
"legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"expr": "gpumon_engine_usage{attrib=\"busy\", engine=\"Video\", node=~\"$node\"}",
"legendFormat": "{{node}}",
"refId": "A"
}
]
},
{
"type": "timeseries",
"title": "VideoEnhance busy",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 12 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "percent",
"min": 0,
"max": 100,
"custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
},
"overrides": [
{ "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] }
]
},
"options": {
"legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"expr": "gpumon_engine_usage{attrib=\"busy\", engine=\"VideoEnhance\", node=~\"$node\"}",
"legendFormat": "{{node}}",
"refId": "A"
}
]
},
{
"type": "timeseries",
"title": "Blitter busy",
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 12 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "percent",
"min": 0,
"max": 100,
"custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
},
"overrides": [
{ "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] }
]
},
"options": {
"legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"expr": "gpumon_engine_usage{attrib=\"busy\", engine=\"Blitter\", node=~\"$node\"}",
"legendFormat": "{{node}}",
"refId": "A"
}
]
},
{
"type": "timeseries",
"title": "GT frequency (actual)",
"description": "0 while the GT sits in RC6.",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 20 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "suffix: MHz",
"min": 0,
"custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
},
"overrides": [
{ "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] }
]
},
"options": {
"legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"expr": "gpumon_frequency{type=\"actual\", node=~\"$node\"}",
"legendFormat": "{{node}}",
"refId": "A"
}
]
},
{
"type": "timeseries",
"title": "GPU power",
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 20 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "watt",
"min": 0,
"custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
},
"overrides": [
{ "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] }
]
},
"options": {
"legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"expr": "gpumon_power{type=\"gpu\", node=~\"$node\"}",
"legendFormat": "{{node}}",
"refId": "A"
}
]
},
{
"type": "timeseries",
"title": "Package power",
"description": "Whole SoC (CPU + GPU + uncore), from the same PMU.",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 28 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "watt",
"min": 0,
"custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
},
"overrides": [
{ "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] }
]
},
"options": {
"legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"expr": "gpumon_power{type=\"pkg\", node=~\"$node\"}",
"legendFormat": "{{node}}",
"refId": "A"
}
]
},
{
"type": "timeseries",
"title": "Memory controller bandwidth",
"description": "Reads solid, writes dashed. The iGPU shares DDR5 with the CPU, so this is the whole package's traffic.",
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 28 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "MiBs",
"min": 0,
"custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
},
"overrides": [
{ "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] },
{ "matcher": { "id": "byRegexp", "options": "/write/" }, "properties": [{ "id": "custom.lineStyle", "value": { "fill": "dash", "dash": [10, 10] } }] }
]
},
"options": {
"legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"expr": "gpumon_imc_read{node=~\"$node\"}",
"legendFormat": "{{node}} read",
"refId": "A"
},
{
"expr": "gpumon_imc_write{node=~\"$node\"}",
"legendFormat": "{{node}} write",
"refId": "B"
}
]
}
]
}
@@ -77,6 +77,9 @@ configMapGenerator:
- name: dashboard-infrastructure-overview
files:
- infrastructure-overview.json
- name: dashboard-intel-gpu
files:
- intel-gpu.json
- name: dashboard-netbird-overview
files:
- netbird-overview.json
@@ -232,7 +232,7 @@ metadata:
lb-pool: bgp
annotations:
io.cilium/lb-ipam-ips: "10.2.99.84,fd99::84"
external-dns.alpha.kubernetes.io/hostname: syslog.renner.uno
external-dns.kubernetes.io/hostname: syslog.renner.uno
spec:
type: LoadBalancer
externalTrafficPolicy: Local
@@ -72,3 +72,73 @@ spec:
tcp:
preferred_ip_protocol: ip6
ip_protocol_fallback: true
# One module per name: the dns prober takes the RESOLVER as its
# target and reads query_name from the module, so a name cannot be
# passed per scrape. The Probes in monitoring-rules/probes-dns.yaml
# point these at the UDM.
#
# Both answer checks are load-bearing. valid_rcodes alone passes a
# deleted record two labels deep (NOERROR, no answer), and the public
# *.renner.uno wildcard answers one label deep with a CNAME to
# proxy.renner.uno — NOERROR with the WAN proxy behind it. Requiring
# an A and rejecting a CNAME separates a real record from both.
dns_lan_ha:
prober: dns
timeout: 5s
dns:
query_name: ha.renner.uno
query_type: A
valid_rcodes: [NOERROR]
validate_answer_rrs:
fail_if_not_matches_regexp: ['\sIN\s+A\s']
fail_if_matches_regexp: ['\sIN\s+CNAME\s']
dns_lan_mqtt:
prober: dns
timeout: 5s
dns:
query_name: mqtt.renner.uno
query_type: A
valid_rcodes: [NOERROR]
validate_answer_rrs:
fail_if_not_matches_regexp: ['\sIN\s+A\s']
fail_if_matches_regexp: ['\sIN\s+CNAME\s']
dns_lan_ldap:
prober: dns
timeout: 5s
dns:
query_name: ldap.renner.uno
query_type: A
valid_rcodes: [NOERROR]
validate_answer_rrs:
fail_if_not_matches_regexp: ['\sIN\s+A\s']
fail_if_matches_regexp: ['\sIN\s+CNAME\s']
dns_lan_syslog:
prober: dns
timeout: 5s
dns:
query_name: syslog.renner.uno
query_type: A
valid_rcodes: [NOERROR]
validate_answer_rrs:
fail_if_not_matches_regexp: ['\sIN\s+A\s']
fail_if_matches_regexp: ['\sIN\s+CNAME\s']
dns_lan_ftp_scanner:
prober: dns
timeout: 5s
dns:
query_name: ftp.paperless.renner.uno
query_type: A
valid_rcodes: [NOERROR]
validate_answer_rrs:
fail_if_not_matches_regexp: ['\sIN\s+A\s']
fail_if_matches_regexp: ['\sIN\s+CNAME\s']
dns_lan_syncthing:
prober: dns
timeout: 5s
dns:
query_name: syncthing.paperless.renner.uno
query_type: A
valid_rcodes: [NOERROR]
validate_answer_rrs:
fail_if_not_matches_regexp: ['\sIN\s+A\s']
fail_if_matches_regexp: ['\sIN\s+CNAME\s']
@@ -91,12 +91,11 @@ spec:
# Additional scrape targets not covered by ServiceMonitors.
additionalScrapeConfigs:
# Gitea exposes /metrics anonymously on its main HTTP port. The
# `gitea_*` metrics power the gitea-overview dashboard. Gitea now runs
# in-cluster (StatefulSet in apps ns; LXC 114 retired), so scrape the
# ClusterIP Service directly instead of the dead LXC hostname.
# `gitea_*` metrics power the gitea-overview dashboard. Scrapes the
# ClusterIP Service of the StatefulSet in the gitea namespace.
- job_name: gitea
static_configs:
- targets: ["gitea.apps.svc.cluster.local:3000"]
- targets: ["gitea.gitea.svc.cluster.local:3000"]
metrics_path: /metrics
# Grafana
@@ -0,0 +1,81 @@
# Intel iGPU utilisation, frequency and power for every node (#752).
#
# Not privileged: /dev/dri comes from the device plugin's monitoring resource
# (every GPU on the node, no hostPath), and the i915 PMU that intel_gpu_top
# reads needs CAP_PERFMON — the one capability granted. No hostPID, no
# hostNetwork.
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: intel-gpu-exporter
namespace: monitoring
labels:
app.kubernetes.io/name: intel-gpu-exporter
spec:
selector:
matchLabels:
app.kubernetes.io/name: intel-gpu-exporter
template:
metadata:
labels:
app.kubernetes.io/name: intel-gpu-exporter
spec:
# Every node is a control plane; without this the exporter covers nothing.
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule
securityContext:
seccompProfile:
type: RuntimeDefault
containers:
- name: intel-gpu-exporter
image: ghcr.io/clambin/intel-gpu-exporter:0.7.4@sha256:d78761fef018834c759db154fe9dcf3dd9db5974703fa50ca21f6abc35221be1
args:
# No -device: intel_gpu_top picks the only GPU the plugin injects.
- -interval=5s
ports:
- name: http-metrics
containerPort: 9100
protocol: TCP
securityContext:
# Root so the PERFMON capability is effective without file
# capabilities on the binary.
runAsUser: 0
runAsNonRoot: false
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
add: ["PERFMON"]
resources:
requests:
cpu: 10m
memory: 32Mi
gpu.intel.com/monitoring: 1
limits:
memory: 128Mi
gpu.intel.com/monitoring: 1
livenessProbe:
httpGet:
path: /metrics
port: http-metrics
initialDelaySeconds: 15
periodSeconds: 30
---
apiVersion: v1
kind: Service
metadata:
name: intel-gpu-exporter
namespace: monitoring
labels:
app.kubernetes.io/name: intel-gpu-exporter
spec:
clusterIP: None
selector:
app.kubernetes.io/name: intel-gpu-exporter
ports:
- name: http-metrics
port: 9100
targetPort: http-metrics
protocol: TCP
@@ -16,6 +16,7 @@ resources:
- prometheus-gate-ingress.yaml
- ci-diagnostics-rbac.yaml
- smartctl-exporter.yaml
- intel-gpu-exporter.yaml
- dashboards/
# prometheusrule-garage.yaml lives in ./monitoring-rules/ and is applied by a
# separate Flux Kustomization that dependsOn `monitoring` — otherwise the
@@ -28,7 +28,7 @@ spec:
# net1 instead. master: ens19 resolves on every k8s-{1,2,3}/edge
# worker, though it is no longer the same kind of device everywhere:
# on the bare-metal control-plane nodes it is a VLAN 30 sub-interface
# of enp2s0f0np0 rather than a NIC, which is why the datapath needs
# of uplink rather than a NIC, which is why the datapath needs
# bpf.vlanBypass to carry it (issue #592).
k8s.v1.cni.cncf.io/networks: music-assistant-iot
spec:
@@ -55,7 +55,7 @@ spec:
# stdlib only; Flux keeps the tag in lockstep via the shared image
# policy) — no extra dependency.
- name: reconcile-publish-ip
image: ghcr.io/music-assistant/server:2.10.2 # {"$imagepolicy": "flux-system:music-assistant"}
image: ghcr.io/music-assistant/server:2.10.4 # {"$imagepolicy": "flux-system:music-assistant"}
command: ["/app/venv/bin/python", "/scripts/reconcile.py"]
volumeMounts:
- name: data
@@ -65,7 +65,7 @@ spec:
readOnly: true
containers:
- name: music-assistant
image: ghcr.io/music-assistant/server:2.10.2 # {"$imagepolicy": "flux-system:music-assistant"}
image: ghcr.io/music-assistant/server:2.10.4 # {"$imagepolicy": "flux-system:music-assistant"}
env:
- name: LOG_LEVEL
value: info
+10 -15
View File
@@ -43,7 +43,7 @@ spec:
mountPath: /consume
containers:
- name: vsftpd
image: gitea.renner.uno/k.renner/vsftpd:build.1784731469 # {"$imagepolicy": "flux-system:vsftpd"}
image: gitea.renner.uno/k.renner/vsftpd:build.1789630783 # {"$imagepolicy": "flux-system:vsftpd"}
ports:
- name: ftp-ctrl
containerPort: 21
@@ -70,24 +70,19 @@ spec:
- name: consume
mountPath: /home/scanner
subPath: scanner
# FTP has no health verb, so the control port accepting a connection
# is what can be asserted. It is enough for the failure that matters:
# if vsftpd dies the scanner cannot deposit anything and nothing else
# in the chain notices, because paperless only ever sees an empty
# consume directory — which is indistinguishable from nobody having
# scanned.
#
# Only the control port is probed. The passive range is negotiated
# per transfer and is not listening at rest.
# Asserts the control socket is listening without opening an FTP
# connection: vsftpd 3.0.5 segfaults its listener on a fraction of
# the connections it accepts. The passive range is negotiated per
# transfer and is not listening at rest, so it is not probed.
livenessProbe:
tcpSocket:
port: ftp-ctrl
exec:
command: ["sh", "-c", "netstat -lnt | grep -q ':21 '"]
initialDelaySeconds: 15
periodSeconds: 30
timeoutSeconds: 5
readinessProbe:
tcpSocket:
port: ftp-ctrl
exec:
command: ["sh", "-c", "netstat -lnt | grep -q ':21 '"]
initialDelaySeconds: 5
periodSeconds: 10
timeoutSeconds: 5
@@ -124,7 +119,7 @@ metadata:
# label — FTP must not be exposed via the cargo tunnel (clear-text
# protocol, public exposure not appropriate). The brother scanner
# uses this hostname instead of the LB IP.
external-dns.alpha.kubernetes.io/hostname: ftp.paperless.renner.uno
external-dns.kubernetes.io/hostname: ftp.paperless.renner.uno
spec:
type: LoadBalancer
externalTrafficPolicy: Local
+1 -1
View File
@@ -22,7 +22,7 @@ spec:
type: RuntimeDefault
containers:
- name: gotenberg
image: gotenberg/gotenberg:8.36.0 # {"$imagepolicy": "flux-system:gotenberg"}
image: gotenberg/gotenberg:8.37.0 # {"$imagepolicy": "flux-system:gotenberg"}
args:
- "gotenberg"
- "--chromium-disable-javascript=true"
+6 -9
View File
@@ -25,18 +25,15 @@ resources:
configMapGenerator:
# vsftpd config for the ftp-scanner deployment. The content-hash name
# suffix is intentionally LEFT ENABLED (unlike home-assistant's nginx
# config): vsftpd templates pasv_address from PASV_ADDRESS at container
# start, so a changed value must roll the pod. With the hash suffix,
# editing any value here renames the generated ConfigMap, which mutates
# the Deployment's pod template and triggers an automatic rolling
# restart. Without it, an IP change (e.g. an LB re-allocation) silently
# leaves the running pod advertising the stale PASV address, which
# breaks classic-PASV clients (older Brother scanners) while EPSV
# clients keep working.
# config): vsftpd reads these into vsftpd.conf at container start, so a
# changed value must roll the pod, which the renamed ConfigMap does.
#
# No PASV_ADDRESS: the LB address is declared once, on the Service
# (io.cilium/lb-ipam-ips). The image default 0.0.0.0 tells PASV clients
# to reuse the control-connection address.
- name: ftp-scanner-config
namespace: paperless
literals:
- PASV_ADDRESS=10.2.99.82
- PASV_MIN_PORT=21100
- PASV_MAX_PORT=21104
- FTP_USER=scanner
+4 -2
View File
@@ -27,6 +27,8 @@ data:
PAPERLESS_ALLOWED_HOSTS: "*"
PAPERLESS_TIME_ZONE: "Europe/Berlin"
PAPERLESS_OCR_LANGUAGE: "deu+eng"
# Default is cpu_count / TASK_WORKERS, 20 on k8s-1; OCR memory scales with it.
PAPERLESS_THREADS_PER_WORKER: "5"
PAPERLESS_CONSUMER_RECURSIVE: "true"
PAPERLESS_CONSUMER_DELETE_DUPLICATES: "true"
PAPERLESS_CONSUMER_ENABLE_BARCODES: "true"
@@ -183,7 +185,7 @@ spec:
drop: ["ALL"]
containers:
- name: paperless
image: ghcr.io/paperless-ngx/paperless-ngx:3.1.3 # {"$imagepolicy": "flux-system:paperless-ngx"}
image: ghcr.io/paperless-ngx/paperless-ngx:3.2.1 # {"$imagepolicy": "flux-system:paperless-ngx"}
ports:
- name: http
containerPort: 8000
@@ -273,7 +275,7 @@ spec:
cpu: 200m
memory: 512Mi
limits:
memory: 2Gi
memory: 4Gi
securityContext:
runAsNonRoot: false
allowPrivilegeEscalation: true
+1 -1
View File
@@ -25,7 +25,7 @@ spec:
type: RuntimeDefault
containers:
- name: redis
image: redis:8.10.1-alpine # {"$imagepolicy": "flux-system:redis-paperless"}
image: redis:8.10.2-alpine # {"$imagepolicy": "flux-system:redis-paperless"}
args:
- redis-server
- --save
+1 -1
View File
@@ -145,7 +145,7 @@ metadata:
lb-pool: bgp
annotations:
io.cilium/lb-ipam-ips: "10.2.99.85,fd99::85"
external-dns.alpha.kubernetes.io/hostname: syncthing.paperless.renner.uno
external-dns.kubernetes.io/hostname: syncthing.paperless.renner.uno
spec:
type: LoadBalancer
externalTrafficPolicy: Local
+2 -2
View File
@@ -134,7 +134,7 @@ spec:
# is a device-bricking class of risk. Bumps are a reviewed, manual
# act — see kubernetes/infrastructure/image-automation/README.md.
- name: terminus
image: ghcr.io/usetrmnl/terminus:0.72.0 # {"$imagepolicy": "flux-system:terminus"}
image: ghcr.io/usetrmnl/terminus:0.74.0 # {"$imagepolicy": "flux-system:terminus"}
ports:
- name: http
containerPort: 2345
@@ -197,7 +197,7 @@ spec:
capabilities:
drop: ["ALL"]
- name: sidekiq
image: ghcr.io/usetrmnl/terminus:0.72.0 # {"$imagepolicy": "flux-system:terminus"}
image: ghcr.io/usetrmnl/terminus:0.74.0 # {"$imagepolicy": "flux-system:terminus"}
command: ["bundle", "exec", "sidekiq", "-r", "./config/sidekiq.rb"]
envFrom:
- configMapRef:
+1 -1
View File
@@ -18,7 +18,7 @@ spec:
spec:
containers:
- name: whisper
image: rhasspy/wyoming-whisper:3.7.0 # {"$imagepolicy": "flux-system:wyoming-whisper"}
image: rhasspy/wyoming-whisper:3.8.1 # {"$imagepolicy": "flux-system:wyoming-whisper"}
args:
- --model=base-int8
- --language=de
+137
View File
@@ -0,0 +1,137 @@
---
# ServiceAccount is the OpenBao identity (auth/kubernetes/role/wiki-index); no
# Kubernetes API access needed.
apiVersion: v1
kind: ServiceAccount
metadata:
name: wiki-index
namespace: ai
labels:
app.kubernetes.io/name: wiki-index
---
# Index (~17 MB JSON for 1700 notes). Survives restarts so a pod start does not
# re-embed the wiki; a lost volume only costs one full sync (~5 min).
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: wiki-index-data
namespace: ai
spec:
accessModes:
- ReadWriteOnce
storageClassName: ceph-block
resources:
requests:
storage: 1Gi
---
# Embedding index over the Notes repo's wiki/ (ci/images/wiki-index). Syncs from
# Gitea on push webhook and every RESYNC_SECONDS, embeds via wiki-embed, serves
# POST /rank to ~/Notes/bin/wiki_rank.py through wiki-index.renner.uno.
apiVersion: apps/v1
kind: Deployment
metadata:
name: wiki-index
namespace: ai
labels:
app.kubernetes.io/name: wiki-index
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app.kubernetes.io/name: wiki-index
template:
metadata:
labels:
app.kubernetes.io/name: wiki-index
spec:
serviceAccountName: wiki-index
securityContext:
runAsNonRoot: true
runAsUser: 65534
runAsGroup: 65534
fsGroup: 65534
seccompProfile:
type: RuntimeDefault
containers:
- name: wiki-index
# Per-commit tag (build-ci-images.yml): immutable, no same-day overwrite.
image: gitea.renner.uno/gitea_admin/wiki-index:2026-09-20-4b529cd1
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
env:
- name: GITEA_URL
value: https://gitea.renner.uno
- name: NOTES_REPO
value: k.renner/notes
- name: NOTES_BRANCH
value: main
- name: EMBED_URL
value: http://wiki-embed.ai.svc.cluster.local:8080
- name: BAO_ADDR
value: http://openbao.openbao.svc.cluster.local:8200
- name: BAO_ROLE
value: wiki-index
- name: BAO_SECRET_PATH
value: homelab/data/apps/ai/wiki-index
- name: INDEX_PATH
value: /data/index.json
- name: RESYNC_SECONDS
value: "1800"
ports:
- name: http
containerPort: 8080
# Pure-Python cosine over 1700 x 1024 floats per query; the index itself is ~60 MB in memory.
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: "1"
memory: 512Mi
# /health: the process serves (liveness). /ready: an index exists; a cold
# start on an empty PVC embeds every note (~5 min) before traffic arrives.
startupProbe:
httpGet:
path: /health
port: http
periodSeconds: 5
failureThreshold: 12
readinessProbe:
httpGet:
path: /ready
port: http
periodSeconds: 10
livenessProbe:
httpGet:
path: /health
port: http
periodSeconds: 60
timeoutSeconds: 10
failureThreshold: 3
volumeMounts:
- name: data
mountPath: /data
volumes:
- name: data
persistentVolumeClaim:
claimName: wiki-index-data
---
apiVersion: v1
kind: Service
metadata:
name: wiki-index
namespace: ai
labels:
app.kubernetes.io/name: wiki-index
spec:
selector:
app.kubernetes.io/name: wiki-index
ports:
- name: http
port: 8080
targetPort: http
+56
View File
@@ -0,0 +1,56 @@
---
# wiki-index.renner.uno — LAN-only (no dns.renner.uno/public label), machine
# client (bin/wiki_rank.py), so Traefik BasicAuth like wiki-embed.renner.uno.
# Gitea's webhook does not pass here; it targets the ClusterIP Service directly.
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: wiki-index-basic-auth
namespace: ai
spec:
basicAuth:
secret: wiki-index-basic-auth
realm: wiki-index
---
# OpenBao key apps/ai/wiki-index-basic-auth, field `users` = htpasswd line.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: wiki-index-basic-auth
namespace: ai
spec:
refreshInterval: 1h
secretStoreRef:
kind: ClusterSecretStore
name: openbao
target:
name: wiki-index-basic-auth
dataFrom:
- extract:
key: apps/ai/wiki-index-basic-auth
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: wiki-index
namespace: ai
annotations:
cert-manager.io/cluster-issuer: letsencrypt-cluster
traefik.ingress.kubernetes.io/router.middlewares: ai-wiki-index-basic-auth@kubernetescrd
spec:
ingressClassName: traefik
tls:
- hosts:
- wiki-index.renner.uno
secretName: wiki-index-tls
rules:
- host: wiki-index.renner.uno
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: wiki-index
port:
number: 8080
@@ -0,0 +1,5 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
- ingress.yaml
+1 -1
View File
@@ -34,7 +34,7 @@ spec:
# Image automation, range 3.x (see image-automation/woodpecker.yaml).
# MUST stay version-matched with woodpecker-server (gRPC parity) — both
# track the same range/filter so they bump together on each release.
image: woodpeckerci/woodpecker-agent:v3.18.0 # {"$imagepolicy": "flux-system:woodpecker-agent"}
image: woodpeckerci/woodpecker-agent:v3.18.1 # {"$imagepolicy": "flux-system:woodpecker-agent"}
env:
- name: WOODPECKER_SERVER
value: woodpecker.woodpecker.svc.cluster.local:9000 # in-cluster gRPC, replaces LXC :9000
+1 -1
View File
@@ -40,7 +40,7 @@ spec:
# minor + patch auto-update within 3.x; only the 4.0 major is held for
# a reviewed bump. Safe now the DB is postgres (the old sqlite-surprise
# -bump hazard is gone). Stays in lockstep with the agent.
image: woodpeckerci/woodpecker-server:v3.18.0 # {"$imagepolicy": "flux-system:woodpecker-server"}
image: woodpeckerci/woodpecker-server:v3.18.1 # {"$imagepolicy": "flux-system:woodpecker-server"}
env:
- name: WOODPECKER_HOST
value: https://ci.renner.uno
@@ -21,7 +21,7 @@ spec:
# Service; nothing on this path needs the host's network namespace.
containers:
- name: zone-configurator
image: everythingsmarthome/everything-presence-mmwave-configurator:2.2.4 # {"$imagepolicy": "flux-system:mmwave-configurator"}
image: everythingsmarthome/everything-presence-mmwave-configurator:2.2.5 # {"$imagepolicy": "flux-system:mmwave-configurator"}
env:
- name: HA_BASE_URL
value: "http://home-assistant:80"
+123 -103
View File
@@ -179,16 +179,18 @@ phase_runs() {
OPENBAO_NS="openbao"
# Where the secrets come from. On a workstation, the operator's pass store. In
# the in-cluster Job there is no pass and no GPG key — and there should not be,
# because the key that protects OpenBao has no business living in the cluster
# OpenBao secures. Set OPENBAO_SECRETS_DIR and the run reads them from files
# instead, which the operator streams in when the run asks for them:
# the in-cluster Job there is no pass and no GPG key — and there should not be:
# the pass store opens everything, and it stays on the operator's machine. Set
# OPENBAO_SECRETS_DIR and the run reads them from files instead, which the
# operator streams in when the run asks for them:
#
# pass show homelab/openbao/unseal-key \
# | kubectl -n vmtest exec -i <pod> -- sh -c 'cat > /secrets/unseal-key'
# pass show homelab/openbao/seal-key \
# | kubectl -n vmtest exec -i <pod> -- sh -c 'cat > /secrets/seal-key'
#
# They exist in the pod's filesystem for the length of the run and are never
# written to a Secret, so they do not land in etcd or in any backup of it.
# written to a Secret in this cluster, so they do not land in its etcd or in
# any backup of it. The seal key alone becomes a Secret in the cluster being
# built, which is what lets that OpenBao unseal itself.
OPENBAO_SECRETS_DIR="${OPENBAO_SECRETS_DIR:-}"
# Every secret the run needs, named once, in one place. The list is here and
@@ -243,7 +245,9 @@ OPENBAO_REQUIRED_SECRETS=(
"git-token|token for flux bootstrap, for whichever forge FLUX_SOURCE names|git_token_for_source"
"s3-access|Hetzner restore access key|pass show homelab/hetztner/k8s-restore-access"
"s3-secret|Hetzner restore secret key|pass show homelab/hetztner/k8s-restore-secret"
"unseal-key|OpenBao unseal key|pass show homelab/openbao/unseal-key"
# The static seal key. The snapshots are encrypted under it, so it has to
# be the one the backed-up cluster ran with — a fresh key restores nothing.
"seal-key|OpenBao seal key|pass show homelab/openbao/seal-key"
# Read by the node registry auth (see registry_auth_patch): the images an app
# needs while the cluster is still being built are mirrored to a private
# ghcr.io namespace, and a node has to be able to pull them before anything
@@ -277,7 +281,7 @@ OPENBAO_REQUIRED_SECRETS=(
# That phase used to need nothing but the machines, and `bootstrap.sh talos` is
# documented as the part that can run unattended. Node registry auth changed
# that: the machine config now carries a pull credential, so the phase needs
# exactly one secret. Demanding all six there would drag in the OpenBao unseal
# exactly one secret. Demanding all six there would drag in the OpenBao seal
# key, which deliberately never reaches CI — and demanding none, as before,
# left the phase to discover the missing file halfway through `talosctl gen`.
require_secrets() {
@@ -408,7 +412,7 @@ read_secret() {
# the kernel link name enp2s0f1np1, and that name cannot be manufactured here
# without assuming a Hetzner private network passes VLAN tags. See
# talos/patches/site-hcloud.yaml. Nor Flux and everything after it: that phase
# needs the OpenBao unseal key, which deliberately never reaches CI, and this
# needs the OpenBao seal key, which deliberately never reaches CI, and this
# test is meant to run unattended.
# Talos version and Image Factory schematic, read out of the installer image in
# cluster.yaml. Both throwaway-cluster phases need them and neither may state
@@ -606,8 +610,8 @@ if [[ "$PHASE" == "hcloudtest" ]]; then
# Constraints on the choice: cpx11/cpx21 are US-only and cax* are ARM, which
# would not match the amd64 schematic this cluster installs. That leaves the
# cx*/cpx*2 families in the German locations.
# Today's traffic-class split, not the target one in
# PROPOSAL-traffic-class-separation.md: the node address carries management,
# Two segments only, unlike the homelab's split in
# NETWORK-traffic-classes.md: here the node address carries management,
# cluster-internal and external alike, and Ceph replication has its own.
#
# The order is load-bearing: it decides which kernel interface each segment
@@ -1121,7 +1125,7 @@ EOF
# nothing has demanded the secrets yet — the full require_secrets is gated
# on the flux phase. Ask for the one this needs, and only that one: pulling
# in the whole list would make an unattended talos run depend on the OpenBao
# unseal key, which never reaches CI.
# seal key, which never reaches CI.
require_secrets ghcr-pull
REGISTRY_PATCH=$(registry_auth_patch)
[[ -s "$REGISTRY_PATCH" ]] \
@@ -1406,9 +1410,11 @@ fi
# waits on infrastructure-crds. A rebuild could not reach a reconciled state
# without someone at a keyboard.
#
# The unseal key stays where it was — the operator's `pass` store — and is read
# The seal key stays where it was — the operator's `pass` store — and is read
# here at run time. It is never written to the repo and never reaches CI, which
# is the reason this was left manual in the first place.
# is the reason this was left manual in the first place. It reaches the cluster
# as the openbao-seal-key Secret, created by the flux phase before Flux starts
# OpenBao, and from then on every replica unseals itself.
#
# Everything below is idempotent: it is safe to re-run against an OpenBao that
# is already unsealed and configured, which is what makes it usable as part of
@@ -1498,61 +1504,63 @@ restore_openbao() {
echo " extracted $(stat -c%s "$work/restore.snap") bytes"
# A snapshot can only be loaded into a running OpenBao, so the fresh one is
# initialised and unsealed with a throwaway key first. That key is used for
# the next few seconds and then stops existing: restoring replaces the
# keyring with the snapshot's, after which the real key from pass applies.
# -force is required for exactly that reason — without it OpenBao refuses a
# snapshot whose keys do not match the cluster's own.
echo -n " initialising the empty server with a throwaway key... "
local init_json throwaway
init_json=$(bao_exec "$pod" bao operator init -key-shares=1 -key-threshold=1 -format=json 2>&1) \
# initialised first. Under the static seal, init hands out a recovery key
# rather than an unseal key and the server unseals itself; the only thing
# used from its output is the root token, for the next few seconds, and
# then it stops existing: restoring replaces the keyring with the
# snapshot's, after which the root token and recovery key from pass apply.
#
# No -force. Without it OpenBao checks that the snapshot was written under
# the seal key this server runs with, and refuses a mismatch by name
# ("possibly the snapshot is using a different autoseal key"). That check
# is the one guard against restoring under the wrong seal-key Secret;
# -force would skip it and leave a server that can never unseal.
#
# The init output is not echoed on a parse failure: it carries the root
# token and the recovery key of a server that is about to be overwritten,
# but a log line is a log line.
echo -n " initialising the empty server (throwaway recovery key)... "
local init_json root
init_json=$(bao_exec "$pod" bao operator init -recovery-shares=1 -recovery-threshold=1 -format=json 2>&1) \
|| { echo; echo "$init_json" >&2; fail "could not initialise OpenBao"; }
throwaway=$(jq -r '.unseal_keys_b64[0]' <<<"$init_json")
[[ -n "$throwaway" && "$throwaway" != "null" ]] || fail "could not read the throwaway unseal key from init output"
root=$(jq -r '.root_token // empty' <<<"$init_json")
[[ -n "$root" ]] || fail "init output carried no root token"
ok
echo -n " unsealing with it... "
# The key goes as an argument, which OpenBao's own error message asks for:
#
# you are executing outside of a terminal (tty) ... If this is not an
# option, the unseal key can be provided as the first argument
#
# Two earlier shapes failed. `unseal -` is Vault syntax; OpenBao takes
# argv[1] literally, so the dash was sent as the key. Omitting the argument
# makes it prompt, and the prompt needs a real tty — a pipe gives "file
# descriptor 0 is not a terminal". There is no stdin path at all.
#
# This does put the key in the pod's process list for the length of the
# call, which the code here previously went out of its way to avoid. It is
# accepted rather than worked around: reading that list requires exec into
# the very pod that holds the key in memory, and the alternative — a
# port-forward plus curl against /v1/sys/unseal — puts a background process
# in the most critical path of a recovery. The openbao image has no curl,
# wget or nc, so it would have to run from here.
bao_exec "$pod" bao operator unseal "$throwaway" >/dev/null \
|| fail "could not unseal with the throwaway key"
ok
# Sealed after init means the static seal is not working — the Secret is
# missing or malformed — and the restore below cannot run against it.
echo -n " waiting for it to unseal itself"
for i in $(seq 1 30); do
if bao_exec "$pod" bao status -format=json 2>"$OPENBAO_SECRETS_DIR/.baoerr" \
| jq -e '.sealed == false' >/dev/null; then
echo; break
fi
printf '.'
sleep 2
if [[ $i -eq 30 ]]; then
echo
cat "$OPENBAO_SECRETS_DIR/.baoerr" >&2
fail "$pod did not unseal itself after init — check the openbao-seal-key Secret and the pod log"
fi
done
echo -n " loading the snapshot... "
kubectl -n "$OPENBAO_NS" cp "$work/restore.snap" "$pod:/tmp/restore.snap" >/dev/null \
|| fail "could not copy the snapshot into $pod"
local root
root=$(jq -r '.root_token' <<<"$init_json")
local out
out=$(bao_exec "$pod" env BAO_TOKEN="$root" bao operator raft snapshot restore -force /tmp/restore.snap 2>&1) \
out=$(bao_exec "$pod" env BAO_TOKEN="$root" bao operator raft snapshot restore /tmp/restore.snap 2>&1) \
|| { echo; echo "$out" >&2; fail "snapshot restore rejected"; }
ok
rm -rf "$work"
# The server is now sealed again, against the keyring that came with the
# snapshot. unseal_openbao carries on from here with the real key.
echo " restored — the server now expects the original unseal key"
# The server seals itself against the keyring that came with the snapshot
# and unseals again from the static seal — the same key the snapshot was
# written under. bring_up_openbao waits for that on every replica.
echo " restored — the server re-seals and unseals itself from the seal key"
}
unseal_openbao() {
step 0 "Unsealing OpenBao..."
[[ -n "$OPENBAO_SECRETS_DIR" ]] || command -v pass >/dev/null \
|| fail "pass not found — the unseal key lives in the operator's pass store, or set OPENBAO_SECRETS_DIR and stream it in"
bring_up_openbao() {
step 0 "Bringing OpenBao up..."
command -v jq >/dev/null || fail "jq not found (needed to read OpenBao's seal status)"
# OpenBao is deployed by the infrastructure Kustomization. What this step
@@ -1658,40 +1666,28 @@ unseal_openbao() {
restore_openbao "${pods[0]}"
fi
local key
key=$(read_secret unseal-key) \
|| fail "could not read the unseal key"
# Unseal every replica the StatefulSet wants, not the ones that happen to
# exist right now. On a cold start only openbao-0 exists when this runs, and
# unsealing just that one deadlocks the StatefulSet: podManagementPolicy is
# OrderedReady, a sealed pod never passes its readiness probe, so openbao-1
# is created only after openbao-0 is unsealed and openbao-2 is never created
# at all. The cluster comes up at 1 of 3.
# Wait for every replica the StatefulSet wants, not the ones that happen to
# exist right now. On a cold start only openbao-0 exists when this runs:
# podManagementPolicy is OrderedReady, so openbao-1 is created once
# openbao-0 passes its readiness probe and openbao-2 after that. Each
# unseals itself from the static seal; nothing here types a key.
#
# That alone would be survivable. What makes it fatal is that the openbao
# Service sets publishNotReadyAddresses — the chart does this so replicas can
# reach each other to join raft before any of them is unsealed — so the
# sealed openbao-1 is a live backend on the service IP. Clients are balanced
# onto a node that cannot answer, and the failure surfaces nowhere near here:
# external-secrets' ClusterSecretStore flaps between "store validated" and
# `dial tcp <svc>:8200: connect: no route to host`, every ExternalSecret in
# the cluster reports "could not get secret data from provider", and
# infrastructure-crds — which waits on those — never goes Ready, taking all
# twenty app Kustomizations with it.
#
# So this converges instead of iterating once: re-list the pods each round,
# unseal whatever is sealed, and finish when spec.replicas of them are
# unsealed. A pod that is still starting cannot be unsealed yet and is not an
# error — it is retried next round, which is exactly how the OrderedReady
# chain is meant to be walked.
# Waiting for the whole chain rather than the first pod matters because the
# openbao Service sets publishNotReadyAddresses — the chart does this so
# replicas can reach each other to join raft before any of them is Ready —
# so a replica that is still starting is a live backend on the service IP.
# Clients balanced onto it get `dial tcp <svc>:8200: connect: no route to
# host`, and the failure surfaces nowhere near here: external-secrets'
# ClusterSecretStore flaps, every ExternalSecret reports "could not get
# secret data from provider", and infrastructure-crds — which waits on those
# — never goes Ready, taking all twenty app Kustomizations with it.
local want
want=$(kubectl -n "$OPENBAO_NS" get statefulset openbao -o jsonpath='{.spec.replicas}') \
|| fail "could not read the openbao StatefulSet's replica count"
[[ "$want" =~ ^[0-9]+$ ]] || fail "openbao StatefulSet reported a non-numeric replica count: '$want'"
echo " unsealing $want replica(s)"
echo " waiting for $want replica(s) to unseal themselves"
local unsealed=0 round pod err
local unsealed=0 round pod
local -A seen=()
for round in $(seq 1 180); do
mapfile -t pods < <(kubectl -n "$OPENBAO_NS" get pods -l app.kubernetes.io/name=openbao \
@@ -1700,29 +1696,21 @@ unseal_openbao() {
for pod in "${pods[@]}"; do
if bao_exec "$pod" bao status -format=json 2>"$OPENBAO_SECRETS_DIR/.baoerr" \
| jq -e '.sealed == false' >/dev/null; then
[[ -n "${seen[$pod]:-}" ]] || { echo " $pod: already unsealed"; seen[$pod]=1; }
[[ -n "${seen[$pod]:-}" ]] || { echo -e " $pod: ${GREEN}unsealed${NC}"; seen[$pod]=1; }
unsealed=$(( unsealed + 1 ))
continue
fi
# Argument, not stdin — see the restore path above for why: OpenBao
# has no stdin form, and its prompt requires a tty a Job does not
# have. The key is exposed in the pod's process list for the length
# of the call.
if err=$(bao_exec "$pod" bao operator unseal "$key" 2>&1); then
echo -e " $pod: ${GREEN}unsealed${NC}"
seen[$pod]=1
unsealed=$(( unsealed + 1 ))
else
# Not fatal on its own: a container still in Init or still
# pulling answers this way, and the next round finds it ready.
# Only running out of rounds is a failure.
echo " $pod: not yet (${err##*$'\n'})"
fi
done
(( unsealed >= want )) && break
sleep 10
if [[ $round -eq 180 ]]; then
fail "only $unsealed of $want OpenBao replicas came unsealed after 30 minutes"
# A replica that is running and still sealed did not get a working
# seal. Say which, with its own answer, rather than "timed out".
for pod in "${pods[@]}"; do
[[ -n "${seen[$pod]:-}" ]] && continue
echo " $pod:" >&2
bao_exec "$pod" bao status 2>&1 | sed 's/^/ /' >&2 || true
done
fail "only $unsealed of $want OpenBao replicas unsealed after 30 minutes — a sealed one means the static seal failed (openbao-seal-key Secret, pod log)"
fi
done
ok
@@ -1747,7 +1735,7 @@ if phase_runs flux; then
command -v flux >/dev/null || fail "flux not found"
# In the in-cluster Job neither the SOPS key nor the Gitea token is present,
# and neither should be baked in. Both are streamed into the secrets
# directory when the run asks for them, the same way the unseal key is.
# directory when the run asks for them, the same way the seal key is.
# Unconditional: the same collection step on a laptop and in the Job, so the
# recovery path is the one that gets exercised.
require_secrets
@@ -1770,6 +1758,37 @@ if phase_runs flux; then
--dry-run=client -o yaml | kubectl apply -f -
ok
# The static seal key, before Flux brings OpenBao up: a pod whose env
# references a Secret that does not exist is never started, and OpenBao
# reads the key at start. Left alone when the cluster already holds the
# same value. A DIFFERENT value is refused rather than applied: the data
# is encrypted under whichever key OpenBao started with, and swapping the
# Secret would leave every replica sealed for good at its next restart.
# That is a migration, not a bootstrap step.
#
# --from-literal, not --from-file: `pass show` ends its output with a
# newline, the substitution strips it, and OpenBao rejects a key that is
# not exactly 32 bytes of base64. The key sits in this process's argument
# list for the length of the call, as it did in the pod's before.
echo -n "Creating OpenBao seal key secret... "
seal_key=$(read_secret seal-key) || fail "could not read the seal key"
kubectl create namespace "$OPENBAO_NS" --dry-run=client -o yaml | kubectl apply -f -
if existing=$(kubectl -n "$OPENBAO_NS" get secret openbao-seal-key \
-o go-template='{{index .data "key" | base64decode}}' 2>"$OPENBAO_SECRETS_DIR/.err"); then
[[ "$existing" == "$seal_key" ]] \
|| fail "openbao-seal-key in $OPENBAO_NS holds a different key than pass — not replacing it (OpenBao's data is written under the key it started with)"
echo "already present"
elif grep -q NotFound "$OPENBAO_SECRETS_DIR/.err"; then
kubectl -n "$OPENBAO_NS" create secret generic openbao-seal-key \
--from-literal=key="$seal_key" >/dev/null \
|| fail "could not create the openbao-seal-key Secret"
ok
else
echo; cat "$OPENBAO_SECRETS_DIR/.err" >&2
fail "could not read the openbao-seal-key Secret"
fi
unset seal_key existing
if [[ "$FLUX_SOURCE" == "github" ]]; then
step 0 "Bootstrapping FluxCD from the GitHub break-glass mirror..."
echo -e "${YELLOW}BREAK-GLASS: Flux source is github.com/${GITHUB_OWNER}/${GITHUB_REPO}, NOT in-cluster Gitea.${NC}"
@@ -1820,7 +1839,7 @@ if phase_runs flux; then
--token-auth --timeout 15m 2>&1
fi
unseal_openbao
bring_up_openbao
step 0 "Waiting for infrastructure..."
for i in $(seq 1 60); do
@@ -1927,6 +1946,7 @@ if phase_runs wait-apps; then
# apps waits on infrastructure-ceph-cluster (Ceph mon/mgr/OSD bring-up
# is genuinely 5–10 min on a fresh OSD-wiped cluster).
wait_kustomization apps "apps" 90
wait_kustomization gitea "Gitea (incl. data-gitea-0 PVC)" 90
wait_kustomization influxdb "InfluxDB (HR install + token Jobs)" 600
wait_kustomization woodpecker "Woodpecker CI (incl. wp-pg CNPG backend)" 600
wait_kustomization authentik "Authentik (incl. ak-pg CNPG backend)" 600
File diff suppressed because it is too large Load Diff
+20
View File
@@ -0,0 +1,20 @@
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: gitea
namespace: flux-system
spec:
# Gitea in its own `gitea` namespace (#878), formerly under the shared `apps` KS.
interval: 10m
retryInterval: 2m
timeout: 15m
sourceRef:
kind: GitRepository
name: flux-system
path: ./kubernetes/apps/gitea
prune: true
wait: true
dependsOn:
- name: infrastructure
- name: infrastructure-crds
- name: infrastructure-ceph-cluster
@@ -0,0 +1,22 @@
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: wiki-index
namespace: flux-system
spec:
interval: 10m
retryInterval: 1m
# First start on an empty PVC embeds every note (~5 min) before /health is 200.
timeout: 20m
sourceRef:
kind: GitRepository
name: flux-system
path: ./kubernetes/apps/wiki-index
prune: true
wait: true
dependsOn:
- name: infrastructure
- name: infrastructure-crds
- name: infrastructure-ceph-cluster
# wiki-embed (the embedding backend) lives in the llama-server app.
- name: llama-server
+103
View File
@@ -0,0 +1,103 @@
# Identical on every node and free of addresses. defines.conf and peers.conf
# are rendered per node by the init container; the first is included above the
# templates that reference it, the second below them.
include "/etc/bird/defines.conf";
router id from "pod0";
log stderr all;
protocol device { scan time 10; }
# Everything a pod may need to reach, in one table, resolved by longest prefix:
# the node's attached networks, the peers' pod CIDRs, and a default for the
# rest. No rule has to carve out exceptions, because the specific routes win on
# their own.
#
# The table is BIRD's alone. If this daemon stops, it empties, the rule finds
# nothing and falls through to main — pod traffic returns to VLAN 40,
# unseparated rather than misrouted. That is why the default route lives here
# and not in the Talos config: a default that outlived the specific routes
# would send pod traffic to the UDM.
# The node's attached networks, taken from the interfaces rather than written
# down. The usual advice is not to export these to the kernel because it
# maintains them itself — true of main, not of a table the kernel never fills.
protocol direct attached {
interface "mgmt0", "ingress0", "pod0", "node0";
ipv4 { table master4; };
ipv6 { table master6; };
}
# Pod egress. GW4/GW6 are the UDM on the ingress segment, published as node
# annotations beside the address that defines them.
protocol static egress4 {
ipv4 { table master4; };
route 0.0.0.0/0 via GW4;
}
protocol static egress6 {
ipv6 { table master6; };
route ::/0 via GW6;
}
# This node's own pod CIDR, via the interface Cilium serves it on. Originated
# rather than read back out of the kernel: reading it back made BIRD's view
# depend on the shape of Cilium's route, and the IPv4 one is rejected outright,
# its next hop sitting inside the prefix it describes.
#
# It belongs in the table, not just in the BGP announcement. A reply from a pod
# on another node is forwarded traffic with a pod source, so the rule matches it
# too; without this route it would find only the default and leave for the UDM.
# Measured — cross-node pod traffic, and with it cluster DNS, stopped.
protocol static own4 {
ipv4 { table master4; };
route OWN_POD4 via "cilium_host";
}
protocol static own6 {
ipv6 { table master6; };
route OWN_POD6 via "cilium_host";
}
protocol kernel k4 {
kernel table 47;
metric 0;
ipv4 { table master4; import none; export all; };
}
protocol kernel k6 {
kernel table 47;
metric 0;
ipv6 { table master6; import none; export all; };
}
# iBGP over pod0. 64514 keeps clear of Cilium's 64513, which peers with the UDM
# from ingress0 and is unrelated.
#
# direct is explicit because iBGP defaults to multihop, which makes the gateway
# recursive — BIRD would try to resolve the peer in the pod RIB. strict bind
# keeps the listener off mgmt0 and ingress0.
#
# Only the node's own CIDR is offered: the export filter drops everything the
# static protocols and the direct protocol produce.
template bgp pod4 {
local as 64514;
source address MY4;
direct;
strict bind;
ipv4 {
next hop self;
import all;
export filter { if net = OWN_POD4 then accept; reject; };
};
}
template bgp pod6 {
local as 64514;
source address MY6;
direct;
strict bind;
ipv6 {
next hop self;
import all;
export filter { if net = OWN_POD6 then accept; reject; };
};
}
include "/etc/bird/peers.conf";
@@ -0,0 +1,115 @@
apiVersion: v1
kind: ServiceAccount
metadata:
name: bird
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: bird
rules:
# The peer list is read from the nodes' pod0 annotations, so no address is
# written into this repo twice.
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: bird
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: bird
subjects:
- kind: ServiceAccount
name: bird
namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: bird
namespace: kube-system
labels:
tier: node
app: bird
spec:
selector:
matchLabels:
app: bird
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
tier: node
app: bird
spec:
# Carries pod east-west once the routing rules land. Without it table 48
# stays empty and the rules fall through to main, i.e. back to VLAN 40.
priorityClassName: system-node-critical
# Writes host routes and peers on host addresses.
hostNetwork: true
serviceAccountName: bird
# Every node here is a control plane node, so this is not optional.
tolerations:
- operator: Exists
effect: NoSchedule
initContainers:
- name: render-config
# By digest, not by the dated tag: the tag is rebuilt within the day
# it names, and the cluster's image path keeps resolving it to the
# first build of that day — imagePullPolicy: Always does not help,
# because the tag itself resolves stale. Update this when the image
# is rebuilt.
image: gitea.renner.uno/gitea_admin/bird@sha256:3bc8500613eeb45b2d8858b5445198bae3990ecf64f6d891aa94797f5a09dc1d
command: ["/bin/sh", "-c"]
args:
- |
set -eu
cp /config/bird.conf /etc/bird/bird.conf
kubectl get node "$NODE_NAME" \
-o go-template-file=/config/defines.tmpl > /etc/bird/defines.conf
kubectl get nodes --field-selector "metadata.name!=$NODE_NAME" \
-o go-template-file=/config/peers.tmpl > /etc/bird/peers.conf
# BIRD's own parser is the check: a node whose annotation is
# missing renders as <no value> and fails here rather than
# crashlooping the daemon or forming a mesh with a hole.
bird -p -c /etc/bird/bird.conf
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
volumeMounts:
- name: config
mountPath: /config
- name: etc-bird
mountPath: /etc/bird
containers:
- name: bird
image: gitea.renner.uno/gitea_admin/bird@sha256:3bc8500613eeb45b2d8858b5445198bae3990ecf64f6d891aa94797f5a09dc1d
resources:
requests:
cpu: "50m"
memory: "32Mi"
limits:
cpu: "200m"
memory: "128Mi"
securityContext:
capabilities:
drop: ["ALL"]
# NET_ADMIN writes the routes; NET_BIND_SERVICE binds 179.
add: ["NET_ADMIN", "NET_BIND_SERVICE"]
volumeMounts:
- name: etc-bird
mountPath: /etc/bird
volumes:
- name: config
configMap:
name: bird-config
- name: etc-bird
emptyDir: {}
@@ -0,0 +1,9 @@
define MY6 = {{ index .metadata.annotations "net.renner.uno/pod0-v6" }};
define MY4 = {{ index .metadata.annotations "net.renner.uno/pod0-v4" }};
define GW6 = {{ index .metadata.annotations "net.renner.uno/egress-gw-v6" }};
define GW4 = {{ index .metadata.annotations "net.renner.uno/egress-gw-v4" }};
{{/* podCIDRs follows the cluster's podSubnets order, v4 first. A flip puts an
IPv6 prefix into an ipv4 protocol, which fails the parse in the init
container rather than passing silently. */}}
define OWN_POD4 = {{ index .spec.podCIDRs 0 }};
define OWN_POD6 = {{ index .spec.podCIDRs 1 }};
@@ -0,0 +1,12 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- daemonset.yaml
configMapGenerator:
# Hashed name, so editing any of these rolls the DaemonSet.
- name: bird-config
namespace: kube-system
files:
- bird.conf
- defines.tmpl
- peers.tmpl
@@ -0,0 +1,7 @@
{{- range $i, $n := .items -}}
{{- $v6 := index $n.metadata.annotations "net.renner.uno/pod0-v6" -}}
{{- $v4 := index $n.metadata.annotations "net.renner.uno/pod0-v4" -}}
# {{ $n.metadata.name }}
protocol bgp peer{{ $i }}_v6 from pod6 { neighbor {{ $v6 }} as 64514; }
protocol bgp peer{{ $i }}_v4 from pod4 { neighbor {{ $v4 }} as 64514; }
{{ end -}}
@@ -118,14 +118,21 @@ spec:
# traffic out to the VIP and back (which fails with L2 announcements).
lbExternalClusterIP: true
cni:
# Multus (kubernetes/infrastructure/multus) runs alongside Cilium and
# writes 00-multus.conf to /etc/cni/net.d so kubelet picks multus
# first; multus then delegates to Cilium's 05-cilium.conflist via
# clusterNetwork. The default `exclusive: true` makes cilium-cni's
# install step rename anything alphabetically before its own config
# to *.cilium_bak on every reconcile, which kept nuking multus.
# Setting it false lets multus's config coexist.
# Multus is the only writer of /etc/cni/net.d (#832): Cilium's conflist
# goes to /etc/cni/net.d/multus/, which the container runtime does not
# read, and Multus delegates to it via clusterNetwork. Without this a
# pod created after a reboot, before Multus rewrote 00-multus.conf, came
# up with Cilium alone. See kubernetes/infrastructure/multus.
confPath: /etc/cni/net.d/multus
# The default `exclusive: true` renames other configs in the conf dir to
# *.cilium_bak on every reconcile, which kept nuking multus.
exclusive: false
# One agent at a time: the chart default of 2 restarts two of three
# nodes' datapath at once.
updateStrategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
# Talos-specific settings
securityContext:
capabilities:
@@ -180,3 +187,17 @@ spec:
# the supernets here cover the LAN as well, which is the point.
ipv4NativeRoutingCIDR: 10.0.0.0/8
ipv6NativeRoutingCIDR: fd00::/8
# Pod east-west leaves the node on pod0 (VLAN 48) rather than the
# management VLAN. Native routing hands forwarding to the kernel;
# autoDirectNodeRoutes installs the remote PodCIDR routes, which is
# allowed because all three nodes share pod0's L2 segment.
#
# directRoutingDevice is explicit on purpose: with several addressed
# devices Cilium falls back to the one carrying the Kubernetes InternalIP,
# which is mgmt0.
#
# Encapsulation is gone, so the datapath MTU goes 1450 -> 1500.
routingMode: native
autoDirectNodeRoutes: true
nodePort:
directRoutingDevice: pod0
@@ -72,6 +72,9 @@ spec:
# they contend for it. Sized for an inference server plus room for
# a transcoder and a benchmark pod alongside it.
- -shared-dev-num=4
# Advertises gpu.intel.com/monitoring: one resource that grants a
# pod every GPU on the node instead of a slot. The exporter needs it.
- -enable-monitoring
securityContext:
seLinuxOptions:
type: "container_device_plugin_t"
+2 -1
View File
@@ -7,6 +7,7 @@ resources:
- cluster-settings/
- cilium/
- multus/
- bird/
- cert-manager/
- snapshot-controller/
- metrics-server/
@@ -18,7 +19,7 @@ resources:
- cloudnative-pg/
- mariadb-operator/
- netbird-ingress/
- netbird-proxy-services/
- netbird-service-sync/
- coredns-config/
- k8s-gateway/
- external-dns-unifi/
@@ -111,10 +111,18 @@ metadata:
name: multus
namespace: kube-system
---
# Multus is the only writer of /etc/cni/net.d (#832). Cilium writes its conflist
# to /etc/cni/net.d/multus/, a subdirectory the container runtime does not read
# (libcni skips directories); Multus picks it up there via multusAutoconfigDir
# and writes 00-multus.conf. Until that file exists the runtime has no network
# config at all, so no pod can be created with Cilium alone after a reboot.
#
# The "-subdir" names date from the node-by-node rollout; renaming would roll
# Multus again for nothing.
kind: ConfigMap
apiVersion: v1
metadata:
name: multus-daemon-config
name: multus-daemon-config-subdir
namespace: kube-system
labels:
tier: node
@@ -127,7 +135,7 @@ data:
"logLevel": "verbose",
"logToStderr": true,
"cniConfigDir": "/host/etc/cni/net.d",
"multusAutoconfigDir": "/host/etc/cni/net.d",
"multusAutoconfigDir": "/host/etc/cni/net.d/multus",
"multusConfigFile": "auto",
"socketDir": "/host/run/multus/"
}
@@ -135,16 +143,16 @@ data:
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-multus-ds
name: kube-multus-ds-subdir
namespace: kube-system
labels:
tier: node
app: multus
name: multus
name: multus-subdir
spec:
selector:
matchLabels:
name: multus
name: multus-subdir
updateStrategy:
type: RollingUpdate
template:
@@ -152,7 +160,7 @@ spec:
labels:
tier: node
app: multus
name: multus
name: multus-subdir
spec:
# Upstream ships this DaemonSet with no priority class, so it runs at 0
# and can preempt nothing. Any event that makes the pod need a fresh
@@ -224,6 +232,10 @@ spec:
volumeMounts:
- name: cni
mountPath: /host/etc/cni/net.d
# Exists before Multus starts, so it waits for Cilium's conflist
# instead of exiting on a missing directory.
- name: cni-cilium-conf
mountPath: /host/etc/cni/net.d/multus
# multus-daemon expects that cnibin path must be identical between pod and container host.
# e.g. if the cni bin is in '/opt/cni/bin' on the container host side, then it should be mount to '/opt/cni/bin' in multus-daemon,
# not to any other directory, like '/opt/bin' or '/usr/bin'.
@@ -279,6 +291,10 @@ spec:
- name: cni
hostPath:
path: /etc/cni/net.d
- name: cni-cilium-conf
hostPath:
path: /etc/cni/net.d/multus
type: DirectoryOrCreate
- name: cnibin
hostPath:
path: /opt/cni/bin
@@ -287,7 +303,7 @@ spec:
path: /
- name: multus-daemon-config
configMap:
name: multus-daemon-config
name: multus-daemon-config-subdir
items:
- key: daemon-config.json
path: daemon-config.json
@@ -39,9 +39,11 @@ spec:
name: whereabouts-reconciler
spec:
serviceAccountName: whereabouts
# Every node here is a control plane node.
# Every node here is a control plane node. Keyed: a keyless Exists would
# also tolerate node.kubernetes.io/unschedulable and ignore cordons.
tolerations:
- operator: Exists
- key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule
containers:
- name: whereabouts-reconciler
@@ -179,6 +179,22 @@ kind: Namespace
metadata:
name: woodpecker
---
# Gitea, moving out of the shared `apps` namespace (issue #878). Declared here
# like `woodpecker` above, so it exists before infrastructure-crds applies its
# CiliumNetworkPolicies. No pod-security labels: the cluster default applies.
apiVersion: v1
kind: Namespace
metadata:
name: gitea
---
# netbird-service-sync, moving out of `openbao` (issue #876). Declared here like
# `woodpecker` above, so it exists before infrastructure-crds applies its
# CiliumNetworkPolicies. No pod-security labels: the cluster default applies.
apiVersion: v1
kind: Namespace
metadata:
name: netbird-service-sync
---
# Local LLM inference (kubernetes/apps/llama-server). Declared here, like
# `woodpecker` above, so the `infrastructure` KS creates it before
# `infrastructure-crds` applies the namespace's CiliumNetworkPolicies.
@@ -1,28 +1,28 @@
# Reconciles NetBird reverse-proxy services from Ingresses that carry
# dns.renner.uno/public: one TLS-passthrough service per host on :443,
# targeting Traefik's ClusterIP through a subnet resource over the service
# CIDR (ci/images/netbird-proxy-services/controller.py).
# CIDR (ci/images/netbird-service-sync/controller.py).
#
# openbao namespace, like the rotators: its default-deny already allows egress
# to OpenBao, DNS, the apiserver and `world` (netbird.renner.uno), and the
# NetBird PAT is read from OpenBao on every pass through the
# netbird-proxy-services role — policies/netbird-proxy-services.hcl, applied
# once by hand like the rotator roles. Until that role exists the pod logs an
# auth error each minute and does nothing.
# Own namespace; its egress (DNS, apiserver, OpenBao, NetBird API) is in
# network-policies/netbird-service-sync-ns.yaml. The NetBird PAT is read from
# OpenBao on every pass through the netbird-service-sync role —
# policies/netbird-service-sync.hcl, applied once by hand like the rotator
# roles. Until that role exists the pod logs an auth error each minute and does
# nothing.
apiVersion: v1
kind: ServiceAccount
metadata:
name: netbird-proxy-services
namespace: openbao
name: netbird-service-sync
namespace: netbird-service-sync
labels:
app.kubernetes.io/name: netbird-proxy-services
app.kubernetes.io/name: netbird-service-sync
---
# Read-only, cluster-wide: public Ingresses live in many namespaces, the
# Traefik Service in ingress/.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: netbird-proxy-services-read
name: netbird-service-sync-read
rules:
- apiGroups: ["networking.k8s.io"]
resources: ["ingresses"]
@@ -34,23 +34,23 @@ rules:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: netbird-proxy-services-read
name: netbird-service-sync-read
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: netbird-proxy-services-read
name: netbird-service-sync-read
subjects:
- kind: ServiceAccount
name: netbird-proxy-services
namespace: openbao
name: netbird-service-sync
namespace: netbird-service-sync
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: netbird-proxy-services
namespace: openbao
name: netbird-service-sync
namespace: netbird-service-sync
labels:
app.kubernetes.io/name: netbird-proxy-services
app.kubernetes.io/name: netbird-service-sync
spec:
replicas: 1
# One writer against the management API at a time.
@@ -58,13 +58,13 @@ spec:
type: Recreate
selector:
matchLabels:
app.kubernetes.io/name: netbird-proxy-services
app.kubernetes.io/name: netbird-service-sync
template:
metadata:
labels:
app.kubernetes.io/name: netbird-proxy-services
app.kubernetes.io/name: netbird-service-sync
spec:
serviceAccountName: netbird-proxy-services
serviceAccountName: netbird-service-sync
securityContext:
runAsNonRoot: true
# Numeric, or the kubelet refuses the image's named user with
@@ -76,7 +76,11 @@ spec:
type: RuntimeDefault
containers:
- name: controller
image: gitea.renner.uno/gitea_admin/netbird-proxy-services:2026-09-07
# Pinned by digest: the date tag is rebuilt on every push of the
# image directory (branch and master alike) and the registry keeps
# only the newest manifest per tag, so the digest must come from
# the MASTER build of the code change.
image: gitea.renner.uno/gitea_admin/netbird-service-sync:2026-09-23@sha256:64f6057c508640e1dbc50457dc26e9b2de058a111fd46ca096021d7444b5bbff
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
@@ -88,7 +92,7 @@ spec:
- name: BAO_ADDR
value: http://openbao.openbao.svc.cluster.local:8200
- name: BAO_ROLE
value: netbird-proxy-services
value: netbird-service-sync
# Must match cluster.network.serviceSubnets in talos/patches/cluster.yaml.
- name: SERVICE_CIDR
value: "10.96.0.0/12"
@@ -126,3 +126,59 @@ spec:
- { type: 135, family: IPv6 }
- { type: 136, family: IPv6 }
- { type: 143, family: IPv6 }
---
# wiki-embed is the one ai pod Traefik may reach: it sits behind BasicAuth
# (apps/llama-server/wiki-embed-ingress.yaml), not behind the Authentik gate.
# Scoped to this pod so the default-deny above keeps Traefik off llama-server.
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: wiki-embed-from-traefik
namespace: ai
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: wiki-embed
ingress:
- fromEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": ingress
app.kubernetes.io/name: traefik
toPorts:
- ports:
- port: "8080"
protocol: TCP
---
# wiki-index: Traefik (LAN clients behind BasicAuth) and Gitea (push webhook)
# may reach it; it reaches OpenBao for its credentials. Gitea itself is reached
# through Traefik, which the namespace egress above already allows.
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: wiki-index
namespace: ai
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: wiki-index
ingress:
- fromEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": ingress
app.kubernetes.io/name: traefik
- matchLabels:
"k8s:io.kubernetes.pod.namespace": gitea
app.kubernetes.io/name: gitea
toPorts:
- ports:
- port: "8080"
protocol: TCP
egress:
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": openbao
app.kubernetes.io/name: openbao
toPorts:
- ports:
- port: "8200"
protocol: TCP

Some files were not shown because too many files have changed in this diff Show More