Pod-to-pod traffic currently rides the management VLAN. In tunnel mode
Cilium encapsulates to the peer's Kubernetes InternalIP, so east-west
follows VLAN 40 wherever it goes -- which blocks moving management onto
the 1G NIC without dragging pod traffic with it.
The node IP stays on VLAN 40. Instead this uses the documented device
selection: native routing hands forwarding to the kernel, and
autoDirectNodeRoutes installs the remote PodCIDR routes, which is
permitted because all three nodes share pod0's L2 segment.
"By default, if a single device was detected or specified via devices
then Cilium will use that device for direct routing. Otherwise,
Cilium will use a device with Kubernetes InternalIP or ExternalIP
set."
directRoutingDevice is therefore set explicitly: mgmt0, ingress0 and
pod0 all carry addresses, so the fallback would pick mgmt0.
Verified beforehand that pod0 is a working path between nodes -- from a
pod on k8s-2, k8s-1's kubelet answers 401 on both 10.2.48.11:10250 and
[fd02:48::11]:10250, so the L2 and both address families work.
ipv4/ipv6NativeRoutingCIDR are already set and become mandatory in this
mode; 10.244.0.0/16 and fd10:244::/48 sit inside them, so pod-to-pod
stays unmasqueraded as before.
Not rolling: dropping encapsulation is a cluster-wide datapath change
and a node that has switched cannot exchange pod traffic with one that
has not. Needs a scheduled window, not a background reconcile.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Phase B step 2 of the traffic-class split. The eBGP session source
address is what the UDM installs as the next-hop for every PodCIDR and
LoadBalancer prefix the cluster advertises. Peering over the management
LAN therefore pinned all world->pod return traffic to the 1G mgmt link,
which is the thing #576 exists to fix. Moving the peers onto VLAN 47
puts that traffic on the 10G ingress link instead.
No localAddress or sourceInterface override is needed: the kernel
already picks the ingress0 address for both families.
ip -6 route get fd02:47::1
fd02:47::1 dev ingress0 proto kernel src fd02:47::5a47:caff:fe74:7027
Both peer addresses were verified reachable from the cluster before
this change -- 10.2.47.1 and fd02:47::1 each answer HTTP with a 301,
and the UDM's neighbour entry on ingress0 is REACHABLE.
The UDM already accepts sessions from this segment: PR #835 added the
10.2.47.0/24 and fd02:47::/64 listen ranges and is uploaded. Its VLAN 40
ranges stay in place for now and get dropped in step 3, once every
session is confirmed established on 47.
gracefulRestart (120s) covers the peer replacement, so the UDM holds the
learned routes across the flap rather than withdrawing the LB VIPs.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
First step of moving the world-to-pod class onto ingress0 (#576). This
changes nothing about where the cluster peers from — it only widens
what the UDM will accept.
The order matters. Cilium's sessions are outbound to the UDM, and the
UDM accepts them through `bgp listen range`. Moving the cluster to
ingress0 before the UDM listens on 10.2.47.0/24 would drop every
session, and with them every advertised LoadBalancer VIP, so external
ingress would go down until the UDM caught up. Listing both segments
first removes that window entirely: a listen range only accepts a
session, it never opens one, so the 47 ranges sit idle until the
cluster actually moves.
The prefix-lists and route-maps are untouched. They filter which
prefixes the UDM accepts — pod CIDRs and the LB pool — and the cluster
advertises the same prefixes either way. Only the next hop changes.
Uploaded by hand at UniFi UI -> Settings -> Routing -> BGP, as before.
Verify with `vtysh -c "show ip bgp summary"`: the six existing sessions
on 10.2.40.x and fd40:: must still be established afterwards, and no
new ones appear until the cluster side moves.
The 40 ranges come out in a later change, once every session is
established on 47.
Refs #576
Adding a second DHCP client per node in #831 broke node DNS. Both
clients announce the same hostname, so the UDM registered each node
against VLAN 47 instead of VLAN 40: `k8s-1.srv.renner.uno` and
`k8s-3.srv.renner.uno` stopped resolving, and the reverse for
10.2.47.11 came back as bare `k8s-1`. The lease's resolver also
displaced the management one, leaving 10.2.47.1 ahead of 10.2.40.1 in
the node's DNS servers.
That matters beyond `talosctl -n <fqdn>`: bootstrap.sh resolves node
FQDNs in the talos phase, so a rebuild depends on those records.
The UDM cannot register a network's clients selectively, so the client
goes rather than the record. DHCP is now off on VLAN 47 entirely and
ingress0 declares .11/.12/.13 by node ordinal, matching pod0's
numbering. Its default route is declared with the same losing metric
2048 the lease supplied, so nothing about routing changes: mgmt0 keeps
both default routes until the flip.
IPv4 only. The problem was DHCPv4 announcing a hostname; IPv6 never had
one. The prefix is announced correctly, so ingress0 keeps taking its
IPv6 address by SLAAC exactly as mgmt0 takes its fd40:: address —
derived from the MAC, stable across reboots, nothing to allocate.
accept_ra therefore stays at its default of 2. An IPv6 default router
is a link-local address that can only be learned from the RA, and with
net.ipv6.conf.all.forwarding=1 — which Cilium sets — Linux ignores RAs
below 2 entirely. Only accept_ra_defrtr is set to 0, because mgmt0 owns
the default until the flip; dropping that one line is what later hands
the IPv6 default to ingress0.
pod0 keeps accept_ra 0: it is a closed segment with no router on it, so
there is genuinely nothing to learn.
No reboot needed. Address, route and sysctl changes apply live, and the
whole change is provable per node with `--mode=try`, which reverts
itself.
Refs #576
First pass of the traffic-class split (#576). Purely additive: nothing
moves onto the new segments yet.
Links by role, not by PCI topology. A udev rule per node matches the
burned-in MAC and renames at boot, the way ceph-san already did:
enp2s0f0np0 -> uplink, enp87s0 -> mgmt. Shared manifests now name roles
and only each node's own patch knows the hardware. The two comments
elsewhere that named the old links go with them.
Two new VLANs on the uplink, both created on the UDM already:
- pod0, VLAN 48. Node-to-node only; the overlay rides it once the
kubelet node IP moves. Statically addressed .11/.12/.13 out of
10.2.48.0/24 and fd02:48::/64, no gateway, no DHCP, accept_ra 0. A
lease would make the cluster's data path depend on the UDM being up in
order to form, on a segment whose traffic never reaches the UDM.
- ingress0, VLAN 47. World-to-pod: BGP session, LoadBalancer next hop,
masquerade source. Publicly routed, so DHCP — but at routeMetric 2048
and with accept_ra_defrtr 0, because DHCPv4Config can only
deprioritise a default route, not suppress one, and two defaults at
1024 tie.
mgmt0 keeps the node IP and both default routes. A mistrunked VLAN 47 or
48 therefore cannot strand a node, and the flip is a later change that
needs no reboot and is provable with `--mode=try`.
Applying needs a rolling reboot: a udev rule cannot rename a link on a
running node (Resource busy). Drain first, one node at a time, and wait
for CephCluster HEALTH_OK rather than node Ready between them.
Refs #576
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The tag build-ci-images published from #827 (pipeline 8573): talosctl
v1.13.9, kubectl v1.36.3, flux 2.9.5. Six references moved from
2026-08-10.
Refs #426
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Regenerated with `flux install --export
--components-extra=image-reflector-controller,image-automation-controller`
from the v2.9.5 CLI; the same component set as before.
Controllers: source v1.8.3 -> v1.9.5, kustomize v1.8.4 -> v1.9.5, helm
v1.5.4 -> v1.6.4, notification v1.8.4 -> v1.9.4, image-reflector v1.1.1
-> v1.2.5, image-automation v1.1.2 -> v1.2.5. CRDs updated, the
allow-webhooks NetworkPolicy now names port 9292.
Prerequisites from the 2.9 release notes, verified before this commit:
the removed image v1beta2 and notification v1beta2 API versions are
neither used in the repo (image.* all v1, Alert/Provider v1beta3) nor
stored in the cluster's CRDs (storedVersions all current), so no
`flux migrate` step is needed. helm-controller 1.6 changes the default
post-render strategy; no HelmRelease here uses postRenderers.
`flux check --pre` passes on Kubernetes 1.36.3 and the manifests pass a
server-side dry run against the live cluster.
Refs #426
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
From the update audit (#426): the tools CI drives the cluster with
trailed what the cluster runs.
ci/images/k8s/Containerfile
- talosctl v1.12.6 -> v1.13.9, the Talos version on the nodes. An older
talosctl can misjudge or reject config generated for a newer machine.
- kubectl v1.36.2 -> v1.36.3, the cluster's Kubernetes version.
- flux CLI 2.9.2 -> 2.9.5, the current 2.9.x. Checksums verified against
the upstream release assets (talos sha256sum.txt, flux checksums.txt)
and against a local download.
.woodpecker/ci-manifests-test.yml
- woodpecker-cli v3.17.0 -> v3.18.0, the version the server and agent
already run; the lint step was one minor behind its own server.
Merging triggers build-ci-images (path ci/images/**), which publishes
ci-k8s:<date>. The six pipelines pinned to ci-k8s:2026-08-10 move to
that tag in a follow-up once it exists.
Refs #426
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The description told on-call to try the recovery key as a last resort,
while RUNBOOK Step 4 in the same change (and upstream: recovery keys are
"purely an authorization mechanism") says it cannot unseal an
auto-sealed server. Say what actually helps: fix the Secret, restart
the pod. Also drop a duplicate root-token computation in
restore_openbao.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Follow-up to #825 (static-seal auto-unseal). The rebuild path and the
operator docs still assumed shamir with a manual unseal per replica.
bootstrap.sh
- Required secrets: `seal-key` (pass show homelab/openbao/seal-key)
replaces `unseal-key`. The snapshots are encrypted under the seal key,
so it is the DR-critical secret now; the former unseal key is the
recovery key and is not needed for a restore (upstream: recovery keys
"are purely an authorization mechanism" and cannot unseal a server
whose auto-seal is broken).
- flux phase creates the `openbao-seal-key` Secret (namespace + Secret,
before `flux bootstrap`) from the collected key. --from-literal via a
substitution, because `pass show` appends a newline and OpenBao
rejects a key that is not exactly 32 bytes of base64 ("unknown
encoding for AES-256 key" — verified). An existing Secret with the
same value is left alone; a different value is refused, since the
data is written under whichever key OpenBao started with.
- restore_openbao: `init -recovery-shares=1 -recovery-threshold=1`,
root token from the output, wait for the self-unseal (init returns
with the node sealed; the server's own retry loop unseals it within
~5 s), then `snapshot restore` WITHOUT -force: OpenBao verifies the
snapshot was written under this server's seal key and refuses a
mismatch by name, which is the one guard against restoring under a
wrong Secret. -force skipped that check and would leave a server that
can never unseal. The throwaway `unseal` step and its argv/tty
discussion are gone — under an auto-seal there is no unseal key.
- unseal_openbao -> bring_up_openbao: waits for spec.replicas pods to
report sealed=false, walking the OrderedReady chain as before, and on
timeout prints each still-sealed pod's `bao status` instead of a bare
"timed out".
Rehearsed locally with two OpenBao servers sharing a static key (file://
form, bao 2.5.5): init on an auto-seal returns an empty unseal_keys_b64,
one recovery key and a root token and the server is unsealed within
seconds; a restore of A's snapshot into fresh B (both with and without
force) re-seals and reloads the root key via the auto seal ~5 s later
("done reloading root key using auto seal"), after which B serves A's
data with A's root token. Restarting B under a different key leaves it
sealed with "failed to decrypt keys from storage: message
authentication failed".
Docs and alerts: RUNBOOK-cold-rebuild Step 4 and MIGRATION-sops-openbao
describe the auto-seal flow and the recovery-key semantics; the
OpenBao alert descriptions and the snapshot CronJob comment no longer
tell the reader to unseal by hand. vmtest-job comment updated.
Not covered: a full VMTEST_FULL=1 rehearsal through the flux phase. The
wiki records none ever reaching past the Flux bootstrap checkpoint, so
that gap predates this change.
Refs #805
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Every restart sealed the replica until a human unsealed it. On
2026-09-07 a rolling talosctl upgrade rebooted all three nodes; the PDB
refused the second eviction while the first replica was sealed, Talos
waited out its 5-minute drain timeout and rebooted anyway, and raft
quorum was gone for 24 minutes — every ExternalSecret and the
Kustomizations behind them with it. A power loss would do the same
with no timeout to blame.
OpenBao 2.6.1 supports a static seal: a 32-byte key the server uses to
unseal itself. The key is generated into pass (homelab/openbao/seal-key)
and created as the openbao-seal-key Secret before this release reconciles;
the chart hands it to the server as BAO_SEAL_KEY from that Secret.
This overturns the principle in bootstrap.sh that the key protecting
OpenBao must not live in the cluster OpenBao secures. Decided by the
owner on 2026-09-08: the alternatives were a transit seal from a second
OpenBao on edge-1, which raft traffic would wear out, or a workstation
unseal script that may not be at hand when needed. The exposure is
bounded by what ESO already does — every synced secret is plaintext in
etcd (encrypted at rest) — and grows only by what lives solely in
OpenBao.
Migration is a live, per-replica procedure with the shamir key becoming
the recovery key; the StatefulSet is OnDelete, so merging this changes
nothing until each pod is restarted in turn. The bootstrap and rebuild
paths (bootstrap.sh, vmtest) follow in a separate change once the
migration is verified.
Refs #805
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
OpenBaoQuorumLost waited for: 5m. Below quorum the secret store is
down — the ClusterSecretStore goes InvalidProviderConfig and every
ExternalSecret and the Kustomizations behind them stall — and a sealed
replica never recovers on its own, so the window only delayed the page.
It fires on the first evaluation now.
OpenBaoUnavailable, the external probe, drops from 5m to 1m: two
consecutive failed probes, enough to skip a single lost probe through
Traefik, not enough to sit on a sealed store.
Both StatefulSet expressions are wrapped in max by (namespace,
statefulset). kube-state-metrics stamps instance/pod on the series, so
a rescheduled exporter starts a new series and Prometheus resets the
for: timer with it — on 2026-09-07 that made OpenBaoReplicaNotReady fire
14 minutes late. Alertmanager routes match alertname/source only, so the
dropped labels change nothing downstream.
Refs #805
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Grafana dashboard for the gpumon_* series from intel-gpu-exporter:
current Render/3D busy, GPU power and GT frequency per node as stat
tiles, then one time series per engine (Render/3D, Video, VideoEnhance,
Blitter) with node as the series, GT frequency, GPU and package power,
and memory-controller read/write bandwidth.
Node colours are fixed per node across every panel — blue, orange, aqua
for k8s-1/2/3 — rather than assigned by series order, so a node keeps
its colour whatever the $node filter shows. The trio validates for
colour-vision deficiency and normal vision on both Grafana themes
(worst all-pairs ΔE 9.4 / 20.9). Reads and writes share one axis and
differ by dash, not hue. One axis per panel throughout.
gpumon_clients_count is deliberately absent: per-client attribution
needs hostPID, which the exporter does not have, so it would read 0
forever and imply an idle GPU.
Refs #752
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The -device flag named /dev/dri/card0. That is the right node today on
every host, but nothing guarantees the number: a second GPU or a
different probe order and the exporter would silently watch the wrong
device or none. The plugin's monitoring resource injects every GPU on
the node and intel_gpu_top selects it on its own — verified with a probe
pod on k8s-2, which produced the same gpumon_* series without the flag.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A DaemonSet running clambin/intel-gpu-exporter (intel_gpu_top behind a
Prometheus endpoint) with a ServiceMonitor, giving gpumon_engine_usage,
gpumon_frequency, gpumon_power, gpumon_imc_* and gpumon_clients_count
per node.
Privilege model, validated with a one-shot pod before this commit:
/dev/dri comes from the device plugin's gpu.intel.com/monitoring
resource (every GPU on the node, no hostPath); the i915 PMU that
intel_gpu_top reads is a system-wide event source and the nodes run
perf_event_paranoid=3, so CAP_PERFMON is the one capability added.
No privileged, no hostPID, no hostNetwork, read-only root filesystem.
Root rather than a fixed uid because the capability has to be effective
without file capabilities on the binary.
End-to-end check on k8s-2 during a 40 s chat completion on llama-server:
Render/3D busy 98 %, actual frequency 1449 MHz, GPU power 18.8 W, while
the pod's CPU stayed near 100m — the "is it really on the GPU" question
this exists to answer.
XPU Manager was ruled out (Arc Pro / data-centre parts only); the
archived onedr0p exporter needs privileged + hostPID.
Fixes#752
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Enables -enable-monitoring on the device plugin. With 0.36.0 that
advertises gpu.intel.com/monitoring, a single resource per node that
hands a pod every GPU rather than one of the shared slots; it is what a
metrics exporter requests to see the whole iGPU, and it is the only way
to get /dev/dri into a pod without a hostPath.
Nothing consumes it yet. The nodes currently advertise gpu.intel.com/i915
only — the issue assumed the chart default had this on, but the
DaemonSet here is hand-written and did not.
Refs #752
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Both consume their ConfigMap with envFrom, which Kubernetes does not
watch: a changed value applies in git and in the cluster while the
running container keeps the environment it started with. Paperless hit
exactly this in #739 and got the annotation in #741; these two were the
remaining envFrom consumers without one.
ftp-scanner, the third Deployment named in the issue, needs nothing: its
ConfigMap comes from a configMapGenerator that deliberately keeps the
content-hash suffix, so any change renames the ConfigMap, mutates the
pod template and rolls the pod on its own.
Reloader already watches all namespaces (kubernetes/infrastructure/reloader).
Fixes#742
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
#817 pinned the digest from the branch build (pipeline 8452). Merging
#816 rebuilt the same date tag on master (pipeline 8457), Gitea kept
only the new manifest, and the pod has been in ImagePullBackOff since:
"failed to resolve reference ...@sha256:0d266b02...". With the Recreate
strategy no controller is running, so the services are unchanged.
The kaniko step of the master build pushed sha256:caffa308...; that is
the digest to pin. Comment states the rule.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Pins the controller image by digest. The registry tag is the build
date and pipeline 8452 rebuilt today's tag in place, so the tag alone
cannot carry a new build to the cluster; the digest can.
Merge only after the Traefik change from 1/2 has reconciled: from this
build on every service target sends a PROXY protocol v2 header, and
Traefik drops such connections unless websecure trusts the pod CIDRs.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
TLS-passthrough services hand Traefik a raw TCP stream, so every WAN
request now logs the routing peer's pod IP (10.244.4.64/.112 in the
access log) instead of the client. CrowdSec's detection parses exactly
that log; with the HTTP-mode services gone it would attribute every
external request to two pod IPs, and a re-armed bouncer would repeat the
2026-06-27 mass ban.
The controller now sets options.proxy_protocol on every target (PROXY
protocol v2, TCP/TLS modes only per NetBird) and treats a service
without it as out of date, so the eleven existing services are updated
on the next pass. Traefik's websecure entrypoint accepts the header from
the pod CIDRs, mirroring forwardedHeaders.trustedIPs. Traefik takes
connections with or without a header from a trusted IP, so mesh and LAN
traffic that carries none is unaffected and the Traefik change can land
before the controller image.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
First rollout of #814 sat in CreateContainerConfigError: "container has
runAsNonRoot and image has non-numeric user (controller), cannot verify
user is non-root". The kubelet cannot resolve a named user; nobody/nogroup
in the pod spec is enough, the script needs no home and no writable path.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Deployment for the controller of the previous commit: openbao namespace
like the rotators (its default-deny already allows OpenBao, DNS, the
apiserver and world), own ServiceAccount with a read-only ClusterRole
(Ingresses list/watch, Services get), one replica, Recreate. The image tag
is the branch build that pushed this exact code.
The OpenBao role is applied once by hand (policies/netbird-proxy-services
.hcl); until then the pod logs an auth error per minute and changes nothing.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
security-bot
k.renner
claude-botandClaude Fable 5.1