Compare commits

..
Author SHA1 Message Date
claude-botandClaude Opus 5 ebd9625926 feat(multus): add whereabouts IPAM for segments without a DHCP server
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Groundwork for moving Ceph's cluster network off host networking. Nothing
references whereabouts yet, so this lands inert: no existing attachment
changes, and the DaemonSet only installs a CNI binary that nothing invokes
until a NetworkAttachmentDefinition names it.

The Ceph SAN on VLAN 49 carries Ceph-internal traffic and nothing else, by
design. Neither IPAM already available covers it:

  dhcp        cni-dhcp-daemon works where a DHCP server is on the wire.
              There is none on the SAN and there must not be one.
  host-local  installed by cni-plugins-installer, but it keeps allocation
              state per node with no cluster-wide coordination. On a shared
              L2 segment two nodes hand out the same address.

Whereabouts coordinates through the Kubernetes API — allocations live in the
ippools and overlappingrangeipreservations CRDs. Nothing about assigning an
address touches the segment being addressed, which is the property that makes
it usable on an isolated SAN.

The DaemonSet runs hostNetwork and privileged. That is upstream's shape, and
it matches what multus-daemonset.yaml, cni-dhcp-daemon.yaml and
cni-plugins-installer.yaml already do: the pod writes a CNI binary and its
config into /opt/cni/bin and /etc/cni/net.d, which is node plumbing, and it
has to work before the attachment it serves exists. It is the fourth instance
of an existing exception, not a new one.

The reconciler is split out deliberately. It needs no host access — only the
API, where the allocations are — so it runs as an ordinary pod-network
Deployment. It matters more than it looks: the normal release path is the CNI
DEL when a pod goes away, which does not run when a node is lost hard, and on
a /29 a handful of leaked addresses is the difference between an OSD getting
one and not.

Vendored from upstream v0.9.4 with the image tag pinned — upstream ships
`:latest`. Tag rather than digest, matching the other images in this
directory. The three CRDs are vendored unmodified so a future upstream diff
stays readable. nodeslicepools is included because the ClusterRole references
it, though the node-slice controller itself is not deployed.

Verified: kustomize builds; all nine resources pass a server-side dry run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-26 21:24:52 +02:00
k.renner 6716f7dc50 Merge pull request 'chore(talos): move to v1.12.11 and add i915 to the schematic' (#634) from talos-1-12-11 into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#634
2026-08-26 16:44:36 +00:00
claude-botandClaude Opus 5 d505503616 chore(talos): move to v1.12.11 and add i915 to the schematic
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The nodes run v1.12.6; v1.12.11 is the last patch of the 1.12 series. Upstream
tests migration only between adjacent minors and recommends going to the latest
patch of each intermediate minor, so 1.12.11 is the required stop before 1.13,
not an optional one.

Adds siderolabs/i915 while the image is being rebuilt anyway. Nothing requests
/dev/dri today — every manifest was checked, the only devices any workload asks
for are Ceph RBD and the NVMe disks by-id. It is included for the reason
thunderbolt already is: the extension set is fixed at install and upgrade time,
so adding it later would mean another rolling upgrade of all three nodes rather
than a config change. The MS-01's Alder Lake-P Iris Xe is present in hardware
either way.

New schematic 0f6fb8bd…, identical to a21cec8f… plus i915. Registered with the
Image Factory and the resulting installer image verified to exist for v1.12.11.

The absences are deliberate and now written down, so they are not re-derived as
gaps later: the X710 (i40e) and I226 (igc) NICs need no firmware extension —
intel-ice-firmware covers E810/ice only; mei is a prerequisite for Intel Arc
discrete GPUs and there are none; iscsi-tools, nvme-cli and util-linux-tools
have no consumer, since storage is Ceph RBD and device health comes from Ceph
rather than a node-level exporter.

Both runbooks move to the new schematic and version.

Note for whoever applies this: talosctl upgrade --image does NOT write
.machine.install.image, so this config has to be applied separately or the node
keeps pointing at whatever it was installed with. That is how the running nodes
ended up on ghcr.io/siderolabs/installer:v1.12.6 despite this file naming a
Factory image — which is also why talosctl get extensions is empty on all
three.

Kubernetes is untouched; talosctl upgrade does not move it. It stays on v1.35.2
and is a separate upgrade-k8s step.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-26 18:33:37 +02:00
k.renner 2ba950497d Merge pull request 'feat(home-assistant): relay IoT-VLAN mDNS into HA's zeroconf as unicast' (#633) from mdns-relay into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline failed
Reviewed-on: k.renner/homelab#633
2026-08-26 15:46:41 +00:00
claude-botandClaude Opus 5 664bc72eb4 fix(home-assistant): confine the mDNS relay's receive sockets to net1
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Both sockets bound the wildcard, so they accepted datagrams on eth0 as well as
net1. The namespace CNP grants every pod in home-assistant unrestricted access
to every other pod on every port, so any sibling pod could reach :5353 over the
cluster network and have records forwarded into HA's zeroconf cache. The
header comment described only the IoT-VLAN side of that.

so-bindtodevice=net1 confines delivery to datagrams that actually arrived on
the IoT VLAN interface. Binding to net1's address is still not possible — it is
DHCP-leased and unknown at exec time — but binding to the device gets the same
confinement without needing the address, and needs no capability: verified that
SO_BINDTODEVICE succeeds as uid 65534 with all capabilities dropped. Also
verified it costs no reception: 6 packets in 12s with the option, 6 in 12s
without.

Scope of what this fixes, stated exactly: the relay is not what makes HA's
zeroconf reachable from inside the cluster. HA binds :5353 on its pod address
(v4) and the wildcard (v6) — confirmed in /proc/net/udp and /proc/net/udp6 —
and the namespace CNP already permits any sibling pod to reach it directly.
This removes a second route to that listener, not the first.

Adds ipv6-v6only=1 on the v6 socket. The two containers share the pod's network
namespace, so a dual-stack v6 socket overlaps the v4 container's port. Measured
behaviour without it was no contention — both bound cleanly and the v4 receiver
kept taking packets throughout the v6 container's lifetime — so this is not a
latent bug being fixed, but the outcome depended on the kernel's bindv6only
default rather than on anything the manifest said. The roles are now explicit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-26 17:11:33 +02:00
claude-botandClaude Opus 5 14e45a540c fix(home-assistant): relay mDNS over IPv6 as well, not just IPv4
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The relay joined 224.0.0.251 only. mDNS is two protocols, and a socket joined
to one group hears nothing of the other, so every IPv6 announcement was
dropped — and a device that announces only over v6 would have stayed exactly
as invisible as before the relay existed.

Not hypothetical for this cluster: HA's zeroconf holds ff02::fb on eth0
alongside the v4 group, verified in /proc/net/igmp6. The listener was already
waiting for traffic the relay was never going to send it.

Adds a second socat container for the v6 group. Two processes rather than one
because socat carries a single address pair per invocation and the families
need different receive and send addresses. They fail independently and the
kubelet restarts each on its own.

Joining ff02::fb needs no address configuration on net1. The group is
link-local, so the kernel's automatic link-local address on the macvlan is
enough to receive on it; whether v30iot hands out global v6 addresses does not
affect reception.

The delivery Service becomes dual-stack (PreferDualStack with both families
listed, matching mosquitto-iot in this namespace), so each family is forwarded
over its own transport instead of folding v6 announcements into a v4 hop.
UDP4-SENDTO takes the A record and UDP6-SENDTO the AAAA from the same name.

Verified: the v6 socat address syntax parses and joins; a server-side dry run
allocates both families to the Service; all three manifests still apply.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-26 16:54:43 +02:00
claude-botandClaude Opus 5 4965f53990 feat(home-assistant): relay IoT-VLAN mDNS into HA's zeroconf as unicast
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
HA's zeroconf listener joins 224.0.0.251 on eth0 and hears nothing. The HA
pod has received 0 multicast frames against 1.9M packets total. Two
independent reasons: multicast is link-local, so IoT-VLAN announcements never
reach the pod CIDR; and Cilium runs tunnel mode (VXLAN) with no
`enable-multicast`, so it does not replicate multicast between pods either.

The visible cost is that the ESPHome integration dials each device at the
address cached in its config entry and has no way to learn when that address
changes. A rotated lease produces `[Errno 113] Connect call failed` until
someone repoints the entry by hand. All eight ESPHome config entries were
created by `source=zeroconf` and now carry `discovery_keys: dhcp` — both
paths are inert today, so the addresses are frozen at whatever they were when
last discovered.

An avahi-style reflector cannot fix this. It receives on one interface and
re-emits as multicast on the other, and the second half is exactly what the
CNI drops. The published recipes for this all put avahi on the node, which
Talos does not permit.

So the relay bridges the last hop as unicast, which is the one direction the
CNI carries. It holds the multicast membership on v30iot through its own
macvlan and forwards each datagram to HA's :5353.

This works because python-zeroconf does not filter on how a packet arrived —
verified against _listener.py: datagram_received checks size, deduplicates,
and parses, with no test on destination address, source address, or source
port. Unicast mDNS is RFC 6762's QU mechanism, so this stays inside the
protocol.

Scope of the fix, stated plainly: passive reception only. RFC 6762 requires a
responder to re-announce when its address changes, which is the failure this
addresses, and new devices announcing themselves are seen too. Queries HA
sends still leave on eth0 and die there, so nothing that depends on
soliciting a response is restored.

No macvlan on the HA pod itself — that stays off-table, and the outcome since
it was removed supports keeping it that way. A separate relay was already the
recorded intent; this is that relay.

Shape follows shelly-manager-iot, which is the working precedent for L2
presence on v30iot: macvlan with DHCP IPAM, chained with sbr so the leased
default gateway cannot displace eth0's route. The forward path runs over eth0,
so that route has to survive.

Delivery target is a dedicated ClusterIP rather than the pod. socat resolves
its target once at start and cannot chase pod IPs; a ClusterIP is stable and
Cilium re-DNATs to the current backend. Kept separate from the
`home-assistant` Service, which fronts the nginx sidecar and is the XFF trust
anchor.

Exposure, since it is not zero: socat binds :5353 on all interfaces including
net1, because net1's address is DHCP-leased and unknown at exec time. Any host
on the IoT VLAN can send it a datagram that reaches HA's zeroconf cache — the
same trust already extended to real mDNS from that segment, which any device
there can equally forge. macvlan traffic bypasses Cilium policy, so the
namespace CNP does not constrain that interface. The container runs socat and
nothing else.

No CNP change needed: home-assistant-ns.yaml already allows intra-namespace
traffic on all ports.

Verified before committing: the socat address syntax joins the group and reads
in a loop without forking per datagram (UDP4-RECV, not UDP4-RECVFROM); it runs
as uid 65534 with no privileged operation; the image digest is pinned from the
registry; kustomize builds; all three manifests pass a server-side dry run.

Not yet verified, and only observable once deployed: that HA acts on the
forwarded records. The check is `discovery_keys` flipping back to zeroconf, or
a device surviving a lease change without hand-editing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-26 14:02:15 +02:00
k.renner e7c70ea368 Merge pull request 'fix(ansible): stop rotating netbird_proxy_token — Ansible does not own both ends' (#632) from netbird-rotate-migration into master
ci/woodpecker/cron/e2e Pipeline failed
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#632
2026-08-26 11:04:55 +00:00
k.renner 4187aa91cb Merge pull request 'docs(runbook): record the half-open socket failure after a node move' (#630) from runbook-stale-sockets into master
Reviewed-on: k.renner/homelab#630
2026-08-26 11:04:38 +00:00
claude-botandClaude Opus 5 6ff5cb7d7f fix(ansible): stop rotating netbird_proxy_token — Ansible does not own both ends
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Rotated turn_secret and relay_secret in production; both are correct and stay
rotated. proxy_token was in the same set and should never have been: rotating
it locked every proxy out of the management server.

The misjudgement was specific. proxy_token appears in proxy.env on the server
AND in proxy.env on each proxy, both Ansible templates, which looked exactly
like the symmetry test the other two pass. It is not: the management server
never reads it from a file Ansible writes. The token is registered in the
management DATABASE, so rotating it changed only what the proxies present.

  management: proxy auth failed: rpc error: code = ResourceExhausted
              desc = too many failed authentication attempts   (~75/min, steady)
  proxy:      http: TLS handshake error ... unknown domain "home.renner.uno"

The second follows from the first — a proxy that cannot authenticate never
receives its domain configuration, so it rejects the domains it holds valid
certificates for.

Recovered by restoring the original proxy_token from the pre-rotation vault and
re-applying both roles. Auth failures went to zero and the domain errors
stopped. turn_secret and relay_secret keep their new values; both appear in
management.json.j2, so both ends really are Ansible's.

vault_netbird_proxy_token moves back into vault.yml. It was in
netbird-rotatable.yml, which this play rewrites wholesale — the next run would
have silently dropped it. Verified by re-running both deploys: changed=0.

Adds the test that would have caught this, in the play's own words: a secret is
safe to rotate here only if it appears in management.json.j2. Templating both
sides of a wire is not the same as owning both ends of a trust relationship.

Also corrects the shadow-key guard. vault.yml carried BOTH
vault_netbird_turn_secret and a bare netbird_turn_secret; the role default maps
one to the other, but a group var outranks a role default, so the bare key won
and rotating only the prefixed one would have changed nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-26 12:56:05 +02:00
claude-botandClaude Opus 5 c8b0d1cd4b docs(runbook): record the half-open socket failure after a node move
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Anything holding a long-lived TCP session to a device outside the cluster keeps
a half-open socket when the node under it moves. The peer drops its side; the
pod never learns, because no RST arrives and no keepalive fires.

This cost real time after the conversion. Home Assistant held seven ESTABLISHED
sockets to ESPHome devices that were themselves reporting "not connected". It
logged nothing and retried nothing, because as far as it knew the connections
were fine — so the search went through logs that were empty by construction.
The one device Home Assistant did log was the only one genuinely offline.

Records the check that settles it in one command, reading /proc/net/tcp in the
pod, and notes that reloading the integration is the lighter fix than
restarting the pod.

Placed in "Verify and restore" rather than the known-stale list: it is something
to check before calling a conversion finished, not a trap in the tooling.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-26 12:13:21 +02:00
k.renner 19462a3792 Merge pull request 'chore(ansible): remove monitoring-agent, which shipped to endpoints that are gone' (#629) from drop-monitoring-agent into master
ci/woodpecker/push/deploy-claude-workstation Pipeline failed
ci/woodpecker/cron/issue-fix Pipeline failed
ci/woodpecker/cron/alert-monitor Pipeline failed
Reviewed-on: k.renner/homelab#629
2026-08-26 10:11:13 +00:00
k.renner 75710eab2e Merge pull request 'chore(ansible): remove dead plays and the 26 unreferenced vault keys' (#627) from ansible-drop-dead-plays into master
Reviewed-on: k.renner/homelab#627
2026-08-26 10:10:46 +00:00
claude-botandClaude Opus 5 b76a948e28 chore(ansible): remove monitoring-agent, which shipped to endpoints that are gone
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
common.yml applied monitoring-agent (Alloy) to every inventory host, pointing at
loki.srv.renner.uno:3100 and prometheus.srv.renner.uno:9090. Both moved into the
cluster and neither name resolves any more, so nas-1, claude-ws, netbird and the
proxies have been shipping logs and metrics nowhere.

Repointing them was the obvious fix and the wrong one. The cluster endpoints are
authenticated — loki.renner.uno returns 401 and prometheus.renner.uno sits behind
an Authentik forward-auth gate — while monitoring-agent has no auth support at
all. Making it work would mean per-host credentials, htpasswd entries in the
cluster secret, and some answer for remote-write past the gate. Nobody reads
those metrics: edge-1 is the host that matters, and edge-2 when it exists.

edge hosts are unaffected. install-edge-host.yml is self-contained and
deliberately never carried monitoring-agent; they ship via edge-log-shipper,
which already does this correctly — authenticated push to the cluster ingress
with a per-host credential.

install-netbird-monitoring.yml goes too; applying monitoring-agent to
netbird_server was its only content.

Comments referring to the role are corrected in install-edge-host.yml,
install-claude-workstation.yml, the claude-workstation role and its RUNBOOK.
Three more described the old srv Loki as "firewalled off the IoT VLAN" — it is
not firewalled, it is gone. The reason edge hosts use the authenticated ingress
is unchanged, but the stated reason was no longer true.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-26 12:09:37 +02:00
30 changed files with 909 additions and 665 deletions
+2 -2
View File
@@ -170,8 +170,8 @@ role keeps logs off local flash and monitors wear:
WAL/positions sit on `/run/alloy` (tmpfs), so the agent writes nothing to
the eMMC either. Net effect: no log writes hit local flash — the trade-off
is that logs are RAM-only and lost on reboot unless already shipped.
- **Loki endpoint:** the unauthenticated srv-network Loki
(`loki.srv.renner.uno:3100`) is firewalled off the IoT VLAN, so Alloy pushes
- **Loki endpoint:** the old srv-network Loki
(`loki.srv.renner.uno:3100`) no longer exists — Loki moved into the cluster, so Alloy pushes
to the authenticated cluster ingress `https://loki.renner.uno/loki/api/v1/push`
with basic-auth (`edge_loki_username` / `vault_edge_loki_password`).
**Delivery only works once the `edge-1` htpasswd line has reconciled into
@@ -0,0 +1,13 @@
$ANSIBLE_VAULT;1.1;AES256
32373134633930343261336362373335623735656634313034393731393661643361346634623439
3561363963333533313264643036663731656465343162330a333964383462373934303766323533
63613065313063626261383236396363656565623538666565646661353766643133306334396561
3961376234623236620a616339326238366538333666316538396465666365373834313064653963
38623233616633383435373836613438333763343932636239386263306238376137643531373161
35386437313465643437343832623935353238316565366537646530323233363837656631383962
32653332333162623966366163633732356637323461613966313263313066643966373335313162
30306362346561633566316166313164323635356363643034663933363464663562343136646137
66643536363734316533663032643637356136363237376365613330633961616665323430633135
36323733633464306330666232663433616533373466633939353061663136666665323432373666
36393930333435333238306238373931393232393861333334643139306138653336633534626362
61323038303838396438
+28 -41
View File
@@ -1,42 +1,29 @@
$ANSIBLE_VAULT;1.1;AES256
39333932383736353162346462663539633763663565366639646463316636333364386365363863
3034356438663432363366633537373738363061633764300a393236373834323237623434353361
32346331666234616335663432386163316161616230396265373337613035323863393663333665
6463623335626238640a306666616366363130363534623032353730313865343537373931343433
61636634306565356566303431623533623734343466343636376664646539363363383733666638
39316362393066666435326438323036353735666630306532353433663163303231376565313838
33616137303261366330653862613934303138653634323063363232303636346336303337333537
33643030613662343765303638303339336564633063626462656265646536616563303565323266
36373765663732666436363866623133366337646265616333633536666630396465363438353865
37383130373532303131646530353236383637613034616433383333333234353137633166663866
30366265323537623863336461313666656662383535623435306231363763383364306335623938
34376363633762303664356262383837313866633362663037616335626135376366613336356132
39323166366437386236386237396236356631623133373866663831656330313936363361656465
31373663653736653438643165383338333034613139303333373937626666376432613665656564
61616365663033633438666562363530323862613436663964356339373635356366386264373437
36646138313838666561646331316265383637313237616632643831303835313663363034633862
62623064383539366366386634313938323833323163316365356430633531306134366630623261
35656461306566666531366364323834663737313835666264316132376638333436343136656233
39343338653264636435383736613763633531356438643464633164633766383435323331633630
63316436343863383561653631333434613761343863643465366139353262333463646263633566
38306266393439346265666133343566353164353765366331343631666163613262363135643938
31643538363136376462633431643135373932373932303435303365383063646236656532643263
34663637616535373235336537633339303263383035323264303665643562336431303436653435
37613162353133636361623431356361646433313161316163333766336531363164316334343061
38303263643263386662383731656537393731356663353861316630353464363932346666323037
31343933346530306461363665383235303239373234336631316531353862643937393235643134
32303265306331336665653162383639353634373462666461643463633531373366633438353538
31376437386538613531323231396437363434663236663635666638373137353434373733633234
66363732393166633534316439343561383234636661316439366362326631656134346264623432
35626332646162386236336434393533626435313462303635313439326163636330636133613335
38636664376166613461316466323161386436396235376534393063663532333538336164643730
30636137633663666532666466363730633137396363653832643837366266626461376636333736
33343832326566353264353266376664643061306630633038313534386563616236636438393434
61643030616531363963313337646263613366663462326561393261383732376231383036303633
66633164623232343537316163636666653932646266643561303561323536313532666538643031
61653663613139383234656465393637323031666462663165323138303164313461623832323236
32373866636432653135623330333464626661396465353133373965376436353138313866633138
30376339626165386262396139393737343036333437623365313534636239373830663437643531
63643233663137646335343338663661623431636162306363363839306631356638356462643336
62643861336131616433666265316538346264376236386434343539343839313130633535616236
32336632306165376635313130346436663939303432353966323662326335666564
30353730633631376539373334376531353263396339313464373538396231616166653738383161
6438656132616365646463653464396236313437323537620a363632643061333036353537626461
34393065613836343536383565303066326133363937386263653961663666643430663139653961
6462363235343063350a636165376232366636363233376637636231363839343738346134623562
63663839326435303964313730336436396365343163343366613966323938656263303166326435
65303931393233623064343763633161353161643635396365316362333137353732346366643162
36386132386638623333623839373161386330353635346331393735366662393935353264646439
36386539393432373765396562613033656339646532323864366230393638313161383363636636
32353266376563393239393734316461313566386233333164623135303132346164353165323338
62306333363838393236346435303339393639623231353232643966613163313134336265633636
64366539326639303362393933303465376532626663346431613563353032383037363532646637
38376266323537343437643635343634636164393230313632663533666434306134366261383230
66383135303565383763646331383338646339613539623835366438393635363832366134316531
63636466653239383231616139663231623636376564376234383762383066613030313937616533
33396135616236643034303236353230376434313365326633303961303232363766633161353434
64333832623535343733353462313862343934633363616237353331386264386632643538333939
34636437393934326131363236346636663665613537343134313636623439356665636663613938
37346634356164343537396363663730306232623964636635333333653738383538343236383535
66326462373964333234343435336630643930333433393034303439366631386666393330343238
61336237373236313533333732343064336637326636663433396362346631343165656163313561
30383331326231383539343639343130323731333432306630353835343338613861376631646264
66383539616663376237663234313431313064663566313566366664386439316161353135306338
35373962636537653535333935306564303235356232313463386237353737313736386463396164
62666165336539316565623634326562356434313833643833353635303361353937666663356263
35653665323736386263333437666432623332333936373938376637633463653930326466383139
38336239646135316565356564373731386662383234386433633632393165343331353835623437
30663138633430356334353832633434633939373337663437326263336537356633313763353337
64373764666430643433
-6
View File
@@ -11,12 +11,6 @@
ignore_unreachable: true
gather_facts: true
vars:
# Monitoring agent endpoints
loki_endpoint: "http://loki.srv.renner.uno:3100"
prometheus_remote_write_endpoint: "http://prometheus.srv.renner.uno:9090/api/v1/write"
roles:
- common
- qemu-guest-agent
- monitoring-agent
+1 -1
View File
@@ -10,7 +10,7 @@
- name: Deploy the Claude Code workstation
hosts: claude_workstation
become: true
# qemu-guest-agent + monitoring-agent (Alloy -> Loki/Prometheus) are applied
# qemu-guest-agent is applied
# to every inventory host by the umbrella plays/common.yml, same as for the
# netbird-proxy host — run that against this host too (see RUNBOOK Step 5).
roles:
+4 -3
View File
@@ -18,9 +18,10 @@
#
# Self-contained: `certificates` and `common` are the parts of plays/common.yml
# that apply to an edge host, carried here so bootstrapping one is a single
# unrestricted run over the group. Not carried over: qemu-guest-agent (bare
# metal, not VMs) and monitoring-agent (Debian-oriented; these hosts ship logs
# via edge-log-shipper).
# unrestricted run over the group. Not carried over: qemu-guest-agent, since
# these are bare metal rather than VMs. Log shipping is edge-log-shipper's job
# and lives below — it pushes to the authenticated cluster ingress with a
# per-host credential, which is the only Loki these hosts can reach.
#
# Order is dependency order: edge-flash-wear before docker, so the journal is
# volatile before containers start logging into it.
@@ -1,38 +0,0 @@
---
# Deploy monitoring agent to NetBird server (external datacenter)
# Uses NetBird mesh IPs to reach Loki and Prometheus
# Usage: ansible-playbook -i inventory/ plays/setup-netbird-monitoring.yml
- name: Deploy Monitoring Agent to NetBird Server
hosts: netbird_server
become: yes
gather_facts: yes
vars:
# Use ARM64 binary for Hetzner ARM server
alloy_arch: "linux-arm64"
# Monitoring endpoints via NetBird mesh
loki_endpoint: "http://loki.nb.renner.uno:3100"
prometheus_remote_write_endpoint: "http://prometheus.nb.renner.uno:9090/api/v1/write"
# NetBird server labels
alloy_extra_labels:
environment: "production"
datacenter: "hetzner"
type: "netbird-server"
service: "netbird"
roles:
- monitoring-agent
post_tasks:
- name: Verify Alloy agent is running
systemd:
name: alloy
state: started
register: alloy_status
- name: Display agent status
debug:
msg: "Monitoring agent deployed to {{ inventory_hostname }} via NetBird mesh"
+27 -10
View File
@@ -15,7 +15,14 @@
#
# ansible-vault edit inventory/group_vars/all/vault.yml
# # delete vault_netbird_turn_secret, vault_netbird_relay_secret,
# # vault_netbird_proxy_token
# # and netbird_turn_secret
#
# That last one is easy to miss and breaks the rotation silently. vault.yml
# carried BOTH vault_netbird_turn_secret and a bare netbird_turn_secret. The
# role default is `netbird_turn_secret: "{{ vault_netbird_turn_secret }}"`, but
# a group var outranks a role default — so the bare key wins, and rotating only
# the vault_-prefixed one changes nothing that reaches the template. The check
# below matches both spellings for that reason.
#
# The first rotation then populates netbird-rotatable.yml with fresh values, so
# nothing needs copying across.
@@ -29,10 +36,24 @@
# cannot disagree across a run.
# vault_netbird_relay_secret management.json on the server and relay.env on
# the proxies. Two hosts, both managed here.
# vault_netbird_proxy_token proxy.env on the server and the proxies.
#
# The test for "safe to rotate here" is whether the value appears in
# management.json.j2. If the management server does not read it from a file
# Ansible writes, Ansible does not own both ends, whatever else it templates.
#
# WHAT IS NOT, and must not be added:
#
# vault_netbird_proxy_token LOOKS symmetric — it is in proxy.env on both
# the server and the proxies, all Ansible
# templates — but the management server does not
# validate it from any of them. It is registered
# in the management DATABASE. Rotating it changes
# only what the proxies present, so every proxy
# is locked out: "proxy auth failed: too many
# failed authentication attempts", and the
# proxies stop receiving their domain
# configuration. Learned by doing it.
#
# vault_netbird_datastore_key encrypts NetBird's datastore at rest. Changing
# it without re-encrypting the store makes the
# existing data unreadable. Rotating this needs a
@@ -114,7 +135,7 @@
ansible-vault view
--vault-password-file {{ playbook_dir }}/../vault-password.sh
{{ playbook_dir }}/../inventory/group_vars/all/vault.yml
| grep -cE '^vault_netbird_(turn_secret|relay_secret|proxy_token):'
| grep -cE '^(vault_)?netbird_(turn_secret|relay_secret):'
register: _stale_keys
changed_when: false
# grep exits 1 on "no matches", which is the good case here.
@@ -130,8 +151,8 @@
rotates. group_vars files merge alphabetically and vault.yml sorts
last, so it would override netbird-rotatable.yml on the next deploy
and revert the secrets. Remove vault_netbird_turn_secret,
vault_netbird_relay_secret and vault_netbird_proxy_token from
vault.yml first (one-time migration), then re-run.
netbird_turn_secret and vault_netbird_relay_secret from vault.yml
first (one-time migration), then re-run.
success_msg: vault.yml does not shadow the rotated keys.
tasks:
@@ -150,7 +171,6 @@
ansible.builtin.set_fact:
new_turn_secret: "{{ lookup('ansible.builtin.password', '/dev/null length=48 chars=ascii_letters,digits') }}"
new_relay_secret: "{{ lookup('ansible.builtin.password', '/dev/null length=48 chars=ascii_letters,digits') }}"
new_proxy_token: "{{ lookup('ansible.builtin.password', '/dev/null length=48 chars=ascii_letters,digits') }}"
no_log: true
- name: Render the new values to a temporary file
@@ -163,7 +183,6 @@
# the next rotation overwrites this file wholesale.
vault_netbird_turn_secret: "{{ new_turn_secret }}"
vault_netbird_relay_secret: "{{ new_relay_secret }}"
vault_netbird_proxy_token: "{{ new_proxy_token }}"
no_log: true
# Encrypting the whole file rather than inline !vault strings: these three
@@ -198,7 +217,6 @@
vars:
netbird_turn_secret: "{{ hostvars['localhost']['new_turn_secret'] }}"
netbird_relay_secret: "{{ hostvars['localhost']['new_relay_secret'] }}"
netbird_proxy_token: "{{ hostvars['localhost']['new_proxy_token'] }}"
roles:
- netbird-server
@@ -213,7 +231,6 @@
vars:
netbird_relay_secret: "{{ hostvars['localhost']['new_relay_secret'] }}"
netbird_proxy_token: "{{ hostvars['localhost']['new_proxy_token'] }}"
roles:
- netbird-proxy
@@ -227,7 +244,7 @@
- name: Remaining manual steps
ansible.builtin.debug:
msg:
- "Rotated: turn_secret, relay_secret, proxy_token."
- "Rotated: turn_secret and relay_secret."
- "Commit inventory/group_vars/all/netbird-rotatable.yml — it is encrypted."
- "NOT rotated: datastore_key (needs dump/restore), auth_client_secret (Authentik side)."
- "OpenBao is unaffected by design — its NetBird credentials are separate ones."
+1 -2
View File
@@ -169,8 +169,7 @@ Merge the PR, then trigger the pipeline (Woodpecker → Run → `PIPELINE ==
"deploy-claude-workstation"`), or push a change under the role's paths. CI SSHes
to `10.3.40.60` over the tunnel and applies `common`, `netbird-client`, and this
role. Also run the umbrella `plays/common.yml` against the new host once, so it
picks up `qemu-guest-agent` and `monitoring-agent` (Alloy → Loki/Prometheus)
like every other inventory host.
picks up `qemu-guest-agent` like every other inventory host.
---
@@ -3,8 +3,8 @@
# Debian 12 VM on the off-site nas-2 (Synology VMM). See README for the manual
# bootstrap (VM creation, Netbird route, secret material) that precedes this.
#
# common.yml already applies `common`, `qemu-guest-agent` and `monitoring-agent`
# (Alloy -> Loki/Prometheus) to every inventory host, so this role owns only the
# common.yml already applies `common` and `qemu-guest-agent`
# to every inventory host, so this role owns only the
# Claude-specific surface: toolchain, the two credential-isolated users, the
# read-only creds, notes sync, the unattended timers, and egress lockdown.
@@ -1,8 +1,8 @@
---
# Journal -> Loki shipping.
#
# The srv-network Loki (loki.srv.renner.uno:3100, no auth) is firewalled off
# the IoT VLAN; the only endpoint these hosts can reach is the authenticated
# The srv-network Loki (loki.srv.renner.uno:3100) no longer exists — it moved
# into the cluster. The only endpoint these hosts can reach is the authenticated
# cluster ingress. edge_loki_password comes from the vault.
edge_alloy_config_file: /etc/alloy/edge.alloy
@@ -7,8 +7,8 @@
// Loki. Its own WAL/positions live on /run (tmpfs, see edge-alloy.service),
// so this agent writes nothing to the eMMC either.
//
// The srv-network Loki (loki.srv.renner.uno:3100, unauthenticated) is not
// reachable from the IoT VLAN; the only reachable endpoint is the cluster
// The srv-network Loki (loki.srv.renner.uno:3100) no longer exists — it moved
// into the cluster. The only reachable endpoint is the cluster
// ingress, which enforces Traefik basic-auth — hence the credential below.
loki.relabel "journal" {
@@ -1,59 +0,0 @@
---
# Monitoring Agent Configuration (Grafana Alloy)
# Enable/disable agent deployment
monitoring_agent_enabled: true
# Version and download
alloy_version: "1.13.2"
alloy_arch: "linux-amd64"
alloy_download_url: "https://github.com/grafana/alloy/releases/download/v{{ alloy_version }}/alloy-{{ alloy_arch }}.zip"
alloy_bin_path: "/usr/local/bin/alloy"
# Directories
alloy_config_dir: "/etc/alloy"
alloy_config_file: "{{ alloy_config_dir }}/config.alloy"
alloy_data_dir: "/var/lib/alloy"
# Endpoints - MUST be configured in group_vars
loki_endpoint: "http://loki.srv.renner.uno:3100"
prometheus_remote_write_endpoint: "http://prometheus.srv.renner.uno:9090/api/v1/write"
# Collection settings
alloy_collect_journal: true
alloy_collect_syslog: true
alloy_collect_node_metrics: true
alloy_collect_docker: "{{ docker_socket_exists | default(false) }}"
# Docker container monitoring (cAdvisor)
# Set to true on hosts that should have cAdvisor deployed
docker_monitoring_enabled: false
# Scrape interval
alloy_scrape_interval: "30s"
# Labels applied to all metrics/logs
alloy_extra_labels:
environment: "production"
datacenter: "homelab"
# Systemd service
alloy_user: "alloy"
alloy_group: "alloy"
# Agent port for self-metrics
alloy_http_listen_port: 12345
# Extra scrape targets for custom exporters
# Format: list of {name: "...", address: "host:port", job: "..."}
alloy_extra_scrape_targets: []
# Syslog receiver (for receiving logs from remote devices like Synology NAS)
# Enable this on the central log collection host (e.g., loki.srv.renner.uno)
alloy_syslog_receiver_enabled: false
alloy_syslog_receiver_port: 1514 # Use non-privileged port, Synology can specify port
alloy_syslog_receiver_protocol: "udp" # udp or tcp
alloy_syslog_receiver_format: "rfc5424" # rfc3164, rfc5424, or raw (requires experimental)
# Stability level for experimental features (e.g., raw syslog format)
alloy_stability_level: "generally-available" # generally-available, public-preview, experimental
@@ -1,17 +0,0 @@
---
- name: Restart alloy
systemd:
name: alloy
state: restarted
daemon_reload: yes
when: monitoring_agent_enabled | bool
- name: Reload systemd
systemd:
daemon_reload: yes
- name: Restart cadvisor
systemd:
name: cadvisor
state: restarted
daemon_reload: yes
@@ -1,170 +0,0 @@
---
# Monitoring Agent (Grafana Alloy) Installation and Configuration
# This role is designed to be included in common.yml for all hosts
- name: Skip if monitoring agent is disabled
meta: end_host
when: not (monitoring_agent_enabled | bool)
- name: Install dependencies
apt:
name:
- unzip
- curl
state: present
update_cache: yes
when: ansible_facts['os_family'] == 'Debian'
- name: Check if Docker socket exists
stat:
path: /var/run/docker.sock
register: docker_socket
- name: Set Docker collection fact
set_fact:
alloy_collect_docker: "{{ docker_socket.stat.exists }}"
- name: Deploy cAdvisor container for Docker metrics
when: docker_monitoring_enabled | bool and docker_socket.stat.exists
block:
- name: Create cAdvisor systemd service
copy:
dest: /etc/systemd/system/cadvisor.service
mode: '0644'
content: |
[Unit]
Description=cAdvisor Container Monitoring
After=docker.service
Requires=docker.service
[Service]
Type=simple
Restart=always
RestartSec=10
ExecStartPre=-/usr/bin/docker rm -f cadvisor
ExecStart=/usr/bin/docker run --rm \
--name=cadvisor \
--privileged \
--device=/dev/kmsg \
--volume=/:/rootfs:ro \
--volume=/var/run:/var/run:ro \
--volume=/sys:/sys:ro \
--volume=/var/lib/docker/:/var/lib/docker:ro \
--volume=/dev/disk/:/dev/disk:ro \
--volume=/cgroup:/cgroup:ro \
--publish=127.0.0.1:8080:8080 \
gcr.io/cadvisor/cadvisor:v0.49.1 \
--store_container_labels=true
ExecStop=/usr/bin/docker stop cadvisor
[Install]
WantedBy=multi-user.target
notify:
- Reload systemd
- Restart cadvisor
- name: Enable and start cAdvisor service
systemd:
name: cadvisor
state: started
enabled: yes
daemon_reload: yes
- name: Create alloy user
user:
name: "{{ alloy_user }}"
system: yes
shell: /usr/sbin/nologin
create_home: no
groups: "{{ 'docker' if alloy_collect_docker else omit }}"
append: yes
- name: Create directories
file:
path: "{{ item }}"
state: directory
owner: "{{ alloy_user }}"
group: "{{ alloy_group }}"
mode: '0755'
loop:
- "{{ alloy_config_dir }}"
- "{{ alloy_data_dir }}"
- name: Check if Alloy is already installed
stat:
path: "{{ alloy_bin_path }}"
register: alloy_binary
- name: Get installed Alloy version
command: "{{ alloy_bin_path }} --version"
register: installed_version
changed_when: false
failed_when: false
when: alloy_binary.stat.exists
- name: Download Alloy
get_url:
url: "{{ alloy_download_url }}"
dest: "/tmp/alloy.zip"
mode: '0644'
when: not alloy_binary.stat.exists or (installed_version.stdout is defined and alloy_version not in installed_version.stdout)
- name: Unzip Alloy
unarchive:
src: "/tmp/alloy.zip"
dest: "/tmp"
remote_src: yes
when: not alloy_binary.stat.exists or (installed_version.stdout is defined and alloy_version not in installed_version.stdout)
- name: Find Alloy binary in extracted files
find:
paths: "/tmp"
patterns: "alloy-{{ alloy_arch }}"
file_type: file
register: alloy_extracted_binary
when: not alloy_binary.stat.exists or (installed_version.stdout is defined and alloy_version not in installed_version.stdout)
- name: Install Alloy binary from extracted file
copy:
src: "{{ alloy_extracted_binary.files[0].path }}"
dest: "{{ alloy_bin_path }}"
mode: '0755'
remote_src: yes
when:
- not alloy_binary.stat.exists or (installed_version.stdout is defined and alloy_version not in installed_version.stdout)
- alloy_extracted_binary.files is defined and alloy_extracted_binary.files | length > 0
notify: Restart alloy
- name: Create Alloy configuration
template:
src: config.alloy.j2
dest: "{{ alloy_config_file }}"
owner: "{{ alloy_user }}"
group: "{{ alloy_group }}"
mode: '0644'
notify: Restart alloy
- name: Create Alloy systemd service
template:
src: alloy.service.j2
dest: /etc/systemd/system/alloy.service
mode: '0644'
notify:
- Reload systemd
- Restart alloy
- name: Enable and start Alloy service
systemd:
name: alloy
state: started
enabled: yes
daemon_reload: yes
- name: Clean up downloaded files
file:
path: "{{ item }}"
state: absent
loop:
- "/tmp/alloy.zip"
- "/tmp/alloy-{{ alloy_arch }}"
ignore_errors: yes
@@ -1,37 +0,0 @@
[Unit]
Description=Grafana Alloy Monitoring Agent
Documentation=https://grafana.com/docs/alloy/latest/
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User={{ alloy_user }}
Group={{ alloy_group }}
ExecStart={{ alloy_bin_path }} run \
--server.http.listen-addr=0.0.0.0:{{ alloy_http_listen_port }} \
--storage.path={{ alloy_data_dir }} \
{% if alloy_stability_level is defined and alloy_stability_level != "generally-available" %}
--stability.level={{ alloy_stability_level }} \
{% endif %}
{{ alloy_config_file }}
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=10s
LimitNOFILE=65536
{% if ansible_virtualization_type is defined and ansible_virtualization_type != 'lxc' %}
# Security hardening (disabled in LXC containers due to namespace restrictions)
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths={{ alloy_data_dir }} /var/log
ReadOnlyPaths=/var/run/docker.sock
PrivateTmp=yes
{% endif %}
# Allow reading journal
SupplementaryGroups=systemd-journal{% if alloy_collect_docker %} docker{% endif %}
[Install]
WantedBy=multi-user.target
@@ -1,266 +0,0 @@
// Grafana Alloy Configuration
// Host: {{ inventory_hostname }}
// Generated by Ansible
// =====================================================
// Logging Configuration (Loki)
// =====================================================
{% if alloy_collect_journal %}
// Collect systemd journal logs
loki.source.journal "journal" {
forward_to = [loki.write.default.receiver]
labels = {
job = "systemd-journal",
host = "{{ inventory_hostname }}",
{% for key, value in alloy_extra_labels.items() %}
{{ key }} = "{{ value }}",
{% endfor %}
}
relabel_rules = loki.relabel.journal.rules
}
loki.relabel "journal" {
forward_to = []
rule {
source_labels = ["__journal__systemd_unit"]
target_label = "unit"
}
rule {
source_labels = ["__journal_priority_keyword"]
target_label = "level"
}
rule {
source_labels = ["__journal__hostname"]
target_label = "hostname"
}
}
{% endif %}
{% if alloy_collect_syslog %}
// Collect syslog files
local.file_match "syslog" {
path_targets = [
{__path__ = "/var/log/syslog", job = "syslog"},
{__path__ = "/var/log/auth.log", job = "auth"},
{__path__ = "/var/log/kern.log", job = "kernel"},
]
}
loki.source.file "syslog" {
targets = local.file_match.syslog.targets
forward_to = [loki.write.default.receiver]
tail_from_end = true
}
{% endif %}
{% if alloy_collect_docker %}
// Docker container logs
discovery.docker "containers" {
host = "unix:///var/run/docker.sock"
}
discovery.relabel "docker" {
targets = discovery.docker.containers.targets
rule {
source_labels = ["__meta_docker_container_name"]
target_label = "container"
regex = "/(.*)"
}
rule {
source_labels = ["__meta_docker_container_label_com_docker_compose_service"]
target_label = "service"
}
rule {
source_labels = ["__meta_docker_container_label_com_docker_compose_project"]
target_label = "project"
}
}
loki.source.docker "docker" {
host = "unix:///var/run/docker.sock"
targets = discovery.relabel.docker.output
forward_to = [loki.process.docker.receiver]
labels = {
job = "docker",
host = "{{ inventory_hostname }}",
{% for key, value in alloy_extra_labels.items() %}
{{ key }} = "{{ value }}",
{% endfor %}
}
}
loki.process "docker" {
forward_to = [loki.write.default.receiver]
stage.json {
expressions = {
log = "log",
stream = "stream",
time = "time",
}
}
stage.labels {
values = {
stream = "",
}
}
stage.output {
source = "log"
}
}
{% endif %}
{% if alloy_syslog_receiver_enabled %}
// Syslog receiver for remote devices (Synology NAS, network devices, etc.)
loki.source.syslog "syslog_receiver" {
listener {
address = "0.0.0.0:{{ alloy_syslog_receiver_port }}"
protocol = "{{ alloy_syslog_receiver_protocol }}"
syslog_format = "{{ alloy_syslog_receiver_format }}"
labels = {
job = "syslog-remote",
}
}
{% if alloy_syslog_receiver_format == "raw" %}
forward_to = [loki.process.syslog_receiver.receiver]
}
// Parse raw syslog messages (BSD/RFC 3164 format)
// Synology's "IETF" format actually sends: <priority>MMM DD HH:MM:SS hostname app[pid]: message
// Example: <14>Mar 1 09:02:47 nas-1 Connection: User [pve] from...
loki.process "syslog_receiver" {
forward_to = [loki.write.default.receiver]
stage.regex {
expression = "^\\w+\\s+\\d+\\s+\\d+:\\d+:\\d+\\s+(?P<hostname>\\S+)\\s+(?P<app>[^:\\[]+)(?:\\[\\d+\\])?:\\s*(?P<message>.*)$"
}
stage.labels {
values = {
hostname = "",
app = "",
}
}
stage.output {
source = "message"
}
}
{% else %}
forward_to = [loki.write.default.receiver]
}
{% endif %}
{% endif %}
// Write logs to Loki
loki.write "default" {
endpoint {
url = "{{ loki_endpoint }}/loki/api/v1/push"
}
external_labels = {
host = "{{ inventory_hostname }}",
{% for key, value in alloy_extra_labels.items() %}
{{ key }} = "{{ value }}",
{% endfor %}
}
}
// =====================================================
// Metrics Configuration (Prometheus)
// =====================================================
{% if alloy_collect_node_metrics %}
// Node metrics (CPU, memory, disk, network)
prometheus.exporter.unix "node" {
// Using default settings for comprehensive node metrics
}
prometheus.scrape "node" {
targets = prometheus.exporter.unix.node.targets
forward_to = [prometheus.remote_write.default.receiver]
scrape_interval = "{{ alloy_scrape_interval }}"
job_name = "node"
}
{% endif %}
{% if alloy_collect_docker and docker_monitoring_enabled %}
// Docker/cAdvisor metrics - scrape from cAdvisor container
prometheus.scrape "cadvisor" {
targets = [
{"__address__" = "127.0.0.1:8080"},
]
forward_to = [prometheus.relabel.cadvisor.receiver]
scrape_interval = "{{ alloy_scrape_interval }}"
job_name = "cadvisor"
}
// Relabel cAdvisor metrics to add host label
prometheus.relabel "cadvisor" {
forward_to = [prometheus.remote_write.default.receiver]
rule {
target_label = "host"
replacement = "{{ inventory_hostname }}"
}
}
{% endif %}
// Self-monitoring metrics
prometheus.exporter.self "alloy" {}
prometheus.scrape "alloy_self" {
targets = prometheus.exporter.self.alloy.targets
forward_to = [prometheus.remote_write.default.receiver]
scrape_interval = "{{ alloy_scrape_interval }}"
job_name = "alloy"
}
{% if alloy_extra_scrape_targets | length > 0 %}
// Extra scrape targets (custom exporters)
{% for target in alloy_extra_scrape_targets %}
prometheus.scrape "{{ target.name }}" {
targets = [
{"__address__" = "{{ target.address }}"},
]
forward_to = [prometheus.relabel.extra_{{ target.name }}.receiver]
scrape_interval = "{{ alloy_scrape_interval }}"
job_name = "{{ target.job }}"
}
prometheus.relabel "extra_{{ target.name }}" {
forward_to = [prometheus.remote_write.default.receiver]
rule {
target_label = "host"
replacement = "{{ inventory_hostname }}"
}
}
{% endfor %}
{% endif %}
// Remote write to Prometheus
prometheus.remote_write "default" {
endpoint {
url = "{{ prometheus_remote_write_endpoint }}"
}
external_labels = {
host = "{{ inventory_hostname }}",
{% for key, value in alloy_extra_labels.items() %}
{{ key }} = "{{ value }}",
{% endfor %}
}
}
// =====================================================
// Agent HTTP Server (for health checks and metrics)
// =====================================================
// Agent serves metrics on port {{ alloy_http_listen_port }}
+30 -2
View File
@@ -31,8 +31,8 @@ all in state that outlived the VM.
two hosts every PG is undersized with min_size 2, so one further OSD loss
blocks IO on every CephFS mount cluster-wide.
- [ ] Talos metal ISO on a USB stick. Schematic
`a21cec8f35ae32ab5e9a452e7b13500ca8776053930b8e182b9cb23eed186b64`
(`net.ifnames=1`, intel-ucode, thunderbolt; no qemu-guest-agent).
`0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9`
(`net.ifnames=1`, intel-ucode, thunderbolt, i915; no qemu-guest-agent).
Verify by comparing partition offsets and sizes against the ISO with
`fdisk -l` — blkid labels lie after a rewrite.
- [ ] Physical access. Secure Boot must be turned off (see below), and that is
@@ -383,6 +383,34 @@ raised to `high_recovery_ops` to speed the evacuation.
The retired Crucial leaves a stale entry in `ceph device ls`. Remove it with
`ceph config-key rm device/<devid>`; the mgr's devicehealth view may lag.
### Long-lived TCP sessions to devices outside the cluster die silently
Check this before declaring the conversion done. Anything holding a persistent
connection to a device on another VLAN — the ESPHome integration is the case
that bit us — keeps a half-open socket when the node underneath it moves. The
peer drops its side; the pod's side never learns, because no RST arrives and no
keepalive fires.
The symptom is that there is no symptom. Home Assistant logged nothing, retried
nothing, and reported the devices as connected. The devices themselves showed
"not connected". Hours went into reading logs that were empty by construction.
`/proc/net/tcp` in the pod settles it in one command — `01` is ESTABLISHED,
`02` is SYN_SENT, and the port is hex:
```sh
kubectl -n home-assistant exec <ha-pod> -c home-assistant -- \
sh -c 'grep ":17A5" /proc/net/tcp' # 17A5 = 6053, the ESPHome API
```
Seven `01` entries to devices insisting they were disconnected is the tell. A
`02` alongside them is a genuinely dead device, not a stale socket — that one
Home Assistant *was* retrying and *did* log.
Restarting the consuming pod clears it. Reloading just the integration from the
UI does the same with less disruption. Neither is discoverable from logs, which
is why it belongs here.
## Known-stale things that will bite
- `bootstrap.sh` cannot generate a config at all right now (issue #581), so a
+1 -1
View File
@@ -87,7 +87,7 @@ all three nodes identically from there, since `apply-config --insecure` does
not care whether the target is a VM or bare metal.
```
https://factory.talos.dev/image/a21cec8f35ae32ab5e9a452e7b13500ca8776053930b8e182b9cb23eed186b64/v1.12.6/metal-amd64.iso
https://factory.talos.dev/image/0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9/v1.12.11/metal-amd64.iso
```
That schematic is deliberately not the one `.woodpecker/deploy-infrastructure.yml`
@@ -37,6 +37,13 @@ resources:
- esphome-service.yaml
- esphome-gate-service.yaml
- esphome-ingress.yaml
# mDNS relay — carries IoT-VLAN announcements into HA's zeroconf listener as
# unicast, which is the only direction the CNI will move them. Without it HA
# never learns that a device's DHCP lease moved. See the header comment in
# mdns-relay-deployment.yaml.
- mdns-relay-network-attachment.yaml
- mdns-relay-service.yaml
- mdns-relay-deployment.yaml
# Zigbee2MQTT — ZHA replacement bridge. Ships scaled to 0; the cutover
# runbook (zigbee2mqtt-migration-runbook.md) scales it up. The MQTT
# `zigbee2mqtt` broker user is provisioned by mosquitto-bootstrap.yaml
@@ -0,0 +1,190 @@
---
# mDNS relay: gives Home Assistant sight of the IoT VLAN's mDNS announcements
# without putting a macvlan on the HA pod itself.
#
# THE PROBLEM
#
# HA's zeroconf listener joins both mDNS groups on eth0 — 224.0.0.251 and
# ff02::fb — and never hears anything on either.
# Multicast is link-local, so IoT-VLAN announcements do not reach the pod
# CIDR; and Cilium runs in tunnel mode (VXLAN) without `enable-multicast`, so
# it does not replicate multicast between pods either. Both halves of the path
# are dead, which is why an avahi-style reflector — receive on one interface,
# re-emit as multicast on the other — cannot work here: the re-emitted packets
# would be dropped by the CNI.
#
# Consequence for HA: the ESPHome integration dials devices at the address
# cached in its config entry. When a lease rotates, every attempt fails with
# `[Errno 113] Connect call failed` and the device stays unreachable, because
# the mDNS announcement that would carry the new address never arrives. New
# devices are never discovered at all.
#
# THE MECHANISM
#
# This relay holds the multicast membership on the IoT VLAN through its own
# macvlan (mdns-relay-network-attachment.yaml) and forwards each datagram it
# receives to HA as ordinary UNICAST UDP. That is the one direction of travel
# the CNI will carry.
#
# It works because python-zeroconf does not filter on how a packet arrived:
# its datagram handler validates size, deduplicates, and parses, but performs
# no check on destination address, source address, or source port. A unicast
# datagram delivered to :5353 is processed exactly like a multicast one. (mDNS
# unicast responses are part of RFC 6762 — the QU bit — so this is within the
# protocol, not a trick.)
#
# WHAT THIS DOES AND DOES NOT RESTORE
#
# Restored: passive reception. RFC 6762 requires a responder to re-announce
# when its address changes, so a device that moves to a new lease announces
# the move, HA sees it, and the ESPHome integration updates the config entry.
# That is the failure this exists to fix. New devices announcing themselves
# are likewise seen.
#
# NOT restored: HA-initiated browsing. Queries HA sends leave on eth0 as
# multicast and die there, so HA cannot actively solicit a response. Anything
# depending on a query rather than an announcement stays broken. Making that
# work needs a querier on the VLAN side, which this relay deliberately is not.
#
# EXPOSURE
#
# The receive sockets carry so-bindtodevice=net1, so only datagrams that
# actually arrived on the IoT VLAN interface are delivered. Binding to net1's
# address instead is not possible — it is DHCP-leased and unknown at exec time
# — but binding to the device achieves the same confinement without needing to
# know the address, and needs no capability: SO_BINDTODEVICE succeeds as an
# unprivileged uid here.
#
# Without it socat would bind the wildcard, and since the namespace CNP grants
# every pod in home-assistant unrestricted access to every other pod on every
# port, any sibling pod could reach :5353 over the cluster network and have
# records forwarded into HA's zeroconf cache.
#
# What remains, and cannot be closed here: any host on the IoT VLAN can send
# this port a datagram. That is the same trust already extended to genuine
# mDNS from that segment, which any device there can equally forge — the relay
# widens who can reach HA's listener, not what can be claimed. macvlan traffic
# bypasses Cilium policy entirely, so the namespace CNP does not constrain
# this interface either way.
#
# Note that the relay is not what makes HA's zeroconf reachable from inside
# the cluster: HA binds :5353 on the pod address (v4) and the wildcard (v6),
# and the namespace CNP already permits any sibling pod to reach it directly.
# Confining the relay removes a second route to that listener; it does not
# remove the first.
#
# Nothing else is exposed: the containers run socat and only socat, with no
# second listener to reach.
apiVersion: apps/v1
kind: Deployment
metadata:
name: mdns-relay
namespace: home-assistant
labels:
app.kubernetes.io/name: mdns-relay
spec:
# One replica. Extra replicas would deliver a duplicate of every announcement
# — harmless, since zeroconf deduplicates, but they buy nothing: this is a
# cache-refresh path, and a missed window is corrected by the next
# announcement.
replicas: 1
# Recreate, not RollingUpdate: two relays would briefly hold two DHCP leases
# on v30iot for one workload.
strategy:
type: Recreate
selector:
matchLabels:
app.kubernetes.io/name: mdns-relay
template:
metadata:
labels:
app.kubernetes.io/name: mdns-relay
annotations:
k8s.v1.cni.cncf.io/networks: mdns-relay-iot
spec:
enableServiceLinks: false
securityContext:
runAsNonRoot: true
runAsUser: 65534
runAsGroup: 65534
seccompProfile:
type: RuntimeDefault
# One container per address family. mDNS is two protocols — 224.0.0.251
# and ff02::fb — and a socket joined to one hears nothing of the other.
# HA listens on both (its zeroconf holds both groups on eth0), so a
# v4-only relay would silently drop every IPv6 announcement, and devices
# that announce only over v6 would stay invisible exactly as before.
#
# Two processes rather than one: socat carries a single address pair per
# invocation, and the families need different receive and send addresses.
# They are independent — one crashing does not blind the other — and each
# is restarted by the kubelet on its own.
containers:
# -u is load-bearing on both: unidirectional, left to right. Without it
# socat would also carry HA's replies back out, and this must stay a
# one-way tap.
#
# UDP{4,6}-RECV (not -RECVFROM) keeps reading from one socket instead of
# forking per datagram — the IoT VLAN's mDNS chatter is continuous, and
# a fork per packet would be a process storm.
#
# The group is joined on net1 specifically, so the relay subscribes on
# the IoT VLAN rather than on the pod network.
#
# Both send targets are the same ClusterIP name, resolved once at start;
# UDP4-SENDTO takes its A record and UDP6-SENDTO its AAAA. See
# mdns-relay-service.yaml for why that indirection is required and why
# the Service is dual-stack.
- name: socat-ipv4
image: alpine/socat:1.8.0.3@sha256:beb4a68d9e4fe6b0f21ea774a0fde6c31f580dde6368939ed70100c5385b015e
imagePullPolicy: IfNotPresent
args:
- -u
- "UDP4-RECV:5353,ip-add-membership=224.0.0.251:net1,so-bindtodevice=net1,reuseaddr"
- "UDP4-SENDTO:home-assistant-mdns.home-assistant.svc.cluster.local:5353"
# No probes. socat is a single process with no health surface: if it
# exits, the container exits and the kubelet restarts it, which is the
# entire liveness story. A probe here could only re-assert that PID 1
# is running.
resources:
requests:
cpu: 10m
memory: 16Mi
limits:
memory: 64Mi
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
# Joining ff02::fb needs no address configuration on net1: the group is
# link-local and the kernel's automatic link-local address on the
# macvlan is sufficient to receive on it. Whether v30iot hands out
# global v6 addresses does not affect reception here.
#
# ipv6-v6only=1 keeps this socket to v6 only. Both containers share the
# pod's network namespace, and a dual-stack v6 socket would overlap the
# v4 container's port. Measured behaviour without it was no contention —
# the v4 receiver kept taking packets while the v6 one ran — but that
# depends on the kernel's bindv6only default rather than on anything
# stated here, so the roles are made explicit instead of inherited.
- name: socat-ipv6
image: alpine/socat:1.8.0.3@sha256:beb4a68d9e4fe6b0f21ea774a0fde6c31f580dde6368939ed70100c5385b015e
imagePullPolicy: IfNotPresent
args:
- -u
- "UDP6-RECV:5353,ipv6-join-group=[ff02::fb]:net1,so-bindtodevice=net1,ipv6-v6only=1,reuseaddr"
- "UDP6-SENDTO:home-assistant-mdns.home-assistant.svc.cluster.local:5353"
resources:
requests:
cpu: 10m
memory: 16Mi
limits:
memory: 64Mi
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
@@ -0,0 +1,52 @@
---
# mdns-relay's second NIC: macvlan onto v30iot, so the relay has real L2
# presence on the IoT VLAN and can join both mDNS multicast groups
# (224.0.0.251 and ff02::fb, port 5353). The v6 group needs no address
# configuration of its own: it is link-local, and the kernel's automatic
# link-local address on the macvlan is enough to receive on it, independently
# of whether v30iot hands out global v6 addresses.
# Multicast is link-local — it does not cross the L3
# boundary between the cluster pod CIDR and v30iot — so that membership has to
# be held by a pod sitting physically on that segment.
#
# The address is leased from the UniFi DHCP server on v30iot. Nothing
# references it: the relay only receives on this interface and sends onward
# over eth0. See mdns-relay-deployment.yaml for what the relay does with it.
#
# Multus's `master` is an interface name. On the k8s-{1,2,3} nodes the v30iot
# NIC is `ens19` — a stable, busPath-derived predictable name (net.ifnames=1,
# via the Image Factory schematic), identical across nodes and independent of
# driver probe order. This is the same master shelly-manager-iot uses.
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
name: mdns-relay-iot
namespace: home-assistant
spec:
# Chained CNI, identical in shape to shelly-manager-iot: macvlan leases the
# v30iot address by DHCP, then `sbr` confines net1's routes to a
# source-based routing table. Without sbr the DHCP-supplied default gateway
# (10.2.30.1, option 3) installs a second default route at the same metric
# as eth0's, and cluster-internal traffic egresses the IoT VLAN and is
# dropped. The relay's forward path to Home Assistant runs over eth0, so
# that route has to stay intact.
#
# Requires the cni-dhcp-daemon DaemonSet in infrastructure/multus.
config: |
{
"cniVersion": "0.3.1",
"name": "mdns-relay-iot",
"plugins": [
{
"type": "macvlan",
"master": "ens19",
"mode": "bridge",
"ipam": {
"type": "dhcp"
}
},
{
"type": "sbr"
}
]
}
@@ -0,0 +1,42 @@
---
# Stable delivery target for the relay's forwarded mDNS traffic.
#
# The relay resolves its send target once, at process start (socat has no
# re-resolution loop), so it cannot chase the HA pod IP across restarts. A
# ClusterIP is the fix: the IP is stable for the lifetime of the Service, and
# Cilium re-DNATs to whatever pod currently backs it.
#
# Deliberately a separate Service from `home-assistant`, which targets the
# nginx proxy sidecar on 8124 and is the XFF trust anchor for the public UI.
# This one bypasses the sidecar and addresses the home-assistant container's
# own port directly — containers in a pod share a network namespace, so
# targetPort 5353 lands on python-zeroconf's listener regardless of which
# container opened it. Keeping the two apart means the UI path and the
# discovery path cannot be changed by accident through one another.
#
# No CiliumNetworkPolicy change is needed: home-assistant-ns.yaml already
# permits intra-namespace traffic on all ports, and the relay is in this
# namespace.
apiVersion: v1
kind: Service
metadata:
name: home-assistant-mdns
namespace: home-assistant
labels:
app.kubernetes.io/name: home-assistant
spec:
# Dual-stack, because mDNS is two protocols and HA listens on both: its
# zeroconf socket holds 224.0.0.251 on eth0 and ff02::fb alongside it. The
# relay forwards each family over its own transport rather than folding v6
# announcements into a v4 hop, so the Service has to answer on both.
ipFamilyPolicy: PreferDualStack
ipFamilies:
- IPv4
- IPv6
selector:
app.kubernetes.io/name: home-assistant
ports:
- name: mdns
port: 5353
protocol: UDP
targetPort: 5353
@@ -4,3 +4,12 @@ resources:
- cni-plugins-installer.yaml
- cni-dhcp-daemon.yaml
- multus-daemonset.yaml
# Whereabouts — cluster-coordinated IPAM, for attachments on segments with
# no DHCP server. CRDs first: the DaemonSet's control loop reads them at
# startup. See the header in whereabouts-daemonset.yaml for why neither
# dhcp nor host-local covers the Ceph SAN.
- whereabouts-crd-ippools.yaml
- whereabouts-crd-overlappingrangeipreservations.yaml
- whereabouts-crd-nodeslicepools.yaml
- whereabouts-daemonset.yaml
- whereabouts-reconciler.yaml
@@ -0,0 +1,71 @@
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
controller-gen.kubebuilder.io/version: v0.14.0
name: ippools.whereabouts.cni.cncf.io
spec:
group: whereabouts.cni.cncf.io
names:
kind: IPPool
listKind: IPPoolList
plural: ippools
singular: ippool
scope: Namespaced
versions:
- name: v1alpha1
schema:
openAPIV3Schema:
description: IPPool is the Schema for the ippools API
properties:
apiVersion:
description: |-
APIVersion defines the versioned schema of this representation of an object.
Servers should convert recognized schemas to the latest internal value, and
may reject unrecognized values.
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
type: string
kind:
description: |-
Kind is a string value representing the REST resource this object represents.
Servers may infer this from the endpoint the client submits requests to.
Cannot be updated.
In CamelCase.
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
type: string
metadata:
type: object
spec:
description: IPPoolSpec defines the desired state of IPPool
properties:
allocations:
additionalProperties:
description: IPAllocation represents metadata about the pod/container
owner of a specific IP
properties:
id:
type: string
ifname:
type: string
podref:
type: string
required:
- id
- podref
type: object
description: |-
Allocations is the set of allocated IPs for the given range. Its` indices are a direct mapping to the
IP with the same index/offset for the pool's range.
type: object
range:
description: Range is a RFC 4632/4291-style string that represents
an IP address and prefix length in CIDR notation
type: string
required:
- allocations
- range
type: object
type: object
served: true
storage: true
@@ -0,0 +1,79 @@
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
controller-gen.kubebuilder.io/version: v0.14.0
name: nodeslicepools.whereabouts.cni.cncf.io
spec:
group: whereabouts.cni.cncf.io
names:
kind: NodeSlicePool
listKind: NodeSlicePoolList
plural: nodeslicepools
singular: nodeslicepool
scope: Namespaced
versions:
- name: v1alpha1
schema:
openAPIV3Schema:
description: NodeSlicePool is the Schema for the nodesliceippools API
properties:
apiVersion:
description: |-
APIVersion defines the versioned schema of this representation of an object.
Servers should convert recognized schemas to the latest internal value, and
may reject unrecognized values.
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
type: string
kind:
description: |-
Kind is a string value representing the REST resource this object represents.
Servers may infer this from the endpoint the client submits requests to.
Cannot be updated.
In CamelCase.
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
type: string
metadata:
type: object
spec:
description: NodeSlicePoolSpec defines the desired state of NodeSlicePool
properties:
range:
description: |-
Range is a RFC 4632/4291-style string that represents an IP address and prefix length in CIDR notation
this refers to the entire range where the node is allocated a subset
type: string
sliceSize:
description: SliceSize is the size of subnets or slices of the range
that each node will be assigned
type: string
required:
- range
- sliceSize
type: object
status:
description: NodeSlicePoolStatus defines the desired state of NodeSlicePool
properties:
allocations:
description: Allocations holds the allocations of nodes to slices
items:
properties:
nodeName:
description: NodeName is the name of the node assigned to this
slice, empty node name is an available slice for assignment
type: string
sliceRange:
description: SliceRange is the subnet of this slice
type: string
required:
- nodeName
- sliceRange
type: object
type: array
required:
- allocations
type: object
type: object
served: true
storage: true
@@ -0,0 +1,57 @@
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
controller-gen.kubebuilder.io/version: v0.14.0
name: overlappingrangeipreservations.whereabouts.cni.cncf.io
spec:
group: whereabouts.cni.cncf.io
names:
kind: OverlappingRangeIPReservation
listKind: OverlappingRangeIPReservationList
plural: overlappingrangeipreservations
singular: overlappingrangeipreservation
scope: Namespaced
versions:
- name: v1alpha1
schema:
openAPIV3Schema:
description: OverlappingRangeIPReservation is the Schema for the OverlappingRangeIPReservations
API
properties:
apiVersion:
description: |-
APIVersion defines the versioned schema of this representation of an object.
Servers should convert recognized schemas to the latest internal value, and
may reject unrecognized values.
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
type: string
kind:
description: |-
Kind is a string value representing the REST resource this object represents.
Servers may infer this from the endpoint the client submits requests to.
Cannot be updated.
In CamelCase.
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
type: string
metadata:
type: object
spec:
description: OverlappingRangeIPReservationSpec defines the desired state
of OverlappingRangeIPReservation
properties:
containerid:
type: string
ifname:
type: string
podref:
type: string
required:
- podref
type: object
required:
- spec
type: object
served: true
storage: true
@@ -0,0 +1,196 @@
---
# Whereabouts: cluster-wide IPAM for Multus attachments.
#
# WHY THIS EXISTS
#
# The attachments in this cluster lease their addresses by DHCP
# (cni-dhcp-daemon.yaml). That works where a DHCP server is on the wire —
# v30iot has one. It does not work on the Ceph SAN (VLAN 49), which carries
# Ceph-internal traffic and nothing else, by design: there is no DHCP server
# there and there must not be one.
#
# `host-local`, already installed by cni-plugins-installer.yaml, cannot stand
# in. It keeps its allocation state per node with no cluster-wide
# coordination, so on a shared L2 segment two nodes hand out the same address.
# Whereabouts coordinates through the Kubernetes API instead — allocations
# live in the ippools/overlappingrangeipreservations CRDs, so nothing about
# the address assignment touches the segment being addressed. That property is
# what makes it usable on an isolated SAN.
#
# HOST ACCESS
#
# The DaemonSet runs with hostNetwork and privileged. That is how upstream
# ships it, and it is what the three CNI components already here do —
# multus-daemonset.yaml, cni-dhcp-daemon.yaml and cni-plugins-installer.yaml
# all carry both. The reason is the same in each case: the pod writes a CNI
# binary and its config into /opt/cni/bin and /etc/cni/net.d on the node,
# which is node plumbing rather than a workload, and it must work before the
# attachment it provides addresses for exists.
#
# The reconciler is deliberately NOT in here. It needs neither, and runs as an
# ordinary pod-network Deployment — see whereabouts-reconciler.yaml.
#
# Vendored from upstream v0.9.4 (doc/crds/daemonset-install.yaml), with the
# image tag pinned; upstream ships `:latest`. The CRDs it needs are the three
# whereabouts-crd-*.yaml files alongside this one, vendored unmodified so an
# upstream diff stays readable.
apiVersion: v1
kind: ServiceAccount
metadata:
name: whereabouts
namespace: kube-system
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: whereabouts
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: whereabouts-cni
subjects:
- kind: ServiceAccount
name: whereabouts
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: whereabouts-cni
rules:
- apiGroups:
- whereabouts.cni.cncf.io
resources:
- ippools
- overlappingrangeipreservations
- nodeslicepools
verbs:
- get
- list
- watch
- create
- update
- patch
- delete
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- '*'
- apiGroups: [""]
resources:
- pods
verbs:
- list
- watch
- get
- apiGroups: [""]
resources:
- nodes
verbs:
- get
- list
- watch
- apiGroups: ["k8s.cni.cncf.io"]
resources:
- network-attachment-definitions
verbs:
- get
- list
- watch
- apiGroups:
- ""
- events.k8s.io
resources:
- events
verbs:
- create
- patch
- update
- get
---
apiVersion: v1
kind: ConfigMap
metadata:
name: whereabouts-config
namespace: kube-system
annotations:
kubernetes.io/description: |
Configmap containing user customizable cronjob schedule
data:
# Read by the reconciler (whereabouts-reconciler.yaml), not by this
# DaemonSet. Governs how often stale allocations are swept — the case that
# matters here is a node lost hard enough that the CNI DEL never ran, which
# otherwise leaves an address held by a pod that no longer exists.
cron-expression: "30 4 * * *"
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: whereabouts
namespace: kube-system
labels:
tier: node
app: whereabouts
spec:
selector:
matchLabels:
name: whereabouts
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
tier: node
app: whereabouts
name: whereabouts
spec:
hostNetwork: true
serviceAccountName: whereabouts
# Every node here is a control plane node, so the NoSchedule toleration
# is not optional — without it this schedules nowhere.
tolerations:
- operator: Exists
effect: NoSchedule
containers:
- name: whereabouts
command: ["/bin/sh"]
args:
- -c
- |
SLEEP=false source /install-cni.sh
/token-watcher.sh &
/ip-control-loop -log-level debug
image: ghcr.io/k8snetworkplumbingwg/whereabouts:v0.9.4
env:
- name: NODENAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
- name: WHEREABOUTS_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
resources:
requests:
cpu: "100m"
memory: "100Mi"
limits:
cpu: "100m"
memory: "200Mi"
securityContext:
privileged: true
volumeMounts:
- name: cnibin
mountPath: /host/opt/cni/bin
- name: cni-net-dir
mountPath: /host/etc/cni/net.d
volumes:
- name: cnibin
hostPath:
path: /opt/cni/bin
- name: cni-net-dir
hostPath:
path: /etc/cni/net.d
@@ -0,0 +1,75 @@
---
# Sweeps whereabouts allocations that no live pod holds any more.
#
# The normal release path is the CNI DEL when a pod goes away. It does not run
# when a node is lost hard — power cut, kernel panic, a reboot that never
# comes back — and the allocation then stays claimed by a pod that no longer
# exists. On a small range that matters: the Ceph SAN is a /29, so a handful
# of leaked addresses is the difference between an OSD getting an address and
# not.
#
# Runs as an ordinary pod-network Deployment. Unlike the DaemonSet next to it,
# this needs no host access at all — it only talks to the Kubernetes API,
# where the allocations live.
#
# replicas stays at 1: there is no leader election, so a second replica would
# reconcile the same records concurrently.
#
# Vendored from upstream v0.9.4 (doc/crds/reconciler-deployment.yaml) with the
# image tag pinned; upstream ships `:latest`. The schedule comes from the
# whereabouts-config ConfigMap in whereabouts-daemonset.yaml.
apiVersion: apps/v1
kind: Deployment
metadata:
name: whereabouts-reconciler
namespace: kube-system
labels:
tier: node
app: whereabouts-reconciler
spec:
replicas: 1
selector:
matchLabels:
name: whereabouts-reconciler
template:
metadata:
labels:
tier: node
app: whereabouts-reconciler
name: whereabouts-reconciler
spec:
serviceAccountName: whereabouts
# Every node here is a control plane node.
tolerations:
- operator: Exists
effect: NoSchedule
containers:
- name: whereabouts-reconciler
command: ["/ip-reconciler"]
args:
- -log-level
- debug
image: ghcr.io/k8snetworkplumbingwg/whereabouts:v0.9.4
env:
- name: WHEREABOUTS_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
volumeMounts:
- name: cron-scheduler-configmap
mountPath: /cron-schedule
volumes:
- name: cron-scheduler-configmap
configMap:
name: whereabouts-config
defaultMode: 0744
items:
- key: "cron-expression"
path: "config"
+18 -4
View File
@@ -20,16 +20,30 @@ machine:
# the MS-01 mesh later needs neither a new image
# nor a reinstall — which is exactly the property
# lost by installing without it.
# siderolabs/i915 Intel GPU microcode and kernel modules for the
# Alder Lake-P Iris Xe in the MS-01. No workload
# requests /dev/dri today; it is here for the same
# reason as thunderbolt, so that wanting hardware
# transcoding later is a config change rather than
# another rolling upgrade of every node.
#
# a21cec8f… is the bare-metal schematic described in RUNBOOK-cold-rebuild.md:
# same net.ifnames=1 as the Proxmox one (the predictable interface names the
# per-node VLAN parents match against), minus qemu-guest-agent.
# Not included, and why, so the absences are not mistaken for oversights:
# the X710 (i40e) and I226 (igc) NICs need no firmware extension —
# intel-ice-firmware covers E810/ice only. mei is a prerequisite for Intel
# Arc discrete GPUs, of which there are none. iscsi-tools, nvme-cli and
# util-linux-tools have no consumer: storage is Ceph RBD, and device health
# is reported by Ceph itself rather than a node-level exporter.
#
# 0f6fb8bd… is the bare-metal schematic: same net.ifnames=1 as the Proxmox
# one (the predictable interface names the per-node VLAN parents match
# against), minus qemu-guest-agent. It supersedes a21cec8f…, which was
# identical but without i915 — see RUNBOOK-cold-rebuild.md.
#
# Only read at install and upgrade time. Changing it here does nothing to a
# running node until that node is upgraded. Bump the tag with the Talos
# version — it must match the running version, or the upgrade is a downgrade.
install:
image: factory.talos.dev/metal-installer/a21cec8f35ae32ab5e9a452e7b13500ca8776053930b8e182b9cb23eed186b64:v1.12.6
image: factory.talos.dev/metal-installer/0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9:v1.12.11
features:
rbac: true
stableHostname: true