All three nodes run the sole-writer layout. Converge the Multus side and
drop the per-node Cilium overrides:
- kube-multus-ds-subdir becomes the only Multus DaemonSet, no node
affinity; kube-multus-ds (0 pods) and its ConfigMap are pruned.
- /etc/cni/net.d/multus is mounted DirectoryOrCreate so Multus waits for
the Cilium conflist instead of exiting on a missing directory.
- CiliumNodeConfigs removed. They only act when a cilium pod is created;
stage B2 moves the same path into the HelmRelease (cni.confPath) and
rolls the agents.
Refs #832
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Nothing changes until a node is labelled:
- post-crds/cilium-nodeconfig.yaml: CiliumNodeConfig cni-uninstall
(label cni.renner.uno/cilium-uninstall) arms the agent's preStop
conflist cleanup; multus-sole-writer (label
cni.renner.uno/multus-sole-writer) moves write-cni-conf-when-ready to
/host/etc/cni/net.d/multus/05-cilium.conflist, a subdirectory libcni
does not read.
- multus/multus-subdir.yaml: kube-multus-ds-subdir with
multusAutoconfigDir /host/etc/cni/net.d/multus, scheduled only on
labelled nodes; kube-multus-ds gets the inverse affinity.
The kube-multus-ds template change rolls Multus node by node once on
merge.
Refs #832
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Groundwork for moving Ceph's cluster network off host networking. Nothing
references whereabouts yet, so this lands inert: no existing attachment
changes, and the DaemonSet only installs a CNI binary that nothing invokes
until a NetworkAttachmentDefinition names it.
The Ceph SAN on VLAN 49 carries Ceph-internal traffic and nothing else, by
design. Neither IPAM already available covers it:
dhcp cni-dhcp-daemon works where a DHCP server is on the wire.
There is none on the SAN and there must not be one.
host-local installed by cni-plugins-installer, but it keeps allocation
state per node with no cluster-wide coordination. On a shared
L2 segment two nodes hand out the same address.
Whereabouts coordinates through the Kubernetes API — allocations live in the
ippools and overlappingrangeipreservations CRDs. Nothing about assigning an
address touches the segment being addressed, which is the property that makes
it usable on an isolated SAN.
The DaemonSet runs hostNetwork and privileged. That is upstream's shape, and
it matches what multus-daemonset.yaml, cni-dhcp-daemon.yaml and
cni-plugins-installer.yaml already do: the pod writes a CNI binary and its
config into /opt/cni/bin and /etc/cni/net.d, which is node plumbing, and it
has to work before the attachment it serves exists. It is the fourth instance
of an existing exception, not a new one.
The reconciler is split out deliberately. It needs no host access — only the
API, where the allocations are — so it runs as an ordinary pod-network
Deployment. It matters more than it looks: the normal release path is the CNI
DEL when a pod goes away, which does not run when a node is lost hard, and on
a /29 a handful of leaked addresses is the difference between an OSD getting
one and not.
Vendored from upstream v0.9.4 with the image tag pinned — upstream ships
`:latest`. Tag rather than digest, matching the other images in this
directory. The three CRDs are vendored unmodified so a future upstream diff
stays readable. nodeslicepools is included because the ClusterRole references
it, though the node-slice controller itself is not deployed.
Verified: kustomize builds; all nine resources pass a server-side dry run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The three macvlan NADs on the IoT VLAN (matter-server, otbr,
shelly-manager) used static IPAM with hand-picked addresses .226/.227/
.228, chosen because a ping sweep on 2026-05-13 found them idle. All
three sit inside the UniFi DHCP pool (10.2.30.100-.254), so the UDM
leased them out from under the pods.
The controller has been logging CLIENT_IP_CONFLICT continuously since
2026-05-13:
.226 vs Miele oven (00:1d:63:5a:95:4e) 16 alerts
.227 vs Miele fridge (00:1d:63:87:9d:b1) 8 alerts
.228 vs itself 4 alerts
Nobody noticed because these attachments exist only for mDNS/multicast
L2 presence, and multicast reception survives losing the ARP race, so no
pod ever logged an error. The .228 case involves no real device at all:
Multus randomizes the macvlan MAC per sandbox, so a rolling update had
the terminating and starting pods claiming the same static address under
different MACs.
A ping sweep shows which hosts answer at that instant. It says nothing
about what DHCP will lease later. Relocating the addresses to a quieter
part of the subnet would preserve the actual defect -- an address that no
allocator owns. So remove the addresses instead: let the UDM allocate,
as it does for every other device on this VLAN.
Nothing depends on the values. Verified across the repo, every live
cluster object including decoded Secrets, and Home Assistant's stored
config entries: no consumer references .226/.227/.228, and no UniFi
firewall rule, port-forward, or firewall group mentions them. All three
pods are reached via their ClusterIP Services; matter-server binds by
interface name (--primary-interface=net1) and otbr by
BACKBONE_INTERFACE=net1.
Adds cni-dhcp-daemon to infrastructure/multus, which the dhcp IPAM
plugin requires -- the plugin is only a client of a per-node daemon on
/run/cni/dhcp.sock. The dhcp binary is already on the nodes; the sibling
cni-plugins-installer extracts the whole v1.5.1 tarball. Flux's apps
Kustomization already dependsOn infrastructure, so the daemon lands
before the NADs reconcile, and running pods keep their addresses until
restarted.
No stable MAC is pinned: under DHCP the random per-sandbox MAC is what
lets a rolling update take two independent leases rather than collide on
one reservation.
Also corrects two stale comments: matter-server/deployment.yaml still
described the NAD as pinning .226, and bgp-service.yaml claimed the
home-assistant-iot Service at 10.2.30.80 was retained for hard-coded
clients -- it was retired 2026-05-20 and no LoadBalancer Service holds a
10.2.30.x address any more.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Talos's /opt/cni/bin only carries what Cilium installs (cilium-cni,
bandwidth, …); the standard containernetworking/plugins binaries are
missing. Without them multus's macvlan attachment for matter-server
fails at sandbox setup with:
failed to find plugin "macvlan" in path [/opt/cni/bin]
There is no `siderolabs/cni-plugins` Talos system extension (verified
against github.com/siderolabs/extensions/tree/main/network — only
bird2/cloudflared/lldpd/nebula/netbird/newt/tailscale/zerotier).
Building a custom Talos system extension would mean a new OCI image,
schematic regeneration, installer image rebuild, and rolling node
reboots for one missing binary; not worth that surface area.
A small DaemonSet that drops the official containernetworking/plugins
v1.5.1 release tarball into /opt/cni/bin via hostPath is the same
mechanism Cilium and Multus already use successfully on this Talos
cluster — no reboot, no schematic change. Kept as a separate DS rather
than an extra initContainer in multus-daemonset.yaml so the upstream
multus manifest can be re-vendored cleanly.
Pinned to v1.5.1; bump together with the multus version when refreshing
upstream pins.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The HelmRelease pointed at https://k8snetworkplumbingwg.github.io/helm-charts
which 404s — the upstream helm-charts repo only ships charts for
local-clone install, not as a published Helm repository. Fetched in
fallback by Flux's source-controller failed and infrastructure stalled
in Reconciling, blocking everything that dependsOn it (matter-server
included).
Vendor the upstream daemonset manifest pinned to v4.2.4. Going thick
because v4.2.4-thin and unsuffixed v4.2.4 are not published as image
tags — only v4.2.4-thick / v4.2.4-debug exist for the release. The
thick variant chroots into /hostroot to exec CNI plugins; Talos's
/opt/cni/bin is writable (Cilium already uses it) so this is fine.
Header comment captures the why-vendored / why-thick / Talos-OK
reasoning so the next reader doesn't have to re-derive it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The prior agent's hostNetwork: true was the wrong abstraction. Earlier
in this PR I tried a Cilium-L2 LoadBalancer Service on 10.2.30.226,
but that only forwards a single TCP port (5580 ws) and does not give
the pod L2 presence on the IoT subnet — outbound is still cluster pod
network, no multicast, no Matter operational ports. Matter device
commissioning and discovery need real mDNS send/receive on the IoT L2.
Install multus-cni alongside Cilium and attach matter-server to a
NetworkAttachmentDefinition that does macvlan on k8s-edge's eth1 (the
v30iot bridge — verified via `talosctl -n k8s-edge get routes`,
10.2.30.0/24 is connected on eth1). Static IPAM pins 10.2.30.226 with
no gateway, so default-route traffic stays on the Cilium-managed
primary interface and only IoT-subnet traffic crosses net1.
The wiki note hostnetwork-app-workload-antipattern still applies, but
unlike the otbr precedent (commit 1972c9b) — where ClusterIP alone is
sufficient — matter-server actually does need IoT-VLAN reach, just not
via hostNetwork. multus is the documented escape hatch the wiki has
flagged but never exercised in this homelab; first install lands here.
iot-pool ledger now: .224 = home-assistant (LB), .225 = mosquitto
(LB), .226 = matter-server (multus, *not* a Cilium LB). LB Services
all pin explicit IPs via lbipam.cilium.io/ips so no auto-allocation
collides with the multus IP.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
claude-botandClaude Opus 5.5
k.rennerandClaude Opus 4.8