Lease v30iot macvlan addresses by DHCP instead of pinning them #135

2026-07-09T10:04:41Z

k.renner commented

2026-07-09 10:04:41 +00:00

Problem

UniFi raised a High-severity Device IP Address Conflict on 10.2.30.226 today. It is not new, and not one address.

The three macvlan NADs on the IoT VLAN pinned hand-picked addresses via static IPAM. All three sit inside the UniFi DHCP pool (10.2.30.100–.254, confirmed against the controller), so the UDM leased them out from under the pods:

IP	Contender	`CLIENT_IP_CONFLICT` alerts
`.226`	Miele oven — `00:1d:63:5a:95:4e` (EG Küche Herd)	16
`.227`	Miele fridge — `00:1d:63:87:9d:b1` (EG Küche Kühlschrank)	8
`.228`	itself, across rolling updates	4

Alerts have fired continuously from 2026-05-13 — the day of the ping sweep that picked the addresses — through 2026-07-09. Confirmed live: arping from inside VLAN 30 gets replies for .226 from both the pod and the Miele, and the Miele wins the ARP cache.

Nothing noticed because these attachments exist purely for mDNS/multicast L2 presence, and multicast reception survives losing an ARP race. matter-server logs zero network errors. What actually breaks is IPv4 unicast: matter-server advertises .226 in its mDNS A record, so a Matter controller resolving that record talks to a dishwasher.

.228 involves no real device at all. Multus randomizes the macvlan MAC per pod sandbox, so during a rolling update the terminating and starting pods both claimed .228 under different MACs. A hardcoded address made the app fight its own next version.

Why not just move the addresses

Because that preserves the defect. An address inside the pool and an address outside the pool are both addresses no allocator owns, justified by evidence of current absence. A ping sweep shows which hosts answer at that instant; it says nothing about what DHCP will lease next month. That reasoning is what produced this bug, and the identical failure already happened once before — the Cilium iot-pool at .224–.239 was abandoned after losing ARP races and moved below the pool.

So the addresses are removed rather than relocated. The UDM allocates, as it does for every other device on this VLAN.

Change

All three NADs → "ipam": {"type": "dhcp"}.
New cni-dhcp-daemon DaemonSet in infrastructure/multus/. The dhcp IPAM plugin is only a client of a per-node daemon on /run/cni/dhcp.sock; without it the attachment fails with failed to connect to dhcp daemon.
Two stale comments corrected.

Nothing depends on these addresses

Verified before changing anything:

Repo: the only occurrences were the NAD definitions themselves plus comments.
Live cluster: Deployments, DaemonSets, StatefulSets, Services, Ingresses, ConfigMaps, and Secrets decoded from base64 — zero references.
UniFi controller: no firewall rule, port-forward, or firewall group mentions them; the only collection that does is the client table.
Home Assistant: no hits in .storage; it reaches OTBR at otbr.home-assistant.svc.cluster.local:8081.
Runtime binds: matter-server uses --primary-interface=net1, otbr uses BACKBONE_INTERFACE=net1 / REST_HOST=0.0.0.0, shelly-manager binds 0.0.0.0:8080. All three are consumed via ClusterIP.

Manifest details verified on the cluster

dhcp v1.5.1 is already in /opt/cni/bin on every node (the sibling cni-plugins-installer extracts the whole tarball).
kube-system carries no Pod Security labels; Multus itself runs privileged.
/run/netns exists on the node; /run/cni does not → type: DirectoryOrCreate.
mountPropagation: HostToContainer on /run/netns copied from Multus, which needs it for the same reason.
Host has /var/run → /run, and alpine:3.21 has /var/run → ../run, so the /var/run/netns/cni-* paths containerd passes as CNI_NETNS resolve without -hostprefix.
All three NADs render valid CNI JSON; kubectl kustomize builds clean.

No stable MAC is pinned deliberately — under DHCP the random per-sandbox MAC is what lets a rolling update take two independent leases instead of colliding on one reservation.

Rollout

apps already dependsOn: infrastructure, so the daemon is created before the NADs reconcile. NAD changes affect only new sandboxes, so running pods keep their current addresses until restarted. The conflict persists until the three pods are restarted.

Open risk — please weigh in

The dhcp daemon tracks leases in memory. I have not verified what happens to already-attached pods when the daemon restarts (node reboot, DaemonSet update): renewals may stop while the pod keeps its address configured, which after the 24h lease could let the UDM re-lease it. Worth testing with a controlled daemon restart before this is considered settled.

## Problem UniFi raised a High-severity `Device IP Address Conflict` on `10.2.30.226` today. It is not new, and not one address. The three macvlan NADs on the IoT VLAN pinned hand-picked addresses via static IPAM. All three sit **inside** the UniFi DHCP pool (`10.2.30.100–.254`, confirmed against the controller), so the UDM leased them out from under the pods: | IP | Contender | `CLIENT_IP_CONFLICT` alerts | |---|---|---| | `.226` | Miele oven — `00:1d:63:5a:95:4e` (*EG Küche Herd*) | 16 | | `.227` | Miele fridge — `00:1d:63:87:9d:b1` (*EG Küche Kühlschrank*) | 8 | | `.228` | **itself**, across rolling updates | 4 | Alerts have fired continuously from **2026-05-13** — the day of the ping sweep that picked the addresses — through **2026-07-09**. Confirmed live: `arping` from inside VLAN 30 gets replies for `.226` from both the pod and the Miele, and the Miele wins the ARP cache. Nothing noticed because these attachments exist purely for mDNS/multicast L2 presence, and multicast reception survives losing an ARP race. `matter-server` logs zero network errors. What actually breaks is IPv4 unicast: `matter-server` advertises `.226` in its mDNS A record, so a Matter controller resolving that record talks to a dishwasher. `.228` involves no real device at all. Multus randomizes the macvlan MAC per pod sandbox, so during a rolling update the terminating and starting pods both claimed `.228` under different MACs. A hardcoded address made the app fight its own next version. ## Why not just move the addresses Because that preserves the defect. An address inside the pool and an address outside the pool are both addresses **no allocator owns**, justified by evidence of current absence. A ping sweep shows which hosts answer at that instant; it says nothing about what DHCP will lease next month. That reasoning is what produced this bug, and the identical failure already happened once before — the Cilium `iot-pool` at `.224–.239` was abandoned after losing ARP races and moved below the pool. So the addresses are removed rather than relocated. The UDM allocates, as it does for every other device on this VLAN. ## Change - All three NADs → `"ipam": {"type": "dhcp"}`. - New `cni-dhcp-daemon` DaemonSet in `infrastructure/multus/`. The `dhcp` IPAM plugin is only a client of a per-node daemon on `/run/cni/dhcp.sock`; without it the attachment fails with *failed to connect to dhcp daemon*. - Two stale comments corrected. ## Nothing depends on these addresses Verified before changing anything: - **Repo**: the only occurrences were the NAD definitions themselves plus comments. - **Live cluster**: Deployments, DaemonSets, StatefulSets, Services, Ingresses, ConfigMaps, and Secrets decoded from base64 — zero references. - **UniFi controller**: no firewall rule, port-forward, or firewall group mentions them; the only collection that does is the client table. - **Home Assistant**: no hits in `.storage`; it reaches OTBR at `otbr.home-assistant.svc.cluster.local:8081`. - **Runtime binds**: `matter-server` uses `--primary-interface=net1`, `otbr` uses `BACKBONE_INTERFACE=net1` / `REST_HOST=0.0.0.0`, `shelly-manager` binds `0.0.0.0:8080`. All three are consumed via ClusterIP. ## Manifest details verified on the cluster - `dhcp v1.5.1` is already in `/opt/cni/bin` on every node (the sibling `cni-plugins-installer` extracts the whole tarball). - `kube-system` carries no Pod Security labels; Multus itself runs `privileged`. - `/run/netns` exists on the node; `/run/cni` does not → `type: DirectoryOrCreate`. - `mountPropagation: HostToContainer` on `/run/netns` copied from Multus, which needs it for the same reason. - Host has `/var/run → /run`, and `alpine:3.21` has `/var/run → ../run`, so the `/var/run/netns/cni-*` paths containerd passes as `CNI_NETNS` resolve without `-hostprefix`. - All three NADs render valid CNI JSON; `kubectl kustomize` builds clean. No stable MAC is pinned deliberately — under DHCP the random per-sandbox MAC is what lets a rolling update take two independent leases instead of colliding on one reservation. ## Rollout `apps` already `dependsOn: infrastructure`, so the daemon is created before the NADs reconcile. NAD changes affect only new sandboxes, so running pods keep their current addresses until restarted. The conflict persists until the three pods are restarted. ## Open risk — please weigh in The `dhcp` daemon tracks leases in memory. I have **not** verified what happens to already-attached pods when the daemon restarts (node reboot, DaemonSet update): renewals may stop while the pod keeps its address configured, which after the 24h lease could let the UDM re-lease it. Worth testing with a controlled daemon restart before this is considered settled.

k.renner added 1 commit 2026-07-09 10:04:41 +00:00

Lease v30iot macvlan addresses by DHCP instead of pinning them 19a44821a6

The three macvlan NADs on the IoT VLAN (matter-server, otbr,
shelly-manager) used static IPAM with hand-picked addresses .226/.227/
.228, chosen because a ping sweep on 2026-05-13 found them idle. All
three sit inside the UniFi DHCP pool (10.2.30.100-.254), so the UDM
leased them out from under the pods.

The controller has been logging CLIENT_IP_CONFLICT continuously since
2026-05-13:

  .226  vs Miele oven   (00:1d:63:5a:95:4e)   16 alerts
  .227  vs Miele fridge (00:1d:63:87:9d:b1)    8 alerts
  .228  vs itself                              4 alerts

Nobody noticed because these attachments exist only for mDNS/multicast
L2 presence, and multicast reception survives losing the ARP race, so no
pod ever logged an error. The .228 case involves no real device at all:
Multus randomizes the macvlan MAC per sandbox, so a rolling update had
the terminating and starting pods claiming the same static address under
different MACs.

A ping sweep shows which hosts answer at that instant. It says nothing
about what DHCP will lease later. Relocating the addresses to a quieter
part of the subnet would preserve the actual defect -- an address that no
allocator owns. So remove the addresses instead: let the UDM allocate,
as it does for every other device on this VLAN.

Nothing depends on the values. Verified across the repo, every live
cluster object including decoded Secrets, and Home Assistant's stored
config entries: no consumer references .226/.227/.228, and no UniFi
firewall rule, port-forward, or firewall group mentions them. All three
pods are reached via their ClusterIP Services; matter-server binds by
interface name (--primary-interface=net1) and otbr by
BACKBONE_INTERFACE=net1.

Adds cni-dhcp-daemon to infrastructure/multus, which the dhcp IPAM
plugin requires -- the plugin is only a client of a per-node daemon on
/run/cni/dhcp.sock. The dhcp binary is already on the nodes; the sibling
cni-plugins-installer extracts the whole v1.5.1 tarball. Flux's apps
Kustomization already dependsOn infrastructure, so the daemon lands
before the NADs reconcile, and running pods keep their addresses until
restarted.

No stable MAC is pinned: under DHCP the random per-sandbox MAC is what
lets a rolling update take two independent leases rather than collide on
one reservation.

Also corrects two stale comments: matter-server/deployment.yaml still
described the NAD as pinning .226, and bgp-service.yaml claimed the
home-assistant-iot Service at 10.2.30.80 was retained for hard-coded
clients -- it was retired 2026-05-20 and no LoadBalancer Service holds a
10.2.30.x address any more.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

k.renner merged commit 18c6773df7 into master

2026-07-09 10:25:32 +00:00

k.renner referenced this issue from a commit

2026-07-09 10:25:34 +00:00

Merge pull request 'Lease v30iot macvlan addresses by DHCP instead of pinning them' (#135) from fix/v30iot-macvlan-dhcp-ipam into master

k.renner referenced this issue from a commit

2026-07-09 10:43:44 +00:00

Confine the DHCP-leased default route with sbr on the v30iot macvlans

k.renner referenced this pull request

2026-07-09 10:44:27 +00:00

Confine the DHCP-leased default route with sbr on the v30iot macvlans #137

k.renner referenced this issue from a commit