Confine the DHCP-leased default route with sbr on the v30iot macvlans #137

2026-07-09T10:44:27Z

k.renner commented

2026-07-09 10:44:27 +00:00

Regression from #135

shelly-manager is down right now — NotReady, ingress returns 504 Gateway Timeout.

#135 was right that these addresses should be allocator-owned, and wrong about what else a lease carries. A DHCP lease arrives with a default gateway (option 3, 10.2.30.1). The dhcp IPAM plugin installs it, so the pod ends up with two default routes at the same metric:

Iface  Destination  Gateway
net1   00000000     011E020A   <- 10.2.30.1, from DHCP
eth0   00000000     3C02F40A   <- 10.244.2.60, Cilium

Replies to cluster-internal peers egress the IoT VLAN and are dropped. Traefik times out.

The static IPAM this replaced avoided the problem by never setting a gateway at all — and said so:

No gateway is set on this attachment: default route stays on the pod's primary (Cilium-managed) interface, so non-IoT egress doesn't leak through eth1.

I deleted the addresses and the gateway suppression together. A leased address comes with a gateway, so it has to be confined, not omitted.

Why it looked healthy

The kubelet readiness probe originates from the node address, which the net1 default route happens to reach. It passed for the first few seconds. I sampled ready=true in that window and reported the change verified. The endpoint has since flipped to ready=false. Multicast/mDNS on net1 was genuinely fine — it just had nothing to do with the failure.

Fix

Chain the sbr (source-based routing) plugin after macvlan. It moves net1's routes into a routing table selected by net1's source address, leaving eth0's default alone in the main table. This restores the original invariant — no non-IoT egress through the second NIC — while keeping the address DHCP-leased.

sbr is containernetworking/plugins v1.5.1, already present in /opt/cni/bin on every node courtesy of the existing cni-plugins-installer DaemonSet. No new binary, no new DaemonSet.

matter-server: macvlan → tuning → sbr
otbr, shelly-manager: converted from single-plugin config to a plugins array, macvlan → sbr

Blast radius

Only shelly-manager runs a post-#135 sandbox, so it is the only pod currently broken. matter-server and otbr still hold their pre-#135 static addresses and a single eth0 default route — but their NADs already say dhcp, so any restart (node reboot, eviction, OOM, rollout) would break them identically. This PR is a strict improvement for them whether or not sbr proves perfect.

Not verified — please read before merging

sbr's effect on the IPv6 OMR route. matter-server needs the fd5f::/64 RIO that OTBR publishes on v30iot for Matter CASE sessions to Thread devices. Source-based routing should keep it reachable for traffic sourced from net1's address, but I have not demonstrated it. It is called out in the NAD comment too.

Suggested sequence after merge:

Restart shelly-manager (non-critical). Confirm exactly one default route, via eth0; endpoint ready=true; https://shelly-manager.home.renner.uno/inventory returns 200.
Only then restart otbr, and check ip -6 route plus Thread attachment.
Only then restart matter-server, and check ip -6 route for fd5f::/64 and a live CASE session.

Still outstanding from #135 and unrelated to this fix: whether a cni-dhcp-daemon restart keeps renewing an attached pod's lease.

## Regression from #135 `shelly-manager` is **down right now** — `NotReady`, ingress returns `504 Gateway Timeout`. #135 was right that these addresses should be allocator-owned, and wrong about what else a lease carries. A DHCP lease arrives with a **default gateway** (option 3, `10.2.30.1`). The `dhcp` IPAM plugin installs it, so the pod ends up with two default routes at the same metric: ``` Iface Destination Gateway net1 00000000 011E020A <- 10.2.30.1, from DHCP eth0 00000000 3C02F40A <- 10.244.2.60, Cilium ``` Replies to cluster-internal peers egress the IoT VLAN and are dropped. Traefik times out. The static IPAM this replaced avoided the problem by never setting a gateway at all — and said so: > No gateway is set on this attachment: default route stays on the pod's primary (Cilium-managed) interface, so non-IoT egress doesn't leak through eth1. I deleted the addresses and the gateway suppression together. A leased address comes with a gateway, so it has to be **confined**, not omitted. ### Why it looked healthy The kubelet readiness probe originates from the **node** address, which the `net1` default route happens to reach. It passed for the first few seconds. I sampled `ready=true` in that window and reported the change verified. The endpoint has since flipped to `ready=false`. Multicast/mDNS on `net1` was genuinely fine — it just had nothing to do with the failure. ## Fix Chain the `sbr` (source-based routing) plugin after `macvlan`. It moves `net1`'s routes into a routing table selected by `net1`'s source address, leaving `eth0`'s default alone in the main table. This restores the original invariant — no non-IoT egress through the second NIC — while keeping the address DHCP-leased. `sbr` is `containernetworking/plugins v1.5.1`, already present in `/opt/cni/bin` on every node courtesy of the existing `cni-plugins-installer` DaemonSet. No new binary, no new DaemonSet. - `matter-server`: `macvlan → tuning → sbr` - `otbr`, `shelly-manager`: converted from single-plugin config to a `plugins` array, `macvlan → sbr` ## Blast radius Only `shelly-manager` runs a post-#135 sandbox, so it is the only pod currently broken. `matter-server` and `otbr` still hold their pre-#135 static addresses and a single `eth0` default route — **but their NADs already say `dhcp`**, so any restart (node reboot, eviction, OOM, rollout) would break them identically. This PR is a strict improvement for them whether or not `sbr` proves perfect. ## Not verified — please read before merging `sbr`'s effect on the **IPv6 OMR route**. `matter-server` needs the `fd5f::/64` RIO that OTBR publishes on v30iot for Matter CASE sessions to Thread devices. Source-based routing *should* keep it reachable for traffic sourced from `net1`'s address, but I have not demonstrated it. It is called out in the NAD comment too. Suggested sequence after merge: 1. Restart `shelly-manager` (non-critical). Confirm exactly one default route, via `eth0`; endpoint `ready=true`; `https://shelly-manager.home.renner.uno/inventory` returns 200. 2. Only then restart `otbr`, and check `ip -6 route` plus Thread attachment. 3. Only then restart `matter-server`, and check `ip -6 route` for `fd5f::/64` and a live CASE session. Still outstanding from #135 and unrelated to this fix: whether a `cni-dhcp-daemon` restart keeps renewing an attached pod's lease.

k.renner added 1 commit 2026-07-09 10:44:28 +00:00

Confine the DHCP-leased default route with sbr on the v30iot macvlans df61407f84

PR #135 moved the three v30iot macvlan attachments from static IPAM to
DHCP. That was right about ownership and wrong about routing: a leased
address arrives with a default gateway (DHCP option 3, 10.2.30.1), and
the dhcp IPAM plugin installs it as a *second* default route on net1 at
the same metric as eth0's Cilium-managed one.

Cluster-internal replies then egress the IoT VLAN and are dropped. The
kubelet readiness probe comes from the node address and keeps passing for
a few seconds, which makes the pod look healthy; anything from the pod
CIDR -- Traefik -- times out. shelly-manager went NotReady and its
ingress returned 504.

The old static IPAM sidestepped this by simply never setting a gateway,
as its comment said: "No gateway is set on this attachment: default route
stays on the pod's primary (Cilium-managed) interface, so non-IoT egress
doesn't leak through eth1." A leased address comes with a gateway, so it
has to be confined rather than omitted.

Chain the `sbr` plugin (source-based routing, containernetworking/plugins
v1.5.1, already on every node from cni-plugins-installer) after macvlan.
It moves net1's routes into a routing table selected by net1's source
address, leaving eth0's default alone in the main table.

matter-server's chain becomes macvlan -> tuning -> sbr; otbr and
shelly-manager convert from a single-plugin config to a plugins array.

Only shelly-manager currently runs a post-#135 sandbox, so it is the only
pod actually broken right now. matter-server and otbr still hold their
pre-#135 static addresses and single eth0 default route -- but their NADs
already say dhcp, so they would break the same way on any restart. This
change is a strict improvement for them either way.

UNVERIFIED, called out in the matter-server NAD: sbr's effect on the IPv6
OMR route. Matter CASE sessions to Thread devices need the fd5f::/64 RIO
that OTBR publishes on v30iot. Source-based routing should keep it
reachable for traffic sourced from net1's address, but this has not been
demonstrated. Check `ip -6 route` inside matter-server after its first
restart under this config.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

k.renner merged commit 4250958425 into master

2026-07-09 10:45:02 +00:00

k.renner referenced this issue from a commit

2026-07-09 10:45:03 +00:00

Merge pull request 'Confine the DHCP-leased default route with sbr on the v30iot macvlans' (#137) from fix/v30iot-sbr-confine-dhcp-default-route into master

k.renner referenced this pull request

2026-07-09 11:44:01 +00:00

Gate shelly-manager behind Authentik passkey step-up #138

k.renner referenced this pull request

2026-07-09 12:09:33 +00:00

otbr: poll for the persisted dataset instead of sampling it once #139

Sign in to join this conversation.