Remove k8s node host addressing on the IoT VLAN (security + hairpin fix) #369

2026-07-23T11:04:07Z

claude-bot commented

2026-07-23 11:04:07 +00:00

The IoT VLAN is treated as compromised, yet all three k8s nodes held addresses on it. DHCP-leased v4 + RA-derived v6 on the v30iot NIC gave every IoT device an L3-reachable path to the node (kubelet, apid, any host-bound service) — a standing exposure independent of anything in this session.

The same addressing was also the root cause of the IoT hairpin failure chased today: the leased address installs a connected 10.2.30.0/24 route, so node replies to IoT-subnet clients egress the IoT leg and bypass the UDM. Ordinary traffic tolerates it; UDM-DNATed traffic does not — the SYN-ACK reaches the client still sourced 10.2.99.86, never un-translated, and the client kernel RSTs it (verified live: DNAT counters increment, br40 shows SYN-then-RST, edge-1 TcpOutRsts delta = SYN-retransmit count).

Change

Per node (k8s-{1,2,3}.yaml), the 0000:06:13.0 NIC goes from dhcp: true + routeMetric: 4096 to:

dhcp: false — no v4 address, no connected route
machine.sysctls: net.ipv6.conf.eth1.disable_ipv6: "1" — no v6, including link-local

The NIC stays up as the macvlan parent for the v30iot pod attachments (matter-server, otbr, shelly-manager); their children DHCP in their own netns and never needed host addressing. The old comment's rationale (Cilium L2 announce for an IoT LB pool) is stale — there is no L2AnnouncementPolicy and the BGP pool is the only LB pool.

⚠️ Name dependency: the sysctl keys on the literal eth1 (a sysctl cannot mirror the interface block's busPath selection). 0000:06:13.0 → eth1 is verified on k8s-{1,2,3}, but a rename would no-op the sysctl silently and leave the host v6-addressed on IoT — so the per-node addressless check below is a hard gate, not a formality.

Chosen over alternatives: reflexive SNAT (papers over the asymmetry, loses client IPs) and Talos policy routing (RoutingRuleConfig is 1.13+; cluster is on 1.12.6). This is the repo's established suppression pattern — cf. edge-host 10-vlan40.network and the HA macvlan /32-not-/24 design.

Residual risk (explicitly NOT solved here)

The NIC remains a macvlan parent, so the node kernel still processes frames from the IoT L2. This PR removes the node's addressable presence, not its L2 attachment. Full isolation would require relocating how those three pods reach IoT devices (dedicated appliance / routed access) — a separate, larger decision worth its own issue.

Rollout (Talos machineconfig — applied by `talosctl`, not CI) — STAGED, per node

Apply to k8s-1 only, then before touching k8s-2/3 confirm ALL of:

Node stays Ready.
talosctl -n k8s-1 get links shows eth1 UP with no v4/v6 address (HARD STOP — a wrong name here fails silently; if eth1 still has an address, do not proceed).
matter-server / otbr / shelly-manager pods on k8s-1 stay Ready and still reach their IoT devices.
macvlan child IPv6 intact: inside the matter-server pod, ip -6 addr / ip -6 route show dev net1 still shows the Thread OMR prefix (fd5f::/64) RIO — i.e. disabling v6 on the parent did not starve the child's own stack.
Reboot k8s-1 (not just a live apply) and re-confirm 2–4 — proves eth1 comes up addressless-but-UP for the macvlan parent across a cold boot. If Talos won't bring an addressless NIC up, fall back to a dummy link: up mechanism before continuing.
From an IoT client (cold DNS), a public app name serves the cluster cert when k8s-1 is the serving node.

Then repeat for k8s-2, k8s-3. I run the cert-serial + canary battery per node as you apply.

🤖 Generated with Claude Code

https://claude.ai/code/session_014S5CLfizEtzHS51kxJFQgc

**The IoT VLAN is treated as compromised, yet all three k8s nodes held addresses on it.** DHCP-leased v4 + RA-derived v6 on the v30iot NIC gave every IoT device an L3-reachable path to the node (kubelet, apid, any host-bound service) — a standing exposure independent of anything in this session. The same addressing was also the root cause of the IoT hairpin failure chased today: the leased address installs a connected `10.2.30.0/24` route, so node replies to IoT-subnet clients egress the IoT leg and **bypass the UDM**. Ordinary traffic tolerates it; UDM-DNATed traffic does not — the SYN-ACK reaches the client still sourced `10.2.99.86`, never un-translated, and the client kernel RSTs it (verified live: DNAT counters increment, br40 shows SYN-then-RST, edge-1 `TcpOutRsts` delta = SYN-retransmit count). ## Change Per node (`k8s-{1,2,3}.yaml`), the `0000:06:13.0` NIC goes from `dhcp: true` + `routeMetric: 4096` to: - `dhcp: false` — no v4 address, no connected route - `machine.sysctls: net.ipv6.conf.eth1.disable_ipv6: "1"` — no v6, including link-local The NIC **stays up as the macvlan parent** for the v30iot pod attachments (matter-server, otbr, shelly-manager); their children DHCP in their own netns and never needed host addressing. The old comment's rationale (Cilium L2 announce for an IoT LB pool) is stale — there is no L2AnnouncementPolicy and the BGP pool is the only LB pool. ⚠️ **Name dependency:** the sysctl keys on the literal `eth1` (a sysctl cannot mirror the interface block's `busPath` selection). `0000:06:13.0` → `eth1` is verified on k8s-{1,2,3}, but a rename would no-op the sysctl **silently** and leave the host v6-addressed on IoT — so the per-node addressless check below is a hard gate, not a formality. Chosen over alternatives: reflexive SNAT (papers over the asymmetry, loses client IPs) and Talos policy routing (`RoutingRuleConfig` is 1.13+; cluster is on 1.12.6). This is the repo's established suppression pattern — cf. edge-host `10-vlan40.network` and the HA macvlan `/32`-not-`/24` design. ## Residual risk (explicitly NOT solved here) The NIC remains a macvlan parent, so the node kernel still processes frames from the IoT L2. This PR removes the node's **addressable** presence, not its L2 attachment. Full isolation would require relocating how those three pods reach IoT devices (dedicated appliance / routed access) — a separate, larger decision worth its own issue. ## Rollout (Talos machineconfig — applied by `talosctl`, not CI) — STAGED, per node Apply to **k8s-1 only**, then before touching k8s-2/3 confirm ALL of: 1. Node stays Ready. 2. `talosctl -n k8s-1 get links` shows **eth1 UP with no v4/v6 address** (HARD STOP — a wrong name here fails silently; if eth1 still has an address, do not proceed). 3. matter-server / otbr / shelly-manager pods on k8s-1 stay Ready and still reach their IoT devices. 4. **macvlan child IPv6 intact:** inside the matter-server pod, `ip -6 addr` / `ip -6 route show dev net1` still shows the Thread OMR prefix (`fd5f::/64`) RIO — i.e. disabling v6 on the parent did not starve the child's own stack. 5. **Reboot k8s-1** (not just a live apply) and re-confirm 2–4 — proves eth1 comes up addressless-**but-UP** for the macvlan parent across a cold boot. If Talos won't bring an addressless NIC up, fall back to a dummy `link: up` mechanism before continuing. 6. From an IoT client (cold DNS), a public app name serves the **cluster** cert when k8s-1 is the serving node. Then repeat for k8s-2, k8s-3. I run the cert-serial + canary battery per node as you apply. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_014S5CLfizEtzHS51kxJFQgc

claude-bot added 1 commit 2026-07-23 11:04:07 +00:00

fix(talos): remove node host addressing on the v30iot NIC

ci/woodpecker/pr/pr-review Pipeline was successful

Details

b077de8556

Security + routing. The k8s nodes held a DHCP-leased v4 address and an
RA-derived v6 address on the IoT VLAN — a network that must be treated
as compromised. That gave every IoT device an L3-reachable path to the
node (kubelet, SSH-equivalent apid, any host service) and, as a side
effect, installed a connected 10.2.30.0/24 route so node replies to
IoT-subnet clients bypassed the UDM entirely — breaking every UDM-DNATed
flow (the hairpin NAT: un-translated SYN-ACK -> client kernel RST,
verified live 2026-07-23).

The NIC (busPath 0000:06:13.0) stays up as the macvlan PARENT for the
v30iot pod attachments (matter-server, otbr, shelly-manager), whose
children do their own DHCP in their own netns — host addressing was
never needed for that. dhcp: false drops the v4 address+route;
sysctl net.ipv6.conf.eth1.disable_ipv6=1 drops v6 including link-local,
so the host has no addressable presence on the L2 at all. The old
comment's claimed purpose (Cilium L2 announce for an IoT LB pool) does
not exist — no L2AnnouncementPolicy, BGP-only pool.

RESIDUAL (not addressed here): the NIC remains a macvlan parent, so the
node kernel still processes frames from the IoT L2. Full node isolation
from IoT would require relocating how those three pods reach IoT devices
— a separate, larger change.

Same suppression pattern the repo already uses (edge-host 10-vlan40,
HA macvlan /32) rather than policy routing (Talos 1.12 has no
RoutingRuleConfig; that is 1.13+).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S5CLfizEtzHS51kxJFQgc

review-bot reviewed

2026-07-23 11:06:53 +00:00

review-bot left a comment

🤖 Automated review — model claude-opus-4-8 (complexity: complex), commit b077de8556ee39200b100b65a17bd110256c3428.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

Removes host addressing from the v30iot NIC on all three k8s nodes: dhcp: false
(no v4 address / connected 10.2.30.0/24 route) plus a per-interface sysctl
net.ipv6.conf.eth1.disable_ipv6: "1" (no v6, incl. link-local). The NIC is kept
as the macvlan parent for the IoT pod attachments. The change is small, well
motivated, and the one assumption that could break it — that v30iot is eth1 on
these nodes — checks out. No blocking issues.

🔴 Blocking

None.

🟡 Suggestions

Interface-name assumption is correct, and worth keeping honest.
k8s-1.yaml:46 (and 2/3) key the sysctl on the literal name eth1 while the
interface block selects the same NIC by busPath: "0000:06:13.0". These two
identifiers must agree. They do today: the three IoT NADs all hardcode
"master": "eth1" for this NIC and cite talosctl get links verification that
0000:06:13.0 → eth1 on k8s-{1,2,3} (matter-server/network-attachment.yaml:60-62,
otbr/network-attachment.yaml:46-48,
home-assistant/shelly-manager-network-attachment.yaml:24-29,57). The risk is
future coupling: this sysctl is now a third place that silently depends on PCI
probe order producing eth1, and unlike the NADs (which fail loudly — pod
NotReady) a wrong name here fails silently (sysctl no-ops or errors on a
non-existent iface, and the host keeps its v6). The rollout step that greps
talosctl get links for eth1 addressless-UP is the right gate; just make it a
hard stop per node.
Confirm macvlan children keep IPv6 after the parent is disabled.
matter-server/network-attachment.yaml:46-53 depends on net1 (a macvlan child
of eth1) receiving the Thread OMR-prefix RA (fd5f::/64) for CASE sessions.
Disabling disable_ipv6 on the parent should not affect a child netdev's own
IPv6 stack, so this ought to be fine — but it's the one interaction this PR
introduces that the pods actually rely on. Worth an explicit
ip -6 route/ip -6 addr check inside the matter-server pod on the first node
you apply to, alongside the cert canary already planned.
Macvlan-parent-must-stay-UP is the real soak risk (already flagged in the PR
body). With dhcp: false and no addresses, the change relies on Talos still
bringing eth1 administratively UP as a configured interface so the macvlan
parent works after a cold boot. The PR body calls this out and proposes a dummy
link: up fallback — good. Since this only manifests on reboot, make sure the
per-node acceptance includes an actual reboot, not just a live talosctl apply.

🟢 Nits

Out of scope (not in this diff, so don't block on it), but adjacent: the
shelly-manager-deployment.yaml:53 comment says "the NAD's master: eth2
lines up with v30iot" — stale/wrong; the NAD uses eth1, matching this PR.
After this change lands, that's the last lingering eth2-means-v30iot
reference; a cleanup would remove a future trap.

Tests

This is Talos machineconfig; the repo has no automated test harness for it and
none is expected. Verification is the staged, per-node rollout described in the
PR body (node stays Ready, eth1 UP + addressless, IoT pods Ready and still see
devices, cert canary from an IoT client). That plan is appropriate and matches
the risk. The only additions I'd fold into it are the two live checks above
(matter-server net1 IPv6 route; a reboot to prove addressless-UP survives).

Recommendation

Looks reasonable — no blocking issues; comments above are optional.

🤖 **Automated review** — model `claude-opus-4-8` (complexity: complex), commit `b077de8556ee39200b100b65a17bd110256c3428`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary Removes host addressing from the v30iot NIC on all three k8s nodes: `dhcp: false` (no v4 address / connected `10.2.30.0/24` route) plus a per-interface sysctl `net.ipv6.conf.eth1.disable_ipv6: "1"` (no v6, incl. link-local). The NIC is kept as the macvlan parent for the IoT pod attachments. The change is small, well motivated, and the one assumption that could break it — that v30iot is `eth1` on these nodes — checks out. No blocking issues. ## 🔴 Blocking None. ## 🟡 Suggestions - **Interface-name assumption is correct, and worth keeping honest.** `k8s-1.yaml:46` (and 2/3) key the sysctl on the literal name `eth1` while the interface block selects the same NIC by `busPath: "0000:06:13.0"`. These two identifiers must agree. They do today: the three IoT NADs all hardcode `"master": "eth1"` for this NIC and cite `talosctl get links` verification that `0000:06:13.0` → `eth1` on k8s-{1,2,3} (`matter-server/network-attachment.yaml:60-62`, `otbr/network-attachment.yaml:46-48`, `home-assistant/shelly-manager-network-attachment.yaml:24-29,57`). The risk is future coupling: this sysctl is now a third place that silently depends on PCI probe order producing `eth1`, and unlike the NADs (which fail loudly — pod NotReady) a wrong name here fails *silently* (sysctl no-ops or errors on a non-existent iface, and the host keeps its v6). The rollout step that greps `talosctl get links` for eth1 addressless-UP is the right gate; just make it a hard stop per node. - **Confirm macvlan children keep IPv6 after the parent is disabled.** `matter-server/network-attachment.yaml:46-53` depends on `net1` (a macvlan child of eth1) receiving the Thread OMR-prefix RA (`fd5f::/64`) for CASE sessions. Disabling `disable_ipv6` on the *parent* should not affect a child netdev's own IPv6 stack, so this ought to be fine — but it's the one interaction this PR introduces that the pods actually rely on. Worth an explicit `ip -6 route`/`ip -6 addr` check inside the matter-server pod on the first node you apply to, alongside the cert canary already planned. - **Macvlan-parent-must-stay-UP is the real soak risk (already flagged in the PR body).** With `dhcp: false` and no addresses, the change relies on Talos still bringing eth1 administratively UP as a configured interface so the macvlan parent works after a cold boot. The PR body calls this out and proposes a dummy `link: up` fallback — good. Since this only manifests on reboot, make sure the per-node acceptance includes an actual reboot, not just a live `talosctl apply`. ## 🟢 Nits - Out of scope (not in this diff, so don't block on it), but adjacent: the `shelly-manager-deployment.yaml:53` comment says "the NAD's `master: eth2` lines up with v30iot" — stale/wrong; the NAD uses `eth1`, matching this PR. After this change lands, that's the last lingering `eth2`-means-v30iot reference; a cleanup would remove a future trap. ## Tests This is Talos machineconfig; the repo has no automated test harness for it and none is expected. Verification is the staged, per-node rollout described in the PR body (node stays Ready, eth1 UP + addressless, IoT pods Ready and still see devices, cert canary from an IoT client). That plan is appropriate and matches the risk. The only additions I'd fold into it are the two live checks above (matter-server `net1` IPv6 route; a reboot to prove addressless-UP survives). ## Recommendation Looks reasonable — no blocking issues; comments above are optional.

claude-bot added 1 commit 2026-07-23 11:08:42 +00:00

docs(talos,shelly): flag the eth1 name dependency; fix stale eth2 comment

ci/woodpecker/pr/pr-review Pipeline was successful

Details

02c6128dc6

Review follow-ups (non-blocking):
- The disable_ipv6 sysctl keys on the literal 'eth1' while the interface
  is selected by busPath; a rename would no-op the sysctl SILENTLY and
  leave the host v6-addressed on IoT. Document the dependency and that
  the rollout must verify eth1 is addressless per node.
- shelly-manager-deployment comment said 'master: eth2' — the NAD uses
  eth1. Last lingering eth2-means-v30iot reference, corrected.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S5CLfizEtzHS51kxJFQgc

claude-bot commented

2026-07-23 11:09:01 +00:00

Review addressed in 02c6128d — no blocking items, all suggestions folded in:

🟡 eth1 name dependency (silent failure) — added an in-code NAME DEPENDENCY note at the sysctl: busPath selection can't be mirrored by a sysctl (kernel-name only), so a rename no-ops it silently and leaves the host v6-addressed on IoT. The comment makes the rollout's per-node addressless check a hard gate.

🟡 macvlan child IPv6 — agreed, and it should be unaffected (disabling v6 on the parent netdev doesn't touch a macvlan child's own stack in its own netns). Added to the rollout as an explicit check: ip -6 addr/ip -6 route inside the matter-server pod for the fd5f::/64 Thread OMR RIO on net1, on the first node applied.

🟡 addressless-UP survives reboot — rollout now requires an actual reboot per node (not just live apply) before proceeding, with the dummy link: up fallback ready if Talos won't bring an addressless NIC up.

🟢 nit (stale eth2) — fixed; shelly-manager-deployment now reads master: eth1, matching the NAD. That was the last eth2-means-v30iot reference.

Updating the PR body's rollout section with the two added live checks.

🤖 Generated with Claude Code

Review addressed in `02c6128d` — no blocking items, all suggestions folded in: **🟡 eth1 name dependency (silent failure)** — added an in-code NAME DEPENDENCY note at the sysctl: busPath selection can't be mirrored by a sysctl (kernel-name only), so a rename no-ops it silently and leaves the host v6-addressed on IoT. The comment makes the rollout's per-node addressless check a hard gate. **🟡 macvlan child IPv6** — agreed, and it should be unaffected (disabling v6 on the parent netdev doesn't touch a macvlan child's own stack in its own netns). Added to the rollout as an explicit check: `ip -6 addr`/`ip -6 route` inside the matter-server pod for the `fd5f::/64` Thread OMR RIO on `net1`, on the first node applied. **🟡 addressless-UP survives reboot** — rollout now requires an actual reboot per node (not just live apply) before proceeding, with the dummy `link: up` fallback ready if Talos won't bring an addressless NIC up. **🟢 nit (stale eth2)** — fixed; `shelly-manager-deployment` now reads `master: eth1`, matching the NAD. That was the last eth2-means-v30iot reference. Updating the PR body's rollout section with the two added live checks. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

review-bot requested changes

2026-07-23 11:11:23 +00:00

review-bot left a comment

🤖 Automated review — model claude-opus-4-8 (complexity: complex), commit 02c6128dc6e751f06e6184ecf089f96ba295ef66.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

This PR removes host-level v30iot addressing (v4 DHCP + v6 RA) from the three
worker nodes by switching the 0000:06:13.0 NIC to dhcp: false and disabling
IPv6 on eth1 via machine.sysctls, keeping the NIC up only as a macvlan
parent. The security rationale and the hairpin/DNAT diagnosis are sound and the
comment cleanup (eth2→eth1) is correct. However, the change overlooks
music-assistant, which is a hostNetwork pod pinned to exactly these three
nodes and hard-depends on a v30iot host address — this will break it.

🔴 Blocking

music-assistant fails to start on every eligible node after this change.

music-assistant runs with hostNetwork: true
(kubernetes/apps/music-assistant/deployment.yaml:30) and, per the comment at
deployment.yaml:33-38, carries no edge toleration — so the
workload-class=pinned:NoSchedule taint keeps it off k8s-edge and it can
only schedule on k8s-{1,2,3}.
Its reconcile-publish-ip initContainer runs
kubernetes/apps/music-assistant/configmap-net-reconcile.yaml, which walks the
host interfaces (SIOCGIFADDR, configmap-net-reconcile.yaml:57-66) looking
for an address inside 10.2.30.0/24, and fails closed when none is found:
return 1 with "FATAL: no address ... refusing to start"
(configmap-net-reconcile.yaml:70-77).
This PR removes precisely that host address on all three nodes
(k8s-1.yaml:33, k8s-2.yaml:14, k8s-3.yaml:14: dhcp: false). Because MA
cannot land on edge, there is no node left where the initContainer can find a
10.2.30.0/24 host address, so MA will crash-loop on init after rollout.

This is not just an init-time check: MA's whole design here needs the host on the
IoT L2 with an address to advertise as publish_ip to Snapcast/SlimProto/Sendspin
players (deployment.yaml:20-29, configmap-net-reconcile.yaml:13-22). Unlike
matter-server/otbr/shelly-manager — which attach via macvlan children in
their own netns and genuinely never needed host addressing — MA consumes the host
address directly. The PR body enumerates only those three pods as the v30iot
attachments and does not account for MA.

Options: migrate MA to a macvlan-child attachment on v30iot (as the other three
pods already do) so it gets its own leased IoT address, or keep host addressing on
one designated node and pin MA there, or otherwise resolve MA's publish_ip
source before removing host addressing. As written, the rollout's step 1 (apply to
k8s-1) will surface this the moment MA reschedules onto a converted node.

🟡 Suggestions

Verify the macvlan parent stays UP with dhcp: false. The PR body already
flags this as a reboot/soak risk. Worth confirming on k8s-1 that eth1 comes up
UP addressless after a cold reboot before converting k8s-2/3, since all three
IoT NADs (master: eth1) and MA's L2 adjacency silently depend on the parent
being up; if Talos won't bring an addressless NIC up, the link: up fallback
the body mentions is needed.

🟢 Nits

The eth1/eth2 comment fix in
shelly-manager-deployment.yaml:53 correctly matches the NAD
(shelly-manager-network-attachment.yaml:57, master: eth1) — good catch.

Tests

This is Talos machineconfig; there is no unit-test surface in-repo and the PR
correctly documents a manual staged rollout. The staged plan is reasonable, but it
should be extended to explicitly assert that a music-assistant pod scheduled onto
each converted node reaches Ready (not just matter/otbr/shelly) — that is the check
that would have caught the blocking issue above.

Recommendation

Request changes — 1 blocking issue (music-assistant loses its required v30iot host
address and fails closed on every node it can schedule on).

🤖 **Automated review** — model `claude-opus-4-8` (complexity: complex), commit `02c6128dc6e751f06e6184ecf089f96ba295ef66`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary This PR removes host-level v30iot addressing (v4 DHCP + v6 RA) from the three worker nodes by switching the `0000:06:13.0` NIC to `dhcp: false` and disabling IPv6 on `eth1` via `machine.sysctls`, keeping the NIC up only as a macvlan parent. The security rationale and the hairpin/DNAT diagnosis are sound and the comment cleanup (`eth2`→`eth1`) is correct. However, the change overlooks `music-assistant`, which is a `hostNetwork` pod pinned to exactly these three nodes and hard-depends on a v30iot **host** address — this will break it. ## 🔴 Blocking **`music-assistant` fails to start on every eligible node after this change.** - `music-assistant` runs with `hostNetwork: true` (`kubernetes/apps/music-assistant/deployment.yaml:30`) and, per the comment at `deployment.yaml:33-38`, carries **no** edge toleration — so the `workload-class=pinned:NoSchedule` taint keeps it off `k8s-edge` and it can only schedule on `k8s-{1,2,3}`. - Its `reconcile-publish-ip` initContainer runs `kubernetes/apps/music-assistant/configmap-net-reconcile.yaml`, which walks the host interfaces (`SIOCGIFADDR`, `configmap-net-reconcile.yaml:57-66`) looking for an address inside `10.2.30.0/24`, and **fails closed** when none is found: `return 1` with "FATAL: no address ... refusing to start" (`configmap-net-reconcile.yaml:70-77`). - This PR removes precisely that host address on all three nodes (`k8s-1.yaml:33`, `k8s-2.yaml:14`, `k8s-3.yaml:14`: `dhcp: false`). Because MA cannot land on edge, there is no node left where the initContainer can find a `10.2.30.0/24` host address, so MA will crash-loop on init after rollout. This is not just an init-time check: MA's whole design here needs the host on the IoT L2 with an address to advertise as `publish_ip` to Snapcast/SlimProto/Sendspin players (`deployment.yaml:20-29`, `configmap-net-reconcile.yaml:13-22`). Unlike matter-server/otbr/shelly-manager — which attach via macvlan **children** in their own netns and genuinely never needed host addressing — MA consumes the host address directly. The PR body enumerates only those three pods as the v30iot attachments and does not account for MA. Options: migrate MA to a macvlan-child attachment on v30iot (as the other three pods already do) so it gets its own leased IoT address, or keep host addressing on one designated node and pin MA there, or otherwise resolve MA's `publish_ip` source before removing host addressing. As written, the rollout's step 1 (apply to k8s-1) will surface this the moment MA reschedules onto a converted node. ## 🟡 Suggestions - **Verify the macvlan parent stays UP with `dhcp: false`.** The PR body already flags this as a reboot/soak risk. Worth confirming on k8s-1 that `eth1` comes up `UP` addressless after a cold reboot before converting k8s-2/3, since all three IoT NADs (`master: eth1`) and MA's L2 adjacency silently depend on the parent being up; if Talos won't bring an addressless NIC up, the `link: up` fallback the body mentions is needed. ## 🟢 Nits - The `eth1`/`eth2` comment fix in `shelly-manager-deployment.yaml:53` correctly matches the NAD (`shelly-manager-network-attachment.yaml:57`, `master: eth1`) — good catch. ## Tests This is Talos machineconfig; there is no unit-test surface in-repo and the PR correctly documents a manual staged rollout. The staged plan is reasonable, but it should be extended to explicitly assert that a `music-assistant` pod scheduled onto each converted node reaches Ready (not just matter/otbr/shelly) — that is the check that would have caught the blocking issue above. ## Recommendation Request changes — 1 blocking issue (music-assistant loses its required v30iot host address and fails closed on every node it can schedule on).

claude-bot added 1 commit 2026-07-23 11:31:52 +00:00

fix(music-assistant): macvlan child on v30iot instead of hostNetwork

ci/woodpecker/pr/pr-review Pipeline was successful

Details

5a9693bdd8

Unblocks the node-unaddressing change: MA ran hostNetwork and its
reconcile initContainer fails closed if no 10.2.30.0/24 address exists
on a host interface — removing node host addressing would crash-loop it
on every node it can schedule to (no edge toleration).

MA now attaches net1 via macvlan on v30iot (DHCP-leased), exactly like
matter-server/otbr/shelly-manager: its own IoT address, direct L2
adjacency for Snapcast/SlimProto/Sendspin audio sync (macvlan bridge
mode puts frames straight on the wire — same property hostNetwork gave,
without the node holding an IoT address). The reconcile script is
unchanged in logic: it scans the pod's interfaces for the 10.2.30.0/24
address and now finds net1 (Multus attaches it with its lease before any
container runs) instead of the host eth1. bind_ip stays 0.0.0.0 -> binds
eth0 + net1; publish_ip = net1's leased IP. NAD omits matter's Thread
accept_ra tuning (MA has no Thread); keeps sbr (leased default gateway
would otherwise blackhole cluster-internal replies).

Folded into this PR so the change is atomic: on merge Flux migrates MA
to its own IoT lease before the manual node-unaddressing rollout removes
the host addresses.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S5CLfizEtzHS51kxJFQgc

claude-bot commented

2026-07-23 11:32:26 +00:00

Blocker fixed in 5a9693bd — thank you, this was a real one I missed (and I compounded it by not reading this re-review until prompted).

You're right on every point: MA is hostNetwork, has no edge toleration, and its reconcile initContainer fails closed without a 10.2.30.0/24 host address — so unaddressing the nodes would crash-loop it everywhere it can land.

Fix: MA migrates to the macvlan-child pattern the other three v30iot pods already use, rather than keeping a node addressed (which would retreat from the security goal). Added network-attachment.yaml (macvlan on eth1 + DHCP + sbr; no Thread accept_ra tuning — MA doesn't speak Thread), dropped hostNetwork/dnsPolicy, added the Multus annotation. The reconcile script is unchanged in logic — it scans the pod's interfaces for the v30iot address and now finds net1 (Multus attaches it with its lease before any container runs) instead of the host eth1; bind_ip stays 0.0.0.0 (binds eth0 + net1), publish_ip = net1's leased IP.

Folded into this PR so it's atomic: on merge Flux migrates MA to its own IoT lease before the manual node-unaddressing rollout removes the host addresses — no window where MA is broken.

Rollout gets one more per-node gate: a music-assistant pod scheduled onto the converted node reaches Ready (the check that would have caught this). Two honest risks to watch on first apply, since this is finicky audio: (1) Snapcast/Sendspin audio sync over macvlan-bridge vs hostNetwork — should be equivalent direct-L2, but wants a real listen test; (2) the reconcile initContainer depends on the DHCP lease completing during Multus attach — verify net1 has its address at init time.

🤖 Generated with Claude Code

**Blocker fixed in `5a9693bd`** — thank you, this was a real one I missed (and I compounded it by not reading this re-review until prompted). You're right on every point: MA is `hostNetwork`, has no edge toleration, and its reconcile initContainer fails closed without a `10.2.30.0/24` host address — so unaddressing the nodes would crash-loop it everywhere it can land. **Fix: MA migrates to the macvlan-child pattern** the other three v30iot pods already use, rather than keeping a node addressed (which would retreat from the security goal). Added `network-attachment.yaml` (macvlan on eth1 + DHCP + sbr; no Thread `accept_ra` tuning — MA doesn't speak Thread), dropped `hostNetwork`/`dnsPolicy`, added the Multus annotation. The reconcile script is unchanged in logic — it scans the pod's interfaces for the v30iot address and now finds `net1` (Multus attaches it with its lease before any container runs) instead of the host eth1; `bind_ip` stays 0.0.0.0 (binds eth0 + net1), `publish_ip` = net1's leased IP. Folded into this PR so it's atomic: on merge Flux migrates MA to its own IoT lease **before** the manual node-unaddressing rollout removes the host addresses — no window where MA is broken. Rollout gets one more per-node gate: **a `music-assistant` pod scheduled onto the converted node reaches Ready** (the check that would have caught this). Two honest risks to watch on first apply, since this is finicky audio: (1) Snapcast/Sendspin audio sync over macvlan-bridge vs hostNetwork — should be equivalent direct-L2, but wants a real listen test; (2) the reconcile initContainer depends on the DHCP lease completing during Multus attach — verify net1 has its address at init time. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

review-bot reviewed

2026-07-23 11:36:19 +00:00

review-bot left a comment

🤖 Automated review — model claude-opus-4-8 (complexity: complex), commit 5a9693bdd8371f520c841f7ca106713c777cf885.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

Removes all host addressing (v4 DHCP + v6 RA) from the v30iot NIC on k8s-{1,2,3},
keeping it as a link-only macvlan parent, and migrates music-assistant off
hostNetwork onto a macvlan NetworkAttachmentDefinition so it no longer depends
on the node holding a v30iot address. The change is well-reasoned, closely mirrors
the already-proven matter-server / shelly-manager NADs, and the code I could
verify is internally consistent. No blocking issues found — the real risks are
operational (Talos link-up behavior, IPv6-on-parent) and are explicitly gated by
the PR's staged rollout, which CI does not execute.

🟡 Suggestions

Talos bringing an addressless NIC UP is the load-bearing assumption, and it's
unverifiable from the repo. With dhcp: false and no addresses
(kubernetes/talos/patches/k8s-1.yaml:33), the whole design depends on Talos
still bringing eth1 link-UP so the macvlan children get frames. I confirmed
this matters cluster-wide: matter-server, otbr, shelly-manager and now
music-assistant all float across k8s-{1,2,3} with no node pinning
(matter-server/deployment.yaml:34, otbr/deployment.yaml:43), so every node's
eth1 must come up UP-but-addressless or whichever v30iot pod lands there breaks.
Your rollout step 5 (reboot k8s-1, confirm eth1 UP, dummy link: up fallback)
is exactly the right gate — just flagging that this is the single point that a
code review cannot clear for you; it has to be the reboot test.
IPv6 disable on the macvlan parent vs. the children's own v6 stacks.
net.ipv6.conf.eth1.disable_ipv6: "1" (k8s-1.yaml:44) targets the parent only;
macvlan children get their v6 config from net.ipv6.conf.default.* at creation,
so matter-server/otbr's Thread OMR (fd5f::/64) route on net1 should
survive. This is standard kernel behavior, but it's an assumption worth the
explicit check you already list (rollout step 4). No change needed — noting it's
the second thing that only the live test settles.
music-assistant binds 0.0.0.0, so its web/admin surface (:8095) is
reachable on net1 from the IoT VLAN, unlike matter-server and
shelly-manager, which deliberately pin their control API to the pod eth0
address precisely because macvlan bypasses Cilium policy
(matter-server/deployment.yaml:65, shelly-manager-deployment.yaml:86). This
is not a regression — under the old hostNetwork MA already bound the host's
v30iot eth1, so :8095 was equally exposed — and the audio servers
(Sendspin/Snapcast/SlimProto) must bind net1, so a blanket pin isn't possible.
Still, since this PR is framed around shrinking the IoT trust surface, it's worth
a follow-up thought on whether MA's HTTP admin port specifically could be pinned
to eth0 the way the two sibling pods are. Out of scope here.

🟢 Nits

The reconcile timing claim checks out: net1 (with its DHCP lease from the
dhcp IPAM plugin) is set up during CNI ADD before any container — including
initContainers — starts, so reconcile.py's interface scan
(configmap-net-reconcile.yaml:59-68, if_nameindex() → match 10.2.30.0/24)
will find net1's address. The fail-closed path is preserved. Good.
shelly-manager-deployment.yaml:9 comment fix eth2 → eth1 is correct — the
actual NAD master is eth1 (shelly-manager-network-attachment.yaml:57).
Removing dnsPolicy: ClusterFirstWithHostNet alongside hostNetwork
(deployment.yaml) is the right call: a normal pod defaults to ClusterFirst.

Tests

No automated tests accompany this change, which is consistent with the repo — these
are Talos machineconfig patches and Multus NADs applied out-of-band by talosctl,
not by CI/Flux, so they can't be unit-tested here. The PR compensates with a
detailed per-node manual verification battery (Ready state, talosctl get links
addressless check, child pod reachability, child IPv6 intact, cold-reboot
re-confirm, IoT cert canary). That is the correct form of "test" for this class of
change; the critical gates (addressless-but-UP after reboot; child v6 survives) are
the ones that must not be skipped when rolling k8s-2 and k8s-3.

Recommendation

Looks reasonable — no blocking issues; comments above are optional. The design is
sound and mirrors proven siblings; residual risk is operational and correctly gated
by the staged rollout.

🤖 **Automated review** — model `claude-opus-4-8` (complexity: complex), commit `5a9693bdd8371f520c841f7ca106713c777cf885`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary Removes all host addressing (v4 DHCP + v6 RA) from the v30iot NIC on `k8s-{1,2,3}`, keeping it as a link-only macvlan parent, and migrates `music-assistant` off `hostNetwork` onto a macvlan `NetworkAttachmentDefinition` so it no longer depends on the node holding a v30iot address. The change is well-reasoned, closely mirrors the already-proven `matter-server` / `shelly-manager` NADs, and the code I could verify is internally consistent. **No blocking issues found** — the real risks are operational (Talos link-up behavior, IPv6-on-parent) and are explicitly gated by the PR's staged rollout, which CI does not execute. ## 🟡 Suggestions - **Talos bringing an addressless NIC UP is the load-bearing assumption, and it's unverifiable from the repo.** With `dhcp: false` and no `addresses` (`kubernetes/talos/patches/k8s-1.yaml:33`), the whole design depends on Talos still bringing `eth1` link-UP so the macvlan children get frames. I confirmed this matters cluster-wide: `matter-server`, `otbr`, `shelly-manager` and now `music-assistant` all float across `k8s-{1,2,3}` with no node pinning (`matter-server/deployment.yaml:34`, `otbr/deployment.yaml:43`), so *every* node's `eth1` must come up UP-but-addressless or whichever v30iot pod lands there breaks. Your rollout step 5 (reboot k8s-1, confirm eth1 UP, dummy `link: up` fallback) is exactly the right gate — just flagging that this is the single point that a code review cannot clear for you; it has to be the reboot test. - **IPv6 disable on the macvlan parent vs. the children's own v6 stacks.** `net.ipv6.conf.eth1.disable_ipv6: "1"` (`k8s-1.yaml:44`) targets the parent only; macvlan children get their v6 config from `net.ipv6.conf.default.*` at creation, so `matter-server`/`otbr`'s Thread OMR (`fd5f::/64`) route on `net1` should survive. This is standard kernel behavior, but it's an assumption worth the explicit check you already list (rollout step 4). No change needed — noting it's the second thing that only the live test settles. - **`music-assistant` binds `0.0.0.0`, so its web/admin surface (`:8095`) is reachable on `net1` from the IoT VLAN**, unlike `matter-server` and `shelly-manager`, which deliberately pin their control API to the pod `eth0` address precisely because macvlan bypasses Cilium policy (`matter-server/deployment.yaml:65`, `shelly-manager-deployment.yaml:86`). This is **not a regression** — under the old `hostNetwork` MA already bound the host's v30iot `eth1`, so `:8095` was equally exposed — and the audio servers (Sendspin/Snapcast/SlimProto) *must* bind `net1`, so a blanket pin isn't possible. Still, since this PR is framed around shrinking the IoT trust surface, it's worth a follow-up thought on whether MA's HTTP admin port specifically could be pinned to `eth0` the way the two sibling pods are. Out of scope here. ## 🟢 Nits - The reconcile timing claim checks out: `net1` (with its DHCP lease from the `dhcp` IPAM plugin) is set up during CNI ADD before any container — including initContainers — starts, so `reconcile.py`'s interface scan (`configmap-net-reconcile.yaml:59-68`, `if_nameindex()` → match `10.2.30.0/24`) will find `net1`'s address. The fail-closed path is preserved. Good. - `shelly-manager-deployment.yaml:9` comment fix `eth2` → `eth1` is correct — the actual NAD master is `eth1` (`shelly-manager-network-attachment.yaml:57`). - Removing `dnsPolicy: ClusterFirstWithHostNet` alongside `hostNetwork` (`deployment.yaml`) is the right call: a normal pod defaults to `ClusterFirst`. ## Tests No automated tests accompany this change, which is consistent with the repo — these are Talos machineconfig patches and Multus NADs applied out-of-band by `talosctl`, not by CI/Flux, so they can't be unit-tested here. The PR compensates with a detailed per-node manual verification battery (Ready state, `talosctl get links` addressless check, child pod reachability, child IPv6 intact, cold-reboot re-confirm, IoT cert canary). That is the correct form of "test" for this class of change; the critical gates (addressless-but-UP after reboot; child v6 survives) are the ones that must not be skipped when rolling k8s-2 and k8s-3. ## Recommendation Looks reasonable — no blocking issues; comments above are optional. The design is sound and mirrors proven siblings; residual risk is operational and correctly gated by the staged rollout.

k.renner merged commit 44c0abd484 into master

2026-07-23 11:42:10 +00:00

k.renner referenced this issue from a commit

2026-07-23 11:42:12 +00:00

Merge pull request 'Remove k8s node host addressing on the IoT VLAN (security + hairpin fix)' (#369) from claude-bot/iot-nic-unaddress into master

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#369

Remove k8s node host addressing on the IoT VLAN (security + hairpin fix) #369

Change

Residual risk (explicitly NOT solved here)

Rollout (Talos machineconfig — applied by talosctl, not CI) — STAGED, per node

Summary

🔴 Blocking

🟡 Suggestions

🟢 Nits

Tests

Recommendation

Summary

🔴 Blocking

🟡 Suggestions

🟢 Nits

Tests

Recommendation

Summary

🟡 Suggestions

🟢 Nits

Tests

Recommendation

Rollout (Talos machineconfig — applied by `talosctl`, not CI) — STAGED, per node