Stable predictable interface names (net.ifnames=1) — fixes the v30iot NIC shuffle #375
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: konairius/homelab#375
Reference in New Issue
Block a user
Root cause
The k8s nodes' interface names (
eth0/1/2) shuffled across reboots. Cause: the Talos nocloud platform driver unconditionally injectsnet.ifnames=0, disabling systemd predictable naming — so names followed driver-probe completion order, a race that mixed-driver VMs (two virtio NICs + one i40e PCI-passthrough) hit reliably. A k8s-3 reboot renamed the v30iot NICeth1→eth2and mgmt→eth1, which silently (a) pointed the v6-disable sysctl at the mgmt NIC (broke node IPv6) and (b) attached macvlan children to the Server VLAN (crashed music-assistant, silently mis-attached matter).This was verified by three independent source-traced research passes:
LinkAliasConfigcannot fix it — it sets a cosmeticIFLA_IFALIAS, invisible to both the sysctl path and CNImaster.net.ifnames=0names by probe order (not PCI order), so a hypervisor slot-pin can't fix it either.machine.install.extraKernelArgsis silently ignored — the cmdline is baked into the image).Fix
New schematic
88e3752d…= current +extraKernelArgs: [-net.ifnames=0, net.ifnames=1], delivered viatalosctl upgrade. Predictable busPath-derived names are topology-stable and identical across the three identical VMs:ens19ens18enp2s0np1Re-keys onto
ens19: the four v30iot macvlan NADs (matter/otbr/shelly/music-assistantmaster) and the per-node v6-disable sysctl. Bumps the schematic id in.woodpecker/deploy-infrastructure.yml.Applied live (this branch documents the already-applied state)
Rolling
talosctl upgrade, one node at a time with full Ceph HEALTH_OK recovery between each (k8s-3 → k8s-1 → k8s-2). Verified after:ens19identical on all three, the v30iot NIC addressless v4+v6 on every node, macvlan children retain their own v6 (matter net1 has fd30:: + GUA), all v30iot pods Running onens19, Ceph HEALTH_OK.Follow-up (not in this PR)
The names resolved to the slot-based
ensNtier (hotplug slots are exposed). A QEMU machine-type bump could flipens↔enp; pinmachine = pc-q35-<version>in opentofu to eliminate even that. Tracked separately.🤖 Generated with Claude Code
https://claude.ai/code/session_014S5CLfizEtzHS51kxJFQgc
🤖 Automated review — model
claude-sonnet-5(complexity: normal), commit4ee087ec5b3fa527d0d7eeaf0551d409a9e2519e.Summary
This PR re-enables systemd predictable interface naming (
net.ifnames=1) via a new Talos Image Factory schematic, and renames allmaster: eth1references (four v30iot macvlan NADs) plus the per-node IPv6-disable sysctl to the new stable nameens19, matching the busPath→name table in the PR description. The rename itself is applied consistently across all four NADs and the three node patches, and the already-upgraded live cluster is documented as verified. However, the CI step that provisions this image onto Proxmox nodes has a stale-cache bug that would silently undo the fix for any future VM (re)provisioning.🔴 Blocking
.woodpecker/deploy-infrastructure.yml:133-144— image-exists check doesn't account for the schematic change, so future provisioning will silently use the old (bugged) image.TALOS_FILENAMEis derived only fromTALOS_VERSION(talos-v1.12.6-nocloud-amd64.raw), which is unchanged by this PR — onlyTALOS_SCHEMATICchanged (line 134). The "ensure image exists" loop (lines 138-144) checks Proxmox storage for a file with that name and skips downloading if found (EXISTS -gt 0→continue). Since jupiter/saturn/neptune already havetalos-v1.12.6-nocloud-amd64.rawcached from the old schematic (the one this PR is replacing — the cluster was live-upgraded viatalosctl upgrade, not reimaged), this check will find the stale file and never fetch the new-schematic image.opentofu/talos_image.tf:1-4(talos_filename = "talos-${local.talos_version}-nocloud-amd64.raw", unrelated to schematic) then hands that same stale local file to any VM create/replace. Net effect: a futuretofu applythat creates or replaces a k8s-1/2/3 VM (or any node using this image, e.g. viaTOFU_REPLACE/TOFU_TARGET) would provision from the oldnet.ifnames=0image, silently reintroducing exactly the interface-shuffle bug this PR fixes. Fix by keying the cached filename/existence check on the schematic ID (e.g. embed the schematic hash inTALOS_FILENAME, or delete-then-reupload on schematic change) so a schematic bump forces a re-download, and updateopentofu/talos_image.tfto match.🟡 Suggestions
kubernetes/apps/music-assistant/configmap-net-reconcile.yaml:20-22now contradicts the comment this PR edited indeployment.yaml:43-44. Before this PR, both files carried the same explanation ("MA's auto-detect picks the mgmt default-route IP ...eth1/v30iot has routeMetric 4096, never the default route"). This PR rewrote thedeployment.yamlcopy to a different claim ("MA's auto-detect picks the pod eth0 (cluster CIDR) address") but left theconfigmap-net-reconcile.yamldocstring with the original routeMetric/eth1 explanation. The two comments, describing the same fact for the same code path, now disagree — worth reconciling so a future reader debuggingreconcile.pyisn't misled by whichever one they read first.kubernetes/talos/patches/cluster.yaml:22-24("...depending on interface ordering can land on eth1/v30iot...") wasn't touched by this PR and is now doubly stale: the interface isens19, noteth1, and — more importantly — the "interface ordering" ambiguity this comment warns about is precisely the race this PR eliminates. Worth a follow-up touch-up for consistency, though it's a comment in an untouched file so not blocking here.kubernetes/apps/matter-server/deployment.yaml:24-25keeps the pre-existing claim thatens19is "present on every node (k8s-1/2/3 and k8s-edge alike)". Per the PR description, thetalosctl upgraderollout covered only k8s-3 → k8s-1 → k8s-2; nok8s-edge.yamlTalos patch exists inkubernetes/talos/patches/and k8s-edge doesn't appear in the busPath table. It's likely inconsequential in practice — theworkload-class=pinned:NoScheduletaint (no toleration here) keeps this pod off k8s-edge regardless — but the comment's factual claim about k8s-edge's interface naming wasn't verified by this PR and may not hold if k8s-edge is on a different image/hardware topology.🟢 Nits
None beyond the above.
Tests
This is a GitOps/infra config repo with no automated test coverage for Talos machine-config or NAD YAML; that matches the repo's existing pattern (no kustomize/kubeconform lint step in
.woodpecker/touches these paths). The PR body documents manual, live verification of the actual rollout (interface names, addresslessness, pod health, Ceph status) — reasonable for this kind of change. The blocking issue above is exactly the class of failure mode that manual verification of the already-upgraded cluster cannot catch, since it only manifests on the next fresh provision/replace.Recommendation
Request changes — 1 blocking issue.
Blocker fixed in
7eb2e02b— genuine catch, thank you. This was exactly the failure mode live verification of the upgraded cluster can't see.🔴 Image cache keyed on schematic —
TALOS_FILENAMEderived from version only, so bumping the schematic left the cached filename unchanged; the Proxmox exists-check would find the stale old-schematic file and skip the download, reintroducing net.ifnames=0 on the next VM (re)provision. Fixed by embedding the schematic hash in the filename in BOTH places (pipelineTALOS_FILENAMEandopentofu/talos_image.tf), kept in sync — verified they produce the identical nametalos-v1.12.6-88e3752db032-nocloud-amd64.raw, so a schematic bump now forces a fresh download. tofu validates.🟡 All three reconciled:
configmap-net-reconcile.yamldocstring still described the hostNetwork-era auto-detect (mgmt default-route / eth1 routeMetric) — updated to match the deployment.yaml macvlan reality (auto-detect picks pod eth0; v30iot lives on net1).cluster.yamlkubelet nodeIP comment: eth1/interface-ordering → ens19 (and the ordering race is what this PR eliminates).🤖 Generated with Claude Code
🤖 Automated review — model
claude-sonnet-5(complexity: normal), commit7eb2e02b1dc163533ffdeb598c51d890683d5039.Summary
This PR re-enables systemd predictable network interface naming (
net.ifnames=1) via a new Talos Image Factory schematic, and re-keys every place that hard-coded the old driver-probe-order nameeth1(four macvlan NADmasterfields, one per-node IPv6-disable sysctl, and assorted comments) onto the new stable nameens19. It also fixes a latent image-caching bug: the Talos image filename now embeds the schematic hash, so a schematic bump forces a fresh download/import instead of silently reusing a stale cached image under the old fixed filename. All functional value changes (master: eth1→master: ens19,net.ipv6.conf.eth1...→net.ipv6.conf.ens19..., schematic IDs, filenames) are consistent across every file that needed to change — I verified all four NAD files, all three per-node Talos patches,cluster.yaml, the CI script, and the opentofu locals. No blocking issues found. There is one pre-existing but now-incongruous comment left half-updated (see Suggestions).🟡 Suggestions
kubernetes/apps/home-assistant/shelly-manager-network-attachment.yaml:24-29— this PR updated the closing sentence of this comment block (line 35: "master: ens19... is used here") but left the preceding paragraph untouched. It still asserts, in the present tense, "Multus'smasteris an interface name. On every cluster node v30iot lands oneth1: k8s-edge has only two NICs (mgmt=eth0, v30iot=eth1) and on k8s-{1,2,3} the PCI probe order is mgmt virtio (eth0) → v30iot virtio (eth1) → X710 i40e (eth2)." That is now stale — it describes the exact probe-order naming this PR removes, and directly contradicts this PR's own root-cause writeup (probe-order naming is what caused the NIC shuffle bug). It's comment-only (the actualmaster: ens19value at line 57 is correct), but worth tightening so the file doesn't read as self-contradictory to the next person debugging this.kubernetes/apps/matter-server/network-attachment.yaml:2-4andkubernetes/apps/otbr/network-attachment.yaml:2-3have a similar, pre-existing (not introduced by this PR) header comment describing the NAD as attached to "k8s-edge's v30iot interface (eth1)", which already contradicted the deployment.yaml comments (both pods are now pinned off k8s-edge). Out of scope for this PR, but since you're doing a naming-comment sweep here, it'd be a cheap add-on to fix while the context is fresh.Tests
No automated tests exist for this repo's Talos/Kubernetes manifests or CI script (consistent with the rest of the codebase — this is GitOps config, not application code). The PR body documents that the change was validated by a live rolling
talosctl upgradeacross all three nodes with verification ofens19naming, addressless v4/v6 on the v30iot NIC, macvlan child connectivity, and Ceph HEALTH_OK — that's the appropriate verification method for this kind of change and I have no way to independently re-run it from a static checkout. One thing I could not verify statically: the busPath→name mapping (0000:06:13.0→ens19,0000:06:12.0→ens18) depends on the QEMU-exposed PCI slot/hotplug index, which is asserted as empirically verified in the PR body rather than derivable from the diff alone — reasonable to trust given the described live testing, but flagging it as unverified-by-me for the record.Recommendation
Looks reasonable — no blocking issues; comments above are optional.
🤖 Automated review — model
claude-sonnet-5(complexity: normal), commitb762cc79417001bdbaf1abc3ba279ee7eec3de2c.Summary
This PR re-enables systemd predictable interface naming (
net.ifnames=1) via a new Talos Image Factory schematic, and re-keys the four v30iot macvlan NADs plus the per-node IPv6-disable sysctl from the old driver-probe-order nameeth1to the new stable busPath-derived nameens19. Theopentofu/talos_image.tfand.woodpecker/deploy-infrastructure.ymlschematic/filename changes are consistent with each other. There is one concrete bug in the CI script change, and the doc "sweep" the last commit claims to finish actually missed a few files.🔴 Blocking
.woodpecker/deploy-infrastructure.yml:136—TALOS_FILENAME="talos-$TALOS_VERSION-${TALOS_SCHEMATIC:0:12}-nocloud-amd64.raw"uses bash-only substring parameter expansion (${var:offset:length}). This step runs in theapplyjob onimage: gitea.renner.uno/gitea_admin/ci-tofu:2026-04-29, which is builtFROM alpine:3.21(ci/images/tofu/Containerfile:1) — even though the image installsbashviaapk add, Woodpecker executescommands:through the image's default/bin/sh, which on Alpine is busybox ash, not bash. Busybox ash does not support${var:offset:length}and will fail with "Bad substitution", breaking the Talos-image-existence-check step the first time this pipeline runs withTOFU_APPLY=true.This repo has already hit and documented this exact ash-vs-bash gap twice elsewhere:
.woodpecker/security-scan.yml:97("busybox ash's tracer (set -x) misparsed...") and.woodpecker/issue-fix.yml:97("has no portable string-< in busybox ash"), and.woodpecker/security-scan.yml:79deliberately uses the POSIX-portablecut -c1-7instead of bash substring syntax for the same "first N chars of a string" task. Fix: replace${TALOS_SCHEMATIC:0:12}with something ash-portable, e.g.$(echo "$TALOS_SCHEMATIC" | cut -c1-12).Note:
opentofu/talos_image.tf:11'ssubstr(local.talos_schematic, 0, 12)is fine — that's Terraform HCL, not shell, and is unaffected.🟡 Suggestions
eth1name as the host's v30iot interface were missed and are now stale/incorrect:kubernetes/apps/music-assistant/network-attachment.yaml:3— "interface (eth1 — the connected 10.2.30.0/24 on every k8s-{1,2,3}/edge worker)"kubernetes/apps/music-assistant/network-attachment.yaml:14— "the same way it used to discover the host's eth1 address"kubernetes/apps/otbr/network-attachment.yaml:27— "...doesn't leak through eth1."kubernetes/apps/matter-server/network-attachment.yaml:26— "...doesn't leak through eth1."These are all comment-only (no functional effect — the actual
master:JSON fields in these files were correctly updated toens19), but given the PR is specifically about correcting this NIC-naming class of stale documentation, and the last commit explicitly claims the sweep is finished, these are worth a follow-up fix for consistency with the sibling fileshelly-manager-network-attachment.yaml, which got the equivalent header fully rewritten.🟢 Nits
Tests
No automated tests apply to this change (Talos machine config / NAD manifests / CI pipeline script are declarative infra, and this repo doesn't appear to have a manifest-linting step in the diff). The PR body documents a real rolling-upgrade verification already performed against the live cluster (
ens19on all three nodes, addressless v4/v6 on the parent, macvlan children retaining v6, Ceph HEALTH_OK), which is reasonable evidence for the Talos/NAD side. The CI script bug above (bash substring syntax) would only surface on the next manualapplyrun since it wasn't exercised as part of that live verification — worth a quick manual dry run of that step (or switching to thecut -c1-12form) before relying on it.Recommendation
Request changes — 1 blocking issue.
Review blocker: my previous filename fix used bash substring syntax ${TALOS_SCHEMATIC:0:12}, but the apply step runs under busybox ash (Alpine /bin/sh) which errors 'Bad substitution' — the repo already uses cut for this exact reason. Replaced with $(echo "$TALOS_SCHEMATIC" | cut -c1-12). The HCL substr() in talos_image.tf is unaffected (not shell). Also finishes the comment sweep the prior commit claimed but didn't: 4 more stale 'eth1' host-NIC references in the matter/otbr/music-assistant NAD headers are now ens19. Grep confirms zero eth1 left in the v30iot NAD/deployment files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014S5CLfizEtzHS51kxJFQgc🤖 Automated review — model
claude-sonnet-5(complexity: normal), commit3939fe51f6a4cacd07bb093b4fa07b7be1545fd1.Summary
This PR re-keys the four v30iot macvlan attachments (matter-server, otbr, shelly-manager, music-assistant), the per-node IPv6-disable sysctl, and the Talos image schematic from the driver-probe-order name
eth1to the busPath-derived predictable nameens19, bumping the Image Factory schematic in both.woodpecker/deploy-infrastructure.ymlandopentofu/talos_image.tf. I checked every changed reference against the underlying Terraform (vm_150/151/152_k8s_*.tf) and Talos patches (k8s-1/2/3.yaml,cluster.yaml) — the busPath selectors (0000:06:12.0mgmt,0000:06:13.0v30iot) and the VM NIC/hostpci ordering are identical across all three nodes, which supports the PR's claim thatens19is stable and consistent. A repo-wide search confirms noeth1reference was missed. No blocking issues found.🟡 Suggestions
kubernetes/infrastructure/post-crds/cilium-bgp.yaml:31,80— these (untouched) comments still say nodes source BGP frometh0for the mgmt LAN. Per this PR's own busPath table, mgmt is nowens18, noteth0, on k8s-{1,2,3}. There's no functional dependency (nolocalAddressoverride is set, so this doesn't affect BGP behavior), but the comment is now stale in the same way theeth1comments were before this PR — worth a follow-up touch-up for consistency, since accurate interface-name comments are clearly a priority here..woodpecker/deploy-infrastructure.yml:136/opentofu/talos_image.tf:11— the new filename scheme (talos-$VERSION-<schematic12>-nocloud-amd64.raw) means the previous image (talos-v1.12.6-nocloud-amd64.raw) is orphaned on each Proxmox node's local storage rather than replaced — minor, unbounded storage growth on future schematic bumps, not this PR's concern to fix but worth tracking.🟢 Nits
kubernetes/apps/music-assistant/deployment.yaml:28andkubernetes/apps/otbr/deployment.yaml:44word the master-interface comment as "present on every k8s-{1,2,3}/edge worker" / "present on every node", but both pods are excluded fromk8s-edgevia theworkload-class=pinnedtaint (no toleration), so whetherens19actually exists on edge is moot. This imprecision predates this PR (it was mechanically carried over from the oldeth1wording) and isn't introduced here — flagging only because the PR was otherwise very careful about comment accuracy elsewhere.Tests
No automated tests apply to this type of change (Talos/OpenTofu/Kubernetes manifest IaC). The PR body documents live verification instead: a rolling
talosctl upgradeacross k8s-3 → k8s-1 → k8s-2 with Ceph HEALTH_OK checked between each, and post-upgrade confirmation thatens19is identical on all three nodes, addressless on v4/v6, and that the four v30iot pods came up Running. I can't independently verify the live cluster state from the checkout, but the static cross-references (busPath selectors, VM definitions, schematic hash match between the two files that must stay in sync) are all internally consistent, which is the strongest check available from a diff review.Recommendation
Looks reasonable — no blocking issues; comments above are optional.