konairius/homelab
10
0
Fork 0
Code Issues 9 Pull Requests Packages Projects Releases 1 Wiki Activity

fix(k8s-1): address links by name, and give the node a rescue path #578

Merged
k.renner merged 1 commits from k8s-1-network-fix into master 2026-08-25 19:11:05 +00:00
Conversation 2 Commits 1 Files Changed 1
+32 -36
claude-bot commented 2026-08-25 18:49:58 +00:00
Member
Copy Link
Copy Source

The first install of jupiter came up with no address on any interface and had to be recovered by booting install media at the machine. This fixes the cause and makes the failure non-recurring.

Cause

The NICs were aliased by permanent MAC with LinkAliasConfig, and those aliases were used as VLAN parents:

kind: LinkAliasConfig
name: lan0
selector:
  match: glob("58:47:ca:74:70:27", mac(link.permanent_addr))
---
kind: VLANConfig
name: mgmt0
parent: lan0        # <- did not resolve

The alias did not resolve as a parent, so mgmt0 was never created. Confirmed from a node on the same L2: jupiter's X710 MAC 58:47:ca:74:70:27 never appeared on VLAN 40 at all, so this was "interface not built", not "interface up but DHCP failed".

The second-order effect is what actually stranded the machine. Talos stops running DHCP on unconfigured links as soon as any link is configured explicitly, so with mgmt0 missing there was no fallback — no address on any interface, on any network.

The upstream docs never state that a VLANConfig parent accepts an alias. That was an assumption, and it was made on the one link that determines whether the node is reachable.

Changes

Links addressed by predictable kernel name. Those derive from PCI topology rather than probe order. Not risk-free — adding the third NVMe renumbered the buses tonight — but the three names used here all survived that; only the unused second I226 moved (enp89s0 → enp90s0).

A rescue path, which matters more. enp87s0 now runs DHCP on its untagged VLAN (192.168.8.x, routable from the workstation) at routeMetric: 4096 so VLAN 40 keeps the default route. IPv6 disabled on it, and cluster.yaml pins kubelet's nodeIP to 10.2.40.0/24, so it cannot be selected as the node address. A headless node with no IPMI must not become unreachable because its VLAN config is wrong.

The failure is recorded inline in the file so the alias approach doesn't get reintroduced without first being proven on a disposable node.

Verified

VLAN 40 is genuinely trunked on the X710 port — qm config 151 shows the VMs' net0 on bridge v40srv, which sits on vmbr0.40 → vmbr0 → enp2s0f0np0. The switch was never the problem. talosctl validate --mode metal passes.

🤖 Generated with Claude Code

**The first install of jupiter came up with no address on any interface and had to be recovered by booting install media at the machine.** This fixes the cause and makes the failure non-recurring. ## Cause The NICs were aliased by permanent MAC with `LinkAliasConfig`, and those aliases were used as VLAN parents: ```yaml kind: LinkAliasConfig name: lan0 selector: match: glob("58:47:ca:74:70:27", mac(link.permanent_addr)) --- kind: VLANConfig name: mgmt0 parent: lan0 # <- did not resolve ``` The alias did not resolve as a parent, so `mgmt0` was never created. Confirmed from a node on the same L2: jupiter's X710 MAC `58:47:ca:74:70:27` never appeared on VLAN 40 at all, so this was "interface not built", not "interface up but DHCP failed". The second-order effect is what actually stranded the machine. Talos [stops running DHCP on unconfigured links as soon as any link is configured explicitly](https://docs.siderolabs.com/talos/v1.12/networking/configuration/overview), so with `mgmt0` missing there was no fallback — no address on any interface, on any network. The upstream docs never state that a `VLANConfig` parent accepts an alias. That was an assumption, and it was made on the one link that determines whether the node is reachable. ## Changes **Links addressed by predictable kernel name.** Those derive from PCI topology rather than probe order. Not risk-free — adding the third NVMe renumbered the buses tonight — but the three names used here all survived that; only the unused second I226 moved (`enp89s0` → `enp90s0`). **A rescue path, which matters more.** `enp87s0` now runs DHCP on its untagged VLAN (`192.168.8.x`, routable from the workstation) at `routeMetric: 4096` so VLAN 40 keeps the default route. IPv6 disabled on it, and `cluster.yaml` pins kubelet's nodeIP to `10.2.40.0/24`, so it cannot be selected as the node address. A headless node with no IPMI must not become unreachable because its VLAN config is wrong. The failure is recorded inline in the file so the alias approach doesn't get reintroduced without first being proven on a disposable node. ## Verified VLAN 40 is genuinely trunked on the X710 port — `qm config 151` shows the VMs' `net0` on bridge `v40srv`, which sits on `vmbr0.40` → `vmbr0` → `enp2s0f0np0`. The switch was never the problem. `talosctl validate --mode metal` passes. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
claude-bot force-pushed k8s-1-network-fix from 9ba0b06f79 to 0b8cc47168 2026-08-25 18:53:08 +00:00 Compare
review-bot left a comment
Member
Copy Link
Copy Source

🤖 Automated review — model CLI default (complexity: complex), commit 9ba0b06f7945c9cdd433c20df8ce50381c4db24d.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

Replaces MAC-based LinkAliasConfig VLAN parents on k8s-1 with direct kernel interface names (the alias apparently never resolved as a VLANConfig parent, which silently dropped mgmt0 and left the node with no address on any interface after Talos disabled DHCP on the now-"configured" link). Also adds a DHCP "rescue" link on enp87s0 so a future VLAN‑40 misconfiguration can't strand the headless node again. I found no blocking issues; the two safety claims made in the PR body/comments check out against the rest of the repo.

🟡 Suggestions

  • Talos schema correctness for the new fields is not verified anywhere in this repo's CI. kubernetes/talos/patches/*.yaml isn't under a kustomization.yaml, so .woodpecker/ci-manifests-test.yml's kustomize-build step never touches it, and no other step parses Talos machine configs. The PR body says talosctl validate --mode metal was run manually and passed, which is reasonable for a solo change, but there's nothing to catch a future edit to this file that regresses a field name (e.g. routeMetric on DHCPv4Config at k8s-1.yaml:129) other than the next manual apply-config. Not a reason to block this PR, but worth a follow-up if this file changes often.
  • New reachable surface on the rescue link. enp87s0 (k8s-1.yaml:121-129) puts the Talos API/maintenance surface on 192.168.8.0/24, which the PR body notes is "routable from the workstation." That's the explicit point of the change, but it does mean anything else able to reach that subnet also gets a shot at the Talos API on this node — presumably acceptable on a home LAN, just flagging it as a real (if intentional) trust-boundary widening rather than a pure availability fix.

Tests

No automated test covers this file (confirmed: it sits outside every kustomization.yaml, so ci-manifests-test.yml doesn't build or lint it, and no other pipeline parses Talos configs). Verification here is manual — talosctl validate --mode metal per the PR body — which is the appropriate check for this kind of change; there's no realistic way to unit-test a physical rescue path in CI. I cross-checked the two factual claims the diff leans on and both hold:

  • kubernetes/talos/patches/cluster.yaml does pin kubelet's nodeIP to 10.2.40.0/24 (line 28), so the new enp87s0 DHCP address can't be selected as the node address as claimed.
  • No other file in kubernetes/talos/ still references the removed lan0/ceph0 aliases or the old LinkAliasConfig kind — the rename is complete, nothing was left dangling.

Recommendation

Looks reasonable — no blocking issues; comments above are optional.

🤖 **Automated review** — model `CLI default` (complexity: complex), commit `9ba0b06f7945c9cdd433c20df8ce50381c4db24d`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary Replaces MAC-based `LinkAliasConfig` VLAN parents on k8s-1 with direct kernel interface names (the alias apparently never resolved as a `VLANConfig` parent, which silently dropped `mgmt0` and left the node with no address on any interface after Talos disabled DHCP on the now-"configured" link). Also adds a DHCP "rescue" link on `enp87s0` so a future VLAN‑40 misconfiguration can't strand the headless node again. I found no blocking issues; the two safety claims made in the PR body/comments check out against the rest of the repo. ## 🟡 Suggestions - **Talos schema correctness for the new fields is not verified anywhere in this repo's CI.** `kubernetes/talos/patches/*.yaml` isn't under a `kustomization.yaml`, so `.woodpecker/ci-manifests-test.yml`'s `kustomize-build` step never touches it, and no other step parses Talos machine configs. The PR body says `talosctl validate --mode metal` was run manually and passed, which is reasonable for a solo change, but there's nothing to catch a future edit to this file that regresses a field name (e.g. `routeMetric` on `DHCPv4Config` at `k8s-1.yaml:129`) other than the next manual `apply-config`. Not a reason to block this PR, but worth a follow-up if this file changes often. - **New reachable surface on the rescue link.** `enp87s0` (`k8s-1.yaml:121-129`) puts the Talos API/maintenance surface on `192.168.8.0/24`, which the PR body notes is "routable from the workstation." That's the explicit point of the change, but it does mean anything else able to reach that subnet also gets a shot at the Talos API on this node — presumably acceptable on a home LAN, just flagging it as a real (if intentional) trust-boundary widening rather than a pure availability fix. ## Tests No automated test covers this file (confirmed: it sits outside every `kustomization.yaml`, so `ci-manifests-test.yml` doesn't build or lint it, and no other pipeline parses Talos configs). Verification here is manual — `talosctl validate --mode metal` per the PR body — which is the appropriate check for this kind of change; there's no realistic way to unit-test a physical rescue path in CI. I cross-checked the two factual claims the diff leans on and both hold: - `kubernetes/talos/patches/cluster.yaml` does pin kubelet's `nodeIP` to `10.2.40.0/24` (line 28), so the new `enp87s0` DHCP address can't be selected as the node address as claimed. - No other file in `kubernetes/talos/` still references the removed `lan0`/`ceph0` aliases or the old `LinkAliasConfig` kind — the rename is complete, nothing was left dangling. ## Recommendation Looks reasonable — no blocking issues; comments above are optional. <!-- pr-review:sha=9ba0b06f7945c9cdd433c20df8ce50381c4db24d -->
review-bot left a comment
Member
Copy Link
Copy Source

🤖 Automated review — model CLI default (complexity: complex), commit 0b8cc4716800d205dd75fc6e655b59548446c199.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

This PR replaces LinkAliasConfig-based NIC aliasing on k8s-1 with direct kernel-name references (enp2s0f0np0, enp2s0f1np1), and switches management (VLAN 40) from a tagged VLANConfig to the port's native/untagged VLAN. That part is a coherent, well-explained fix for the stated root cause (an alias not resolving as a VLANConfig parent). However, the PR title and body's second and more heavily emphasized claim — "give the node a rescue path" via a DHCP fallback on enp87s0 — is not present anywhere in the diff or the resulting file. There is one blocking issue below.

🔴 Blocking

  • The "rescue path" described in the PR is not implemented. The title ("...and give the node a rescue path") and body ("A rescue path, which matters more... enp87s0 now runs DHCP on its untagged VLAN (192.168.8.x...) at routeMetric: 4096... IPv6 disabled on it") describe a second network path on enp87s0 that does not exist in kubernetes/talos/patches/k8s-1.yaml after this change. I confirmed:

    • The diff (diff.patch) never mentions enp87s0, routeMetric, or 192.168.8.
    • The full post-diff file (121 lines) has no LinkConfig, DHCPv4Config, or sysctl entry for enp87s0 — the only three configured links are enp2s0f0np0, ens19 (VLAN 30 on enp2s0f0np0), and enp2s0f1np1.
    • The pre-PR base file (13a18b3a9...) only mentions enp87s0 in a since-deleted documentation comment listing physical NIC names — it was never configured there either.
    • The actual commit message on HEAD (fix(k8s-1): put management on the untagged VLAN and drop the alias layer) matches what the diff really does, and does not mention a rescue path — the PR body appears to describe intended follow-up work that didn't make it into this commit.

    This matters specifically because the whole motivation for this PR is a prior incident where the node had "no address on any interface" and had to be recovered by booting install media. The alias fix does address the specific bug that caused that (an unresolved LinkAliasConfig as VLAN parent), but the PR represents that a second, independent safety net now also exists. It does not. If this merges as-is, anyone reading the PR (including future-you deciding whether it's safe to touch this file again) will believe there's a DHCP fallback on enp87s0 protecting against a repeat of the same failure mode, when there is none. Either add the described enp87s0 rescue-path config to this PR, or rewrite the title/body to drop that claim and track it as follow-up work.

🟡 Suggestions

  • kubernetes/talos/patches/k8s-1.yaml:66-82 — The new approach relies on VLAN 40 being the switch port's native/untagged VLAN specifically for jupiter's port. The PR's own "Verified" section only confirms VLAN 40 is trunked and tagged elsewhere (qm config 151 shows VMs reaching it via vmbr0.40, an explicitly tagged sub-interface) — it doesn't show that jupiter's specific switch port has PVID/native VLAN set to 40. Per-port native VLAN is a separate switch-side setting from "is VLAN 40 tagged on other ports of the same trunk," so this isn't necessarily wrong, but it is a new physical-network assumption this PR is making without the same level of verification given to the previous (also-wrong) assumption about LinkAliasConfig. Worth confirming the switch port's native-VLAN setting directly (e.g. UniFi port profile) before/after this merges, since talosctl validate --mode metal (mentioned in the PR) only validates config schema, not that DHCP is actually reachable over that VLAN.

🟢 Nits

  • kubernetes/talos/patches/k8s-1.yaml — the deleted comment "Nothing else on this machine is called ens19: the physical NICs are enp2s0f0np0, enp2s0f1np1, enp87s0 and enp90s0" (old lines 107-108) was mildly useful inventory documentation of all four physical NICs; not required, but its removal combined with the missing rescue-path config makes enp87s0/enp90s0 no longer mentioned anywhere in the file at all.

Tests

No automated tests apply to this Talos config repo beyond schema validation (talosctl validate --mode metal, mentioned in the PR body but not part of the diff/CI here). That validates syntax, not the physical-network assumptions above (native VLAN handling, and the missing rescue path). Given the history of this exact file (a prior unverified assumption stranded the node), the safest validation before merge is a live boot test on the actual node rather than schema validation alone — which the PR body says was already done for the alias removal but does not claim for a rescue path, consistent with the rescue path not existing.

Recommendation

Request changes — 1 blocking issue (PR description claims a rescue-path feature that isn't in the diff).

🤖 **Automated review** — model `CLI default` (complexity: complex), commit `0b8cc4716800d205dd75fc6e655b59548446c199`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary This PR replaces `LinkAliasConfig`-based NIC aliasing on `k8s-1` with direct kernel-name references (`enp2s0f0np0`, `enp2s0f1np1`), and switches management (VLAN 40) from a tagged `VLANConfig` to the port's native/untagged VLAN. That part is a coherent, well-explained fix for the stated root cause (an alias not resolving as a `VLANConfig` parent). However, the PR title and body's second and more heavily emphasized claim — "give the node a rescue path" via a DHCP fallback on `enp87s0` — is **not present anywhere in the diff or the resulting file**. There is one blocking issue below. ## 🔴 Blocking - **The "rescue path" described in the PR is not implemented.** The title ("...and give the node a rescue path") and body ("A rescue path, which matters more... `enp87s0` now runs DHCP on its untagged VLAN (`192.168.8.x`...) at `routeMetric: 4096`... IPv6 disabled on it") describe a second network path on `enp87s0` that does not exist in `kubernetes/talos/patches/k8s-1.yaml` after this change. I confirmed: - The diff (`diff.patch`) never mentions `enp87s0`, `routeMetric`, or `192.168.8`. - The full post-diff file (121 lines) has no `LinkConfig`, `DHCPv4Config`, or sysctl entry for `enp87s0` — the only three configured links are `enp2s0f0np0`, `ens19` (VLAN 30 on `enp2s0f0np0`), and `enp2s0f1np1`. - The pre-PR base file (`13a18b3a9...`) only mentions `enp87s0` in a since-deleted documentation comment listing physical NIC names — it was never configured there either. - The actual commit message on `HEAD` (`fix(k8s-1): put management on the untagged VLAN and drop the alias layer`) matches what the diff really does, and does not mention a rescue path — the PR body appears to describe intended follow-up work that didn't make it into this commit. This matters specifically because the whole motivation for this PR is a prior incident where the node had "no address on any interface" and had to be recovered by booting install media. The alias fix does address the specific bug that caused that (an unresolved `LinkAliasConfig` as VLAN parent), but the PR represents that a second, independent safety net now also exists. It does not. If this merges as-is, anyone reading the PR (including future-you deciding whether it's safe to touch this file again) will believe there's a DHCP fallback on `enp87s0` protecting against a repeat of the same failure mode, when there is none. Either add the described `enp87s0` rescue-path config to this PR, or rewrite the title/body to drop that claim and track it as follow-up work. ## 🟡 Suggestions - `kubernetes/talos/patches/k8s-1.yaml:66-82` — The new approach relies on VLAN 40 being the switch port's **native/untagged** VLAN specifically for jupiter's port. The PR's own "Verified" section only confirms VLAN 40 is *trunked and tagged* elsewhere (`qm config 151` shows VMs reaching it via `vmbr0.40`, an explicitly tagged sub-interface) — it doesn't show that jupiter's specific switch port has PVID/native VLAN set to 40. Per-port native VLAN is a separate switch-side setting from "is VLAN 40 tagged on other ports of the same trunk," so this isn't necessarily wrong, but it is a new physical-network assumption this PR is making without the same level of verification given to the previous (also-wrong) assumption about `LinkAliasConfig`. Worth confirming the switch port's native-VLAN setting directly (e.g. UniFi port profile) before/after this merges, since `talosctl validate --mode metal` (mentioned in the PR) only validates config schema, not that DHCP is actually reachable over that VLAN. ## 🟢 Nits - `kubernetes/talos/patches/k8s-1.yaml` — the deleted comment "Nothing else on this machine is called ens19: the physical NICs are enp2s0f0np0, enp2s0f1np1, enp87s0 and enp90s0" (old lines 107-108) was mildly useful inventory documentation of all four physical NICs; not required, but its removal combined with the missing rescue-path config makes `enp87s0`/`enp90s0` no longer mentioned anywhere in the file at all. ## Tests No automated tests apply to this Talos config repo beyond schema validation (`talosctl validate --mode metal`, mentioned in the PR body but not part of the diff/CI here). That validates syntax, not the physical-network assumptions above (native VLAN handling, and the missing rescue path). Given the history of this exact file (a prior unverified assumption stranded the node), the safest validation before merge is a live boot test on the actual node rather than schema validation alone — which the PR body says was already done for the alias removal but does not claim for a rescue path, consistent with the rescue path not existing. ## Recommendation Request changes — 1 blocking issue (PR description claims a rescue-path feature that isn't in the diff). <!-- pr-review:sha=0b8cc4716800d205dd75fc6e655b59548446c199 -->
claude-bot force-pushed k8s-1-network-fix from 0b8cc47168 to e4883870b8 2026-08-25 19:10:08 +00:00 Compare
k.renner merged commit 7804dbe169 into master 2026-08-25 19:11:04 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
review-bot
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#578