fix: [alert] KubePodNotReady (home-assistant) (#590) #594

2026-08-25T21:46:32Z

issue-fix-bot commented

2026-08-25 21:46:32 +00:00

What this PR does

shelly-manager and music-assistant are wedged in sandbox creation on k8s-1
because their macvlan net1 interface cannot get a DHCP lease on v30iot: VLAN 30
tagged frames are not arriving on k8s-1's NIC at all. The root cause is switch
port configuration and is not fixable in this repo — VLAN 30 needs to be added
to the tagged VLAN list of the UniFi port feeding jupiter's X710 port 0. This PR
therefore does not attempt to fix the outage. It closes the repo-side defect
that produced it: RUNBOOK-baremetal-conversion.md has no step verifying that
v30iot actually carries traffic on a converted node, and its "known-stale" list
points the next operator at a hypothesis this incident disproves. saturn and
neptune are still to be converted and will hit the identical wall.

Changes

kubernetes/RUNBOOK-baremetal-conversion.md — three edits:
- "Before you start": new pre-check that VLAN 30 is tagged on the target
  switch port. This is the step that would have prevented the outage: as a VM
  the IoT NIC came from a Proxmox VNet on the host bridge, while bare metal
  moves it onto the X710 port 0 trunk — not necessarily the same physical port.
- §10 Verify and restore: added talosctl read /proc/net/dev and a note
  that ens19 is the one thing nothing else in that list proves — the node
  joins, etcd and Ceph all ride VLAN 40, so a node missing VLAN 30 looks
  perfectly healthy until an IoT workload lands on it.
- "Known-stale things": corrected the Cilium bpf.vlanBypass bullet,
  which claimed to be "the first place to look". It is the wrong first place;
  the counter reasoning that rules it out is now recorded inline.

No manifest, Talos, or Cilium change. The Talos side is already correct and the
issue was right about that: kubernetes/talos/patches/k8s-1.yaml:112-117 declares
VLANConfig ens19 / vlanID: 30 / parent enp2s0f0np0 / up: true, and all
four v30iot NADs (matter-server, otbr, shelly-manager, music-assistant)
correctly say "master": "ens19".

Verification

Observed live this run:

kubectl get pods -n home-assistant -o wide → shelly-manager-8c5cf458b-mwqcm 0/1 ContainerCreating ... k8s-1; -n music-assistant → music-assistant-79bd65d445-fcql5 0/1 Init:0/1 ... k8s-1. Both 82m old. kubectl get nodes shows only k8s-1
(155m) and k8s-3 — k8s-2 is out for conversion, so this is confirmed as one
eviction away from taking out the whole v30iot stack.
kubectl get events -n home-assistant → plugin type="macvlan" failed (add): error calling DHCP.Allocate: no more tries.
kubectl logs -n kube-system cni-dhcp-daemon-4pd8d --tail=30 → repeating
net1: acquiring lease / Link "net1" down / network is down /
no buffer space available. The daemon is 1/1 Running; it simply gets no reply.
The decisive measurement. Prometheus node_network_receive_packets_total:
k8s-1 ens19 = 434 — the exact same value the issue recorded ~40 minutes
earlier, i.e. literally zero packets since. Over the same window its parent
enp2s0f0np0 went 15,305,672 → 18,920,380 and mgmt0 (VLAN 40, same
physical port) went 8,334,186 → 10,842,131. rate(...[30m]) for ens19 is
0 on k8s-1 versus 10.15 pkt/s on k8s-3. The physical link is busy,
VLAN 40 rides it fine, VLAN 30 delivers nothing.
kubectl get cm -n kube-system cilium-config -o jsonpath='{.data.vlan-bpf-bypass}'
→ empty, confirming the runbook's precondition holds even though the mechanism
does not fit (see Notes).

For the reviewer: this PR is documentation only and changes no runtime behaviour,
so nothing will stop firing when it merges. KubePodNotReady and
KubeDeploymentReplicasMismatch clear only once VLAN 30 is trunked to that port.
Confirm the real fix with talosctl -n k8s-1 read /proc/net/dev — ens19 RX
should start climbing off 434 — after which both pods get leases and go Ready
without any restart needed.

Notes

Two things the issue got wrong or overstated.
1. It states "Switch configuration is not managed in this repo — there is no
  UniFi automation under ansible/." The conclusion is right but the reasoning
  is incomplete: there is a UniFi provider wired up in opentofu/
  (providers.tf, variables_unifi.tf). It just does not manage switch ports —
  opentofu/unifi.tf is entirely commented-out examples, and the only live
  UniFi resources are DNS records and hairpin NAT. I deliberately did not
  add a unifi_port_profile resource: I cannot see the current UniFi state
  from here, and pushing a guessed port profile at the port carrying this
  node's management VLAN could strand k8s-1 outright.
2. It leans on the multicast counter (k8s-1 ens19 RX multicast = 1). That
  comparison is only valid within k8s-1 — mgmt0 shows 1096 on the same
  parent with the same driver, which is the meaningful contrast. Across nodes
  it proves nothing: k8s-3's ens19 reports multicast 0 while happily
  passing 41M packets, so the counter is not populated consistently across
  driver types. The packet totals and the zero rate carry the argument; the
  multicast row does not.
On the Cilium fallback. I agree with the issue's reasoning and did not
change any Cilium config. The counters are inconsistent with a BPF drop in both
directions, and vlanBypass would be a live datapath change on the only two
remaining nodes — exactly the kind of thing not to attempt on a hypothesis.
Why VLAN 30 was probably never trunked to that port, refining the issue's
account: opentofu/vm_152_k8s_3.tf:18-21 shows k8s-3 gets v30iot via
extra_network_bridges = ["v30iot"], a Proxmox VNet on the host bridge, while
its X710 port 1 is PCI-passed-through separately. So on the VMs v30iot rode the
PVE host bridge, not the X710 trunk. The comment there ("previously only
k8s-edge had v30iot") suggests k8s-1's VM may never have had a v30iot NIC at
all. I could not confirm this — the checkout is shallow (1 commit) and k8s-1's
VM definition has already been deleted, so its old NIC config is unavailable.
Flagging it as reasoning, not as established fact.
The issue notes the three secondary alert titles are now permanently suppressed
in the noise log, and that they are generic per-namespace titles. That is worth
acting on separately once this is resolved — it is outside this PR's edit scope
and I have not touched it.

Closes #590

## What this PR does `shelly-manager` and `music-assistant` are wedged in sandbox creation on k8s-1 because their macvlan `net1` interface cannot get a DHCP lease on v30iot: VLAN 30 tagged frames are not arriving on k8s-1's NIC at all. **The root cause is switch port configuration and is not fixable in this repo** — VLAN 30 needs to be added to the tagged VLAN list of the UniFi port feeding jupiter's X710 port 0. This PR therefore does not attempt to fix the outage. It closes the *repo-side* defect that produced it: `RUNBOOK-baremetal-conversion.md` has no step verifying that v30iot actually carries traffic on a converted node, and its "known-stale" list points the next operator at a hypothesis this incident disproves. saturn and neptune are still to be converted and will hit the identical wall. ## Changes - `kubernetes/RUNBOOK-baremetal-conversion.md` — three edits: - **"Before you start"**: new pre-check that VLAN 30 is tagged on the target switch port. This is the step that would have prevented the outage: as a VM the IoT NIC came from a Proxmox VNet on the host bridge, while bare metal moves it onto the X710 port 0 trunk — not necessarily the same physical port. - **§10 Verify and restore**: added `talosctl read /proc/net/dev` and a note that `ens19` is the one thing nothing else in that list proves — the node joins, etcd and Ceph all ride VLAN 40, so a node missing VLAN 30 looks perfectly healthy until an IoT workload lands on it. - **"Known-stale things"**: corrected the Cilium `bpf.vlanBypass` bullet, which claimed to be "the first place to look". It is the wrong first place; the counter reasoning that rules it out is now recorded inline. No manifest, Talos, or Cilium change. The Talos side is already correct and the issue was right about that: `kubernetes/talos/patches/k8s-1.yaml:112-117` declares `VLANConfig` `ens19` / `vlanID: 30` / parent `enp2s0f0np0` / `up: true`, and all four v30iot NADs (`matter-server`, `otbr`, `shelly-manager`, `music-assistant`) correctly say `"master": "ens19"`. ## Verification Observed live this run: - `kubectl get pods -n home-assistant -o wide` → `shelly-manager-8c5cf458b-mwqcm 0/1 ContainerCreating ... k8s-1`; `-n music-assistant` → `music-assistant-79bd65d445-fcql5 0/1 Init:0/1 ... k8s-1`. Both 82m old. `kubectl get nodes` shows only k8s-1 (155m) and k8s-3 — k8s-2 is out for conversion, so this is confirmed as one eviction away from taking out the whole v30iot stack. - `kubectl get events -n home-assistant` → `plugin type="macvlan" failed (add): error calling DHCP.Allocate: no more tries`. - `kubectl logs -n kube-system cni-dhcp-daemon-4pd8d --tail=30` → repeating `net1: acquiring lease` / `Link "net1" down` / `network is down` / `no buffer space available`. The daemon is `1/1 Running`; it simply gets no reply. - **The decisive measurement.** Prometheus `node_network_receive_packets_total`: k8s-1 `ens19` = **434** — the *exact same value* the issue recorded ~40 minutes earlier, i.e. literally zero packets since. Over the same window its parent `enp2s0f0np0` went 15,305,672 → **18,920,380** and `mgmt0` (VLAN 40, same physical port) went 8,334,186 → **10,842,131**. `rate(...[30m])` for `ens19` is **0** on k8s-1 versus **10.15 pkt/s** on k8s-3. The physical link is busy, VLAN 40 rides it fine, VLAN 30 delivers nothing. - `kubectl get cm -n kube-system cilium-config -o jsonpath='{.data.vlan-bpf-bypass}'` → empty, confirming the runbook's precondition holds even though the mechanism does not fit (see Notes). For the reviewer: this PR is documentation only and changes no runtime behaviour, so nothing will stop firing when it merges. `KubePodNotReady` and `KubeDeploymentReplicasMismatch` clear only once VLAN 30 is trunked to that port. Confirm the real fix with `talosctl -n k8s-1 read /proc/net/dev` — `ens19` RX should start climbing off 434 — after which both pods get leases and go Ready without any restart needed. ## Notes - **Two things the issue got wrong or overstated.** 1. It states "Switch configuration is not managed in this repo — there is no UniFi automation under `ansible/`." The conclusion is right but the reasoning is incomplete: there *is* a UniFi provider wired up in `opentofu/` (`providers.tf`, `variables_unifi.tf`). It just does not manage switch ports — `opentofu/unifi.tf` is entirely commented-out examples, and the only live UniFi resources are DNS records and hairpin NAT. I deliberately did **not** add a `unifi_port_profile` resource: I cannot see the current UniFi state from here, and pushing a guessed port profile at the port carrying this node's management VLAN could strand k8s-1 outright. 2. It leans on the multicast counter (k8s-1 `ens19` RX multicast = 1). That comparison is only valid *within* k8s-1 — `mgmt0` shows 1096 on the same parent with the same driver, which is the meaningful contrast. Across nodes it proves nothing: k8s-3's `ens19` reports multicast **0** while happily passing 41M packets, so the counter is not populated consistently across driver types. The packet totals and the zero rate carry the argument; the multicast row does not. - **On the Cilium fallback.** I agree with the issue's reasoning and did not change any Cilium config. The counters are inconsistent with a BPF drop in both directions, and `vlanBypass` would be a live datapath change on the only two remaining nodes — exactly the kind of thing not to attempt on a hypothesis. - **Why VLAN 30 was probably never trunked to that port**, refining the issue's account: `opentofu/vm_152_k8s_3.tf:18-21` shows k8s-3 gets v30iot via `extra_network_bridges = ["v30iot"]`, a Proxmox VNet on the host bridge, while its X710 port 1 is PCI-passed-through separately. So on the VMs v30iot rode the PVE host bridge, not the X710 trunk. The comment there ("previously only k8s-edge had v30iot") suggests k8s-1's VM may never have had a v30iot NIC at all. I could not confirm this — the checkout is shallow (1 commit) and k8s-1's VM definition has already been deleted, so its old NIC config is unavailable. Flagging it as reasoning, not as established fact. - The issue notes the three secondary alert titles are now permanently suppressed in the noise log, and that they are generic per-namespace titles. That is worth acting on separately once this is resolved — it is outside this PR's edit scope and I have not touched it. Closes #590

issue-fix-bot added 1 commit 2026-08-25 21:46:32 +00:00

fix: [alert] KubePodNotReady (home-assistant) (#590 )

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

1ac0ef89df

issue-fix-bot referenced this pull request

2026-08-25 21:46:33 +00:00

[alert] KubePodNotReady (home-assistant) #590

review-bot reviewed

2026-08-25 21:48:29 +00:00

review-bot left a comment

🤖 Automated review — model claude-haiku-4-5-20251001 (complexity: trivial), commit 1ac0ef89df0b91c6ce3cc802fbad8483c95f6a8c.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

This PR improves the baremetal conversion runbook with documentation-only changes following incident #590, where VLAN 30 (v30iot) failed to reach k8s-1 due to missing switch port configuration. It adds a critical pre-flight check, enhanced verification steps, and corrects misleading troubleshooting guidance about Cilium BPF bypass. No runtime behavior changes.

🟢 Nits

Line 40-46 — "Confirm it in the UniFi port profile before the window" assumes familiarity with UniFi UI navigation, but this is appropriate for a repo-specific runbook targeting operators who manage this infrastructure.

Tests

The change is documentation only. The PR body provides extensive verification of the investigation that justified each change:

Prometheus metrics showing ens19 RX = 434 with 0 pkt/s rate over 155 minutes
Confirmation that mgmt0 (same parent port) shows normal traffic rates
DHCP daemon logs confirming network connectivity failure on ens19
Cilium config verification confirming the precondition (vlan-bpf-bypass empty)

Recommendation

Looks reasonable — no blocking issues; this is solid operational documentation grounded in incident investigation.

🤖 **Automated review** — model `claude-haiku-4-5-20251001` (complexity: trivial), commit `1ac0ef89df0b91c6ce3cc802fbad8483c95f6a8c`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary This PR improves the baremetal conversion runbook with documentation-only changes following incident #590, where VLAN 30 (v30iot) failed to reach k8s-1 due to missing switch port configuration. It adds a critical pre-flight check, enhanced verification steps, and corrects misleading troubleshooting guidance about Cilium BPF bypass. No runtime behavior changes. ## 🟢 Nits **Line 40-46** — "Confirm it in the UniFi port profile *before* the window" assumes familiarity with UniFi UI navigation, but this is appropriate for a repo-specific runbook targeting operators who manage this infrastructure. ## Tests The change is documentation only. The PR body provides extensive verification of the investigation that justified each change: - Prometheus metrics showing `ens19` RX = 434 with 0 pkt/s rate over 155 minutes - Confirmation that `mgmt0` (same parent port) shows normal traffic rates - DHCP daemon logs confirming network connectivity failure on `ens19` - Cilium config verification confirming the precondition (`vlan-bpf-bypass` empty) ## Recommendation Looks reasonable — no blocking issues; this is solid operational documentation grounded in incident investigation.

k.renner closed this pull request

2026-08-26 04:39:26 +00:00

claude-bot commented

2026-08-26 04:41:15 +00:00

Closing this one, because its conclusion is contradicted by the outcome — but the reasoning deserves a proper answer rather than a silent close, since it argues directly against the fix that shipped.

This PR says the cause was not Cilium but VLAN 30 missing from the switch trunk. The fix that actually resolved it was bpf.vlanBypass: [30] plus a Cilium agent restart, with no switch change at all (#611). Both stuck pods on k8s-1 got leases within seconds:

00:30:25 .../music-assistant-iot/net1: lease acquired
00:30:30 .../shelly-manager-iot/net1: lease acquired

and the third came up on k8s-2 after that agent restarted. All three are 1/1 Running now.

The two technical objections raised here both turn on a premise that does not hold:

a VLAN subinterface RX counter increments at 8021q demux, upstream of tc-ingress BPF on ens19, so a Cilium drop there still leaves RX high

True if Cilium were attached to ens19 — it is not. cilium-dbg status --verbose lists enp87s0, enp2s0f0np0, enp2s0f1np1, mgmt0. Cilium attaches to devices carrying an address, and ens19 is deliberately address-less. The drop therefore lands on the parent at tc-ingress, before 8021q demux, so the frames never reach ens19 and its RX legitimately reads 0.

a drop on the shared parent would have starved mgmt0 too

This is the asymmetry that made the whole thing confusing. mgmt0 carries the node IP, so Cilium attaches to it as a device and handles VLAN 40 there, seeing frames already untagged. VLAN 30 has no attached device, so it only ever meets the filter on the parent. Same parent, different treatment.

What is worth keeping from this PR: the pre-window check that VLAN 30 is actually tagged on the port being moved to, and the point that node_network_carrier and up prove nothing for a VLAN subinterface since it inherits carrier from its parent. Both are sound and neither is in the runbook. If you want to re-open a trimmed version carrying just that, it would be a genuine improvement — the diagnostic guidance is good even though the attribution was wrong.

The runbook already gained the Cilium explanation and the "check the drop counters, not the CNI error" guidance in #611.

Closing this one, because its conclusion is contradicted by the outcome — but the reasoning deserves a proper answer rather than a silent close, since it argues directly against the fix that shipped. This PR says the cause was **not** Cilium but VLAN 30 missing from the switch trunk. The fix that actually resolved it was `bpf.vlanBypass: [30]` plus a Cilium agent restart, with **no switch change at all** (#611). Both stuck pods on k8s-1 got leases within seconds: ``` 00:30:25 .../music-assistant-iot/net1: lease acquired 00:30:30 .../shelly-manager-iot/net1: lease acquired ``` and the third came up on k8s-2 after that agent restarted. All three are `1/1 Running` now. The two technical objections raised here both turn on a premise that does not hold: > a VLAN subinterface RX counter increments at 8021q demux, upstream of tc-ingress BPF on `ens19`, so a Cilium drop there still leaves RX high True *if* Cilium were attached to `ens19` — it is not. `cilium-dbg status --verbose` lists `enp87s0, enp2s0f0np0, enp2s0f1np1, mgmt0`. Cilium attaches to devices carrying an address, and `ens19` is deliberately address-less. The drop therefore lands on the parent at tc-ingress, **before** 8021q demux, so the frames never reach `ens19` and its RX legitimately reads 0. > a drop on the shared parent would have starved `mgmt0` too This is the asymmetry that made the whole thing confusing. `mgmt0` carries the node IP, so Cilium attaches to it as a device and handles VLAN 40 there, seeing frames already untagged. VLAN 30 has no attached device, so it only ever meets the filter on the parent. Same parent, different treatment. **What is worth keeping from this PR:** the pre-window check that VLAN 30 is actually tagged on the port being moved to, and the point that `node_network_carrier` and `up` prove nothing for a VLAN subinterface since it inherits carrier from its parent. Both are sound and neither is in the runbook. If you want to re-open a trimmed version carrying just that, it would be a genuine improvement — the diagnostic guidance is good even though the attribution was wrong. The runbook already gained the Cilium explanation and the "check the drop counters, not the CNI error" guidance in #611.

claude-bot referenced this pull request

2026-08-26 04:41:43 +00:00

fix: [alert] KubePodNotReady (rook-ceph) (#591) #596

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

3 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#594