fix: [alert] KubeContainerWaiting (home-assistant) (#592) #606

2026-08-25T23:47:27Z

issue-fix-bot commented

2026-08-25 23:47:27 +00:00

What this PR does

KubeContainerWaiting fires for home-assistant/shelly-manager (and its
siblings) because Cilium's eBPF VLAN filter is dropping the macvlan children's
DHCP exchange on the bare-metal nodes. Since jupiter (k8s-1) was converted from
a Proxmox VM, ens19 is no longer a virtio NIC — it is a VLAN 30 sub-interface
of the enp2s0f0np0 trunk, so a macvlan child's DHCP DISCOVER now leaves the
node tagged. Cilium drops every tagged frame it sees unless bpf.vlanBypass
lists the id, and that key was set nowhere in the repo and was absent from the
live cilium-config. The CNI DHCP daemon reports the tc-layer drop as no buffer space available (an AF_PACKET sender sees a tc drop as ENOBUFS, not
as a timeout), retries forever, and the sandbox never comes up. This adds
vlanBypass: [30, 40] to the Cilium HelmRelease and mirrors it into the two
committed bootstrap copies so a cold rebuild does not regress.

The repo predicted this exactly — kubernetes/RUNBOOK-baremetal-conversion.md:392-395
says the datapath "silently drops VLAN-tagged traffic unless bpf.vlanBypass
is set … the first place to look if the v30iot macvlan workloads misbehave".

Changes

kubernetes/infrastructure/cilium/helmrelease.yaml — add values.bpf.vlanBypass: [30, 40]
alongside the existing bpf block. This is the live fix; it renders to the
vlan-bpf-bypass config key. [30] alone would resolve this alert, but
mgmt0 (VLAN 40) is the other VLAN sub-interface a converted node terminates
and the runbook names both — the list is still a least-privilege allowlist,
unlike the [0] "bypass every VLAN id" option.
kubernetes/talos/cilium.yaml — add the matching vlan-bpf-bypass: "30,40".
kubernetes/talos/patches/cilium-cni.yaml — same key in the inline bootstrap
manifest. Bootstrap installs Cilium from this file, so without it a cold
rebuild comes up with every v30iot macvlan pod unable to DHCP until Flux
reconciles. Both files' own comments demand this lockstep ("Nothing enforces
any of this").

Comment lines aside, the functional change is three lines. No deletions.

Verification

Verified live this run (all read-only, via ./.ci-diag/kubectl):

The key really is absent. Full dump of configmap/cilium-config .data
contains no vlan-bpf-bypass entry. grep -rn "vlanBypass|vlan-bpf-bypass"
over kubernetes/ matched only the runbook prediction — the repo set it
nowhere.
Both pods fail on macvlan DHCP, not on anything else. get events for
shelly-manager-8c5cf458b-mwqcm and music-assistant-79bd65d445-fcql5:
plugin type="macvlan" failed (add): error calling DHCP.Allocate: no more tries.
The drop counters correlate three ways. sum by (pod,direction,reason) (increase(cilium_drop_count_total{reason=~"VLAN.*"}[60m]))
via the Prometheus API, reason VLAN traffic disallowed by VLAN filter:

cilium agent node ens19 is EGRESS drops/60m macvlan DHCP

cilium-v7wht k8s-1 VLAN sub-if 883 broken

cilium-z9fq6 k8s-2 VLAN sub-if 330 broken

cilium-8ssqh k8s-3 virtio NIC 0 works

EGRESS drops appear on exactly the two nodes where the macvlan parent is a
VLAN sub-interface, and on neither the node where it isn't. That is the
transmit path the DHCP DISCOVER takes.
The DHCP daemons agree. logs -l app=cni-dhcp-daemon: the k8s-1 and k8s-2
daemons are both looping no buffer space available / retrying in 63.9 seconds; the k8s-3 daemon is quiet.

For the reviewer — this will not take effect on its own. daemonset/cilium
carries no config-checksum annotation on its pod template (verified:
.spec.template.metadata.annotations holds only default-container and a
restartedAt from 2026-05-19). Flux will update the ConfigMap and the agents
will keep running the old datapath config. An agent restart is required.

Note the DaemonSet is maxSurge: 0, maxUnavailable: 2, so a plain rollout restart drops 2 of 3 agents at once — the same hazard the HelmRelease's
own chart-version comment warns about. Prefer deleting one agent pod at a time
and waiting for Ready in between.

After the restart, expect: vlan-bpf-bypass: "30,40" present in
cilium-config; cilium_drop_count_total{reason=~"VLAN.*"} EGRESS flat on
k8s-1/k8s-2; lease acquired in the k8s-1/k8s-2 DHCP daemon logs; the stuck
pods reaching Running; KubeContainerWaiting clearing for both
home-assistant and music-assistant.

Notes

The issue's live state is already stale, in a way that matters. It reports
matter-server as Running on k8s-3 and lists the k8s-2 rejoin as a
forward-looking risk. Both have moved: k8s-2 is now Ready (89m old, bare
metal, zone=saturn) and matter-server has rescheduled onto it and is
0/1 ContainerCreating with the identical macvlan DHCP error. The predicted
regression already happened. Three v30iot workloads are down, not two, and
k8s-3 is the only node where they still work.

I deliberately did not apply the issue's suggested interim mitigation. It
proposes pinning the v30iot Deployments to topology.kubernetes.io/zone: neptune
(k8s-3). That would be actively counterproductive now: the most recent commit on
main is "drop k8s-3 from the storage fleet for neptune's conversion", so
k8s-3 is the next node to be converted and would break the same way — the pin
would have to be reverted almost immediately, and it pins three workloads to a
single node in the meantime. The datapath fix makes the stopgap unnecessary.

One thing I could not verify, stated plainly. I could not confirm which
devices Cilium has attached its datapath to; that needs cilium-dbg datapath show or cilium status --verbose from inside the agent pod, which requires
exec that this agent's ServiceAccount does not have. So the issue's "loose
end" — why VLAN 40 (mgmt0) works today while VLAN 30 does not — remains
inference, not measurement: the most consistent explanation is that host
traffic on mgmt0 is processed by Cilium untagged and marked before the VLAN
tag is pushed at the trunk, whereas macvlan child traffic never traverses a
Cilium program and arrives at the trunk tagged and unmarked. This does not
affect the fix: allowlisting a VLAN only converts a drop into a pass, so adding
40 cannot break the management path that already works, and the reason string
plus the three-way correlation above are unambiguous about VLAN 30.

Out of scope, left alone (worth a follow-up):

kubernetes/RUNBOOK-baremetal-conversion.md:394 still reads "Not observed on
jupiter" — it is now observed on both jupiter and saturn.
Stale invariant comments now that ens19 is not a NIC on every node, e.g.
kubernetes/apps/music-assistant/deployment.yaml:28 ("master: ens19 is the
v30iot NIC on every k8s-{1,2,3}/edge worker") and
kubernetes/talos/patches/k8s-1.yaml:1-2 ("k8s-2 and k8s-3 are still Proxmox
VMs" — k8s-2 no longer is).
Unrelated pre-existing bootstrap drift: kubernetes/talos/cilium.yaml has
enable-ipv6: "false" while the live config and the HelmRelease both have it
true. Not touched here.

Nothing in the issue asked for changes outside the allowed edit paths.

Closes #592

## What this PR does `KubeContainerWaiting` fires for `home-assistant/shelly-manager` (and its siblings) because Cilium's eBPF VLAN filter is dropping the macvlan children's DHCP exchange on the bare-metal nodes. Since jupiter (k8s-1) was converted from a Proxmox VM, `ens19` is no longer a virtio NIC — it is a VLAN 30 sub-interface of the `enp2s0f0np0` trunk, so a macvlan child's DHCP DISCOVER now leaves the node **tagged**. Cilium drops every tagged frame it sees unless `bpf.vlanBypass` lists the id, and that key was set nowhere in the repo and was absent from the live `cilium-config`. The CNI DHCP daemon reports the tc-layer drop as `no buffer space available` (an `AF_PACKET` sender sees a tc drop as `ENOBUFS`, not as a timeout), retries forever, and the sandbox never comes up. This adds `vlanBypass: [30, 40]` to the Cilium HelmRelease and mirrors it into the two committed bootstrap copies so a cold rebuild does not regress. The repo predicted this exactly — `kubernetes/RUNBOOK-baremetal-conversion.md:392-395` says the datapath "silently drops VLAN-tagged traffic unless `bpf.vlanBypass` is set … the first place to look if the v30iot macvlan workloads misbehave". ## Changes - `kubernetes/infrastructure/cilium/helmrelease.yaml` — add `values.bpf.vlanBypass: [30, 40]` alongside the existing `bpf` block. This is the live fix; it renders to the `vlan-bpf-bypass` config key. `[30]` alone would resolve this alert, but `mgmt0` (VLAN 40) is the other VLAN sub-interface a converted node terminates and the runbook names both — the list is still a least-privilege allowlist, unlike the `[0]` "bypass every VLAN id" option. - `kubernetes/talos/cilium.yaml` — add the matching `vlan-bpf-bypass: "30,40"`. - `kubernetes/talos/patches/cilium-cni.yaml` — same key in the inline bootstrap manifest. Bootstrap installs Cilium from this file, so without it a cold rebuild comes up with every v30iot macvlan pod unable to DHCP until Flux reconciles. Both files' own comments demand this lockstep ("Nothing enforces any of this"). Comment lines aside, the functional change is three lines. No deletions. ## Verification Verified live this run (all read-only, via `./.ci-diag/kubectl`): - **The key really is absent.** Full dump of `configmap/cilium-config` `.data` contains no `vlan-bpf-bypass` entry. `grep -rn "vlanBypass|vlan-bpf-bypass"` over `kubernetes/` matched only the runbook prediction — the repo set it nowhere. - **Both pods fail on macvlan DHCP, not on anything else.** `get events` for `shelly-manager-8c5cf458b-mwqcm` and `music-assistant-79bd65d445-fcql5`: `plugin type="macvlan" failed (add): error calling DHCP.Allocate: no more tries`. - **The drop counters correlate three ways.** `sum by (pod,direction,reason) (increase(cilium_drop_count_total{reason=~"VLAN.*"}[60m]))` via the Prometheus API, reason `VLAN traffic disallowed by VLAN filter`: | cilium agent | node | ens19 is | EGRESS drops/60m | macvlan DHCP | |---|---|---|---|---| | `cilium-v7wht` | k8s-1 | VLAN sub-if | 883 | broken | | `cilium-z9fq6` | k8s-2 | VLAN sub-if | 330 | broken | | `cilium-8ssqh` | k8s-3 | virtio NIC | **0** | works | EGRESS drops appear on exactly the two nodes where the macvlan parent is a VLAN sub-interface, and on neither the node where it isn't. That is the transmit path the DHCP DISCOVER takes. - **The DHCP daemons agree.** `logs -l app=cni-dhcp-daemon`: the k8s-1 and k8s-2 daemons are both looping `no buffer space available` / `retrying in 63.9 seconds`; the k8s-3 daemon is quiet. **For the reviewer — this will not take effect on its own.** `daemonset/cilium` carries no config-checksum annotation on its pod template (verified: `.spec.template.metadata.annotations` holds only `default-container` and a `restartedAt` from 2026-05-19). Flux will update the ConfigMap and the agents will keep running the old datapath config. An agent restart is required. Note the DaemonSet is `maxSurge: 0, maxUnavailable: 2`, so a plain `rollout restart` drops **2 of 3 agents at once** — the same hazard the HelmRelease's own chart-version comment warns about. Prefer deleting one agent pod at a time and waiting for Ready in between. After the restart, expect: `vlan-bpf-bypass: "30,40"` present in `cilium-config`; `cilium_drop_count_total{reason=~"VLAN.*"}` EGRESS flat on k8s-1/k8s-2; `lease acquired` in the k8s-1/k8s-2 DHCP daemon logs; the stuck pods reaching Running; `KubeContainerWaiting` clearing for both `home-assistant` and `music-assistant`. ## Notes **The issue's live state is already stale, in a way that matters.** It reports `matter-server` as Running on k8s-3 and lists the k8s-2 rejoin as a *forward-looking* risk. Both have moved: k8s-2 is now Ready (89m old, bare metal, `zone=saturn`) and `matter-server` has rescheduled onto it and is `0/1 ContainerCreating` with the identical macvlan DHCP error. The predicted regression already happened. Three v30iot workloads are down, not two, and k8s-3 is the only node where they still work. **I deliberately did not apply the issue's suggested interim mitigation.** It proposes pinning the v30iot Deployments to `topology.kubernetes.io/zone: neptune` (k8s-3). That would be actively counterproductive now: the most recent commit on `main` is *"drop k8s-3 from the storage fleet for neptune's conversion"*, so k8s-3 is the next node to be converted and would break the same way — the pin would have to be reverted almost immediately, and it pins three workloads to a single node in the meantime. The datapath fix makes the stopgap unnecessary. **One thing I could not verify, stated plainly.** I could not confirm which devices Cilium has attached its datapath to; that needs `cilium-dbg datapath show` or `cilium status --verbose` from inside the agent pod, which requires `exec` that this agent's ServiceAccount does not have. So the issue's "loose end" — why VLAN 40 (`mgmt0`) works today while VLAN 30 does not — remains *inference*, not measurement: the most consistent explanation is that host traffic on `mgmt0` is processed by Cilium untagged and marked before the VLAN tag is pushed at the trunk, whereas macvlan child traffic never traverses a Cilium program and arrives at the trunk tagged and unmarked. This does not affect the fix: allowlisting a VLAN only converts a drop into a pass, so adding 40 cannot break the management path that already works, and the reason string plus the three-way correlation above are unambiguous about VLAN 30. **Out of scope, left alone (worth a follow-up):** - `kubernetes/RUNBOOK-baremetal-conversion.md:394` still reads "Not observed on jupiter" — it is now observed on both jupiter and saturn. - Stale invariant comments now that `ens19` is not a NIC on every node, e.g. `kubernetes/apps/music-assistant/deployment.yaml:28` ("master: ens19 is the v30iot NIC on every k8s-{1,2,3}/edge worker") and `kubernetes/talos/patches/k8s-1.yaml:1-2` ("k8s-2 and k8s-3 are still Proxmox VMs" — k8s-2 no longer is). - Unrelated pre-existing bootstrap drift: `kubernetes/talos/cilium.yaml` has `enable-ipv6: "false"` while the live config and the HelmRelease both have it true. Not touched here. Nothing in the issue asked for changes outside the allowed edit paths. Closes #592

issue-fix-bot added 1 commit 2026-08-25 23:47:27 +00:00

fix: [alert] KubeContainerWaiting (home-assistant) (#592 )

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

fdd303d7a0

issue-fix-bot referenced this pull request

2026-08-25 23:47:27 +00:00

[alert] KubeContainerWaiting (home-assistant) #592

review-bot reviewed

2026-08-25 23:51:41 +00:00

review-bot left a comment

🤖 Automated review — model CLI default (complexity: complex), commit fdd303d7a0c780b5dbe5a16dbfb5ccc8164446eb.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

Adds bpf.vlanBypass: [30, 40] to the Cilium HelmRelease and mirrors the equivalent vlan-bpf-bypass: "30,40" ConfigMap key into the two committed bootstrap copies (kubernetes/talos/cilium.yaml, kubernetes/talos/patches/cilium-cni.yaml). Purely additive (28 insertions, 0 deletions, 3 files), no logic beyond config values. The change is well-grounded: it matches an exact prediction already present in the repo (kubernetes/RUNBOOK-baremetal-conversion.md:392-395, key name bpf.vlanBypass) and the VLAN topology it targets is independently confirmed by the Talos VLANConfig definitions for ens19/VLAN 30 and mgmt0/VLAN 40 on enp2s0f0np0 in kubernetes/talos/patches/k8s-1.yaml:92-116 and kubernetes/talos/patches/k8s-2.yaml:64-88. No blocking issues found.

🟡 Suggestions

I could not independently verify against upstream Cilium chart docs that bpf.vlanBypass (list of ints in Helm values) is the exact schema for chart version range >=1.20.0 <1.21.0 and that it renders precisely to the ConfigMap key vlan-bpf-bypass as a comma-joined string (no internet access in this review environment). The corroborating evidence is strong — the runbook text at kubernetes/RUNBOOK-baremetal-conversion.md:392-393 independently names the identical key — but since this doesn't take effect until an agent restart (per the PR's own note), it'd be worth a helm template cilium/cilium --version <pinned> --set bpf.vlanBypass='{30,40}' dry run before/at merge to confirm the rendered ConfigMap key and value format match kubernetes/infrastructure/cilium/helmrelease.yaml:94 exactly, rather than discovering a mismatch only after the manual agent restart.
The three-way lockstep between helmrelease.yaml / cilium.yaml / cilium-cni.yaml is enforced only by comments ("Nothing enforces any of this" — pre-existing wording this PR doesn't add to but does rely on). Not introduced by this PR, but it now adds a fourth key to that manually-synced set across the same three files. Worth a lint/CI check at some point (out of scope here).

Tests

No automated tests apply — this is a GitOps config repo and none of the three touched files have any existing test coverage; git grep confirms this class of change (bpf ConfigMap keys) has never had one (e.g. policyMapMax/bpf-policy-map-max follows the exact same three-file pattern with no test). The PR body documents live-cluster verification instead (drop-counter correlation, DHCP daemon logs, absence of the key in the running ConfigMap), which is consistent with how this repo validates infra changes. Nothing missing relative to repo norms.

Recommendation

Looks reasonable — no blocking issues; comments above are optional.

🤖 **Automated review** — model `CLI default` (complexity: complex), commit `fdd303d7a0c780b5dbe5a16dbfb5ccc8164446eb`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary Adds `bpf.vlanBypass: [30, 40]` to the Cilium HelmRelease and mirrors the equivalent `vlan-bpf-bypass: "30,40"` ConfigMap key into the two committed bootstrap copies (`kubernetes/talos/cilium.yaml`, `kubernetes/talos/patches/cilium-cni.yaml`). Purely additive (28 insertions, 0 deletions, 3 files), no logic beyond config values. The change is well-grounded: it matches an exact prediction already present in the repo (`kubernetes/RUNBOOK-baremetal-conversion.md:392-395`, key name `bpf.vlanBypass`) and the VLAN topology it targets is independently confirmed by the Talos `VLANConfig` definitions for `ens19`/VLAN 30 and `mgmt0`/VLAN 40 on `enp2s0f0np0` in `kubernetes/talos/patches/k8s-1.yaml:92-116` and `kubernetes/talos/patches/k8s-2.yaml:64-88`. No blocking issues found. ## 🟡 Suggestions - I could not independently verify against upstream Cilium chart docs that `bpf.vlanBypass` (list of ints in Helm values) is the exact schema for chart version range `>=1.20.0 <1.21.0` and that it renders precisely to the ConfigMap key `vlan-bpf-bypass` as a comma-joined string (no internet access in this review environment). The corroborating evidence is strong — the runbook text at `kubernetes/RUNBOOK-baremetal-conversion.md:392-393` independently names the identical key — but since this doesn't take effect until an agent restart (per the PR's own note), it'd be worth a `helm template cilium/cilium --version <pinned> --set bpf.vlanBypass='{30,40}'` dry run before/at merge to confirm the rendered ConfigMap key and value format match `kubernetes/infrastructure/cilium/helmrelease.yaml:94` exactly, rather than discovering a mismatch only after the manual agent restart. - The three-way lockstep between `helmrelease.yaml` / `cilium.yaml` / `cilium-cni.yaml` is enforced only by comments ("Nothing enforces any of this" — pre-existing wording this PR doesn't add to but does rely on). Not introduced by this PR, but it now adds a fourth key to that manually-synced set across the same three files. Worth a lint/CI check at some point (out of scope here). ## Tests No automated tests apply — this is a GitOps config repo and none of the three touched files have any existing test coverage; `git grep` confirms this class of change (bpf ConfigMap keys) has never had one (e.g. `policyMapMax`/`bpf-policy-map-max` follows the exact same three-file pattern with no test). The PR body documents live-cluster verification instead (drop-counter correlation, DHCP daemon logs, absence of the key in the running ConfigMap), which is consistent with how this repo validates infra changes. Nothing missing relative to repo norms. ## Recommendation Looks reasonable — no blocking issues; comments above are optional.

claude-bot commented

2026-08-26 00:44:08 +00:00

Superseded by #611, which is merged, and closing this to avoid a conflicting change landing on the same three files.

The diagnosis here was correct and genuinely useful — the datapath analysis in #592 is what found the root cause, after I had spent a long time chasing the CNI plugin's misleading ENOBUFS. Credit where it is due.

One substantive difference: this PR sets vlan-bpf-bypass: "30,40". #611 sets [30] only. VLAN 40 does not need a bypass entry, and the reason is worth recording:

$ cilium-dbg status --verbose
Devices: enp87s0, enp2s0f0np0, enp2s0f1np1, mgmt0 (Direct Routing)

mgmt0 is a device Cilium attaches to — it carries the node IP — so Cilium owns it and sees frames the 8021q layer has already untagged; there is no tag for a VLAN filter to match. ens19 is deliberately address-less, so Cilium never attaches to it and VLAN 30 frames only meet the filter on the shared parent, still tagged.

Listing 40 would therefore change nothing while touching the datapath that carries etcd and the API. That distinction came out of a direct question — "why does 40 work and 30 not" — rather than from either of our first drafts.

Verified working: all three v30iot workloads are 1/1 Running after the agents were restarted. #613 is open to close the separate gap that made the restart manual.

Superseded by #611, which is merged, and closing this to avoid a conflicting change landing on the same three files. The diagnosis here was correct and genuinely useful — the datapath analysis in #592 is what found the root cause, after I had spent a long time chasing the CNI plugin's misleading ENOBUFS. Credit where it is due. One substantive difference: this PR sets `vlan-bpf-bypass: "30,40"`. #611 sets `[30]` only. VLAN 40 does not need a bypass entry, and the reason is worth recording: ``` $ cilium-dbg status --verbose Devices: enp87s0, enp2s0f0np0, enp2s0f1np1, mgmt0 (Direct Routing) ``` `mgmt0` is a device Cilium attaches to — it carries the node IP — so Cilium owns it and sees frames the 8021q layer has already untagged; there is no tag for a VLAN filter to match. `ens19` is deliberately address-less, so Cilium never attaches to it and VLAN 30 frames only meet the filter on the shared parent, still tagged. Listing 40 would therefore change nothing while touching the datapath that carries etcd and the API. That distinction came out of a direct question — "why does 40 work and 30 not" — rather than from either of our first drafts. Verified working: all three v30iot workloads are `1/1 Running` after the agents were restarted. #613 is open to close the separate gap that made the restart manual.

k.renner closed this pull request

2026-08-26 04:37:52 +00:00

claude-bot referenced this pull request

2026-08-26 04:41:43 +00:00

fix: [alert] KubePodNotReady (rook-ceph) (#591) #596

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

3 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#606

cilium agent	node	ens19 is	EGRESS drops/60m	macvlan DHCP
`cilium-v7wht`	k8s-1	VLAN sub-if	883	broken
`cilium-z9fq6`	k8s-2	VLAN sub-if	330	broken
`cilium-8ssqh`	k8s-3	virtio NIC	0	works