fix: [alert] KubeContainerWaiting (home-assistant) (#592 )

Merge pull request 'chore(rook-ceph): drop k8s-3 from the storage fleet for neptune's conversion' (#597 ) from ceph-drop-neptune into master
Reviewed-on: k.renner/homelab#597
2026-08-25 23:47:26 +00:00 · 2026-08-25 22:54:19 +00:00 · 2026-08-26 00:47:52 +02:00
4 changed files with 32 additions and 7 deletions
@@ -77,6 +77,21 @@ spec:
      # redirects the socket directly to the backend pod instead of sending
      # traffic out to the VIP and back (which fails with L2 announcements).
      lbExternalClusterIP: true
+      # VLAN ids whose tagged frames are passed to the stack instead of being
+      # dropped by Cilium's VLAN filter. Unset, the filter drops every tagged
+      # frame the datapath sees — which is what strands the v30iot macvlan
+      # pods on a converted node: there `ens19` (VLAN 30) and `mgmt0`
+      # (VLAN 40) are VLAN sub-interfaces of the enp2s0f0np0 trunk, so a
+      # macvlan child's DHCP DISCOVER leaves tagged and dies as
+      # DROP_VLAN_FILTER. The CNI DHCP daemon reports that as "no buffer
+      # space available" (a tc drop surfaces to an AF_PACKET sender as
+      # ENOBUFS, not as a timeout) and the pod sits in ContainerCreating
+      # forever — issue #592, predicted in RUNBOOK-baremetal-conversion.md.
+      # Lists only the two VLANs a node terminates; [0] would bypass the
+      # filter for every id. Takes effect on agent restart.
+      # Keep in lockstep with vlan-bpf-bypass in
+      # kubernetes/talos/patches/cilium-cni.yaml and kubernetes/talos/cilium.yaml.
+      vlanBypass: [30, 40]
    cni:
      # Multus (kubernetes/infrastructure/multus) runs alongside Cilium and
      # writes 00-multus.conf to /etc/cni/net.d so kubelet picks multus
@@ -167,10 +167,6 @@ spec:
  # impossible: the OSD is recreated within a reconcile and starts backfilling
  # onto the drive being replaced.
  #
-  # nvme0n1 on k8s-3 is the NVMe drive PCI-passed-through from neptune, and is
-  # the only block device that guest can see, so a bare device name is
-  # unambiguous there.
-  #
  # k8s-1 and k8s-2 are bare metal and see three NVMe drives each, so they name
  # their OSDs by /dev/disk/by-id path instead. A bare name would be actively
  # wrong: Talos enumerates these machines' drives in a different order than
@@ -192,9 +188,10 @@ spec:
    useAllNodes: false
    useAllDevices: false
    nodes:
-      - name: k8s-3
-        devices:
-          - name: nvme0n1
+      # k8s-3 is absent while neptune's OSD drive is swapped and the node is
+      # converted to bare metal. RE-ADD IT once the PM983 is in and the node has
+      # rejoined, naming the device by /dev/disk/by-id as the other two do —
+      # bare metal sees three NVMe drives and the bare name is not the OSD.
      - name: k8s-1
        devices:
          - fullpath: /dev/disk/by-id/nvme-SAMSUNG_MZQLB960HAJR-00007_S437NX0M400426
@@ -306,6 +306,12 @@ data:
  # Specifies the ratio (0.0-1.0] of total system memory to use for dynamic
  # sizing of the TCP CT, non-TCP CT, NAT and policy BPF maps.
  bpf-map-dynamic-size-ratio: "0.0025"
+  # VLAN ids whose tagged frames bypass the eBPF datapath rather than being
+  # dropped by the VLAN filter. 30 (v30iot, the macvlan parent) and 40 (mgmt)
+  # are VLAN sub-interfaces of the trunk on a bare-metal node. Keep in lockstep
+  # with patches/cilium-cni.yaml and values.bpf.vlanBypass in
+  # kubernetes/infrastructure/cilium/helmrelease.yaml.
+  vlan-bpf-bypass: "30,40"
  # bpf-policy-map-max specifies the maximum number of entries in endpoint
  # policy map (per endpoint). Keep in lockstep with patches/cilium-cni.yaml and
  # values.bpf.policyMapMax in kubernetes/infrastructure/cilium/helmrelease.yaml.
@@ -324,6 +324,13 @@ cluster:
          # Specifies the ratio (0.0-1.0] of total system memory to use for dynamic
          # sizing of the TCP CT, non-TCP CT, NAT and policy BPF maps.
          bpf-map-dynamic-size-ratio: "0.0025"
+          # VLAN ids whose tagged frames bypass the eBPF datapath rather than being
+          # dropped by the VLAN filter. 30 (v30iot, the macvlan parent) and 40 (mgmt)
+          # are VLAN sub-interfaces of the trunk on a bare-metal node, so without this
+          # a cold rebuild comes up with the v30iot macvlan pods unable to DHCP. Keep
+          # in lockstep with kubernetes/talos/cilium.yaml and values.bpf.vlanBypass in
+          # kubernetes/infrastructure/cilium/helmrelease.yaml.
+          vlan-bpf-bypass: "30,40"
          # bpf-policy-map-max specifies the maximum number of entries in endpoint
          # policy map (per endpoint). Keep in lockstep with values.bpf.policyMapMax
          # in kubernetes/infrastructure/cilium/helmrelease.yaml, and with the third
Author	SHA1	Message	Date
issue-fix-bot	fdd303d7a0	fix: [alert] KubeContainerWaiting (home-assistant) (#592 ) ci/woodpecker/pr/ci-manifests-test Pipeline was successful Details ci/woodpecker/pr/ci-scripts-test Pipeline was successful Details ci/woodpecker/pr/pr-review Pipeline was successful Details	2026-08-25 23:47:26 +00:00
k.renner	1f15062b80	Merge pull request 'chore(rook-ceph): drop k8s-3 from the storage fleet for neptune's conversion' (#597 ) from ceph-drop-neptune into master ci/woodpecker/cron/issue-fix Pipeline was successful Details ci/woodpecker/cron/alert-monitor Pipeline was successful Details Reviewed-on: k.renner/homelab#597	2026-08-25 22:54:19 +00:00
claude-botandClaude Opus 5	8e97253d6c	chore(rook-ceph): drop k8s-3 from the storage fleet for neptune's conversion ci/woodpecker/pr/ci-scripts-test Pipeline was successful Details ci/woodpecker/pr/ci-manifests-test Pipeline was successful Details ci/woodpecker/pr/pr-review Pipeline was successful Details neptune is the last node to convert to bare metal, and its OSD drive is swapped in the same window. Removing k8s-3 from the explicit node list first means Rook stops managing that OSD before the drive it lives on goes away, rather than recreating it mid-reconcile and backfilling onto a disk that is about to be pulled. The device comment for k8s-3 goes with it: it described the PCI-passed-through drive that node saw as a Proxmox guest, which stops being true here. Ceph runs on two OSDs and size-3 pools stay degraded until the node rejoins. cephfs-metadata is already at 2/1 for exactly this window. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-26 00:47:52 +02:00