fix: [alert] KubePodNotReady (home-assistant) (#590 )

2026-08-25 21:46:31 +00:00
1 changed files with 22 additions and 4 deletions
@@ -37,6 +37,13 @@ all in state that outlived the VM.
      `fdisk -l` — blkid labels lie after a rewrite.
 - [ ] Physical access. Secure Boot must be turned off (see below), and that is
      the one step nothing can do remotely.
+- [ ] **VLAN 30 (v30iot) is tagged on the switch port you are moving to.** As a
+      VM the IoT NIC came from a Proxmox VNet on the host bridge; bare metal
+      moves it onto the X710 port 0 trunk, and that need not be the same
+      physical port. Confirm it in the UniFi port profile *before* the window —
+      switch config is not managed here (`opentofu/unifi.tf` is examples only).
+      jupiter went in without this check and `ens19` took 434 packets in 155
+      minutes, wedging every v30iot macvlan pod in `ContainerCreating` (#590).

 ## 1. Evacuate the OSD

@@ -319,8 +326,16 @@ talosctl -n <node> etcd members                     # 3 members
 ceph status                                         # 3 mons in quorum, 3 osds
 ceph osd tree                                       # OSD under the right zone/host
 ceph device ls                                      # PM983 backing the new OSD
+talosctl -n <node> read /proc/net/dev                # ens19 RX climbing
 ```

+`ens19` is the one thing nothing else here proves. The node joins, etcd and Ceph
+all ride VLAN 40, so a node with VLAN 30 missing looks entirely healthy until an
+IoT workload happens to land on it. Read `/proc/net/dev` twice a minute apart:
+if `ens19` RX does not move while `mgmt0` does, VLAN 30 is not reaching this
+port — go back to the pre-check. `node_network_carrier` and `up` prove nothing,
+a VLAN subinterface inherits carrier from its parent.
+
 Then undo the window changes: `cephfs-metadata` back to size 3, immich back to
 1 replica, and `ceph config set osd osd_mclock_profile balanced` if it was
 raised to `high_recovery_ops` to speed the evacuation.
@@ -334,7 +349,10 @@ The retired Crucial leaves a stale entry in `ceph device ls`. Remove it with
  cold rebuild is broken independently of this work.
 - Manual Woodpecker pipelines are broken (issue #582), so anything gated on
  `PIPELINE == "..."` must be run another way.
- Cilium's eBPF datapath silently drops VLAN-tagged traffic unless
-  `bpf.vlanBypass` is set. Both `mgmt0` and `ens19` are VLAN interfaces on a
-  converted node. Not observed on jupiter, but it is the first place to look if
-  the v30iot macvlan workloads misbehave there.
+- Cilium's eBPF datapath can silently drop VLAN-tagged traffic unless
+  `bpf.vlanBypass` is set, and `vlan-bpf-bypass` is indeed empty in
+  `cilium-config`. Worth knowing — but it was *not* what killed jupiter's
+  `ens19` (#590), so check the packet counters before reaching for it. A VLAN
+  subinterface's RX counter increments at 8021q demux, upstream of tc-ingress
+  BPF on `ens19`, so a Cilium drop there still leaves RX high; and a drop on the
+  shared parent would have starved `mgmt0` too. Both were the other way round.