diff --git a/kubernetes/RUNBOOK-baremetal-conversion.md b/kubernetes/RUNBOOK-baremetal-conversion.md index 55506642..f7827ca4 100644 --- a/kubernetes/RUNBOOK-baremetal-conversion.md +++ b/kubernetes/RUNBOOK-baremetal-conversion.md @@ -37,6 +37,13 @@ all in state that outlived the VM. `fdisk -l` — blkid labels lie after a rewrite. - [ ] Physical access. Secure Boot must be turned off (see below), and that is the one step nothing can do remotely. +- [ ] **VLAN 30 (v30iot) is tagged on the switch port you are moving to.** As a + VM the IoT NIC came from a Proxmox VNet on the host bridge; bare metal + moves it onto the X710 port 0 trunk, and that need not be the same + physical port. Confirm it in the UniFi port profile *before* the window — + switch config is not managed here (`opentofu/unifi.tf` is examples only). + jupiter went in without this check and `ens19` took 434 packets in 155 + minutes, wedging every v30iot macvlan pod in `ContainerCreating` (#590). ## 1. Evacuate the OSD @@ -319,8 +326,16 @@ talosctl -n etcd members # 3 members ceph status # 3 mons in quorum, 3 osds ceph osd tree # OSD under the right zone/host ceph device ls # PM983 backing the new OSD +talosctl -n read /proc/net/dev # ens19 RX climbing ``` +`ens19` is the one thing nothing else here proves. The node joins, etcd and Ceph +all ride VLAN 40, so a node with VLAN 30 missing looks entirely healthy until an +IoT workload happens to land on it. Read `/proc/net/dev` twice a minute apart: +if `ens19` RX does not move while `mgmt0` does, VLAN 30 is not reaching this +port — go back to the pre-check. `node_network_carrier` and `up` prove nothing, +a VLAN subinterface inherits carrier from its parent. + Then undo the window changes: `cephfs-metadata` back to size 3, immich back to 1 replica, and `ceph config set osd osd_mclock_profile balanced` if it was raised to `high_recovery_ops` to speed the evacuation. @@ -334,7 +349,10 @@ The retired Crucial leaves a stale entry in `ceph device ls`. Remove it with cold rebuild is broken independently of this work. - Manual Woodpecker pipelines are broken (issue #582), so anything gated on `PIPELINE == "..."` must be run another way. -- Cilium's eBPF datapath silently drops VLAN-tagged traffic unless - `bpf.vlanBypass` is set. Both `mgmt0` and `ens19` are VLAN interfaces on a - converted node. Not observed on jupiter, but it is the first place to look if - the v30iot macvlan workloads misbehave there. +- Cilium's eBPF datapath can silently drop VLAN-tagged traffic unless + `bpf.vlanBypass` is set, and `vlan-bpf-bypass` is indeed empty in + `cilium-config`. Worth knowing — but it was *not* what killed jupiter's + `ens19` (#590), so check the packet counters before reaching for it. A VLAN + subinterface's RX counter increments at 8021q demux, upstream of tc-ingress + BPF on `ens19`, so a Cilium drop there still leaves RX high; and a drop on the + shared parent would have starved `mgmt0` too. Both were the other way round.