fix: [alert] KubePodNotReady (home-assistant) (#590)

Reviewed-on: k.renner/homelab#589

kubernetes/RUNBOOK-baremetal-conversion.md

@@ -37,6 +37,13 @@ all in state that outlived the VM.
       `fdisk -l` — blkid labels lie after a rewrite.
 - [ ] Physical access. Secure Boot must be turned off (see below), and that is
       the one step nothing can do remotely.
+- [ ] **VLAN 30 (v30iot) is tagged on the switch port you are moving to.** As a
+      VM the IoT NIC came from a Proxmox VNet on the host bridge; bare metal
+      moves it onto the X710 port 0 trunk, and that need not be the same
+      physical port. Confirm it in the UniFi port profile *before* the window —
+      switch config is not managed here (`opentofu/unifi.tf` is examples only).
+      jupiter went in without this check and `ens19` took 434 packets in 155
+      minutes, wedging every v30iot macvlan pod in `ContainerCreating` (#590).
 
 ## 1. Evacuate the OSD
 
@@ -319,8 +326,16 @@ talosctl -n <node> etcd members                     # 3 members
 ceph status                                         # 3 mons in quorum, 3 osds
 ceph osd tree                                       # OSD under the right zone/host
 ceph device ls                                      # PM983 backing the new OSD
+talosctl -n <node> read /proc/net/dev                # ens19 RX climbing
 ```
 
+`ens19` is the one thing nothing else here proves. The node joins, etcd and Ceph
+all ride VLAN 40, so a node with VLAN 30 missing looks entirely healthy until an
+IoT workload happens to land on it. Read `/proc/net/dev` twice a minute apart:
+if `ens19` RX does not move while `mgmt0` does, VLAN 30 is not reaching this
+port — go back to the pre-check. `node_network_carrier` and `up` prove nothing,
+a VLAN subinterface inherits carrier from its parent.
+
 Then undo the window changes: `cephfs-metadata` back to size 3, immich back to
 1 replica, and `ceph config set osd osd_mclock_profile balanced` if it was
 raised to `high_recovery_ops` to speed the evacuation.
@@ -334,7 +349,10 @@ The retired Crucial leaves a stale entry in `ceph device ls`. Remove it with
   cold rebuild is broken independently of this work.
 - Manual Woodpecker pipelines are broken (issue #582), so anything gated on
   `PIPELINE == "..."` must be run another way.
-- Cilium's eBPF datapath silently drops VLAN-tagged traffic unless
-  `bpf.vlanBypass` is set. Both `mgmt0` and `ens19` are VLAN interfaces on a
-  converted node. Not observed on jupiter, but it is the first place to look if
-  the v30iot macvlan workloads misbehave there.
+- Cilium's eBPF datapath can silently drop VLAN-tagged traffic unless
+  `bpf.vlanBypass` is set, and `vlan-bpf-bypass` is indeed empty in
+  `cilium-config`. Worth knowing — but it was *not* what killed jupiter's
+  `ens19` (#590), so check the packet counters before reaching for it. A VLAN
+  subinterface's RX counter increments at 8021q demux, upstream of tc-ingress
+  BPF on `ens19`, so a Cilium drop there still leaves RX high; and a drop on the
+  shared parent would have starved `mgmt0` too. Both were the other way round.

opentofu/modules/proxmox_vm/variables.tf

@@ -47,8 +47,13 @@ variable "tags" {
 }
 
 variable "template_name" {
-  type    = string
-  default = "ubuntu-24-04-template"
+  type = string
+  # No default clone source. The ubuntu-24.04-template it used to name was
+  # deleted with saturn's conversion to bare metal, and Proxmox is being
+  # retired, so a default pointing at a template that does not exist would
+  # only fail at apply time. Set it explicitly if a clone source is ever
+  # wanted again.
+  default = null
 }
 
 variable "full_clone" {

opentofu/vm_605_template.tf

@@ -1,71 +0,0 @@
-resource "proxmox_virtual_environment_vm" "ubuntu_24_04_template" {
-  name      = "ubuntu-24.04-template"
-  node_name = "saturn"
-  vm_id     = 605
-
-  started = false
-  template = true
-  on_boot = false
-
-  keyboard_layout = "en-us"
-
-  bios = "seabios"
-
-  agent {
-    enabled = true
-  }
-
-  cpu {
-    cores = 2
-    type  = "x86-64-v3"
-  }
-
-  memory {
-    dedicated = 2048
-  }
-
-  initialization {
-    datastore_id = "local-zfs"
-
-    ip_config {
-      ipv4 {
-        address = "dhcp"
-      }
-    }
-
-    user_account {
-      username = "ladmin"
-      keys = [
-        "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPzFNBP3bq2m/6JqDqod9w+FP/uxXxHMolqhq3Ojflr1 krenner@fortress"
-      ]
-    }
-  }
-
-  disk {
-    datastore_id = "local-zfs"
-    interface    = "scsi0"
-    size         = 32
-    file_format  = "raw"
-    iothread     = false
-  }
-
-  network_device {
-    bridge = "v40srv"
-    model  = "virtio"
-  }
-
-  serial_device {}
-
-  scsi_hardware = "virtio-scsi-pci"
-
-  lifecycle {
-    ignore_changes = [
-      node_name,
-      started,
-      initialization,
-      disk,
-      cpu,
-      memory
-    ]
-  }
-}