fix: [alert] KubeContainerWaiting (home-assistant) (#592)

Reviewed-on: k.renner/homelab#597

kubernetes/RUNBOOK-baremetal-conversion.md

@@ -172,12 +172,22 @@ use the secureboot installer image too. That is a separate project.
 ## 5. Install
 
 Boot the stick. Maintenance mode DHCPs untagged on every link; on these hosts
-that lands on `192.168.8.x`, which is routable from the workstation.
+that lands on `192.168.8.x`, which is routable from the workstation. It DHCPs on
+*every* link, so one machine answers on several addresses at once — they are the
+same host, not several hosts. `get disks` on each confirms it.
 
-Maintenance mode is effectively write-only — `talosctl get` has no `--insecure`
-and `version` returns "API is not implemented in maintenance mode". You cannot
-inspect disks or links before applying. Verify serials and MACs from the host's
-PVE beforehand, in step 3.
+Maintenance mode does serve `talosctl get`. `--insecure` is a flag on the
+subcommand rather than a global, so it must come *after* `get` — placed before
+it, cobra resolves to the legacy `talosctl disks` command and reports
+`unknown flag: --insecure`:
+
+```sh
+talosctl -n <maintenance-ip> get disks --insecure
+talosctl -n <maintenance-ip> get links --insecure
+```
+
+Verify serials and MACs here, on the machine itself, against the config about to
+be applied.
 
 Build the config from the **live cluster's secrets**, not a fresh
 `talosctl gen config`, or the node forms a new cluster instead of joining:
@@ -202,13 +212,58 @@ Strip the generated `HostnameConfig` document — it collides with
 against a base with no `inlineManifests` (issue #581). Use only the additive
 second document.
 
-Validate, then apply:
+Validate:
 
 ```sh
 talosctl validate --config install.yaml --mode metal
-talosctl apply-config --insecure -n <maintenance-ip> --file install.yaml
 ```
 
+### Install minimal, complete on the running node
+
+Do **not** apply the full per-node config in maintenance mode. A document that
+references an interface which does not exist yet hangs the install: the node
+sits at `STAGE: Installing` indefinitely with `KUBELET n/a`, `APISERVER n/a`,
+and the console scrolling
+
+```
+k8s.NodeApplyController: error getting node: nodes "k8s-N" not found
+```
+
+That message is a symptom of an install that never finished, not a Kubernetes or
+networking fault — the node holds a DHCP address and is reachable throughout. On
+saturn the offenders were the `ens19` VLAN document and the sysctl naming it,
+`net.ipv6.conf.ens19.disable_ipv6`. Three cold installs hung on it; the identical
+config applied cleanly once the node was up. Wiping disks makes no difference —
+it is not leftover state.
+
+**Stage 1** — the smallest config that installs and joins, plus `EPHEMERAL`,
+which is one-shot (section 7) and so cannot be deferred to stage 2:
+
+```
+machine.network.hostname, machine.install.diskSelector, certSANs, nodeLabels
+LinkConfig    <uplink>
+VLANConfig    mgmt0
+DHCPv4Config  mgmt0
+VolumeConfig  EPHEMERAL
+```
+
+```sh
+talosctl apply-config --insecure -n <maintenance-ip> --file install-stage1.yaml
+```
+
+**Stage 2** — once the node is `Ready` and `talosctl etcd members` lists it,
+apply the full config to the running node. `--mode=try` reverts on its own if the
+network config is wrong, so it cannot strand the node; re-apply with
+`--mode=auto` to persist.
+
+```sh
+talosctl -e <node-ip> -n <node-ip> apply-config --mode=try --timeout=3m --file install.yaml
+talosctl -e <node-ip> -n <node-ip> apply-config --mode=auto --file install.yaml
+```
+
+Both applied without a reboot on saturn. Confirm `ens19`, `mgmt0` and the Ceph
+link are up afterwards with `talosctl get links` and `get addresses`.
+
 ## 6. The network config will strand the node if it is wrong
 
 Talos stops running default DHCP operators **for the entire machine** as soon as

kubernetes/infrastructure/cilium/helmrelease.yaml

@@ -77,6 +77,21 @@ spec:
       # redirects the socket directly to the backend pod instead of sending
       # traffic out to the VIP and back (which fails with L2 announcements).
       lbExternalClusterIP: true
+      # VLAN ids whose tagged frames are passed to the stack instead of being
+      # dropped by Cilium's VLAN filter. Unset, the filter drops every tagged
+      # frame the datapath sees — which is what strands the v30iot macvlan
+      # pods on a converted node: there `ens19` (VLAN 30) and `mgmt0`
+      # (VLAN 40) are VLAN sub-interfaces of the enp2s0f0np0 trunk, so a
+      # macvlan child's DHCP DISCOVER leaves tagged and dies as
+      # DROP_VLAN_FILTER. The CNI DHCP daemon reports that as "no buffer
+      # space available" (a tc drop surfaces to an AF_PACKET sender as
+      # ENOBUFS, not as a timeout) and the pod sits in ContainerCreating
+      # forever — issue #592, predicted in RUNBOOK-baremetal-conversion.md.
+      # Lists only the two VLANs a node terminates; [0] would bypass the
+      # filter for every id. Takes effect on agent restart.
+      # Keep in lockstep with vlan-bpf-bypass in
+      # kubernetes/talos/patches/cilium-cni.yaml and kubernetes/talos/cilium.yaml.
+      vlanBypass: [30, 40]
     cni:
       # Multus (kubernetes/infrastructure/multus) runs alongside Cilium and
       # writes 00-multus.conf to /etc/cni/net.d so kubelet picks multus

kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml

@@ -167,34 +167,37 @@ spec:
   # impossible: the OSD is recreated within a reconcile and starts backfilling
   # onto the drive being replaced.
   #
-  # nvme0n1 on k8s-2/k8s-3 is the NVMe drive PCI-passed-through from saturn and
-  # neptune, and is the only block device those guests can see, so a bare device
-  # name is unambiguous there.
-  #
-  # k8s-1 is bare metal and sees three NVMe drives, so it names its OSD by
-  # /dev/disk/by-id path instead. A bare name would be actively wrong: Talos
-  # enumerates this machine's drives in a different order than Proxmox did, and
-  # nvme0n1 here is the Kingston SYSTEM disk. Rook would probably decline it for
-  # carrying partitions, but that is luck rather than intent. The by-id path is
-  # derived from model and serial and does not depend on enumeration at all.
+  # k8s-1 and k8s-2 are bare metal and see three NVMe drives each, so they name
+  # their OSDs by /dev/disk/by-id path instead. A bare name would be actively
+  # wrong: Talos enumerates these machines' drives in a different order than
+  # Proxmox did, and nvme0n1 there is the Kingston SYSTEM disk. Rook would
+  # probably decline it for carrying partitions, but that is luck rather than
+  # intent. The by-id path is derived from model and serial and does not depend
+  # on enumeration at all.
   #
+  #   k8s-1 (jupiter)
   #   nvme0n1  50026B7382B48B86  Kingston  — Talos system disk
   #   nvme1n1  2332E86B749A      Crucial   — Talos EPHEMERAL (/var)
   #   nvme2n1  S437NX0M400426    PM983     — this OSD
+  #
+  #   k8s-2 (saturn)
+  #   nvme0n1  50026B7382DA5F6E  Kingston  — Talos system disk
+  #   nvme1n1  2332E86B74BC      Crucial   — Talos EPHEMERAL (/var)
+  #   nvme2n1  S437NX0M403161    PM983     — this OSD
   storage:
     useAllNodes: false
     useAllDevices: false
     nodes:
-      # k8s-2 is absent while saturn's OSD drive is swapped and the node is
+      # k8s-3 is absent while neptune's OSD drive is swapped and the node is
       # converted to bare metal. RE-ADD IT once the PM983 is in and the node has
-      # rejoined, naming the device by /dev/disk/by-id as k8s-1 does — bare
-      # metal sees three NVMe drives and the bare name is not the OSD.
-      - name: k8s-3
-        devices:
-          - name: nvme0n1
+      # rejoined, naming the device by /dev/disk/by-id as the other two do —
+      # bare metal sees three NVMe drives and the bare name is not the OSD.
       - name: k8s-1
         devices:
           - fullpath: /dev/disk/by-id/nvme-SAMSUNG_MZQLB960HAJR-00007_S437NX0M400426
+      - name: k8s-2
+        devices:
+          - fullpath: /dev/disk/by-id/nvme-SAMSUNG_MZQLB960HAJR-00007_S437NX0M403161
     config:
       osdsPerDevice: "1"
   # Storage layer must never be evicted by kubelet under node pressure

kubernetes/talos/cilium.yaml

@@ -306,6 +306,12 @@ data:
   # Specifies the ratio (0.0-1.0] of total system memory to use for dynamic
   # sizing of the TCP CT, non-TCP CT, NAT and policy BPF maps.
   bpf-map-dynamic-size-ratio: "0.0025"
+  # VLAN ids whose tagged frames bypass the eBPF datapath rather than being
+  # dropped by the VLAN filter. 30 (v30iot, the macvlan parent) and 40 (mgmt)
+  # are VLAN sub-interfaces of the trunk on a bare-metal node. Keep in lockstep
+  # with patches/cilium-cni.yaml and values.bpf.vlanBypass in
+  # kubernetes/infrastructure/cilium/helmrelease.yaml.
+  vlan-bpf-bypass: "30,40"
   # bpf-policy-map-max specifies the maximum number of entries in endpoint
   # policy map (per endpoint). Keep in lockstep with patches/cilium-cni.yaml and
   # values.bpf.policyMapMax in kubernetes/infrastructure/cilium/helmrelease.yaml.

kubernetes/talos/patches/cilium-cni.yaml

@@ -324,6 +324,13 @@ cluster:
           # Specifies the ratio (0.0-1.0] of total system memory to use for dynamic
           # sizing of the TCP CT, non-TCP CT, NAT and policy BPF maps.
           bpf-map-dynamic-size-ratio: "0.0025"
+          # VLAN ids whose tagged frames bypass the eBPF datapath rather than being
+          # dropped by the VLAN filter. 30 (v30iot, the macvlan parent) and 40 (mgmt)
+          # are VLAN sub-interfaces of the trunk on a bare-metal node, so without this
+          # a cold rebuild comes up with the v30iot macvlan pods unable to DHCP. Keep
+          # in lockstep with kubernetes/talos/cilium.yaml and values.bpf.vlanBypass in
+          # kubernetes/infrastructure/cilium/helmrelease.yaml.
+          vlan-bpf-bypass: "30,40"
           # bpf-policy-map-max specifies the maximum number of entries in endpoint
           # policy map (per endpoint). Keep in lockstep with values.bpf.policyMapMax
           # in kubernetes/infrastructure/cilium/helmrelease.yaml, and with the third

opentofu/modules/proxmox_vm/variables.tf

@@ -47,8 +47,13 @@ variable "tags" {
 }
 
 variable "template_name" {
-  type    = string
-  default = "ubuntu-24-04-template"
+  type = string
+  # No default clone source. The ubuntu-24.04-template it used to name was
+  # deleted with saturn's conversion to bare metal, and Proxmox is being
+  # retired, so a default pointing at a template that does not exist would
+  # only fail at apply time. Set it explicitly if a clone source is ever
+  # wanted again.
+  default = null
 }
 
 variable "full_clone" {

opentofu/vm_605_template.tf

@@ -1,71 +0,0 @@
-resource "proxmox_virtual_environment_vm" "ubuntu_24_04_template" {
-  name      = "ubuntu-24.04-template"
-  node_name = "saturn"
-  vm_id     = 605
-
-  started = false
-  template = true
-  on_boot = false
-
-  keyboard_layout = "en-us"
-
-  bios = "seabios"
-
-  agent {
-    enabled = true
-  }
-
-  cpu {
-    cores = 2
-    type  = "x86-64-v3"
-  }
-
-  memory {
-    dedicated = 2048
-  }
-
-  initialization {
-    datastore_id = "local-zfs"
-
-    ip_config {
-      ipv4 {
-        address = "dhcp"
-      }
-    }
-
-    user_account {
-      username = "ladmin"
-      keys = [
-        "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPzFNBP3bq2m/6JqDqod9w+FP/uxXxHMolqhq3Ojflr1 krenner@fortress"
-      ]
-    }
-  }
-
-  disk {
-    datastore_id = "local-zfs"
-    interface    = "scsi0"
-    size         = 32
-    file_format  = "raw"
-    iothread     = false
-  }
-
-  network_device {
-    bridge = "v40srv"
-    model  = "virtio"
-  }
-
-  serial_device {}
-
-  scsi_hardware = "virtio-scsi-pci"
-
-  lifecycle {
-    ignore_changes = [
-      node_name,
-      started,
-      initialization,
-      disk,
-      cpu,
-      memory
-    ]
-  }
-}