fix: [alert] KubePodNotReady (home-assistant) (#590)

kubernetes/RUNBOOK-baremetal-conversion.md

@@ -37,6 +37,13 @@ all in state that outlived the VM.
       `fdisk -l` — blkid labels lie after a rewrite.
 - [ ] Physical access. Secure Boot must be turned off (see below), and that is
       the one step nothing can do remotely.
+- [ ] **VLAN 30 (v30iot) is tagged on the switch port you are moving to.** As a
+      VM the IoT NIC came from a Proxmox VNet on the host bridge; bare metal
+      moves it onto the X710 port 0 trunk, and that need not be the same
+      physical port. Confirm it in the UniFi port profile *before* the window —
+      switch config is not managed here (`opentofu/unifi.tf` is examples only).
+      jupiter went in without this check and `ens19` took 434 packets in 155
+      minutes, wedging every v30iot macvlan pod in `ContainerCreating` (#590).
 
 ## 1. Evacuate the OSD
 
@@ -172,22 +179,12 @@ use the secureboot installer image too. That is a separate project.
 ## 5. Install
 
 Boot the stick. Maintenance mode DHCPs untagged on every link; on these hosts
-that lands on `192.168.8.x`, which is routable from the workstation. It DHCPs on
-*every* link, so one machine answers on several addresses at once — they are the
-same host, not several hosts. `get disks` on each confirms it.
+that lands on `192.168.8.x`, which is routable from the workstation.
 
-Maintenance mode does serve `talosctl get`. `--insecure` is a flag on the
-subcommand rather than a global, so it must come *after* `get` — placed before
-it, cobra resolves to the legacy `talosctl disks` command and reports
-`unknown flag: --insecure`:
-
-```sh
-talosctl -n <maintenance-ip> get disks --insecure
-talosctl -n <maintenance-ip> get links --insecure
-```
-
-Verify serials and MACs here, on the machine itself, against the config about to
-be applied.
+Maintenance mode is effectively write-only — `talosctl get` has no `--insecure`
+and `version` returns "API is not implemented in maintenance mode". You cannot
+inspect disks or links before applying. Verify serials and MACs from the host's
+PVE beforehand, in step 3.
 
 Build the config from the **live cluster's secrets**, not a fresh
 `talosctl gen config`, or the node forms a new cluster instead of joining:
@@ -212,58 +209,13 @@ Strip the generated `HostnameConfig` document — it collides with
 against a base with no `inlineManifests` (issue #581). Use only the additive
 second document.
 
-Validate:
+Validate, then apply:
 
 ```sh
 talosctl validate --config install.yaml --mode metal
+talosctl apply-config --insecure -n <maintenance-ip> --file install.yaml
 ```
 
-### Install minimal, complete on the running node
-
-Do **not** apply the full per-node config in maintenance mode. A document that
-references an interface which does not exist yet hangs the install: the node
-sits at `STAGE: Installing` indefinitely with `KUBELET n/a`, `APISERVER n/a`,
-and the console scrolling
-
-```
-k8s.NodeApplyController: error getting node: nodes "k8s-N" not found
-```
-
-That message is a symptom of an install that never finished, not a Kubernetes or
-networking fault — the node holds a DHCP address and is reachable throughout. On
-saturn the offenders were the `ens19` VLAN document and the sysctl naming it,
-`net.ipv6.conf.ens19.disable_ipv6`. Three cold installs hung on it; the identical
-config applied cleanly once the node was up. Wiping disks makes no difference —
-it is not leftover state.
-
-**Stage 1** — the smallest config that installs and joins, plus `EPHEMERAL`,
-which is one-shot (section 7) and so cannot be deferred to stage 2:
-
-```
-machine.network.hostname, machine.install.diskSelector, certSANs, nodeLabels
-LinkConfig    <uplink>
-VLANConfig    mgmt0
-DHCPv4Config  mgmt0
-VolumeConfig  EPHEMERAL
-```
-
-```sh
-talosctl apply-config --insecure -n <maintenance-ip> --file install-stage1.yaml
-```
-
-**Stage 2** — once the node is `Ready` and `talosctl etcd members` lists it,
-apply the full config to the running node. `--mode=try` reverts on its own if the
-network config is wrong, so it cannot strand the node; re-apply with
-`--mode=auto` to persist.
-
-```sh
-talosctl -e <node-ip> -n <node-ip> apply-config --mode=try --timeout=3m --file install.yaml
-talosctl -e <node-ip> -n <node-ip> apply-config --mode=auto --file install.yaml
-```
-
-Both applied without a reboot on saturn. Confirm `ens19`, `mgmt0` and the Ceph
-link are up afterwards with `talosctl get links` and `get addresses`.
-
 ## 6. The network config will strand the node if it is wrong
 
 Talos stops running default DHCP operators **for the entire machine** as soon as
@@ -374,8 +326,16 @@ talosctl -n <node> etcd members                     # 3 members
 ceph status                                         # 3 mons in quorum, 3 osds
 ceph osd tree                                       # OSD under the right zone/host
 ceph device ls                                      # PM983 backing the new OSD
+talosctl -n <node> read /proc/net/dev                # ens19 RX climbing
 ```
 
+`ens19` is the one thing nothing else here proves. The node joins, etcd and Ceph
+all ride VLAN 40, so a node with VLAN 30 missing looks entirely healthy until an
+IoT workload happens to land on it. Read `/proc/net/dev` twice a minute apart:
+if `ens19` RX does not move while `mgmt0` does, VLAN 30 is not reaching this
+port — go back to the pre-check. `node_network_carrier` and `up` prove nothing,
+a VLAN subinterface inherits carrier from its parent.
+
 Then undo the window changes: `cephfs-metadata` back to size 3, immich back to
 1 replica, and `ceph config set osd osd_mclock_profile balanced` if it was
 raised to `high_recovery_ops` to speed the evacuation.
@@ -389,7 +349,10 @@ The retired Crucial leaves a stale entry in `ceph device ls`. Remove it with
   cold rebuild is broken independently of this work.
 - Manual Woodpecker pipelines are broken (issue #582), so anything gated on
   `PIPELINE == "..."` must be run another way.
-- Cilium's eBPF datapath silently drops VLAN-tagged traffic unless
-  `bpf.vlanBypass` is set. Both `mgmt0` and `ens19` are VLAN interfaces on a
-  converted node. Not observed on jupiter, but it is the first place to look if
-  the v30iot macvlan workloads misbehave there.
+- Cilium's eBPF datapath can silently drop VLAN-tagged traffic unless
+  `bpf.vlanBypass` is set, and `vlan-bpf-bypass` is indeed empty in
+  `cilium-config`. Worth knowing — but it was *not* what killed jupiter's
+  `ens19` (#590), so check the packet counters before reaching for it. A VLAN
+  subinterface's RX counter increments at 8021q demux, upstream of tc-ingress
+  BPF on `ens19`, so a Cilium drop there still leaves RX high; and a drop on the
+  shared parent would have starved `mgmt0` too. Both were the other way round.

kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml

@@ -167,44 +167,34 @@ spec:
   # impossible: the OSD is recreated within a reconcile and starts backfilling
   # onto the drive being replaced.
   #
-  # All three nodes are bare metal and see three NVMe drives each, so they name
-  # their OSDs by /dev/disk/by-id path. A bare name would be actively wrong:
-  # Talos enumerates these machines' drives in a different order than Proxmox
-  # did, and the kernel name lands on the Kingston SYSTEM disk or on EPHEMERAL
-  # depending on the boot. Rook would probably decline the system disk for
+  # nvme0n1 on k8s-2/k8s-3 is the NVMe drive PCI-passed-through from saturn and
+  # neptune, and is the only block device those guests can see, so a bare device
+  # name is unambiguous there.
+  #
+  # k8s-1 is bare metal and sees three NVMe drives, so it names its OSD by
+  # /dev/disk/by-id path instead. A bare name would be actively wrong: Talos
+  # enumerates this machine's drives in a different order than Proxmox did, and
+  # nvme0n1 here is the Kingston SYSTEM disk. Rook would probably decline it for
   # carrying partitions, but that is luck rather than intent. The by-id path is
   # derived from model and serial and does not depend on enumeration at all.
   #
-  # k8s-3 demonstrated why: it enumerated the Kingston as nvme0n1 on one boot
-  # and the Crucial as nvme0n1 on the very next one.
-  #
-  #   k8s-1 (jupiter)
-  #   50026B7382B48B86  Kingston  — Talos system disk
-  #   2332E86B749A      Crucial   — Talos EPHEMERAL (/var)
-  #   S437NX0M400426    PM983     — this OSD
-  #
-  #   k8s-2 (saturn)
-  #   50026B7382DA5F6E  Kingston  — Talos system disk
-  #   2332E86B74BC      Crucial   — Talos EPHEMERAL (/var)
-  #   S437NX0M403161    PM983     — this OSD
-  #
-  #   k8s-3 (neptune)
-  #   50026B7382B48B90  Kingston  — Talos system disk
-  #   2326E6E8E67D      Crucial   — Talos EPHEMERAL (/var)
-  #   S437NF0M501176    PM983     — this OSD
+  #   nvme0n1  50026B7382B48B86  Kingston  — Talos system disk
+  #   nvme1n1  2332E86B749A      Crucial   — Talos EPHEMERAL (/var)
+  #   nvme2n1  S437NX0M400426    PM983     — this OSD
   storage:
     useAllNodes: false
     useAllDevices: false
     nodes:
+      # k8s-2 is absent while saturn's OSD drive is swapped and the node is
+      # converted to bare metal. RE-ADD IT once the PM983 is in and the node has
+      # rejoined, naming the device by /dev/disk/by-id as k8s-1 does — bare
+      # metal sees three NVMe drives and the bare name is not the OSD.
+      - name: k8s-3
+        devices:
+          - name: nvme0n1
       - name: k8s-1
         devices:
           - fullpath: /dev/disk/by-id/nvme-SAMSUNG_MZQLB960HAJR-00007_S437NX0M400426
-      - name: k8s-2
-        devices:
-          - fullpath: /dev/disk/by-id/nvme-SAMSUNG_MZQLB960HAJR-00007_S437NX0M403161
-      - name: k8s-3
-        devices:
-          - fullpath: /dev/disk/by-id/nvme-SAMSUNG_MZQLB960HAJR-00007_S437NF0M501176
     config:
       osdsPerDevice: "1"
   # Storage layer must never be evicted by kubelet under node pressure

kubernetes/infrastructure/rook-ceph-cluster/storageclass.yaml

@@ -5,24 +5,31 @@ metadata:
   namespace: rook-ceph
 spec:
   failureDomain: host
-  # size 3. This pool sat at 2 while the OSDs were Crucial P3 QLC drives: with
-  # 3 OSDs on 3 hosts, size=3 put every write on every drive, so per-drive write
-  # volume equalled total cluster write volume and all three wore at an
-  # identical rate (measured 2026-08-19: 80/82/87% endurance used, ~0.13
-  # points/day, all three projected to exhaust between Nov 2026 and Jan 2027 —
-  # see #434). Replication does not protect against that, because the wear was
-  # correlated by construction rather than independent.
+  # size 2, NOT 3. With 3 OSDs on 3 hosts, size=3 meant every write landed on
+  # every drive in the cluster: per-drive write volume equalled total cluster
+  # write volume, and all three drives wore at an identical rate and would have
+  # reached end-of-rated-life together (measured 2026-08-19: 80/82/87% endurance
+  # used, ~0.13 points/day, all three exhausted between Nov 2026 and Jan 2027 —
+  # see #434). Replication does not protect against that failure mode, because
+  # wear-out here is correlated by construction rather than independent.
   #
-  # The OSDs are Samsung PM983 enterprise drives now, at 2-3% of rated
-  # endurance with a far higher write budget, so wear-out is no longer the
-  # binding constraint and the correlation argument no longer applies.
+  # size 2 cuts per-drive write volume by a third and desynchronises the wear.
   #
-  # min_size is left at the size-3 default of 2. The explicit min_size 1
-  # override existed only because Ceph does not lower min_size when size is
-  # reduced, which would have left the pool 2/2 — where any single OSD down
+  # min_size 1 is REQUIRED and deliberate. Ceph does not lower min_size when
+  # size is reduced — it was 2 here, and leaving it would make the pool 2/2,
+  # where any single OSD down (every rolling node reboot, every OSD restart)
   # blocks IO for its PGs and hangs every RBD volume in the cluster.
+  #
+  # The trade this accepts: while the pool is degraded (one OSD down), a second
+  # drive failing loses data, and with all three drives at 80%+ endurance that
+  # window is not hypothetical. Mitigations that already exist: CNPG barman and
+  # Velero both ship offsite to Hetzner (opentofu/object_storage.tf), so the
+  # exposure is the RPO of those backups, not total loss. Keep node reboots
+  # serialised and wait for HEALTH_OK between them.
   replicated:
-    size: 3
+    size: 2
+  parameters:
+    min_size: "1"
 ---
 apiVersion: storage.k8s.io/v1
 kind: StorageClass
@@ -80,26 +87,39 @@ metadata:
   name: cephfs
   namespace: rook-ceph
 spec:
-  # size 3 with no min_size override. This pool holds ~253 MiB against the data
-  # pool's 280 GiB, and losing it would destroy the whole filesystem rather
-  # than just the objects on the lost OSD, so it is the last pool that should
-  # ever run thin.
+  # TEMPORARY — RESTORE TO size 3 / no min_size OVERRIDE ONCE SATURN AND
+  # NEPTUNE HAVE BOTH BEEN CONVERTED TO BARE METAL. It stays at 2/1 across all
+  # three conversions rather than being toggled per node: each change
+  # re-replicates the pool, and the window reopens with the next node anyway.
   #
-  # It drops to 2/1 whenever one of the three hosts is out for longer than a
-  # reboot. size 3 on two remaining hosts leaves every PG undersized at two
-  # copies with min_size 2, so a single further OSD loss falls below min_size
-  # and blocks IO on every CephFS mount cluster-wide. At 2/1 the same two
-  # copies are active+clean instead, and a further loss degrades rather than
-  # blocks. See RUNBOOK-baremetal-conversion.md.
+  # The standing trade for this pool is size 3: it holds ~253 MiB against the
+  # data pool's 280 GiB, so it contributes nothing measurable to drive wear —
+  # the entire reason data0 dropped to 2 — while losing it would destroy the
+  # whole filesystem, not just the objects on the lost OSD. Paying 3x on a
+  # quarter of a gigabyte to keep that intact is the right trade in steady
+  # state.
+  #
+  # It does not hold while one of three hosts is out. size 3 on two remaining
+  # hosts leaves every PG undersized at two copies with min_size 2, so a single
+  # further OSD loss drops the pool below min_size and blocks IO on every
+  # CephFS mount cluster-wide. At 2/1 the same two copies are active+clean
+  # instead, and a further loss degrades rather than blocks — availability
+  # bought at the cost of accepting writes onto a single copy while degraded.
+  #
+  # See RUNBOOK-baremetal-conversion.md.
   metadataPool:
     replicated:
-      size: 3
+      size: 2
+    parameters:
+      min_size: "1"
   dataPools:
     - name: data0
-      # size 3 for the same reasons as replicapool above — see the comment
-      # there for why this pool sat at 2 while the Crucials backed the OSDs.
+      # size 2 / min_size 1 for the same reasons as replicapool above — see
+      # the comment there for the full rationale and the accepted trade.
       replicated:
-        size: 3
+        size: 2
+      parameters:
+        min_size: "1"
   metadataServer:
     activeCount: 1
     activeStandby: true

kubernetes/talos/patches/k8s-2.yaml

@@ -1,6 +1,7 @@
 # k8s-2 runs Talos directly on saturn's MS-01 hardware. Structurally identical
 # to k8s-1.yaml — see that file for the reasoning behind each choice; only the
-# serials, MACs and addresses differ.
+# serials, MACs and addresses differ. k8s-3 is still a Proxmox VM and keeps the
+# VM-era layout until neptune is converted.
 machine:
   network:
     hostname: k8s-2

kubernetes/talos/patches/k8s-3.yaml

@@ -1,82 +1,31 @@
-# k8s-3 runs Talos directly on neptune's MS-01 hardware. Structurally identical
-# to k8s-1.yaml — see that file for the reasoning behind each choice; only the
-# serials and addresses differ. neptune was the last node converted, so all
-# three control-plane nodes now share this layout and no Proxmox remains.
 machine:
   network:
     hostname: k8s-3
+    # Shared control-plane VIP — see k8s-1.yaml for the busPath rationale.
+    interfaces:
+      - deviceSelector:
+          busPath: "0000:06:12.0"
+        dhcp: true
+        vip:
+          ip: 10.2.40.60
+      # v30iot: link only, no host addressing — see k8s-1.yaml.
+      - deviceSelector:
+          busPath: "0000:06:13.0"
+        dhcp: false
+      # See k8s-1.yaml for context on the dedicated Ceph cluster network.
+      - deviceSelector:
+          driver: i40e
+        addresses:
+          - 10.99.0.3/29
+        mtu: 9000
   sysctls:
     # v30iot NIC carries no host addressing — see k8s-1.yaml.
     net.ipv6.conf.ens19.disable_ipv6: "1"
   install:
-    # Three NVMe drives are visible on bare metal. Select the installer target
-    # by serial so it cannot land on the OSD drive or on EPHEMERAL, whatever
-    # order the kernel enumerates them in — Talos and Proxmox do not agree on
-    # that order.
-    diskSelector:
-      serial: 50026B7382B48B90
+    disk: /dev/vda
   certSANs:
     - k8s-3.srv.renner.uno
   nodeLabels:
     topology.kubernetes.io/zone: neptune
 cluster:
   allowSchedulingOnControlPlanes: true
----
-# EPHEMERAL (/var) goes on the retired Crucial P3, not the system disk. That
-# drive is at 83% of its rated endurance and is finished as an OSD, but
-# everything EPHEMERAL holds is reconstructible, so it is consumable in this
-# role. Available spare is still 100% and it reports no media errors.
-#
-# ONLY TAKES EFFECT ON FIRST PROVISION. Talos applies volume configuration
-# solely when the volume does not yet exist; applied later it is silently
-# ignored and relocating needs a disk wipe and a boot from install media.
-#
-# Matched by serial deliberately: a broad selector such as
-# `disk.transport == 'nvme' && !system_disk` would match the PM983 as well.
-apiVersion: v1alpha1
-kind: VolumeConfig
-name: EPHEMERAL
-provisioning:
-  diskSelector:
-    match: disk.serial == '2326E6E8E67D'
-  minSize: 50GiB
-  grow: true
----
-apiVersion: v1alpha1
-kind: LinkConfig
-name: enp2s0f0np0
-up: true
----
-apiVersion: v1alpha1
-kind: VLANConfig
-name: mgmt0
-vlanID: 40
-parent: enp2s0f0np0
-up: true
----
-apiVersion: v1alpha1
-kind: DHCPv4Config
-name: mgmt0
----
-# Shared control-plane VIP, held by whichever node wins the etcd campaign.
-apiVersion: v1alpha1
-kind: Layer2VIPConfig
-name: 10.2.40.60
-link: mgmt0
----
-# v30iot: link only, no host addressing — see k8s-1.yaml.
-apiVersion: v1alpha1
-kind: VLANConfig
-name: ens19
-vlanID: 30
-parent: enp2s0f0np0
-up: true
----
-# Dedicated Ceph cluster network — see k8s-1.yaml.
-apiVersion: v1alpha1
-kind: LinkConfig
-name: enp2s0f1np1
-up: true
-mtu: 9000
-addresses:
-  - address: 10.99.0.3/29