fix: [alert] KubePodNotReady (rook-ceph) (#591)

Reviewed-on: k.renner/homelab#595

kubernetes/RUNBOOK-baremetal-conversion.md

@@ -80,6 +80,29 @@ pulled anyway and wiping a terabyte of it is wasted time. `--graceful` leaves
 etcd cleanly. Delete the Node object — it carries the `unschedulable` flag from
 the drain, which the rebuilt node would otherwise inherit.
 
+### From here the operator stops reconciling Ceph entirely
+
+Expect this. With `mon.count: 3` and `allowMultiplePerNode: false`
+(`infrastructure/rook-ceph-cluster/cephcluster.yaml`), two nodes fail Rook's
+pre-flight validation, so the operator aborts the *whole* CephCluster reconcile
+and retries with backoff for the length of the window:
+
+```
+failed to perform validation before cluster creation:
+cannot start 3 mons on 2 node(s) when allowMultiplePerNode is false
+```
+
+Ceph is unmanaged, not merely degraded — no mon, mgr, OSD or pool change applies
+until the third node is back. Two knock-on effects worth knowing in advance:
+
+- `infrastructure-ceph-cluster` has `wait: true`, `timeout: 20m` and a
+  CephCluster `status.phase == Ready` health expression, so it fails its health
+  check every cycle. Nearly every app Kustomization `dependsOn` it, so app
+  changes stop landing too. Both clear on their own once the node rejoins.
+- Do **not** drop `mon.count` to 2 or set `allowMultiplePerNode: true` to make
+  the error go away. Two mons have no quorum tolerance at all, and co-locating
+  mons defeats the anti-affinity the three-node layout depends on.
+
 ## 2a. Destroy the VM in OpenTofu — while the host is still a PVE node
 
 Remove `opentofu/vm_15N_k8s_N.tf`, then apply. This **must** happen before the
@@ -172,12 +195,22 @@ use the secureboot installer image too. That is a separate project.
 ## 5. Install
 
 Boot the stick. Maintenance mode DHCPs untagged on every link; on these hosts
-that lands on `192.168.8.x`, which is routable from the workstation.
+that lands on `192.168.8.x`, which is routable from the workstation. It DHCPs on
+*every* link, so one machine answers on several addresses at once — they are the
+same host, not several hosts. `get disks` on each confirms it.
 
-Maintenance mode is effectively write-only — `talosctl get` has no `--insecure`
-and `version` returns "API is not implemented in maintenance mode". You cannot
-inspect disks or links before applying. Verify serials and MACs from the host's
-PVE beforehand, in step 3.
+Maintenance mode does serve `talosctl get`. `--insecure` is a flag on the
+subcommand rather than a global, so it must come *after* `get` — placed before
+it, cobra resolves to the legacy `talosctl disks` command and reports
+`unknown flag: --insecure`:
+
+```sh
+talosctl -n <maintenance-ip> get disks --insecure
+talosctl -n <maintenance-ip> get links --insecure
+```
+
+Verify serials and MACs here, on the machine itself, against the config about to
+be applied.
 
 Build the config from the **live cluster's secrets**, not a fresh
 `talosctl gen config`, or the node forms a new cluster instead of joining:
@@ -202,13 +235,58 @@ Strip the generated `HostnameConfig` document — it collides with
 against a base with no `inlineManifests` (issue #581). Use only the additive
 second document.
 
-Validate, then apply:
+Validate:
 
 ```sh
 talosctl validate --config install.yaml --mode metal
-talosctl apply-config --insecure -n <maintenance-ip> --file install.yaml
 ```
 
+### Install minimal, complete on the running node
+
+Do **not** apply the full per-node config in maintenance mode. A document that
+references an interface which does not exist yet hangs the install: the node
+sits at `STAGE: Installing` indefinitely with `KUBELET n/a`, `APISERVER n/a`,
+and the console scrolling
+
+```
+k8s.NodeApplyController: error getting node: nodes "k8s-N" not found
+```
+
+That message is a symptom of an install that never finished, not a Kubernetes or
+networking fault — the node holds a DHCP address and is reachable throughout. On
+saturn the offenders were the `ens19` VLAN document and the sysctl naming it,
+`net.ipv6.conf.ens19.disable_ipv6`. Three cold installs hung on it; the identical
+config applied cleanly once the node was up. Wiping disks makes no difference —
+it is not leftover state.
+
+**Stage 1** — the smallest config that installs and joins, plus `EPHEMERAL`,
+which is one-shot (section 7) and so cannot be deferred to stage 2:
+
+```
+machine.network.hostname, machine.install.diskSelector, certSANs, nodeLabels
+LinkConfig    <uplink>
+VLANConfig    mgmt0
+DHCPv4Config  mgmt0
+VolumeConfig  EPHEMERAL
+```
+
+```sh
+talosctl apply-config --insecure -n <maintenance-ip> --file install-stage1.yaml
+```
+
+**Stage 2** — once the node is `Ready` and `talosctl etcd members` lists it,
+apply the full config to the running node. `--mode=try` reverts on its own if the
+network config is wrong, so it cannot strand the node; re-apply with
+`--mode=auto` to persist.
+
+```sh
+talosctl -e <node-ip> -n <node-ip> apply-config --mode=try --timeout=3m --file install.yaml
+talosctl -e <node-ip> -n <node-ip> apply-config --mode=auto --file install.yaml
+```
+
+Both applied without a reboot on saturn. Confirm `ens19`, `mgmt0` and the Ceph
+link are up afterwards with `talosctl get links` and `get addresses`.
+
 ## 6. The network config will strand the node if it is wrong
 
 Talos stops running default DHCP operators **for the entire machine** as soon as

kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml

@@ -167,34 +167,40 @@ spec:
   # impossible: the OSD is recreated within a reconcile and starts backfilling
   # onto the drive being replaced.
   #
-  # nvme0n1 on k8s-2/k8s-3 is the NVMe drive PCI-passed-through from saturn and
-  # neptune, and is the only block device those guests can see, so a bare device
-  # name is unambiguous there.
+  # nvme0n1 on k8s-3 is the NVMe drive PCI-passed-through from neptune, and is
+  # the only block device that guest can see, so a bare device name is
+  # unambiguous there.
   #
-  # k8s-1 is bare metal and sees three NVMe drives, so it names its OSD by
-  # /dev/disk/by-id path instead. A bare name would be actively wrong: Talos
-  # enumerates this machine's drives in a different order than Proxmox did, and
-  # nvme0n1 here is the Kingston SYSTEM disk. Rook would probably decline it for
-  # carrying partitions, but that is luck rather than intent. The by-id path is
-  # derived from model and serial and does not depend on enumeration at all.
+  # k8s-1 and k8s-2 are bare metal and see three NVMe drives each, so they name
+  # their OSDs by /dev/disk/by-id path instead. A bare name would be actively
+  # wrong: Talos enumerates these machines' drives in a different order than
+  # Proxmox did, and nvme0n1 there is the Kingston SYSTEM disk. Rook would
+  # probably decline it for carrying partitions, but that is luck rather than
+  # intent. The by-id path is derived from model and serial and does not depend
+  # on enumeration at all.
   #
+  #   k8s-1 (jupiter)
   #   nvme0n1  50026B7382B48B86  Kingston  — Talos system disk
   #   nvme1n1  2332E86B749A      Crucial   — Talos EPHEMERAL (/var)
   #   nvme2n1  S437NX0M400426    PM983     — this OSD
+  #
+  #   k8s-2 (saturn)
+  #   nvme0n1  50026B7382DA5F6E  Kingston  — Talos system disk
+  #   nvme1n1  2332E86B74BC      Crucial   — Talos EPHEMERAL (/var)
+  #   nvme2n1  S437NX0M403161    PM983     — this OSD
   storage:
     useAllNodes: false
     useAllDevices: false
     nodes:
-      # k8s-2 is absent while saturn's OSD drive is swapped and the node is
-      # converted to bare metal. RE-ADD IT once the PM983 is in and the node has
-      # rejoined, naming the device by /dev/disk/by-id as k8s-1 does — bare
-      # metal sees three NVMe drives and the bare name is not the OSD.
       - name: k8s-3
         devices:
           - name: nvme0n1
       - name: k8s-1
         devices:
           - fullpath: /dev/disk/by-id/nvme-SAMSUNG_MZQLB960HAJR-00007_S437NX0M400426
+      - name: k8s-2
+        devices:
+          - fullpath: /dev/disk/by-id/nvme-SAMSUNG_MZQLB960HAJR-00007_S437NX0M403161
     config:
       osdsPerDevice: "1"
   # Storage layer must never be evicted by kubelet under node pressure

opentofu/modules/proxmox_vm/variables.tf

@@ -47,8 +47,13 @@ variable "tags" {
 }
 
 variable "template_name" {
-  type    = string
-  default = "ubuntu-24-04-template"
+  type = string
+  # No default clone source. The ubuntu-24.04-template it used to name was
+  # deleted with saturn's conversion to bare metal, and Proxmox is being
+  # retired, so a default pointing at a template that does not exist would
+  # only fail at apply time. Set it explicitly if a clone source is ever
+  # wanted again.
+  default = null
 }
 
 variable "full_clone" {

opentofu/vm_605_template.tf

@@ -1,71 +0,0 @@
-resource "proxmox_virtual_environment_vm" "ubuntu_24_04_template" {
-  name      = "ubuntu-24.04-template"
-  node_name = "saturn"
-  vm_id     = 605
-
-  started = false
-  template = true
-  on_boot = false
-
-  keyboard_layout = "en-us"
-
-  bios = "seabios"
-
-  agent {
-    enabled = true
-  }
-
-  cpu {
-    cores = 2
-    type  = "x86-64-v3"
-  }
-
-  memory {
-    dedicated = 2048
-  }
-
-  initialization {
-    datastore_id = "local-zfs"
-
-    ip_config {
-      ipv4 {
-        address = "dhcp"
-      }
-    }
-
-    user_account {
-      username = "ladmin"
-      keys = [
-        "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPzFNBP3bq2m/6JqDqod9w+FP/uxXxHMolqhq3Ojflr1 krenner@fortress"
-      ]
-    }
-  }
-
-  disk {
-    datastore_id = "local-zfs"
-    interface    = "scsi0"
-    size         = 32
-    file_format  = "raw"
-    iothread     = false
-  }
-
-  network_device {
-    bridge = "v40srv"
-    model  = "virtio"
-  }
-
-  serial_device {}
-
-  scsi_hardware = "virtio-scsi-pci"
-
-  lifecycle {
-    ignore_changes = [
-      node_name,
-      started,
-      initialization,
-      disk,
-      cpu,
-      memory
-    ]
-  }
-}