feat(rook-ceph): restore replicapool and cephfs-data0 to size 3

These two sat at size 2 because of #434, not because of the conversion. With
three Crucial P3 QLC OSDs on three hosts, size 3 put every write on every
drive, so per-drive write volume equalled cluster write volume and all three
wore at an identical rate — 80/82/87% of rated endurance, projected to exhaust
together between Nov 2026 and Jan 2027. Replication is no defence against that,
because the wear was correlated by construction.

The OSDs are Samsung PM983 enterprise drives now, at 2-3% of rated endurance
with a far higher write budget. Wear-out is no longer the binding constraint
and the correlation argument no longer applies, so the pools return to the
durability the cluster is sized for.

The min_size 1 overrides go with them. They existed only because Ceph does not
lower min_size when size is reduced, which would have left the pools 2/2 and
blocked IO on any single OSD going down.

cephfs-metadata's comment loses the clause explaining that data0 sits at 2,
which stops being true here.

cephfs-ci stays at size 2: that pool holds throwaway CI workspaces and is at 2
rather than 1 only because an unreplicated pool makes `ceph osd ok-to-stop`
refuse for every OSD. Unrelated to #434.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kubernetes/RUNBOOK-baremetal-conversion.md

@@ -172,12 +172,22 @@ use the secureboot installer image too. That is a separate project.
 ## 5. Install
 
 Boot the stick. Maintenance mode DHCPs untagged on every link; on these hosts
-that lands on `192.168.8.x`, which is routable from the workstation.
+that lands on `192.168.8.x`, which is routable from the workstation. It DHCPs on
+*every* link, so one machine answers on several addresses at once — they are the
+same host, not several hosts. `get disks` on each confirms it.
 
-Maintenance mode is effectively write-only — `talosctl get` has no `--insecure`
-and `version` returns "API is not implemented in maintenance mode". You cannot
-inspect disks or links before applying. Verify serials and MACs from the host's
-PVE beforehand, in step 3.
+Maintenance mode does serve `talosctl get`. `--insecure` is a flag on the
+subcommand rather than a global, so it must come *after* `get` — placed before
+it, cobra resolves to the legacy `talosctl disks` command and reports
+`unknown flag: --insecure`:
+
+```sh
+talosctl -n <maintenance-ip> get disks --insecure
+talosctl -n <maintenance-ip> get links --insecure
+```
+
+Verify serials and MACs here, on the machine itself, against the config about to
+be applied.
 
 Build the config from the **live cluster's secrets**, not a fresh
 `talosctl gen config`, or the node forms a new cluster instead of joining:
@@ -202,13 +212,58 @@ Strip the generated `HostnameConfig` document — it collides with
 against a base with no `inlineManifests` (issue #581). Use only the additive
 second document.
 
-Validate, then apply:
+Validate:
 
 ```sh
 talosctl validate --config install.yaml --mode metal
-talosctl apply-config --insecure -n <maintenance-ip> --file install.yaml
 ```
 
+### Install minimal, complete on the running node
+
+Do **not** apply the full per-node config in maintenance mode. A document that
+references an interface which does not exist yet hangs the install: the node
+sits at `STAGE: Installing` indefinitely with `KUBELET n/a`, `APISERVER n/a`,
+and the console scrolling
+
+```
+k8s.NodeApplyController: error getting node: nodes "k8s-N" not found
+```
+
+That message is a symptom of an install that never finished, not a Kubernetes or
+networking fault — the node holds a DHCP address and is reachable throughout. On
+saturn the offenders were the `ens19` VLAN document and the sysctl naming it,
+`net.ipv6.conf.ens19.disable_ipv6`. Three cold installs hung on it; the identical
+config applied cleanly once the node was up. Wiping disks makes no difference —
+it is not leftover state.
+
+**Stage 1** — the smallest config that installs and joins, plus `EPHEMERAL`,
+which is one-shot (section 7) and so cannot be deferred to stage 2:
+
+```
+machine.network.hostname, machine.install.diskSelector, certSANs, nodeLabels
+LinkConfig    <uplink>
+VLANConfig    mgmt0
+DHCPv4Config  mgmt0
+VolumeConfig  EPHEMERAL
+```
+
+```sh
+talosctl apply-config --insecure -n <maintenance-ip> --file install-stage1.yaml
+```
+
+**Stage 2** — once the node is `Ready` and `talosctl etcd members` lists it,
+apply the full config to the running node. `--mode=try` reverts on its own if the
+network config is wrong, so it cannot strand the node; re-apply with
+`--mode=auto` to persist.
+
+```sh
+talosctl -e <node-ip> -n <node-ip> apply-config --mode=try --timeout=3m --file install.yaml
+talosctl -e <node-ip> -n <node-ip> apply-config --mode=auto --file install.yaml
+```
+
+Both applied without a reboot on saturn. Confirm `ens19`, `mgmt0` and the Ceph
+link are up afterwards with `talosctl get links` and `get addresses`.
+
 ## 6. The network config will strand the node if it is wrong
 
 Talos stops running default DHCP operators **for the entire machine** as soon as

kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml

@@ -167,34 +167,44 @@ spec:
   # impossible: the OSD is recreated within a reconcile and starts backfilling
   # onto the drive being replaced.
   #
-  # nvme0n1 on k8s-2/k8s-3 is the NVMe drive PCI-passed-through from saturn and
-  # neptune, and is the only block device those guests can see, so a bare device
-  # name is unambiguous there.
-  #
-  # k8s-1 is bare metal and sees three NVMe drives, so it names its OSD by
-  # /dev/disk/by-id path instead. A bare name would be actively wrong: Talos
-  # enumerates this machine's drives in a different order than Proxmox did, and
-  # nvme0n1 here is the Kingston SYSTEM disk. Rook would probably decline it for
+  # All three nodes are bare metal and see three NVMe drives each, so they name
+  # their OSDs by /dev/disk/by-id path. A bare name would be actively wrong:
+  # Talos enumerates these machines' drives in a different order than Proxmox
+  # did, and the kernel name lands on the Kingston SYSTEM disk or on EPHEMERAL
+  # depending on the boot. Rook would probably decline the system disk for
   # carrying partitions, but that is luck rather than intent. The by-id path is
   # derived from model and serial and does not depend on enumeration at all.
   #
-  #   nvme0n1  50026B7382B48B86  Kingston  — Talos system disk
-  #   nvme1n1  2332E86B749A      Crucial   — Talos EPHEMERAL (/var)
-  #   nvme2n1  S437NX0M400426    PM983     — this OSD
+  # k8s-3 demonstrated why: it enumerated the Kingston as nvme0n1 on one boot
+  # and the Crucial as nvme0n1 on the very next one.
+  #
+  #   k8s-1 (jupiter)
+  #   50026B7382B48B86  Kingston  — Talos system disk
+  #   2332E86B749A      Crucial   — Talos EPHEMERAL (/var)
+  #   S437NX0M400426    PM983     — this OSD
+  #
+  #   k8s-2 (saturn)
+  #   50026B7382DA5F6E  Kingston  — Talos system disk
+  #   2332E86B74BC      Crucial   — Talos EPHEMERAL (/var)
+  #   S437NX0M403161    PM983     — this OSD
+  #
+  #   k8s-3 (neptune)
+  #   50026B7382B48B90  Kingston  — Talos system disk
+  #   2326E6E8E67D      Crucial   — Talos EPHEMERAL (/var)
+  #   S437NF0M501176    PM983     — this OSD
   storage:
     useAllNodes: false
     useAllDevices: false
     nodes:
-      # k8s-2 is absent while saturn's OSD drive is swapped and the node is
-      # converted to bare metal. RE-ADD IT once the PM983 is in and the node has
-      # rejoined, naming the device by /dev/disk/by-id as k8s-1 does — bare
-      # metal sees three NVMe drives and the bare name is not the OSD.
-      - name: k8s-3
-        devices:
-          - name: nvme0n1
       - name: k8s-1
         devices:
           - fullpath: /dev/disk/by-id/nvme-SAMSUNG_MZQLB960HAJR-00007_S437NX0M400426
+      - name: k8s-2
+        devices:
+          - fullpath: /dev/disk/by-id/nvme-SAMSUNG_MZQLB960HAJR-00007_S437NX0M403161
+      - name: k8s-3
+        devices:
+          - fullpath: /dev/disk/by-id/nvme-SAMSUNG_MZQLB960HAJR-00007_S437NF0M501176
     config:
       osdsPerDevice: "1"
   # Storage layer must never be evicted by kubelet under node pressure

kubernetes/infrastructure/rook-ceph-cluster/storageclass.yaml

@@ -5,31 +5,24 @@ metadata:
   namespace: rook-ceph
 spec:
   failureDomain: host
-  # size 2, NOT 3. With 3 OSDs on 3 hosts, size=3 meant every write landed on
-  # every drive in the cluster: per-drive write volume equalled total cluster
-  # write volume, and all three drives wore at an identical rate and would have
-  # reached end-of-rated-life together (measured 2026-08-19: 80/82/87% endurance
-  # used, ~0.13 points/day, all three exhausted between Nov 2026 and Jan 2027 —
-  # see #434). Replication does not protect against that failure mode, because
-  # wear-out here is correlated by construction rather than independent.
+  # size 3. This pool sat at 2 while the OSDs were Crucial P3 QLC drives: with
+  # 3 OSDs on 3 hosts, size=3 put every write on every drive, so per-drive write
+  # volume equalled total cluster write volume and all three wore at an
+  # identical rate (measured 2026-08-19: 80/82/87% endurance used, ~0.13
+  # points/day, all three projected to exhaust between Nov 2026 and Jan 2027 —
+  # see #434). Replication does not protect against that, because the wear was
+  # correlated by construction rather than independent.
   #
-  # size 2 cuts per-drive write volume by a third and desynchronises the wear.
+  # The OSDs are Samsung PM983 enterprise drives now, at 2-3% of rated
+  # endurance with a far higher write budget, so wear-out is no longer the
+  # binding constraint and the correlation argument no longer applies.
   #
-  # min_size 1 is REQUIRED and deliberate. Ceph does not lower min_size when
-  # size is reduced — it was 2 here, and leaving it would make the pool 2/2,
-  # where any single OSD down (every rolling node reboot, every OSD restart)
+  # min_size is left at the size-3 default of 2. The explicit min_size 1
+  # override existed only because Ceph does not lower min_size when size is
+  # reduced, which would have left the pool 2/2 — where any single OSD down
   # blocks IO for its PGs and hangs every RBD volume in the cluster.
-  #
-  # The trade this accepts: while the pool is degraded (one OSD down), a second
-  # drive failing loses data, and with all three drives at 80%+ endurance that
-  # window is not hypothetical. Mitigations that already exist: CNPG barman and
-  # Velero both ship offsite to Hetzner (opentofu/object_storage.tf), so the
-  # exposure is the RPO of those backups, not total loss. Keep node reboots
-  # serialised and wait for HEALTH_OK between them.
   replicated:
-    size: 2
-  parameters:
-    min_size: "1"
+    size: 3
 ---
 apiVersion: storage.k8s.io/v1
 kind: StorageClass
@@ -87,39 +80,26 @@ metadata:
   name: cephfs
   namespace: rook-ceph
 spec:
-  # TEMPORARY — RESTORE TO size 3 / no min_size OVERRIDE ONCE SATURN AND
-  # NEPTUNE HAVE BOTH BEEN CONVERTED TO BARE METAL. It stays at 2/1 across all
-  # three conversions rather than being toggled per node: each change
-  # re-replicates the pool, and the window reopens with the next node anyway.
+  # size 3 with no min_size override. This pool holds ~253 MiB against the data
+  # pool's 280 GiB, and losing it would destroy the whole filesystem rather
+  # than just the objects on the lost OSD, so it is the last pool that should
+  # ever run thin.
   #
-  # The standing trade for this pool is size 3: it holds ~253 MiB against the
-  # data pool's 280 GiB, so it contributes nothing measurable to drive wear —
-  # the entire reason data0 dropped to 2 — while losing it would destroy the
-  # whole filesystem, not just the objects on the lost OSD. Paying 3x on a
-  # quarter of a gigabyte to keep that intact is the right trade in steady
-  # state.
-  #
-  # It does not hold while one of three hosts is out. size 3 on two remaining
-  # hosts leaves every PG undersized at two copies with min_size 2, so a single
-  # further OSD loss drops the pool below min_size and blocks IO on every
-  # CephFS mount cluster-wide. At 2/1 the same two copies are active+clean
-  # instead, and a further loss degrades rather than blocks — availability
-  # bought at the cost of accepting writes onto a single copy while degraded.
-  #
-  # See RUNBOOK-baremetal-conversion.md.
+  # It drops to 2/1 whenever one of the three hosts is out for longer than a
+  # reboot. size 3 on two remaining hosts leaves every PG undersized at two
+  # copies with min_size 2, so a single further OSD loss falls below min_size
+  # and blocks IO on every CephFS mount cluster-wide. At 2/1 the same two
+  # copies are active+clean instead, and a further loss degrades rather than
+  # blocks. See RUNBOOK-baremetal-conversion.md.
   metadataPool:
     replicated:
-      size: 2
-    parameters:
-      min_size: "1"
+      size: 3
   dataPools:
     - name: data0
-      # size 2 / min_size 1 for the same reasons as replicapool above — see
-      # the comment there for the full rationale and the accepted trade.
+      # size 3 for the same reasons as replicapool above — see the comment
+      # there for why this pool sat at 2 while the Crucials backed the OSDs.
       replicated:
-        size: 2
-      parameters:
-        min_size: "1"
+        size: 3
   metadataServer:
     activeCount: 1
     activeStandby: true

kubernetes/talos/patches/k8s-2.yaml

@@ -1,7 +1,6 @@
 # k8s-2 runs Talos directly on saturn's MS-01 hardware. Structurally identical
 # to k8s-1.yaml — see that file for the reasoning behind each choice; only the
-# serials, MACs and addresses differ. k8s-3 is still a Proxmox VM and keeps the
-# VM-era layout until neptune is converted.
+# serials, MACs and addresses differ.
 machine:
   network:
     hostname: k8s-2

kubernetes/talos/patches/k8s-3.yaml

@@ -1,31 +1,82 @@
+# k8s-3 runs Talos directly on neptune's MS-01 hardware. Structurally identical
+# to k8s-1.yaml — see that file for the reasoning behind each choice; only the
+# serials and addresses differ. neptune was the last node converted, so all
+# three control-plane nodes now share this layout and no Proxmox remains.
 machine:
   network:
     hostname: k8s-3
-    # Shared control-plane VIP — see k8s-1.yaml for the busPath rationale.
-    interfaces:
-      - deviceSelector:
-          busPath: "0000:06:12.0"
-        dhcp: true
-        vip:
-          ip: 10.2.40.60
-      # v30iot: link only, no host addressing — see k8s-1.yaml.
-      - deviceSelector:
-          busPath: "0000:06:13.0"
-        dhcp: false
-      # See k8s-1.yaml for context on the dedicated Ceph cluster network.
-      - deviceSelector:
-          driver: i40e
-        addresses:
-          - 10.99.0.3/29
-        mtu: 9000
   sysctls:
     # v30iot NIC carries no host addressing — see k8s-1.yaml.
     net.ipv6.conf.ens19.disable_ipv6: "1"
   install:
-    disk: /dev/vda
+    # Three NVMe drives are visible on bare metal. Select the installer target
+    # by serial so it cannot land on the OSD drive or on EPHEMERAL, whatever
+    # order the kernel enumerates them in — Talos and Proxmox do not agree on
+    # that order.
+    diskSelector:
+      serial: 50026B7382B48B90
   certSANs:
     - k8s-3.srv.renner.uno
   nodeLabels:
     topology.kubernetes.io/zone: neptune
 cluster:
   allowSchedulingOnControlPlanes: true
+---
+# EPHEMERAL (/var) goes on the retired Crucial P3, not the system disk. That
+# drive is at 83% of its rated endurance and is finished as an OSD, but
+# everything EPHEMERAL holds is reconstructible, so it is consumable in this
+# role. Available spare is still 100% and it reports no media errors.
+#
+# ONLY TAKES EFFECT ON FIRST PROVISION. Talos applies volume configuration
+# solely when the volume does not yet exist; applied later it is silently
+# ignored and relocating needs a disk wipe and a boot from install media.
+#
+# Matched by serial deliberately: a broad selector such as
+# `disk.transport == 'nvme' && !system_disk` would match the PM983 as well.
+apiVersion: v1alpha1
+kind: VolumeConfig
+name: EPHEMERAL
+provisioning:
+  diskSelector:
+    match: disk.serial == '2326E6E8E67D'
+  minSize: 50GiB
+  grow: true
+---
+apiVersion: v1alpha1
+kind: LinkConfig
+name: enp2s0f0np0
+up: true
+---
+apiVersion: v1alpha1
+kind: VLANConfig
+name: mgmt0
+vlanID: 40
+parent: enp2s0f0np0
+up: true
+---
+apiVersion: v1alpha1
+kind: DHCPv4Config
+name: mgmt0
+---
+# Shared control-plane VIP, held by whichever node wins the etcd campaign.
+apiVersion: v1alpha1
+kind: Layer2VIPConfig
+name: 10.2.40.60
+link: mgmt0
+---
+# v30iot: link only, no host addressing — see k8s-1.yaml.
+apiVersion: v1alpha1
+kind: VLANConfig
+name: ens19
+vlanID: 30
+parent: enp2s0f0np0
+up: true
+---
+# Dedicated Ceph cluster network — see k8s-1.yaml.
+apiVersion: v1alpha1
+kind: LinkConfig
+name: enp2s0f1np1
+up: true
+mtu: 9000
+addresses:
+  - address: 10.99.0.3/29

opentofu/modules/proxmox_vm/variables.tf

@@ -47,8 +47,13 @@ variable "tags" {
 }
 
 variable "template_name" {
-  type    = string
-  default = "ubuntu-24-04-template"
+  type = string
+  # No default clone source. The ubuntu-24.04-template it used to name was
+  # deleted with saturn's conversion to bare metal, and Proxmox is being
+  # retired, so a default pointing at a template that does not exist would
+  # only fail at apply time. Set it explicitly if a clone source is ever
+  # wanted again.
+  default = null
 }
 
 variable "full_clone" {

opentofu/vm_605_template.tf

@@ -1,71 +0,0 @@
-resource "proxmox_virtual_environment_vm" "ubuntu_24_04_template" {
-  name      = "ubuntu-24.04-template"
-  node_name = "saturn"
-  vm_id     = 605
-
-  started = false
-  template = true
-  on_boot = false
-
-  keyboard_layout = "en-us"
-
-  bios = "seabios"
-
-  agent {
-    enabled = true
-  }
-
-  cpu {
-    cores = 2
-    type  = "x86-64-v3"
-  }
-
-  memory {
-    dedicated = 2048
-  }
-
-  initialization {
-    datastore_id = "local-zfs"
-
-    ip_config {
-      ipv4 {
-        address = "dhcp"
-      }
-    }
-
-    user_account {
-      username = "ladmin"
-      keys = [
-        "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPzFNBP3bq2m/6JqDqod9w+FP/uxXxHMolqhq3Ojflr1 krenner@fortress"
-      ]
-    }
-  }
-
-  disk {
-    datastore_id = "local-zfs"
-    interface    = "scsi0"
-    size         = 32
-    file_format  = "raw"
-    iothread     = false
-  }
-
-  network_device {
-    bridge = "v40srv"
-    model  = "virtio"
-  }
-
-  serial_device {}
-
-  scsi_hardware = "virtio-scsi-pci"
-
-  lifecycle {
-    ignore_changes = [
-      node_name,
-      started,
-      initialization,
-      disk,
-      cpu,
-      memory
-    ]
-  }
-}