feat(rook-ceph): restore replicapool and cephfs-data0 to size 3

These two sat at size 2 because of #434, not because of the conversion. With
three Crucial P3 QLC OSDs on three hosts, size 3 put every write on every
drive, so per-drive write volume equalled cluster write volume and all three
wore at an identical rate — 80/82/87% of rated endurance, projected to exhaust
together between Nov 2026 and Jan 2027. Replication is no defence against that,
because the wear was correlated by construction.

The OSDs are Samsung PM983 enterprise drives now, at 2-3% of rated endurance
with a far higher write budget. Wear-out is no longer the binding constraint
and the correlation argument no longer applies, so the pools return to the
durability the cluster is sized for.

The min_size 1 overrides go with them. They existed only because Ceph does not
lower min_size when size is reduced, which would have left the pools 2/2 and
blocked IO on any single OSD going down.

cephfs-metadata's comment loses the clause explaining that data0 sits at 2,
which stops being true here.

cephfs-ci stays at size 2: that pool holds throwaway CI workspaces and is at 2
rather than 1 only because an unreplicated pool makes `ceph osd ok-to-stop`
refuse for every OSD. Unrelated to #434.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kubernetes/infrastructure/rook-ceph-cluster/storageclass.yaml

@@ -5,31 +5,24 @@ metadata:
   namespace: rook-ceph
 spec:
   failureDomain: host
-  # size 2, NOT 3. With 3 OSDs on 3 hosts, size=3 meant every write landed on
-  # every drive in the cluster: per-drive write volume equalled total cluster
-  # write volume, and all three drives wore at an identical rate and would have
-  # reached end-of-rated-life together (measured 2026-08-19: 80/82/87% endurance
-  # used, ~0.13 points/day, all three exhausted between Nov 2026 and Jan 2027 —
-  # see #434). Replication does not protect against that failure mode, because
-  # wear-out here is correlated by construction rather than independent.
+  # size 3. This pool sat at 2 while the OSDs were Crucial P3 QLC drives: with
+  # 3 OSDs on 3 hosts, size=3 put every write on every drive, so per-drive write
+  # volume equalled total cluster write volume and all three wore at an
+  # identical rate (measured 2026-08-19: 80/82/87% endurance used, ~0.13
+  # points/day, all three projected to exhaust between Nov 2026 and Jan 2027 —
+  # see #434). Replication does not protect against that, because the wear was
+  # correlated by construction rather than independent.
   #
-  # size 2 cuts per-drive write volume by a third and desynchronises the wear.
+  # The OSDs are Samsung PM983 enterprise drives now, at 2-3% of rated
+  # endurance with a far higher write budget, so wear-out is no longer the
+  # binding constraint and the correlation argument no longer applies.
   #
-  # min_size 1 is REQUIRED and deliberate. Ceph does not lower min_size when
-  # size is reduced — it was 2 here, and leaving it would make the pool 2/2,
-  # where any single OSD down (every rolling node reboot, every OSD restart)
+  # min_size is left at the size-3 default of 2. The explicit min_size 1
+  # override existed only because Ceph does not lower min_size when size is
+  # reduced, which would have left the pool 2/2 — where any single OSD down
   # blocks IO for its PGs and hangs every RBD volume in the cluster.
-  #
-  # The trade this accepts: while the pool is degraded (one OSD down), a second
-  # drive failing loses data, and with all three drives at 80%+ endurance that
-  # window is not hypothetical. Mitigations that already exist: CNPG barman and
-  # Velero both ship offsite to Hetzner (opentofu/object_storage.tf), so the
-  # exposure is the RPO of those backups, not total loss. Keep node reboots
-  # serialised and wait for HEALTH_OK between them.
   replicated:
-    size: 2
-  parameters:
-    min_size: "1"
+    size: 3
 ---
 apiVersion: storage.k8s.io/v1
 kind: StorageClass
@@ -87,39 +80,26 @@ metadata:
   name: cephfs
   namespace: rook-ceph
 spec:
-  # TEMPORARY — RESTORE TO size 3 / no min_size OVERRIDE ONCE SATURN AND
-  # NEPTUNE HAVE BOTH BEEN CONVERTED TO BARE METAL. It stays at 2/1 across all
-  # three conversions rather than being toggled per node: each change
-  # re-replicates the pool, and the window reopens with the next node anyway.
+  # size 3 with no min_size override. This pool holds ~253 MiB against the data
+  # pool's 280 GiB, and losing it would destroy the whole filesystem rather
+  # than just the objects on the lost OSD, so it is the last pool that should
+  # ever run thin.
   #
-  # The standing trade for this pool is size 3: it holds ~253 MiB against the
-  # data pool's 280 GiB, so it contributes nothing measurable to drive wear —
-  # the entire reason data0 dropped to 2 — while losing it would destroy the
-  # whole filesystem, not just the objects on the lost OSD. Paying 3x on a
-  # quarter of a gigabyte to keep that intact is the right trade in steady
-  # state.
-  #
-  # It does not hold while one of three hosts is out. size 3 on two remaining
-  # hosts leaves every PG undersized at two copies with min_size 2, so a single
-  # further OSD loss drops the pool below min_size and blocks IO on every
-  # CephFS mount cluster-wide. At 2/1 the same two copies are active+clean
-  # instead, and a further loss degrades rather than blocks — availability
-  # bought at the cost of accepting writes onto a single copy while degraded.
-  #
-  # See RUNBOOK-baremetal-conversion.md.
+  # It drops to 2/1 whenever one of the three hosts is out for longer than a
+  # reboot. size 3 on two remaining hosts leaves every PG undersized at two
+  # copies with min_size 2, so a single further OSD loss falls below min_size
+  # and blocks IO on every CephFS mount cluster-wide. At 2/1 the same two
+  # copies are active+clean instead, and a further loss degrades rather than
+  # blocks. See RUNBOOK-baremetal-conversion.md.
   metadataPool:
     replicated:
-      size: 2
-    parameters:
-      min_size: "1"
+      size: 3
   dataPools:
     - name: data0
-      # size 2 / min_size 1 for the same reasons as replicapool above — see
-      # the comment there for the full rationale and the accepted trade.
+      # size 3 for the same reasons as replicapool above — see the comment
+      # there for why this pool sat at 2 while the Crucials backed the OSDs.
       replicated:
-        size: 2
-      parameters:
-        min_size: "1"
+        size: 3
   metadataServer:
     activeCount: 1
     activeStandby: true

kubernetes/talos/patches/k8s-2.yaml

@@ -1,7 +1,6 @@
 # k8s-2 runs Talos directly on saturn's MS-01 hardware. Structurally identical
 # to k8s-1.yaml — see that file for the reasoning behind each choice; only the
-# serials, MACs and addresses differ. k8s-3 is still a Proxmox VM and keeps the
-# VM-era layout until neptune is converted.
+# serials, MACs and addresses differ.
 machine:
   network:
     hostname: k8s-2

kubernetes/talos/patches/k8s-3.yaml

@@ -1,31 +1,82 @@
+# k8s-3 runs Talos directly on neptune's MS-01 hardware. Structurally identical
+# to k8s-1.yaml — see that file for the reasoning behind each choice; only the
+# serials and addresses differ. neptune was the last node converted, so all
+# three control-plane nodes now share this layout and no Proxmox remains.
 machine:
   network:
     hostname: k8s-3
-    # Shared control-plane VIP — see k8s-1.yaml for the busPath rationale.
-    interfaces:
-      - deviceSelector:
-          busPath: "0000:06:12.0"
-        dhcp: true
-        vip:
-          ip: 10.2.40.60
-      # v30iot: link only, no host addressing — see k8s-1.yaml.
-      - deviceSelector:
-          busPath: "0000:06:13.0"
-        dhcp: false
-      # See k8s-1.yaml for context on the dedicated Ceph cluster network.
-      - deviceSelector:
-          driver: i40e
-        addresses:
-          - 10.99.0.3/29
-        mtu: 9000
   sysctls:
     # v30iot NIC carries no host addressing — see k8s-1.yaml.
     net.ipv6.conf.ens19.disable_ipv6: "1"
   install:
-    disk: /dev/vda
+    # Three NVMe drives are visible on bare metal. Select the installer target
+    # by serial so it cannot land on the OSD drive or on EPHEMERAL, whatever
+    # order the kernel enumerates them in — Talos and Proxmox do not agree on
+    # that order.
+    diskSelector:
+      serial: 50026B7382B48B90
   certSANs:
     - k8s-3.srv.renner.uno
   nodeLabels:
     topology.kubernetes.io/zone: neptune
 cluster:
   allowSchedulingOnControlPlanes: true
+---
+# EPHEMERAL (/var) goes on the retired Crucial P3, not the system disk. That
+# drive is at 83% of its rated endurance and is finished as an OSD, but
+# everything EPHEMERAL holds is reconstructible, so it is consumable in this
+# role. Available spare is still 100% and it reports no media errors.
+#
+# ONLY TAKES EFFECT ON FIRST PROVISION. Talos applies volume configuration
+# solely when the volume does not yet exist; applied later it is silently
+# ignored and relocating needs a disk wipe and a boot from install media.
+#
+# Matched by serial deliberately: a broad selector such as
+# `disk.transport == 'nvme' && !system_disk` would match the PM983 as well.
+apiVersion: v1alpha1
+kind: VolumeConfig
+name: EPHEMERAL
+provisioning:
+  diskSelector:
+    match: disk.serial == '2326E6E8E67D'
+  minSize: 50GiB
+  grow: true
+---
+apiVersion: v1alpha1
+kind: LinkConfig
+name: enp2s0f0np0
+up: true
+---
+apiVersion: v1alpha1
+kind: VLANConfig
+name: mgmt0
+vlanID: 40
+parent: enp2s0f0np0
+up: true
+---
+apiVersion: v1alpha1
+kind: DHCPv4Config
+name: mgmt0
+---
+# Shared control-plane VIP, held by whichever node wins the etcd campaign.
+apiVersion: v1alpha1
+kind: Layer2VIPConfig
+name: 10.2.40.60
+link: mgmt0
+---
+# v30iot: link only, no host addressing — see k8s-1.yaml.
+apiVersion: v1alpha1
+kind: VLANConfig
+name: ens19
+vlanID: 30
+parent: enp2s0f0np0
+up: true
+---
+# Dedicated Ceph cluster network — see k8s-1.yaml.
+apiVersion: v1alpha1
+kind: LinkConfig
+name: enp2s0f1np1
+up: true
+mtu: 9000
+addresses:
+  - address: 10.99.0.3/29