feat(rook-ceph): restore replicapool and cephfs-data0 to size 3

These two sat at size 2 because of #434, not because of the conversion. With
three Crucial P3 QLC OSDs on three hosts, size 3 put every write on every
drive, so per-drive write volume equalled cluster write volume and all three
wore at an identical rate — 80/82/87% of rated endurance, projected to exhaust
together between Nov 2026 and Jan 2027. Replication is no defence against that,
because the wear was correlated by construction.

The OSDs are Samsung PM983 enterprise drives now, at 2-3% of rated endurance
with a far higher write budget. Wear-out is no longer the binding constraint
and the correlation argument no longer applies, so the pools return to the
durability the cluster is sized for.

The min_size 1 overrides go with them. They existed only because Ceph does not
lower min_size when size is reduced, which would have left the pools 2/2 and
blocked IO on any single OSD going down.

cephfs-metadata's comment loses the clause explaining that data0 sits at 2,
which stops being true here.

cephfs-ci stays at size 2: that pool holds throwaway CI workspaces and is at 2
rather than 1 only because an unreplicated pool makes `ceph osd ok-to-stop`
refuse for every OSD. Unrelated to #434.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kubernetes/infrastructure/rook-ceph-cluster/storageclass.yaml

@@ -5,31 +5,24 @@ metadata:
   namespace: rook-ceph
 spec:
   failureDomain: host
-  # size 2, NOT 3. With 3 OSDs on 3 hosts, size=3 meant every write landed on
+  # size 3. This pool sat at 2 while the OSDs were Crucial P3 QLC drives: with
-  # every drive in the cluster: per-drive write volume equalled total cluster
+  # 3 OSDs on 3 hosts, size=3 put every write on every drive, so per-drive write
-  # write volume, and all three drives wore at an identical rate and would have
+  # volume equalled total cluster write volume and all three wore at an
-  # reached end-of-rated-life together (measured 2026-08-19: 80/82/87% endurance
+  # identical rate (measured 2026-08-19: 80/82/87% endurance used, ~0.13
-  # used, ~0.13 points/day, all three exhausted between Nov 2026 and Jan 2027 —
+  # points/day, all three projected to exhaust between Nov 2026 and Jan 2027 —
-  # see #434). Replication does not protect against that failure mode, because
+  # see #434). Replication does not protect against that, because the wear was
-  # wear-out here is correlated by construction rather than independent.
+  # correlated by construction rather than independent.
   #
-  # size 2 cuts per-drive write volume by a third and desynchronises the wear.
+  # The OSDs are Samsung PM983 enterprise drives now, at 2-3% of rated
+  # endurance with a far higher write budget, so wear-out is no longer the
+  # binding constraint and the correlation argument no longer applies.
   #
-  # min_size 1 is REQUIRED and deliberate. Ceph does not lower min_size when
+  # min_size is left at the size-3 default of 2. The explicit min_size 1
-  # size is reduced — it was 2 here, and leaving it would make the pool 2/2,
+  # override existed only because Ceph does not lower min_size when size is
-  # where any single OSD down (every rolling node reboot, every OSD restart)
+  # reduced, which would have left the pool 2/2 — where any single OSD down
   # blocks IO for its PGs and hangs every RBD volume in the cluster.
-  #
-  # The trade this accepts: while the pool is degraded (one OSD down), a second
-  # drive failing loses data, and with all three drives at 80%+ endurance that
-  # window is not hypothetical. Mitigations that already exist: CNPG barman and
-  # Velero both ship offsite to Hetzner (opentofu/object_storage.tf), so the
-  # exposure is the RPO of those backups, not total loss. Keep node reboots
-  # serialised and wait for HEALTH_OK between them.
   replicated:
-    size: 2
+    size: 3
-  parameters:
-    min_size: "1"
 ---
 apiVersion: storage.k8s.io/v1
 kind: StorageClass
@@ -88,10 +81,9 @@ metadata:
   namespace: rook-ceph
 spec:
   # size 3 with no min_size override. This pool holds ~253 MiB against the data
-  # pool's 280 GiB, so it contributes nothing measurable to drive wear — the
+  # pool's 280 GiB, and losing it would destroy the whole filesystem rather
-  # entire reason data0 sits at 2 — while losing it would destroy the whole
+  # than just the objects on the lost OSD, so it is the last pool that should
-  # filesystem, not just the objects on the lost OSD. Paying 3x on a quarter of
+  # ever run thin.
-  # a gigabyte to keep that intact is the right trade.
   #
   # It drops to 2/1 whenever one of the three hosts is out for longer than a
   # reboot. size 3 on two remaining hosts leaves every PG undersized at two
@@ -104,12 +96,10 @@ spec:
       size: 3
   dataPools:
     - name: data0
-      # size 2 / min_size 1 for the same reasons as replicapool above — see
+      # size 3 for the same reasons as replicapool above — see the comment
-      # the comment there for the full rationale and the accepted trade.
+      # there for why this pool sat at 2 while the Crucials backed the OSDs.
       replicated:
-        size: 2
+        size: 3
-      parameters:
-        min_size: "1"
   metadataServer:
     activeCount: 1
     activeStandby: true