fix(ceph): replicapool and cephfs data to size 2, min_size 1

With 3 OSDs on 3 hosts, size=3 meant every write landed on every drive in the cluster. Per-drive write volume equalled TOTAL cluster write volume, so all three drives wore at an identical rate and would have reached end-of-rated-life together: measured 2026-08-19 from the drives' own SMART counters, 80/82/87% endurance used, advancing ~0.13 points/day, exhausted between Nov 2026 and Jan 2027 (#434). Replication gives no protection there — the wear-out is correlated by construction, not independent. size 2 cuts per-drive write volume by a third and desynchronises the wear. min_size 1 is required, not incidental. Ceph does NOT lower min_size when size is reduced; it was 2, and leaving it would make replicapool 2/2, where a single OSD down blocks IO for its PGs. That would hang every RBD volume in the cluster on every rolling node reboot and every OSD restart. Accepted trade, stated explicitly: while degraded (one OSD down), a second drive failure loses data, and at 80%+ endurance that window is not hypothetical. What limits the blast radius is that CNPG barman and Velero both ship offsite to Hetzner, so exposure is the RPO of those backups rather than total loss. Node reboots must stay serialised with HEALTH_OK between them. CephFS metadataPool deliberately stays at size 3: 467 MiB against data0's 275 GiB, so it contributes nothing measurable to wear, while losing it destroys the entire filesystem rather than the objects on one OSD. The wear saving comes from data0. The RGW pools in objectstore.yaml were already size 2. One-time cost on apply: Ceph deletes the third replica of everything, which is background IO, and raw used drops from ~1.4 TiB to ~0.95 TiB. Refs #434 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Merge pull request 'fix(ceph): stop deferring small writes on SSD (halves write amplification)' (#518 ) from fix-ceph-deferred-write-amplification into master
2026-08-19 14:35:58 +02:00 · 2026-08-19 12:30:21 +00:00
1 changed files with 35 additions and 2 deletions
@@ -5,8 +5,31 @@ metadata:
  namespace: rook-ceph
 spec:
  failureDomain: host
+  # size 2, NOT 3. With 3 OSDs on 3 hosts, size=3 meant every write landed on
+  # every drive in the cluster: per-drive write volume equalled total cluster
+  # write volume, and all three drives wore at an identical rate and would have
+  # reached end-of-rated-life together (measured 2026-08-19: 80/82/87% endurance
+  # used, ~0.13 points/day, all three exhausted between Nov 2026 and Jan 2027 —
+  # see #434). Replication does not protect against that failure mode, because
+  # wear-out here is correlated by construction rather than independent.
+  #
+  # size 2 cuts per-drive write volume by a third and desynchronises the wear.
+  #
+  # min_size 1 is REQUIRED and deliberate. Ceph does not lower min_size when
+  # size is reduced — it was 2 here, and leaving it would make the pool 2/2,
+  # where any single OSD down (every rolling node reboot, every OSD restart)
+  # blocks IO for its PGs and hangs every RBD volume in the cluster.
+  #
+  # The trade this accepts: while the pool is degraded (one OSD down), a second
+  # drive failing loses data, and with all three drives at 80%+ endurance that
+  # window is not hypothetical. Mitigations that already exist: CNPG barman and
+  # Velero both ship offsite to Hetzner (opentofu/object_storage.tf), so the
+  # exposure is the RPO of those backups, not total loss. Keep node reboots
+  # serialised and wait for HEALTH_OK between them.
  replicated:
-    size: 3
+    size: 2
+  parameters:
+    min_size: "1"
 ---
 apiVersion: storage.k8s.io/v1
 kind: StorageClass
@@ -64,13 +87,23 @@ metadata:
  name: cephfs
  namespace: rook-ceph
 spec:
+  # metadataPool STAYS at size 3 while the data pool drops to 2. It holds
+  # 467 MiB against the data pool's 275 GiB, so it contributes nothing
+  # measurable to drive wear — the entire reason for the size reduction — while
+  # losing it would destroy the whole filesystem, not just the objects on the
+  # lost OSD. Paying 3x on half a gigabyte to keep that intact is the right
+  # trade; the wear saving comes from data0.
  metadataPool:
    replicated:
      size: 3
  dataPools:
    - name: data0
+      # size 2 / min_size 1 for the same reasons as replicapool above — see
+      # the comment there for the full rationale and the accepted trade.
      replicated:
-        size: 3
+        size: 2
+      parameters:
+        min_size: "1"
  metadataServer:
    activeCount: 1
    activeStandby: true