feat(ceph): deliberate cutover to Ceph 20.2.1 (Tentacle)

The 19.x pin existed to keep this cutover deliberate and tied to the
Rook operator version. That coupling has already come apart in the
dangerous direction: Rook floating constraint pulled v1.20 in June, and
with it cephcsi v3.17.1, whose client libraries are Ceph 20.2.1 — so
every CSI pod has been talking to 19.2.6 mons with a client a whole
major ahead of them.

That is not academic. The 19.2.6 mons mint auth keys as aes256k, and
the 20.2.1 client cannot decode them ("auth: failed to decode key").
Nothing has broken only because the four CSI users predate the cipher
change and still carry legacy aes keys — which Ceph itself flags daily
as HEALTH_WARN (AUTH_INSECURE_CLIENT_KEY_TYPE, 5 entities). The moment
one of those is recreated — ceph auth rotate, a deleted Secret, Rook
resyncing — the provisioner stops and no PVC binds. A cold rebuild hits
it immediately: a fresh cluster has no legacy keys at all.

So the cutover happens now, with the version the client already is.
Rook v1.20 supports Tentacle from v20.2.1; v20.2.0 is excluded
deliberately — upstream reports data corruption there with read
affinity enabled. The two imagepolicy-marked lines are bumped in the
same commit so image automation finds nothing left to do.

Rollout is Rook-managed and rolling: osdMaxUpdatesInParallel: 1 and
upgradeOSDRequiresHealthyPGs: true bound it to one OSD at a time with
clean PGs between steps. Expect mons, then mgrs, then three sequential
OSD restarts with recovery between them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kubernetes/bootstrap.sh

@@ -159,8 +159,16 @@ if [[ "$PHASE" == "vmtest" ]]; then
 
     # sun_path caps a unix socket path at 108 bytes and the provisioner puts a
     # qemu monitor socket under the state directory, so this has to stay short.
-    # A path under the repo or a long mktemp dir silently fails to boot.
-    VM_STATE=/tmp/bootstrap-vmtest
+    # A path under the repo or a long mktemp dir silently fails to boot. The
+    # caller may override it to aim the qcow2 images at a sized volume; the
+    # guard turns "too long" from a silent boot failure into an error.
+    VM_STATE="${VM_STATE:-/tmp/bootstrap-vmtest}"
+    [[ ${#VM_STATE} -le 50 ]] \
+        || fail "VM_STATE '$VM_STATE' is ${#VM_STATE} bytes; keep it under 50 so the qemu monitor socket fits in sun_path's 108"
+    # cleanup_vmtest runs `sudo rm -rf "$VM_STATE"`, so refuse anything that is
+    # not a nested absolute path.
+    [[ $VM_STATE == /*/?* ]] \
+        || fail "VM_STATE '$VM_STATE' must be an absolute path at least two levels deep"
     VM_CLUSTER=bootstrap-vmtest
     VM_DIR=$(mktemp -d)
     TALOSCTL_BIN=$(command -v talosctl)

kubernetes/infrastructure/image-automation/ceph.yaml

@@ -9,9 +9,26 @@ spec:
 ---
 # v-prefix tags (v19.2.3). Upstream also publishes date-suffixed builds
 # (v19.2.3-20250717) and moving major/minor tags (v19, v19.2); the
-# filter pins us to the bare three-component release. Range stays at
-# 19.x — the 19 → 20 (Tentacle) cutover should be a deliberate
-# upgrade tied to the Rook operator version.
+# filter pins us to the bare three-component release.
+#
+# The range was 19.x, to keep the 19 -> 20 (Tentacle) cutover a deliberate
+# upgrade tied to the Rook operator version. That coupling had already come
+# apart without anyone noticing: Rook's floating constraint pulled v1.20 in
+# June, and with it a ceph-csi image whose client libraries are Ceph 20.2.1.
+# The cluster stayed on 19.2.6, so every CSI pod has since been talking to
+# the mons with a client a whole major ahead of them.
+#
+# That is not academic. The 19.2.6 mons mint auth keys as aes256k, and the
+# 20.2.1 client cannot decode them — "auth: failed to decode key 'AgB...'".
+# Nothing broke only because the four CSI users predate the cipher change and
+# still carry legacy aes keys, which Ceph itself flags as insecure. The moment
+# one is recreated — ceph auth rotate, a deleted Secret, Rook resyncing — the
+# provisioner stops and no PVC binds. A rebuild rehearsal hits it immediately,
+# because a fresh cluster has no legacy keys at all.
+#
+# So the cutover happens now, with the version the client already is. Rook
+# v1.20 supports Tentacle from v20.2.1; v20.2.0 is excluded deliberately —
+# upstream reports data corruption there when read affinity is enabled.
 apiVersion: image.toolkit.fluxcd.io/v1
 kind: ImagePolicy
 metadata:
@@ -25,4 +42,4 @@ spec:
     extract: '$ver'
   policy:
     semver:
-      range: "19.x"
+      range: ">=20.2.1 <21.0.0"

kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml

@@ -5,7 +5,7 @@ metadata:
   namespace: rook-ceph
 spec:
   cephVersion:
-    image: quay.io/ceph/ceph:v19.2.6 # {"$imagepolicy": "flux-system:ceph"}
+    image: quay.io/ceph/ceph:v20.2.1 # {"$imagepolicy": "flux-system:ceph"}
     allowUnsupported: false
   dataDirHostPath: /var/lib/rook
   # Defaults to false. With three OSDs and size 3 there is no spare copy, so

kubernetes/infrastructure/rook-ceph-cluster/toolbox.yaml

@@ -38,7 +38,7 @@ spec:
       serviceAccountName: rook-ceph-default
       containers:
         - name: rook-ceph-tools
-          image: quay.io/ceph/ceph:v19.2.6 # {"$imagepolicy": "flux-system:ceph"}
+          image: quay.io/ceph/ceph:v20.2.1 # {"$imagepolicy": "flux-system:ceph"}
           command:
             - /bin/bash
             - -c