fix(rook-ceph): rotate CephX daemon keys to close CVE-2025-30156

Ceph v19.2.6 ships the fix for CVE-2025-30156, a CephX authentication
bypass: AES-128-CBC service tickets carry no HMAC and use a hard-coded
IV, so a holder of any low-privilege key can bit-flip a ticket into
cluster-admin in linear time. Upgrading the binaries does not resolve
it — the keys themselves must be rotated to the aes256k cipher.

The cluster is mid-procedure. auth_allowed_ciphers (aes, aes256k),
auth_preferred_cipher (aes256k) and auth_service_cipher (aes256k) are
already set, and the rotating service keys were wiped, but no actual
key rotation has happened: every entity in status.cephx is still
keyGeneration 1 with keyType aes, created under 19.2.3.

Rook automates daemon-key rotation but it is opt-in and defaults to
Disabled, so spec.security.cephx was left empty and nothing rotated.
Rook v1.20.6 and Ceph v19.2.6 are both past the version floor the
feature requires.

Rotating daemon keys clears AUTH_INSECURE_SERVICE_KEY_TYPE, the only
one of the six new checks at ERR severity. That is the operationally
urgent part: Rook skips CephFS and RGW reconciliation entirely while
health is HEALTH_ERR — which is what deadlocked the mds/rgw rollout in
issue #527 — and the check is presently held down only by a mon-side
mute with a finite TTL. When that mute lapses the cluster returns to
HEALTH_ERR and the reconcile freeze comes back.

CSI keys are deliberately left on aes. aes256k over the CSI mount path
requires a host kernel of 7.0 or newer and these nodes run 6.18, so
`aes` must stay in auth_allowed_ciphers for CSI to authenticate at all.
AUTH_INSECURE_CLIENT_KEY_TYPE, AUTH_INSECURE_KEYS_ALLOWED and
AUTH_INSECURE_KEYS_CREATABLE therefore stay raised at WARN until the
nodes run a kernel that supports it.

Rotation restarts most Ceph daemons. With size 2 pools across 3 OSDs
this should be applied deliberately rather than alongside other churn,
and Ceph should reach HEALTH_OK before anything else touches the
cluster.

Refs #527, #528

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kubernetes/apps/immich/redis.yaml

@@ -27,7 +27,7 @@ spec:
         # Pinned to 7.4-alpine to match the LXC 225 compose exactly.
         # Any redis major-version bump is a separate migration.
         - name: redis
-          image: redis:7.4.10-alpine # {"$imagepolicy": "flux-system:redis-immich"}
+          image: redis:7.4.11-alpine # {"$imagepolicy": "flux-system:redis-immich"}
           args:
             - redis-server
             - --save

kubernetes/apps/monitoring-rules/prometheusrule-ceph.yaml

@@ -12,18 +12,65 @@ spec:
     - name: ceph.rules
       rules:
         - alert: CephHealthWarn
-          # ceph_health_status: 0 = HEALTH_OK, 1 = HEALTH_WARN, 2 = HEALTH_ERR.
-          # 6h is deliberately long. Ceph warns for self-clearing conditions —
-          # a slow BlueStore op during the Sunday restore sweep, a mon briefly
-          # below its disk threshold — and paging on those trains people to
-          # ignore the alert. What matters is a warning that does not clear.
-          expr: ceph_health_status == 1
+          # Was `ceph_health_status == 1`, which had been FIRING CONTINUOUSLY
+          # since at least 2026-08-21T00:26Z and was therefore telling us
+          # nothing: four AUTH_INSECURE_* checks are permanently active
+          # (CVE-2025-30156, issue #527) and cannot clear until Rook >=1.20.5.
+          # A permanently-firing alert masks every subsequent Ceph warning,
+          # which is the opposite of what this rule is for.
+          #
+          # ceph_health_detail exposes each check individually, so alert on any
+          # check that is NOT a known-accepted one. Verified against live data:
+          # 5 checks active, 0 unexpected — so this is quiet now and fires the
+          # moment a genuinely new condition appears.
+          #
+          # Accepted, each for a documented reason:
+          #   AUTH_INSECURE_*       CVE-2025-30156, tracked in #527
+          #   POOL_NO_REDUNDANCY    cephfs-ci is size 1 ON PURPOSE (CI
+          #                         workspaces are throwaway). Non-CI pools
+          #                         losing redundancy is caught by
+          #                         CephPoolLostRedundancy below, since this
+          #                         check does not say WHICH pool.
+          #
+          # 6h is kept: Ceph warns for self-clearing conditions — a slow
+          # BlueStore op during the Sunday restore sweep, a mon briefly below
+          # its disk threshold — and paging on those trains people to ignore
+          # the alert. What matters is a warning that does not clear.
+          expr: ceph_health_detail{name!~"AUTH_INSECURE_.+|POOL_NO_REDUNDANCY"} == 1
           for: 6h
           labels:
             severity: warning
           annotations:
-            summary: "Ceph has been HEALTH_WARN for 6h"
-            description: "The Ceph cluster has held HEALTH_WARN for six hours, so it is not a self-clearing blip. Get the detail with `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail`."
+            summary: "Ceph health check {{ $labels.name }} active for 6h"
+            description: "Ceph health check {{ $labels.name }} has been active for six hours, so it is not a self-clearing blip. Known-accepted checks (AUTH_INSECURE_*, POOL_NO_REDUNDANCY) are excluded from this alert, so this is a new condition. Detail: `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail`."
+
+        - alert: CephPoolLostRedundancy
+          # Restores what excluding POOL_NO_REDUNDANCY from CephHealthWarn
+          # gives up. That check fires cluster-wide without naming a pool, so
+          # accepting it for cephfs-ci would also hide a PRODUCTION pool
+          # dropping to one replica.
+          #
+          # There is no pool-size metric, but bytes_used/stored IS the
+          # replication factor. Verified live: replicapool 2, cephfs-data0 2,
+          # cephfs-metadata 3, cephfs-ci-metadata 1.
+          #
+          # The >1e6 floor excludes trivially small pools. It is not needed to
+          # suppress the near-empty rgw pools — their ratios are 9-36 because
+          # per-object overhead dominates, so they never match `< 1.5` anyway.
+          #
+          # Both directions were tested against live data before committing:
+          # including cephfs-ci the expression matches (pool 13), excluding it
+          # the expression is empty. An alert that cannot fire is worse than no
+          # alert, so the positive control matters as much as the negative.
+          expr: |
+            (ceph_pool_bytes_used / ceph_pool_stored < 1.5 and ceph_pool_stored > 1e6)
+              and on(pool_id) ceph_pool_metadata{name!~"cephfs-ci-.+"}
+          for: 30m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Ceph pool {{ $labels.pool_id }} is running unreplicated"
+            description: "Pool {{ $labels.pool_id }} has a replication factor below 1.5, meaning a single OSD loss destroys data. Only the cephfs-ci pools are meant to be unreplicated. Check with `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd pool ls detail`."
 
         - alert: CephHealthError
           # HEALTH_ERR means data availability or redundancy is at risk.

kubernetes/apps/paperless/redis.yaml

@@ -25,7 +25,7 @@ spec:
           type: RuntimeDefault
       containers:
         - name: redis
-          image: redis:8.10.0-alpine # {"$imagepolicy": "flux-system:redis-paperless"}
+          image: redis:8.10.1-alpine # {"$imagepolicy": "flux-system:redis-paperless"}
           args:
             - redis-server
             - --save

kubernetes/apps/woodpecker/agent.yaml

@@ -50,12 +50,39 @@ spec:
             # clone + subsequent steps share the workspace PVC and the k8s backend
             # can schedule them on different nodes -> an RWO volume then fails to
             # mount on the second node. CephFS (RWX) lets the workspace follow
-            # the pipeline across nodes. The -ephemeral variant (reclaim Delete,
-            # rook-ceph-cluster/storageclass.yaml) is REQUIRED here: workspaces
-            # are throwaway, and the Retain class leaked one Released PV per
-            # finished pipeline (issue #262 follow-up).
+            # the pipeline across nodes. Reclaim MUST be Delete: workspaces are
+            # throwaway, and a Retain class leaked one Released PV per finished
+            # pipeline (issue #262 follow-up).
+            #
+            # ceph-filesystem-ci, not ceph-filesystem-ephemeral: a dedicated
+            # CephFS whose pools are BOTH size 1 (infrastructure/
+            # rook-ceph-cluster/cephfs-ci.yaml). The shared filesystem
+            # replicates data0 2x and metadata 3x, which is the correct trade
+            # for real data and pure waste for a git clone that is deleted
+            # minutes later.
+            #
+            # Measured 2026-08-21, polling both pools every 15s (a 5-minute
+            # interval misses an entire pipeline -- PVC created, used and
+            # deleted inside ~2 minutes):
+            #   CI active  1.5min: data0 143.2 MB, metadata 17.2 MB
+            #   CI idle   16.0min: data0   0.0 MB, metadata 11.0 MB
+            # data0 write volume is 100% CI. Per pipeline this moves the drive
+            # cost from 71.6x2 + 8.2x3 = ~168 MB to ~80 MB.
+            #
+            # ceph-filesystem-ephemeral is NOT repointed, and the reason is a
+            # technical constraint rather than a safety one. Its other consumer
+            # is velero/restore-test.sh, whose probe PVCs are snapshot CLONES:
+            # `ceph fs subvolume snapshot clone` operates within one filesystem,
+            # so a snapshot taken in `cephfs` cannot be cloned into `cephfs-ci`.
+            #
+            # Those probes are just as throwaway as a CI workspace -- the
+            # authoritative copy is the backup on Hetzner, and a lost probe just
+            # means re-running the test -- so they are equally undeserving of
+            # replication. Landing them unreplicated would mean a second size-1
+            # data pool inside `cephfs`, and restore-test runs WEEKLY against
+            # CI's ~96 pipelines/day, so it is not worth the complexity.
             - name: WOODPECKER_BACKEND_K8S_STORAGE_CLASS
-              value: ceph-filesystem-ephemeral
+              value: ceph-filesystem-ci
             - name: WOODPECKER_BACKEND_K8S_STORAGE_RWX
               value: "true"
             # MUST be a JSON object literal — the backend json.Unmarshal's this

kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml

@@ -17,6 +17,33 @@ spec:
     modules:
       - name: pg_autoscaler
         enabled: true
+  # CVE-2025-30156: CephX AES-128-CBC service tickets carry no HMAC and use a
+  # hard-coded IV, so a holder of any low-privilege key can bit-flip a ticket
+  # into cluster-admin. The fix is rotating keys to the aes256k cipher —
+  # upgrading the binaries alone does not close it. Rotation is opt-in and
+  # defaults to Disabled, so it must be requested here.
+  #
+  # Daemon keys only. CSI keys stay on aes: aes256k over the CSI mount path
+  # needs a host kernel of 7.0 or newer and these nodes run 6.18. `aes`
+  # therefore has to remain in auth_allowed_ciphers for CSI to authenticate,
+  # which leaves AUTH_INSECURE_CLIENT_KEY_TYPE, AUTH_INSECURE_KEYS_ALLOWED and
+  # AUTH_INSECURE_KEYS_CREATABLE raised. Revisit once the nodes run a kernel
+  # that supports it.
+  #
+  # Rotating the daemon keys clears AUTH_INSECURE_SERVICE_KEY_TYPE, the only
+  # one of the six checks at ERR severity. That matters beyond the CVE itself:
+  # Rook skips CephFS and RGW reconciliation entirely while health is
+  # HEALTH_ERR, and that check is currently held down only by a mon-side mute
+  # with a finite TTL.
+  #
+  # keyType is deliberately unset — Rook picks the best type for daemon keys.
+  # To rotate again later, increment keyGeneration; Rook restarts the daemons
+  # to pick up the new keys.
+  security:
+    cephx:
+      daemon:
+        keyRotationPolicy: KeyGeneration
+        keyGeneration: 2
   # Bump CephFS subvolume cloner pool from 4 → 16. Velero CSI Snapshot
   # Data Movement provisions a backupPVC per source PVC by cloning the
   # VolumeSnapshot, and a 4-thread default stalls migrate-immich's

kubernetes/infrastructure/rook-ceph-cluster/cephfs-ci.yaml

@@ -0,0 +1,86 @@
+---
+# A SEPARATE CephFS whose only purpose is Woodpecker CI workspaces, with both
+# pools at size 1.
+#
+# WHY THIS EXISTS
+# Woodpecker's Kubernetes backend creates a PVC per pipeline, fills it with a
+# checkout plus build output, and deletes it when the pipeline ends. Measured
+# 2026-08-21 by polling both pools at 15s (a 5-minute interval misses a whole
+# pipeline — its PVC is created, used and deleted inside ~2 minutes):
+#
+#   one pipeline, a single 15s interval:  data0 +71.6 MB   metadata +8.2 MB
+#   every interval without a CI pod:      data0   ~0        metadata   ~0
+#
+# CI drove essentially all of it. On the shared filesystem that costs
+# 71.6x2 + 8.2x3 = ~168 MB of drive writes per pipeline. Here it costs ~80 MB.
+#
+# WHY size 1 IS CORRECT HERE, HAVING BEEN REJECTED FOR THE MAIN FILESYSTEM
+# `cephfs` keeps metadataPool at size 3 because losing it destroys the whole
+# filesystem, not the objects on one OSD, and 550 GiB in cephfs-data0 depends
+# on it. None of that applies to this filesystem. Every byte in it is a fresh
+# git clone; the worst case of total loss is that in-flight pipelines fail and
+# get re-run. Replicating throwaway data is the thing being paid for, so it is
+# the thing removed.
+#
+# Losing an OSD takes these PGs offline and kills running pipelines. That is
+# not a regression: CI already breaks during Ceph maintenance.
+apiVersion: ceph.rook.io/v1
+kind: CephFilesystem
+metadata:
+  name: cephfs-ci
+  namespace: rook-ceph
+spec:
+  metadataPool:
+    replicated:
+      size: 1
+      # Rook refuses size 1 without this — it is asserting that an unreplicated
+      # pool is deliberate rather than a typo. It is.
+      requireSafeReplicaSize: false
+  dataPools:
+    - name: data0
+      replicated:
+        size: 1
+        requireSafeReplicaSize: false
+  metadataServer:
+    activeCount: 1
+    # No standby-replay, unlike `cephfs`. A standby exists there so a failed
+    # MDS does not hang every CephFS mount cluster-wide; here the only casualty
+    # is a running pipeline, and a second daemon to protect throwaway work is
+    # not worth its memory.
+    activeStandby: false
+    resources:
+      # Deliberately NOT the 4Gi that `cephfs` needs. That figure is driven by
+      # immich's millions of cached inodes; a CI workspace is a few thousand
+      # short-lived files that vanish with the pipeline.
+      requests:
+        cpu: 100m
+        memory: 512Mi
+      limits:
+        memory: 1Gi
+---
+# CI-only class. `ceph-filesystem-ephemeral` is deliberately left pointing at
+# the main filesystem: velero/restore-test.sh uses it to restore REAL data for
+# verification, which must not land on unreplicated pools.
+#
+# A new class rather than repointing the old one — StorageClass.parameters are
+# immutable, so `fsName` cannot be edited in place.
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+  name: ceph-filesystem-ci
+provisioner: rook-ceph.cephfs.csi.ceph.com
+parameters:
+  clusterID: rook-ceph
+  fsName: cephfs-ci
+  pool: cephfs-ci-data0
+  csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-provisioner
+  csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
+  csi.storage.k8s.io/controller-expand-secret-name: rook-csi-cephfs-provisioner
+  csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
+  csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-node
+  csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
+# Delete, for the same reason ceph-filesystem-ephemeral uses it: with Retain,
+# every finished pipeline leaks a Released PV pinning a subvolume (191 of them
+# by 2026-07-16, issue #262 follow-up).
+reclaimPolicy: Delete
+allowVolumeExpansion: true

kubernetes/infrastructure/rook-ceph-cluster/kustomization.yaml

@@ -4,5 +4,6 @@ resources:
   - cephcluster.yaml
   - toolbox.yaml
   - storageclass.yaml
+  - cephfs-ci.yaml
   - volumesnapshotclass.yaml
   - objectstore.yaml