fix(rook-ceph): rotate CephX daemon keys to close CVE-2025-30156

Ceph v19.2.6 ships the fix for CVE-2025-30156, a CephX authentication bypass: AES-128-CBC service tickets carry no HMAC and use a hard-coded IV, so a holder of any low-privilege key can bit-flip a ticket into cluster-admin in linear time. Upgrading the binaries does not resolve it — the keys themselves must be rotated to the aes256k cipher. The cluster is mid-procedure. auth_allowed_ciphers (aes, aes256k), auth_preferred_cipher (aes256k) and auth_service_cipher (aes256k) are already set, and the rotating service keys were wiped, but no actual key rotation has happened: every entity in status.cephx is still keyGeneration 1 with keyType aes, created under 19.2.3. Rook automates daemon-key rotation but it is opt-in and defaults to Disabled, so spec.security.cephx was left empty and nothing rotated. Rook v1.20.6 and Ceph v19.2.6 are both past the version floor the feature requires. Rotating daemon keys clears AUTH_INSECURE_SERVICE_KEY_TYPE, the only one of the six new checks at ERR severity. That is the operationally urgent part: Rook skips CephFS and RGW reconciliation entirely while health is HEALTH_ERR — which is what deadlocked the mds/rgw rollout in issue #527 — and the check is presently held down only by a mon-side mute with a finite TTL. When that mute lapses the cluster returns to HEALTH_ERR and the reconcile freeze comes back. CSI keys are deliberately left on aes. aes256k over the CSI mount path requires a host kernel of 7.0 or newer and these nodes run 6.18, so `aes` must stay in auth_allowed_ciphers for CSI to authenticate at all. AUTH_INSECURE_CLIENT_KEY_TYPE, AUTH_INSECURE_KEYS_ALLOWED and AUTH_INSECURE_KEYS_CREATABLE therefore stay raised at WARN until the nodes run a kernel that supports it. Rotation restarts most Ceph daemons. With size 2 pools across 3 OSDs this should be applied deliberately rather than alongside other churn, and Ceph should reach HEALTH_OK before anything else touches the cluster. Refs #527, #528 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fix(monitoring): CephHealthWarn has been firing continuously and hiding everything
2026-08-21 22:05:10 +02:00 · 2026-08-21 09:53:37 +02:00 · 2026-08-21 07:32:11 +00:00
2 changed files with 82 additions and 8 deletions
@@ -12,18 +12,65 @@ spec:
    - name: ceph.rules
      rules:
        - alert: CephHealthWarn
-          # ceph_health_status: 0 = HEALTH_OK, 1 = HEALTH_WARN, 2 = HEALTH_ERR.
-          # 6h is deliberately long. Ceph warns for self-clearing conditions —
-          # a slow BlueStore op during the Sunday restore sweep, a mon briefly
-          # below its disk threshold — and paging on those trains people to
-          # ignore the alert. What matters is a warning that does not clear.
-          expr: ceph_health_status == 1
+          # Was `ceph_health_status == 1`, which had been FIRING CONTINUOUSLY
+          # since at least 2026-08-21T00:26Z and was therefore telling us
+          # nothing: four AUTH_INSECURE_* checks are permanently active
+          # (CVE-2025-30156, issue #527) and cannot clear until Rook >=1.20.5.
+          # A permanently-firing alert masks every subsequent Ceph warning,
+          # which is the opposite of what this rule is for.
+          #
+          # ceph_health_detail exposes each check individually, so alert on any
+          # check that is NOT a known-accepted one. Verified against live data:
+          # 5 checks active, 0 unexpected — so this is quiet now and fires the
+          # moment a genuinely new condition appears.
+          #
+          # Accepted, each for a documented reason:
+          #   AUTH_INSECURE_*       CVE-2025-30156, tracked in #527
+          #   POOL_NO_REDUNDANCY    cephfs-ci is size 1 ON PURPOSE (CI
+          #                         workspaces are throwaway). Non-CI pools
+          #                         losing redundancy is caught by
+          #                         CephPoolLostRedundancy below, since this
+          #                         check does not say WHICH pool.
+          #
+          # 6h is kept: Ceph warns for self-clearing conditions — a slow
+          # BlueStore op during the Sunday restore sweep, a mon briefly below
+          # its disk threshold — and paging on those trains people to ignore
+          # the alert. What matters is a warning that does not clear.
+          expr: ceph_health_detail{name!~"AUTH_INSECURE_.+|POOL_NO_REDUNDANCY"} == 1
          for: 6h
          labels:
            severity: warning
          annotations:
-            summary: "Ceph has been HEALTH_WARN for 6h"
-            description: "The Ceph cluster has held HEALTH_WARN for six hours, so it is not a self-clearing blip. Get the detail with `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail`."
+            summary: "Ceph health check {{ $labels.name }} active for 6h"
+            description: "Ceph health check {{ $labels.name }} has been active for six hours, so it is not a self-clearing blip. Known-accepted checks (AUTH_INSECURE_*, POOL_NO_REDUNDANCY) are excluded from this alert, so this is a new condition. Detail: `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail`."
+
+        - alert: CephPoolLostRedundancy
+          # Restores what excluding POOL_NO_REDUNDANCY from CephHealthWarn
+          # gives up. That check fires cluster-wide without naming a pool, so
+          # accepting it for cephfs-ci would also hide a PRODUCTION pool
+          # dropping to one replica.
+          #
+          # There is no pool-size metric, but bytes_used/stored IS the
+          # replication factor. Verified live: replicapool 2, cephfs-data0 2,
+          # cephfs-metadata 3, cephfs-ci-metadata 1.
+          #
+          # The >1e6 floor excludes trivially small pools. It is not needed to
+          # suppress the near-empty rgw pools — their ratios are 9-36 because
+          # per-object overhead dominates, so they never match `< 1.5` anyway.
+          #
+          # Both directions were tested against live data before committing:
+          # including cephfs-ci the expression matches (pool 13), excluding it
+          # the expression is empty. An alert that cannot fire is worse than no
+          # alert, so the positive control matters as much as the negative.
+          expr: |
+            (ceph_pool_bytes_used / ceph_pool_stored < 1.5 and ceph_pool_stored > 1e6)
+              and on(pool_id) ceph_pool_metadata{name!~"cephfs-ci-.+"}
+          for: 30m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Ceph pool {{ $labels.pool_id }} is running unreplicated"
+            description: "Pool {{ $labels.pool_id }} has a replication factor below 1.5, meaning a single OSD loss destroys data. Only the cephfs-ci pools are meant to be unreplicated. Check with `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd pool ls detail`."

        - alert: CephHealthError
          # HEALTH_ERR means data availability or redundancy is at risk.
@@ -17,6 +17,33 @@ spec:
    modules:
      - name: pg_autoscaler
        enabled: true
+  # CVE-2025-30156: CephX AES-128-CBC service tickets carry no HMAC and use a
+  # hard-coded IV, so a holder of any low-privilege key can bit-flip a ticket
+  # into cluster-admin. The fix is rotating keys to the aes256k cipher —
+  # upgrading the binaries alone does not close it. Rotation is opt-in and
+  # defaults to Disabled, so it must be requested here.
+  #
+  # Daemon keys only. CSI keys stay on aes: aes256k over the CSI mount path
+  # needs a host kernel of 7.0 or newer and these nodes run 6.18. `aes`
+  # therefore has to remain in auth_allowed_ciphers for CSI to authenticate,
+  # which leaves AUTH_INSECURE_CLIENT_KEY_TYPE, AUTH_INSECURE_KEYS_ALLOWED and
+  # AUTH_INSECURE_KEYS_CREATABLE raised. Revisit once the nodes run a kernel
+  # that supports it.
+  #
+  # Rotating the daemon keys clears AUTH_INSECURE_SERVICE_KEY_TYPE, the only
+  # one of the six checks at ERR severity. That matters beyond the CVE itself:
+  # Rook skips CephFS and RGW reconciliation entirely while health is
+  # HEALTH_ERR, and that check is currently held down only by a mon-side mute
+  # with a finite TTL.
+  #
+  # keyType is deliberately unset — Rook picks the best type for daemon keys.
+  # To rotate again later, increment keyGeneration; Rook restarts the daemons
+  # to pick up the new keys.
+  security:
+    cephx:
+      daemon:
+        keyRotationPolicy: KeyGeneration
+        keyGeneration: 2
  # Bump CephFS subvolume cloner pool from 4 → 16. Velero CSI Snapshot
  # Data Movement provisions a backupPVC per source PVC by cloning the
  # VolumeSnapshot, and a 4-thread default stalls migrate-immich's