fix(rook-ceph): rotate CephX daemon keys to close CVE-2025-30156 #545

Merged

k.renner merged 1 commits from rotate-cephx-daemon-keys into master 2026-08-21 20:07:26 +00:00

Conversation 1 Commits 1 Files Changed 1

+27

claude-bot commented 2026-08-21 20:06:30 +00:00

Member

Copy Link

Copy Source

Closes the operationally urgent half of #527, and lets #528 be closed.

What is wrong

Ceph v19.2.6 ships the fix for CVE-2025-30156, a CephX authentication bypass — AES-128-CBC service tickets carry no HMAC and use a hard-coded IV, so a holder of any low-privilege key can bit-flip a ticket into cluster-admin in linear time. Upgrading the binaries does not resolve it. The keys themselves must be rotated to the aes256k cipher.

The cluster is stalled mid-procedure. The config half is done:

auth_allowed_ciphers    aes, aes256k
auth_preferred_cipher   aes256k
auth_service_cipher     aes256k

but no key has actually rotated — every entity in status.cephx is still keyGeneration: 1, keyType: aes, created under 19.2.3:

mon / osd / mgr / csi / crashCollector / cephExporter / rbdMirrorPeer
  keyGeneration 1, keyCephVersion 19.2.3-0

Rook automates daemon-key rotation, but it is opt-in and defaults to Disabled. spec.security.cephx was left empty ({csi:{}, daemon:{}, rbdMirrorPeer:{}}), so nothing rotated. Rook v1.20.6 and Ceph v19.2.6 are both past the version floor the feature needs (v1.20.5 / v19.2.6).

Why this is urgent beyond the CVE

Rotating daemon keys clears AUTH_INSECURE_SERVICE_KEY_TYPE — the only one of the six new checks at ERR severity.

Rook skips CephFS and RGW reconciliation entirely while health is HEALTH_ERR. That is what deadlocked the mds/rgw rollout described in #527. Right now the check is suppressed only by a mon-side mute with a finite TTL; when it lapses the cluster returns to HEALTH_ERR and the reconcile freeze returns with it.

What this does NOT fix

CSI keys stay on aes deliberately. aes256k over the CSI mount path needs a host kernel of 7.0 or newer, and these nodes run 6.18.18-talos. So aes has to remain in auth_allowed_ciphers for CSI to authenticate at all, and these stay raised at WARN until the nodes run a kernel that supports it:

• AUTH_INSECURE_CLIENT_KEY_TYPE
• AUTH_INSECURE_KEYS_ALLOWED
• AUTH_INSECURE_KEYS_CREATABLE

All three are already excluded from CephHealthWarn by #544, so this will not generate alert noise. Worth noting the rule annotation there calls them "known-accepted checks", which reads as a settled decision rather than a kernel-blocked remediation — worth rewording separately.

Applying this

Rotation restarts most Ceph daemons. With size 2 pools across 3 OSDs there is no redundancy margin during an OSD restart, so this should land on its own rather than alongside other churn, and Ceph should reach HEALTH_OK before anything else touches the cluster.

Verify with:

kubectl -n rook-ceph get cephcluster rook-ceph -o jsonpath='{.status.cephx}'
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail

Daemon entries should move to keyGeneration: 2 with keyType: aes256k, and AUTH_INSECURE_SERVICE_KEY_TYPE should disappear — at which point the mon-side mute is no longer load-bearing and can be dropped.

To rotate again later, increment keyGeneration.

Refs #527, #528

🤖 Generated with Claude Code

Closes the operationally urgent half of #527, and lets #528 be closed. ## What is wrong Ceph v19.2.6 ships the fix for [CVE-2025-30156](https://docs.ceph.com/en/latest/security/CVE-2025-30156/), a CephX authentication bypass — AES-128-CBC service tickets carry no HMAC and use a hard-coded IV, so a holder of any low-privilege key can bit-flip a ticket into cluster-admin in linear time. **Upgrading the binaries does not resolve it.** The keys themselves must be rotated to the `aes256k` cipher. The cluster is stalled mid-procedure. The config half is done: ``` auth_allowed_ciphers aes, aes256k auth_preferred_cipher aes256k auth_service_cipher aes256k ``` but no key has actually rotated — every entity in `status.cephx` is still `keyGeneration: 1`, `keyType: aes`, created under 19.2.3: ``` mon / osd / mgr / csi / crashCollector / cephExporter / rbdMirrorPeer keyGeneration 1, keyCephVersion 19.2.3-0 ``` Rook automates daemon-key rotation, but it is opt-in and defaults to `Disabled`. `spec.security.cephx` was left empty (`{csi:{}, daemon:{}, rbdMirrorPeer:{}}`), so nothing rotated. Rook v1.20.6 and Ceph v19.2.6 are both past the version floor the feature needs (v1.20.5 / v19.2.6). ## Why this is urgent beyond the CVE Rotating daemon keys clears `AUTH_INSECURE_SERVICE_KEY_TYPE` — the only one of the six new checks at **ERR** severity. Rook skips CephFS and RGW reconciliation entirely while health is `HEALTH_ERR`. That is what deadlocked the mds/rgw rollout described in #527. Right now the check is suppressed only by a mon-side mute with a finite TTL; when it lapses the cluster returns to `HEALTH_ERR` and the reconcile freeze returns with it. ## What this does NOT fix CSI keys stay on `aes` deliberately. `aes256k` over the CSI mount path needs a host kernel of **7.0 or newer**, and these nodes run **6.18.18-talos**. So `aes` has to remain in `auth_allowed_ciphers` for CSI to authenticate at all, and these stay raised at WARN until the nodes run a kernel that supports it: - `AUTH_INSECURE_CLIENT_KEY_TYPE` - `AUTH_INSECURE_KEYS_ALLOWED` - `AUTH_INSECURE_KEYS_CREATABLE` All three are already excluded from `CephHealthWarn` by #544, so this will not generate alert noise. Worth noting the rule annotation there calls them "known-accepted checks", which reads as a settled decision rather than a kernel-blocked remediation — worth rewording separately. ## Applying this **Rotation restarts most Ceph daemons.** With size 2 pools across 3 OSDs there is no redundancy margin during an OSD restart, so this should land on its own rather than alongside other churn, and Ceph should reach `HEALTH_OK` before anything else touches the cluster. Verify with: ``` kubectl -n rook-ceph get cephcluster rook-ceph -o jsonpath='{.status.cephx}' kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail ``` Daemon entries should move to `keyGeneration: 2` with `keyType: aes256k`, and `AUTH_INSECURE_SERVICE_KEY_TYPE` should disappear — at which point the mon-side mute is no longer load-bearing and can be dropped. To rotate again later, increment `keyGeneration`. Refs #527, #528 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit 2026-08-21 20:06:30 +00:00

fix(rook-ceph): rotate CephX daemon keys to close CVE-2025-30156 ... 2abd946c05

Ceph v19.2.6 ships the fix for CVE-2025-30156, a CephX authentication
bypass: AES-128-CBC service tickets carry no HMAC and use a hard-coded
IV, so a holder of any low-privilege key can bit-flip a ticket into
cluster-admin in linear time. Upgrading the binaries does not resolve
it — the keys themselves must be rotated to the aes256k cipher.

The cluster is mid-procedure. auth_allowed_ciphers (aes, aes256k),
auth_preferred_cipher (aes256k) and auth_service_cipher (aes256k) are
already set, and the rotating service keys were wiped, but no actual
key rotation has happened: every entity in status.cephx is still
keyGeneration 1 with keyType aes, created under 19.2.3.

Rook automates daemon-key rotation but it is opt-in and defaults to
Disabled, so spec.security.cephx was left empty and nothing rotated.
Rook v1.20.6 and Ceph v19.2.6 are both past the version floor the
feature requires.

Rotating daemon keys clears AUTH_INSECURE_SERVICE_KEY_TYPE, the only
one of the six new checks at ERR severity. That is the operationally
urgent part: Rook skips CephFS and RGW reconciliation entirely while
health is HEALTH_ERR — which is what deadlocked the mds/rgw rollout in
issue #527 — and the check is presently held down only by a mon-side
mute with a finite TTL. When that mute lapses the cluster returns to
HEALTH_ERR and the reconcile freeze comes back.

CSI keys are deliberately left on aes. aes256k over the CSI mount path
requires a host kernel of 7.0 or newer and these nodes run 6.18, so
`aes` must stay in auth_allowed_ciphers for CSI to authenticate at all.
AUTH_INSECURE_CLIENT_KEY_TYPE, AUTH_INSECURE_KEYS_ALLOWED and
AUTH_INSECURE_KEYS_CREATABLE therefore stay raised at WARN until the
nodes run a kernel that supports it.

Rotation restarts most Ceph daemons. With size 2 pools across 3 OSDs
this should be applied deliberately rather than alongside other churn,
and Ceph should reach HEALTH_OK before anything else touches the
cluster.

Refs #527, #528

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

k.renner merged commit 04b8c93113 into master 2026-08-21 20:07:26 +00:00

k.renner referenced this issue from a commit 2026-08-21 20:07:26 +00:00

Merge pull request 'fix(rook-ceph): rotate CephX daemon keys to close CVE-2025-30156' (#545) from rotate-cephx-daemon-keys into master

review-bot requested changes 2026-08-21 20:12:01 +00:00

review-bot left a comment

Member

Copy Link

Copy Source

🤖 Automated review — model CLI default (complexity: complex), commit 2abd946c05765b56b3b660c2daadfd908e6d2bb5.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

---

Summary

This PR sets spec.security.cephx.daemon (keyRotationPolicy: KeyGeneration, keyGeneration: 2) in kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml to make Rook rotate internal Ceph daemon keys (mon/osd/mgr/crashCollector/cephExporter) to the aes256k cipher, closing the ERR-severity AUTH_INSECURE_SERVICE_KEY_TYPE check for CVE-2025-30156. CSI keys are deliberately left untouched because the CSI mount path needs a kernel ≥7.0 and these nodes run 6.18. The change itself is small, well-commented, and structurally consistent with the rest of the file (indentation, sibling key placement). The one real gap is that the mon-side cipher-negotiation settings this rotation depends on for daemons to actually authenticate with aes256k are not declared anywhere in this repository.

🔴 Blocking

• kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml: the auth_allowed_ciphers / auth_preferred_cipher / auth_service_cipher settings this PR depends on are not present anywhere in the repo. The PR body quotes them as already applied live (ceph config dump-style output), but I grepped the entire tree for auth_allowed_ciphers, auth_preferred_cipher, auth_service_cipher, and aes256k and the only hits are the prose comments this diff adds (lines 20-41) — there is no cephConfig entry (e.g. a global: section, alongside the existing cephConfig.mon / cephConfig.mgr / cephConfig.osd blocks at lines 53-160) that declares them, and no other tracked file sets them either. That means the cipher allow/prefer list this fix depends on currently exists only as untracked, out-of-band cluster state.

  This matters specifically because this file's own conventions elsewhere argue against exactly that: the bluestore_prefer_deferred_size_ssd/bluestore_deferred_batch_ops_ssd block (lines 156-160) is deliberately set to explicit values "so the previous override is actively overwritten rather than merely removed," and the mon section carries a "BARE-METAL MIGRATION" note (lines 68-73) showing the author already cares about this cluster's config being reproducible from git. If this cluster's mon config store is ever rebuilt, restored, or reset from this repo alone, keyRotationPolicy: KeyGeneration will rotate daemon keys to aes256k, but nothing in git will have told Ceph to allow or prefer that cipher — which, if the shipped Ceph default excludes aes256k, breaks daemon authentication rather than fixing it.

  I can't confirm from the checkout alone whether these settings are genuinely live on the cluster right now (the PR author's quoted ceph config dump output implies yes) or whether Rook's CephX feature itself sets them automatically as a side effect of keyRotationPolicy (I found nothing in this repo suggesting Rook manages cipher negotiation, and the CVE's own upstream guidance treats these as Ceph-level config the operator must set independently of Rook's key-rotation CRD field). The settling check: kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph config dump | grep auth_ — if that confirms the three settings are live, they should still be added to this file's cephConfig (likely a global: section) in this PR so the fix is captured in git rather than living only as manual cluster state.

🟡 Suggestions

• CI's kustomize-build step (.woodpecker/ci-manifests-test.yml) only runs kubectl kustomize, which renders the YAML but does not validate it against the installed CephCluster CRD (no API server, no schema check). A typo'd field under the new spec.security.cephx.daemon block (e.g. a wrong enum value for keyRotationPolicy) would render clean here and only surface at apply time. Worth a kubectl apply --dry-run=server (or equivalent) against a real cluster/CRD before merging, since this is new-to-the-repo CRD surface with no prior usage to sanity-check the field names against.

Tests

No automated test in this repo exercises CephCluster semantics beyond the offline kubectl kustomize render noted above (confirmed by reading .woodpecker/ci-manifests-test.yml), so there's nothing here that would catch a wrong field name or enum value before it reaches the cluster. The PR body's own post-apply verification (kubectl -n rook-ceph get cephcluster rook-ceph -o jsonpath='{.status.cephx}' and ceph health detail, checking for keyGeneration: 2 / keyType: aes256k and the disappearance of AUTH_INSECURE_SERVICE_KEY_TYPE) is the right manual check and should be run and watched as described, including the operational caution already called out in the body (land this alone, not alongside other churn, given size-2 pools across 3 OSDs).

Recommendation

Request changes — 1 blocking issue.

🤖 **Automated review** — model `CLI default` (complexity: complex), commit `2abd946c05765b56b3b660c2daadfd908e6d2bb5`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary This PR sets `spec.security.cephx.daemon` (`keyRotationPolicy: KeyGeneration`, `keyGeneration: 2`) in `kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml` to make Rook rotate internal Ceph daemon keys (mon/osd/mgr/crashCollector/cephExporter) to the `aes256k` cipher, closing the ERR-severity `AUTH_INSECURE_SERVICE_KEY_TYPE` check for CVE-2025-30156. CSI keys are deliberately left untouched because the CSI mount path needs a kernel ≥7.0 and these nodes run 6.18. The change itself is small, well-commented, and structurally consistent with the rest of the file (indentation, sibling key placement). The one real gap is that the mon-side cipher-negotiation settings this rotation depends on for daemons to actually authenticate with `aes256k` are not declared anywhere in this repository. ## 🔴 Blocking - **`kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml`: the `auth_allowed_ciphers` / `auth_preferred_cipher` / `auth_service_cipher` settings this PR depends on are not present anywhere in the repo.** The PR body quotes them as already applied live (`ceph config dump`-style output), but I grepped the entire tree for `auth_allowed_ciphers`, `auth_preferred_cipher`, `auth_service_cipher`, and `aes256k` and the only hits are the prose comments this diff adds (lines 20-41) — there is no `cephConfig` entry (e.g. a `global:` section, alongside the existing `cephConfig.mon` / `cephConfig.mgr` / `cephConfig.osd` blocks at lines 53-160) that declares them, and no other tracked file sets them either. That means the cipher allow/prefer list this fix depends on currently exists only as untracked, out-of-band cluster state. This matters specifically because this file's own conventions elsewhere argue against exactly that: the `bluestore_prefer_deferred_size_ssd`/`bluestore_deferred_batch_ops_ssd` block (lines 156-160) is deliberately set to explicit values "so the previous override is actively overwritten rather than merely removed," and the mon section carries a "BARE-METAL MIGRATION" note (lines 68-73) showing the author already cares about this cluster's config being reproducible from git. If this cluster's mon config store is ever rebuilt, restored, or reset from this repo alone, `keyRotationPolicy: KeyGeneration` will rotate daemon keys to `aes256k`, but nothing in git will have told Ceph to allow or prefer that cipher — which, if the shipped Ceph default excludes `aes256k`, breaks daemon authentication rather than fixing it. I can't confirm from the checkout alone whether these settings are genuinely live on the cluster right now (the PR author's quoted `ceph config dump` output implies yes) or whether Rook's CephX feature itself sets them automatically as a side effect of `keyRotationPolicy` (I found nothing in this repo suggesting Rook manages cipher negotiation, and the CVE's own upstream guidance treats these as Ceph-level config the operator must set independently of Rook's key-rotation CRD field). The settling check: `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph config dump | grep auth_` — if that confirms the three settings are live, they should still be added to this file's `cephConfig` (likely a `global:` section) in this PR so the fix is captured in git rather than living only as manual cluster state. ## 🟡 Suggestions - CI's `kustomize-build` step (`.woodpecker/ci-manifests-test.yml`) only runs `kubectl kustomize`, which renders the YAML but does not validate it against the installed CephCluster CRD (no API server, no schema check). A typo'd field under the new `spec.security.cephx.daemon` block (e.g. a wrong enum value for `keyRotationPolicy`) would render clean here and only surface at apply time. Worth a `kubectl apply --dry-run=server` (or equivalent) against a real cluster/CRD before merging, since this is new-to-the-repo CRD surface with no prior usage to sanity-check the field names against. ## Tests No automated test in this repo exercises CephCluster semantics beyond the offline `kubectl kustomize` render noted above (confirmed by reading `.woodpecker/ci-manifests-test.yml`), so there's nothing here that would catch a wrong field name or enum value before it reaches the cluster. The PR body's own post-apply verification (`kubectl -n rook-ceph get cephcluster rook-ceph -o jsonpath='{.status.cephx}'` and `ceph health detail`, checking for `keyGeneration: 2` / `keyType: aes256k` and the disappearance of `AUTH_INSECURE_SERVICE_KEY_TYPE`) is the right manual check and should be run and watched as described, including the operational caution already called out in the body (land this alone, not alongside other churn, given size-2 pools across 3 OSDs). ## Recommendation Request changes — 1 blocking issue. <!-- pr-review:sha=2abd946c05765b56b3b660c2daadfd908e6d2bb5 -->

claude-bot referenced this issue from a commit 2026-08-21 20:21:20 +00:00

fix(rook-ceph): raise cephfs-ci to size 2 — size 1 blocks all OSD maintenance

claude-bot referenced this pull request 2026-08-21 20:21:42 +00:00

fix(rook-ceph): raise cephfs-ci to size 2 — size 1 blocks all OSD maintenance #547

claude-bot referenced this pull request 2026-08-21 20:24:42 +00:00

docs(rook-ceph): cut the narrative out of the ceph manifest comments #548

claude-bot referenced this issue from a commit 2026-08-21 20:27:17 +00:00

docs(rook-ceph): cut the narrative out of the ceph manifest comments

claude-bot referenced this pull request 2026-08-21 20:32:40 +00:00

CVE-2025-30156: Ceph 19.2.6 auto-upgrade left cluster HEALTH_ERR and deadlocked the MDS/RGW rollout #527

Sign in to join this conversation.

Reviewers

No Reviewers

review-bot

Labels

Clear labels

apollo

Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant

atlas

Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring

clio

Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites

daedalus

Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes

hestia

Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#545