konairius/homelab
10
0
Fork 0
Code Issues 8 Pull Requests Packages Projects Releases 1 Wiki Activity

[alert] CephHealthError (rook-ceph) #525

New Issue
Closed
opened 2026-08-19 16:06:13 +00:00 by alert-monitor-bot · 2 comments
alert-monitor-bot commented 2026-08-19 16:06:13 +00:00
Member
Copy Link
Copy Source

What is firing

CephHealthError (severity critical), namespace rook-ceph, firing since 2026-08-19 15:53:20 UTC and still active (ALERTS{alertname="CephHealthError"} = 1 at 16:04 UTC).

Rule: kubernetes/apps/monitoring-rules/prometheusrule-ceph.yaml:28 — ceph_health_status == 2 for 5m.

Summary

Ceph really is in HEALTH_ERR, but no data is at risk. The Ceph v19.2.5 -> v19.2.6 upgrade that Flux image-automation committed ten minutes before the alert (694389b) added six new AUTH_INSECURE_* cephx health checks, two of them ERR-severity. They report pre-existing keyring state — every cephx key in this cluster uses the legacy aes key type — not a new fault.

Evidence

The upgrade completed successfully. Prometheus query of ceph_mon_metadata or ceph_osd_metadata or ceph_mgr_metadata (via kubectl get --raw against the Prometheus proxy): all 3 mons, 3 OSDs and 2 mgrs report ceph version 19.2.6 (f9fd95b4...) squid (stable). kubectl -n rook-ceph get pods shows every Ceph daemon Running with 0 restarts.

Storage is healthy. Same query path: ceph_osd_up and ceph_osd_in are 1 for osd.0/1/2; across all 12 pools ceph_pg_active == ceph_pg_clean == ceph_pg_total, and ceph_pg_degraded, ceph_pg_inconsistent, ceph_pg_undersized are all 0.

The ERR comes only from auth checks. ceph_health_detail{namespace="rook-ceph"} > 0 returns exactly six series — ERR on AUTH_INSECURE_SERVICE_KEY_TYPE and AUTH_INSECURE_SERVICE_TICKETS, WARN on AUTH_INSECURE_CLIENT_KEY_TYPE, AUTH_INSECURE_KEYS_ALLOWED, AUTH_INSECURE_KEYS_CREATABLE, AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE. No non-auth check is failing.

mon.a cluster log (Loki, {namespace="rook-ceph"} |~ "(?i)insecure"):

[ERR] Health check failed: 7 auth service entities with insecure key types (AUTH_INSECURE_SERVICE_KEY_TYPE)
[ERR] Health check failed: Monitors are configured to issue insecure service tickets (AUTH_INSECURE_SERVICE_TICKETS)
[WRN] AUTH_INSECURE_KEYS_ALLOWED: insecure cipher aes allowed for auth

Flagged client entities, each using insecure key type: aes: client.admin, client.ceph-exporter, client.crash, client.csi-cephfs-node.1, client.csi-cephfs-provisioner.1, client.csi-rbd-node.1, client.csi-rbd-provisioner.1, client.rbd-mirror-peer, client.rgw.migration.a. Rotating service keys for mon, mds, osd, mgr are flagged the same way.

Timeline:

Time (UTC) Event
15:43:41 694389b bumps quay.io/ceph/ceph to v19.2.6 (Flux image-automation)
15:45:41 mon.a logs the six health-check failures; ceph_health_status 0 -> 2
15:53:20 CephHealthError fires

ceph_health_status was 0 (HEALTH_OK) for the entire preceding 6 hours (query_range, step 600s), and count(ceph_health_detail{name=~"AUTH_INSECURE.*"}) has no samples at all before the new mgr pod started — confirming these checks ship with 19.2.6 rather than reflecting a change in cluster state.

The Startup probe failed warnings for rook-ceph-osd-0/1/2 in the context snapshot were the rolling upgrade restarting each OSD; all three are now Running with 0 restarts, so they are stale. Likewise the waiting for ceph monitors upgrade to finish reconcile errors on cephobjectstore/migration, cephfilesystem/cephfs and cephblockpool/replicapool — the mon upgrade has since completed.

Root cause

Verified. Ceph 19.2.6 introduced cephx key-type health checks that flag the legacy aes key type as insecure. Every keyring in this cluster predates that change and uses aes, so the patch upgrade converted a long-standing security posture into a HEALTH_ERR. Nothing about data durability or availability changed — this is a reporting change, not a regression.

Hypothesis, not verified: the exact remediation mechanism. The mon log shows what is flagged but not the config options that govern key-type policy, and this agent has no Ceph admin socket access. Settle it before changing anything:

kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph config ls
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph auth ls

read alongside the Ceph v19.2.6 release notes for the cephx key-type transition.

Where to fix it

There are two separable decisions here.

1. Fix the alerting assumption first (repo-only, low risk).

kubernetes/apps/monitoring-rules/prometheusrule-ceph.yaml:29 carries the comment "HEALTH_ERR means data availability or redundancy is at risk." As of 19.2.6 that is no longer true — a pure auth-policy check can drive HEALTH_ERR. This matters beyond tidiness: while this condition persists, a genuine data-risk HEALTH_ERR is indistinguishable from it, because ceph_health_status == 2 is already saturated. Options: alert on specific ceph_health_detail checks, or exclude the AUTH_INSECURE_* family from the critical rule and cover it with a separate lower-severity alert. This is the more urgent change, since it restores the cluster's ability to page for real availability loss.

2. Remediate the auth posture (NOT fixable in the repo alone).

Rotating cephx keys to the modern key type is live-cluster state, not manifest state. It touches client.admin, both CSI provisioner/node key pairs and the RGW key; the CSI keys are consumed by Rook-managed secrets, so this must be driven through Rook or the toolbox rather than hand-edited — rotating them carelessly will break RBD/CephFS volume provisioning cluster-wide. A human should plan this deliberately against the 19.2.6 release notes.

Any durable configuration that comes out of that work belongs in spec.cephConfig in kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml (which already has mon:, osd: and mgr: blocks), so it survives reconciliation rather than being set imperatively and silently lost.

Also worth reviewing: kubernetes/infrastructure/image-automation/ceph.yaml auto-merges any 19.x patch release with no human gate. That is how a health-check behaviour change landed unattended on a Saturday-equivalent. The policy comment already reserves the 19 -> 20 cutover for a deliberate upgrade; this incident suggests patch releases may warrant at least a notification gate too.

Filed automatically by the alert-monitor pipeline (2026-08-19). The issue-fix pipeline may pick this up and open a PR.

## What is firing `CephHealthError` (severity `critical`), namespace `rook-ceph`, firing since **2026-08-19 15:53:20 UTC** and still active (`ALERTS{alertname="CephHealthError"}` = 1 at 16:04 UTC). Rule: `kubernetes/apps/monitoring-rules/prometheusrule-ceph.yaml:28` — `ceph_health_status == 2` for 5m. ## Summary Ceph really is in HEALTH_ERR, but **no data is at risk**. The Ceph v19.2.5 -> v19.2.6 upgrade that Flux image-automation committed ten minutes before the alert (`694389b`) added six new `AUTH_INSECURE_*` cephx health checks, two of them ERR-severity. They report **pre-existing keyring state** — every cephx key in this cluster uses the legacy `aes` key type — not a new fault. ## Evidence **The upgrade completed successfully.** Prometheus query of `ceph_mon_metadata or ceph_osd_metadata or ceph_mgr_metadata` (via `kubectl get --raw` against the Prometheus proxy): all 3 mons, 3 OSDs and 2 mgrs report `ceph version 19.2.6 (f9fd95b4...) squid (stable)`. `kubectl -n rook-ceph get pods` shows every Ceph daemon `Running` with 0 restarts. **Storage is healthy.** Same query path: `ceph_osd_up` and `ceph_osd_in` are `1` for osd.0/1/2; across all 12 pools `ceph_pg_active` == `ceph_pg_clean` == `ceph_pg_total`, and `ceph_pg_degraded`, `ceph_pg_inconsistent`, `ceph_pg_undersized` are all `0`. **The ERR comes only from auth checks.** `ceph_health_detail{namespace="rook-ceph"} > 0` returns exactly six series — ERR on `AUTH_INSECURE_SERVICE_KEY_TYPE` and `AUTH_INSECURE_SERVICE_TICKETS`, WARN on `AUTH_INSECURE_CLIENT_KEY_TYPE`, `AUTH_INSECURE_KEYS_ALLOWED`, `AUTH_INSECURE_KEYS_CREATABLE`, `AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE`. No non-auth check is failing. **mon.a cluster log** (Loki, `{namespace="rook-ceph"} |~ "(?i)insecure"`): ``` [ERR] Health check failed: 7 auth service entities with insecure key types (AUTH_INSECURE_SERVICE_KEY_TYPE) [ERR] Health check failed: Monitors are configured to issue insecure service tickets (AUTH_INSECURE_SERVICE_TICKETS) [WRN] AUTH_INSECURE_KEYS_ALLOWED: insecure cipher aes allowed for auth ``` Flagged client entities, each `using insecure key type: aes`: `client.admin`, `client.ceph-exporter`, `client.crash`, `client.csi-cephfs-node.1`, `client.csi-cephfs-provisioner.1`, `client.csi-rbd-node.1`, `client.csi-rbd-provisioner.1`, `client.rbd-mirror-peer`, `client.rgw.migration.a`. Rotating service keys for `mon`, `mds`, `osd`, `mgr` are flagged the same way. **Timeline:** | Time (UTC) | Event | |---|---| | 15:43:41 | `694389b` bumps `quay.io/ceph/ceph` to `v19.2.6` (Flux image-automation) | | 15:45:41 | mon.a logs the six health-check failures; `ceph_health_status` 0 -> 2 | | 15:53:20 | `CephHealthError` fires | `ceph_health_status` was `0` (HEALTH_OK) for the entire preceding 6 hours (`query_range`, step 600s), and `count(ceph_health_detail{name=~"AUTH_INSECURE.*"})` has **no samples at all** before the new mgr pod started — confirming these checks ship with 19.2.6 rather than reflecting a change in cluster state. The `Startup probe failed` warnings for `rook-ceph-osd-0/1/2` in the context snapshot were the rolling upgrade restarting each OSD; all three are now `Running` with 0 restarts, so they are stale. Likewise the `waiting for ceph monitors upgrade to finish` reconcile errors on `cephobjectstore/migration`, `cephfilesystem/cephfs` and `cephblockpool/replicapool` — the mon upgrade has since completed. ## Root cause **Verified.** Ceph 19.2.6 introduced cephx key-type health checks that flag the legacy `aes` key type as insecure. Every keyring in this cluster predates that change and uses `aes`, so the patch upgrade converted a long-standing security posture into a HEALTH_ERR. Nothing about data durability or availability changed — this is a reporting change, not a regression. **Hypothesis, not verified:** the exact remediation mechanism. The mon log shows *what* is flagged but not the config options that govern key-type policy, and this agent has no Ceph admin socket access. Settle it before changing anything: ``` kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph config ls kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph auth ls ``` read alongside the Ceph v19.2.6 release notes for the cephx key-type transition. ## Where to fix it There are two separable decisions here. **1. Fix the alerting assumption first (repo-only, low risk).** `kubernetes/apps/monitoring-rules/prometheusrule-ceph.yaml:29` carries the comment *"HEALTH_ERR means data availability or redundancy is at risk."* As of 19.2.6 that is no longer true — a pure auth-policy check can drive HEALTH_ERR. This matters beyond tidiness: **while this condition persists, a genuine data-risk HEALTH_ERR is indistinguishable from it**, because `ceph_health_status == 2` is already saturated. Options: alert on specific `ceph_health_detail` checks, or exclude the `AUTH_INSECURE_*` family from the critical rule and cover it with a separate lower-severity alert. This is the more urgent change, since it restores the cluster's ability to page for real availability loss. **2. Remediate the auth posture (NOT fixable in the repo alone).** Rotating cephx keys to the modern key type is live-cluster state, not manifest state. It touches `client.admin`, both CSI provisioner/node key pairs and the RGW key; the CSI keys are consumed by Rook-managed secrets, so this must be driven through Rook or the toolbox rather than hand-edited — rotating them carelessly will break RBD/CephFS volume provisioning cluster-wide. A human should plan this deliberately against the 19.2.6 release notes. Any *durable configuration* that comes out of that work belongs in `spec.cephConfig` in `kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml` (which already has `mon:`, `osd:` and `mgr:` blocks), so it survives reconciliation rather than being set imperatively and silently lost. **Also worth reviewing:** `kubernetes/infrastructure/image-automation/ceph.yaml` auto-merges any `19.x` patch release with no human gate. That is how a health-check behaviour change landed unattended on a Saturday-equivalent. The policy comment already reserves the 19 -> 20 cutover for a deliberate upgrade; this incident suggests patch releases may warrant at least a notification gate too. --- <!-- alert-monitor:fingerprint=53a1840cb77f1b4b --> _Filed automatically by the alert-monitor pipeline (2026-08-19). The issue-fix pipeline may pick this up and open a PR._
issue-fix-bot commented 2026-08-19 16:46:29 +00:00
Member
Copy Link
Copy Source

🤖 Opened k.renner/homelab#526 with a proposed fix.

🤖 Opened https://gitea.renner.uno/k.renner/homelab/pulls/526 with a proposed fix.
claude-bot commented 2026-08-20 04:41:16 +00:00
Member
Copy Link
Copy Source

Resolved and superseded by #527.

This fired because the automated bump to Ceph v19.2.6 (commit 694389b5) introduced six new AUTH_INSECURE_* health checks, two at [ERR] severity, which flipped the cluster to HEALTH_ERR at 15:46 UTC. Not a data problem — mon quorum, OSDs, MDS and RGW were all healthy throughout.

The serious part was a deadlock: Rook IsReadyToReconcile gate skips the CephFS and CephObjectStore controllers while health is ERR, so 2 MDS and 1 RGW were stranded on v19.2.5 for three hours while the rest of the cluster ran v19.2.6.

Resolved by muting the two ERR checks with a bounded TTL, which let Rook finish:

before: 3 daemons on 19.2.5, CephObjectStore Progressing for 3h
after:  11 daemons ALL on v19.2.6, CephObjectStore Ready, 169/169 PGs active+clean

Cluster health is now HEALTH_WARN (the four remaining auth warnings are left visible on purpose).

The underlying CVE is NOT fixed — see #527. It needs key rotation to aes256k, which requires Rook >= v1.20.5 (unreleased; we run v1.20.4). The mute expires 2026-09-02, tracked there.

Closing this alert issue; #527 carries the remaining work.

🤖 Generated with Claude Code

Resolved and superseded by #527. This fired because the automated bump to Ceph v19.2.6 (commit `694389b5`) introduced six new `AUTH_INSECURE_*` health checks, two at `[ERR]` severity, which flipped the cluster to HEALTH_ERR at 15:46 UTC. Not a data problem — mon quorum, OSDs, MDS and RGW were all healthy throughout. The serious part was a **deadlock**: Rook `IsReadyToReconcile` gate skips the CephFS and CephObjectStore controllers while health is ERR, so 2 MDS and 1 RGW were stranded on v19.2.5 for three hours while the rest of the cluster ran v19.2.6. Resolved by muting the two ERR checks with a bounded TTL, which let Rook finish: ``` before: 3 daemons on 19.2.5, CephObjectStore Progressing for 3h after: 11 daemons ALL on v19.2.6, CephObjectStore Ready, 169/169 PGs active+clean ``` Cluster health is now HEALTH_WARN (the four remaining auth warnings are left visible on purpose). **The underlying CVE is NOT fixed** — see #527. It needs key rotation to `aes256k`, which requires Rook >= v1.20.5 (unreleased; we run v1.20.4). The mute expires **2026-09-02**, tracked there. Closing this alert issue; #527 carries the remaining work. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
3 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#525