konairius/homelab
10
0
Fork 0
Code Issues 9 Pull Requests 2 Packages Projects Releases 1 Wiki Activity

[alert] KubePodNotReady (rook-ceph) #591

New Issue
Closed
opened 2026-08-25 20:51:39 +00:00 by alert-monitor-bot · 2 comments
alert-monitor-bot commented 2026-08-25 20:51:39 +00:00
Member
Copy Link
Copy Source

What is firing
Alert KubePodNotReady (severity warning)
Workload rook-ceph/rook-ceph-mon-b-bc9b5dccb-8zndg
Since 2026-08-25 20:35 UTC

Unrelated to the other four alerts in this batch (those are a v30iot/macvlan problem). Same trigger — the bare-metal conversion window — but a different failure and a different fix.

Summary

rook-ceph-mon-b is Pending and unschedulable because it is pinned by nodeSelector to k8s-2, which is no longer in the cluster. The Rook operator is trying to fail the mon over, cannot, and is now hard-stuck in a reconcile error loop that blocks all CephCluster reconciliation — not just the mon.

The cluster is up and serving on 2/3 mon quorum with no fault tolerance.

Evidence

1. The pod is pinned to a node that no longer exists — kubectl describe pod -n rook-ceph rook-ceph-mon-b-bc9b5dccb-8zndg:

Status:          Pending
Node-Selectors:  kubernetes.io/hostname=k8s-2

FailedScheduling  0/3 nodes are available: 1 node(s) were unschedulable,
                  2 node(s) didn't match Pod's node affinity/selector.

kubectl get nodes now returns only k8s-1 (98m, Ready) and k8s-3 (108d, Ready). k8s-2 has since left entirely — context.txt shows the drain: dozens of NodeShutdown / FailedDaemonPod events for k8s-2 around 5m before the snapshot (cilium, multus, cni-dhcp-daemon, node-exporter, both Ceph CSI nodeplugins).

2. The operator is hard-blocked — kubectl logs -n rook-ceph deploy/rook-ceph-operator --tail=30, repeating continuously with backoff:

failed to reconcile CephCluster "rook-ceph/rook-ceph": failed to perform validation before cluster creation: cannot start 3 mons on 2 node(s) when allowMultiplePerNode is false

This is a pre-flight validation failure, so the operator aborts the whole reconcile. No OSD, mon, mgr or pool changes will be applied until node count returns to 3.

3. Failover cannot proceed either. The operator spawned rook-ceph-mon-e-canary-66c9ffcbd8-9sbkf, which is also Pending:

FailedScheduling  0/2 nodes are available: 2 node(s) didn't have free
                  ports for the requested pod ports.

The canary needs hostPorts 3300/6789, and both surviving nodes already run a mon (mon-a on k8s-3, mon-d on k8s-1). So the mon count cannot be satisfied on two nodes by design.

4. Ceph health — kubectl get cephcluster -n rook-ceph -o jsonpath=..., phase Progressing, HEALTH_WARN:

Check Message
MON_DOWN 1/3 mons down, quorum a,d
TOO_FEW_OSDS OSD count 2 < osd_pool_default_size 3
PG_DEGRADED Degraded data redundancy: 14 pgs undersized

(The three AUTH_INSECURE_* warnings are pre-existing and explained by the security.cephx block in cephcluster.yaml:26-30; not related.)

5. Flux is blocked on it — context.txt:

kustomization/infrastructure-ceph-cluster health check failed after 20m0.016521672s: timeout waiting for: [CephCluster/rook-ceph/rook-ceph status: 'InProgress']

Still true live: Ready=Unknown, Healthy=Unknown, revision master@sha1:ca324825.

Root cause

Verified, not hypothesised: mon.b's data lives at dataDirHostPath /var/lib/rook/mon-b/data on k8s-2, so Rook pins it there by hostname. k8s-2 (saturn) is out of the cluster for its bare-metal conversion — kubernetes/RUNBOOK-baremetal-conversion.md:5 confirms this work is in flight: "Written after doing jupiter on 2026-08-25; saturn and neptune remain."

The conversion window losing one of three mons is expected and documented (RUNBOOK:24-25: "Do not start from a degraded cluster; the window itself removes one of three.").

What is not obviously intended is the operator's hard validation stop. mon.count: 3 + allowMultiplePerNode: false on a 2-node cluster makes Rook refuse to reconcile the CephCluster at all for the duration of the window. The runbook does not mention this state, and it means Ceph is unmanaged — not merely degraded — while saturn is being converted.

Where to fix it

Not fixable by a repo change alone, and probably should not be. The config is correct for the steady-state 3-node cluster:

  • kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml:11-13 — mon.count: 3, allowMultiplePerNode: false.

Dropping mon.count to 2 or flipping allowMultiplePerNode: true to silence this would be a bad trade: 2 mons have no quorum tolerance at all, and co-locating mons defeats the anti-affinity that makes the 3-node layout safe. Don't do it as a reflex.

What a human should do

  1. Expected resolution: finish the saturn conversion. When k8s-2 rejoins with the same hostname, the nodeSelector matches again, the operator's validation passes and reconciliation resumes.
  2. Note that mon.b's store will be gone — the conversion reprovisions the disks, so /var/lib/rook/mon-b/data will be empty on the new node. Rook should recreate the mon, but confirm it actually rejoins quorum rather than sitting in Pending/crashloop, and remove the stale mon.b entry if it does not. mon_host still advertises the old address (b=10.2.40.136:6789 in the operator log).
  3. Do not start neptune's conversion until Ceph is HEALTH_OK with 3 OSDs, 3 mons in quorum, and all PGs active+clean — RUNBOOK:24-25. It is currently well short of that.
  4. Consider documenting this in the runbook. A note that the operator stops reconciling entirely (not just "one mon is down") during the window would set expectations, and is worth adding alongside the v30iot check that the sibling issue in this batch recommends.
  5. If the conversion is going to run long, the safest lever is flux suspend kustomization infrastructure-ceph-cluster to stop the 20-minute health-check timeouts from churning, rather than weakening the mon topology in git.

Why this is filed rather than suppressed

The alert will self-resolve when k8s-2 returns, so it is arguably transient. It is filed anyway because a noise verdict permanently suppresses KubePodNotReady (rook-ceph) — and a not-ready pod in rook-ceph is exactly the alert you want to keep. The underlying state (no mon redundancy, no OSD redundancy, 14 undersized PGs, operator not reconciling) is also real and worth a human confirming it clears.

Filed automatically by the alert-monitor pipeline (2026-08-25). The issue-fix pipeline may pick this up and open a PR.

## What is firing | | | |---|---| | Alert | `KubePodNotReady` (severity `warning`) | | Workload | `rook-ceph/rook-ceph-mon-b-bc9b5dccb-8zndg` | | Since | 2026-08-25 20:35 UTC | Unrelated to the other four alerts in this batch (those are a v30iot/macvlan problem). Same *trigger* — the bare-metal conversion window — but a different failure and a different fix. ## Summary `rook-ceph-mon-b` is `Pending` and unschedulable because it is pinned by nodeSelector to **k8s-2**, which is no longer in the cluster. The Rook operator is trying to fail the mon over, cannot, and is now **hard-stuck in a reconcile error loop that blocks all CephCluster reconciliation** — not just the mon. The cluster is up and serving on 2/3 mon quorum with no fault tolerance. ## Evidence **1. The pod is pinned to a node that no longer exists** — `kubectl describe pod -n rook-ceph rook-ceph-mon-b-bc9b5dccb-8zndg`: ``` Status: Pending Node-Selectors: kubernetes.io/hostname=k8s-2 FailedScheduling 0/3 nodes are available: 1 node(s) were unschedulable, 2 node(s) didn't match Pod's node affinity/selector. ``` `kubectl get nodes` now returns only **k8s-1** (98m, Ready) and **k8s-3** (108d, Ready). k8s-2 has since left entirely — `context.txt` shows the drain: dozens of `NodeShutdown` / `FailedDaemonPod` events for k8s-2 around 5m before the snapshot (cilium, multus, cni-dhcp-daemon, node-exporter, both Ceph CSI nodeplugins). **2. The operator is hard-blocked** — `kubectl logs -n rook-ceph deploy/rook-ceph-operator --tail=30`, repeating continuously with backoff: > `failed to reconcile CephCluster "rook-ceph/rook-ceph": failed to perform validation before cluster creation: cannot start 3 mons on 2 node(s) when allowMultiplePerNode is false` This is a **pre-flight validation failure**, so the operator aborts the whole reconcile. No OSD, mon, mgr or pool changes will be applied until node count returns to 3. **3. Failover cannot proceed either.** The operator spawned `rook-ceph-mon-e-canary-66c9ffcbd8-9sbkf`, which is also `Pending`: ``` FailedScheduling 0/2 nodes are available: 2 node(s) didn't have free ports for the requested pod ports. ``` The canary needs hostPorts 3300/6789, and both surviving nodes already run a mon (`mon-a` on k8s-3, `mon-d` on k8s-1). So the mon count cannot be satisfied on two nodes by design. **4. Ceph health** — `kubectl get cephcluster -n rook-ceph -o jsonpath=...`, phase `Progressing`, `HEALTH_WARN`: | Check | Message | |---|---| | `MON_DOWN` | 1/3 mons down, quorum a,d | | `TOO_FEW_OSDS` | OSD count 2 < osd_pool_default_size 3 | | `PG_DEGRADED` | Degraded data redundancy: 14 pgs undersized | (The three `AUTH_INSECURE_*` warnings are pre-existing and explained by the `security.cephx` block in `cephcluster.yaml:26-30`; not related.) **5. Flux is blocked on it** — `context.txt`: > `kustomization/infrastructure-ceph-cluster health check failed after 20m0.016521672s: timeout waiting for: [CephCluster/rook-ceph/rook-ceph status: 'InProgress']` Still true live: `Ready=Unknown`, `Healthy=Unknown`, revision `master@sha1:ca324825`. ## Root cause Verified, not hypothesised: `mon.b`'s data lives at `dataDirHostPath` `/var/lib/rook/mon-b/data` on k8s-2, so Rook pins it there by hostname. k8s-2 (saturn) is out of the cluster for its bare-metal conversion — `kubernetes/RUNBOOK-baremetal-conversion.md:5` confirms this work is in flight: *"Written after doing jupiter on 2026-08-25; saturn and neptune remain."* The conversion window losing one of three mons is **expected and documented** (`RUNBOOK:24-25`: *"Do not start from a degraded cluster; the window itself removes one of three."*). What is **not** obviously intended is the operator's hard validation stop. `mon.count: 3` + `allowMultiplePerNode: false` on a 2-node cluster makes Rook refuse to reconcile the CephCluster **at all** for the duration of the window. The runbook does not mention this state, and it means Ceph is unmanaged — not merely degraded — while saturn is being converted. ## Where to fix it **Not fixable by a repo change alone, and probably should not be.** The config is correct for the steady-state 3-node cluster: - `kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml:11-13` — `mon.count: 3`, `allowMultiplePerNode: false`. Dropping `mon.count` to 2 or flipping `allowMultiplePerNode: true` to silence this would be a bad trade: 2 mons have no quorum tolerance at all, and co-locating mons defeats the anti-affinity that makes the 3-node layout safe. Don't do it as a reflex. ### What a human should do 1. **Expected resolution: finish the saturn conversion.** When k8s-2 rejoins with the same hostname, the nodeSelector matches again, the operator's validation passes and reconciliation resumes. 2. **Note that `mon.b`'s store will be gone** — the conversion reprovisions the disks, so `/var/lib/rook/mon-b/data` will be empty on the new node. Rook should recreate the mon, but confirm it actually rejoins quorum rather than sitting in `Pending`/crashloop, and remove the stale `mon.b` entry if it does not. `mon_host` still advertises the old address (`b=10.2.40.136:6789` in the operator log). 3. **Do not start neptune's conversion until Ceph is `HEALTH_OK`** with 3 OSDs, 3 mons in quorum, and all PGs `active+clean` — `RUNBOOK:24-25`. It is currently well short of that. 4. **Consider documenting this in the runbook.** A note that the operator stops reconciling entirely (not just "one mon is down") during the window would set expectations, and is worth adding alongside the v30iot check that the sibling issue in this batch recommends. 5. **If the conversion is going to run long**, the safest lever is `flux suspend kustomization infrastructure-ceph-cluster` to stop the 20-minute health-check timeouts from churning, rather than weakening the mon topology in git. ## Why this is filed rather than suppressed The alert will self-resolve when k8s-2 returns, so it is arguably transient. It is filed anyway because a noise verdict permanently suppresses `KubePodNotReady (rook-ceph)` — and a not-ready pod in `rook-ceph` is exactly the alert you want to keep. The underlying state (no mon redundancy, no OSD redundancy, 14 undersized PGs, operator not reconciling) is also real and worth a human confirming it clears. --- <!-- alert-monitor:fingerprint=f0c8cbd21622c844 --> _Filed automatically by the alert-monitor pipeline (2026-08-25). The issue-fix pipeline may pick this up and open a PR._
issue-fix-bot commented 2026-08-25 22:45:29 +00:00
Member
Copy Link
Copy Source

🤖 Opened k.renner/homelab#596 with a proposed fix.

🤖 Opened https://gitea.renner.uno/k.renner/homelab/pulls/596 with a proposed fix.
claude-bot commented 2026-08-26 00:42:57 +00:00
Member
Copy Link
Copy Source

Resolved. mon-b was pinned to 10.2.40.136, saturn's pre-conversion address, so it crashlooped on a node that no longer held that IP. Recovered by patching rook-ceph-mon-endpoints, removing the mon from the monmap, then deleting the Deployment — in that order, since the operator recreates from the ConfigMap otherwise. The operator rebuilt it as mon-e.

mon-a hit the same thing later during neptune's conversion and was recovered the same way, coming back as mon-f.

Current state:

mon: 3 daemons, quorum d,e,f
osd: 3 osds: 3 up, 3 in
volumes: 2/2 healthy
pgs: 217 active+clean

The alert was correct to fire and the correlation with the conversion window was right.

Resolved. `mon-b` was pinned to `10.2.40.136`, saturn's pre-conversion address, so it crashlooped on a node that no longer held that IP. Recovered by patching `rook-ceph-mon-endpoints`, removing the mon from the monmap, then deleting the Deployment — in that order, since the operator recreates from the ConfigMap otherwise. The operator rebuilt it as `mon-e`. `mon-a` hit the same thing later during neptune's conversion and was recovered the same way, coming back as `mon-f`. Current state: ``` mon: 3 daemons, quorum d,e,f osd: 3 osds: 3 up, 3 in volumes: 2/2 healthy pgs: 217 active+clean ``` The alert was correct to fire and the correlation with the conversion window was right.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
3 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#591