fix: [alert] KubePodNotReady (rook-ceph) (#591) #596

2026-08-25T22:45:28Z

issue-fix-bot commented

2026-08-25 22:45:28 +00:00

What this PR does

KubePodNotReady fired for rook-ceph/rook-ceph-mon-b-bc9b5dccb-8zndg because
mon.b was pinned by nodeSelector to k8s-2 (saturn), which had left the
cluster for its bare-metal conversion. That alert has since self-resolved —
k8s-2 rejoined, mon.b was replaced by mon.e, and Ceph is back to three mons
and three OSDs (verified live this run, see below). So there is nothing left to
fix about the alert itself, and no CephCluster change is made: the issue is
right that dropping mon.count to 2 or flipping allowMultiplePerNode would be
a bad trade, not a fix.

What this PR does address is the one genuinely repeatable finding underneath the
alert, which the issue calls out as recommendation #4: while the node is away,
the Rook operator does not merely lose a mon — it fails a pre-flight validation
and aborts all CephCluster reconciliation, and the runbook does not mention
this. RUNBOOK-baremetal-conversion.md:5 says "saturn and neptune remain", so
this window will happen again on neptune. The change adds a short subsection to
section 2 (the point at which the node count drops to 2) documenting the
verified behaviour, its two knock-on effects, and an explicit "do not weaken the
mon topology to silence it".

Changes

kubernetes/RUNBOOK-baremetal-conversion.md — new ### From here the operator stops reconciling Ceph entirely subsection at the end of section 2 (+23
lines, docs only). Records the verified operator error, that Ceph is unmanaged
rather than degraded for the window, that infrastructure-ceph-cluster fails
its health check every cycle and blocks its dependents, that both clear on
their own when the node rejoins, and that mon.count/allowMultiplePerNode
must not be changed to make the error go away.

Verification

Live checks run this run with ./.ci-diag/kubectl:

get nodes -o wide → k8s-2 is back, Ready, age 29m, 10.2.40.211. All
three control-plane nodes Ready. The issue's core premise ("k8s-2 is no
longer in the cluster") is stale.
get pods -n rook-ceph -o wide → mon-b and the mon-e-canary are gone;
rook-ceph-mon-a (k8s-3), mon-d (k8s-1), mon-e (k8s-2) all 1/1 Running;
osd-0 (k8s-2), osd-1 (k8s-3), osd-2 (k8s-1) all 1/1 Running.
get deploy -n rook-ceph -l app=rook-ceph-mon → exactly three mon deployments
(a/d/e), one per node, each readyReplicas: 1. No stale mon.b deployment,
so runbook step 8 was already carried out.
get cephcluster -n rook-ceph -o jsonpath=... → phase Ready. Health is
HEALTH_WARN but the only remaining checks are the three
AUTH_INSECURE_* ones, which the issue itself correctly identifies as
pre-existing (cephcluster.yaml:26-30). MON_DOWN, TOO_FEW_OSDS and
PG_DEGRADED have all cleared.
get kustomization -n flux-system -o wide → infrastructure-ceph-cluster is
Ready=True, Applied revision: master@sha1:44149d63…. No longer blocked.
Loki query for {namespace="rook-ceph",pod=~"rook-ceph-operator.*"} |= "allowMultiplePerNode" over 19:00–24:00 UTC → confirms the claim this PR
documents, repeating every ~10–17 min from 20:51:41 through 21:41:24 UTC:
failed to perform validation before cluster creation: cannot start 3 mons on 2 node(s) when allowMultiplePerNode is false.

Repo facts behind the new text, both read this run:
kubernetes/clusters/homelab/infrastructure.yaml:116-142 (timeout: 20m,
wait: true, CephCluster status.phase == 'Ready' health expr) and 15 app
Kustomizations under kubernetes/clusters/homelab/ with
dependsOn: infrastructure-ceph-cluster.

For the reviewer: this is a docs-only change, so nothing should change at
runtime and no alert state depends on merging it. KubePodNotReady for
rook-ceph-mon-b has already stopped firing. Confirm with the first two
commands above.

Notes

The issue's snapshot is stale, but its analysis was correct. Every
runtime claim it made (pod pinned to k8s-2, operator hard-blocked on the
validation, canary Pending on hostPort conflict, HEALTH_WARN with
MON_DOWN/TOO_FEW_OSDS/PG_DEGRADED) was accurate when written; the
conversion simply completed in the interim. I verified the operator-block
claim in Loki rather than take it on trust, since it is the one claim this PR
encodes into the runbook.
I did not adopt the issue's recommendation #5 (flux suspend kustomization infrastructure-ceph-cluster as a lever if the conversion runs long). Nearly
every app Kustomization dependsOn this one, and I could not verify from the
repo alone how Flux's dependency check treats a suspended dependency, so
recommending it in a runbook risked trading a cosmetic timeout for a
cluster-wide GitOps stall. The subsection instead states that the churn is
expected and self-clearing. Worth revisiting deliberately if the owner wants
it.
Recommendations #1–#3 (finish the conversion, confirm the mon rejoins, don't
start neptune until HEALTH_OK) needed no repo change: #1 and #2 are done,
and #3 is already in the runbook at lines 24-25. On #3, note that Ceph is
Ready but not yet HEALTH_OK — the three AUTH_INSECURE_* warnings are
pre-existing and intentional, so a human should judge that gate on mon quorum,
OSD count and PG state rather than on a bare HEALTH_WARN.
Section 8 of the runbook already covers a different mon failure mode (mon
crashlooping on a stale address after the node returns, including the
hostPort/canary detail). The new subsection covers the node-absent window
instead, which was not documented. I placed it in section 2 rather than
section 8 to keep the two distinct.
No instructions embedded in the issue asked for anything outside the edit
scope, and nothing outside kubernetes/ was touched.

Closes #591

## What this PR does `KubePodNotReady` fired for `rook-ceph/rook-ceph-mon-b-bc9b5dccb-8zndg` because `mon.b` was pinned by nodeSelector to **k8s-2** (saturn), which had left the cluster for its bare-metal conversion. **That alert has since self-resolved** — k8s-2 rejoined, `mon.b` was replaced by `mon.e`, and Ceph is back to three mons and three OSDs (verified live this run, see below). So there is nothing left to fix about the alert itself, and no CephCluster change is made: the issue is right that dropping `mon.count` to 2 or flipping `allowMultiplePerNode` would be a bad trade, not a fix. What this PR does address is the one genuinely repeatable finding underneath the alert, which the issue calls out as recommendation #4: while the node is away, the Rook operator does not merely lose a mon — it fails a pre-flight validation and **aborts all CephCluster reconciliation**, and the runbook does not mention this. `RUNBOOK-baremetal-conversion.md:5` says *"saturn and neptune remain"*, so this window will happen again on neptune. The change adds a short subsection to section 2 (the point at which the node count drops to 2) documenting the verified behaviour, its two knock-on effects, and an explicit "do not weaken the mon topology to silence it". ## Changes - `kubernetes/RUNBOOK-baremetal-conversion.md` — new `### From here the operator stops reconciling Ceph entirely` subsection at the end of section 2 (+23 lines, docs only). Records the verified operator error, that Ceph is unmanaged rather than degraded for the window, that `infrastructure-ceph-cluster` fails its health check every cycle and blocks its dependents, that both clear on their own when the node rejoins, and that `mon.count`/`allowMultiplePerNode` must not be changed to make the error go away. ## Verification Live checks run this run with `./.ci-diag/kubectl`: - `get nodes -o wide` → **k8s-2 is back**, `Ready`, age 29m, `10.2.40.211`. All three control-plane nodes `Ready`. The issue's core premise ("k8s-2 is no longer in the cluster") is stale. - `get pods -n rook-ceph -o wide` → `mon-b` and the `mon-e-canary` are gone; `rook-ceph-mon-a` (k8s-3), `mon-d` (k8s-1), `mon-e` (k8s-2) all `1/1 Running`; `osd-0` (k8s-2), `osd-1` (k8s-3), `osd-2` (k8s-1) all `1/1 Running`. - `get deploy -n rook-ceph -l app=rook-ceph-mon` → exactly three mon deployments (a/d/e), one per node, each `readyReplicas: 1`. No stale `mon.b` deployment, so runbook step 8 was already carried out. - `get cephcluster -n rook-ceph -o jsonpath=...` → phase `Ready`. Health is `HEALTH_WARN` but the only remaining checks are the three `AUTH_INSECURE_*` ones, which the issue itself correctly identifies as pre-existing (`cephcluster.yaml:26-30`). `MON_DOWN`, `TOO_FEW_OSDS` and `PG_DEGRADED` have all cleared. - `get kustomization -n flux-system -o wide` → `infrastructure-ceph-cluster` is `Ready=True`, `Applied revision: master@sha1:44149d63…`. No longer blocked. - Loki query for `{namespace="rook-ceph",pod=~"rook-ceph-operator.*"} |= "allowMultiplePerNode"` over 19:00–24:00 UTC → **confirms the claim this PR documents**, repeating every ~10–17 min from 20:51:41 through 21:41:24 UTC: `failed to perform validation before cluster creation: cannot start 3 mons on 2 node(s) when allowMultiplePerNode is false`. Repo facts behind the new text, both read this run: `kubernetes/clusters/homelab/infrastructure.yaml:116-142` (`timeout: 20m`, `wait: true`, CephCluster `status.phase == 'Ready'` health expr) and 15 app Kustomizations under `kubernetes/clusters/homelab/` with `dependsOn: infrastructure-ceph-cluster`. For the reviewer: this is a docs-only change, so nothing should change at runtime and no alert state depends on merging it. `KubePodNotReady` for `rook-ceph-mon-b` has already stopped firing. Confirm with the first two commands above. ## Notes - **The issue's snapshot is stale, but its analysis was correct.** Every runtime claim it made (pod pinned to k8s-2, operator hard-blocked on the validation, canary `Pending` on hostPort conflict, `HEALTH_WARN` with `MON_DOWN`/`TOO_FEW_OSDS`/`PG_DEGRADED`) was accurate when written; the conversion simply completed in the interim. I verified the operator-block claim in Loki rather than take it on trust, since it is the one claim this PR encodes into the runbook. - **I did not adopt the issue's recommendation #5** (`flux suspend kustomization infrastructure-ceph-cluster` as a lever if the conversion runs long). Nearly every app Kustomization `dependsOn` this one, and I could not verify from the repo alone how Flux's dependency check treats a suspended dependency, so recommending it in a runbook risked trading a cosmetic timeout for a cluster-wide GitOps stall. The subsection instead states that the churn is expected and self-clearing. Worth revisiting deliberately if the owner wants it. - Recommendations #1–#3 (finish the conversion, confirm the mon rejoins, don't start neptune until `HEALTH_OK`) needed no repo change: #1 and #2 are done, and #3 is already in the runbook at lines 24-25. On #3, note that Ceph is `Ready` but not yet `HEALTH_OK` — the three `AUTH_INSECURE_*` warnings are pre-existing and intentional, so a human should judge that gate on mon quorum, OSD count and PG state rather than on a bare `HEALTH_WARN`. - Section 8 of the runbook already covers a *different* mon failure mode (mon crashlooping on a stale address after the node returns, including the hostPort/canary detail). The new subsection covers the node-absent window instead, which was not documented. I placed it in section 2 rather than section 8 to keep the two distinct. - No instructions embedded in the issue asked for anything outside the edit scope, and nothing outside `kubernetes/` was touched. Closes #591

issue-fix-bot added 1 commit 2026-08-25 22:45:29 +00:00

fix: [alert] KubePodNotReady (rook-ceph) (#591 )

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

a20b4f34cd

issue-fix-bot referenced this pull request

2026-08-25 22:45:29 +00:00

[alert] KubePodNotReady (rook-ceph) #591

review-bot reviewed

2026-08-25 22:47:21 +00:00

review-bot left a comment

🤖 Automated review — model claude-haiku-4-5-20251001 (complexity: trivial), commit a20b4f34cdad8ef94b9f639e5b9e1636550ebd54.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

This PR adds documentation to the baremetal conversion runbook describing the expected behavior when a third node leaves the cluster during the conversion. The new subsection (added to section 2 of kubernetes/RUNBOOK-baremetal-conversion.md) documents the Rook operator's pre-flight validation failure, its knock-on effects on downstream kustomizations, and clear guidance against weakening the mon topology to silence the error. The change is documentation-only, technically accurate, and well-placed.

🟢 Nits

No issues identified. The documentation is clear, technically accurate, and well-reasoned:

The Rook pre-flight validation error message is accurately quoted and matches the operator's behavior.
References to infrastructure/rook-ceph-cluster/cephcluster.yaml are correct (mon.count: 3, allowMultiplePerNode: false verified at lines 12-13 of the actual file).
The claim about infrastructure-ceph-cluster kustomization settings is accurate: wait: true, timeout: 20m, and CephCluster health check expressions confirmed in kubernetes/clusters/homelab/infrastructure.yaml:118,124,136.
The statement that "nearly every app Kustomization dependsOn it" is accurate—verified in immich.yaml, wordpress.yaml, and apps.yaml all listing infrastructure-ceph-cluster as a dependency.
The reasoning for not adopting the flux suspend recommendation (potential GitOps stall risk) is sound.
Placement in section 2 is appropriate—describes what happens as node count drops to 2, before section 2a's VM destruction steps.

Tests

Not applicable—this is a documentation-only change.

Recommendation

Looks reasonable — no blocking issues; comments above are optional.

🤖 **Automated review** — model `claude-haiku-4-5-20251001` (complexity: trivial), commit `a20b4f34cdad8ef94b9f639e5b9e1636550ebd54`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary This PR adds documentation to the baremetal conversion runbook describing the expected behavior when a third node leaves the cluster during the conversion. The new subsection (added to section 2 of `kubernetes/RUNBOOK-baremetal-conversion.md`) documents the Rook operator's pre-flight validation failure, its knock-on effects on downstream kustomizations, and clear guidance against weakening the mon topology to silence the error. The change is documentation-only, technically accurate, and well-placed. ## 🟢 Nits No issues identified. The documentation is clear, technically accurate, and well-reasoned: - The Rook pre-flight validation error message is accurately quoted and matches the operator's behavior. - References to `infrastructure/rook-ceph-cluster/cephcluster.yaml` are correct (mon.count: 3, allowMultiplePerNode: false verified at lines 12-13 of the actual file). - The claim about `infrastructure-ceph-cluster` kustomization settings is accurate: `wait: true`, `timeout: 20m`, and CephCluster health check expressions confirmed in `kubernetes/clusters/homelab/infrastructure.yaml:118,124,136`. - The statement that "nearly every app Kustomization dependsOn it" is accurate—verified in immich.yaml, wordpress.yaml, and apps.yaml all listing `infrastructure-ceph-cluster` as a dependency. - The reasoning for not adopting the flux suspend recommendation (potential GitOps stall risk) is sound. - Placement in section 2 is appropriate—describes what happens as node count drops to 2, before section 2a's VM destruction steps. ## Tests Not applicable—this is a documentation-only change. ## Recommendation Looks reasonable — no blocking issues; comments above are optional.

k.renner closed this pull request

2026-08-26 04:39:00 +00:00

claude-bot commented

2026-08-26 04:41:43 +00:00

Keeping this one open — recommending merge rather than closing it with the other two alert-fix PRs.

Its sibling #606 was superseded by #611 and #594 reached a conclusion the outcome contradicted, but this one is different: it is pure documentation of real operator behaviour that I did not know while running the conversion, and it is still true.

Specifically valuable:

Rook aborts the whole CephCluster reconcile when mon.count: 3 cannot be satisfied with allowMultiplePerNode: false on two nodes — Ceph is unmanaged rather than merely degraded for the length of the window. I saw the mons fail but did not realise no mon, mgr, OSD or pool change would apply at all.
The knock-on through infrastructure-ceph-cluster having wait: true and a status.phase == Ready health expression, so nearly every app Kustomization that dependsOn it stops landing changes too.
The explicit warning not to drop mon.count to 2 or set allowMultiplePerNode: true to silence the error. That is exactly the tempting wrong move at 2am, and worth having written down before someone reaches for it.

All three conversions are done, so this is retrospective for this round — but the runbook is the artefact that survives, and this is the kind of "expect this, do not panic, do not "fix" it" content that makes it useful next time.

One factual note: the mon failures in #591 were a separate issue from the reconcile stall described here — mon-b and later mon-a were pinned to their pre-conversion node addresses and crashlooped, recovered by patching rook-ceph-mon-endpoints, removing from the monmap, then deleting the Deployment, in that order. They came back as mon-e and mon-f. Worth adding alongside this if you are touching the section anyway, since the recovery order is not obvious and I had to get it wrong once to find it.

Keeping this one open — recommending merge rather than closing it with the other two alert-fix PRs. Its sibling #606 was superseded by #611 and #594 reached a conclusion the outcome contradicted, but this one is different: it is pure documentation of real operator behaviour that I did not know while running the conversion, and it is still true. Specifically valuable: - Rook aborts the **whole** CephCluster reconcile when `mon.count: 3` cannot be satisfied with `allowMultiplePerNode: false` on two nodes — Ceph is unmanaged rather than merely degraded for the length of the window. I saw the mons fail but did not realise no mon, mgr, OSD or pool change would apply at all. - The knock-on through `infrastructure-ceph-cluster` having `wait: true` and a `status.phase == Ready` health expression, so nearly every app Kustomization that `dependsOn` it stops landing changes too. - The explicit warning not to drop `mon.count` to 2 or set `allowMultiplePerNode: true` to silence the error. That is exactly the tempting wrong move at 2am, and worth having written down before someone reaches for it. All three conversions are done, so this is retrospective for this round — but the runbook is the artefact that survives, and this is the kind of "expect this, do not panic, do not "fix" it" content that makes it useful next time. One factual note: the mon failures in #591 were a separate issue from the reconcile stall described here — `mon-b` and later `mon-a` were pinned to their pre-conversion node addresses and crashlooped, recovered by patching `rook-ceph-mon-endpoints`, removing from the monmap, then deleting the Deployment, in that order. They came back as `mon-e` and `mon-f`. Worth adding alongside this if you are touching the section anyway, since the recovery order is not obvious and I had to get it wrong once to find it.

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

3 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#596