fix: [alert] KubeDeploymentReplicasMismatch (rook-ceph) (#571) #574

2026-08-25T17:45:47Z

issue-fix-bot commented

2026-08-25 17:45:47 +00:00

What this PR does

rook-ceph-mon-c is 0/1 Pending because its node, k8s-1, is out for the
jupiter bare-metal conversion — that part is planned and is not a repo bug.
What is worth fixing is the secondary effect: Rook's mon health checker is
stuck in an un-completable failover loop. Since the issue was triaged, k8s-1
stopped being merely cordoned and is now gone from the cluster as a Node
object entirely, which flips the operator onto its "assigned node not found"
path — it now fails over immediately on every 45 s health-check tick instead
of after the 600 s out-of-quorum timeout the issue described. Each attempt
creates a rook-ceph-mon-d-canary Deployment and pod that cannot schedule
(mons are hostNetwork on 3300/6789 and allowMultiplePerNode: false, so with
k8s-1 gone there is no free port pair), gives up after ~2.5 min, deletes it,
and starts over. This PR sets spec.healthCheck.daemonHealth.mon.disabled: true on the CephCluster to park the checker for the duration of the window,
with a comment tying its revert to the two revert notes already in the repo.

Changes

kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml — add
healthCheck.daemonHealth.mon.disabled: true (4 lines of YAML; the rest of
the diff is the explanatory comment this file's conventions call for). Placed
directly under the mon: block so the revert sits next to count: 3. The
comment explicitly pairs the revert with the existing RE-ADD IT note under
storage.nodes and the RESTORE TO 1 note in
kubernetes/apps/immich/machine-learning.yaml:9, so all three go back
together when k8s-1 rejoins.

Verification

Everything below was observed live this run via ./.ci-diag/kubectl.

The issue's premise is stale — k8s-1 is deleted, not cordoned.
get nodes -o wide returns only k8s-2 and k8s-3, both Ready. The issue
quoted spec.unschedulable: true on a k8s-1 Node object; there is no k8s-1
Node object now. The scheduler messages show the transition — the older mon-c
event says 0/3 nodes are available: 1 node(s) were unschedulable, the newer
one says 0/2 nodes are available: 2 node(s) didn't match Pod's node affinity/selector.

The failover loop is real, and tighter than the issue reported.
logs deploy/rook-ceph-operator -n rook-ceph --tail=40:

17:38:31 W | op-mon: mon "c" NOT found in quorum and its assigned node "k8s-1" is not found, failing over immediately
17:40:56 E | op-mon: failed to schedule mon "d". failed to schedule canary pod(s)
17:40:59 E | op-mon: failed to failover mon "c". failed to place new mon on a node: failed to schedule mons
17:41:44 W | op-mon: mon "c" NOT found in quorum and its assigned node "k8s-1" is not found, failing over immediately

The 17:40:59 → 17:41:44 gap is 45 s, matching Rook's default mon health-check
interval — this is the loop the change stops. The issue's quoted
Retrying with a timeout of 600.00 seconds before failover no longer applies.

The canary cannot land. get pods -n rook-ceph -l app=rook-ceph-mon -o wide
showed a rook-ceph-mon-d-canary-6fbb8f4548-28422 that was 42 s old, next
to mon-a (k8s-3) and mon-b (k8s-2) both 1/1 Running for 3d21h.
get events -n rook-ceph --field-selector reason=FailedScheduling lists ~19
distinct rook-ceph-mon-d-canary-* pods, every one of them
0/2 nodes are available: 2 node(s) didn't have free ports for the requested pod ports.

The field is valid on this cluster.
explain cephcluster.spec.healthCheck.daemonHealth.mon returns
disabled <boolean> alongside interval and timeout, on Rook
docker.io/rook/ceph:v1.20.6. The live CephCluster already carries the
CRD-defaulted {"daemonHealth":{"mon":{},"osd":{},"status":{}}}.
apply --dry-run=client on the edited file (client-side parse only, no server
mutation) yields {"daemonHealth":{"mon":{"disabled":true},"osd":{}, "status":{}}} with spec.mon still {"allowMultiplePerNode":false,"count":3}.

Ceph state, for the record.
get cephcluster -n rook-ceph -o jsonpath='{.items[0].status.ceph}' →
MON_DOWN: 1/3 mons down, quorum a,b, plus PG_DEGRADED (12 pgs undersized)
and TOO_FEW_OSDS: OSD count 2 < osd_pool_default_size 3. Overall
HEALTH_WARN.

After Flux reconciles, expect: no more rook-ceph-mon-d-canary Deployment
appearing in get deploy -n rook-ceph, no new FailedScheduling events for
rook-ceph-mon-d-canary-*, and the operator log quiet of Failing over monitor "c". Nothing else should change.

What should NOT stop firing, deliberately: KubeDeploymentReplicasMismatch
on rook-ceph-mon-c and KubePodNotReady on its pod both keep firing, because
the deployment stays at 0/1 — and daemonHealth.status is untouched, so
MON_DOWN stays in CephCluster.status.ceph and CephHealthWarn keeps
alerting. This change removes the churn, not the signal.

Notes

This is not the fix the issue suggested. The issue's option 2 was
mon.count: 2. I did not take it: telling Rook two mons is the target makes
it remove mon-c from the monmap and delete its Deployment, which both
destroys mon-c's identity and makes KubeDeploymentReplicasMismatch stop
firing by deleting the object it watches — i.e. it silences the alert as a
side effect. Disabling the health checker stops only the un-completable
retry loop and leaves every alert and every Ceph health signal intact.
Fault tolerance is unchanged by this PR. Quorum is a,b — 2 of 3, no
redundancy — before and after. The operative advice from the issue still
stands and is not something a repo change can enforce: do not reboot or
drain k8s-2 or k8s-3 until k8s-1 rejoins, or quorum is lost and all
RBD/CephFS storage goes offline. The failover loop was never going to save
you there; with k8s-1 absent there is no node a replacement mon could use.
The revert is mandatory, and its order matters. If /var/lib/rook/mon-c
survives the reinstall, re-enabling the checker after k8s-1 rejoins is fine —
mon-c will already have rescheduled and rejoined. If the store did not
survive (the conversion reinstalls the node, so this is a real possibility —
the issue flagged it as an unverified hypothesis and I did not verify it
either, since it cannot be settled until the node is back), re-enable the
checker and let Rook fail mon-c over to a fresh mon-d. Leaving disabled: true in place past the window is the one genuinely bad outcome, since it
would also suppress failover for a real single-mon failure later.
I did not touch container image tags, thresholds, PrometheusRules, or
anything outside kubernetes/. The issue contained no instructions to act
on outside the edit scope, and no "Verified owner guidance" section.
Not verified this run: whether the immich/wordpress/woodpecker Insufficient cpu pressure and the k8s-3 ImageGCFailed mentioned in the issue's Impact
section are still present. They are separate from this alert and out of
scope here.

Closes #571

## What this PR does `rook-ceph-mon-c` is `0/1 Pending` because its node, k8s-1, is out for the jupiter bare-metal conversion — that part is planned and is not a repo bug. What *is* worth fixing is the secondary effect: Rook's mon health checker is stuck in an un-completable failover loop. Since the issue was triaged, k8s-1 stopped being merely cordoned and is now **gone from the cluster as a Node object entirely**, which flips the operator onto its "assigned node not found" path — it now fails over *immediately* on every 45 s health-check tick instead of after the 600 s out-of-quorum timeout the issue described. Each attempt creates a `rook-ceph-mon-d-canary` Deployment and pod that cannot schedule (mons are `hostNetwork` on 3300/6789 and `allowMultiplePerNode: false`, so with k8s-1 gone there is no free port pair), gives up after ~2.5 min, deletes it, and starts over. This PR sets `spec.healthCheck.daemonHealth.mon.disabled: true` on the CephCluster to park the checker for the duration of the window, with a comment tying its revert to the two revert notes already in the repo. ## Changes - `kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml` — add `healthCheck.daemonHealth.mon.disabled: true` (4 lines of YAML; the rest of the diff is the explanatory comment this file's conventions call for). Placed directly under the `mon:` block so the revert sits next to `count: 3`. The comment explicitly pairs the revert with the existing `RE-ADD IT` note under `storage.nodes` and the `RESTORE TO 1` note in `kubernetes/apps/immich/machine-learning.yaml:9`, so all three go back together when k8s-1 rejoins. ## Verification Everything below was observed live this run via `./.ci-diag/kubectl`. **The issue's premise is stale — k8s-1 is deleted, not cordoned.** `get nodes -o wide` returns only `k8s-2` and `k8s-3`, both `Ready`. The issue quoted `spec.unschedulable: true` on a k8s-1 Node object; there is no k8s-1 Node object now. The scheduler messages show the transition — the older mon-c event says `0/3 nodes are available: 1 node(s) were unschedulable`, the newer one says `0/2 nodes are available: 2 node(s) didn't match Pod's node affinity/selector`. **The failover loop is real, and tighter than the issue reported.** `logs deploy/rook-ceph-operator -n rook-ceph --tail=40`: ``` 17:38:31 W | op-mon: mon "c" NOT found in quorum and its assigned node "k8s-1" is not found, failing over immediately 17:40:56 E | op-mon: failed to schedule mon "d". failed to schedule canary pod(s) 17:40:59 E | op-mon: failed to failover mon "c". failed to place new mon on a node: failed to schedule mons 17:41:44 W | op-mon: mon "c" NOT found in quorum and its assigned node "k8s-1" is not found, failing over immediately ``` The 17:40:59 → 17:41:44 gap is 45 s, matching Rook's default mon health-check interval — this is the loop the change stops. The issue's quoted `Retrying with a timeout of 600.00 seconds before failover` no longer applies. **The canary cannot land.** `get pods -n rook-ceph -l app=rook-ceph-mon -o wide` showed a `rook-ceph-mon-d-canary-6fbb8f4548-28422` that was **42 s old**, next to `mon-a` (k8s-3) and `mon-b` (k8s-2) both `1/1 Running` for 3d21h. `get events -n rook-ceph --field-selector reason=FailedScheduling` lists ~19 distinct `rook-ceph-mon-d-canary-*` pods, every one of them `0/2 nodes are available: 2 node(s) didn't have free ports for the requested pod ports`. **The field is valid on this cluster.** `explain cephcluster.spec.healthCheck.daemonHealth.mon` returns `disabled <boolean>` alongside `interval` and `timeout`, on Rook `docker.io/rook/ceph:v1.20.6`. The live CephCluster already carries the CRD-defaulted `{"daemonHealth":{"mon":{},"osd":{},"status":{}}}`. `apply --dry-run=client` on the edited file (client-side parse only, no server mutation) yields `{"daemonHealth":{"mon":{"disabled":true},"osd":{}, "status":{}}}` with `spec.mon` still `{"allowMultiplePerNode":false,"count":3}`. **Ceph state, for the record.** `get cephcluster -n rook-ceph -o jsonpath='{.items[0].status.ceph}'` → `MON_DOWN: 1/3 mons down, quorum a,b`, plus `PG_DEGRADED` (12 pgs undersized) and `TOO_FEW_OSDS: OSD count 2 < osd_pool_default_size 3`. Overall `HEALTH_WARN`. **After Flux reconciles**, expect: no more `rook-ceph-mon-d-canary` Deployment appearing in `get deploy -n rook-ceph`, no new `FailedScheduling` events for `rook-ceph-mon-d-canary-*`, and the operator log quiet of `Failing over monitor "c"`. Nothing else should change. **What should NOT stop firing**, deliberately: `KubeDeploymentReplicasMismatch` on `rook-ceph-mon-c` and `KubePodNotReady` on its pod both keep firing, because the deployment stays at 0/1 — and `daemonHealth.status` is untouched, so `MON_DOWN` stays in `CephCluster.status.ceph` and `CephHealthWarn` keeps alerting. This change removes the churn, not the signal. ## Notes - **This is not the fix the issue suggested.** The issue's option 2 was `mon.count: 2`. I did not take it: telling Rook two mons is the target makes it *remove* mon-c from the monmap and delete its Deployment, which both destroys mon-c's identity and makes `KubeDeploymentReplicasMismatch` stop firing by deleting the object it watches — i.e. it silences the alert as a side effect. Disabling the health checker stops only the un-completable retry loop and leaves every alert and every Ceph health signal intact. - **Fault tolerance is unchanged by this PR.** Quorum is `a,b` — 2 of 3, no redundancy — before and after. The operative advice from the issue still stands and is not something a repo change can enforce: **do not reboot or drain k8s-2 or k8s-3 until k8s-1 rejoins**, or quorum is lost and all RBD/CephFS storage goes offline. The failover loop was never going to save you there; with k8s-1 absent there is no node a replacement mon could use. - **The revert is mandatory, and its order matters.** If `/var/lib/rook/mon-c` survives the reinstall, re-enabling the checker after k8s-1 rejoins is fine — mon-c will already have rescheduled and rejoined. If the store did *not* survive (the conversion reinstalls the node, so this is a real possibility — the issue flagged it as an unverified hypothesis and I did not verify it either, since it cannot be settled until the node is back), re-enable the checker and let Rook fail mon-c over to a fresh mon-d. Leaving `disabled: true` in place past the window is the one genuinely bad outcome, since it would also suppress failover for a real single-mon failure later. - I did not touch container image tags, thresholds, PrometheusRules, or anything outside `kubernetes/`. The issue contained no instructions to act on outside the edit scope, and no "Verified owner guidance" section. - Not verified this run: whether the immich/wordpress/woodpecker `Insufficient cpu` pressure and the k8s-3 `ImageGCFailed` mentioned in the issue's Impact section are still present. They are separate from this alert and out of scope here. Closes #571

issue-fix-bot added 1 commit 2026-08-25 17:45:47 +00:00

fix: [alert] KubeDeploymentReplicasMismatch (rook-ceph) (#571 )

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

86a685f56b

issue-fix-bot referenced this pull request

2026-08-25 17:45:47 +00:00

[alert] KubeDeploymentReplicasMismatch (rook-ceph) #571

review-bot reviewed

2026-08-25 17:49:17 +00:00

review-bot left a comment

🤖 Automated review — model CLI default (complexity: complex), commit 86a685f56bcc628a2fd13180ebffe34b88e9521d.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

Adds spec.healthCheck.daemonHealth.mon.disabled: true to the rook-ceph CephCluster CR to stop Rook's mon health-checker from endlessly (every 45 s) attempting an un-completable failover of mon-c now that its assigned node (k8s-1) has been removed from the cluster entirely. The change is a single 4-line YAML addition plus a long explanatory comment, following the same "TEMPORARY, revert once k8s-1 rejoins" convention already used for the RE-ADD IT note later in this same file and the RESTORE TO 1 note in kubernetes/apps/immich/machine-learning.yaml. No blocking issues found.

🟡 Suggestions

kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml:47-49 — healthCheck.daemonHealth.mon.disabled and its correct nesting under spec (not under mon:) can't be verified from the static checkout — there's no vendored CRD schema in the repo, and the CI kustomize-build check (.woodpecker/ci-manifests-test.yml) only validates that kustomizations build, not CRD field validity. The PR body documents kubectl explain and a client-side apply --dry-run against the live cluster confirming both the field and its effect, which is the right way to check this and I have no reason to doubt it, but it is worth a maintainer's quick double-check against the live rook-ceph:v1.20.6 CRD before merge since a typo'd/misnested field here would silently no-op rather than error.
The PR closes #571 on merge. Both this comment and the immich/machine-learning.yaml RESTORE TO 1 note rely purely on someone reading inline YAML comments during the eventual k8s-1-rejoins cleanup — closing the tracking issue that motivated the change removes one signal that the revert is still pending. This matches the repo's existing convention (the RE-ADD IT / RESTORE TO 1 notes aren't tied to open issues either), so it's not a new risk introduced by this PR, just worth flagging since two separate files now need a coordinated revert with no issue tracking it.

Tests

No automated test applies here — this is a GitOps config repo, and the repo's own CI (ci-manifests-test.yml) only runs kubectl kustomize builds offline, which will pass regardless of whether this CRD field is valid (see suggestion above). The PR body's live verification (mon failover log lines, FailedScheduling events, kubectl explain, apply --dry-run=client) is the appropriate substitute for a repo this shape, and it's unusually thorough and specific about what changes and what should not change (the alert itself keeps firing, MON_DOWN/CephHealthWarn keep firing) — that's the right invariant to call out for a change whose entire point is suppressing noise without suppressing signal.

Recommendation

Looks reasonable — no blocking issues; comments above are optional.

🤖 **Automated review** — model `CLI default` (complexity: complex), commit `86a685f56bcc628a2fd13180ebffe34b88e9521d`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary Adds `spec.healthCheck.daemonHealth.mon.disabled: true` to the `rook-ceph` `CephCluster` CR to stop Rook's mon health-checker from endlessly (every 45 s) attempting an un-completable failover of `mon-c` now that its assigned node (`k8s-1`) has been removed from the cluster entirely. The change is a single 4-line YAML addition plus a long explanatory comment, following the same "TEMPORARY, revert once k8s-1 rejoins" convention already used for the `RE-ADD IT` note later in this same file and the `RESTORE TO 1` note in `kubernetes/apps/immich/machine-learning.yaml`. No blocking issues found. ## 🟡 Suggestions - `kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml:47-49` — `healthCheck.daemonHealth.mon.disabled` and its correct nesting under `spec` (not under `mon:`) can't be verified from the static checkout — there's no vendored CRD schema in the repo, and the CI `kustomize-build` check (`.woodpecker/ci-manifests-test.yml`) only validates that kustomizations build, not CRD field validity. The PR body documents `kubectl explain` and a client-side `apply --dry-run` against the live cluster confirming both the field and its effect, which is the right way to check this and I have no reason to doubt it, but it is worth a maintainer's quick double-check against the live `rook-ceph:v1.20.6` CRD before merge since a typo'd/misnested field here would silently no-op rather than error. - The PR closes #571 on merge. Both this comment and the `immich/machine-learning.yaml` `RESTORE TO 1` note rely purely on someone reading inline YAML comments during the eventual k8s-1-rejoins cleanup — closing the tracking issue that motivated the change removes one signal that the revert is still pending. This matches the repo's existing convention (the `RE-ADD IT` / `RESTORE TO 1` notes aren't tied to open issues either), so it's not a new risk introduced by this PR, just worth flagging since two separate files now need a coordinated revert with no issue tracking it. ## Tests No automated test applies here — this is a GitOps config repo, and the repo's own CI (`ci-manifests-test.yml`) only runs `kubectl kustomize` builds offline, which will pass regardless of whether this CRD field is valid (see suggestion above). The PR body's live verification (mon failover log lines, `FailedScheduling` events, `kubectl explain`, `apply --dry-run=client`) is the appropriate substitute for a repo this shape, and it's unusually thorough and specific about what changes and what should *not* change (the alert itself keeps firing, `MON_DOWN`/`CephHealthWarn` keep firing) — that's the right invariant to call out for a change whose entire point is suppressing noise without suppressing signal. ## Recommendation Looks reasonable — no blocking issues; comments above are optional.

k.renner closed this pull request

2026-08-26 00:11:35 +00:00

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#574