konairius/homelab
10
0
Fork 0
Code Issues 11 Pull Requests 1 Packages Projects Releases 1 Wiki Activity

[alert] KubeDeploymentReplicasMismatch (rook-ceph) #571

New Issue
Closed
opened 2026-08-25 16:05:10 +00:00 by alert-monitor-bot · 1 comment
alert-monitor-bot commented 2026-08-25 16:05:10 +00:00
Member
Copy Link
Copy Source

What is firing

  • Alert: KubeDeploymentReplicasMismatch (severity warning), namespace rook-ceph, deployment rook-ceph-mon-c
  • Since: 2026-08-25 15:54:14Z (still active at time of triage)
  • Companion alert: KubePodNotReady for rook-ceph-mon-c-56b666fc59-tk648 — same root cause, filed as noise against this issue.

Evidence

context.txt — the only non-running pod in the cluster:

rook-ceph  rook-ceph-mon-c-56b666fc59-tk648  0/1  Pending  0  21m

./.ci-diag/kubectl get pod rook-ceph-mon-c-56b666fc59-tk648 -n rook-ceph -o yaml — the pod is pinned to the cordoned node:

nodeSelector:
  kubernetes.io/hostname: k8s-1

reason: Unschedulable
message: '0/3 nodes are available: 1 node(s) were unschedulable,
  2 node(s) didn''t match Pod''s node affinity/selector.'

./.ci-diag/kubectl get node k8s-1 -o yaml — k8s-1 was cordoned 15 minutes before the alert fired:

spec:
  unschedulable: true
  taints:
  - effect: NoSchedule
    key: node.kubernetes.io/unschedulable
    timeAdded: "2026-08-25T15:39:00Z"

./.ci-diag/kubectl get cephcluster -n rook-ceph -o yaml — Ceph is degraded but still serving:

MON_DOWN:   1/3 mons down, quorum a,b
health:     HEALTH_WARN

Root cause (verified)

k8s-1 (Proxmox host jupiter) is intentionally cordoned for the bare-metal
conversion documented in kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml
and kubernetes/apps/immich/machine-learning.yaml. Rook pins each mon to a
fixed node because its store lives on a hostPath (/var/lib/rook/mon-c/data),
so mon-c cannot move and stays Pending for the duration of the window.

The part that is not just "planned maintenance": Rook's automatic mon
failover has started and cannot succeed. From
./.ci-diag/kubectl logs deploy/rook-ceph-operator -n rook-ceph --tail=40:

15:50:13  mon "c" NOT found in quorum after timeout. Mon pod is not scheduled.
          Retrying with a timeout of 600.00 seconds before failover
16:01:32  mon "c" NOT found in quorum and timeout exceeded, mon will be failed over
16:01:32  Failing over monitor "c" ... starting new mon: rook-ceph-mon-d
16:01:32  created canary deployment rook-ceph-mon-d-canary

The replacement canary is itself unschedulable —
./.ci-diag/kubectl get events -n rook-ceph --field-selector reason=FailedScheduling:

rook-ceph-mon-d-canary-6fbb8f4548-wj649
  0/3 nodes are available: 1 node(s) were unschedulable,
  2 node(s) didn't have free ports for the requested pod ports.

Mons run with hostNetwork: true on ports 3300/6789, and mon-a/mon-b
already occupy those ports on k8s-3 and k8s-2
(./.ci-diag/kubectl get pods -n rook-ceph -l app=rook-ceph-mon -o wide).
With mon.count: 3 and allowMultiplePerNode: false, there is no third
place to put a mon while k8s-1 is out. The operator will therefore keep
cycling 600s failover attempts and leaving rook-ceph-mon-d-canary pods
Pending until k8s-1 returns.

Impact

Ceph holds quorum on a,b — 2 of 3, with no fault tolerance. If k8s-2 or
k8s-3 reboots or is drained during the jupiter window, quorum is lost and all
RBD/CephFS storage in the cluster goes offline. Note the window has already
made the remaining two nodes tight: context.txt shows repeated
Insufficient cpu FailedScheduling across wordpress, woodpecker,
immich, influxdb, paperless, trmnl, and ImageGCFailed on k8s-3 at
81% image-filesystem usage.

Where to fix it

This is not a repo bug — the node is down on purpose. The decisions a
human needs to make:

  1. Accept and ride it out. Do not reboot or drain k8s-2/k8s-3 until k8s-1
    rejoins. Nothing to change in the repo; close this issue with that note.
  2. Stop the failover churn (optional). Temporarily set mon.count: 2 in
    kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml:11-13, which
    tells Rook two mons is the expected state and stops the un-completable
    mon-d failover loop. This must be reverted to 3 when k8s-1 rejoins —
    pair it with the existing RE-ADD IT note at
    kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml:170-172 and
    the RESTORE TO 1 note in kubernetes/apps/immich/machine-learning.yaml:9
    so all three revert together. Trade-off: it removes redundancy from Ceph's
    own point of view and, on a homelab already running two OSDs
    (TOO_FEW_OSDS: OSD count 2 < osd_pool_default_size 3), may not be worth
    the extra revert step.

Hypothesis, not verified: that mon-c rejoins cleanly on its own once
k8s-1 is uncordoned and the mon store on /var/lib/rook/mon-c/data survives
the bare-metal conversion. The conversion reinstalls the node, so that data
may be lost and mon-c may need to be recreated from scratch. Settle it after
the window with ./.ci-diag/kubectl logs deploy/rook-ceph-operator -n rook-ceph --tail=50
and ./.ci-diag/kubectl get cephcluster -n rook-ceph -o jsonpath='{.items[0].status.ceph.health}'.

Verification once k8s-1 is back

./.ci-diag/kubectl get pods -n rook-ceph -l app=rook-ceph-mon -o wide
./.ci-diag/kubectl get cephcluster -n rook-ceph -o yaml

Expect three Running mons and MON_DOWN gone from status.ceph.details.

Filed automatically by the alert-monitor pipeline (2026-08-25). The issue-fix pipeline may pick this up and open a PR.

## What is firing - **Alert:** `KubeDeploymentReplicasMismatch` (severity `warning`), namespace `rook-ceph`, deployment `rook-ceph-mon-c` - **Since:** 2026-08-25 15:54:14Z (still active at time of triage) - **Companion alert:** `KubePodNotReady` for `rook-ceph-mon-c-56b666fc59-tk648` — same root cause, filed as noise against this issue. ## Evidence `context.txt` — the only non-running pod in the cluster: ``` rook-ceph rook-ceph-mon-c-56b666fc59-tk648 0/1 Pending 0 21m ``` `./.ci-diag/kubectl get pod rook-ceph-mon-c-56b666fc59-tk648 -n rook-ceph -o yaml` — the pod is pinned to the cordoned node: ``` nodeSelector: kubernetes.io/hostname: k8s-1 ``` ``` reason: Unschedulable message: '0/3 nodes are available: 1 node(s) were unschedulable, 2 node(s) didn''t match Pod''s node affinity/selector.' ``` `./.ci-diag/kubectl get node k8s-1 -o yaml` — k8s-1 was cordoned 15 minutes before the alert fired: ``` spec: unschedulable: true taints: - effect: NoSchedule key: node.kubernetes.io/unschedulable timeAdded: "2026-08-25T15:39:00Z" ``` `./.ci-diag/kubectl get cephcluster -n rook-ceph -o yaml` — Ceph is degraded but still serving: ``` MON_DOWN: 1/3 mons down, quorum a,b health: HEALTH_WARN ``` ## Root cause (verified) k8s-1 (Proxmox host `jupiter`) is intentionally cordoned for the bare-metal conversion documented in `kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml` and `kubernetes/apps/immich/machine-learning.yaml`. Rook pins each mon to a fixed node because its store lives on a hostPath (`/var/lib/rook/mon-c/data`), so mon-c cannot move and stays `Pending` for the duration of the window. **The part that is not just "planned maintenance": Rook's automatic mon failover has started and cannot succeed.** From `./.ci-diag/kubectl logs deploy/rook-ceph-operator -n rook-ceph --tail=40`: ``` 15:50:13 mon "c" NOT found in quorum after timeout. Mon pod is not scheduled. Retrying with a timeout of 600.00 seconds before failover 16:01:32 mon "c" NOT found in quorum and timeout exceeded, mon will be failed over 16:01:32 Failing over monitor "c" ... starting new mon: rook-ceph-mon-d 16:01:32 created canary deployment rook-ceph-mon-d-canary ``` The replacement canary is itself unschedulable — `./.ci-diag/kubectl get events -n rook-ceph --field-selector reason=FailedScheduling`: ``` rook-ceph-mon-d-canary-6fbb8f4548-wj649 0/3 nodes are available: 1 node(s) were unschedulable, 2 node(s) didn't have free ports for the requested pod ports. ``` Mons run with `hostNetwork: true` on ports 3300/6789, and `mon-a`/`mon-b` already occupy those ports on k8s-3 and k8s-2 (`./.ci-diag/kubectl get pods -n rook-ceph -l app=rook-ceph-mon -o wide`). With `mon.count: 3` and `allowMultiplePerNode: false`, there is no third place to put a mon while k8s-1 is out. The operator will therefore keep cycling 600s failover attempts and leaving `rook-ceph-mon-d-canary` pods Pending until k8s-1 returns. ## Impact Ceph holds quorum on `a,b` — **2 of 3, with no fault tolerance**. If k8s-2 or k8s-3 reboots or is drained during the jupiter window, quorum is lost and all RBD/CephFS storage in the cluster goes offline. Note the window has already made the remaining two nodes tight: `context.txt` shows repeated `Insufficient cpu` FailedScheduling across `wordpress`, `woodpecker`, `immich`, `influxdb`, `paperless`, `trmnl`, and `ImageGCFailed` on k8s-3 at 81% image-filesystem usage. ## Where to fix it This is **not a repo bug** — the node is down on purpose. The decisions a human needs to make: 1. **Accept and ride it out.** Do not reboot or drain k8s-2/k8s-3 until k8s-1 rejoins. Nothing to change in the repo; close this issue with that note. 2. **Stop the failover churn** (optional). Temporarily set `mon.count: 2` in `kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml:11-13`, which tells Rook two mons is the expected state and stops the un-completable mon-d failover loop. This must be reverted to `3` when k8s-1 rejoins — pair it with the existing `RE-ADD IT` note at `kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml:170-172` and the `RESTORE TO 1` note in `kubernetes/apps/immich/machine-learning.yaml:9` so all three revert together. Trade-off: it removes redundancy from Ceph's own point of view and, on a homelab already running two OSDs (`TOO_FEW_OSDS: OSD count 2 < osd_pool_default_size 3`), may not be worth the extra revert step. **Hypothesis, not verified:** that mon-c rejoins cleanly on its own once k8s-1 is uncordoned and the mon store on `/var/lib/rook/mon-c/data` survives the bare-metal conversion. The conversion reinstalls the node, so that data may be lost and mon-c may need to be recreated from scratch. Settle it after the window with `./.ci-diag/kubectl logs deploy/rook-ceph-operator -n rook-ceph --tail=50` and `./.ci-diag/kubectl get cephcluster -n rook-ceph -o jsonpath='{.items[0].status.ceph.health}'`. ## Verification once k8s-1 is back ``` ./.ci-diag/kubectl get pods -n rook-ceph -l app=rook-ceph-mon -o wide ./.ci-diag/kubectl get cephcluster -n rook-ceph -o yaml ``` Expect three Running mons and `MON_DOWN` gone from `status.ceph.details`. --- <!-- alert-monitor:fingerprint=84e1b1a66b117934 --> _Filed automatically by the alert-monitor pipeline (2026-08-25). The issue-fix pipeline may pick this up and open a PR._
issue-fix-bot commented 2026-08-25 17:45:47 +00:00
Member
Copy Link
Copy Source

🤖 Opened k.renner/homelab#574 with a proposed fix.

🤖 Opened https://gitea.renner.uno/k.renner/homelab/pulls/574 with a proposed fix.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#571