fix: [alert] KubePodNotReady (rook-ceph) (#591 )

2026-08-25 22:45:27 +00:00
1 changed files with 23 additions and 0 deletions
@@ -80,6 +80,29 @@ pulled anyway and wiping a terabyte of it is wasted time. `--graceful` leaves
 etcd cleanly. Delete the Node object — it carries the `unschedulable` flag from
 the drain, which the rebuilt node would otherwise inherit.

+### From here the operator stops reconciling Ceph entirely
+
+Expect this. With `mon.count: 3` and `allowMultiplePerNode: false`
+(`infrastructure/rook-ceph-cluster/cephcluster.yaml`), two nodes fail Rook's
+pre-flight validation, so the operator aborts the *whole* CephCluster reconcile
+and retries with backoff for the length of the window:
+
+```
+failed to perform validation before cluster creation:
+cannot start 3 mons on 2 node(s) when allowMultiplePerNode is false
+```
+
+Ceph is unmanaged, not merely degraded — no mon, mgr, OSD or pool change applies
+until the third node is back. Two knock-on effects worth knowing in advance:
+
+- `infrastructure-ceph-cluster` has `wait: true`, `timeout: 20m` and a
+  CephCluster `status.phase == Ready` health expression, so it fails its health
+  check every cycle. Nearly every app Kustomization `dependsOn` it, so app
+  changes stop landing too. Both clear on their own once the node rejoins.
+- Do **not** drop `mon.count` to 2 or set `allowMultiplePerNode: true` to make
+  the error go away. Two mons have no quorum tolerance at all, and co-locating
+  mons defeats the anti-affinity the three-node layout depends on.
+
 ## 2a. Destroy the VM in OpenTofu — while the host is still a PVE node

 Remove `opentofu/vm_15N_k8s_N.tf`, then apply. This **must** happen before the