diff --git a/kubernetes/RUNBOOK-baremetal-conversion.md b/kubernetes/RUNBOOK-baremetal-conversion.md index 46d9adef..8192a63c 100644 --- a/kubernetes/RUNBOOK-baremetal-conversion.md +++ b/kubernetes/RUNBOOK-baremetal-conversion.md @@ -80,6 +80,29 @@ pulled anyway and wiping a terabyte of it is wasted time. `--graceful` leaves etcd cleanly. Delete the Node object — it carries the `unschedulable` flag from the drain, which the rebuilt node would otherwise inherit. +### From here the operator stops reconciling Ceph entirely + +Expect this. With `mon.count: 3` and `allowMultiplePerNode: false` +(`infrastructure/rook-ceph-cluster/cephcluster.yaml`), two nodes fail Rook's +pre-flight validation, so the operator aborts the *whole* CephCluster reconcile +and retries with backoff for the length of the window: + +``` +failed to perform validation before cluster creation: +cannot start 3 mons on 2 node(s) when allowMultiplePerNode is false +``` + +Ceph is unmanaged, not merely degraded — no mon, mgr, OSD or pool change applies +until the third node is back. Two knock-on effects worth knowing in advance: + +- `infrastructure-ceph-cluster` has `wait: true`, `timeout: 20m` and a + CephCluster `status.phase == Ready` health expression, so it fails its health + check every cycle. Nearly every app Kustomization `dependsOn` it, so app + changes stop landing too. Both clear on their own once the node rejoins. +- Do **not** drop `mon.count` to 2 or set `allowMultiplePerNode: true` to make + the error go away. Two mons have no quorum tolerance at all, and co-locating + mons defeats the anti-affinity the three-node layout depends on. + ## 2a. Destroy the VM in OpenTofu — while the host is still a PVE node Remove `opentofu/vm_15N_k8s_N.tf`, then apply. This **must** happen before the