fix: [alert] KubeDeploymentReplicasMismatch (rook-ceph) (#571 )

2026-08-25 17:45:46 +00:00
1 changed files with 37 additions and 0 deletions
@@ -11,6 +11,43 @@ spec:
  mon:
    count: 3
    allowMultiplePerNode: false
+  # TEMPORARY — SET BACK TO false (or drop this block) ONCE k8s-1 HAS REJOINED.
+  #
+  # Stops Rook's mon failover loop for the jupiter window. k8s-1 is no longer a
+  # Node object at all, so the health checker takes the "assigned node not
+  # found" path and fails over *immediately* instead of after the 600s
+  # out-of-quorum timeout:
+  #
+  #   op-mon: mon "c" NOT found in quorum and its assigned node "k8s-1" is not
+  #           found, failing over immediately
+  #
+  # The failover can never complete. Mons run hostNetwork on 3300/6789 and
+  # allowMultiplePerNode is false, so with k8s-1 gone the replacement's canary
+  # has nowhere to land ("2 node(s) didn't have free ports for the requested
+  # pod ports"). The operator gives up after ~2.5 min, deletes the canary, and
+  # the 45s health-check interval starts the next attempt. Each cycle churns a
+  # Deployment plus pod, rewrites the mon-endpoints ConfigMap and the ceph-csi
+  # connection/clientProfile CRs, and buries everything else in the operator log.
+  #
+  # This does NOT hide the degradation: daemonHealth.status is untouched, so
+  # MON_DOWN stays in CephCluster.status.ceph and CephHealthWarn still fires,
+  # and rook-ceph-mon-c stays 0/1 Pending so KubeDeploymentReplicasMismatch
+  # keeps firing. Nothing is given up, because there is no node the failover
+  # could have succeeded on anyway.
+  #
+  # It also keeps mon "c" intact for the end of the window: with the checker
+  # live, the moment k8s-1 rejoins the operator races to stand mon-d up there
+  # and drop "c" from the monmap. Off, mon-c reschedules onto k8s-1 and rejoins
+  # with its existing store under /var/lib/rook/mon-c. If that store did not
+  # survive the reinstall, re-enable the checker first — then failover is
+  # exactly what you want.
+  #
+  # Reverts together with the `RE-ADD IT` note under storage.nodes below and
+  # the `RESTORE TO 1` note in kubernetes/apps/immich/machine-learning.yaml.
+  healthCheck:
+    daemonHealth:
+      mon:
+        disabled: true
  mgr:
    count: 2
    allowMultiplePerNode: false