fix: [alert] KubeStatefulSetReplicasMismatch (openbao) (#645)

Reviewed-on: k.renner/homelab#646

kubernetes/MIGRATION-sops-openbao.md

@@ -51,7 +51,11 @@ kubectl -n openbao exec -it openbao-0 -- bao operator unseal   # paste key
 ```
 
 **Manual unseal is required after every pod restart** (node drain, upgrade,
-crash): `kubectl -n openbao exec -it openbao-0 -- bao operator unseal`.
+crash), **per replica** — the server runs three raft peers (see
+`infrastructure/openbao/helmrelease.yaml`), and a sealed one is neither a
+voter nor a standby, so it stays out of the cluster until unsealed:
+`kubectl -n openbao exec -it openbao-<N> -- bao operator unseal`, same key for
+all three. Three peers mean one sealed replica is survivable; two are not.
 Accepted for now; auto-unseal needs a second trust root (transit/KMS) and is
 explicitly out of scope. Symptom of sealed: ESO stores go NotReady, synced
 Secrets keep their last value (they are real k8s Secrets — pods keep running).

kubernetes/RUNBOOK-cold-rebuild.md

@@ -149,8 +149,8 @@ Secrets, and every secret-consuming app stays Pending. This step is **manual**
 (the unseal key lives in `pass`, off-cluster by design).
 
 ```sh
-# 1. Wait for the OpenBao pod to exist (sealed, 0/1 Ready is expected).
-kubectl -n openbao get pod openbao-0
+# 1. Wait for the OpenBao pods to exist (all three sealed, 0/1 Ready is expected).
+kubectl -n openbao get pods
 
 # 2. Restore the most recent raft snapshot. Source it from the snapshot PVC
 #    (infrastructure/openbao/snapshot-cronjob.yaml) or, in a total-loss rebuild,
@@ -158,8 +158,12 @@ kubectl -n openbao get pod openbao-0
 kubectl -n openbao cp <snapshot>.snap openbao-0:/tmp/restore.snap
 kubectl -n openbao exec -it openbao-0 -- bao operator raft snapshot restore /tmp/restore.snap
 
-# 3. Unseal with the key from pass (threshold 1).
+# 3. Unseal with the key from pass (threshold 1) — EVERY replica, same key.
+#    A sealed server is neither a raft voter nor a standby, so unsealing only
+#    openbao-0 leaves the three-peer cluster one node short of quorum.
 kubectl -n openbao exec -it openbao-0 -- bao operator unseal   # paste homelab/openbao/unseal-key
+kubectl -n openbao exec -it openbao-1 -- bao operator unseal   # same key
+kubectl -n openbao exec -it openbao-2 -- bao operator unseal   # same key
 ```
 
 Verify the store recovers before moving on:
@@ -170,7 +174,9 @@ kubectl -n <ns> get externalsecret            # want: SecretSynced
 ```
 
 > **Manual unseal is required after every OpenBao pod restart**, not just
-> during rebuilds. Details + rationale: `MIGRATION-sops-openbao.md`.
+> during rebuilds, and for *each* replica that restarts — a rolling node
+> drain seals all three in turn. Details + rationale:
+> `MIGRATION-sops-openbao.md`.
 
 ## Step 5 — apps: let the rest reconcile
 

kubernetes/infrastructure/openbao/helmrelease.yaml

@@ -42,12 +42,28 @@ spec:
       # the agent-injector mutating webhook is unused surface.
       enabled: false
     server:
-      # Single-node raft (not standalone/file) so we get `bao operator raft
-      # snapshot` as a consistent backup primitive. TLS terminates at the
-      # Traefik ingress; in-cluster traffic is plain HTTP.
+      # Raft (not standalone/file) for two reasons: `bao operator raft
+      # snapshot` as a consistent backup primitive, and real HA across three
+      # nodes. TLS terminates at the Traefik ingress; in-cluster traffic is
+      # plain HTTP.
       ha:
         enabled: true
-        replicas: 1
+        # Three, not two. Raft needs a quorum, and the quorum of two is two —
+        # a two-replica cluster tolerates no failures at all and would stop
+        # the moment one pod is sealed or rescheduled, which is worse than
+        # the single replica it replaced. Three tolerates one.
+        #
+        # What this buys, and what it does not: a node drain seals the replica
+        # on it, and raft elects a new leader from the remaining standbys
+        # within seconds, so the service does not go down. It does NOT remove
+        # the manual unseal — a sealed server can neither serve requests nor
+        # act as standby, so that replica stays out until someone unseals it.
+        #
+        # The consequence for rolling node maintenance is worth stating: a
+        # Talos upgrade drains all three nodes in turn, sealing each replica
+        # as it goes. Unseal each one as it comes back, or the third drain
+        # takes the last unsealed replica with it and quorum is lost anyway.
+        replicas: 3
         raft:
           enabled: true
           setNodeId: true
@@ -65,8 +81,32 @@ spec:
               cluster_address = "[::]:8201"
             }
 
+            # retry_join is what makes a replica join the existing raft
+            # cluster instead of coming up as an isolated, uninitialized
+            # node. Without it, scaling the StatefulSet produces pods that
+            # never become peers and would each need a manual
+            # `bao operator raft join`.
+            #
+            # All three peers are listed on every node, including the node
+            # itself. That is deliberate: the stanza is a list of possible
+            # leaders, a node ignores its own entry, and listing all three
+            # means the set is identical on every replica and stays correct
+            # regardless of which pod currently leads.
+            #
+            # Addressed through the headless service, so each entry resolves
+            # to one specific pod rather than load-balancing across them.
             storage "raft" {
               path = "/openbao/data"
+
+              retry_join {
+                leader_api_addr = "http://openbao-0.openbao-internal:8200"
+              }
+              retry_join {
+                leader_api_addr = "http://openbao-1.openbao-internal:8200"
+              }
+              retry_join {
+                leader_api_addr = "http://openbao-2.openbao-internal:8200"
+              }
             }
 
             service_registration "kubernetes" {}

kubernetes/infrastructure/openbao/snapshot-cronjob.yaml

@@ -53,7 +53,13 @@ spec:
               command: ["/bin/sh", "-ec"]
               args:
                 - |
-                  export BAO_ADDR=http://openbao.openbao.svc.cluster.local:8200
+                  # openbao-active, not openbao: with more than one replica
+                  # the plain service selects any pod, and a raft snapshot has
+                  # to be taken against the leader. Standbys forward ordinary
+                  # requests, but this is not one to leave to chance — the
+                  # active service selects the leader by label, so the
+                  # snapshot is always taken where it is consistent.
+                  export BAO_ADDR=http://openbao-active.openbao.svc.cluster.local:8200
                   BAO_TOKEN=$(bao write -field=token auth/kubernetes/login \
                     role=openbao-snapshot \
                     jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)")

kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml

@@ -8,6 +8,17 @@ spec:
     image: quay.io/ceph/ceph:v19.2.6 # {"$imagepolicy": "flux-system:ceph"}
     allowUnsupported: false
   dataDirHostPath: /var/lib/rook
+  # Do not start restarting OSDs while PGs are still recovering. The default
+  # is false, so Rook would proceed regardless — and paired with the old
+  # osdMaxUpdatesInParallel default that means it can take the next OSD down
+  # while the previous one's data is still being caught up.
+  #
+  # With three OSDs and size 3 there is no spare copy to fall back on: the
+  # gate between one OSD returning and the next going down is the only thing
+  # that keeps a rolling change from becoming an outage. See
+  # storage.osdMaxUpdatesInParallel below, which bounds the same operation
+  # from the other side.
+  upgradeOSDRequiresHealthyPGs: true
   mon:
     count: 3
     allowMultiplePerNode: false
@@ -147,19 +158,65 @@ spec:
     ssl: false
   monitoring:
     enabled: true
-  # Dedicated Ceph cluster network on VLAN 49 / 10.99.0.0/29, carried by an
-  # X710 SFP+ port PCI-passed-through to each Talos K8s VM (see
-  # talos/patches/k8s-{1,2,3}.yaml and opentofu/vm_15{0,1,2}_k8s_*.tf).
-  # `addressRanges` requires `provider: host` — Rook v1.18 explicitly notes
-  # the default (CNI) provider ignores addressRanges. Public range matches
-  # the existing v40srv subnet that the controlplane VMs already live on.
+  # Dedicated Ceph cluster network on VLAN 49, carried by the second X710
+  # SFP+ port (enp2s0f1np1) on each node, attached to the OSDs through Multus.
+  #
+  # WHY NOT provider: host, WHICH ALSO REACHES THAT LINK
+  #
+  # Because it reaches it for every daemon, not just the ones that need it.
+  # The cluster network is an OSD concern: per the Ceph documentation it
+  # carries OSD heartbeat, replication and recovery, and "Ceph Monitors always
+  # operate on the public network". `addressRanges` is only honoured under
+  # provider: host, so using it put the mons on host networking as a side
+  # effect — making a mon's address the node's DHCP lease.
+  #
+  # That is not theoretical. When a lease moved, the mon could no longer bind
+  # ("Cannot assign requested address") and crash-looped. Rook detected it and
+  # started a failover, but with three mons on three nodes each holding the
+  # mon host ports, the replacement could not be scheduled anywhere — the
+  # canary sat Pending on "didn't have free ports". Two of three nodes hit
+  # this during the Talos v1.12.11 upgrade and both needed hand intervention.
+  #
+  # Only `cluster` is selected here. With `public` left unset Rook puts client
+  # traffic on the pod network, so mons, MDS, MGR and the CSI pods keep
+  # Rook-managed identities and hold no host ports — which is what removes
+  # that failure mode rather than working around it.
+  #
+  # ISOLATION IS PRESERVED
+  #
+  # Rook's Multus prerequisites about hosts routing to the Multus network
+  # apply to a Multus *public* network. Selecting only `cluster` means they do
+  # not apply, so VLAN 49 stays what it is: Ceph-internal traffic and nothing
+  # else, no router, no other participants. After this settles, the node
+  # addresses on that segment (10.99.0.1-.3) have no remaining purpose and
+  # should come out of the Talos configs — at which point the attachment's
+  # pool can take the whole /29.
+  #
+  # Reached from provider: "" rather than directly from host, which Rook's CRD
+  # rejects. See the phase A commit.
+  #
+  # ONE-TIME STEP THAT THIS FILE CANNOT EXPRESS, recorded because it is not
+  # reproducible from the manifests alone. Removing addressRanges here does
+  # not clear public_network and cluster_network from Ceph's own config
+  # database — the same trap the bluestore comment above describes: `ceph
+  # config` retains a value that is only deleted from this file. During the
+  # phase A rollout the first OSD came up on the pod network, still had
+  # public_network 10.2.40.0/24 in the config database, found no interface in
+  # it and crash-looped on "Failed to pick public address". Both keys were
+  # removed with `ceph config rm global public_network` and the same for
+  # cluster_network; Rook then wrote cluster_network afresh from the
+  # attachment below. Anyone rebuilding this cluster from scratch will not hit
+  # it — the config database starts empty — but anyone repeating a provider
+  # change on a live cluster will.
+  #
+  # Bounded by upgradeOSDRequiresHealthyPGs and storage.osdMaxUpdatesInParallel
+  # above: one OSD restarts at a time and only once PGs are clean again.
+  #
+  # The attachment itself is san-network-attachment.yaml.
   network:
-    provider: host
-    addressRanges:
-      public:
-        - "10.2.40.0/24"
-      cluster:
-        - "10.99.0.0/29"
+    provider: multus
+    selectors:
+      cluster: rook-ceph/ceph-san
   # Nodes are listed explicitly rather than useAllNodes, so that a node can be
   # taken out of the storage fleet by removing it here. Under useAllNodes the
   # operator re-provisions an OSD on any node whose device still carries a
@@ -193,6 +250,15 @@ spec:
   #   2326E6E8E67D      Crucial   — Talos EPHEMERAL (/var)
   #   S437NF0M501176    PM983     — this OSD
   storage:
+    # One OSD restarted at a time. The default is 20, which on a three-OSD
+    # cluster means all of them at once — every PG inactive, every pool
+    # unreachable, for as long as the restart takes. The default suits a
+    # cluster large enough that 20 OSDs is a small fraction of it; here it is
+    # the whole thing.
+    #
+    # This binds any change that rewrites the OSD deployment spec, not just
+    # Ceph version upgrades — a network provider change is one such.
+    osdMaxUpdatesInParallel: 1
     useAllNodes: false
     useAllDevices: false
     nodes:

kubernetes/infrastructure/rook-ceph-cluster/kustomization.yaml

@@ -1,6 +1,10 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
+  # Attachment for the VLAN 49 cluster network. Inert until cephcluster.yaml
+  # names it under network.selectors.cluster — creating it changes nothing on
+  # its own.
+  - san-network-attachment.yaml
   - cephcluster.yaml
   - toolbox.yaml
   - storageclass.yaml

kubernetes/infrastructure/rook-ceph-cluster/san-network-attachment.yaml

@@ -0,0 +1,82 @@
+---
+# The Ceph cluster network as a Multus attachment, so OSDs reach VLAN 49
+# without every Ceph daemon being put on host networking to get there.
+#
+# WHAT THIS REPLACES, AND WHY
+#
+# `addressRanges` only takes effect under `provider: host` — Rook ignores it
+# for the default CNI provider. So carrying replication over the dedicated
+# 10G link meant setting provider: host, which is not per-daemon: it puts
+# mons on host networking too. A mon's address is then the node's DHCP lease,
+# and when that lease moves the mon cannot bind and fails. Worse, three mons
+# on three nodes each hold the mon host ports, so Rook cannot schedule a
+# replacement anywhere and the failover it starts cannot finish.
+#
+# None of that has anything to do with the goal. Per the Ceph documentation
+# the cluster network carries OSD heartbeat, replication and recovery only —
+# "Ceph Monitors always operate on the public network". Attaching it here
+# gives the OSDs the link and leaves everything else on the pod network,
+# where Rook manages identity itself.
+#
+# ISOLATION
+#
+# VLAN 49 carries Ceph-internal traffic and nothing else. Two consequences
+# are deliberate:
+#
+#   No gateway and no routes. There is no router on this segment and there
+#   must not be one. whereabouts adds neither unless asked, so nothing here
+#   can leak onto or off the SAN. This also means no `sbr` chaining is
+#   needed — unlike the v30iot attachments, there is no DHCP-supplied default
+#   route to confine, because there is no DHCP and no default route.
+#
+#   The pool stays inside 10.99.0.0/29, on-link with the node addresses, and
+#   this is load-bearing during the migration rather than a detail. OSDs move
+#   one at a time, so the cluster runs mixed for a while: an OSD already on
+#   this attachment has to reach an OSD still using the node's host address.
+#   There is no router on this segment, so anything outside the /29 would be
+#   unreachable from the un-migrated side and the two would mark each other
+#   down — during precisely the window that is supposed to be safe.
+#
+#   range_start/range_end reserve .4-.6 and leave .1-.3 to the nodes. That is
+#   exactly one address per OSD with nothing spare, which is sufficient
+#   because osdMaxUpdatesInParallel bounds the migration to one OSD at a
+#   time: allocations grow 1, 2, 3 and never overlap. Once the switch is
+#   confirmed and the node addresses come out of the Talos configs, the pool
+#   can take the whole /29 and the spare returns.
+#
+# IPAM
+#
+# whereabouts, because neither alternative works on this segment: dhcp needs
+# a server that must not exist here, and host-local keeps its state per node
+# with no cluster-wide coordination, so two nodes would hand out the same
+# address on shared L2. See infrastructure/multus/whereabouts-daemonset.yaml.
+#
+# MTU 9000 matches the link. macvlan children inherit the master's MTU, but
+# it is stated so that a change to the parent does not silently halve
+# replication throughput without anything in the tree recording the
+# expectation.
+apiVersion: k8s.cni.cncf.io/v1
+kind: NetworkAttachmentDefinition
+metadata:
+  name: ceph-san
+  namespace: rook-ceph
+spec:
+  # `master` is an interface name. enp2s0f1np1 is the second X710 SFP+ port,
+  # a busPath-derived predictable name (net.ifnames=1 via the Image Factory
+  # schematic) that is identical on k8s-1/2/3 and cannot shuffle across
+  # reboots.
+  config: |
+    {
+      "cniVersion": "0.3.1",
+      "name": "ceph-san",
+      "type": "macvlan",
+      "master": "enp2s0f1np1",
+      "mode": "bridge",
+      "mtu": 9000,
+      "ipam": {
+        "type": "whereabouts",
+        "range": "10.99.0.0/29",
+        "range_start": "10.99.0.4",
+        "range_end": "10.99.0.6"
+      }
+    }

kubernetes/talos/patches/k8s-1.yaml

@@ -117,13 +117,20 @@ parent: enp2s0f0np0
 up: true
 ---
 # Dedicated Ceph cluster network on X710 port 1, VLAN 49 / 10.99.0.0/29,
-# untagged on the wire. Used by Rook for OSD<->OSD replication via
-# CephCluster.spec.network.addressRanges.cluster. Not routed off its own
-# segment, and deliberately has no default route.
+# untagged on the wire. Carries OSD<->OSD replication and nothing else.
+#
+# The node deliberately holds NO address here. Rook attaches this link to the
+# OSD pods through a Multus macvlan (infrastructure/rook-ceph-cluster/
+# san-network-attachment.yaml), and a macvlan parent only has to be up — it
+# does not need an address of its own for its children to work.
+#
+# It used to carry 10.99.0.1/29, because reaching this link required
+# CephCluster.spec.network.provider: host, and that put every Ceph daemon on
+# host networking, mons included. Nothing needs it now, and by the rule this
+# segment is built on — Ceph-internal traffic and nothing else — the node has
+# no business on it either.
 apiVersion: v1alpha1
 kind: LinkConfig
 name: enp2s0f1np1
 up: true
 mtu: 9000
-addresses:
-  - address: 10.99.0.1/29

kubernetes/talos/patches/k8s-2.yaml

@@ -89,10 +89,17 @@ up: true
 # Dedicated Ceph cluster network on X710 port 1, VLAN 49 / 10.99.0.0/29,
 # untagged on the wire. OSD<->OSD replication only. Not routed off its own
 # segment and deliberately has no default route.
+#
+# The node deliberately holds NO address here. Rook attaches this link to the
+# OSD pods through a Multus macvlan (infrastructure/rook-ceph-cluster/
+# san-network-attachment.yaml), and a macvlan parent only has to be up — it
+# does not need an address of its own for its children to work.
+#
+# It used to carry 10.99.0.2/29, because reaching this link required
+# CephCluster.spec.network.provider: host, which put every Ceph daemon on
+# host networking, mons included. Nothing needs it now.
 apiVersion: v1alpha1
 kind: LinkConfig
 name: enp2s0f1np1
 up: true
 mtu: 9000
-addresses:
-  - address: 10.99.0.2/29

kubernetes/talos/patches/k8s-3.yaml

@@ -73,10 +73,16 @@ parent: enp2s0f0np0
 up: true
 ---
 # Dedicated Ceph cluster network — see k8s-1.yaml.
+# The node deliberately holds NO address here. Rook attaches this link to the
+# OSD pods through a Multus macvlan (infrastructure/rook-ceph-cluster/
+# san-network-attachment.yaml), and a macvlan parent only has to be up — it
+# does not need an address of its own for its children to work.
+#
+# It used to carry 10.99.0.3/29, because reaching this link required
+# CephCluster.spec.network.provider: host, which put every Ceph daemon on
+# host networking, mons included. Nothing needs it now.
 apiVersion: v1alpha1
 kind: LinkConfig
 name: enp2s0f1np1
 up: true
 mtu: 9000
-addresses:
-  - address: 10.99.0.3/29