fix: [alert] KubeStatefulSetReplicasMismatch (openbao) (#645) #648

Merged
k.renner merged 1 commits from alert-fix/issue-645 into master 2026-08-27 04:53:57 +00:00
2 changed files with 15 additions and 5 deletions
+5 -1
View File
@@ -51,7 +51,11 @@ kubectl -n openbao exec -it openbao-0 -- bao operator unseal # paste key
```
**Manual unseal is required after every pod restart** (node drain, upgrade,
crash): `kubectl -n openbao exec -it openbao-0 -- bao operator unseal`.
crash), **per replica** — the server runs three raft peers (see
`infrastructure/openbao/helmrelease.yaml`), and a sealed one is neither a
voter nor a standby, so it stays out of the cluster until unsealed:
`kubectl -n openbao exec -it openbao-<N> -- bao operator unseal`, same key for
all three. Three peers mean one sealed replica is survivable; two are not.
Accepted for now; auto-unseal needs a second trust root (transit/KMS) and is
explicitly out of scope. Symptom of sealed: ESO stores go NotReady, synced
Secrets keep their last value (they are real k8s Secrets — pods keep running).
+10 -4
View File
@@ -149,8 +149,8 @@ Secrets, and every secret-consuming app stays Pending. This step is **manual**
(the unseal key lives in `pass`, off-cluster by design).
```sh
# 1. Wait for the OpenBao pod to exist (sealed, 0/1 Ready is expected).
kubectl -n openbao get pod openbao-0
# 1. Wait for the OpenBao pods to exist (all three sealed, 0/1 Ready is expected).
kubectl -n openbao get pods
# 2. Restore the most recent raft snapshot. Source it from the snapshot PVC
# (infrastructure/openbao/snapshot-cronjob.yaml) or, in a total-loss rebuild,
@@ -158,8 +158,12 @@ kubectl -n openbao get pod openbao-0
kubectl -n openbao cp <snapshot>.snap openbao-0:/tmp/restore.snap
kubectl -n openbao exec -it openbao-0 -- bao operator raft snapshot restore /tmp/restore.snap
# 3. Unseal with the key from pass (threshold 1).
# 3. Unseal with the key from pass (threshold 1) — EVERY replica, same key.
# A sealed server is neither a raft voter nor a standby, so unsealing only
# openbao-0 leaves the three-peer cluster one node short of quorum.
kubectl -n openbao exec -it openbao-0 -- bao operator unseal # paste homelab/openbao/unseal-key
kubectl -n openbao exec -it openbao-1 -- bao operator unseal # same key
kubectl -n openbao exec -it openbao-2 -- bao operator unseal # same key
```
Verify the store recovers before moving on:
@@ -170,7 +174,9 @@ kubectl -n <ns> get externalsecret # want: SecretSynced
```
> **Manual unseal is required after every OpenBao pod restart**, not just
> during rebuilds. Details + rationale: `MIGRATION-sops-openbao.md`.
> during rebuilds, and for *each* replica that restarts — a rolling node
> drain seals all three in turn. Details + rationale:
> `MIGRATION-sops-openbao.md`.
## Step 5 — apps: let the rest reconcile