fix: [alert] KubeStatefulSetReplicasMismatch (openbao) (#645) #648
@@ -51,7 +51,11 @@ kubectl -n openbao exec -it openbao-0 -- bao operator unseal # paste key
|
||||
```
|
||||
|
||||
**Manual unseal is required after every pod restart** (node drain, upgrade,
|
||||
crash): `kubectl -n openbao exec -it openbao-0 -- bao operator unseal`.
|
||||
crash), **per replica** — the server runs three raft peers (see
|
||||
`infrastructure/openbao/helmrelease.yaml`), and a sealed one is neither a
|
||||
voter nor a standby, so it stays out of the cluster until unsealed:
|
||||
`kubectl -n openbao exec -it openbao-<N> -- bao operator unseal`, same key for
|
||||
all three. Three peers mean one sealed replica is survivable; two are not.
|
||||
Accepted for now; auto-unseal needs a second trust root (transit/KMS) and is
|
||||
explicitly out of scope. Symptom of sealed: ESO stores go NotReady, synced
|
||||
Secrets keep their last value (they are real k8s Secrets — pods keep running).
|
||||
|
||||
@@ -149,8 +149,8 @@ Secrets, and every secret-consuming app stays Pending. This step is **manual**
|
||||
(the unseal key lives in `pass`, off-cluster by design).
|
||||
|
||||
```sh
|
||||
# 1. Wait for the OpenBao pod to exist (sealed, 0/1 Ready is expected).
|
||||
kubectl -n openbao get pod openbao-0
|
||||
# 1. Wait for the OpenBao pods to exist (all three sealed, 0/1 Ready is expected).
|
||||
kubectl -n openbao get pods
|
||||
|
||||
# 2. Restore the most recent raft snapshot. Source it from the snapshot PVC
|
||||
# (infrastructure/openbao/snapshot-cronjob.yaml) or, in a total-loss rebuild,
|
||||
@@ -158,8 +158,12 @@ kubectl -n openbao get pod openbao-0
|
||||
kubectl -n openbao cp <snapshot>.snap openbao-0:/tmp/restore.snap
|
||||
kubectl -n openbao exec -it openbao-0 -- bao operator raft snapshot restore /tmp/restore.snap
|
||||
|
||||
# 3. Unseal with the key from pass (threshold 1).
|
||||
# 3. Unseal with the key from pass (threshold 1) — EVERY replica, same key.
|
||||
# A sealed server is neither a raft voter nor a standby, so unsealing only
|
||||
# openbao-0 leaves the three-peer cluster one node short of quorum.
|
||||
kubectl -n openbao exec -it openbao-0 -- bao operator unseal # paste homelab/openbao/unseal-key
|
||||
kubectl -n openbao exec -it openbao-1 -- bao operator unseal # same key
|
||||
kubectl -n openbao exec -it openbao-2 -- bao operator unseal # same key
|
||||
```
|
||||
|
||||
Verify the store recovers before moving on:
|
||||
@@ -170,7 +174,9 @@ kubectl -n <ns> get externalsecret # want: SecretSynced
|
||||
```
|
||||
|
||||
> **Manual unseal is required after every OpenBao pod restart**, not just
|
||||
> during rebuilds. Details + rationale: `MIGRATION-sops-openbao.md`.
|
||||
> during rebuilds, and for *each* replica that restarts — a rolling node
|
||||
> drain seals all three in turn. Details + rationale:
|
||||
> `MIGRATION-sops-openbao.md`.
|
||||
|
||||
## Step 5 — apps: let the rest reconcile
|
||||
|
||||
|
||||
Reference in New Issue
Block a user