diff --git a/kubernetes/MIGRATION-sops-openbao.md b/kubernetes/MIGRATION-sops-openbao.md index 503e92f9..82cdecda 100644 --- a/kubernetes/MIGRATION-sops-openbao.md +++ b/kubernetes/MIGRATION-sops-openbao.md @@ -51,7 +51,11 @@ kubectl -n openbao exec -it openbao-0 -- bao operator unseal # paste key ``` **Manual unseal is required after every pod restart** (node drain, upgrade, -crash): `kubectl -n openbao exec -it openbao-0 -- bao operator unseal`. +crash), **per replica** — the server runs three raft peers (see +`infrastructure/openbao/helmrelease.yaml`), and a sealed one is neither a +voter nor a standby, so it stays out of the cluster until unsealed: +`kubectl -n openbao exec -it openbao- -- bao operator unseal`, same key for +all three. Three peers mean one sealed replica is survivable; two are not. Accepted for now; auto-unseal needs a second trust root (transit/KMS) and is explicitly out of scope. Symptom of sealed: ESO stores go NotReady, synced Secrets keep their last value (they are real k8s Secrets — pods keep running). diff --git a/kubernetes/RUNBOOK-cold-rebuild.md b/kubernetes/RUNBOOK-cold-rebuild.md index d49f58a6..ac0acdf7 100644 --- a/kubernetes/RUNBOOK-cold-rebuild.md +++ b/kubernetes/RUNBOOK-cold-rebuild.md @@ -149,8 +149,8 @@ Secrets, and every secret-consuming app stays Pending. This step is **manual** (the unseal key lives in `pass`, off-cluster by design). ```sh -# 1. Wait for the OpenBao pod to exist (sealed, 0/1 Ready is expected). -kubectl -n openbao get pod openbao-0 +# 1. Wait for the OpenBao pods to exist (all three sealed, 0/1 Ready is expected). +kubectl -n openbao get pods # 2. Restore the most recent raft snapshot. Source it from the snapshot PVC # (infrastructure/openbao/snapshot-cronjob.yaml) or, in a total-loss rebuild, @@ -158,8 +158,12 @@ kubectl -n openbao get pod openbao-0 kubectl -n openbao cp .snap openbao-0:/tmp/restore.snap kubectl -n openbao exec -it openbao-0 -- bao operator raft snapshot restore /tmp/restore.snap -# 3. Unseal with the key from pass (threshold 1). +# 3. Unseal with the key from pass (threshold 1) — EVERY replica, same key. +# A sealed server is neither a raft voter nor a standby, so unsealing only +# openbao-0 leaves the three-peer cluster one node short of quorum. kubectl -n openbao exec -it openbao-0 -- bao operator unseal # paste homelab/openbao/unseal-key +kubectl -n openbao exec -it openbao-1 -- bao operator unseal # same key +kubectl -n openbao exec -it openbao-2 -- bao operator unseal # same key ``` Verify the store recovers before moving on: @@ -170,7 +174,9 @@ kubectl -n get externalsecret # want: SecretSynced ``` > **Manual unseal is required after every OpenBao pod restart**, not just -> during rebuilds. Details + rationale: `MIGRATION-sops-openbao.md`. +> during rebuilds, and for *each* replica that restarts — a rolling node +> drain seals all three in turn. Details + rationale: +> `MIGRATION-sops-openbao.md`. ## Step 5 — apps: let the rest reconcile