Issue #645 reported a 2h13m KubeStatefulSetReplicasMismatch on openbao/openbao: the single raft replica was replaced at 18:17Z, came up sealed (no auto-unseal), stayed 0/1 until a human unsealed it at 20:30Z, and took the ClusterSecretStore — and with it ~45 ExternalSecrets — down for the duration. The availability root cause is already fixed in the repo: PR #646 (feat(openbao): run three raft replicas so a node drain is not an outage, merged as 71d6bd3, the commit this branch is based on) raised server.ha.replicas to 3 with retry_join for all three peers. I verified live that it is deployed and healthy, so there is nothing left to change in the OpenBao manifests. What #646 did not do is propagate the consequence to the operator docs, which still tell you to unseal openbao-0 and only openbao-0. Following them now leaves two of three peers sealed — i.e. no raft quorum — which is precisely the failure #646 was meant to prevent. This PR corrects those two documents.
Changes
kubernetes/RUNBOOK-cold-rebuild.md — Step 4 ("restore the snapshot + unseal") now waits on all three pods and unseals openbao-0/-1/-2 with the same key, with a one-line note on why (a sealed server is neither a voter nor a standby). The closing callout now says unseal is needed for each replica that restarts, since a rolling node drain seals all three in turn.
kubernetes/MIGRATION-sops-openbao.md — the "Manual unseal is required after every pod restart" rule is now stated per-replica and points at infrastructure/openbao/helmrelease.yaml for the three-peer topology, plus the operative fact for on-call: one sealed replica is survivable, two are not. The one-time bao operator init step above it is unchanged — init is correctly a single-node operation.
Docs only. No manifest, chart, threshold, or alerting-rule change.
Verification
Observed live this run with ./.ci-diag/kubectl:
get pods -n openbao -o wide → openbao-0, openbao-1, openbao-2 all 1/1 Running, 0 restarts, on k8s-1/k8s-3/k8s-2.
get statefulset -n openbao openbao -o jsonpath=... → spec.replicas=3 status.replicas=3 ready=3. The 3-replica change is reconciled, not just merged.
get svc,endpoints -n openbao → openbao-active has exactly one endpoint (10.244.0.122 = openbao-0, the leader) and openbao-standby has the other two. Those labels are set by OpenBao's service_registration "kubernetes" only on an unsealed node, so this is direct evidence all three peers are unsealed with a leader elected — the state the runbook must be able to reproduce.
get clustersecretstore openbao -o jsonpath=... → Ready=True reason=Valid msg="store validated" at=2026-08-26T20:33:25Z. Recovered, matching the issue's timeline.
Prometheus ALERTS{namespace="openbao"} via the API proxy → empty result. Nothing is firing now.
get cronjob,job -n openbao → openbao-raft-snapshot-29795100Complete 1/1 21h ago (and ...-29793660Complete 45h ago). This answers the issue's open question #3: backups were not affected.
Reviewer check: this is a documentation change, so there is nothing for Flux to reconcile and no alert that should start or stop firing. The substance is verifiable by reading the runbook against the live topology above — three peers, one leader, one shared unseal key.
Notes
One claim in the issue is wrong and I did not act on it. Recommendation 1 argues for "a dedicated, faster, higher-severity 'OpenBao sealed' alert routed to a real receiver, added to kubernetes/apps/monitoring/helmrelease-prometheus.yaml, so unseal starts in minutes rather than hours," on the premise that "the alert only became visible after the fact." Detection was not the problem:
The live rule definition (/api/v1/rules?rule_name[]=KubeStatefulSetReplicasMismatch) is duration: 900 — it fires at 15 minutes, so it would have gone active ~18:35Z.
Alertmanager's config (helmrelease-prometheus.yaml:289-324) has a single default route to receiver email → admin@renner.uno, with only Watchdog and KubeCPUOvercommit nulled. Every severity already reaches the only receiver that exists.
So a duplicate rule at severity: critical would send the same mail to the same address ~13 minutes earlier. The ~2h gap was human response time, not detection latency, and a new PrometheusRule would not have shortened it. I've left the alerting config alone rather than add a rule whose only effect is a second copy of an email that already sent.
Recommendation 2 (auto-unseal) is out of scope for this pipeline: it needs a second trust root (transit/KMS), which is a new dependency and an architecture decision, not a surgical fix. It remains the only change that would actually remove the manual step.
No openbao ServiceMonitor exists (confirmed via get servicemonitor -A — nothing in the openbao namespace), so bao_core_unsealed is not in Prometheus. Any future "sealed replica" alert would have to either add scraping of OpenBao's /v1/sys/metrics first, or be built on kube-state-metrics readiness — worth knowing before someone tries the metric-based version.
Two things the issue flagged that I confirmed are already handled, so no change was needed: snapshot-cronjob.yaml already targets openbao-active rather than the plain service, with an in-line comment explaining that a raft snapshot must hit the leader; and the KubePdbNotEnoughHealthyPods companion alert should stop being a single-pod tripwire now that the chart's disruption budget covers three replicas. I could not read the PDB to confirm the second — the CI ServiceAccount is denied poddisruptionbudgets (Error from server (Forbidden)) — so that one is inference from the replica count, not an observation.
Not verified, and left alone: why the pod was replaced at 18:17Z. The issue's Ceph-eviction and manual-delete hypotheses both remain open; the namespace no longer retains the event. It does not affect this change either way — sealed-on-restart is the same regardless of what triggered the restart.
## What this PR does
Issue #645 reported a 2h13m `KubeStatefulSetReplicasMismatch` on `openbao/openbao`: the single raft replica was replaced at 18:17Z, came up **sealed** (no auto-unseal), stayed `0/1` until a human unsealed it at 20:30Z, and took the `ClusterSecretStore` — and with it ~45 ExternalSecrets — down for the duration. **The availability root cause is already fixed in the repo**: PR #646 (`feat(openbao): run three raft replicas so a node drain is not an outage`, merged as `71d6bd3`, the commit this branch is based on) raised `server.ha.replicas` to 3 with `retry_join` for all three peers. I verified live that it is deployed and healthy, so there is nothing left to change in the OpenBao manifests. What #646 did **not** do is propagate the consequence to the operator docs, which still tell you to unseal `openbao-0` and only `openbao-0`. Following them now leaves two of three peers sealed — i.e. no raft quorum — which is precisely the failure #646 was meant to prevent. This PR corrects those two documents.
## Changes
- `kubernetes/RUNBOOK-cold-rebuild.md` — Step 4 ("restore the snapshot + unseal") now waits on all three pods and unseals `openbao-0`/`-1`/`-2` with the same key, with a one-line note on why (a sealed server is neither a voter nor a standby). The closing callout now says unseal is needed for *each* replica that restarts, since a rolling node drain seals all three in turn.
- `kubernetes/MIGRATION-sops-openbao.md` — the "Manual unseal is required after every pod restart" rule is now stated per-replica and points at `infrastructure/openbao/helmrelease.yaml` for the three-peer topology, plus the operative fact for on-call: one sealed replica is survivable, two are not. The one-time `bao operator init` step above it is unchanged — init is correctly a single-node operation.
Docs only. No manifest, chart, threshold, or alerting-rule change.
## Verification
Observed live this run with `./.ci-diag/kubectl`:
- `get pods -n openbao -o wide` → `openbao-0`, `openbao-1`, `openbao-2` all `1/1 Running`, 0 restarts, on `k8s-1`/`k8s-3`/`k8s-2`.
- `get statefulset -n openbao openbao -o jsonpath=...` → `spec.replicas=3 status.replicas=3 ready=3`. The 3-replica change is reconciled, not just merged.
- `get svc,endpoints -n openbao` → `openbao-active` has exactly one endpoint (`10.244.0.122` = `openbao-0`, the leader) and `openbao-standby` has the other two. Those labels are set by OpenBao's `service_registration "kubernetes"` only on an **unsealed** node, so this is direct evidence all three peers are unsealed with a leader elected — the state the runbook must be able to reproduce.
- `get clustersecretstore openbao -o jsonpath=...` → `Ready=True reason=Valid msg="store validated" at=2026-08-26T20:33:25Z`. Recovered, matching the issue's timeline.
- Prometheus `ALERTS{namespace="openbao"}` via the API proxy → empty result. Nothing is firing now.
- `get cronjob,job -n openbao` → `openbao-raft-snapshot-29795100` `Complete 1/1` 21h ago (and `...-29793660` `Complete` 45h ago). This answers the issue's open question #3: backups were not affected.
Reviewer check: this is a documentation change, so there is nothing for Flux to reconcile and no alert that should start or stop firing. The substance is verifiable by reading the runbook against the live topology above — three peers, one leader, one shared unseal key.
## Notes
**One claim in the issue is wrong and I did not act on it.** Recommendation 1 argues for "a dedicated, faster, higher-severity 'OpenBao sealed' alert routed to a real receiver, added to `kubernetes/apps/monitoring/helmrelease-prometheus.yaml`, so unseal starts in minutes rather than hours," on the premise that "the alert only became visible after the fact." Detection was not the problem:
- The live rule definition (`/api/v1/rules?rule_name[]=KubeStatefulSetReplicasMismatch`) is `duration: 900` — it fires at 15 minutes, so it would have gone active ~18:35Z.
- Alertmanager's config (`helmrelease-prometheus.yaml:289-324`) has a single default route to receiver `email` → `admin@renner.uno`, with only `Watchdog` and `KubeCPUOvercommit` nulled. **Every** severity already reaches the only receiver that exists.
So a duplicate rule at `severity: critical` would send the same mail to the same address ~13 minutes earlier. The ~2h gap was human response time, not detection latency, and a new PrometheusRule would not have shortened it. I've left the alerting config alone rather than add a rule whose only effect is a second copy of an email that already sent.
**Recommendation 2 (auto-unseal) is out of scope** for this pipeline: it needs a second trust root (transit/KMS), which is a new dependency and an architecture decision, not a surgical fix. It remains the only change that would actually remove the manual step.
**No openbao ServiceMonitor exists** (confirmed via `get servicemonitor -A` — nothing in the `openbao` namespace), so `bao_core_unsealed` is not in Prometheus. Any future "sealed replica" alert would have to either add scraping of OpenBao's `/v1/sys/metrics` first, or be built on kube-state-metrics readiness — worth knowing before someone tries the metric-based version.
**Two things the issue flagged that I confirmed are already handled**, so no change was needed: `snapshot-cronjob.yaml` already targets `openbao-active` rather than the plain service, with an in-line comment explaining that a raft snapshot must hit the leader; and the `KubePdbNotEnoughHealthyPods` companion alert should stop being a single-pod tripwire now that the chart's disruption budget covers three replicas. I could not read the PDB to confirm the second — the CI ServiceAccount is denied `poddisruptionbudgets` (`Error from server (Forbidden)`) — so that one is inference from the replica count, not an observation.
**Not verified, and left alone:** why the pod was replaced at 18:17Z. The issue's Ceph-eviction and manual-delete hypotheses both remain open; the namespace no longer retains the event. It does not affect this change either way — sealed-on-restart is the same regardless of what triggered the restart.
Closes #645
🤖Automated review — model claude-haiku-4-5-20251001 (complexity: trivial), commit 781b849adb8d163c2d78cea938c652704c191ed6.
Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.
Summary
This PR updates two documentation files to reflect the 3-replica OpenBao Raft cluster deployed in PR #646. The changes correctly document that all three replicas must be unsealed manually after pod restarts, and explain the technical reasoning (sealed servers cannot act as voters or standbys). The documentation is accurate, complete, and addresses the root cause of issue #645.
🟢 Nits
Path reference consistency (minor): kubernetes/MIGRATION-sops-openbao.md:55 references infrastructure/openbao/helmrelease.yaml without the kubernetes/ prefix. While this follows the existing convention elsewhere in the file (lines 73, 235), line 32 uses the full path kubernetes/infrastructure/openbao/. The actual file is at kubernetes/infrastructure/openbao/helmrelease.yaml. For maximum clarity, consider standardizing on the full path across the documentation.
Tests
No tests needed — this is a documentation-only change. The PR body includes verification of the deployed state (3 pods running, StatefulSet spec confirms replicas=3, status=3, ready=3, ClusterSecretStore is Ready, no alerts firing, backups completing), which validates that the new documentation matches the actual deployed system.
Verification Notes
Verified key technical claims:
✓ Helmrelease.yaml at kubernetes/infrastructure/openbao/helmrelease.yaml confirms replicas: 3 with retry_join for all three peers (lines 66, 101-109)
✓ Snapshot cronjob correctly targets openbao-active (the leader) for consistency, as documented at line 56-62
✓ Base commit 71d6bd3... is the merge of PR #646, which introduced the 3-replica setup
✓ Technical explanation is sound: in Raft, sealed servers cannot serve as quorum participants
Recommendation
Looks reasonable — no blocking issues; the path reference inconsistency is pre-existing and minor.
🤖 **Automated review** — model `claude-haiku-4-5-20251001` (complexity: trivial), commit `781b849adb8d163c2d78cea938c652704c191ed6`.
> Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review.
---
## Summary
This PR updates two documentation files to reflect the 3-replica OpenBao Raft cluster deployed in PR #646. The changes correctly document that all three replicas must be unsealed manually after pod restarts, and explain the technical reasoning (sealed servers cannot act as voters or standbys). The documentation is accurate, complete, and addresses the root cause of issue #645.
## 🟢 Nits
**Path reference consistency (minor):**
`kubernetes/MIGRATION-sops-openbao.md:55` references `infrastructure/openbao/helmrelease.yaml` without the `kubernetes/` prefix. While this follows the existing convention elsewhere in the file (lines 73, 235), line 32 uses the full path `kubernetes/infrastructure/openbao/`. The actual file is at `kubernetes/infrastructure/openbao/helmrelease.yaml`. For maximum clarity, consider standardizing on the full path across the documentation.
## Tests
No tests needed — this is a documentation-only change. The PR body includes verification of the deployed state (3 pods running, StatefulSet spec confirms replicas=3, status=3, ready=3, ClusterSecretStore is Ready, no alerts firing, backups completing), which validates that the new documentation matches the actual deployed system.
## Verification Notes
Verified key technical claims:
- ✓ Helmrelease.yaml at `kubernetes/infrastructure/openbao/helmrelease.yaml` confirms `replicas: 3` with `retry_join` for all three peers (lines 66, 101-109)
- ✓ Snapshot cronjob correctly targets `openbao-active` (the leader) for consistency, as documented at line 56-62
- ✓ Base commit `71d6bd3...` is the merge of PR #646, which introduced the 3-replica setup
- ✓ Technical explanation is sound: in Raft, sealed servers cannot serve as quorum participants
## Recommendation
Looks reasonable — no blocking issues; the path reference inconsistency is pre-existing and minor.
<!-- pr-review:sha=781b849adb8d163c2d78cea938c652704c191ed6 -->
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
review-bot
What this PR does
Issue #645 reported a 2h13m
KubeStatefulSetReplicasMismatchonopenbao/openbao: the single raft replica was replaced at 18:17Z, came up sealed (no auto-unseal), stayed0/1until a human unsealed it at 20:30Z, and took theClusterSecretStore— and with it ~45 ExternalSecrets — down for the duration. The availability root cause is already fixed in the repo: PR #646 (feat(openbao): run three raft replicas so a node drain is not an outage, merged as71d6bd3, the commit this branch is based on) raisedserver.ha.replicasto 3 withretry_joinfor all three peers. I verified live that it is deployed and healthy, so there is nothing left to change in the OpenBao manifests. What #646 did not do is propagate the consequence to the operator docs, which still tell you to unsealopenbao-0and onlyopenbao-0. Following them now leaves two of three peers sealed — i.e. no raft quorum — which is precisely the failure #646 was meant to prevent. This PR corrects those two documents.Changes
kubernetes/RUNBOOK-cold-rebuild.md— Step 4 ("restore the snapshot + unseal") now waits on all three pods and unsealsopenbao-0/-1/-2with the same key, with a one-line note on why (a sealed server is neither a voter nor a standby). The closing callout now says unseal is needed for each replica that restarts, since a rolling node drain seals all three in turn.kubernetes/MIGRATION-sops-openbao.md— the "Manual unseal is required after every pod restart" rule is now stated per-replica and points atinfrastructure/openbao/helmrelease.yamlfor the three-peer topology, plus the operative fact for on-call: one sealed replica is survivable, two are not. The one-timebao operator initstep above it is unchanged — init is correctly a single-node operation.Docs only. No manifest, chart, threshold, or alerting-rule change.
Verification
Observed live this run with
./.ci-diag/kubectl:get pods -n openbao -o wide→openbao-0,openbao-1,openbao-2all1/1 Running, 0 restarts, onk8s-1/k8s-3/k8s-2.get statefulset -n openbao openbao -o jsonpath=...→spec.replicas=3 status.replicas=3 ready=3. The 3-replica change is reconciled, not just merged.get svc,endpoints -n openbao→openbao-activehas exactly one endpoint (10.244.0.122=openbao-0, the leader) andopenbao-standbyhas the other two. Those labels are set by OpenBao'sservice_registration "kubernetes"only on an unsealed node, so this is direct evidence all three peers are unsealed with a leader elected — the state the runbook must be able to reproduce.get clustersecretstore openbao -o jsonpath=...→Ready=True reason=Valid msg="store validated" at=2026-08-26T20:33:25Z. Recovered, matching the issue's timeline.ALERTS{namespace="openbao"}via the API proxy → empty result. Nothing is firing now.get cronjob,job -n openbao→openbao-raft-snapshot-29795100Complete 1/121h ago (and...-29793660Complete45h ago). This answers the issue's open question #3: backups were not affected.Reviewer check: this is a documentation change, so there is nothing for Flux to reconcile and no alert that should start or stop firing. The substance is verifiable by reading the runbook against the live topology above — three peers, one leader, one shared unseal key.
Notes
One claim in the issue is wrong and I did not act on it. Recommendation 1 argues for "a dedicated, faster, higher-severity 'OpenBao sealed' alert routed to a real receiver, added to
kubernetes/apps/monitoring/helmrelease-prometheus.yaml, so unseal starts in minutes rather than hours," on the premise that "the alert only became visible after the fact." Detection was not the problem:/api/v1/rules?rule_name[]=KubeStatefulSetReplicasMismatch) isduration: 900— it fires at 15 minutes, so it would have gone active ~18:35Z.helmrelease-prometheus.yaml:289-324) has a single default route to receiveremail→admin@renner.uno, with onlyWatchdogandKubeCPUOvercommitnulled. Every severity already reaches the only receiver that exists.So a duplicate rule at
severity: criticalwould send the same mail to the same address ~13 minutes earlier. The ~2h gap was human response time, not detection latency, and a new PrometheusRule would not have shortened it. I've left the alerting config alone rather than add a rule whose only effect is a second copy of an email that already sent.Recommendation 2 (auto-unseal) is out of scope for this pipeline: it needs a second trust root (transit/KMS), which is a new dependency and an architecture decision, not a surgical fix. It remains the only change that would actually remove the manual step.
No openbao ServiceMonitor exists (confirmed via
get servicemonitor -A— nothing in theopenbaonamespace), sobao_core_unsealedis not in Prometheus. Any future "sealed replica" alert would have to either add scraping of OpenBao's/v1/sys/metricsfirst, or be built on kube-state-metrics readiness — worth knowing before someone tries the metric-based version.Two things the issue flagged that I confirmed are already handled, so no change was needed:
snapshot-cronjob.yamlalready targetsopenbao-activerather than the plain service, with an in-line comment explaining that a raft snapshot must hit the leader; and theKubePdbNotEnoughHealthyPodscompanion alert should stop being a single-pod tripwire now that the chart's disruption budget covers three replicas. I could not read the PDB to confirm the second — the CI ServiceAccount is deniedpoddisruptionbudgets(Error from server (Forbidden)) — so that one is inference from the replica count, not an observation.Not verified, and left alone: why the pod was replaced at 18:17Z. The issue's Ceph-eviction and manual-delete hypotheses both remain open; the namespace no longer retains the event. It does not affect this change either way — sealed-on-restart is the same regardless of what triggered the restart.
Closes #645
🤖 Automated review — model
claude-haiku-4-5-20251001(complexity: trivial), commit781b849adb8d163c2d78cea938c652704c191ed6.Summary
This PR updates two documentation files to reflect the 3-replica OpenBao Raft cluster deployed in PR #646. The changes correctly document that all three replicas must be unsealed manually after pod restarts, and explain the technical reasoning (sealed servers cannot act as voters or standbys). The documentation is accurate, complete, and addresses the root cause of issue #645.
🟢 Nits
Path reference consistency (minor):
kubernetes/MIGRATION-sops-openbao.md:55referencesinfrastructure/openbao/helmrelease.yamlwithout thekubernetes/prefix. While this follows the existing convention elsewhere in the file (lines 73, 235), line 32 uses the full pathkubernetes/infrastructure/openbao/. The actual file is atkubernetes/infrastructure/openbao/helmrelease.yaml. For maximum clarity, consider standardizing on the full path across the documentation.Tests
No tests needed — this is a documentation-only change. The PR body includes verification of the deployed state (3 pods running, StatefulSet spec confirms replicas=3, status=3, ready=3, ClusterSecretStore is Ready, no alerts firing, backups completing), which validates that the new documentation matches the actual deployed system.
Verification Notes
Verified key technical claims:
kubernetes/infrastructure/openbao/helmrelease.yamlconfirmsreplicas: 3withretry_joinfor all three peers (lines 66, 101-109)openbao-active(the leader) for consistency, as documented at line 56-6271d6bd3...is the merge of PR #646, which introduced the 3-replica setupRecommendation
Looks reasonable — no blocking issues; the path reference inconsistency is pre-existing and minor.