OpenBaoQuorumLost paged as critical on one sample at ~12:27:00 UTC on 2026-09-23. Raft had not lost quorum. The rule uses pod readiness as a stand-in for raft quorum, because OpenBao is not scraped. It also had for: 0m, so any single evaluation with fewer than 2 Ready replicas paged at once. During a rolling node reboot, openbao-2 was evicted and stayed Pending on the cordoned k8s-2. So only openbao-0 and openbao-1 existed. Then k8s-1 flapped NotReady for about a second, and the node lifecycle controller marked openbao-0 not-ready for ~30s. That left one Ready replica, even though openbao-0 stayed the active node and kept serving. This PR adds for: 2m to the rule and updates its comment to say why.
Changes
kubernetes/apps/monitoring-rules/prometheusrule-openbao.yaml: OpenBaoQuorumLost changed from for: 0m to for: 2m, and the "No window" comment was replaced with the reasoning above. The expression, severity and other rules are unchanged.
Verification
Checked live this run:
Prometheus, max by (pod)(kube_pod_status_ready{namespace="openbao",condition="true"}) at 15s steps over 12:25–12:30:
openbao-0 was 0 only at 12:26:45 and 12:27:00.
openbao-1 was 1 throughout.
openbao-2 had no series until ~12:29:45.
So readiness was below quorum for ≤ ~30s.
Loki, {namespace="openbao",pod="openbao-0"} between 12:26 and 12:28: openbao-0 kept serving as the active node while it was marked not-ready.
A successful auth/kubernetes/login request/response at 12:27:16.
The only errors were failed to appendEntries to: peer=... openbao-2 ... no such host, which is expected while openbao-2 was evicted.
Loki, {namespace="openbao"} |~ "(?i)(leader|election|quorum|sealed|standby|active)" over 12:20–12:30: the only matches are from openbao-2 rejoining at 12:29:24 (entering follower state, vault is unsealed, unsealed with stored key). openbao-0 and openbao-1 logged no election or leader-loss lines.
After Flux reconciles, confirm with kubectl -n monitoring get prometheusrule openbao -o yaml or the Prometheus /rules page that OpenBaoQuorumLost shows for: 2m. A real loss of quorum, such as a sealed or crashed majority, lasts far longer than 2 minutes and will still page. The next rolling Talos reboot is the practical test: it should not page on a brief node flap.
Notes
This reverses a deliberate choice. The old comment argued that any delay only postpones the page. This incident shows that readiness can drop below quorum while raft is healthy, and existing Secrets keep their last values. So a 2-minute delay costs almost nothing and removes the false critical.
Better long-term fix (out of scope): scrape OpenBao's own telemetry and alert on raft leader or active-node metrics instead of pod readiness. That means enabling unauthenticated_metrics_access (or a metrics token) and adding a ServiceMonitor. It changes the HelmRelease and adds a new scrape target, so it is more than a surgical change.
Maintenance risk the issue raised (no change made): while a node is cordoned during a rolling reboot, OpenBao runs on 2/3 replicas, because hard anti-affinity keeps the evicted pod Pending (~10 min here). The runbook should make sure the next node is drained only after OpenBao is back to 3/3 Ready. I did not check the PDB. The issue says the diagnostics SA cannot list poddisruptionbudgets, and I did not re-test that.
Not checked: the issue's hypothesis that the k8s-1 NotReady flap came from a kube-controller-manager leader failover. The fix does not depend on it.
## What this PR does
`OpenBaoQuorumLost` paged as critical on one sample at ~12:27:00 UTC on 2026-09-23. Raft had not lost quorum. The rule uses pod readiness as a stand-in for raft quorum, because OpenBao is not scraped. It also had `for: 0m`, so any single evaluation with fewer than 2 Ready replicas paged at once. During a rolling node reboot, openbao-2 was evicted and stayed Pending on the cordoned k8s-2. So only openbao-0 and openbao-1 existed. Then k8s-1 flapped NotReady for about a second, and the node lifecycle controller marked openbao-0 not-ready for ~30s. That left one Ready replica, even though openbao-0 stayed the active node and kept serving. This PR adds `for: 2m` to the rule and updates its comment to say why.
## Changes
- `kubernetes/apps/monitoring-rules/prometheusrule-openbao.yaml`: `OpenBaoQuorumLost` changed from `for: 0m` to `for: 2m`, and the "No window" comment was replaced with the reasoning above. The expression, severity and other rules are unchanged.
## Verification
Checked live this run:
- **Prometheus**, `max by (pod)(kube_pod_status_ready{namespace="openbao",condition="true"})` at 15s steps over 12:25–12:30:
- openbao-0 was `0` only at 12:26:45 and 12:27:00.
- openbao-1 was `1` throughout.
- openbao-2 had no series until ~12:29:45.
- So readiness was below quorum for ≤ ~30s.
- **Loki**, `{namespace="openbao",pod="openbao-0"}` between 12:26 and 12:28: openbao-0 kept serving as the active node while it was marked not-ready.
- `12:26:26.753Z [INFO] expiration: revoked lease ...`
- A successful `auth/kubernetes/login` request/response at `12:27:16`.
- The only errors were `failed to appendEntries to: peer=... openbao-2 ... no such host`, which is expected while openbao-2 was evicted.
- **Loki**, `{namespace="openbao"} |~ "(?i)(leader|election|quorum|sealed|standby|active)"` over 12:20–12:30: the only matches are from openbao-2 rejoining at 12:29:24 (`entering follower state`, `vault is unsealed`, `unsealed with stored key`). openbao-0 and openbao-1 logged no election or leader-loss lines.
After Flux reconciles, confirm with `kubectl -n monitoring get prometheusrule openbao -o yaml` or the Prometheus `/rules` page that `OpenBaoQuorumLost` shows `for: 2m`. A real loss of quorum, such as a sealed or crashed majority, lasts far longer than 2 minutes and will still page. The next rolling Talos reboot is the practical test: it should not page on a brief node flap.
## Notes
- **This reverses a deliberate choice.** The old comment argued that any delay only postpones the page. This incident shows that readiness can drop below quorum while raft is healthy, and existing Secrets keep their last values. So a 2-minute delay costs almost nothing and removes the false critical.
- **Better long-term fix (out of scope):** scrape OpenBao's own telemetry and alert on raft leader or active-node metrics instead of pod readiness. That means enabling `unauthenticated_metrics_access` (or a metrics token) and adding a ServiceMonitor. It changes the HelmRelease and adds a new scrape target, so it is more than a surgical change.
- **Maintenance risk the issue raised (no change made):** while a node is cordoned during a rolling reboot, OpenBao runs on 2/3 replicas, because hard anti-affinity keeps the evicted pod Pending (~10 min here). The runbook should make sure the next node is drained only after OpenBao is back to 3/3 Ready. I did not check the PDB. The issue says the diagnostics SA cannot list `poddisruptionbudgets`, and I did not re-test that.
- **Not checked:** the issue's hypothesis that the k8s-1 NotReady flap came from a kube-controller-manager leader failover. The fix does not depend on it.
Closes #920
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
What this PR does
OpenBaoQuorumLostpaged as critical on one sample at ~12:27:00 UTC on 2026-09-23. Raft had not lost quorum. The rule uses pod readiness as a stand-in for raft quorum, because OpenBao is not scraped. It also hadfor: 0m, so any single evaluation with fewer than 2 Ready replicas paged at once. During a rolling node reboot, openbao-2 was evicted and stayed Pending on the cordoned k8s-2. So only openbao-0 and openbao-1 existed. Then k8s-1 flapped NotReady for about a second, and the node lifecycle controller marked openbao-0 not-ready for ~30s. That left one Ready replica, even though openbao-0 stayed the active node and kept serving. This PR addsfor: 2mto the rule and updates its comment to say why.Changes
kubernetes/apps/monitoring-rules/prometheusrule-openbao.yaml:OpenBaoQuorumLostchanged fromfor: 0mtofor: 2m, and the "No window" comment was replaced with the reasoning above. The expression, severity and other rules are unchanged.Verification
Checked live this run:
max by (pod)(kube_pod_status_ready{namespace="openbao",condition="true"})at 15s steps over 12:25–12:30:0only at 12:26:45 and 12:27:00.1throughout.{namespace="openbao",pod="openbao-0"}between 12:26 and 12:28: openbao-0 kept serving as the active node while it was marked not-ready.12:26:26.753Z [INFO] expiration: revoked lease ...auth/kubernetes/loginrequest/response at12:27:16.failed to appendEntries to: peer=... openbao-2 ... no such host, which is expected while openbao-2 was evicted.{namespace="openbao"} |~ "(?i)(leader|election|quorum|sealed|standby|active)"over 12:20–12:30: the only matches are from openbao-2 rejoining at 12:29:24 (entering follower state,vault is unsealed,unsealed with stored key). openbao-0 and openbao-1 logged no election or leader-loss lines.After Flux reconciles, confirm with
kubectl -n monitoring get prometheusrule openbao -o yamlor the Prometheus/rulespage thatOpenBaoQuorumLostshowsfor: 2m. A real loss of quorum, such as a sealed or crashed majority, lasts far longer than 2 minutes and will still page. The next rolling Talos reboot is the practical test: it should not page on a brief node flap.Notes
unauthenticated_metrics_access(or a metrics token) and adding a ServiceMonitor. It changes the HelmRelease and adds a new scrape target, so it is more than a surgical change.poddisruptionbudgets, and I did not re-test that.Closes #920