fix(monitoring): widen OpenBaoUnavailable for: 1m -> 5m (REN-131)
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful

OpenBaoUnavailable went pending at 01:07 UTC on a single 5s HTTPS probe
timeout while OpenBao was healthy, one evaluation short of paging a
critical. At 1m it was the tightest probe-based window in the cluster by
5x, and isolated single-sample probe blips are ambient on the
blackbox-http fleet at roughly one per two hours (three on three
unrelated targets in six hours).

5m matches WoodpeckerDown in the same group. Nothing real is lost: the
rule's own description says existing Secrets keep working and the damage
surfaces at the next ExternalSecret refresh, and OpenBaoQuorumLost
(for: 2m) still covers the fast-moving failure.

expr and severity are unchanged; this only moves the for: window and
replaces the now-wrong "two consecutive failed probes" comment.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
atlas-bot
2026-09-26 03:10:22 +00:00
co-authored by Paperclip Claude Opus 5
parent f177685dbd
commit 97b7f79b5f
@@ -34,9 +34,20 @@ spec:
# query parameter, and an exact match would break silently if the
# instance label ever encodes it differently.
expr: probe_success{job="blackbox-http", instance=~"https://openbao\\.renner\\.uno/v1/sys/health.*"} == 0
# Two consecutive failed probes: enough to skip a single lost probe
# through Traefik, not enough to sit on a sealed store.
for: 1m
# Five minutes of continuously failing probes — about ten consecutive
# samples, since the probe and the rule evaluation both run at 30s
# (probes-blackbox.yaml, helmrelease-prometheus.yaml). Matches
# WoodpeckerDown above.
#
# This was 1m, the tightest probe window in the cluster by 5x, and it
# went pending on a single 5s HTTPS timeout with OpenBao healthy
# (REN-131). Isolated blips are ambient on this probe fleet at roughly
# one per two hours, so a window that clears on three samples pages on
# jitter. The widening costs nothing real: per the description below
# nothing breaks the moment OpenBao goes away, and OpenBaoQuorumLost
# (prometheusrule-openbao.yaml, for: 2m) still covers the fast
# failure, which is quorum loss rather than a lost probe.
for: 5m
labels:
severity: critical
annotations: