fix(monitoring): widen OpenBaoUnavailable for: 1m -> 5m (REN-131)
OpenBaoUnavailable went pending at 01:07 UTC on a single 5s HTTPS probe timeout while OpenBao was healthy, one evaluation short of paging a critical. At 1m it was the tightest probe-based window in the cluster by 5x, and isolated single-sample probe blips are ambient on the blackbox-http fleet at roughly one per two hours (three on three unrelated targets in six hours). 5m matches WoodpeckerDown in the same group. Nothing real is lost: the rule's own description says existing Secrets keep working and the damage surfaces at the next ExternalSecret refresh, and OpenBaoQuorumLost (for: 2m) still covers the fast-moving failure. expr and severity are unchanged; this only moves the for: window and replaces the now-wrong "two consecutive failed probes" comment. Co-Authored-By: Paperclip <noreply@paperclip.ing> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Paperclip
Claude Opus 5
parent
f177685dbd
commit
97b7f79b5f
@@ -34,9 +34,20 @@ spec:
|
||||
# query parameter, and an exact match would break silently if the
|
||||
# instance label ever encodes it differently.
|
||||
expr: probe_success{job="blackbox-http", instance=~"https://openbao\\.renner\\.uno/v1/sys/health.*"} == 0
|
||||
# Two consecutive failed probes: enough to skip a single lost probe
|
||||
# through Traefik, not enough to sit on a sealed store.
|
||||
for: 1m
|
||||
# Five minutes of continuously failing probes — about ten consecutive
|
||||
# samples, since the probe and the rule evaluation both run at 30s
|
||||
# (probes-blackbox.yaml, helmrelease-prometheus.yaml). Matches
|
||||
# WoodpeckerDown above.
|
||||
#
|
||||
# This was 1m, the tightest probe window in the cluster by 5x, and it
|
||||
# went pending on a single 5s HTTPS timeout with OpenBao healthy
|
||||
# (REN-131). Isolated blips are ambient on this probe fleet at roughly
|
||||
# one per two hours, so a window that clears on three samples pages on
|
||||
# jitter. The widening costs nothing real: per the description below
|
||||
# nothing breaks the moment OpenBao goes away, and OpenBaoQuorumLost
|
||||
# (prometheusrule-openbao.yaml, for: 2m) still covers the fast
|
||||
# failure, which is quorum loss rather than a lost probe.
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
|
||||
Reference in New Issue
Block a user