Both were unmonitored. Neither is a leaf service: Woodpecker runs every cron
and deploy in this repo, and every ExternalSecret in the cluster resolves
through OpenBao.
OpenBao is probed at /v1/sys/health rather than at the root, because a sealed
instance still serves its UI. Root would answer 307 -> /ui/ -> 200 while every
secret lookup failed — the exact green-while-broken shape probes-blackbox.yaml
already warns about for the gated monitoring UIs.
standbyok=true is required, not cosmetic: the endpoint returns 200 only on the
active node and 429 on a standby, and the external route currently lands on a
standby. Measured — the bare path returns 429 right now, so an http_2xx probe
without the parameter would have been red from the first scrape. With it, 200
means initialized, unsealed and serving; 503 sealed and 501 uninitialized stay
red.
The alert matches OpenBao by instance prefix instead of the full target string,
because that target carries a query parameter and an exact match would fail
silently if the label ever encodes it differently.
Both alerts are new rules rather than additions to an existing file. Worth
noting while here: there is no generic blackbox-down rule. Only fsn1, netbird
and the two relays have alerts, so gitea, immich, auth, home, prometheus,
alerts and dash are probed and graphed but nothing fires if they go down. Not
addressed here — it needs a decision about whether to write one generic rule
and retire the per-instance ones.
claude-bot