docs(monitoring): record why the gauge beats the counter, with the live numbers
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was canceled

Comment-only. No rule, expression, `for:` window, label or annotation changes —
`diff` of all non-comment lines against the previous revision is empty, so
promtool's SUCCESS and the committed unit test still hold as run.

Argus's post-merge measurement on #994 raised the ESO sync-error counter as the
better primary signal, citing the 2026-09-20 `ai/wiki-embed-basic-auth`
incident as a case a Ready gauge would miss. Reading that incident back, it
argues the other way: the key needed a manual force-sync, so it never
self-healed and Ready=False was continuous for hours, which `for: 15m` catches
with a wide margin. It was a stuck condition, not a flap.

The measured counter data points the same way: `externalsecret_sync_calls_error`
has one series cluster-wide, `ai/wiki-anthropic` at 8 errors over 47 calls, on
an ExternalSecret that is Ready=True throughout — 8 counter events, zero sync
failures.

Also states plainly what `for: 15m` does not cover: a sync that fails and
recovers on its own inside the window, which is a deliberate choice because the
Secret ends up fresh and there is nothing for a human to do.

Refs REN-103
This commit is contained in:
atlas-bot
2026-09-26 00:38:15 +00:00
parent 5f0a491d97
commit bbaf0c6a0d
@@ -69,6 +69,24 @@ spec:
# noisier signal, not the quieter one. And the gauge does
# not flap, because it is set to 1 only by markAsFailed() and cleared by
# the next successful reconcile. There is no unset state in between.
#
# The live numbers say the same thing, measured after #994 landed:
# `externalsecret_sync_calls_error` has exactly one series in the whole
# cluster, `ai/wiki-anthropic`, at 8 errors over 47 calls — and that
# ExternalSecret is Ready=True and has never been otherwise in the
# observation window. So on today's data the counter carries 8 events that
# correspond to no sync failure at all, and the gauge carries none.
#
# The 2026-09-20 `ai/wiki-embed-basic-auth` incident is evidence FOR this
# rule, not against it, though it reads the other way at first. That key
# sat in ESO retry backoff through ~10 failures and NEEDED A MANUAL
# force-sync (the incident is written up in
# clusters/homelab/uptime-monitor.yaml, as the in-repo proof for REN-100).
# Because it never self-healed, Ready=False was continuous from the first
# failure until a human intervened — hours, not minutes. `for: 15m` catches
# that with a wide margin. It was a stuck condition, not a flap; the flap
# reading would only hold if the retries had eventually succeeded on their
# own, and they did not.
- alert: ExternalSecretSyncFailing
expr: >-
max by (exported_namespace, name) (
@@ -142,3 +160,12 @@ spec:
# reconciled emits no series at all, so nothing above can see it. That
# needs object-existence data — kube-state-metrics custom resource
# state — rather than ESO's own metrics, and is not in this change.
# * A sync that fails and then succeeds on its own inside 15m. This is a
# choice, not an oversight: the retry left the Secret holding a freshly
# synced value, so nothing degraded and there is nothing for a human to
# do. Alerting on it would mean paging for a self-healed event. The case
# that would justify revisiting is the same failure recurring for days
# without any single occurrence lasting 15m, which would show up as a
# climbing `externalsecret_sync_calls_error` on an ExternalSecret that
# is Ready=True. Nothing in the cluster shows that shape today, and the
# rule for it needs a measured recurrence rate rather than a guess.