docs(monitoring): record why the gauge beats the counter, with the live numbers
Comment-only. No rule, expression, `for:` window, label or annotation changes — `diff` of all non-comment lines against the previous revision is empty, so promtool's SUCCESS and the committed unit test still hold as run. Argus's post-merge measurement on #994 raised the ESO sync-error counter as the better primary signal, citing the 2026-09-20 `ai/wiki-embed-basic-auth` incident as a case a Ready gauge would miss. Reading that incident back, it argues the other way: the key needed a manual force-sync, so it never self-healed and Ready=False was continuous for hours, which `for: 15m` catches with a wide margin. It was a stuck condition, not a flap. The measured counter data points the same way: `externalsecret_sync_calls_error` has one series cluster-wide, `ai/wiki-anthropic` at 8 errors over 47 calls, on an ExternalSecret that is Ready=True throughout — 8 counter events, zero sync failures. Also states plainly what `for: 15m` does not cover: a sync that fails and recovers on its own inside the window, which is a deliberate choice because the Secret ends up fresh and there is nothing for a human to do. Refs REN-103
This commit is contained in:
@@ -69,6 +69,24 @@ spec:
|
||||
# noisier signal, not the quieter one. And the gauge does
|
||||
# not flap, because it is set to 1 only by markAsFailed() and cleared by
|
||||
# the next successful reconcile. There is no unset state in between.
|
||||
#
|
||||
# The live numbers say the same thing, measured after #994 landed:
|
||||
# `externalsecret_sync_calls_error` has exactly one series in the whole
|
||||
# cluster, `ai/wiki-anthropic`, at 8 errors over 47 calls — and that
|
||||
# ExternalSecret is Ready=True and has never been otherwise in the
|
||||
# observation window. So on today's data the counter carries 8 events that
|
||||
# correspond to no sync failure at all, and the gauge carries none.
|
||||
#
|
||||
# The 2026-09-20 `ai/wiki-embed-basic-auth` incident is evidence FOR this
|
||||
# rule, not against it, though it reads the other way at first. That key
|
||||
# sat in ESO retry backoff through ~10 failures and NEEDED A MANUAL
|
||||
# force-sync (the incident is written up in
|
||||
# clusters/homelab/uptime-monitor.yaml, as the in-repo proof for REN-100).
|
||||
# Because it never self-healed, Ready=False was continuous from the first
|
||||
# failure until a human intervened — hours, not minutes. `for: 15m` catches
|
||||
# that with a wide margin. It was a stuck condition, not a flap; the flap
|
||||
# reading would only hold if the retries had eventually succeeded on their
|
||||
# own, and they did not.
|
||||
- alert: ExternalSecretSyncFailing
|
||||
expr: >-
|
||||
max by (exported_namespace, name) (
|
||||
@@ -142,3 +160,12 @@ spec:
|
||||
# reconciled emits no series at all, so nothing above can see it. That
|
||||
# needs object-existence data — kube-state-metrics custom resource
|
||||
# state — rather than ESO's own metrics, and is not in this change.
|
||||
# * A sync that fails and then succeeds on its own inside 15m. This is a
|
||||
# choice, not an oversight: the retry left the Secret holding a freshly
|
||||
# synced value, so nothing degraded and there is nothing for a human to
|
||||
# do. Alerting on it would mean paging for a self-healed event. The case
|
||||
# that would justify revisiting is the same failure recurring for days
|
||||
# without any single occurrence lasting 15m, which would show up as a
|
||||
# climbing `externalsecret_sync_calls_error` on an ExternalSecret that
|
||||
# is Ready=True. Nothing in the cluster shows that shape today, and the
|
||||
# rule for it needs a measured recurrence rate rather than a guess.
|
||||
|
||||
Reference in New Issue
Block a user