From bbaf0c6a0d778a7344e60e389acff053399a4bbd Mon Sep 17 00:00:00 2001 From: atlas-bot Date: Sat, 26 Sep 2026 00:38:15 +0000 Subject: [PATCH] docs(monitoring): record why the gauge beats the counter, with the live numbers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Comment-only. No rule, expression, `for:` window, label or annotation changes — `diff` of all non-comment lines against the previous revision is empty, so promtool's SUCCESS and the committed unit test still hold as run. Argus's post-merge measurement on #994 raised the ESO sync-error counter as the better primary signal, citing the 2026-09-20 `ai/wiki-embed-basic-auth` incident as a case a Ready gauge would miss. Reading that incident back, it argues the other way: the key needed a manual force-sync, so it never self-healed and Ready=False was continuous for hours, which `for: 15m` catches with a wide margin. It was a stuck condition, not a flap. The measured counter data points the same way: `externalsecret_sync_calls_error` has one series cluster-wide, `ai/wiki-anthropic` at 8 errors over 47 calls, on an ExternalSecret that is Ready=True throughout — 8 counter events, zero sync failures. Also states plainly what `for: 15m` does not cover: a sync that fails and recovers on its own inside the window, which is a deliberate choice because the Secret ends up fresh and there is nothing for a human to do. Refs REN-103 --- .../prometheusrule-external-secrets.yaml | 27 +++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/kubernetes/apps/monitoring-rules/prometheusrule-external-secrets.yaml b/kubernetes/apps/monitoring-rules/prometheusrule-external-secrets.yaml index 16b8098c..668061ab 100644 --- a/kubernetes/apps/monitoring-rules/prometheusrule-external-secrets.yaml +++ b/kubernetes/apps/monitoring-rules/prometheusrule-external-secrets.yaml @@ -69,6 +69,24 @@ spec: # noisier signal, not the quieter one. And the gauge does # not flap, because it is set to 1 only by markAsFailed() and cleared by # the next successful reconcile. There is no unset state in between. + # + # The live numbers say the same thing, measured after #994 landed: + # `externalsecret_sync_calls_error` has exactly one series in the whole + # cluster, `ai/wiki-anthropic`, at 8 errors over 47 calls — and that + # ExternalSecret is Ready=True and has never been otherwise in the + # observation window. So on today's data the counter carries 8 events that + # correspond to no sync failure at all, and the gauge carries none. + # + # The 2026-09-20 `ai/wiki-embed-basic-auth` incident is evidence FOR this + # rule, not against it, though it reads the other way at first. That key + # sat in ESO retry backoff through ~10 failures and NEEDED A MANUAL + # force-sync (the incident is written up in + # clusters/homelab/uptime-monitor.yaml, as the in-repo proof for REN-100). + # Because it never self-healed, Ready=False was continuous from the first + # failure until a human intervened — hours, not minutes. `for: 15m` catches + # that with a wide margin. It was a stuck condition, not a flap; the flap + # reading would only hold if the retries had eventually succeeded on their + # own, and they did not. - alert: ExternalSecretSyncFailing expr: >- max by (exported_namespace, name) ( @@ -142,3 +160,12 @@ spec: # reconciled emits no series at all, so nothing above can see it. That # needs object-existence data — kube-state-metrics custom resource # state — rather than ESO's own metrics, and is not in this change. + # * A sync that fails and then succeeds on its own inside 15m. This is a + # choice, not an oversight: the retry left the Secret holding a freshly + # synced value, so nothing degraded and there is nothing for a human to + # do. Alerting on it would mean paging for a self-healed event. The case + # that would justify revisiting is the same failure recurring for days + # without any single occurrence lasting 15m, which would show up as a + # climbing `externalsecret_sync_calls_error` on an ExternalSecret that + # is Ready=True. Nothing in the cluster shows that shape today, and the + # rule for it needs a measured recurrence rate rather than a guess.