diff --git a/kubernetes/apps/monitoring-rules/prometheusrule-external-secrets.yaml b/kubernetes/apps/monitoring-rules/prometheusrule-external-secrets.yaml index 16b8098c..668061ab 100644 --- a/kubernetes/apps/monitoring-rules/prometheusrule-external-secrets.yaml +++ b/kubernetes/apps/monitoring-rules/prometheusrule-external-secrets.yaml @@ -69,6 +69,24 @@ spec: # noisier signal, not the quieter one. And the gauge does # not flap, because it is set to 1 only by markAsFailed() and cleared by # the next successful reconcile. There is no unset state in between. + # + # The live numbers say the same thing, measured after #994 landed: + # `externalsecret_sync_calls_error` has exactly one series in the whole + # cluster, `ai/wiki-anthropic`, at 8 errors over 47 calls — and that + # ExternalSecret is Ready=True and has never been otherwise in the + # observation window. So on today's data the counter carries 8 events that + # correspond to no sync failure at all, and the gauge carries none. + # + # The 2026-09-20 `ai/wiki-embed-basic-auth` incident is evidence FOR this + # rule, not against it, though it reads the other way at first. That key + # sat in ESO retry backoff through ~10 failures and NEEDED A MANUAL + # force-sync (the incident is written up in + # clusters/homelab/uptime-monitor.yaml, as the in-repo proof for REN-100). + # Because it never self-healed, Ready=False was continuous from the first + # failure until a human intervened — hours, not minutes. `for: 15m` catches + # that with a wide margin. It was a stuck condition, not a flap; the flap + # reading would only hold if the retries had eventually succeeded on their + # own, and they did not. - alert: ExternalSecretSyncFailing expr: >- max by (exported_namespace, name) ( @@ -142,3 +160,12 @@ spec: # reconciled emits no series at all, so nothing above can see it. That # needs object-existence data — kube-state-metrics custom resource # state — rather than ESO's own metrics, and is not in this change. + # * A sync that fails and then succeeds on its own inside 15m. This is a + # choice, not an oversight: the retry left the Secret holding a freshly + # synced value, so nothing degraded and there is nothing for a human to + # do. Alerting on it would mean paging for a self-healed event. The case + # that would justify revisiting is the same failure recurring for days + # without any single occurrence lasting 15m, which would show up as a + # climbing `externalsecret_sync_calls_error` on an ExternalSecret that + # is Ready=True. Nothing in the cluster shows that shape today, and the + # rule for it needs a measured recurrence rate rather than a guess.