fix(uptime-monitor): fail only on the ESO reasons that never retry (REN-106)
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful

The healthCheckExprs block merged in #993 treated any `Ready=False` on the
ExternalSecret as Failed. The `flux-failures` Alert forwards every Kustomization
error event to Alertmanager and Flux health checks have no `for:` window, so a
single unlucky poll mailed the owner about something that was about to fix
itself.

REN-106 asked for `reason == 'SecretSyncedError'`, matching wordpress.yaml.
Checked against the ESO the cluster runs (v2.11.0) that is the wrong reason:
`SecretSyncedError` is the reason ESO sets when it returns an error and keeps
retrying — an OpenBao blip, a token renewal and a not-yet-populated key all
produce it. `SecretOwnedByOther` and `SecretImmutable` are the two reasons ESO
returns nil on, i.e. where it has stopped retrying.

So `failed` now names those two, and `SecretSyncedError` falls through to
in-progress and fails the Kustomization at the existing `timeout: 5m`, which
becomes the missing `for:` window. Detection of a genuinely stuck sync moves
from immediate to ~5m; a transient blip no longer reaches the owner at all.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
atlas-bot
2026-09-26 00:27:54 +00:00
co-authored by Claude Opus 5
parent 3d6c9940d6
commit 16c0c0a7ca
@@ -71,12 +71,46 @@ spec:
# Kustomization would report a CEL evaluation error rather than a clean
# in-progress.
#
# No `inProgress` expression: Flux treats "neither failed nor current" as
# in-progress, which is the behaviour wanted here.
# `failed` NAMES THE TWO REASONS ESO STOPS RETRYING ON, AND DELIBERATELY LEAVES
# OUT `SecretSyncedError` (REN-106). The first cut of this block fired on any
# `Ready=False` at all, and that path is undampened: the `flux-failures` Alert
# (infrastructure/flux-notifications/alerts.yaml) forwards error events from
# every Kustomization to Alertmanager, and a Flux health check has no `for:`
# window, so a single bad poll would have mailed the owner.
#
# Read off the ESO the cluster actually runs, v2.11.0 (the chart range
# `>=2.7.0 <3.0.0` in infrastructure/external-secrets/helmrelease.yaml resolves
# there today). On the Secret target path ESO sets `Ready=False` with exactly
# three reasons, and what separates them is whether it intends to try again
# (pkg/controllers/externalsecret/externalsecret_controller.go):
# SecretSyncedError — provider read, template or update failed. ESO returns
# the error, so controller-runtime requeues with
# backoff. This is the reason an OpenBao blip, a token
# renewal AND a not-yet-populated key all produce — the
# transient case and the REN-100 case share one string,
# so the string cannot tell them apart.
# SecretOwnedByOther — the target Secret is owned by something else.
# SecretImmutable — the target Secret is immutable and needs an update.
# The last two return `nil`, commented in ESO's own source as "cant be fixed by
# retrying": ESO has given up, nothing is in flight, so failing at once is
# correct and cannot be a flap. Narrowing to `reason == 'SecretSyncedError'`
# instead — REN-106's first suggestion — would have inverted that, keeping the
# one flap-prone reason and dropping the two that cannot flap.
#
# A `SecretSyncedError` is dampened, not dropped: it matches neither
# expression, so Flux reports in-progress and keeps polling until `timeout: 5m`
# above expires and the Kustomization fails. That timeout is the `for:` window
# this path otherwise has none of — a blip that clears inside it never reaches
# the owner, a sync that is genuinely stuck still does, about five minutes
# later.
#
# No `inProgress` expression: `current` is the only required expression, and
# Flux's evaluator returns in-progress when nothing matches
# (fluxcd/pkg runtime/cel, StatusEvaluator.Evaluate), which is what is wanted.
healthCheckExprs:
- apiVersion: external-secrets.io/v1
kind: ExternalSecret
failed: "has(status.conditions) && status.conditions.exists(c, c.type == 'Ready' && c.status == 'False')"
failed: "has(status.conditions) && status.conditions.exists(c, c.type == 'Ready' && c.status == 'False' && (c.reason == 'SecretOwnedByOther' || c.reason == 'SecretImmutable'))"
current: "has(status.conditions) && status.conditions.exists(c, c.type == 'Ready' && c.status == 'True')"
dependsOn:
# ESO itself, and the `uptime-monitor` namespace (declared centrally in