fix(uptime-monitor): fail only on the ESO reasons that never retry (REN-106)
The healthCheckExprs block merged in #993 treated any `Ready=False` on the ExternalSecret as Failed. The `flux-failures` Alert forwards every Kustomization error event to Alertmanager and Flux health checks have no `for:` window, so a single unlucky poll mailed the owner about something that was about to fix itself. REN-106 asked for `reason == 'SecretSyncedError'`, matching wordpress.yaml. Checked against the ESO the cluster runs (v2.11.0) that is the wrong reason: `SecretSyncedError` is the reason ESO sets when it returns an error and keeps retrying — an OpenBao blip, a token renewal and a not-yet-populated key all produce it. `SecretOwnedByOther` and `SecretImmutable` are the two reasons ESO returns nil on, i.e. where it has stopped retrying. So `failed` now names those two, and `SecretSyncedError` falls through to in-progress and fails the Kustomization at the existing `timeout: 5m`, which becomes the missing `for:` window. Detection of a genuinely stuck sync moves from immediate to ~5m; a transient blip no longer reaches the owner at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
3d6c9940d6
commit
16c0c0a7ca
@@ -71,12 +71,46 @@ spec:
|
||||
# Kustomization would report a CEL evaluation error rather than a clean
|
||||
# in-progress.
|
||||
#
|
||||
# No `inProgress` expression: Flux treats "neither failed nor current" as
|
||||
# in-progress, which is the behaviour wanted here.
|
||||
# `failed` NAMES THE TWO REASONS ESO STOPS RETRYING ON, AND DELIBERATELY LEAVES
|
||||
# OUT `SecretSyncedError` (REN-106). The first cut of this block fired on any
|
||||
# `Ready=False` at all, and that path is undampened: the `flux-failures` Alert
|
||||
# (infrastructure/flux-notifications/alerts.yaml) forwards error events from
|
||||
# every Kustomization to Alertmanager, and a Flux health check has no `for:`
|
||||
# window, so a single bad poll would have mailed the owner.
|
||||
#
|
||||
# Read off the ESO the cluster actually runs, v2.11.0 (the chart range
|
||||
# `>=2.7.0 <3.0.0` in infrastructure/external-secrets/helmrelease.yaml resolves
|
||||
# there today). On the Secret target path ESO sets `Ready=False` with exactly
|
||||
# three reasons, and what separates them is whether it intends to try again
|
||||
# (pkg/controllers/externalsecret/externalsecret_controller.go):
|
||||
# SecretSyncedError — provider read, template or update failed. ESO returns
|
||||
# the error, so controller-runtime requeues with
|
||||
# backoff. This is the reason an OpenBao blip, a token
|
||||
# renewal AND a not-yet-populated key all produce — the
|
||||
# transient case and the REN-100 case share one string,
|
||||
# so the string cannot tell them apart.
|
||||
# SecretOwnedByOther — the target Secret is owned by something else.
|
||||
# SecretImmutable — the target Secret is immutable and needs an update.
|
||||
# The last two return `nil`, commented in ESO's own source as "cant be fixed by
|
||||
# retrying": ESO has given up, nothing is in flight, so failing at once is
|
||||
# correct and cannot be a flap. Narrowing to `reason == 'SecretSyncedError'`
|
||||
# instead — REN-106's first suggestion — would have inverted that, keeping the
|
||||
# one flap-prone reason and dropping the two that cannot flap.
|
||||
#
|
||||
# A `SecretSyncedError` is dampened, not dropped: it matches neither
|
||||
# expression, so Flux reports in-progress and keeps polling until `timeout: 5m`
|
||||
# above expires and the Kustomization fails. That timeout is the `for:` window
|
||||
# this path otherwise has none of — a blip that clears inside it never reaches
|
||||
# the owner, a sync that is genuinely stuck still does, about five minutes
|
||||
# later.
|
||||
#
|
||||
# No `inProgress` expression: `current` is the only required expression, and
|
||||
# Flux's evaluator returns in-progress when nothing matches
|
||||
# (fluxcd/pkg runtime/cel, StatusEvaluator.Evaluate), which is what is wanted.
|
||||
healthCheckExprs:
|
||||
- apiVersion: external-secrets.io/v1
|
||||
kind: ExternalSecret
|
||||
failed: "has(status.conditions) && status.conditions.exists(c, c.type == 'Ready' && c.status == 'False')"
|
||||
failed: "has(status.conditions) && status.conditions.exists(c, c.type == 'Ready' && c.status == 'False' && (c.reason == 'SecretOwnedByOther' || c.reason == 'SecretImmutable'))"
|
||||
current: "has(status.conditions) && status.conditions.exists(c, c.type == 'Ready' && c.status == 'True')"
|
||||
dependsOn:
|
||||
# ESO itself, and the `uptime-monitor` namespace (declared centrally in
|
||||
|
||||
Reference in New Issue
Block a user