From 16c0c0a7cad934b0a75e64bebb6fc418281d83b5 Mon Sep 17 00:00:00 2001 From: atlas-bot Date: Sat, 26 Sep 2026 00:27:54 +0000 Subject: [PATCH 1/2] fix(uptime-monitor): fail only on the ESO reasons that never retry (REN-106) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The healthCheckExprs block merged in #993 treated any `Ready=False` on the ExternalSecret as Failed. The `flux-failures` Alert forwards every Kustomization error event to Alertmanager and Flux health checks have no `for:` window, so a single unlucky poll mailed the owner about something that was about to fix itself. REN-106 asked for `reason == 'SecretSyncedError'`, matching wordpress.yaml. Checked against the ESO the cluster runs (v2.11.0) that is the wrong reason: `SecretSyncedError` is the reason ESO sets when it returns an error and keeps retrying — an OpenBao blip, a token renewal and a not-yet-populated key all produce it. `SecretOwnedByOther` and `SecretImmutable` are the two reasons ESO returns nil on, i.e. where it has stopped retrying. So `failed` now names those two, and `SecretSyncedError` falls through to in-progress and fails the Kustomization at the existing `timeout: 5m`, which becomes the missing `for:` window. Detection of a genuinely stuck sync moves from immediate to ~5m; a transient blip no longer reaches the owner at all. Co-Authored-By: Claude Opus 5 --- .../clusters/homelab/uptime-monitor.yaml | 40 +++++++++++++++++-- 1 file changed, 37 insertions(+), 3 deletions(-) diff --git a/kubernetes/clusters/homelab/uptime-monitor.yaml b/kubernetes/clusters/homelab/uptime-monitor.yaml index 395c29cd..225cb9c7 100644 --- a/kubernetes/clusters/homelab/uptime-monitor.yaml +++ b/kubernetes/clusters/homelab/uptime-monitor.yaml @@ -71,12 +71,46 @@ spec: # Kustomization would report a CEL evaluation error rather than a clean # in-progress. # - # No `inProgress` expression: Flux treats "neither failed nor current" as - # in-progress, which is the behaviour wanted here. + # `failed` NAMES THE TWO REASONS ESO STOPS RETRYING ON, AND DELIBERATELY LEAVES + # OUT `SecretSyncedError` (REN-106). The first cut of this block fired on any + # `Ready=False` at all, and that path is undampened: the `flux-failures` Alert + # (infrastructure/flux-notifications/alerts.yaml) forwards error events from + # every Kustomization to Alertmanager, and a Flux health check has no `for:` + # window, so a single bad poll would have mailed the owner. + # + # Read off the ESO the cluster actually runs, v2.11.0 (the chart range + # `>=2.7.0 <3.0.0` in infrastructure/external-secrets/helmrelease.yaml resolves + # there today). On the Secret target path ESO sets `Ready=False` with exactly + # three reasons, and what separates them is whether it intends to try again + # (pkg/controllers/externalsecret/externalsecret_controller.go): + # SecretSyncedError — provider read, template or update failed. ESO returns + # the error, so controller-runtime requeues with + # backoff. This is the reason an OpenBao blip, a token + # renewal AND a not-yet-populated key all produce — the + # transient case and the REN-100 case share one string, + # so the string cannot tell them apart. + # SecretOwnedByOther — the target Secret is owned by something else. + # SecretImmutable — the target Secret is immutable and needs an update. + # The last two return `nil`, commented in ESO's own source as "cant be fixed by + # retrying": ESO has given up, nothing is in flight, so failing at once is + # correct and cannot be a flap. Narrowing to `reason == 'SecretSyncedError'` + # instead — REN-106's first suggestion — would have inverted that, keeping the + # one flap-prone reason and dropping the two that cannot flap. + # + # A `SecretSyncedError` is dampened, not dropped: it matches neither + # expression, so Flux reports in-progress and keeps polling until `timeout: 5m` + # above expires and the Kustomization fails. That timeout is the `for:` window + # this path otherwise has none of — a blip that clears inside it never reaches + # the owner, a sync that is genuinely stuck still does, about five minutes + # later. + # + # No `inProgress` expression: `current` is the only required expression, and + # Flux's evaluator returns in-progress when nothing matches + # (fluxcd/pkg runtime/cel, StatusEvaluator.Evaluate), which is what is wanted. healthCheckExprs: - apiVersion: external-secrets.io/v1 kind: ExternalSecret - failed: "has(status.conditions) && status.conditions.exists(c, c.type == 'Ready' && c.status == 'False')" + failed: "has(status.conditions) && status.conditions.exists(c, c.type == 'Ready' && c.status == 'False' && (c.reason == 'SecretOwnedByOther' || c.reason == 'SecretImmutable'))" current: "has(status.conditions) && status.conditions.exists(c, c.type == 'Ready' && c.status == 'True')" dependsOn: # ESO itself, and the `uptime-monitor` namespace (declared centrally in -- 2.54.0 From 5f335847a36f43100ae93d816f399de264877923 Mon Sep 17 00:00:00 2001 From: atlas-bot Date: Sat, 26 Sep 2026 03:08:38 +0000 Subject: [PATCH 2/2] docs(uptime-monitor): give the third ExternalSecret status shape a verdict (REN-106) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Comment-only. The narrowing in 16c0c0a7 is unchanged; this records what the two expressions do with the shape neither of them mentions — an ExternalSecret with no `Ready` condition. Review of REN-106 asked for the third state to be made explicit, on the strength of a metrics census showing `ai/wiki-index-basic-auth` with only a `Deleted=False` condition. That series is real; the object behind it is not. ESO v2.11.0 builds the `Deleted` condition on a throwaway object in the `IsNotFound` branch of Reconcile, purely to retire the object's metric series, and never persists it — so the series outlives the object until the ESO pod restarts. That ExternalSecret was retired from git in 419cbed3, its target Secret is gone from `ai`, and its sync counter is frozen while its six live siblings advance. On a live object ESO writes conditions of type `Ready` only. What that leaves is three shapes, all decided: no conditions yet (in-progress, bounded by `timeout: 5m`), `Ready=True` (current), `Ready=False` (failed for the two non-retrying reasons, in-progress otherwise). Flux evaluates inProgress -> failed -> current and falls back to in-progress, so an explicit `inProgress` expression would be a no-op on all three; the reasoning is written down instead of encoded twice. Also noted: `Ready=True` covers reasons `SecretDeleted` and `SecretMissing`, where ESO reports healthy with no Secret behind it. Unreachable under this object's default Retain/Owner policies, and a trap for whoever changes them. Co-Authored-By: Claude Opus 5 --- .../clusters/homelab/uptime-monitor.yaml | 57 ++++++++++++++++--- 1 file changed, 48 insertions(+), 9 deletions(-) diff --git a/kubernetes/clusters/homelab/uptime-monitor.yaml b/kubernetes/clusters/homelab/uptime-monitor.yaml index 225cb9c7..5cb02c08 100644 --- a/kubernetes/clusters/homelab/uptime-monitor.yaml +++ b/kubernetes/clusters/homelab/uptime-monitor.yaml @@ -83,12 +83,12 @@ spec: # there today). On the Secret target path ESO sets `Ready=False` with exactly # three reasons, and what separates them is whether it intends to try again # (pkg/controllers/externalsecret/externalsecret_controller.go): - # SecretSyncedError — provider read, template or update failed. ESO returns - # the error, so controller-runtime requeues with - # backoff. This is the reason an OpenBao blip, a token - # renewal AND a not-yet-populated key all produce — the - # transient case and the REN-100 case share one string, - # so the string cannot tell them apart. + # SecretSyncedError — provider read, template or update failed. Almost every + # site returns the error, so controller-runtime requeues + # with backoff. This is the reason an OpenBao blip, a + # token renewal AND a not-yet-populated key all produce — + # the transient case and the REN-100 case share one + # string, so the string cannot tell them apart. # SecretOwnedByOther — the target Secret is owned by something else. # SecretImmutable — the target Secret is immutable and needs an update. # The last two return `nil`, commented in ESO's own source as "cant be fixed by @@ -97,6 +97,14 @@ spec: # instead — REN-106's first suggestion — would have inverted that, keeping the # one flap-prone reason and dropping the two that cannot flap. # + # Two details that make `failed` an allowlist of reasons rather than the + # complement of `SecretSyncedError`: one `SecretSyncedError` site + # (`ErrSecretSetCtrlRef`) also returns `nil`, so that string is not a reliable + # "still trying" signal either way; and the generic-target path (`spec.target` + # pointing at something other than a Secret) has its own reason set, + # `ResourceSyncedError`, which is not used here — this ExternalSecret targets a + # Secret — and is left to the in-progress path if it ever is. + # # A `SecretSyncedError` is dampened, not dropped: it matches neither # expression, so Flux reports in-progress and keeps polling until `timeout: 5m` # above expires and the Kustomization fails. That timeout is the `for:` window @@ -104,9 +112,40 @@ spec: # the owner, a sync that is genuinely stuck still does, about five minutes # later. # - # No `inProgress` expression: `current` is the only required expression, and - # Flux's evaluator returns in-progress when nothing matches - # (fluxcd/pkg runtime/cel, StatusEvaluator.Evaluate), which is what is wanted. + # EVERY SHAPE AN ExternalSecret STATUS CAN TAKE GETS A VERDICT FROM THE TWO + # EXPRESSIONS BELOW. There are three, not two, and the third one — no `Ready` + # condition at all — is the one a reader is most likely to miss: + # + # 1. no conditions yet (applied, first reconcile unfinished): the `has()` + # guard makes both expressions false -> in-progress, bounded by + # `timeout: 5m`. + # 2. `Ready=True`: `current` -> Current. Note this includes reasons + # `SecretDeleted` and `SecretMissing`, which ESO reports as Ready=True + # (markAsDone) even though no target Secret exists. Reaching them needs + # `deletionPolicy: Delete`/`Merge` or `creationPolicy: Merge`; this + # ExternalSecret takes the defaults (Retain/Owner), where a vanished + # provider key is a `SecretSyncedError` instead. Anyone who later sets + # either policy here reopens REN-100 through this expression. + # 3. `Ready=False`: `failed` for the two non-retrying reasons above, + # in-progress for the rest. + # + # No `inProgress` expression: only `apiVersion` and `current` are required by + # the CRD (flux-system/gotk-components.yaml, kustomize-controller v1.9.5), and + # Flux evaluates inProgress -> failed -> current and returns in-progress when + # nothing matches (fluxcd/pkg runtime/cel, StatusEvaluator.Evaluate). An + # `inProgress` naming shapes 1 and 3 would therefore be a no-op on all three. + # + # "conditions present but no `Ready` condition" is NOT a fourth shape (REN-106). + # On a live object ESO only ever writes conditions of type `Ready`; the + # `Deleted` type is never persisted — it is built on a throwaway object in the + # `IsNotFound` branch of Reconcile purely to retire that object's metric series + # ("this does not actually set the condition on the ExternalSecret, because it + # does not exist"). So `externalsecret_status_condition{condition="Deleted"}` + # outlives the object it names, until the ESO pod restarts. + # `ai/wiki-index-basic-auth` is that: retired from git in 419cbed3, target + # Secret gone from the namespace, sync counter frozen, still a series in an ESO + # pod older than the prune. Any alert built on this metric (REN-103) has to + # exclude that shape; this health check never sees it. healthCheckExprs: - apiVersion: external-secrets.io/v1 kind: ExternalSecret -- 2.54.0