feat(monitoring): Flux per-object readiness via kube-state-metrics (REN-102) #1013

2026-09-26T03:29:18Z

Ghost commented

2026-09-26 03:29:18 +00:00

Closes the readiness half of REN-97. Follows #992 (the scrape), which is merged. Argus's pre-PR design review is REN-123; findings 1–3 are fixed in the code, 4–6 are answered below.

What changed

Four files.

kubernetes/apps/monitoring/helmrelease-prometheus.yaml — kube-state-metrics.customResourceState over eight Flux kinds, plus rbac.extraRules.
kubernetes/apps/monitoring-rules/prometheusrule-flux.yaml — new. Three not-Ready alerts, four recording rules.
kubernetes/apps/monitoring-rules/kustomization.yaml — one entry.
kubernetes/infrastructure/image-automation/README.md — the "What watches this" section said plainly that both cases were uncovered.

Why it is kube-state-metrics and not the controllers

gotk_reconcile_condition and gotk_suspend_status are still declared by fluxcd/pkg/runtime, but no controller populates them at the tags pinned in gotk-components.yaml, and an unpopulated GaugeVec exports no series. Re-verified now that #992 is live: the PodMonitor yields 2,273 gotk_* series, and count({__name__=~"gotk_reconcile_condition|gotk_suspend_status"}) is still empty. All of the 2,273 are gotk_reconcile_duration_seconds*, gotk_token_* and gotk_event_http_*.

Kinds, and the four that are deliberately omitted

Included (8): Kustomization, HelmRelease, GitRepository, HelmRepository, OCIRepository, ImageRepository, ImagePolicy, ImageUpdateAutomation.

Omitted	Reason
`Bucket`, `Receiver`	zero objects exist in this repo
`HelmChart`	created by source-controller at runtime, one per HelmRelease; a not-Ready HelmChart is always mirrored by its owning HelmRelease, so alerting on both is one failure reported twice
`Alert`, `Provider`	`notification.toolkit.fluxcd.io/v1beta3` Alert and Provider have no `status.conditions` in the CRD at all. Checked in `gotk-components.yaml`. There is no readiness to report, which is why upstream omits `ready` for them too. This corrects the plan, which had listed them as worth including.

Findings 1–3: fixed, and each one is now an executable check

1 — the >= 2600 guard exceeded available history. Re-measured today: (time() - prometheus_tsdb_lowest_timestamp/1000)/86400 = 26.63 d, prometheus_tsdb_storage_blocks_bytes = 27.22 GB against a 32.21 GB limit (84.5%), growing ~1.09 GB/day. I took the first of the two remedies offered: the window is 14 d, not 30 d, and the guard is >= 1250 at a declared interval: 15m — 93% of the 1344 samples a full window holds, so 13.02 d of history against a ≥26.6 d floor, 2.0× margin. The arithmetic is in a comment next to the constant, as asked.

No companion "window exceeds retention" alert: it is unnecessary once the window fits, and it would mail every 12 h for 14 days after any Prometheus PVC recreation. Say the word if you would rather have it.

2 — the distinct-tag counter counted KSM target labels. Fixed with the first-stage record you specified. It is now proved both ways in a promtool unit test: across a simulated pod roll, count(count_over_time(gotk_resource_info{...}[1d])) reads 2 while flux:image_policy_tags_seen:1d reads 1; a genuine tag change still takes it to 2. The max by (...) collapse on the three not-Ready rules is kept, and a test proves the for: 2h clock survives a pod-identity change mid-window.

3 — the 15m interval lived only in prose. interval: 15m is now declared on the recording group, with a comment tying 1250 to it and noting that 45 of the 46 currently loaded groups run at the global 30s.

Findings 4–6: answered

4 — the comparator was direction-agnostic. Replaced max/min >= 1.10 with last_over_time(...[1h]) - min_over_time(...[14d]) >= 1, which only counts net growth. Proved: a pruning event 3299 → 2900 makes the old ratio read 1.1376 (it would have fired) while the new form yields −399 and the candidate series stays empty.

5 — the join matched by naming convention. Fixed properly rather than asserted. source_name: [spec, imageRepositoryRef, name] is now a label on the ImagePolicy series, the repository side is label_replaced onto the same key, and the join is on (source_name, exported_namespace). I re-confirmed your count independently: 34 pairs, 0 mismatches — so this is drift insurance, not a bug fix, and a differently-named policy no longer drops out silently.

6 — gotk_image_policy_ready_transition_timestamp_seconds had no consumer. Dropped. The card asked for it; §4a's rejection of wall-clock superseded the request. 34 series not added.

The staleness half: what I am NOT shipping, and the measurements that forced it

Scope item 4 asked for one staleness alert. This PR ships the recording rules and no alert. That is a deliberate reduction, not an oversight, and it is your call to overrule.

Every candidate threshold was tested against the 191 automated bump commits and against the upstream registries:

Candidate	Result
flat "frozen for N days"	No N exists. gatus went 127 d between bumps and paperless-ngx 126 d, both provably healthy. Six policies have never bumped in 147 d. Any N < 127 fires on the healthy pair; any N > 147 exceeds the observation window.
tag count grew ≥ 10 % in 14 d (the plan's number)	Measurably wrong. Of the 18 policies frozen ≥14 d it fires on exactly three — gotenberg (+87 tags/14 d on 526), valkey (+68 on 301), eclipse-mosquitto (+13 on 90) — all three churning variant tags their filters reject on purpose. It does not fire on tika (+4 on 136 = 3.0 %), the canonical case this card exists for.
tag count grew ≥ 1	Catches tika, but adds velero-plugin-for-aws, whose `1.13.x` pin is deliberate, on top of the same three.
`.status.lastScanResult.latestTags`	Unusable: the CRD defines it as "the first 10 tags when sorting all the tags in descending alphabetical order". Not newest-first for any tag scheme here.

And the cost of getting it wrong is concrete: Alertmanager routes every severity to the same email receiver with repeat_interval: 12h, so the ≥1-tag variant means four standing mails every twelve hours, indefinitely, for four non-faults.

I also have to correct my own plan here: §4b rejected an absolute tag delta because immich and home-assistant would "meet it instantly and permanently". That reasoned about the growth condition in isolation — in conjunction with a 14-day frozen tag those policies never qualify, because they bump. The real reason no threshold works is the variant-churn false positives above.

So the four recording rules materialise the candidate set and start the per-repository tagCount history that a threshold has to be derived from — a series that has never existed here. Recording rules cannot mail anyone, so this is silent while it warms up. A follow-up card attaches the alert once there are 14+ days of the real series.

RBAC

list and watch only. No get — KSM drives reflectors and never point-reads. The rendered ClusterRole was checked: every verb in the whole kube-prometheus-stack-kube-state-metrics ClusterRole, built-ins included, is list or watch. Nothing else.

One added permission to disclose beyond my extraRules: the subchart adds apiextensions.k8s.io/customresourcedefinitions: [list, watch] itself via rbac.customResourceState.createClusterRoleRules (default true). KSM needs it to resolve the GVKs — verified, it will not start the custom-resource stores without it.

Cardinality

Source	Series
`gotk_resource_info` (149 objects across the 8 kinds)	149
`gotk_image_repository_tag_count`	34
recorded (`tag:info`, `tags_seen:1d`, `tag_count:1d`)	~102
Total	~285

+0.12 % of the 238,242 head series, +0.66 % of the 27,792 on the KSM scrape. Ongoing churn is the tag label alone: 191 bumps in 147 days = 1.3 new series/day, which ages out with retention. The recorded series are evaluated at 15m (96 samples/day) rather than the 30s scrape rate.

Dropped from upstream's label set to keep it that small: revision (a git SHA — a new series per commit reaching a Kustomization, on a TSDB at 84.5 % of cap), url, endpoint.

One thing I added that was not in the reviewed plan

errorLogV: 10 on all eight resources. spec.suspend has no default: in the Flux CRDs and no Flux object in this repo sets it, so those label paths do not resolve on nearly every object, and KSM's docs say an unresolved label path logs an error — at 30 s scrapes over ~149 objects that is a large amount of noise shipped straight into Loki. The cost of quieting it is that a genuinely wrong path also becomes quiet, which is exactly why FluxImagePolicyMetricsMissing exists as a structural check rather than relying on logs. Startup config-parse failures are unaffected and still logged.

The same finding is why the rules use suspended!="true" rather than an unless ... suspended="false" join — verified against the real binary, the label is absent, not "false", so the ="false" form would have matched nothing.

Alert tuning, from the corrected census

Re-counted independently: ImageRepository 1h×31, 30m×2 (shelly-manager, vsftpd), 10m×1 (wiki); Kustomization 33 all 10m; HelmRelease 25 all 30m. Matches your numbers.

FluxImageAutomationNotReady, for: 2h — 91 % of ImageRepositories scan at 1h, so anything under 1h mails about a single self-healing missed scan; 2h is two consecutive failed scans.
FluxResourceNotReady, for: 1h — six Kustomization retries or two HelmRelease intervals inside the hour. Known trade, stated rather than hidden: this matcher also covers the 17 HelmRepository and 4 OCIRepository objects, and all 21 are at interval: 24h, so a transient fetch failure there cannot self-heal inside an hour and yields up to two mails. Accepted as correct information; if it proves noisy the fix is a third rule for the 24h kinds, not a global bump.
ready=~"False|Unknown", not ready="False" — a change from the withdrawn draft. Flux sets Unknown while reconciling, and an object wedged in Unknown is stuck just as hard; this repo has already had a Progressing CephCluster deadlock the apps layer. Written as an explicit two-value match rather than !="True" so a series whose ready label is absent entirely stays out.
FluxImagePolicyMetricsMissing, absent(), for: 30m — the rule you flagged. Safe only because this change creates the 34 ImagePolicy series it asserts on; merged against gotk_* alone it would have mailed every 12 h forever. 30m rides out the KSM roll this change itself causes.

Verification — what was actually run, and what was not

Nothing was applied to the cluster. I have read-only Prometheus through the REN-36 proxy and no write access; the paperclip SA cannot even get the KSM Deployment. So this is not verified against the live cluster, and Flux applying it on merge is the first real application. What I did run, offline:

The real kube-state-metrics v2.20.0 binary — extracted from registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.20.0, the exact image the live pod runs — fed the helm-rendered config, against a stub apiserver serving canned Flux objects. It produced exactly the intended series, with no config or path errors:
```
gotk_resource_info{customresource_kind="ImagePolicy",name="tika",exported_namespace="flux-system",
  ready="True",source_name="tika",tag="3.0.0.0-full"} 1
gotk_resource_info{customresource_kind="ImagePolicy",name="navidrome",...,ready="False",
  source_name="navidrome",suspended="true",tag="0.64.0"} 1
gotk_resource_info{customresource_kind="ImagePolicy",name="fresh",source_name="fresh"} 1
gotk_resource_info{customresource_kind="ImageRepository",name="navidrome",...,ready="Unknown"} 1
gotk_image_repository_tag_count{...,name="tika",exported_namespace="flux-system"} 136
gotk_image_repository_tag_count{...,name="navidrome",exported_namespace="flux-system"} 3299
```
Note the three things this proved that reading YAML would not: suspended is absent rather than "false"; ready="Unknown" is real; and the no-status policy (fresh) emits a series with neither ready nor tag, so it is correctly outside the alerts.

This caught a real bug in my own config. The tag_count Gauge originally had labelsFromPath: {name: [metadata, name]} nested under gauge, where KSM resolves it relative to gauge.path — it would have emitted the metric with no name label and the join would have matched nothing. Moved to the metric level and re-verified.
Run again in the exact argument shape the chart renders — --resources=<28 built-in collectors> present, --custom-resource-state-only absent: 11 gotk_* series alongside 72 active built-in stores. This is the check that kube_* survives.
promtool check rules → 7 rules. promtool test rules → 10 unit tests, all passing, covering: the 2h/1h/30m thresholds firing and not firing, the KSM-pod-roll immunity both ways, suspended suppression, Unknown detection, the exhaustive partition (no kind claimed by both rules or neither), the absent-suspended-label shape, the no-status policy, the 14-day warm-up boundary (empty at 12.5 d, populated at 13.23 d), and the tag-count-drop case. Fixtures are attached to REN-102 rather than committed — there is no promtool step in CI yet and no established home for them; that is REN-96's call, not this PR's.
helm template against kube-prometheus-stack 91.5.3 / kube-state-metrics 8.5.0 (what >=72.0.0 resolves to): --custom-resource-state-config-file arg added, ConfigMap rendered, volume mounted, --resources untouched.
kustomize build over every kustomization in kubernetes/ — the CI gate — all green.

Post-merge, to be captured on REN-102: KSM pod Ready after the roll; count({job="kube-state-metrics"}) back at ~27,792 + 183; count({__name__=~"gotk_resource_info"}) = 149; gotk_resource_info{customresource_kind="ImagePolicy"} carrying tag; and the both-directions check — suspend one ImagePolicy, watch suspended="true" suppress the rule, then unsuspend.

Blast radius

What this touches: the kube-state-metrics values and its ClusterRole; one new PrometheusRule.

What breaks if it goes wrong: a customResourceStateConfig that KSM rejects crash-loops the pod, and all 27,792 kube_* series stop, blinding every Kube* alert in the chart. Nothing else: KSM only watches, it writes nothing, and no Flux object, ImagePolicy, workload or ingress controller is touched. Enabling the block rolls the KSM pod exactly once.

Who notices: KubeStateMetricsDown and TargetDown from the chart's own rules, plus Iris on REN-85.

Note for future edits: the subchart puts no config checksum on the pod template, so editing the config later changes the ConfigMap without rolling the pod and KSM keeps serving the old config until it restarts. Called out in a comment in the file.

Revert

Stronger than I claimed on the plan, and you were right to say so. Two layers:

Automatic, no human. The HelmRelease sets upgrade.remediation.retries: 3 with no disableWait. A KSM that crash-loops on a bad config fails the Helm upgrade, and Helm rolls the release back on its own — kube_* returns without anyone intervening.
Manual. git revert the merge commit. That produces a new revision Flux applies cleanly; the KSM pod rolls once more and returns to the current 27,792 series. The ClusterRole rules and the PrometheusRule are both garbage-collected by their Kustomizations, so there is no orphan. One step, one commit.

Do not self-merge — over to Argus.

🤖 Generated with Claude Code

Closes the readiness half of [REN-97](/REN/issues/REN-97). Follows [#992](https://gitea.renner.uno/k.renner/homelab/pulls/992) (the scrape), which is merged. Argus's pre-PR design review is [REN-123](/REN/issues/REN-123); findings 1–3 are fixed in the code, 4–6 are answered below. ## What changed Four files. 1. `kubernetes/apps/monitoring/helmrelease-prometheus.yaml` — `kube-state-metrics.customResourceState` over **eight** Flux kinds, plus `rbac.extraRules`. 2. `kubernetes/apps/monitoring-rules/prometheusrule-flux.yaml` — new. Three not-Ready alerts, four recording rules. 3. `kubernetes/apps/monitoring-rules/kustomization.yaml` — one entry. 4. `kubernetes/infrastructure/image-automation/README.md` — the "What watches this" section said plainly that both cases were uncovered. ## Why it is kube-state-metrics and not the controllers `gotk_reconcile_condition` and `gotk_suspend_status` are still declared by `fluxcd/pkg/runtime`, but no controller populates them at the tags pinned in `gotk-components.yaml`, and an unpopulated `GaugeVec` exports no series. Re-verified now that #992 is live: the PodMonitor yields **2,273 `gotk_*` series**, and `count({__name__=~"gotk_reconcile_condition|gotk_suspend_status"})` is still **empty**. All of the 2,273 are `gotk_reconcile_duration_seconds*`, `gotk_token_*` and `gotk_event_http_*`. ## Kinds, and the four that are deliberately omitted Included (8): Kustomization, HelmRelease, GitRepository, HelmRepository, OCIRepository, ImageRepository, ImagePolicy, ImageUpdateAutomation. | Omitted | Reason | |---|---| | `Bucket`, `Receiver` | zero objects exist in this repo | | `HelmChart` | created by source-controller at runtime, one per HelmRelease; a not-Ready HelmChart is always mirrored by its owning HelmRelease, so alerting on both is one failure reported twice | | `Alert`, `Provider` | **`notification.toolkit.fluxcd.io/v1beta3` Alert and Provider have no `status.conditions` in the CRD at all.** Checked in `gotk-components.yaml`. There is no readiness to report, which is why upstream omits `ready` for them too. This corrects the plan, which had listed them as worth including. | ## Findings 1–3: fixed, and each one is now an executable check **1 — the `>= 2600` guard exceeded available history.** Re-measured today: `(time() - prometheus_tsdb_lowest_timestamp/1000)/86400` = **26.63 d**, `prometheus_tsdb_storage_blocks_bytes` = 27.22 GB against a 32.21 GB limit (**84.5%**), growing ~1.09 GB/day. I took the first of the two remedies offered: **the window is 14 d, not 30 d**, and the guard is `>= 1250` at a declared `interval: 15m` — 93% of the 1344 samples a full window holds, so **13.02 d of history against a ≥26.6 d floor, 2.0× margin**. The arithmetic is in a comment next to the constant, as asked. No companion "window exceeds retention" alert: it is unnecessary once the window fits, and it would mail every 12 h for 14 days after any Prometheus PVC recreation. Say the word if you would rather have it. **2 — the distinct-tag counter counted KSM target labels.** Fixed with the first-stage record you specified. It is now proved both ways in a promtool unit test: across a simulated pod roll, `count(count_over_time(gotk_resource_info{...}[1d]))` reads **2** while `flux:image_policy_tags_seen:1d` reads **1**; a genuine tag change still takes it to **2**. The `max by (...)` collapse on the three not-Ready rules is kept, and a test proves the `for: 2h` clock survives a pod-identity change mid-window. **3 — the `15m` interval lived only in prose.** `interval: 15m` is now declared on the recording group, with a comment tying `1250` to it and noting that 45 of the 46 currently loaded groups run at the global 30s. ## Findings 4–6: answered **4 — the comparator was direction-agnostic.** Replaced `max/min >= 1.10` with `last_over_time(...[1h]) - min_over_time(...[14d]) >= 1`, which only counts net growth. Proved: a pruning event 3299 → 2900 makes the old ratio read **1.1376** (it would have fired) while the new form yields −399 and the candidate series stays empty. **5 — the join matched by naming convention.** Fixed properly rather than asserted. `source_name: [spec, imageRepositoryRef, name]` is now a label on the ImagePolicy series, the repository side is `label_replace`d onto the same key, and the join is `on (source_name, exported_namespace)`. I re-confirmed your count independently: 34 pairs, 0 mismatches — so this is drift insurance, not a bug fix, and a differently-named policy no longer drops out silently. **6 — `gotk_image_policy_ready_transition_timestamp_seconds` had no consumer.** **Dropped.** The card asked for it; §4a's rejection of wall-clock superseded the request. 34 series not added. ## The staleness half: what I am NOT shipping, and the measurements that forced it **Scope item 4 asked for one staleness alert. This PR ships the recording rules and no alert.** That is a deliberate reduction, not an oversight, and it is your call to overrule. Every candidate threshold was tested against the 191 automated bump commits and against the upstream registries: | Candidate | Result | |---|---| | flat "frozen for N days" | No N exists. gatus went **127 d** between bumps and paperless-ngx **126 d**, both provably healthy. Six policies have never bumped in 147 d. Any N < 127 fires on the healthy pair; any N > 147 exceeds the observation window. | | tag count grew **≥ 10 %** in 14 d (the plan's number) | **Measurably wrong.** Of the 18 policies frozen ≥14 d it fires on exactly three — gotenberg (+87 tags/14 d on 526), valkey (+68 on 301), eclipse-mosquitto (+13 on 90) — all three churning variant tags their filters reject on purpose. It does **not** fire on **tika (+4 on 136 = 3.0 %)**, the canonical case this card exists for. | | tag count grew **≥ 1** | Catches tika, but adds velero-plugin-for-aws, whose `1.13.x` pin is deliberate, on top of the same three. | | `.status.lastScanResult.latestTags` | Unusable: the CRD defines it as "the first 10 tags when sorting all the tags in descending **alphabetical** order". Not newest-first for any tag scheme here. | And the cost of getting it wrong is concrete: **Alertmanager routes every severity to the same `email` receiver with `repeat_interval: 12h`**, so the ≥1-tag variant means four standing mails every twelve hours, indefinitely, for four non-faults. I also have to correct my own plan here: §4b rejected an absolute tag delta because immich and home-assistant would "meet it instantly and permanently". That reasoned about the growth condition in isolation — in conjunction with a 14-day frozen tag those policies never qualify, because they bump. The real reason no threshold works is the variant-churn false positives above. So the four recording rules materialise the candidate set **and start the per-repository `tagCount` history that a threshold has to be derived from** — a series that has never existed here. Recording rules cannot mail anyone, so this is silent while it warms up. A follow-up card attaches the alert once there are 14+ days of the real series. ## RBAC `list` and `watch` only. **No `get`** — KSM drives reflectors and never point-reads. The rendered ClusterRole was checked: every verb in the whole `kube-prometheus-stack-kube-state-metrics` ClusterRole, built-ins included, is `list` or `watch`. Nothing else. One added permission to disclose beyond my `extraRules`: the subchart adds `apiextensions.k8s.io/customresourcedefinitions: [list, watch]` itself via `rbac.customResourceState.createClusterRoleRules` (default true). KSM needs it to resolve the GVKs — verified, it will not start the custom-resource stores without it. ## Cardinality | Source | Series | |---|---| | `gotk_resource_info` (149 objects across the 8 kinds) | 149 | | `gotk_image_repository_tag_count` | 34 | | recorded (`tag:info`, `tags_seen:1d`, `tag_count:1d`) | ~102 | | **Total** | **~285** | **+0.12 % of the 238,242 head series, +0.66 % of the 27,792 on the KSM scrape.** Ongoing churn is the `tag` label alone: 191 bumps in 147 days = **1.3 new series/day**, which ages out with retention. The recorded series are evaluated at 15m (96 samples/day) rather than the 30s scrape rate. Dropped from upstream's label set to keep it that small: `revision` (a git SHA — a new series per commit reaching a Kustomization, on a TSDB at 84.5 % of cap), `url`, `endpoint`. ## One thing I added that was not in the reviewed plan `errorLogV: 10` on all eight resources. `spec.suspend` has **no `default:` in the Flux CRDs** and **no Flux object in this repo sets it**, so those label paths do not resolve on nearly every object, and KSM's docs say an unresolved *label* path logs an error — at 30 s scrapes over ~149 objects that is a large amount of noise shipped straight into Loki. The cost of quieting it is that a genuinely wrong path also becomes quiet, which is exactly why `FluxImagePolicyMetricsMissing` exists as a structural check rather than relying on logs. Startup config-parse failures are unaffected and still logged. The same finding is why the rules use `suspended!="true"` rather than an `unless ... suspended="false"` join — verified against the real binary, the label is **absent**, not `"false"`, so the `="false"` form would have matched nothing. ## Alert tuning, from the corrected census Re-counted independently: ImageRepository `1h`×31, `30m`×2 (shelly-manager, vsftpd), `10m`×1 (wiki); Kustomization 33 all `10m`; HelmRelease 25 all `30m`. Matches your numbers. - **`FluxImageAutomationNotReady`, `for: 2h`** — 91 % of ImageRepositories scan at 1h, so anything under 1h mails about a single self-healing missed scan; 2h is two consecutive failed scans. - **`FluxResourceNotReady`, `for: 1h`** — six Kustomization retries or two HelmRelease intervals inside the hour. **Known trade, stated rather than hidden:** this matcher also covers the 17 HelmRepository and 4 OCIRepository objects, and all 21 are at `interval: 24h`, so a transient fetch failure there cannot self-heal inside an hour and yields up to two mails. Accepted as correct information; if it proves noisy the fix is a third rule for the 24h kinds, not a global bump. - **`ready=~"False|Unknown"`, not `ready="False"`** — a change from the withdrawn draft. Flux sets Unknown while reconciling, and an object wedged in Unknown is stuck just as hard; this repo has already had a Progressing CephCluster deadlock the apps layer. Written as an explicit two-value match rather than `!="True"` so a series whose `ready` label is absent entirely stays out. - **`FluxImagePolicyMetricsMissing`, `absent()`, `for: 30m`** — the rule you flagged. Safe only because this change creates the 34 ImagePolicy series it asserts on; merged against `gotk_*` alone it would have mailed every 12 h forever. `30m` rides out the KSM roll this change itself causes. ## Verification — what was actually run, and what was not **Nothing was applied to the cluster.** I have read-only Prometheus through the REN-36 proxy and no write access; the paperclip SA cannot even `get` the KSM Deployment. So this is *not* verified against the live cluster, and Flux applying it on merge is the first real application. What I did run, offline: 1. **The real `kube-state-metrics` v2.20.0 binary** — extracted from `registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.20.0`, the exact image the live pod runs — fed the **helm-rendered** config, against a stub apiserver serving canned Flux objects. It produced exactly the intended series, with no config or path errors: ``` gotk_resource_info{customresource_kind="ImagePolicy",name="tika",exported_namespace="flux-system", ready="True",source_name="tika",tag="3.0.0.0-full"} 1 gotk_resource_info{customresource_kind="ImagePolicy",name="navidrome",...,ready="False", source_name="navidrome",suspended="true",tag="0.64.0"} 1 gotk_resource_info{customresource_kind="ImagePolicy",name="fresh",source_name="fresh"} 1 gotk_resource_info{customresource_kind="ImageRepository",name="navidrome",...,ready="Unknown"} 1 gotk_image_repository_tag_count{...,name="tika",exported_namespace="flux-system"} 136 gotk_image_repository_tag_count{...,name="navidrome",exported_namespace="flux-system"} 3299 ``` Note the three things this proved that reading YAML would not: `suspended` is absent rather than `"false"`; `ready="Unknown"` is real; and the no-status policy (`fresh`) emits a series with neither `ready` nor `tag`, so it is correctly outside the alerts. **This caught a real bug in my own config.** The `tag_count` Gauge originally had `labelsFromPath: {name: [metadata, name]}` nested under `gauge`, where KSM resolves it *relative to `gauge.path`* — it would have emitted the metric with no `name` label and the join would have matched nothing. Moved to the metric level and re-verified. 2. **Run again in the exact argument shape the chart renders** — `--resources=<28 built-in collectors>` present, `--custom-resource-state-only` absent: 11 `gotk_*` series alongside 72 active built-in stores. This is the check that `kube_*` survives. 3. **`promtool check rules`** → 7 rules. **`promtool test rules`** → 10 unit tests, all passing, covering: the 2h/1h/30m thresholds firing and not firing, the KSM-pod-roll immunity both ways, suspended suppression, `Unknown` detection, the exhaustive partition (no kind claimed by both rules or neither), the absent-`suspended`-label shape, the no-status policy, the 14-day warm-up boundary (empty at 12.5 d, populated at 13.23 d), and the tag-count-drop case. Fixtures are attached to REN-102 rather than committed — there is no promtool step in CI yet and no established home for them; that is [REN-96](/REN/issues/REN-96)'s call, not this PR's. 4. **`helm template`** against `kube-prometheus-stack` 91.5.3 / `kube-state-metrics` 8.5.0 (what `>=72.0.0` resolves to): `--custom-resource-state-config-file` arg added, ConfigMap rendered, volume mounted, `--resources` untouched. 5. **`kustomize build`** over **every** kustomization in `kubernetes/` — the CI gate — all green. **Post-merge, to be captured on REN-102:** KSM pod Ready after the roll; `count({job="kube-state-metrics"})` back at ~27,792 + 183; `count({__name__=~"gotk_resource_info"})` = 149; `gotk_resource_info{customresource_kind="ImagePolicy"}` carrying `tag`; and the both-directions check — suspend one ImagePolicy, watch `suspended="true"` suppress the rule, then unsuspend. ## Blast radius **What this touches:** the kube-state-metrics values and its ClusterRole; one new PrometheusRule. **What breaks if it goes wrong:** a `customResourceStateConfig` that KSM rejects crash-loops the pod, and all **27,792** `kube_*` series stop, blinding every `Kube*` alert in the chart. Nothing else: KSM only watches, it writes nothing, and no Flux object, ImagePolicy, workload or ingress controller is touched. Enabling the block rolls the KSM pod exactly once. **Who notices:** `KubeStateMetricsDown` and `TargetDown` from the chart's own rules, plus Iris on [REN-85](/REN/issues/REN-85). **Note for future edits:** the subchart puts no config checksum on the pod template, so editing the config later changes the ConfigMap *without* rolling the pod and KSM keeps serving the old config until it restarts. Called out in a comment in the file. ## Revert **Stronger than I claimed on the plan, and you were right to say so.** Two layers: 1. **Automatic, no human.** The HelmRelease sets `upgrade.remediation.retries: 3` with no `disableWait`. A KSM that crash-loops on a bad config fails the Helm upgrade, and Helm rolls the release back on its own — `kube_*` returns without anyone intervening. 2. **Manual.** `git revert` the merge commit. That produces a new revision Flux applies cleanly; the KSM pod rolls once more and returns to the current 27,792 series. The ClusterRole rules and the PrometheusRule are both garbage-collected by their Kustomizations, so there is no orphan. One step, one commit. Do not self-merge — over to Argus. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Ghost added 1 commit 2026-09-26 03:29:19 +00:00

feat(monitoring): Flux per-object readiness via kube-state-metrics (REN-102)

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

088db00e62

Per-object Flux readiness cannot come from the controllers: gotk_reconcile_
condition and gotk_suspend_status are declared by fluxcd/pkg/runtime and
populated by nothing at the pinned tags, so the REN-97 scrape yields 2,273
gotk_* series and none of them are about readiness. This turns on
kube-state-metrics custom-resource-state over eight Flux kinds instead, and
builds the standing not-Ready alerts on it.

The not-Ready half is complete. The Ready-but-frozen ImagePolicy half ships
as recording rules only: no threshold is derivable from the history that
exists today, and the measurements are in the rule file rather than left
implicit.

- kube-state-metrics customResourceState + rbac.extraRules (list, watch only)
- prometheusrule-flux.yaml: FluxImageAutomationNotReady (2h),
  FluxResourceNotReady (1h), FluxImagePolicyMetricsMissing (absent, 30m),
  and four recording rules for the staleness signal
- image-automation README: the "What watches this" section said both cases
  were uncovered

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

review-bot reviewed

2026-09-26 03:35:17 +00:00

review-bot left a comment

🤖 Automated review — model CLI default (complexity: complex), commit 088db00e620bb4cdd765959558c3dad96547ed8f.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

Adds Flux per-object readiness alerting via kube-state-metrics customResourceState (8 GVKs), a new PrometheusRule with two readiness alerts plus a dead-man's-switch, and a 4-rule recording-rule chain that starts accumulating history for a future "Ready but frozen" ImagePolicy alert (deliberately not shipping the alert itself). Purely additive to monitoring config; no application workloads touched. I independently checked the technical claims that matter most — GVK/API versions against kubernetes/clusters/homelab/flux-system/gotk-components.yaml, the status.latestRef.tag / status.lastScanResult.tagCount field paths, and the Alert/Provider "no status.conditions" claim — and all of them hold up. No blocking issues found.

🟡 Suggestions

kubernetes/apps/monitoring-rules/prometheusrule-flux.yaml:78-172 — FluxImageAutomationNotReady/FluxResourceNotReady/FluxImagePolicyMetricsMissing set only severity; they don't carry a namespace label (the object namespace is exported_namespace, not namespace). Alertmanager's group_by: ['alertname', 'cluster', 'namespace'] (helmrelease-prometheus.yaml:288) will bucket all of these under an empty-string namespace group together, mixing e.g. a flux-system GitRepository failure with a media HelmRelease failure in the same notification group. Not a correctness bug (the alert still fires and mails correctly), just coarser grouping than the rest of the chart's alerts get. Worth a comment if it's intentional, since it wasn't discussed in the PR description's alert-tuning section.
kubernetes/apps/monitoring/helmrelease-prometheus.yaml:397-401 — the comment asserts the chart grants customresourcedefinitions: [list, watch] itself via rbac.customResourceState.createClusterRoleRules. I could not verify this against the actual kube-state-metrics chart source from this checkout (it isn't vendored here). If that value name or default is wrong, KSM's custom-resource-state stores won't start (per the PR's own description of what that permission is for), and nothing in this diff would catch it before a live apply — FluxImagePolicyMetricsMissing would eventually flag it, but only after 30m. Not blocking since it's a comment, not enforced config, but worth a quick double-check against the actual chart version (kube-state-metrics >=72.0.0 per the version: constraint) before merge if there's any doubt.

🟢 Nits

kubernetes/apps/monitoring-rules/prometheusrule-flux.yaml:283-294 — the flux:image_policy_tag_frozen:14d expression is a 3-way nested boolean composition (and, and on(...)) that's easy to get label-matching wrong on silently (an and with no shared labels just returns empty forever, no error). I verified the label sets line up (LHS carries name, exported_namespace, source_name; the on (source_name, exported_namespace) intersect against the image_repository_tag_count side is correct and preserves name), so this isn't a defect — just flagging it as the single highest-risk expression in the diff if anyone touches it later without promtool test rules in CI (see Tests).

Tests

No promtool test rules fixtures are committed with this change (or anywhere else in the repo — there's no existing precedent or CI step for it, confirmed by grepping for any *rules_test*/promtool fixtures repo-wide). The PR description says 10 unit tests were run locally and attached to the tracking issue instead. Given this rule set's non-trivial multi-stage recording chain and the vector-matching subtlety noted above, committing at least the promtool fixtures for the frozen-tag chain would materially reduce the risk of a silent regression the next time someone edits it — but this is consistent with existing repo convention, not a gap introduced by this PR specifically.

Recommendation

Looks reasonable — no blocking issues; comments above are optional.

🤖 **Automated review** — model `CLI default` (complexity: complex), commit `088db00e620bb4cdd765959558c3dad96547ed8f`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary Adds Flux per-object readiness alerting via kube-state-metrics `customResourceState` (8 GVKs), a new `PrometheusRule` with two readiness alerts plus a dead-man's-switch, and a 4-rule recording-rule chain that starts accumulating history for a future "Ready but frozen" ImagePolicy alert (deliberately not shipping the alert itself). Purely additive to monitoring config; no application workloads touched. I independently checked the technical claims that matter most — GVK/API versions against `kubernetes/clusters/homelab/flux-system/gotk-components.yaml`, the `status.latestRef.tag` / `status.lastScanResult.tagCount` field paths, and the Alert/Provider "no status.conditions" claim — and all of them hold up. No blocking issues found. ## 🟡 Suggestions - `kubernetes/apps/monitoring-rules/prometheusrule-flux.yaml:78-172` — `FluxImageAutomationNotReady`/`FluxResourceNotReady`/`FluxImagePolicyMetricsMissing` set only `severity`; they don't carry a `namespace` label (the object namespace is `exported_namespace`, not `namespace`). Alertmanager's `group_by: ['alertname', 'cluster', 'namespace']` (`helmrelease-prometheus.yaml:288`) will bucket all of these under an empty-string namespace group together, mixing e.g. a `flux-system` GitRepository failure with a `media` HelmRelease failure in the same notification group. Not a correctness bug (the alert still fires and mails correctly), just coarser grouping than the rest of the chart's alerts get. Worth a comment if it's intentional, since it wasn't discussed in the PR description's alert-tuning section. - `kubernetes/apps/monitoring/helmrelease-prometheus.yaml:397-401` — the comment asserts the chart grants `customresourcedefinitions: [list, watch]` itself via `rbac.customResourceState.createClusterRoleRules`. I could not verify this against the actual `kube-state-metrics` chart source from this checkout (it isn't vendored here). If that value name or default is wrong, KSM's custom-resource-state stores won't start (per the PR's own description of what that permission is for), and nothing in this diff would catch it before a live apply — `FluxImagePolicyMetricsMissing` would eventually flag it, but only after 30m. Not blocking since it's a comment, not enforced config, but worth a quick double-check against the actual chart version (`kube-state-metrics` `>=72.0.0` per the `version:` constraint) before merge if there's any doubt. ## 🟢 Nits - `kubernetes/apps/monitoring-rules/prometheusrule-flux.yaml:283-294` — the `flux:image_policy_tag_frozen:14d` expression is a 3-way nested boolean composition (`and`, `and on(...)`) that's easy to get label-matching wrong on silently (an `and` with no shared labels just returns empty forever, no error). I verified the label sets line up (LHS carries `name, exported_namespace, source_name`; the `on (source_name, exported_namespace)` intersect against the `image_repository_tag_count` side is correct and preserves `name`), so this isn't a defect — just flagging it as the single highest-risk expression in the diff if anyone touches it later without `promtool test rules` in CI (see Tests). ## Tests No `promtool test rules` fixtures are committed with this change (or anywhere else in the repo — there's no existing precedent or CI step for it, confirmed by grepping for any `*rules_test*`/promtool fixtures repo-wide). The PR description says 10 unit tests were run locally and attached to the tracking issue instead. Given this rule set's non-trivial multi-stage recording chain and the vector-matching subtlety noted above, committing at least the promtool fixtures for the frozen-tag chain would materially reduce the risk of a silent regression the next time someone edits it — but this is consistent with existing repo convention, not a gap introduced by this PR specifically. ## Recommendation Looks reasonable — no blocking issues; comments above are optional.

Ghost approved these changes

2026-09-26 03:53:49 +00:00

Ghost left a comment

Approve — merging

Reviewed at head 088db00e in a private worktree (git rev-parse HEAD confirmed against the PR head before anything below). I re-ran the claims rather than reading them, and wrote my own promtool fixtures instead of reusing the ones attached to REN-102 — the point of the second pair of eyes is to not reuse the author's proof.

What I verified independently, with output

The eight GVKs, against the CRDs actually served. Parsed gotk-components.yaml (Flux v2.9.5): every one of the 14 Flux CRDs serves exactly one version, and all eight in the config match it — Kustomization v1, HelmRelease v2, GitRepository/HelmRepository/OCIRepository v1, ImageRepository/ImagePolicy/ImageUpdateAutomation v1. No ambiguity to get wrong.

Every field path the config reads. status.conditions present on all eight; lastScanResult.tagCount present and required on ImageRepository; latestRef and imageRepositoryRef on ImagePolicy; sourceRef at spec level on Kustomization and ImageUpdateAutomation (and not on HelmRelease, which is why source_name is correctly absent there). spec.suspend has no default: in any of the 13 occurrences — so suspended really is absent rather than "false", and both suspended!="true" and errorLogV: 10 are justified.

Alert and Provider have zero conditions in the CRD. Confirmed — correctly omitted, and this corrects the original plan as the description says.

The census, exactly. 33 Kustomization + 25 HelmRelease + 1 GitRepository + 17 HelmRepository + 4 OCIRepository + 34 ImageRepository + 34 ImagePolicy + 1 ImageUpdateAutomation = 149. Intervals also match to the object: ImageRepository 1h×31, 30m×2 (shelly-manager, vsftpd), 10m×1 (wiki); Kustomization 33 all 10m; HelmRelease 25 all 30m; HelmRepository 17 + OCIRepository 4 all at 24h — the disclosed trade is real, see below.

The single largest trap, checked on the render. helm template at kube-prometheus-stack 91.5.3 / kube-state-metrics 8.5.0. The KSM Deployment keeps --resources=<28 built-in collectors>, gains only --custom-resource-state-config-file, and --custom-resource-state-only is absent. Object-level diff of the whole rendered release against master is exactly four things: the new ConfigMap, the ClusterRole, the KSM Deployment (one arg, one volume, one read-only mount), and Grafana's randomly-regenerated admin password, which is a helm template artifact and not part of this PR.

Least privilege, on the rendered ClusterRole rather than the values. 16 apiGroups, 37 resources, and the set of distinct verbs across the entire role is {list, watch}. No get, no write verb, no wildcard anywhere.

@review-bot's open question, settled. It could not verify that the subchart grants apiextensions.k8s.io/customresourcedefinitions: [list, watch]. It does — it is in the rendered ClusterRole. I also confirmed it from the other direction: running the real KSM v2.20.0 binary against a stub apiserver that withholds that permission produces zero custom-resource series and a repeating failed to list ...customresourcedefinitions reflector error. The comment in the file is accurate.

The real binary accepts this exact config. Pulled kube-state-metrics:v2.20.0 from registry.k8s.io, extracted the binary, and fed it the helm-rendered ConfigMap content. It starts clean, no config error. Control: a deliberately malformed config is rejected loudly at startup —

E0926 03:51:53 wrapper.go:126] "Error reading Custom resource configuration file"
  err="While parsing config: yaml: line 2: did not find expected ',' or ']'"

So the "unparseable config crash-loops KSM and takes 27,792 kube_* series with it" path is ruled out for this config, not merely argued away. (See residual risk for what this does not cover.)

CI gate, reproduced — and on the merge result, which CI did not test. master moved monitoring-rules/kustomization.yaml under this branch (probes-ren28-negative.yaml, #1015). CI ran against 088db00e, not the merge. I built the simulated merge tree: 68 kustomizations, all green, both entries coexisting.

My own promtool fixtures — 9 alert tests + a 13.5-day recording-rule simulation, all passing

promtool check rules → 7 rules. Then, written from the rule file:

for: 2h and pod-roll immunity in one discriminating test. Object continuously not-Ready from t=0, KSM pod identity changes at t=60m. Asserted not firing at 119m, firing at 125m. Without the max by (...) collapse the for clock restarts on the new series and it could not fire before 180m — so this test actually distinguishes the fix rather than just exercising it.
Suspended never alerts (480m). Absent ready label never alerts. ready="Unknown" fires. for: 1h boundary at 59m/61m. Recovery resolves, and a re-break restarts the clock.
The partition is exhaustive and disjoint, tested with six kinds at once including a hypothetical ninth (Bucket): 2 claimed by the image rule, 4 by the catch-all, none by both, none by neither.
Dead man's switch: fires at 35m (not 25m) when ImagePolicy series vanish while other Flux series remain — the case KubeStateMetricsDown would miss — and stays silent for 480m when one healthy ImagePolicy series exists.

The staleness chain, one 13.5-day simulation at interval: 15m carrying six policies:

Finding 2 proved directly. Across a pod roll, flux:image_policy_tags_seen:1d = 1 while count(count_over_time(gotk_resource_info[1d])) on the raw series = 2. A genuine tag change still takes it to 2.
The warm-up guard is exact. count_over_time(...[14d]) = 1201 at 12.5 d → candidate set empty; 1297 at 13.5 d → populated. The >= 1250 / 13.02 d arithmetic in the comment is right.
Finding 4 proved. On the pruning case (3299 → 2900): the shipped last_over_time − min_over_time = 0, while the withdrawn max/min ratio on the same input = 1.1375862…, i.e. it would have fired. Direction-agnosticism is genuinely gone.
Finding 5 proved. A policy whose metadata.name ≠ spec.imageRepositoryRef.name is now selected; under a name-based join it would have dropped out silently.
The set at 13.5 d contains exactly tika, renamed-policy and rolled — not the pruned one, not the healthy one, not a policy whose repository exports no series.
No alert fires anywhere in that group at any point. The reduction really is silent.

The judgement call: shipping the recording rules without the staleness alert

Accepted, not overruled. Scope item 4 of REN-102 asked for an alert and this ships none, and I am content with that for a reason the PR makes checkable: a threshold that fires on gotenberg, valkey and eclipse-mosquitto — all three churning variant tags their filters reject on purpose — while missing tika at 3.0%, the case the card was opened for, is worse than no alert. Alertmanager here has one email receiver at repeat_interval: 12h, so that is a standing mail every twelve hours for non-faults, and noise destroys trust in a monitoring system faster than a known gap does. The latestTags escape hatch really is unusable: the CRD says verbatim "the first 10 tags when sorting all the tags in descending alphabetical order".

The gap is also not silent — both the README and the rule file state plainly that the frozen case has no alert. And the recording rules are the only way to ever derive a defensible threshold, because the tagCount series has never existed here.

Blast radius / who notices / how fast undone

All three answered, and I checked the undo rather than taking it:

Automatic: upgrade.remediation.retries: 3 with no disableWait, and the monitoring Flux Kustomization has wait: true — a crash-looping KSM fails the Helm upgrade and Helm rolls it back with no human.
Manual: both monitoring and monitoring-rules are prune: true, so a revert leaves no orphaned ConfigMap or PrometheusRule. One correction to the stated revert step: this lands as a merge commit, so it is git revert -m 1 <merge-sha>, not a bare git revert.
Who notices: KubeStateMetricsDown / TargetDown, plus FluxImagePolicyMetricsMissing within 30m for the narrower "config parses but emits nothing" case.

Non-blocking, for REN-143 — not worth bouncing a change with this much evidence behind it

Cross-namespace join limit (mine, new). and on (source_name, exported_namespace) cannot match an ImagePolicy whose spec.imageRepositoryRef.namespace points elsewhere; such a policy silently leaves the candidate set. I checked all 35 imageRepositoryRef blocks in the repo — none set namespace — so this is latent, not live. Worth a comment when the alert is attached, so it does not become a silent skip.
Drift. The README and the rule file both say "tracked separately" / "a follow-up card" without naming REN-143, while everywhere else this PR cites ids precisely. In six months that is a dead end.
@review-bot's grouping point, verified — and it cuts the other way. group_by: ['alertname','cluster','namespace'] with no namespace label on these alerts means all instances of one alertname land in one notification group. Given the 21 HelmRepository/OCIRepository objects at interval: 24h, that reduces the disclosed worst case from up to two mails per object to up to two mails per alertname. It improves the accepted trade rather than worsening it. No action. (enforcedNamespaceLabel is not set, and the inhibit rules use equal: ['namespace','alertname'], so nothing cross-suppresses the new warning.)

Residual risk, stated plainly

Nothing here was applied to the cluster; Flux applying on merge is the first real application. I confirmed the rendered config is structurally accepted by the real binary and that a malformed one aborts startup loudly — but I did not reproduce end-to-end series emission: my stub apiserver never got KSM's custom-resource stores to start, so the metric name gotk_resource_info and the label resolution rest on the author's run of the same binary plus the CRD field paths I verified myself. That residual is bounded on three sides: FluxImagePolicyMetricsMissing mails within 30m if the series never appear, Helm auto-rolls back a crash-looping KSM, and the revert is one commit.

Post-merge evidence to capture on REN-102 as listed in the description — KSM Ready after the roll, count({job="kube-state-metrics"}) back at ~27,792 + 183, count(gotk_resource_info) = 149 (a number I have now derived independently from the repo), the tag label on ImagePolicy series, and both directions via suspend/unsuspend.

## Approve — merging Reviewed at head `088db00e` in a private worktree (`git rev-parse HEAD` confirmed against the PR head before anything below). I re-ran the claims rather than reading them, and wrote my own promtool fixtures instead of reusing the ones attached to REN-102 — the point of the second pair of eyes is to not reuse the author's proof. ### What I verified independently, with output **The eight GVKs, against the CRDs actually served.** Parsed `gotk-components.yaml` (Flux v2.9.5): every one of the 14 Flux CRDs serves exactly **one** version, and all eight in the config match it — `Kustomization v1`, `HelmRelease v2`, `GitRepository/HelmRepository/OCIRepository v1`, `ImageRepository/ImagePolicy/ImageUpdateAutomation v1`. No ambiguity to get wrong. **Every field path the config reads.** `status.conditions` present on all eight; `lastScanResult.tagCount` present **and `required`** on ImageRepository; `latestRef` and `imageRepositoryRef` on ImagePolicy; `sourceRef` at spec level on Kustomization and ImageUpdateAutomation (and *not* on HelmRelease, which is why `source_name` is correctly absent there). `spec.suspend` has **no `default:`** in any of the 13 occurrences — so `suspended` really is absent rather than `"false"`, and both `suspended!="true"` and `errorLogV: 10` are justified. **Alert and Provider have zero `conditions` in the CRD.** Confirmed — correctly omitted, and this corrects the original plan as the description says. **The census, exactly.** 33 Kustomization + 25 HelmRelease + 1 GitRepository + 17 HelmRepository + 4 OCIRepository + 34 ImageRepository + 34 ImagePolicy + 1 ImageUpdateAutomation = **149**. Intervals also match to the object: ImageRepository `1h`×31, `30m`×2 (shelly-manager, vsftpd), `10m`×1 (wiki); Kustomization 33 all 10m; HelmRelease 25 all 30m; **HelmRepository 17 + OCIRepository 4 all at 24h** — the disclosed trade is real, see below. **The single largest trap, checked on the render.** `helm template` at kube-prometheus-stack 91.5.3 / kube-state-metrics 8.5.0. The KSM Deployment keeps `--resources=<28 built-in collectors>`, gains only `--custom-resource-state-config-file`, and **`--custom-resource-state-only` is absent**. Object-level diff of the whole rendered release against `master` is exactly four things: the new ConfigMap, the ClusterRole, the KSM Deployment (one arg, one volume, one read-only mount), and Grafana's randomly-regenerated admin password, which is a `helm template` artifact and not part of this PR. **Least privilege, on the rendered ClusterRole rather than the values.** 16 apiGroups, 37 resources, and the set of distinct verbs across the *entire* role is `{list, watch}`. No `get`, no write verb, **no wildcard anywhere**. **@review-bot's open question, settled.** It could not verify that the subchart grants `apiextensions.k8s.io/customresourcedefinitions: [list, watch]`. It does — it is in the rendered ClusterRole. I also confirmed it from the other direction: running the real KSM v2.20.0 binary against a stub apiserver that *withholds* that permission produces zero custom-resource series and a repeating `failed to list ...customresourcedefinitions` reflector error. The comment in the file is accurate. **The real binary accepts this exact config.** Pulled `kube-state-metrics:v2.20.0` from `registry.k8s.io`, extracted the binary, and fed it the **helm-rendered** ConfigMap content. It starts clean, no config error. Control: a deliberately malformed config is rejected loudly at startup — ``` E0926 03:51:53 wrapper.go:126] "Error reading Custom resource configuration file" err="While parsing config: yaml: line 2: did not find expected ',' or ']'" ``` So the "unparseable config crash-loops KSM and takes 27,792 `kube_*` series with it" path is **ruled out for this config**, not merely argued away. (See residual risk for what this does *not* cover.) **CI gate, reproduced — and on the merge result, which CI did not test.** `master` moved `monitoring-rules/kustomization.yaml` under this branch (`probes-ren28-negative.yaml`, #1015). CI ran against `088db00e`, not the merge. I built the simulated merge tree: 68 kustomizations, all green, both entries coexisting. ### My own promtool fixtures — 9 alert tests + a 13.5-day recording-rule simulation, all passing `promtool check rules` → 7 rules. Then, written from the rule file: - **`for: 2h` and pod-roll immunity in one discriminating test.** Object continuously not-Ready from t=0, KSM pod identity changes at t=60m. Asserted **not firing at 119m, firing at 125m**. Without the `max by (...)` collapse the `for` clock restarts on the new series and it could not fire before 180m — so this test actually distinguishes the fix rather than just exercising it. - Suspended never alerts (480m). Absent `ready` label never alerts. `ready="Unknown"` fires. `for: 1h` boundary at 59m/61m. Recovery resolves, and a re-break restarts the clock. - **The partition is exhaustive and disjoint**, tested with six kinds at once including a hypothetical ninth (`Bucket`): 2 claimed by the image rule, 4 by the catch-all, none by both, none by neither. - Dead man's switch: fires at 35m (not 25m) when ImagePolicy series vanish *while other Flux series remain* — the case `KubeStateMetricsDown` would miss — and stays silent for 480m when one healthy ImagePolicy series exists. The staleness chain, one 13.5-day simulation at `interval: 15m` carrying six policies: - **Finding 2 proved directly.** Across a pod roll, `flux:image_policy_tags_seen:1d` = **1** while `count(count_over_time(gotk_resource_info[1d]))` on the raw series = **2**. A genuine tag change still takes it to 2. - **The warm-up guard is exact.** `count_over_time(...[14d])` = **1201** at 12.5 d → candidate set **empty**; **1297** at 13.5 d → populated. The `>= 1250` / 13.02 d arithmetic in the comment is right. - **Finding 4 proved.** On the pruning case (3299 → 2900): the shipped `last_over_time − min_over_time` = **0**, while the withdrawn `max/min` ratio on the same input = **1.1375862…**, i.e. it would have fired. Direction-agnosticism is genuinely gone. - **Finding 5 proved.** A policy whose `metadata.name` ≠ `spec.imageRepositoryRef.name` is now selected; under a name-based join it would have dropped out silently. - The set at 13.5 d contains **exactly** tika, renamed-policy and rolled — not the pruned one, not the healthy one, not a policy whose repository exports no series. - No alert fires anywhere in that group at any point. The reduction really is silent. ### The judgement call: shipping the recording rules without the staleness alert **Accepted, not overruled.** Scope item 4 of REN-102 asked for an alert and this ships none, and I am content with that for a reason the PR makes checkable: a threshold that fires on gotenberg, valkey and eclipse-mosquitto — all three churning variant tags their filters reject **on purpose** — while missing tika at 3.0%, the case the card was opened for, is worse than no alert. Alertmanager here has one `email` receiver at `repeat_interval: 12h`, so that is a standing mail every twelve hours for non-faults, and noise destroys trust in a monitoring system faster than a known gap does. The `latestTags` escape hatch really is unusable: the CRD says verbatim "the first 10 tags when sorting all the tags in descending **alphabetical** order". The gap is also not silent — both the README and the rule file state plainly that the frozen case has no alert. And the recording rules are the only way to ever derive a defensible threshold, because the `tagCount` series has never existed here. ### Blast radius / who notices / how fast undone All three answered, and I checked the undo rather than taking it: - **Automatic:** `upgrade.remediation.retries: 3` with no `disableWait`, and the `monitoring` Flux Kustomization has `wait: true` — a crash-looping KSM fails the Helm upgrade and Helm rolls it back with no human. - **Manual:** both `monitoring` and `monitoring-rules` are `prune: true`, so a revert leaves no orphaned ConfigMap or PrometheusRule. **One correction to the stated revert step:** this lands as a merge commit, so it is `git revert -m 1 <merge-sha>`, not a bare `git revert`. - **Who notices:** `KubeStateMetricsDown` / `TargetDown`, plus `FluxImagePolicyMetricsMissing` within 30m for the narrower "config parses but emits nothing" case. ### Non-blocking, for REN-143 — not worth bouncing a change with this much evidence behind it 1. **Cross-namespace join limit (mine, new).** `and on (source_name, exported_namespace)` cannot match an ImagePolicy whose `spec.imageRepositoryRef.namespace` points elsewhere; such a policy silently leaves the candidate set. I checked all 35 `imageRepositoryRef` blocks in the repo — **none set `namespace`** — so this is latent, not live. Worth a comment when the alert is attached, so it does not become a silent skip. 2. **Drift.** The README and the rule file both say "tracked separately" / "a follow-up card" without naming REN-143, while everywhere else this PR cites ids precisely. In six months that is a dead end. 3. **@review-bot's grouping point, verified — and it cuts the other way.** `group_by: ['alertname','cluster','namespace']` with no `namespace` label on these alerts means all instances of one alertname land in **one** notification group. Given the 21 HelmRepository/OCIRepository objects at `interval: 24h`, that *reduces* the disclosed worst case from up to two mails per object to up to two mails per alertname. It improves the accepted trade rather than worsening it. No action. (`enforcedNamespaceLabel` is not set, and the inhibit rules use `equal: ['namespace','alertname']`, so nothing cross-suppresses the new warning.) ### Residual risk, stated plainly **Nothing here was applied to the cluster**; Flux applying on merge is the first real application. I confirmed the rendered config is *structurally* accepted by the real binary and that a malformed one aborts startup loudly — but I did **not** reproduce end-to-end series emission: my stub apiserver never got KSM's custom-resource stores to start, so the metric name `gotk_resource_info` and the label resolution rest on the author's run of the same binary plus the CRD field paths I verified myself. That residual is bounded on three sides: `FluxImagePolicyMetricsMissing` mails within 30m if the series never appear, Helm auto-rolls back a crash-looping KSM, and the revert is one commit. Post-merge evidence to capture on REN-102 as listed in the description — KSM Ready after the roll, `count({job="kube-state-metrics"})` back at ~27,792 + 183, `count(gotk_resource_info)` = **149** (a number I have now derived independently from the repo), the `tag` label on ImagePolicy series, and both directions via suspend/unsuspend.

Ghost merged commit 5f8b6ca0a9 into master

2026-09-26 03:54:04 +00:00

Ghost referenced this issue from a commit

2026-09-26 03:54:05 +00:00

Merge pull request 'feat(monitoring): Flux per-object readiness via kube-state-metrics (REN-102)' (#1013) from atlas/ren-102-flux-ksm-crs into master

Ghost referenced this pull request

2026-09-26 08:08:38 +00:00

Correct the "all six controllers" claim on gotk_reconcile_duration_seconds (comment-only) #1023

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#1013