Correct the "all six controllers" claim on gotk_reconcile_duration_seconds (comment-only) #1023

2026-09-26T08:08:38Z

Ghost commented

2026-09-26 08:08:38 +00:00

What changed

One comment block in kubernetes/apps/monitoring-rules/flux-podmonitor.yaml. No YAML key,
value, selector or label is touched.

#992's header comment claims RecordDuration "IS called, by all six controllers". That was
established by grepping the tagged upstream source trees, which was a reasonable method before
the scrape existed. Now that #992 is merged the claim is measurable, and it is wrong by one:
five of six controllers emit gotk_reconcile_duration_seconds.

Controller	series	objects it owns
image-reflector	68	ImagePolicy 34 + ImageRepository 34
source	43	HelmChart 21 + HelmRepository 17 + OCIRepository 4 + GitRepository 1
kustomize	33	Kustomization 33
helm	25	HelmRelease 25
image-automation	1	ImageUpdateAutomation 1
notification	0	Alert 2 + Provider 1
Total	170	173

173 live Flux objects, 170 series — the gap is exactly the 2 Alerts and 1 Provider.

Why it matters, and why five is permanent

The obvious reading, "notification-controller has nothing to reconcile", is not the reason — it
owns three objects. Its reconcilers are registered but never walked:
controller_runtime_reconcile_errors_total has exactly 3 series for that pod
(controller="alert", "provider", "receiver") and controller_runtime_reconcile_total has 12
(3 x 4 result values), so they exist — yet all 15 of those counters read 0 over 56.45 h of
process uptime, with controller_runtime_active_workers at 0 for all three. Under Flux v2.9.5 the
metric-recording path is never reached for these kinds; the whole family is absent from that pod's
/metrics, which exports only gotk_event_http_* and gotk_token_cache*.

So creating a Flux Alert or Provider does not produce the sixth series — the cluster
already has both. The comment now says so, and states the two rules that follow:

Never count(... gotk_reconcile_duration_seconds ...) < 6, and never absent() on it per
pod. Such a rule fires forever on notification-controller and trains everyone to ignore the
whole group. This is the trap the correction exists to close, with
REN-102's alerts landing next door.
Liveness for all six pods is up{job="monitoring/flux-controllers"}, which is 6 and all 1.
That pod's reconcile health is controller_runtime_reconcile_errors_total.

Nothing merged keys on either — git grep over kubernetes/ finds no rule referencing
gotk_reconcile_duration_seconds or a controller count, so no shipped alert is wrong today.
This is a guard rail for the next rule anyone adds.

How it was verified

Measured, not asserted. Every number above is a live PromQL read through the read-only
Prometheus path from REN-36, taken 2026-09-26T03:05Z and re-measured 03:56Z — identical.
Independently reproduced by Iris on REN-127.
Proof the diff cannot affect the cluster, which is the whole safety argument for touching a
merged file: all 17 changed lines are pure comment lines, and the comment-stripped file is
byte-identical to master — sha256 9c081e7e310945e8457a... on both sides. YAML comments
vanish at parse, so the object Flux applies is unchanged and no pod rolls.
Not applied by hand. Nothing in the cluster was written; every check was a GET.

Blast radius

Zero. The applied object is identical to what is already running, proven above. Flux
reconciles the PodMonitor to the same spec, Prometheus does not reload a changed target, and no
workload, ingress controller or Flux object is touched. If this diff were somehow wrong the worst
case is a misleading code comment — the same failure mode as today.

Revert

git revert ee4ea92a — one commit, one file, comment lines only. Nothing to undo in the cluster
because nothing changes in it.

Refs REN-127 (the correction), REN-102 (where the alert design lands), REN-97, #992.

🤖 Generated with Claude Code

## What changed One comment block in `kubernetes/apps/monitoring-rules/flux-podmonitor.yaml`. **No YAML key, value, selector or label is touched.** #992's header comment claims `RecordDuration` "IS called, by all six controllers". That was established by grepping the tagged upstream source trees, which was a reasonable method before the scrape existed. Now that #992 is merged the claim is measurable, and it is wrong by one: **five of six controllers emit `gotk_reconcile_duration_seconds`.** | Controller | series | objects it owns | | --- | --- | --- | | image-reflector | 68 | ImagePolicy 34 + ImageRepository 34 | | source | 43 | HelmChart 21 + HelmRepository 17 + OCIRepository 4 + GitRepository 1 | | kustomize | 33 | Kustomization 33 | | helm | 25 | HelmRelease 25 | | image-automation | 1 | ImageUpdateAutomation 1 | | **notification** | **0** | **Alert 2 + Provider 1** | | **Total** | **170** | **173** | 173 live Flux objects, 170 series — the gap is exactly the 2 Alerts and 1 Provider. ## Why it matters, and why five is permanent The obvious reading, "notification-controller has nothing to reconcile", is not the reason — it owns three objects. Its reconcilers are **registered but never walked**: `controller_runtime_reconcile_errors_total` has exactly 3 series for that pod (`controller="alert"`, `"provider"`, `"receiver"`) and `controller_runtime_reconcile_total` has 12 (3 x 4 `result` values), so they exist — yet **all 15 of those counters read 0** over 56.45 h of process uptime, with `controller_runtime_active_workers` at 0 for all three. Under Flux v2.9.5 the metric-recording path is never reached for these kinds; the whole family is absent from that pod's `/metrics`, which exports only `gotk_event_http_*` and `gotk_token_cache*`. So **creating a Flux `Alert` or `Provider` does not produce the sixth series** — the cluster already has both. The comment now says so, and states the two rules that follow: - **Never** `count(... gotk_reconcile_duration_seconds ...) < 6`, and never `absent()` on it per pod. Such a rule fires forever on notification-controller and trains everyone to ignore the whole group. This is the trap the correction exists to close, with [REN-102](https://gitea.renner.uno/k.renner/homelab/pulls/1013)'s alerts landing next door. - Liveness for all six pods is `up{job="monitoring/flux-controllers"}`, which is 6 and all `1`. That pod's reconcile health is `controller_runtime_reconcile_errors_total`. Nothing merged keys on either — `git grep` over `kubernetes/` finds no rule referencing `gotk_reconcile_duration_seconds` or a controller count, so **no shipped alert is wrong today.** This is a guard rail for the next rule anyone adds. ## How it was verified - **Measured, not asserted.** Every number above is a live PromQL read through the read-only Prometheus path from REN-36, taken 2026-09-26T03:05Z and re-measured 03:56Z — identical. Independently reproduced by Iris on REN-127. - **Proof the diff cannot affect the cluster**, which is the whole safety argument for touching a merged file: all 17 changed lines are pure comment lines, and the comment-stripped file is **byte-identical to master** — sha256 `9c081e7e310945e8457a...` on both sides. YAML comments vanish at parse, so the object Flux applies is unchanged and **no pod rolls**. - Not applied by hand. Nothing in the cluster was written; every check was a GET. ## Blast radius **Zero.** The applied object is identical to what is already running, proven above. Flux reconciles the PodMonitor to the same spec, Prometheus does not reload a changed target, and no workload, ingress controller or Flux object is touched. If this diff were somehow wrong the worst case is a misleading code comment — the same failure mode as today. ## Revert `git revert ee4ea92a` — one commit, one file, comment lines only. Nothing to undo in the cluster because nothing changes in it. --- Refs REN-127 (the correction), REN-102 (where the alert design lands), REN-97, #992. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Ghost added 1 commit 2026-09-26 08:08:38 +00:00

Correct the "all six controllers" claim on gotk_reconcile_duration_seconds

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

ee4ea92a9e

#992's header comment says RecordDuration "IS called, by all six controllers",
established by grepping the tagged upstream source trees. Now that the scrape is
merged the claim is measurable, and five of six emit the family: 170 series from
kustomize (33), source (43), image-reflector (68), helm (25) and image-automation
(1), and zero from notification-controller.

Five is permanent, not today-only. notification-controller's three reconcilers
are registered but never walked -- controller_runtime_reconcile_total for
controller=alert/provider/receiver reads 0 over 56 h of uptime with 2 Alerts and
1 Provider live in the apiserver -- so the family is absent from its /metrics
entirely and creating a Flux Alert or Provider does not produce the sixth series.

Comment-only. Every one of the 17 changed lines is a pure comment line and the
comment-stripped file is byte-identical to master (same sha256), so the object
Flux applies is unchanged and no pod rolls.

Refs REN-127, REN-102, REN-97

Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Ghost approved these changes

2026-09-26 08:26:04 +00:00

Ghost left a comment

Approve at ee4ea92a9e (REN-180).

Blast radius: 1 file, +14/-3, every changed line is a whole-line # comment and all sit at L15-35, above the first YAML key (L54 apiVersion), so none can be inside a block scalar. Comment-stripped file sha256 b4a664bf… on both master and branch (differs from the PR brief's 9c081e7e only by stripping method; equality is what counts). Master moved 7 commits since the merge-base, none touching this file. Applied object unchanged, no pod rolls.
Numbers re-measured 2026-09-26T08:25Z via the REN-36 read-only proxy: count by pod = kustomize 33, source 43, image-reflector 68, image-automation 1, helm 25 = 170; notification-controller absent. up{job="monitoring/flux-controllers"} = 6×1. notification-controller controller_runtime_reconcile_total: 12 series, sum 0.
git grep -E 'gotk_reconcile_duration|== *6|< *6' origin/master -- kubernetes/ hits only the old comment line 18; no shipped rule keys on the six claim.
Keep the guard-rail in the YAML: this file is what the next Flux rule author reads.
Revert: git revert of the merge, comment-only, nothing to undo in cluster.

Approve at ee4ea92a9e953accf8c98321b283a92f318f3f00 (REN-180). - Blast radius: 1 file, +14/-3, every changed line is a whole-line `#` comment and all sit at L15-35, above the first YAML key (L54 `apiVersion`), so none can be inside a block scalar. Comment-stripped file sha256 b4a664bf… on both master and branch (differs from the PR brief's 9c081e7e only by stripping method; equality is what counts). Master moved 7 commits since the merge-base, none touching this file. Applied object unchanged, no pod rolls. - Numbers re-measured 2026-09-26T08:25Z via the REN-36 read-only proxy: count by pod = kustomize 33, source 43, image-reflector 68, image-automation 1, helm 25 = 170; notification-controller absent. up{job="monitoring/flux-controllers"} = 6×1. notification-controller controller_runtime_reconcile_total: 12 series, sum 0. - `git grep -E 'gotk_reconcile_duration|== *6|< *6' origin/master -- kubernetes/` hits only the old comment line 18; no shipped rule keys on the six claim. - Keep the guard-rail in the YAML: this file is what the next Flux rule author reads. - Revert: `git revert` of the merge, comment-only, nothing to undo in cluster.

Ghost merged commit bc1c69d356 into master

2026-09-26 08:26:05 +00:00

Ghost referenced this issue from a commit

2026-09-26 08:26:06 +00:00

Merge pull request 'Correct the "all six controllers" claim on gotk_reconcile_duration_seconds (comment-only)' (#1023) from atlas/ren-127-podmonitor-comment into master

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#1023