From c7d1b1ff4a2699db28edf7ee9ef5a6b219b5fd15 Mon Sep 17 00:00:00 2001 From: atlas-bot Date: Sat, 26 Sep 2026 00:26:29 +0000 Subject: [PATCH] feat(monitoring): scrape the Flux controllers (REN-97) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds a PodMonitor for the six Flux controllers in flux-system. Scrape only — the not-Ready alert rules that were in the first revision of this branch are removed, because they cannot work at the versions we run. Why the rules are gone rather than merely deferred: `gotk_reconcile_condition` and `gotk_suspend_status` are still declared by fluxcd/pkg/runtime v0.110.3 (runtime/metrics/recorder.go), but at the tags pinned in gotk-components.yaml (source/kustomize v1.9.5, helm v1.6.4, notification v1.9.4, image-reflector and image-automation v1.2.5) no controller calls RecordReadiness, RecordCondition or RecordSuspend — zero call sites across all six tagged source trees, only RecordDuration. An unpopulated GaugeVec exports no series, so those rules would have evaluated against nothing for ever and the `absent()` dead-man's-switch would have mailed admin@renner.uno every 12h from 30m after merge. Flux moved per-object readiness to kube-state-metrics custom-resource-state (`gotk_resource_info{ready=...}`), which is REN-102 — a prerequisite for the alert, not a follow-up to it. Caught in review by argus-bot on #992. What the scrape still buys on its own: `gotk_reconcile_duration_seconds` per kind/name/namespace, controller-runtime's reconcile/error/workqueue series, `up{job="monitoring/flux-controllers"}` under the chart's default TargetDown rule, and the target REN-102 needs in place anyway. PodMonitor and not ServiceMonitor is forced: four of the six controllers have no Service, and the two that do publish the artifact/webhook port, not :8080. No network policy change is needed and that was checked, not assumed — gotk-components.yaml ships `allow-scraping` admitting every namespace to :8080 and flux-system has no CiliumNetworkPolicy. Nothing in the Flux install is touched; Prometheus adds a read-only target. Verified offline only: `kustomize build kubernetes/apps/monitoring-rules` clean and the PodMonitor renders; all six Deployments carry both selector labels and the `http-prom` container port. Not applied to the cluster — no kubeconfig in this account's grants. Refs REN-97, REN-102. Co-Authored-By: Claude Opus 5 --- .../monitoring-rules/flux-podmonitor.yaml | 80 +++++++++++++++++++ .../apps/monitoring-rules/kustomization.yaml | 5 ++ .../infrastructure/image-automation/README.md | 29 +++++++ 3 files changed, 114 insertions(+) create mode 100644 kubernetes/apps/monitoring-rules/flux-podmonitor.yaml diff --git a/kubernetes/apps/monitoring-rules/flux-podmonitor.yaml b/kubernetes/apps/monitoring-rules/flux-podmonitor.yaml new file mode 100644 index 00000000..0d51d1c0 --- /dev/null +++ b/kubernetes/apps/monitoring-rules/flux-podmonitor.yaml @@ -0,0 +1,80 @@ +# Scrape the Flux controllers' own metrics. Until this existed Prometheus held +# no `gotk_*` series at all, so nothing about Flux's own behaviour was +# queryable and no rule could be written about it (REN-97). +# +# THIS IS THE SCRAPE ONLY, and the readiness alert is not merely deferred — +# it cannot be written from this endpoint. `gotk_reconcile_condition` and +# `gotk_suspend_status` are still DECLARED by fluxcd/pkg/runtime (v0.110.3, +# runtime/metrics/recorder.go), but at the tags pinned in gotk-components.yaml +# no controller calls the recorder that populates them: grepping the six +# tagged source trees for RecordReadiness / RecordCondition / RecordSuspend +# returns zero call sites, and an unpopulated GaugeVec exports no series. So +# the readiness metrics would stay empty however this file is written. Flux +# moved per-object readiness to kube-state-metrics custom-resource-state +# (`gotk_resource_info{ready=...}`); that is REN-102, and it is a prerequisite +# for the not-Ready alert rather than a follow-up to it. +# +# What this scrape does yield, and why it is still worth merging on its own: +# * `gotk_reconcile_duration_seconds` per kind/name/namespace (RecordDuration +# IS called, by all six controllers) — reconcile latency and, by its +# absence, an object that has stopped being reconciled at all. +# * controller-runtime's own series: reconcile totals and errors per +# controller, workqueue depth and retry counts, leader election. +# * the target itself, so `up{job="monitoring/flux-controllers"}` and the +# chart's default TargetDown rule start covering the Flux control plane. +# * the scrape REN-102 needs in place anyway. +# +# A PodMonitor, NOT a ServiceMonitor, and that is forced rather than preferred: +# gotk-components.yaml creates Services for only two of the six controllers +# (source-controller, notification-controller), and both publish port 80 -> +# targetPort `http`, which is the artifact/webhook server on :9090 — not the +# metrics port. kustomize-, helm-, image-reflector- and image-automation- +# controller have no Service at all, so there is nothing for a ServiceMonitor to +# select. All six pods do carry the same two labels and the same `http-prom` +# (:8080) container port, so one PodMonitor covers the set. +# +# NO NETWORK POLICY CHANGE IS NEEDED, verified rather than assumed, and unlike +# external-secrets-podmonitor.yaml which did need one: flux-system has no +# CiliumNetworkPolicy of its own (infrastructure/network-policies/ has no +# flux-system file), and gotk-components.yaml ships an `allow-scraping` +# NetworkPolicy admitting every namespace to :8080 (`from: [namespaceSelector: +# {}]`, port 8080). So `monitoring` is already admitted, as it was for REN-21. +# Nothing in the Flux install is touched. +apiVersion: monitoring.coreos.com/v1 +kind: PodMonitor +metadata: + name: flux-controllers + namespace: monitoring + labels: + app.kubernetes.io/part-of: flux + # Not required for discovery — podMonitorSelector is empty, see + # apps/monitoring/helmrelease-prometheus.yaml — but set for consistency + # with the other monitors in this directory. + release: kube-prometheus-stack +spec: + # Lives in `monitoring` with a namespaceSelector rather than in flux-system, + # matching external-dns-servicemonitor.yaml and external-secrets-podmonitor.yaml. + # Flux owns every object in flux-system via gotk-components.yaml; keeping this + # out of that namespace means this change cannot interact with the Flux install + # itself. + namespaceSelector: + matchNames: + - flux-system + selector: + matchLabels: + app.kubernetes.io/part-of: flux + app.kubernetes.io/instance: flux-system + podMetricsEndpoints: + # Matches the controllers' `http-prom` port (containerPort 8080). Present on + # all six Deployments in gotk-components.yaml; a future controller without + # it simply yields no target rather than a broken scrape config. + - port: http-prom + path: /metrics + # 60s, not the 30s used elsewhere here, matching the ESO PodMonitor: the + # useful series are slow-moving per-object gauges and histograms over a + # few hundred Flux objects, and any rule built on them (REN-102) will use + # a multi-minute `for:`. 30s would double the samples for no gain in + # detection time — and Prometheus is already near its retention cap + # (REN-107). + interval: 60s + scrapeTimeout: 10s diff --git a/kubernetes/apps/monitoring-rules/kustomization.yaml b/kubernetes/apps/monitoring-rules/kustomization.yaml index 1b3afcf9..f7e01483 100644 --- a/kubernetes/apps/monitoring-rules/kustomization.yaml +++ b/kubernetes/apps/monitoring-rules/kustomization.yaml @@ -52,4 +52,9 @@ resources: # infrastructure/network-policies/external-secrets-ns.yaml in the same change, # or it scrapes nothing. See the file. - external-secrets-podmonitor.yaml + # Flux's own control plane. The controllers were never scraped, so nothing + # about Flux's behaviour was queryable. Scrape only: per-object readiness is + # not on this endpoint at the pinned controller versions and needs + # kube-state-metrics instead. See the file. + - flux-podmonitor.yaml - probes-blackbox.yaml diff --git a/kubernetes/infrastructure/image-automation/README.md b/kubernetes/infrastructure/image-automation/README.md index 5bdf396b..b7350427 100644 --- a/kubernetes/infrastructure/image-automation/README.md +++ b/kubernetes/infrastructure/image-automation/README.md @@ -134,3 +134,32 @@ unaffected. - Image findings still surface in the security-scan agent's report (Skipped / "Flux image-automation owns this") so they remain visible even when an image isn't yet enrolled in automation here. + +## What watches this + +Worth stating explicitly, because for a while nothing did and each +plausible reporter deferred to another: the update-audit agent skips +`$imagepolicy`-marked images (`ci/agents/update-audit/prompt.md`), the +security scan records them as "Flux image-automation owns this" (above), +and both are correct to — an enrolled image is supposed to be this +directory's problem. + +What exists today: + +- `flux-notifications/alerts.yaml` emails error-severity events from + every image kind. Immediate, but edge-triggered: one mail at the + moment of transition, then silence. Nothing says "still broken twelve + hours later" and nothing answers "is anything not-Ready right now". +- `apps/monitoring-rules/flux-podmonitor.yaml` scrapes the controllers, + so reconcile latency and controller-level error counts are queryable. + +What does **not** exist yet: a standing per-object "this ImagePolicy is +not Ready" alert. The Flux controllers at the versions pinned in +`clusters/homelab/flux-system/gotk-components.yaml` no longer populate +`gotk_reconcile_condition` — the metric is declared by their shared +runtime library but never recorded — so per-object readiness has to come +from kube-state-metrics custom-resource-state instead. That work, and +the not-Ready alert built on it, is tracked separately; the same change +covers a policy that is perfectly Ready while pinned to a tag its own +filter has frozen (`tika` if upstream ships a 3.x), which no readiness +signal would catch either. -- 2.54.0