Merge pull request 'feat(monitoring): scrape the Flux controllers (REN-97)' (#992) from atlas/ren-97-flux-metrics-alerts into master

kubernetes/apps/monitoring-rules/flux-podmonitor.yaml

@@ -0,0 +1,80 @@
+# Scrape the Flux controllers' own metrics. Until this existed Prometheus held
+# no `gotk_*` series at all, so nothing about Flux's own behaviour was
+# queryable and no rule could be written about it (REN-97).
+#
+# THIS IS THE SCRAPE ONLY, and the readiness alert is not merely deferred —
+# it cannot be written from this endpoint. `gotk_reconcile_condition` and
+# `gotk_suspend_status` are still DECLARED by fluxcd/pkg/runtime (v0.110.3,
+# runtime/metrics/recorder.go), but at the tags pinned in gotk-components.yaml
+# no controller calls the recorder that populates them: grepping the six
+# tagged source trees for RecordReadiness / RecordCondition / RecordSuspend
+# returns zero call sites, and an unpopulated GaugeVec exports no series. So
+# the readiness metrics would stay empty however this file is written. Flux
+# moved per-object readiness to kube-state-metrics custom-resource-state
+# (`gotk_resource_info{ready=...}`); that is REN-102, and it is a prerequisite
+# for the not-Ready alert rather than a follow-up to it.
+#
+# What this scrape does yield, and why it is still worth merging on its own:
+#   * `gotk_reconcile_duration_seconds` per kind/name/namespace (RecordDuration
+#     IS called, by all six controllers) — reconcile latency and, by its
+#     absence, an object that has stopped being reconciled at all.
+#   * controller-runtime's own series: reconcile totals and errors per
+#     controller, workqueue depth and retry counts, leader election.
+#   * the target itself, so `up{job="monitoring/flux-controllers"}` and the
+#     chart's default TargetDown rule start covering the Flux control plane.
+#   * the scrape REN-102 needs in place anyway.
+#
+# A PodMonitor, NOT a ServiceMonitor, and that is forced rather than preferred:
+# gotk-components.yaml creates Services for only two of the six controllers
+# (source-controller, notification-controller), and both publish port 80 ->
+# targetPort `http`, which is the artifact/webhook server on :9090 — not the
+# metrics port. kustomize-, helm-, image-reflector- and image-automation-
+# controller have no Service at all, so there is nothing for a ServiceMonitor to
+# select. All six pods do carry the same two labels and the same `http-prom`
+# (:8080) container port, so one PodMonitor covers the set.
+#
+# NO NETWORK POLICY CHANGE IS NEEDED, verified rather than assumed, and unlike
+# external-secrets-podmonitor.yaml which did need one: flux-system has no
+# CiliumNetworkPolicy of its own (infrastructure/network-policies/ has no
+# flux-system file), and gotk-components.yaml ships an `allow-scraping`
+# NetworkPolicy admitting every namespace to :8080 (`from: [namespaceSelector:
+# {}]`, port 8080). So `monitoring` is already admitted, as it was for REN-21.
+# Nothing in the Flux install is touched.
+apiVersion: monitoring.coreos.com/v1
+kind: PodMonitor
+metadata:
+  name: flux-controllers
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/part-of: flux
+    # Not required for discovery — podMonitorSelector is empty, see
+    # apps/monitoring/helmrelease-prometheus.yaml — but set for consistency
+    # with the other monitors in this directory.
+    release: kube-prometheus-stack
+spec:
+  # Lives in `monitoring` with a namespaceSelector rather than in flux-system,
+  # matching external-dns-servicemonitor.yaml and external-secrets-podmonitor.yaml.
+  # Flux owns every object in flux-system via gotk-components.yaml; keeping this
+  # out of that namespace means this change cannot interact with the Flux install
+  # itself.
+  namespaceSelector:
+    matchNames:
+      - flux-system
+  selector:
+    matchLabels:
+      app.kubernetes.io/part-of: flux
+      app.kubernetes.io/instance: flux-system
+  podMetricsEndpoints:
+    # Matches the controllers' `http-prom` port (containerPort 8080). Present on
+    # all six Deployments in gotk-components.yaml; a future controller without
+    # it simply yields no target rather than a broken scrape config.
+    - port: http-prom
+      path: /metrics
+      # 60s, not the 30s used elsewhere here, matching the ESO PodMonitor: the
+      # useful series are slow-moving per-object gauges and histograms over a
+      # few hundred Flux objects, and any rule built on them (REN-102) will use
+      # a multi-minute `for:`. 30s would double the samples for no gain in
+      # detection time — and Prometheus is already near its retention cap
+      # (REN-107).
+      interval: 60s
+      scrapeTimeout: 10s

kubernetes/apps/monitoring-rules/kustomization.yaml

@@ -52,4 +52,9 @@ resources:
   # infrastructure/network-policies/external-secrets-ns.yaml in the same change,
   # or it scrapes nothing. See the file.
   - external-secrets-podmonitor.yaml
+  # Flux's own control plane. The controllers were never scraped, so nothing
+  # about Flux's behaviour was queryable. Scrape only: per-object readiness is
+  # not on this endpoint at the pinned controller versions and needs
+  # kube-state-metrics instead. See the file.
+  - flux-podmonitor.yaml
   - probes-blackbox.yaml

kubernetes/infrastructure/image-automation/README.md

@@ -134,3 +134,32 @@ unaffected.
 - Image findings still surface in the security-scan agent's report
   (Skipped / "Flux image-automation owns this") so they remain visible
   even when an image isn't yet enrolled in automation here.
+
+## What watches this
+
+Worth stating explicitly, because for a while nothing did and each
+plausible reporter deferred to another: the update-audit agent skips
+`$imagepolicy`-marked images (`ci/agents/update-audit/prompt.md`), the
+security scan records them as "Flux image-automation owns this" (above),
+and both are correct to — an enrolled image is supposed to be this
+directory's problem.
+
+What exists today:
+
+- `flux-notifications/alerts.yaml` emails error-severity events from
+  every image kind. Immediate, but edge-triggered: one mail at the
+  moment of transition, then silence. Nothing says "still broken twelve
+  hours later" and nothing answers "is anything not-Ready right now".
+- `apps/monitoring-rules/flux-podmonitor.yaml` scrapes the controllers,
+  so reconcile latency and controller-level error counts are queryable.
+
+What does **not** exist yet: a standing per-object "this ImagePolicy is
+not Ready" alert. The Flux controllers at the versions pinned in
+`clusters/homelab/flux-system/gotk-components.yaml` no longer populate
+`gotk_reconcile_condition` — the metric is declared by their shared
+runtime library but never recorded — so per-object readiness has to come
+from kube-state-metrics custom-resource-state instead. That work, and
+the not-Ready alert built on it, is tracked separately; the same change
+covers a policy that is perfectly Ready while pinned to a tag its own
+filter has frozen (`tika` if upstream ships a 3.x), which no readiness
+signal would catch either.