diff --git a/kubernetes/apps/monitoring-rules/flux-podmonitor.yaml b/kubernetes/apps/monitoring-rules/flux-podmonitor.yaml new file mode 100644 index 00000000..0d51d1c0 --- /dev/null +++ b/kubernetes/apps/monitoring-rules/flux-podmonitor.yaml @@ -0,0 +1,80 @@ +# Scrape the Flux controllers' own metrics. Until this existed Prometheus held +# no `gotk_*` series at all, so nothing about Flux's own behaviour was +# queryable and no rule could be written about it (REN-97). +# +# THIS IS THE SCRAPE ONLY, and the readiness alert is not merely deferred — +# it cannot be written from this endpoint. `gotk_reconcile_condition` and +# `gotk_suspend_status` are still DECLARED by fluxcd/pkg/runtime (v0.110.3, +# runtime/metrics/recorder.go), but at the tags pinned in gotk-components.yaml +# no controller calls the recorder that populates them: grepping the six +# tagged source trees for RecordReadiness / RecordCondition / RecordSuspend +# returns zero call sites, and an unpopulated GaugeVec exports no series. So +# the readiness metrics would stay empty however this file is written. Flux +# moved per-object readiness to kube-state-metrics custom-resource-state +# (`gotk_resource_info{ready=...}`); that is REN-102, and it is a prerequisite +# for the not-Ready alert rather than a follow-up to it. +# +# What this scrape does yield, and why it is still worth merging on its own: +# * `gotk_reconcile_duration_seconds` per kind/name/namespace (RecordDuration +# IS called, by all six controllers) — reconcile latency and, by its +# absence, an object that has stopped being reconciled at all. +# * controller-runtime's own series: reconcile totals and errors per +# controller, workqueue depth and retry counts, leader election. +# * the target itself, so `up{job="monitoring/flux-controllers"}` and the +# chart's default TargetDown rule start covering the Flux control plane. +# * the scrape REN-102 needs in place anyway. +# +# A PodMonitor, NOT a ServiceMonitor, and that is forced rather than preferred: +# gotk-components.yaml creates Services for only two of the six controllers +# (source-controller, notification-controller), and both publish port 80 -> +# targetPort `http`, which is the artifact/webhook server on :9090 — not the +# metrics port. kustomize-, helm-, image-reflector- and image-automation- +# controller have no Service at all, so there is nothing for a ServiceMonitor to +# select. All six pods do carry the same two labels and the same `http-prom` +# (:8080) container port, so one PodMonitor covers the set. +# +# NO NETWORK POLICY CHANGE IS NEEDED, verified rather than assumed, and unlike +# external-secrets-podmonitor.yaml which did need one: flux-system has no +# CiliumNetworkPolicy of its own (infrastructure/network-policies/ has no +# flux-system file), and gotk-components.yaml ships an `allow-scraping` +# NetworkPolicy admitting every namespace to :8080 (`from: [namespaceSelector: +# {}]`, port 8080). So `monitoring` is already admitted, as it was for REN-21. +# Nothing in the Flux install is touched. +apiVersion: monitoring.coreos.com/v1 +kind: PodMonitor +metadata: + name: flux-controllers + namespace: monitoring + labels: + app.kubernetes.io/part-of: flux + # Not required for discovery — podMonitorSelector is empty, see + # apps/monitoring/helmrelease-prometheus.yaml — but set for consistency + # with the other monitors in this directory. + release: kube-prometheus-stack +spec: + # Lives in `monitoring` with a namespaceSelector rather than in flux-system, + # matching external-dns-servicemonitor.yaml and external-secrets-podmonitor.yaml. + # Flux owns every object in flux-system via gotk-components.yaml; keeping this + # out of that namespace means this change cannot interact with the Flux install + # itself. + namespaceSelector: + matchNames: + - flux-system + selector: + matchLabels: + app.kubernetes.io/part-of: flux + app.kubernetes.io/instance: flux-system + podMetricsEndpoints: + # Matches the controllers' `http-prom` port (containerPort 8080). Present on + # all six Deployments in gotk-components.yaml; a future controller without + # it simply yields no target rather than a broken scrape config. + - port: http-prom + path: /metrics + # 60s, not the 30s used elsewhere here, matching the ESO PodMonitor: the + # useful series are slow-moving per-object gauges and histograms over a + # few hundred Flux objects, and any rule built on them (REN-102) will use + # a multi-minute `for:`. 30s would double the samples for no gain in + # detection time — and Prometheus is already near its retention cap + # (REN-107). + interval: 60s + scrapeTimeout: 10s diff --git a/kubernetes/apps/monitoring-rules/kustomization.yaml b/kubernetes/apps/monitoring-rules/kustomization.yaml index 1b3afcf9..f7e01483 100644 --- a/kubernetes/apps/monitoring-rules/kustomization.yaml +++ b/kubernetes/apps/monitoring-rules/kustomization.yaml @@ -52,4 +52,9 @@ resources: # infrastructure/network-policies/external-secrets-ns.yaml in the same change, # or it scrapes nothing. See the file. - external-secrets-podmonitor.yaml + # Flux's own control plane. The controllers were never scraped, so nothing + # about Flux's behaviour was queryable. Scrape only: per-object readiness is + # not on this endpoint at the pinned controller versions and needs + # kube-state-metrics instead. See the file. + - flux-podmonitor.yaml - probes-blackbox.yaml diff --git a/kubernetes/infrastructure/image-automation/README.md b/kubernetes/infrastructure/image-automation/README.md index 5bdf396b..b7350427 100644 --- a/kubernetes/infrastructure/image-automation/README.md +++ b/kubernetes/infrastructure/image-automation/README.md @@ -134,3 +134,32 @@ unaffected. - Image findings still surface in the security-scan agent's report (Skipped / "Flux image-automation owns this") so they remain visible even when an image isn't yet enrolled in automation here. + +## What watches this + +Worth stating explicitly, because for a while nothing did and each +plausible reporter deferred to another: the update-audit agent skips +`$imagepolicy`-marked images (`ci/agents/update-audit/prompt.md`), the +security scan records them as "Flux image-automation owns this" (above), +and both are correct to — an enrolled image is supposed to be this +directory's problem. + +What exists today: + +- `flux-notifications/alerts.yaml` emails error-severity events from + every image kind. Immediate, but edge-triggered: one mail at the + moment of transition, then silence. Nothing says "still broken twelve + hours later" and nothing answers "is anything not-Ready right now". +- `apps/monitoring-rules/flux-podmonitor.yaml` scrapes the controllers, + so reconcile latency and controller-level error counts are queryable. + +What does **not** exist yet: a standing per-object "this ImagePolicy is +not Ready" alert. The Flux controllers at the versions pinned in +`clusters/homelab/flux-system/gotk-components.yaml` no longer populate +`gotk_reconcile_condition` — the metric is declared by their shared +runtime library but never recorded — so per-object readiness has to come +from kube-state-metrics custom-resource-state instead. That work, and +the not-Ready alert built on it, is tracked separately; the same change +covers a policy that is perfectly Ready while pinned to a tag its own +filter has frozen (`tika` if upstream ships a 3.x), which no readiness +signal would catch either.