ee4ea92a9e
#992's header comment says RecordDuration "IS called, by all six controllers", established by grepping the tagged upstream source trees. Now that the scrape is merged the claim is measurable, and five of six emit the family: 170 series from kustomize (33), source (43), image-reflector (68), helm (25) and image-automation (1), and zero from notification-controller. Five is permanent, not today-only. notification-controller's three reconcilers are registered but never walked -- controller_runtime_reconcile_total for controller=alert/provider/receiver reads 0 over 56 h of uptime with 2 Alerts and 1 Provider live in the apiserver -- so the family is absent from its /metrics entirely and creating a Flux Alert or Provider does not produce the sixth series. Comment-only. Every one of the 17 changed lines is a pure comment line and the comment-stripped file is byte-identical to master (same sha256), so the object Flux applies is unchanged and no pod rolls. Refs REN-127, REN-102, REN-97 Co-Authored-By: Paperclip <noreply@paperclip.ing> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
92 lines
5.2 KiB
YAML
92 lines
5.2 KiB
YAML
# Scrape the Flux controllers' own metrics. Until this existed Prometheus held
|
|
# no `gotk_*` series at all, so nothing about Flux's own behaviour was
|
|
# queryable and no rule could be written about it (REN-97).
|
|
#
|
|
# THIS IS THE SCRAPE ONLY, and the readiness alert is not merely deferred —
|
|
# it cannot be written from this endpoint. `gotk_reconcile_condition` and
|
|
# `gotk_suspend_status` are still DECLARED by fluxcd/pkg/runtime (v0.110.3,
|
|
# runtime/metrics/recorder.go), but at the tags pinned in gotk-components.yaml
|
|
# no controller calls the recorder that populates them: grepping the six
|
|
# tagged source trees for RecordReadiness / RecordCondition / RecordSuspend
|
|
# returns zero call sites, and an unpopulated GaugeVec exports no series. So
|
|
# the readiness metrics would stay empty however this file is written. Flux
|
|
# moved per-object readiness to kube-state-metrics custom-resource-state
|
|
# (`gotk_resource_info{ready=...}`); that is REN-102, and it is a prerequisite
|
|
# for the not-Ready alert rather than a follow-up to it.
|
|
#
|
|
# What this scrape does yield, and why it is still worth merging on its own:
|
|
# * `gotk_reconcile_duration_seconds` per kind/name/namespace — reconcile
|
|
# latency and, by its absence, an object that has stopped being reconciled
|
|
# at all. RecordDuration IS called, but by FIVE of the six controllers, not
|
|
# all six: measured post-merge, 170 series from kustomize (33), source (43),
|
|
# image-reflector (68), image-automation (1) and helm (25), and ZERO from
|
|
# notification-controller. That is not a gap waiting to fill in — its three
|
|
# reconcilers are registered but never walked (controller_runtime_reconcile_
|
|
# total for controller=alert/provider/receiver is 0 over 56 h of uptime with
|
|
# 2 Alerts and 1 Provider live), so the family is absent from that pod's
|
|
# /metrics entirely and five is the permanent value. DO NOT write a rule of
|
|
# the form `count(... gotk_reconcile_duration_seconds ...) < 6`, or an
|
|
# absent() per pod: it fires forever on notification-controller. Liveness for
|
|
# all six pods is `up{job="monitoring/flux-controllers"}` (below); that pod's
|
|
# reconcile health is `controller_runtime_reconcile_errors_total`. REN-127.
|
|
# * controller-runtime's own series: reconcile totals and errors per
|
|
# controller, workqueue depth and retry counts, leader election.
|
|
# * the target itself, so `up{job="monitoring/flux-controllers"}` and the
|
|
# chart's default TargetDown rule start covering the Flux control plane.
|
|
# * the scrape REN-102 needs in place anyway.
|
|
#
|
|
# A PodMonitor, NOT a ServiceMonitor, and that is forced rather than preferred:
|
|
# gotk-components.yaml creates Services for only two of the six controllers
|
|
# (source-controller, notification-controller), and both publish port 80 ->
|
|
# targetPort `http`, which is the artifact/webhook server on :9090 — not the
|
|
# metrics port. kustomize-, helm-, image-reflector- and image-automation-
|
|
# controller have no Service at all, so there is nothing for a ServiceMonitor to
|
|
# select. All six pods do carry the same two labels and the same `http-prom`
|
|
# (:8080) container port, so one PodMonitor covers the set.
|
|
#
|
|
# NO NETWORK POLICY CHANGE IS NEEDED, verified rather than assumed, and unlike
|
|
# external-secrets-podmonitor.yaml which did need one: flux-system has no
|
|
# CiliumNetworkPolicy of its own (infrastructure/network-policies/ has no
|
|
# flux-system file), and gotk-components.yaml ships an `allow-scraping`
|
|
# NetworkPolicy admitting every namespace to :8080 (`from: [namespaceSelector:
|
|
# {}]`, port 8080). So `monitoring` is already admitted, as it was for REN-21.
|
|
# Nothing in the Flux install is touched.
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PodMonitor
|
|
metadata:
|
|
name: flux-controllers
|
|
namespace: monitoring
|
|
labels:
|
|
app.kubernetes.io/part-of: flux
|
|
# Not required for discovery — podMonitorSelector is empty, see
|
|
# apps/monitoring/helmrelease-prometheus.yaml — but set for consistency
|
|
# with the other monitors in this directory.
|
|
release: kube-prometheus-stack
|
|
spec:
|
|
# Lives in `monitoring` with a namespaceSelector rather than in flux-system,
|
|
# matching external-dns-servicemonitor.yaml and external-secrets-podmonitor.yaml.
|
|
# Flux owns every object in flux-system via gotk-components.yaml; keeping this
|
|
# out of that namespace means this change cannot interact with the Flux install
|
|
# itself.
|
|
namespaceSelector:
|
|
matchNames:
|
|
- flux-system
|
|
selector:
|
|
matchLabels:
|
|
app.kubernetes.io/part-of: flux
|
|
app.kubernetes.io/instance: flux-system
|
|
podMetricsEndpoints:
|
|
# Matches the controllers' `http-prom` port (containerPort 8080). Present on
|
|
# all six Deployments in gotk-components.yaml; a future controller without
|
|
# it simply yields no target rather than a broken scrape config.
|
|
- port: http-prom
|
|
path: /metrics
|
|
# 60s, not the 30s used elsewhere here, matching the ESO PodMonitor: the
|
|
# useful series are slow-moving per-object gauges and histograms over a
|
|
# few hundred Flux objects, and any rule built on them (REN-102) will use
|
|
# a multi-minute `for:`. 30s would double the samples for no gain in
|
|
# detection time — and Prometheus is already near its retention cap
|
|
# (REN-107).
|
|
interval: 60s
|
|
scrapeTimeout: 10s
|