feat(monitoring): Flux per-object readiness via kube-state-metrics (REN-102)
Per-object Flux readiness cannot come from the controllers: gotk_reconcile_ condition and gotk_suspend_status are declared by fluxcd/pkg/runtime and populated by nothing at the pinned tags, so the REN-97 scrape yields 2,273 gotk_* series and none of them are about readiness. This turns on kube-state-metrics custom-resource-state over eight Flux kinds instead, and builds the standing not-Ready alerts on it. The not-Ready half is complete. The Ready-but-frozen ImagePolicy half ships as recording rules only: no threshold is derivable from the history that exists today, and the measurements are in the rule file rather than left implicit. - kube-state-metrics customResourceState + rbac.extraRules (list, watch only) - prometheusrule-flux.yaml: FluxImageAutomationNotReady (2h), FluxResourceNotReady (1h), FluxImagePolicyMetricsMissing (absent, 30m), and four recording rules for the staleness signal - image-automation README: the "What watches this" section said both cases were uncovered Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
atlas-bot
co-authored by
Claude Opus 5
Claude Opus 5
parent
10240ae50b
commit
088db00e62
@@ -57,4 +57,9 @@ resources:
|
||||
# not on this endpoint at the pinned controller versions and needs
|
||||
# kube-state-metrics instead. See the file.
|
||||
- flux-podmonitor.yaml
|
||||
# The readiness half the PodMonitor above cannot provide. Reads
|
||||
# `gotk_resource_info`, which comes from the kube-state-metrics
|
||||
# customResourceState block in apps/monitoring/helmrelease-prometheus.yaml —
|
||||
# these two files must land and be reverted together.
|
||||
- prometheusrule-flux.yaml
|
||||
- probes-blackbox.yaml
|
||||
|
||||
@@ -0,0 +1,274 @@
|
||||
# Per-object Flux readiness, and the groundwork for the Ready-but-frozen
|
||||
# ImagePolicy signal (REN-102, closing REN-97).
|
||||
#
|
||||
# EVERYTHING HERE READS `gotk_resource_info`, WHICH IS A kube-state-metrics
|
||||
# SERIES, NOT A FLUX CONTROLLER SERIES. apps/monitoring-rules/flux-podmonitor.yaml
|
||||
# scrapes the controllers and yields reconcile durations and controller-runtime
|
||||
# counters, but readiness is not on that endpoint at the pinned controller
|
||||
# versions: `gotk_reconcile_condition` and `gotk_suspend_status` are declared by
|
||||
# fluxcd/pkg/runtime and populated by nothing. The series these rules read are
|
||||
# created by the `customResourceState` block in
|
||||
# apps/monitoring/helmrelease-prometheus.yaml. If that block is removed, every
|
||||
# rule below goes blind — which is what FluxImagePolicyMetricsMissing exists to
|
||||
# say out loud.
|
||||
#
|
||||
# THE STALENESS HALF DELIBERATELY SHIPS WITHOUT AN ALERT. See the second group.
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
metadata:
|
||||
name: flux
|
||||
namespace: monitoring
|
||||
labels:
|
||||
# ruleSelector matches on this label; without it the rule is created but
|
||||
# never loaded into Prometheus.
|
||||
release: kube-prometheus-stack
|
||||
spec:
|
||||
groups:
|
||||
- name: flux.rules
|
||||
rules:
|
||||
# Why `max by (...)` and not the bare selector: it collapses the KSM
|
||||
# target labels (`pod`, `instance`, `endpoint`, ...). Without it, a
|
||||
# kube-state-metrics rollout changes `pod`, which resolves the alert on
|
||||
# the old series and re-fires it on the new one — a fresh mail for an
|
||||
# unchanged failure. KSM rolled 11 times in the last 30 days, so this is
|
||||
# routine, not hypothetical.
|
||||
#
|
||||
# Why `ready=~"False|Unknown"` and not `ready="False"`: Flux sets Ready
|
||||
# False on failure but Unknown while a reconcile is in flight, and an
|
||||
# object wedged in Unknown is stuck just as hard as one that is False —
|
||||
# this repo has already had a Progressing CephCluster deadlock the whole
|
||||
# apps layer (see rook-servicemonitors.yaml). Matching the two values
|
||||
# explicitly, rather than `ready!="True"`, keeps an object whose `ready`
|
||||
# label is absent entirely (freshly created, condition not yet written)
|
||||
# out of the alert.
|
||||
#
|
||||
# Why `suspended!="true"` and not an `unless` join: `suspended` is a
|
||||
# label on the same Info series, so one matcher does what the withdrawn
|
||||
# draft needed a second vector for. The `!=` form also covers kinds where
|
||||
# the label is absent. There is no deleted-object guard because KSM stops
|
||||
# emitting the series when the object goes away — under `gotk_*` that
|
||||
# guard had to be written, here it is structural.
|
||||
#
|
||||
# for: 2h — 31 of the 34 ImageRepositories scan at `interval: 1h`
|
||||
# (2 at 30m: shelly-manager, vsftpd; 1 at 10m: wiki), so 91% of the
|
||||
# population cannot recover from a transient registry failure faster
|
||||
# than one hour whatever this value says. Anything under 1h mails about
|
||||
# a single self-healing missed scan. 2h means two consecutive failed
|
||||
# scans. Detection latency added by going through KSM is one 30s scrape.
|
||||
- alert: FluxImageAutomationNotReady
|
||||
expr: >-
|
||||
max by (customresource_kind, name, exported_namespace) (
|
||||
gotk_resource_info{
|
||||
customresource_kind=~"ImageRepository|ImagePolicy|ImageUpdateAutomation",
|
||||
ready=~"False|Unknown",
|
||||
suspended!="true"
|
||||
}
|
||||
)
|
||||
for: 2h
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Flux {{ $labels.customresource_kind }}/{{ $labels.name }} has not been Ready for 2h"
|
||||
description: >-
|
||||
An image-automation object has been not-Ready for two reconcile
|
||||
intervals. Nothing is broken in the running cluster, but the image
|
||||
it governs has stopped moving, security rebuilds included.
|
||||
`flux get image all -n {{ $labels.exported_namespace }}` gives the
|
||||
reason; the usual causes are registry credentials, a rate limit, or
|
||||
a tag format the policy's `pattern`/`extract` no longer matches.
|
||||
flux-notifications/alerts.yaml already mailed once at the moment of
|
||||
transition — this is the "still broken two hours later" signal that
|
||||
edge-triggered mail cannot give.
|
||||
|
||||
# The exact negation of the matcher above, so a kind added to
|
||||
# helmrelease-prometheus.yaml's customResourceState config is covered by
|
||||
# one of these two rules without anyone remembering to widen a list.
|
||||
#
|
||||
# for: 1h, derived from the intervals actually in this repo: all 33
|
||||
# Kustomizations are at 10m (six retries inside the hour), all 25
|
||||
# HelmReleases at 30m (two), and the flux-system GitRepository at 1m0s
|
||||
# (sixty). The dominant non-failure source of ready="False" is a
|
||||
# `dependsOn` wait, whose `reason` is not a label and so cannot be
|
||||
# filtered out; an hour clears a whole-cluster reconcile chain.
|
||||
#
|
||||
# KNOWN TRADE, stated rather than hidden: this matcher also picks up the
|
||||
# 17 HelmRepository and 4 OCIRepository objects, and every one of those is
|
||||
# at `interval: 24h`. A failed index or manifest fetch there cannot
|
||||
# self-heal inside an hour, so a transient failure yields up to two mails
|
||||
# (repeat_interval is 12h) before the next scheduled reconcile. That is
|
||||
# accepted as correct — a chart source stale for an hour is worth knowing
|
||||
# — and it is real information rather than a blip. If it turns out to be
|
||||
# the noisy one, the fix is a third rule for the 24h-interval kinds, not
|
||||
# a global bump of this value, which would make Kustomization alerts
|
||||
# useless.
|
||||
- alert: FluxResourceNotReady
|
||||
expr: >-
|
||||
max by (customresource_kind, name, exported_namespace) (
|
||||
gotk_resource_info{
|
||||
customresource_kind!~"ImageRepository|ImagePolicy|ImageUpdateAutomation",
|
||||
ready=~"False|Unknown",
|
||||
suspended!="true"
|
||||
}
|
||||
)
|
||||
for: 1h
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "Flux {{ $labels.customresource_kind }}/{{ $labels.name }} has not been Ready for 1h"
|
||||
description: >-
|
||||
A Flux object outside image automation has been not-Ready for an
|
||||
hour, so the desired state in git is not reaching the cluster.
|
||||
`flux get all -A --status-selector ready=false` lists them with a
|
||||
reason. For a Kustomization or HelmRelease this means a rollout is
|
||||
stuck; for a HelmRepository or OCIRepository it means the chart or
|
||||
artifact index is stale and dependent releases will not upgrade.
|
||||
|
||||
# THE DEAD MAN'S SWITCH, and the reason it is safe here and was not in
|
||||
# #992. This asserts that the 34 ImagePolicy series exist — and they
|
||||
# exist only because this change created them. Merged against `gotk_*`
|
||||
# alone, as the withdrawn draft had it, this rule would have been
|
||||
# permanently true and mailed admin@renner.uno every 12h forever.
|
||||
#
|
||||
# Its value is catching the removal of the customResourceState block, or
|
||||
# a malformed edit to it, which would otherwise take all three rules
|
||||
# above silently blind. `KubeStateMetricsDown` and `TargetDown` overlap it
|
||||
# when the whole pod is down; neither notices a config that parses but
|
||||
# emits nothing.
|
||||
#
|
||||
# for: 30m rides out the kube-state-metrics pod roll that applying this
|
||||
# change itself causes, and any later roll.
|
||||
- alert: FluxImagePolicyMetricsMissing
|
||||
expr: absent(gotk_resource_info{customresource_kind="ImagePolicy"})
|
||||
for: 30m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "Flux per-object readiness metrics have stopped being exported"
|
||||
description: >-
|
||||
kube-state-metrics is exporting no ImagePolicy series, so
|
||||
FluxImageAutomationNotReady and FluxResourceNotReady cannot fire and
|
||||
Flux readiness is unmonitored. Check the `customResourceState` block
|
||||
in apps/monitoring/helmrelease-prometheus.yaml, then the
|
||||
kube-state-metrics pod log — a custom-resource-state config that
|
||||
fails to parse is logged there and yields no series.
|
||||
|
||||
# The Ready-but-frozen ImagePolicy signal: a policy whose `filterTags` or
|
||||
# semver range excludes everything new keeps resolving the tag it resolved
|
||||
# months ago, holds Ready=True the whole time, and no readiness rule can see
|
||||
# it. `tika.yaml`'s own comment names the case.
|
||||
#
|
||||
# THIS GROUP DELIBERATELY CONTAINS NO ALERT, AND THAT IS A MEASURED DECISION,
|
||||
# NOT AN OMISSION. Every candidate threshold was tested against the real
|
||||
# release history and the real registries on 2026-09-26:
|
||||
#
|
||||
# * A flat "frozen for N days" duration does not exist. Over the 191
|
||||
# automated bump commits in the 147 days to 2026-09-25, gatus went 127
|
||||
# days between bumps and paperless-ngx 126, both provably healthy — they
|
||||
# bumped. Six policies have never bumped at all. Any N below 127 fires on
|
||||
# the healthy pair; any N above 147 is longer than the whole observation
|
||||
# window.
|
||||
#
|
||||
# * The "tag count grew >= 10%" comparator from this card's first design is
|
||||
# wrong, and the registries say so. Of the 18 policies frozen 14 days or
|
||||
# more, a 10% threshold fires on exactly three — gotenberg (+87 tags/14d
|
||||
# on 526), valkey (+68 on 301), eclipse-mosquitto (+13 on 90) — and all
|
||||
# three churn variant tags their filters reject on purpose. It does NOT
|
||||
# fire on tika (+4 on 136 = 3.0%), which is the canonical case this card
|
||||
# was opened for.
|
||||
#
|
||||
# * Relaxing to "grew by >= 1 tag" does catch tika, but then also fires on
|
||||
# those same three plus velero-plugin-for-aws, whose `1.13.x` pin is
|
||||
# deliberate. Alertmanager routes every severity to the same email
|
||||
# receiver with repeat_interval: 12h, so that is four standing mails
|
||||
# every twelve hours, indefinitely, for four non-faults.
|
||||
#
|
||||
# * `.status.lastScanResult.latestTags` looks like the way out and is not:
|
||||
# the CRD defines it as "the first 10 tags when sorting all the tags in
|
||||
# descending ALPHABETICAL order", which is not newest-first for any tag
|
||||
# scheme in this repo.
|
||||
#
|
||||
# So the recorded series below materialise the candidate set and, more to the
|
||||
# point, START THE HISTORY that a threshold has to be derived from — a real
|
||||
# time series of tagCount per repository, which has never existed here. The
|
||||
# alert is deferred to a follow-up card that reads 14+ days of it. Recording
|
||||
# rules cannot mail anyone, so this costs nothing while it warms up.
|
||||
#
|
||||
# interval: 15m is LOAD-BEARING and is declared here rather than inherited.
|
||||
# Global evaluationInterval is 30s and no other rule group in this directory
|
||||
# sets one (45 of the 46 loaded groups run at 30s). The `>= 1250` warm-up
|
||||
# guard below counts samples, so at 30s it would be satisfied in 10.4 hours
|
||||
# instead of 13 days and the frozen series would read true for policies that
|
||||
# had not been observed frozen at all.
|
||||
- name: flux-image-policy-staleness.rules
|
||||
interval: 15m
|
||||
rules:
|
||||
# Stage one exists only to strip the KSM target labels. Counting distinct
|
||||
# tags means counting distinct series, and `gotk_resource_info` carries
|
||||
# `pod` and `instance` — there were 11 distinct kube-state-metrics pod
|
||||
# identities in the last 30 days, so counting the raw series would report
|
||||
# "several tags seen" after every roll, including the roll this change
|
||||
# itself causes, and would keep reporting it for the whole window.
|
||||
- record: flux:image_policy_tag:info
|
||||
expr: >-
|
||||
max by (name, exported_namespace, source_name, tag) (
|
||||
gotk_resource_info{customresource_kind="ImagePolicy"}
|
||||
)
|
||||
|
||||
# Distinct resolved tags seen in a trailing day. A tag change retires one
|
||||
# series and starts another, so `count by (...)` over `count_over_time`
|
||||
# is a count of tag values, and `== 1` means the tag did not move.
|
||||
- record: flux:image_policy_tags_seen:1d
|
||||
expr: >-
|
||||
count by (name, exported_namespace, source_name) (
|
||||
count_over_time(flux:image_policy_tag:info[1d])
|
||||
)
|
||||
|
||||
# The repository side, keyed by `source_name` so it joins on the actual
|
||||
# `spec.imageRepositoryRef.name` reference rather than on the convention
|
||||
# that a policy is named after its repository. All 34 pairs match by name
|
||||
# today, so this is drift insurance, not a bug fix.
|
||||
- record: flux:image_repository_tag_count:1d
|
||||
expr: >-
|
||||
max by (source_name, exported_namespace) (
|
||||
label_replace(
|
||||
max_over_time(gotk_image_repository_tag_count[1d]),
|
||||
"source_name", "$1", "name", "(.+)"
|
||||
)
|
||||
)
|
||||
|
||||
# The candidate set: resolved tag unmoved for 14 days while the
|
||||
# repository's tag count is genuinely higher than it was.
|
||||
#
|
||||
# 14d, not the 30d this card was first designed around, because 30d does
|
||||
# not fit. `retentionSize: 30GB` binds long before `retention: 90d`:
|
||||
# (time() - prometheus_tsdb_lowest_timestamp/1000)/86400 measured 26.63
|
||||
# days on 2026-09-26, at 84.5% of the size cap and growing ~1.09 GB/day
|
||||
# (REN-107). A 30d window can never be satisfied. 14d uses 53% of that
|
||||
# floor.
|
||||
#
|
||||
# >= 1250 is the warm-up guard and its arithmetic is: at interval 15m a
|
||||
# full 14d window holds 14*24*4 = 1344 samples, so 1250 is 93% of a full
|
||||
# window = 13.02 days of history, leaving ~23h of slack for Prometheus
|
||||
# restarts and KSM rolls. Without it the series would read true on day
|
||||
# one, when "unmoved for 14 days" is simply not yet knowable. 13.02 days
|
||||
# against a 26.63-day floor is 2.0x margin; re-check it with the
|
||||
# lowest_timestamp query above if retention is ever cut.
|
||||
#
|
||||
# `last_over_time - min_over_time >= 1` and not a max/min ratio: a ratio
|
||||
# is direction-agnostic and is satisfied by a tag count that DROPPED
|
||||
# (registry pruning, an edited exclusionList, a partial scan) exactly as
|
||||
# by one that grew. Comparing the current value against the window
|
||||
# minimum only counts net growth.
|
||||
- record: flux:image_policy_tag_frozen:14d
|
||||
expr: >-
|
||||
(
|
||||
max_over_time(flux:image_policy_tags_seen:1d[14d]) == 1
|
||||
and
|
||||
count_over_time(flux:image_policy_tags_seen:1d[14d]) >= 1250
|
||||
)
|
||||
and on (source_name, exported_namespace) (
|
||||
(
|
||||
last_over_time(flux:image_repository_tag_count:1d[1h])
|
||||
- min_over_time(flux:image_repository_tag_count:1d[14d])
|
||||
) >= 1
|
||||
)
|
||||
@@ -360,6 +360,285 @@ spec:
|
||||
limits:
|
||||
memory: 256Mi
|
||||
|
||||
# --- Flux per-object readiness (REN-102) ---------------------------------
|
||||
#
|
||||
# WHY THIS IS HERE AND NOT ON THE FLUX CONTROLLERS. `gotk_reconcile_condition`
|
||||
# and `gotk_suspend_status` are still declared by fluxcd/pkg/runtime, but no
|
||||
# controller populates them at the tags pinned in
|
||||
# clusters/homelab/flux-system/gotk-components.yaml, and an unpopulated
|
||||
# GaugeVec exports no series. Verified after the REN-97 scrape landed: the
|
||||
# PodMonitor now yields 2,273 `gotk_*` series and
|
||||
# `count({__name__=~"gotk_reconcile_condition|gotk_suspend_status"})` is still
|
||||
# empty. Upstream moved per-object readiness to kube-state-metrics
|
||||
# custom-resource-state, which is what this block turns on.
|
||||
#
|
||||
# DO NOT ADD `collectors: []` OR `--custom-resource-state-only=true`.
|
||||
# Upstream's reference config (fluxcd/flux2-monitoring-example,
|
||||
# monitoring/controllers/kube-prometheus-stack/kube-state-metrics-config.yaml)
|
||||
# opens with exactly those two lines. Copying them replaces every built-in
|
||||
# collector with the Flux ones: measured 2026-09-26, that would take
|
||||
# `count({job="kube-state-metrics"})` from 27,792 series to ~183 and blind
|
||||
# every `Kube*` alert in the chart. Both keys are deliberately absent below.
|
||||
#
|
||||
# ROLLS THE kube-state-metrics POD. Enabling customResourceState adds an arg
|
||||
# and a volume to the Deployment, so the pod is replaced once on apply. That
|
||||
# is the whole blast radius of this block — KSM only watches, it writes
|
||||
# nothing, and no Flux object or workload is touched. During the roll (a few
|
||||
# seconds) `kube_*` has a gap; `KubeStateMetricsDown` and `TargetDown` cover
|
||||
# it if it does not come back.
|
||||
#
|
||||
# NOTE FOR LATER EDITS: the subchart puts no config checksum on the
|
||||
# Deployment's pod template, so editing `config` below changes the ConfigMap
|
||||
# WITHOUT rolling the pod, and KSM keeps serving the old config until it
|
||||
# restarts for another reason. Any future change to this config needs a
|
||||
# deliberate pod restart to take effect.
|
||||
#
|
||||
# RBAC: list + watch, nothing else. `get` is NOT needed — KSM drives
|
||||
# reflectors, never point-reads. Also note the chart adds
|
||||
# `customresourcedefinitions: [list, watch]` on its own via
|
||||
# rbac.customResourceState.createClusterRoleRules (default true); KSM needs it
|
||||
# to resolve the GVKs below. Both land in the existing
|
||||
# kube-prometheus-stack-kube-state-metrics ClusterRole rather than a second
|
||||
# binding to maintain. No write verb, no wildcard, no cluster-admin.
|
||||
rbac:
|
||||
extraRules:
|
||||
- apiGroups:
|
||||
- source.toolkit.fluxcd.io
|
||||
- kustomize.toolkit.fluxcd.io
|
||||
- helm.toolkit.fluxcd.io
|
||||
- image.toolkit.fluxcd.io
|
||||
# Trimmed to the kinds this cluster actually has. Omitted from
|
||||
# upstream's list, each for a checked reason:
|
||||
# buckets, receivers — zero objects exist in this repo.
|
||||
# helmcharts — created by source-controller at
|
||||
# runtime, one per HelmRelease; a not-Ready HelmChart is always
|
||||
# mirrored by the HelmRelease that owns it, so alerting on both
|
||||
# is one failure reported twice.
|
||||
# alerts, providers — notification.toolkit.fluxcd.io/v1beta3
|
||||
# Alert and Provider have NO status.conditions in the CRD at all
|
||||
# (checked in gotk-components.yaml), so there is no readiness to
|
||||
# report; upstream omits `ready` for them for the same reason. A
|
||||
# series carrying only `suspended` is not alertable.
|
||||
resources:
|
||||
- gitrepositories
|
||||
- helmrepositories
|
||||
- ocirepositories
|
||||
- kustomizations
|
||||
- helmreleases
|
||||
- imagerepositories
|
||||
- imagepolicies
|
||||
- imageupdateautomations
|
||||
verbs: ["list", "watch"]
|
||||
|
||||
customResourceState:
|
||||
enabled: true
|
||||
config:
|
||||
spec:
|
||||
resources:
|
||||
# Label set is upstream's, minus the churning ones. Dropped
|
||||
# deliberately: `revision` (status.artifact.revision /
|
||||
# lastAppliedRevision — a git SHA, so every commit that reaches a
|
||||
# Kustomization mints a NEW SERIES), and `url` / `endpoint`.
|
||||
# Prometheus is at 84.5% of its 30GB retentionSize cap (REN-107);
|
||||
# a label that changes several times a day across 33 Kustomizations
|
||||
# is exactly what must not be added. Kept: `ready` and `suspended`
|
||||
# (what the alerts read), `name` / `exported_namespace` (identity),
|
||||
# `source_name` (the join key, see prometheusrule-flux.yaml), plus
|
||||
# `image` and `tag` on the two image kinds, which are stable enough
|
||||
# to ride the existing series.
|
||||
#
|
||||
# `exported_namespace` is upstream's own label name for this, not a
|
||||
# Prometheus collision rename: the chart's KSM ServiceMonitor runs
|
||||
# honorLabels, so `namespace` on a KSM series is already the
|
||||
# OBJECT's namespace and nothing gets renamed. The rules use the
|
||||
# same string because the config declares it.
|
||||
#
|
||||
# KSM adds `customresource_group`, `customresource_version` and
|
||||
# `customresource_kind` to every custom-resource-state series by
|
||||
# itself. The rules key off `customresource_kind`.
|
||||
- groupVersionKind:
|
||||
group: kustomize.toolkit.fluxcd.io
|
||||
version: v1
|
||||
kind: Kustomization
|
||||
metricNamePrefix: gotk
|
||||
errorLogV: 10
|
||||
metrics:
|
||||
- name: "resource_info"
|
||||
help: "The current state of a Flux Kustomization resource."
|
||||
each:
|
||||
type: Info
|
||||
info:
|
||||
labelsFromPath:
|
||||
name: [metadata, name]
|
||||
labelsFromPath:
|
||||
exported_namespace: [metadata, namespace]
|
||||
ready: [status, conditions, "[type=Ready]", status]
|
||||
suspended: [spec, suspend]
|
||||
source_name: [spec, sourceRef, name]
|
||||
- groupVersionKind:
|
||||
group: helm.toolkit.fluxcd.io
|
||||
version: v2
|
||||
kind: HelmRelease
|
||||
metricNamePrefix: gotk
|
||||
errorLogV: 10
|
||||
metrics:
|
||||
- name: "resource_info"
|
||||
help: "The current state of a Flux HelmRelease resource."
|
||||
each:
|
||||
type: Info
|
||||
info:
|
||||
labelsFromPath:
|
||||
name: [metadata, name]
|
||||
labelsFromPath:
|
||||
exported_namespace: [metadata, namespace]
|
||||
ready: [status, conditions, "[type=Ready]", status]
|
||||
suspended: [spec, suspend]
|
||||
- groupVersionKind:
|
||||
group: source.toolkit.fluxcd.io
|
||||
version: v1
|
||||
kind: GitRepository
|
||||
metricNamePrefix: gotk
|
||||
errorLogV: 10
|
||||
metrics:
|
||||
- name: "resource_info"
|
||||
help: "The current state of a Flux GitRepository resource."
|
||||
each:
|
||||
type: Info
|
||||
info:
|
||||
labelsFromPath:
|
||||
name: [metadata, name]
|
||||
labelsFromPath:
|
||||
exported_namespace: [metadata, namespace]
|
||||
ready: [status, conditions, "[type=Ready]", status]
|
||||
suspended: [spec, suspend]
|
||||
- groupVersionKind:
|
||||
group: source.toolkit.fluxcd.io
|
||||
version: v1
|
||||
kind: HelmRepository
|
||||
metricNamePrefix: gotk
|
||||
errorLogV: 10
|
||||
metrics:
|
||||
- name: "resource_info"
|
||||
help: "The current state of a Flux HelmRepository resource."
|
||||
each:
|
||||
type: Info
|
||||
info:
|
||||
labelsFromPath:
|
||||
name: [metadata, name]
|
||||
labelsFromPath:
|
||||
exported_namespace: [metadata, namespace]
|
||||
ready: [status, conditions, "[type=Ready]", status]
|
||||
suspended: [spec, suspend]
|
||||
- groupVersionKind:
|
||||
group: source.toolkit.fluxcd.io
|
||||
version: v1
|
||||
kind: OCIRepository
|
||||
metricNamePrefix: gotk
|
||||
errorLogV: 10
|
||||
metrics:
|
||||
- name: "resource_info"
|
||||
help: "The current state of a Flux OCIRepository resource."
|
||||
each:
|
||||
type: Info
|
||||
info:
|
||||
labelsFromPath:
|
||||
name: [metadata, name]
|
||||
labelsFromPath:
|
||||
exported_namespace: [metadata, namespace]
|
||||
ready: [status, conditions, "[type=Ready]", status]
|
||||
suspended: [spec, suspend]
|
||||
- groupVersionKind:
|
||||
group: image.toolkit.fluxcd.io
|
||||
version: v1
|
||||
kind: ImageRepository
|
||||
metricNamePrefix: gotk
|
||||
errorLogV: 10
|
||||
metrics:
|
||||
- name: "resource_info"
|
||||
help: "The current state of a Flux ImageRepository resource."
|
||||
each:
|
||||
type: Info
|
||||
info:
|
||||
labelsFromPath:
|
||||
name: [metadata, name]
|
||||
labelsFromPath:
|
||||
exported_namespace: [metadata, namespace]
|
||||
ready: [status, conditions, "[type=Ready]", status]
|
||||
suspended: [spec, suspend]
|
||||
image: [spec, image]
|
||||
# The repository-side half of the Ready-but-frozen signal. A
|
||||
# policy can hold Ready=True forever while its filter excludes
|
||||
# every new release, so readiness cannot see it; what can is
|
||||
# "the repository kept finding tags and the policy's answer
|
||||
# never moved". .status.lastScanResult.tagCount is a required
|
||||
# field on this CRD, so the series exists for every object.
|
||||
- name: "image_repository_tag_count"
|
||||
help: "Number of tags found by the ImageRepository's last scan."
|
||||
each:
|
||||
type: Gauge
|
||||
gauge:
|
||||
path: [status, lastScanResult]
|
||||
valueFrom: [tagCount]
|
||||
# `name` belongs HERE and not under `gauge`. KSM resolves
|
||||
# `gauge.labelsFromPath` RELATIVE TO `gauge.path`, so
|
||||
# [metadata, name] under there would be looked up inside
|
||||
# status.lastScanResult and silently yield no label. The
|
||||
# metric-level labelsFromPath resolves from the object root.
|
||||
labelsFromPath:
|
||||
name: [metadata, name]
|
||||
exported_namespace: [metadata, namespace]
|
||||
- groupVersionKind:
|
||||
group: image.toolkit.fluxcd.io
|
||||
version: v1
|
||||
kind: ImagePolicy
|
||||
metricNamePrefix: gotk
|
||||
errorLogV: 10
|
||||
metrics:
|
||||
- name: "resource_info"
|
||||
help: "The current state of a Flux ImagePolicy resource."
|
||||
each:
|
||||
type: Info
|
||||
info:
|
||||
labelsFromPath:
|
||||
name: [metadata, name]
|
||||
labelsFromPath:
|
||||
exported_namespace: [metadata, namespace]
|
||||
ready: [status, conditions, "[type=Ready]", status]
|
||||
suspended: [spec, suspend]
|
||||
# The resolved tag, as a label on the Info series that
|
||||
# already exists — so the policy-side half of the frozen
|
||||
# signal costs zero extra series at rest. It retires one
|
||||
# series and starts another only when the tag actually
|
||||
# changes: 191 automated bumps in 147 days across 34
|
||||
# policies is ~1.3 new series/day, which ages out with
|
||||
# retention.
|
||||
tag: [status, latestRef, tag]
|
||||
# The join key. `metadata.name` happens to equal
|
||||
# `spec.imageRepositoryRef.name` for all 34 pairs in this
|
||||
# repo today, but the REFERENCE is this field, and a policy
|
||||
# named differently from its repository would silently drop
|
||||
# out of a name-based join.
|
||||
source_name: [spec, imageRepositoryRef, name]
|
||||
- groupVersionKind:
|
||||
group: image.toolkit.fluxcd.io
|
||||
version: v1
|
||||
kind: ImageUpdateAutomation
|
||||
metricNamePrefix: gotk
|
||||
errorLogV: 10
|
||||
metrics:
|
||||
- name: "resource_info"
|
||||
help: "The current state of a Flux ImageUpdateAutomation resource."
|
||||
each:
|
||||
type: Info
|
||||
info:
|
||||
labelsFromPath:
|
||||
name: [metadata, name]
|
||||
labelsFromPath:
|
||||
exported_namespace: [metadata, namespace]
|
||||
ready: [status, conditions, "[type=Ready]", status]
|
||||
suspended: [spec, suspend]
|
||||
source_name: [spec, sourceRef, name]
|
||||
|
||||
# Drop the apiserver's high-cardinality latency histograms.
|
||||
#
|
||||
# Measured 2026-08-20: Prometheus held 298,236 active series, of which
|
||||
|
||||
@@ -152,14 +152,38 @@ What exists today:
|
||||
hours later" and nothing answers "is anything not-Ready right now".
|
||||
- `apps/monitoring-rules/flux-podmonitor.yaml` scrapes the controllers,
|
||||
so reconcile latency and controller-level error counts are queryable.
|
||||
Readiness is **not** on that endpoint: at the controller versions
|
||||
pinned in `clusters/homelab/flux-system/gotk-components.yaml`,
|
||||
`gotk_reconcile_condition` and `gotk_suspend_status` are declared by
|
||||
the shared runtime library and recorded by nothing, so they export no
|
||||
series however the scrape is written.
|
||||
- `apps/monitoring/helmrelease-prometheus.yaml` therefore turns on
|
||||
kube-state-metrics `customResourceState` over eight Flux kinds, which
|
||||
is where per-object readiness now comes from: `gotk_resource_info`
|
||||
with `ready` and `suspended` labels. For ImagePolicy it also carries
|
||||
the resolved `tag`, and each ImageRepository gets
|
||||
`gotk_image_repository_tag_count`.
|
||||
- `apps/monitoring-rules/prometheusrule-flux.yaml` builds the standing
|
||||
alerts on those series. `FluxImageAutomationNotReady` fires after an
|
||||
ImageRepository, ImagePolicy or ImageUpdateAutomation has been
|
||||
not-Ready for 2h (two scan intervals for the 31 of 34 repositories at
|
||||
`interval: 1h`); `FluxResourceNotReady` covers every other Flux kind
|
||||
at 1h; `FluxImagePolicyMetricsMissing` is the dead man's switch that
|
||||
says so if the metrics themselves stop. This is the "still broken two
|
||||
hours later" signal the edge-triggered mail above cannot give.
|
||||
|
||||
What does **not** exist yet: a standing per-object "this ImagePolicy is
|
||||
not Ready" alert. The Flux controllers at the versions pinned in
|
||||
`clusters/homelab/flux-system/gotk-components.yaml` no longer populate
|
||||
`gotk_reconcile_condition` — the metric is declared by their shared
|
||||
runtime library but never recorded — so per-object readiness has to come
|
||||
from kube-state-metrics custom-resource-state instead. That work, and
|
||||
the not-Ready alert built on it, is tracked separately; the same change
|
||||
covers a policy that is perfectly Ready while pinned to a tag its own
|
||||
filter has frozen (`tika` if upstream ships a 3.x), which no readiness
|
||||
signal would catch either.
|
||||
What still does **not** exist: an *alert* for a policy that is perfectly
|
||||
Ready while its own filter has frozen it — `tika` if upstream ships a
|
||||
3.x. The series for it are now recorded
|
||||
(`flux:image_policy_tag_frozen:14d` and the two rules under it), but no
|
||||
threshold is attached yet, because none is derivable from the history
|
||||
available today. Measured 2026-09-26 against the 191 automated bump
|
||||
commits and the upstream registries: a flat duration cannot work (gatus
|
||||
went 127 days between bumps while healthy), a 10% tag-count growth
|
||||
comparator fires on three variant-churning repositories and misses
|
||||
`tika` at 3.0%, relaxing it to one new tag adds `velero-plugin-for-aws`,
|
||||
whose pin is deliberate, and `.status.lastScanResult.latestTags` is
|
||||
sorted alphabetically rather than newest-first so it cannot substitute.
|
||||
The recording rules exist to accumulate the per-repository `tagCount`
|
||||
history that a defensible threshold has to be derived from; the alert is
|
||||
tracked separately.
|
||||
|
||||
Reference in New Issue
Block a user