homelab/kubernetes/apps/monitoring-rules/tests

promtool unit suites for the PrometheusRules in this directory

promtool test rules suites, run on every pull request that touches kubernetes/** by the promtool step in .woodpecker/ci-manifests-test.yml via scripts/ci/promtool-rules.sh.

Why this exists

Before this, CI proved that every PrometheusRule in the tree assembled into valid YAML and nothing else. It did not check that an expression parses, that a label join matches anything, or that a for window behaves the way its comment claims.

That was not theoretical. On REN-129 the surge rule in prometheusrule-cnpg-backup.yaml was written for: 2m and reviewed at that value. Against Iris's seven measured nights it fires on six of them — two owner emails a night, 04:16 and 04:22 local, for a WAL backlog that clears itself in under ten minutes. It was caught by a human writing the arithmetic out by hand, twice, in two separate runs, and throwing the work away both times because there was nowhere in the repo to put it. REN-152 then retired the depth rule in favour of one persistence rule (> 5 for 20m).

Layout

File	What it is
cnpg-backup_test.yaml	The suite. Generated — see below.
generate-cnpg-backup-test.py	Regenerates it from the REN-129 measurement table.
external-secrets_test.yaml	Hand-written suite for prometheusrule-external-secrets.yaml (REN-103).
home-assistant_test.yaml	Hand-written suite for prometheusrule-home-assistant.yaml (REN-245).
ceph_test.yaml	Hand-written suite for the CephPoolLostRedundancyBlind canary in prometheusrule-ceph.yaml (REN-93).
disk-wear_test.yaml	Hand-written suite for the EPHEMERAL exclusion in prometheusrule-disk-wear.yaml (REN-419).
gitea_test.yaml	Hand-written suite for GiteaDataVolumeLow in prometheusrule-gitea.yaml (REN-583).
mutations.txt	Negative controls. Each must make the suite fail.

Adding a suite

Name it <something>_test.yaml in this directory. Discovery is by that suffix, and promtool-rules.sh fails if it finds none, so a suite in the wrong place is a red pipeline rather than a silent gap.

rule_files entries are relative to this directory, not the repo root. promtool joins dirname(suite) onto each entry, so a manifest in the parent directory is ../name.yaml. Getting this wrong is not a test failure: promtool prints WARNING: no file match pattern …, loads zero rules, and every exp_alerts: [] assertion then passes against an empty rule set. The first draft of the suite here shipped exactly that way — ninety silence assertions, green, testing nothing. promtool-rules.sh resolves every entry itself before running promtool and fails on a miss, because promtool has no flag that makes it fatal.

promtool cannot read a PrometheusRule directly: it wants a bare groups: document, and a PrometheusRule carries that under spec:. The script strips spec: into a temporary mirrored tree with the same relative layout and runs there, so ../name.yaml resolves in the mirror exactly as it reads here. Nothing is written back into the checkout.

Regenerating cnpg-backup_test.yaml

python3 kubernetes/apps/monitoring-rules/tests/generate-cnpg-backup-test.py \
  > kubernetes/apps/monitoring-rules/tests/cnpg-backup_test.yaml

and commit the result. --report prints the reconstruction self-check table instead of the suite.

The generator exists so the numbers in the suite are auditable: the seven measured nights are rebuilt from the (peak, peak minute, fill, drain) tuples Iris published on REN-129, not hand-typed, so a reader can check them against the measurement rather than taking them on trust. It is not a hermetic artifact, and CI does not regenerate and diff it — the promtool image has no Python, and byte-equality with a generator is not what stops a suite from being weakened. The mutation phase is.

Two things the generator refuses to emit, both because they happened:

• a window that does not cover the bursts. An earlier revision had T0 at 8 * 60 + 8 (08:08Z) instead of 2 * 60 + 8, seven hours after every burst, so all seven nights rebuilt as forty zeroes and their silence assertions passed without touching a threshold. assert_window_covers_bursts() now fails instead.
• a night that clears the threshold sooner than the measured one, which would be a weaker silence test than the night it claims to stand for. Longer is fine and expected — a linear decay at the published mean drain rate holds the tail above 5 a minute or two longer than the real trace on four of the seven nights, which makes the suite harder to pass, not easier.

Negative controls

A suite that passes proves the rules match the assertions. It does not prove the assertions reach the rules — both failure modes above were green. So promtool-rules.sh runs a second phase: for each entry in mutations.txt it breaks one value the rules depend on and requires the suites to fail. A mutation nothing notices is reported as a hole in the suite.

This is also the answer to "why is that case in there". backlog-for-12m is silent on all seven reconstructed nights and fires only on the raw 2026-09-11 worst-night case, so deleting that case makes for: 20m reducible to 12m with the suite still green. The control is what makes the case load-bearing.

Mutations are name|manifest-path|sed-program, applied to the stripped copy in the mirror (two spaces shallower than the manifest on disk — anchor on ^[[:space:]]*). A program that matches nothing is a failure, not a skip: otherwise renaming a rule quietly turns its control into a no-op.

Reading for margins

Margin is pending time, which is one minute less than the run length. An alert goes pending at the first evaluation where the expression is true and fires when now - activeAt >= for, so N consecutive one-minute samples above a threshold accumulate N-1 minutes. A trace with 5 samples above 5 does not fire for: 5m; it fires for: 4m. The >5 pend column in --report is the number to compare a for against — not the sample count next to it.

Running it locally

# promtool must be 3.x: the cluster runs kube-prometheus-stack >= 72.0.0, and
# alert `for` evaluation is version-sensitive, so a 2.x promtool would be
# checking a different evaluator than production. The script enforces this.
sh scripts/ci/promtool-rules.sh

The script's own guards — the discovery floors, the rule_files resolution, the mutation phase — are tested offline and without promtool by scripts/ci/test-promtool-rules.sh, which runs on every pull request via .woodpecker/ci-scripts-test.yml.