feat(monitoring): alert on any blackbox target without a rule of its own #698

2026-08-28T09:11:14Z

claude-bot commented

2026-08-28 09:11:14 +00:00

Neun von dreizehn blackbox-http-Zielen wurden geprobed, geplottet und komplett unalarmiert: gitea, immich, auth, home, prometheus, alerts, dash — plus die zwei internen /-/healthy-Probes, die ausdrücklich existieren, um eine tote App hinter einem lebenden Authentik-Gate zu erwischen.

Das ist genau die Form, die file.renner.uno monatelang 502 liefern ließ.

Die Regel

BlackboxProbeFailing deckt alles ab außer den Instanzen, die schon eine eigene Regel haben.

Gegen die laufende Instanz gemessen, vor dem Commit — der Selektor trifft exakt diese neun:

http://kube-prometheus-stack-alertmanager...:9093/-/healthy
http://kube-prometheus-stack-prometheus...:9090/-/healthy
https://alerts.renner.uno
https://auth.renner.uno
https://dash.renner.uno
https://gitea.renner.uno
https://home.renner.uno
https://immich.renner.uno
https://prometheus.renner.uno

und lässt die sechs mit eigener Behandlung aus. Der vollständige Ausdruck, wie YAML ihn ausliefert, ergibt heute 0 Treffer.

Die Ausschlussliste ist kein Beiwerk

Ausschlussliste und dedizierte Regeln sind eine Invariante, und das steht so in der Datei: eine Instanz gehört nur so lange in die Liste, wie eine Regel anderswo sie abdeckt — und eine neue dedizierte Regel muss ihre Instanz eintragen. Falsch in die eine Richtung feuert der Alarm doppelt, falsch in die andere nie.

Ausgeschlossen	Abgedeckt durch
`fsn1.your-objectstorage.com`	`HetznerBackupEndpointDown` (critical)
`netbird.renner.uno`	`prometheusrule-netbird-availability`
`proxy-1/2:9000/health`	dieselbe Datei, Relay-Redundanz
`ci.renner.uno`	`WoodpeckerDown`
`openbao .../sys/health`	`OpenBaoUnavailable`

Severity ist warning, und zwar absichtlich. Eine Catch-all-Regel kann nicht wissen, was ein einzelnes Ziel bedeutet. Was einen kritischen Alarm verdient, bekommt eine eigene Regel — und landet damit in der Liste oben. Genau das hält die Liste davon ab, willkürlich zu sein.

Und die Prüfung der Prüfung

BlackboxProbesAbsent feuert, wenn der Job überhaupt keine Serien mehr liefert. Jeder Endpunkt-Alarm liest probe_success; hören die Serien auf, kann keiner davon feuern — und dieses Schweigen ist von Gesundheit nicht zu unterscheiden.

Nebenbei korrigiert

prometheusrule-hetzner-backup.yaml behauptete, CNPG-Pod-Metriken würden nicht gescrapet (enablePodMonitor=false) und ein granularer Archivierungs-Alarm sei ein Follow-up. Beides existiert längst: cnpg-podmonitors.yaml scrapet die Instanzen, und CNPGWalArchiveBacklog liest genau diesen Collector.

Geprüft

kubectl apply --dry-run=server -k sauber, prometheusrule/blackbox als created. Nach dem Merge prüfe ich, dass beide Regeln mit health=ok laden und dass ihr Selektor real etwas trifft — ein health=ok beweist nur, dass die Regel parst, nicht dass sie auf existierende Serien zeigt.

Neun von dreizehn `blackbox-http`-Zielen wurden geprobed, geplottet und **komplett unalarmiert**: `gitea`, `immich`, `auth`, `home`, `prometheus`, `alerts`, `dash` — plus die zwei internen `/-/healthy`-Probes, die ausdrücklich existieren, um eine tote App hinter einem lebenden Authentik-Gate zu erwischen. Das ist genau die Form, die `file.renner.uno` monatelang 502 liefern ließ. ## Die Regel `BlackboxProbeFailing` deckt alles ab **außer** den Instanzen, die schon eine eigene Regel haben. **Gegen die laufende Instanz gemessen, vor dem Commit** — der Selektor trifft exakt diese neun: ``` http://kube-prometheus-stack-alertmanager...:9093/-/healthy http://kube-prometheus-stack-prometheus...:9090/-/healthy https://alerts.renner.uno https://auth.renner.uno https://dash.renner.uno https://gitea.renner.uno https://home.renner.uno https://immich.renner.uno https://prometheus.renner.uno ``` und lässt die sechs mit eigener Behandlung aus. Der vollständige Ausdruck, wie YAML ihn ausliefert, ergibt heute 0 Treffer. ## Die Ausschlussliste ist kein Beiwerk Ausschlussliste und dedizierte Regeln sind **eine Invariante**, und das steht so in der Datei: eine Instanz gehört nur so lange in die Liste, wie eine Regel anderswo sie abdeckt — und eine neue dedizierte Regel muss ihre Instanz eintragen. Falsch in die eine Richtung feuert der Alarm doppelt, falsch in die andere nie. | Ausgeschlossen | Abgedeckt durch | |---|---| | `fsn1.your-objectstorage.com` | `HetznerBackupEndpointDown` (critical) | | `netbird.renner.uno` | `prometheusrule-netbird-availability` | | `proxy-1/2:9000/health` | dieselbe Datei, Relay-Redundanz | | `ci.renner.uno` | `WoodpeckerDown` | | `openbao .../sys/health` | `OpenBaoUnavailable` | **Severity ist warning, und zwar absichtlich.** Eine Catch-all-Regel kann nicht wissen, was ein einzelnes Ziel bedeutet. Was einen kritischen Alarm verdient, bekommt eine eigene Regel — und landet damit in der Liste oben. Genau das hält die Liste davon ab, willkürlich zu sein. ## Und die Prüfung der Prüfung `BlackboxProbesAbsent` feuert, wenn der Job überhaupt keine Serien mehr liefert. Jeder Endpunkt-Alarm liest `probe_success`; hören die Serien auf, kann keiner davon feuern — und dieses Schweigen ist von Gesundheit nicht zu unterscheiden. ## Nebenbei korrigiert `prometheusrule-hetzner-backup.yaml` behauptete, CNPG-Pod-Metriken würden nicht gescrapet (`enablePodMonitor=false`) und ein granularer Archivierungs-Alarm sei ein Follow-up. Beides existiert längst: `cnpg-podmonitors.yaml` scrapet die Instanzen, und `CNPGWalArchiveBacklog` liest genau diesen Collector. ## Geprüft `kubectl apply --dry-run=server -k` sauber, `prometheusrule/blackbox` als `created`. Nach dem Merge prüfe ich, dass beide Regeln mit `health=ok` laden **und** dass ihr Selektor real etwas trifft — ein `health=ok` beweist nur, dass die Regel parst, nicht dass sie auf existierende Serien zeigt.

claude-bot added 1 commit 2026-08-28 09:11:14 +00:00

feat(monitoring): alert on any blackbox target without a rule of its own

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

e15067a784

Nine of the thirteen blackbox-http targets were probed, graphed and completely
unalerted: gitea, immich, auth, home, prometheus, alerts and dash, plus the two
internal /-/healthy probes that exist specifically to catch a dead app behind a
live Authentik gate. That is the shape that let file.renner.uno serve 502 for
months.

BlackboxProbeFailing covers everything except the instances that already have a
dedicated rule. Measured against the running Prometheus before committing: the
selector returns exactly those nine and excludes the six with bespoke handling,
and the full expression as YAML delivers it evaluates to zero results today.

The exclusion list and the dedicated rules are one invariant, stated in the
file: an instance belongs in the list only while a rule elsewhere covers it,
and a new dedicated rule has to add its instance. Wrong in one direction the
alert fires twice, wrong in the other it never fires. Severity is warning
because a catch-all cannot know what any given target means — anything
deserving critical earns its own rule, which is also what puts it in the list.
That is what stops the list from being arbitrary.

BlackboxProbesAbsent is the check on the checks. Every endpoint alert reads
probe_success; if the job stops producing series none of them can fire, and
that silence is indistinguishable from health.

Also corrects the comment in prometheusrule-hetzner-backup.yaml claiming CNPG
pod metrics are not scraped and a granular archiving alert is a follow-up. Both
have existed for a while: cnpg-podmonitors.yaml scrapes the instances and
CNPGWalArchiveBacklog reads that collector.

review-bot reviewed

2026-08-28 09:14:16 +00:00

review-bot left a comment

🤖 Automated review — model claude-sonnet-5 (complexity: normal), commit e15067a784a2eb38b2badd0beadf01acb985dd6f.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

Adds a catch-all BlackboxProbeFailing alert for any blackbox-http target lacking a dedicated rule, plus a BlackboxProbesAbsent meta-alert for when the probe job itself stops producing series, and fixes a stale comment in prometheusrule-hetzner-backup.yaml. I checked the new regex and the "corrected" claims against the actual state of the repo (probes-blackbox.yaml, prometheusrule-ci-openbao.yaml, prometheusrule-netbird-availability.yaml, prometheusrule-cnpg-backup.yaml, cnpg-podmonitors.yaml) and everything holds up. No blocking issues.

🟡 Suggestions

kubernetes/apps/monitoring-rules/prometheusrule-blackbox.yaml:35 — the exclusion regex is a full-string alternation (instance!~"A|B|C|D|E"), which works today but is a maintenance trap: the PR body itself calls the exclusion list "one invariant" with the dedicated rules, yet nothing enforces it — a future dedicated rule that forgets to extend this line will only be caught by someone noticing double-firing in Alertmanager. Not asking for a fix, just noting the failure mode is silent-by-default, matching the exact shape of bug this PR exists to close.

🟢 Nits

PR description says "Neun von dreizehn" blackbox-http targets were unalerted. Counting the actual blackbox-http job in probes-blackbox.yaml:24-80, there are 15 targets, not 13 (6 excluded + 9 previously-unalerted = 15). The "nine" figure and the exclusion table are both correct; only the "thirteen" total in the prose is off. Doesn't affect the rule itself, just the PR narrative.

Tests

No test suite applies here — this is a Kubernetes manifest / PrometheusRule change, and the repo's own verification method (kubectl apply --dry-run=server -k, checking prometheusrule/blackbox loads with health=ok) is the appropriate check, which the PR description says was done pre-commit. I independently verified the parts that matter statically:

The exclusion regex's five alternatives each exactly match the corresponding instance label values as configured in probes-blackbox.yaml (including the openbao prefix match against the ?standbyok=true query target).
The double-backslash escaping (\\.) follows the same convention already used in prometheusrule-ci-openbao.yaml and prometheusrule-netbird-availability.yaml, so it round-trips correctly through YAML → PromQL string parsing.
No other rule in monitoring-rules/ already covers the nine now-caught instances (gitea, immich, auth, home, prometheus, alerts, dash, and the two internal /-/healthy targets), so BlackboxProbeFailing doesn't create duplicate alerts for them.
The prometheusrule-hetzner-backup.yaml comment fix is accurate: CNPGWalArchiveBacklog exists in prometheusrule-cnpg-backup.yaml and reads cnpg_collector_pg_wal_archive_status, and cnpg-podmonitors.yaml does scrape the CNPG instances — the old "not scraped yet" comment was indeed stale.
The new file is correctly wired into kustomization.yaml (alphabetically placed before prometheusrule-ceph.yaml), and this directory's kustomization.yaml lists files explicitly rather than globbing, so this is the only place that needed updating.

One thing I could not verify from a static read: whether the live cluster's instance label values exactly match what's in probes-blackbox.yaml (Prometheus Operator's Probe → scrape config relabeling is standard, but I only confirmed it against the YAML, not a running target). The PR body says this was checked live pre-commit, which is the right verification method.

Recommendation

Looks reasonable — no blocking issues; comments above are optional.

🤖 **Automated review** — model `claude-sonnet-5` (complexity: normal), commit `e15067a784a2eb38b2badd0beadf01acb985dd6f`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary Adds a catch-all `BlackboxProbeFailing` alert for any `blackbox-http` target lacking a dedicated rule, plus a `BlackboxProbesAbsent` meta-alert for when the probe job itself stops producing series, and fixes a stale comment in `prometheusrule-hetzner-backup.yaml`. I checked the new regex and the "corrected" claims against the actual state of the repo (`probes-blackbox.yaml`, `prometheusrule-ci-openbao.yaml`, `prometheusrule-netbird-availability.yaml`, `prometheusrule-cnpg-backup.yaml`, `cnpg-podmonitors.yaml`) and everything holds up. No blocking issues. ## 🟡 Suggestions - `kubernetes/apps/monitoring-rules/prometheusrule-blackbox.yaml:35` — the exclusion regex is a full-string alternation (`instance!~"A|B|C|D|E"`), which works today but is a maintenance trap: the PR body itself calls the exclusion list "one invariant" with the dedicated rules, yet nothing enforces it — a future dedicated rule that forgets to extend this line will only be caught by someone noticing double-firing in Alertmanager. Not asking for a fix, just noting the failure mode is silent-by-default, matching the exact shape of bug this PR exists to close. ## 🟢 Nits - PR description says "Neun von dreizehn" blackbox-http targets were unalerted. Counting the actual `blackbox-http` job in `probes-blackbox.yaml:24-80`, there are 15 targets, not 13 (6 excluded + 9 previously-unalerted = 15). The "nine" figure and the exclusion table are both correct; only the "thirteen" total in the prose is off. Doesn't affect the rule itself, just the PR narrative. ## Tests No test suite applies here — this is a Kubernetes manifest / PrometheusRule change, and the repo's own verification method (`kubectl apply --dry-run=server -k`, checking `prometheusrule/blackbox` loads with `health=ok`) is the appropriate check, which the PR description says was done pre-commit. I independently verified the parts that matter statically: - The exclusion regex's five alternatives each exactly match the corresponding `instance` label values as configured in `probes-blackbox.yaml` (including the `openbao` prefix match against the `?standbyok=true` query target). - The double-backslash escaping (`\\.`) follows the same convention already used in `prometheusrule-ci-openbao.yaml` and `prometheusrule-netbird-availability.yaml`, so it round-trips correctly through YAML → PromQL string parsing. - No other rule in `monitoring-rules/` already covers the nine now-caught instances (`gitea`, `immich`, `auth`, `home`, `prometheus`, `alerts`, `dash`, and the two internal `/-/healthy` targets), so `BlackboxProbeFailing` doesn't create duplicate alerts for them. - The `prometheusrule-hetzner-backup.yaml` comment fix is accurate: `CNPGWalArchiveBacklog` exists in `prometheusrule-cnpg-backup.yaml` and reads `cnpg_collector_pg_wal_archive_status`, and `cnpg-podmonitors.yaml` does scrape the CNPG instances — the old "not scraped yet" comment was indeed stale. - The new file is correctly wired into `kustomization.yaml` (alphabetically placed before `prometheusrule-ceph.yaml`), and this directory's `kustomization.yaml` lists files explicitly rather than globbing, so this is the only place that needed updating. One thing I could not verify from a static read: whether the live cluster's `instance` label values exactly match what's in `probes-blackbox.yaml` (Prometheus Operator's Probe → scrape config relabeling is standard, but I only confirmed it against the YAML, not a running target). The PR body says this was checked live pre-commit, which is the right verification method. ## Recommendation Looks reasonable — no blocking issues; comments above are optional.

k.renner merged commit 5c412f0486 into master

2026-08-28 09:16:19 +00:00

k.renner referenced this issue from a commit

2026-08-28 09:16:21 +00:00

Merge pull request 'feat(monitoring): alert on any blackbox target without a rule of its own' (#698) from generic-blackbox-alert into master

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#698