claude-bot
e15067a784
Nine of the thirteen blackbox-http targets were probed, graphed and completely unalerted: gitea, immich, auth, home, prometheus, alerts and dash, plus the two internal /-/healthy probes that exist specifically to catch a dead app behind a live Authentik gate. That is the shape that let file.renner.uno serve 502 for months. BlackboxProbeFailing covers everything except the instances that already have a dedicated rule. Measured against the running Prometheus before committing: the selector returns exactly those nine and excludes the six with bespoke handling, and the full expression as YAML delivers it evaluates to zero results today. The exclusion list and the dedicated rules are one invariant, stated in the file: an instance belongs in the list only while a rule elsewhere covers it, and a new dedicated rule has to add its instance. Wrong in one direction the alert fires twice, wrong in the other it never fires. Severity is warning because a catch-all cannot know what any given target means — anything deserving critical earns its own rule, which is also what puts it in the list. That is what stops the list from being arbitrary. BlackboxProbesAbsent is the check on the checks. Every endpoint alert reads probe_success; if the job stops producing series none of them can fire, and that silence is indistinguishable from health. Also corrects the comment in prometheusrule-hetzner-backup.yaml claiming CNPG pod metrics are not scraped and a granular archiving alert is a follow-up. Both have existed for a while: cnpg-podmonitors.yaml scrapes the instances and CNPGWalArchiveBacklog reads that collector.