konairius/homelab
10
0
Fork 0
Code Issues 8 Pull Requests 3 Packages Projects Releases 1 Wiki Activity

Merge PR #978: record the egress dependency and the real revert recipe for the app-health probes (REN-21)
ci/woodpecker/cron/alert-monitor Pipeline was successful
Details

Browse Source 
Comment-only change to kubernetes/apps/monitoring-rules/probes-blackbox.yaml. Reviewed by argus-bot at head 49bd9271 against base 2b71dda1: 3 documents parse deep-equal, blackbox-http 20 targets before and after, git diff -U0 non-comment non-blank lines = 0, apps/gatus/configmap.yaml untouched. Revert recipe and blast-radius claims re-executed and confirmed.
This commit was merged in pull request #978.
This commit is contained in:
argus-bot (Paperclip service account)
2026-09-25 21:13:11 +00:00
parent 2b71dda105 49bd927147
commit 3a457bd63c
1 changed files with 74 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files
kubernetes/apps/monitoring-rules/probes-blackbox.yaml
+74 -21
View File
@@ -116,16 +116,41 @@ spec:
# destination ingress allow above is worthless if the prober itself
# is in egress enforcement without a matching rule. It is not: an
# endpoint only enters egress enforcement once some policy selecting
# it declares an `egress:` block, and none does for monitoring.
# Verified at 0a82d61f across the whole repo — 73 Cilium policies, 38
# with an egress block, none of them in the monitoring namespace, and
# the single CiliumClusterwideNetworkPolicy is scoped to
# `reserved.health` so it cannot select a pod. 17 other namespaces
# here already have an egress block. If a routine hardening pass ever
# adds one to monitoring, it MUST list the navidrome and paperless
# namespaces or these three cross-namespace targets go red while the
# *.monitoring.svc ones above keep working — a failure mode nobody
# will connect back to the hardening commit.
# it declares an `egress:` block, and none does for monitoring. The
# namespace has exactly two Cilium policies and both are ingress-only:
# infrastructure/network-policies/monitoring-ns.yaml
# (default-deny-ingress) and monitoring-alloy-syslog.yaml
# (allow-syslog-world-ingress). blackbox-exporter runs in that same
# namespace (apps/monitoring/helmrelease-blackbox-exporter.yaml:5).
# Re-verified at 2b71dda1 across all 635 yaml files in the repo — 73
# Cilium objects, 38 carrying an egress or egressDeny block, none of
# them in the monitoring namespace, and the single
# CiliumClusterwideNetworkPolicy is scoped to `reserved.health` so it
# cannot select a pod. 17 other namespaces here already have an egress
# block. If a routine hardening pass ever adds one to monitoring, it
# MUST list the navidrome and paperless namespaces or these three
# cross-namespace targets go red while the *.monitoring.svc ones above
# keep working — a failure mode nobody will connect back to the
# hardening commit. The cnp-cross-namespace-* notes document only the
# ingress half; that silence is what makes this the trap.
#
# BLAST RADIUS if that egress dependency is ever broken. Nothing
# user-facing breaks: these are Prometheus-side Probe targets, not
# gatus checks, so status.renner.uno is unaffected and no workload is
# restarted. What breaks is the signal, and it breaks misleadingly.
# All three targets start failing BlackboxProbeFailing
# (prometheusrule-blackbox.yaml:34 — severity warning, for: 10m); none
# of the three is in that rule's exclusion list. BlackboxProbesAbsent
# does NOT fire, because the other 17 targets keep producing
# probe_success — so it reads as three apps going down, not as a
# policy change. WHO NOTICES: warning routes to the default `email`
# receiver, admin@renner.uno (helmrelease-prometheus.yaml alertmanager
# route, group_wait 30s, grouped by alertname), so the owner gets one
# email roughly 10-11 minutes after the policy lands. HOW FAST UNDONE:
# the REVERT recipe below is a one-line deletion; from merge it is
# live within ~11 minutes (GitRepository interval 1m0s,
# monitoring-rules Kustomization interval 10m), or immediately with
# `flux reconcile kustomization monitoring-rules --with-source`.
#
# /ping on both: unauthenticated, and already the pods' own kubelet
# probe path (navidrome/deployment.yaml:61-74,
@@ -142,16 +167,41 @@ spec:
# probe_http_status_code on the first scrapes; a target that settles
# on a 3xx gets REVERTED, not widened to accept 3xx.
#
# REVERT (recipe tested by review at 0a82d61f): delete everything
# from the "APP-HEALTH probes for the two gated navidrome-namespace
# apps" line above through the lidarr target below. Do NOT use
# `git revert -m 1 16238d39` — it conflicts, and resolving the
# conflict the natural way also drops the paperless target below,
# silently leaving no app-health signal at all. The paperless block
# below reverts on its own with `git revert -m 1 0a82d61f`, but only
# against a tree where this comment block is absent — this commit
# edits that block, so expect one conflict covering the whole
# paperless block, and resolve it by taking the deletion.
# REVERT — an executed recipe, not a claim. Every command below was
# actually run against master 2b71dda1 on 2026-09-25; the results are
# what is recorded here.
#
# NO `git revert` works. All four candidates conflict in THIS file at
# 2b71dda1, because the comment blocks around these targets were
# rewritten after the probes landed (#979):
#
# git revert -m 1 0a82d61f merge of #977, paperless CONFLICT
# git revert -m 1 16238d39 merge of #976, nav + lidarr CONFLICT
# git revert cb9e73bd the paperless commit CONFLICT
# git revert 2219c928 the nav + lidarr commit CONFLICT
#
# Hand-resolving any of them is the trap: the natural --theirs
# resolution of the 16238d39 conflict also drops the paperless target
# below, leaving NO app-health probes and no signal that it happened.
#
# Delete the target lines forward instead. Only the target lines are
# part of the object; the comments are inert. Both of these were run
# and verified:
#
# # all three app-health targets — blackbox-http 20 -> 17
# sed -i '/svc\.cluster\.local:80\/ping$/d;/paperless\.paperless\.svc\.cluster\.local:80\/accounts\/login\/$/d' \
# kubernetes/apps/monitoring-rules/probes-blackbox.yaml
#
# # the paperless target alone — blackbox-http 20 -> 19
# sed -i '/paperless\.paperless\.svc\.cluster\.local:80\/accounts\/login\/$/d' \
# kubernetes/apps/monitoring-rules/probes-blackbox.yaml
#
# Verified after each run: the file still parses, blackbox-http drops
# to 17 and to 19 respectively, blackbox-edge-wan (3) and
# blackbox-tcp-relays (2) are untouched, and `git diff` contains
# nothing but the deleted target lines. Then drop whichever comment
# block is left orphaned, open a PR, and merge — timing as in the
# BLAST RADIUS paragraph above.
- http://navidrome.navidrome.svc.cluster.local:80/ping
- http://lidarr.navidrome.svc.cluster.local:80/ping
# APP-HEALTH probe for paperless. Same rule as the block above
@@ -193,7 +243,10 @@ spec:
#
# This is the target most likely to fail the STATUS CODE check in the
# block above — a Django login view can answer 302 — so check it
# first, and revert rather than widen if it does.
# first, and revert rather than widen if it does. Use the REVERT
# recipe in that block: `git revert -m 1 0a82d61f` conflicts here too
# at 2b71dda1, and the tested one-line deletion for this target alone
# is the second command listed there (20 -> 19 targets).
- http://paperless.paperless.svc.cluster.local:80/accounts/login/
# WAN-path probes of the public edge BY PROXY IP. The name-based
# home.renner.uno probe above resolves in-cluster (CoreDNS/k8s_gateway)