diff --git a/kubernetes/apps/monitoring-rules/probes-blackbox.yaml b/kubernetes/apps/monitoring-rules/probes-blackbox.yaml index 85b63bc8..b3b1b22c 100644 --- a/kubernetes/apps/monitoring-rules/probes-blackbox.yaml +++ b/kubernetes/apps/monitoring-rules/probes-blackbox.yaml @@ -116,16 +116,41 @@ spec: # destination ingress allow above is worthless if the prober itself # is in egress enforcement without a matching rule. It is not: an # endpoint only enters egress enforcement once some policy selecting - # it declares an `egress:` block, and none does for monitoring. - # Verified at 0a82d61f across the whole repo — 73 Cilium policies, 38 - # with an egress block, none of them in the monitoring namespace, and - # the single CiliumClusterwideNetworkPolicy is scoped to - # `reserved.health` so it cannot select a pod. 17 other namespaces - # here already have an egress block. If a routine hardening pass ever - # adds one to monitoring, it MUST list the navidrome and paperless - # namespaces or these three cross-namespace targets go red while the - # *.monitoring.svc ones above keep working — a failure mode nobody - # will connect back to the hardening commit. + # it declares an `egress:` block, and none does for monitoring. The + # namespace has exactly two Cilium policies and both are ingress-only: + # infrastructure/network-policies/monitoring-ns.yaml + # (default-deny-ingress) and monitoring-alloy-syslog.yaml + # (allow-syslog-world-ingress). blackbox-exporter runs in that same + # namespace (apps/monitoring/helmrelease-blackbox-exporter.yaml:5). + # Re-verified at 2b71dda1 across all 635 yaml files in the repo — 73 + # Cilium objects, 38 carrying an egress or egressDeny block, none of + # them in the monitoring namespace, and the single + # CiliumClusterwideNetworkPolicy is scoped to `reserved.health` so it + # cannot select a pod. 17 other namespaces here already have an egress + # block. If a routine hardening pass ever adds one to monitoring, it + # MUST list the navidrome and paperless namespaces or these three + # cross-namespace targets go red while the *.monitoring.svc ones above + # keep working — a failure mode nobody will connect back to the + # hardening commit. The cnp-cross-namespace-* notes document only the + # ingress half; that silence is what makes this the trap. + # + # BLAST RADIUS if that egress dependency is ever broken. Nothing + # user-facing breaks: these are Prometheus-side Probe targets, not + # gatus checks, so status.renner.uno is unaffected and no workload is + # restarted. What breaks is the signal, and it breaks misleadingly. + # All three targets start failing BlackboxProbeFailing + # (prometheusrule-blackbox.yaml:34 — severity warning, for: 10m); none + # of the three is in that rule's exclusion list. BlackboxProbesAbsent + # does NOT fire, because the other 17 targets keep producing + # probe_success — so it reads as three apps going down, not as a + # policy change. WHO NOTICES: warning routes to the default `email` + # receiver, admin@renner.uno (helmrelease-prometheus.yaml alertmanager + # route, group_wait 30s, grouped by alertname), so the owner gets one + # email roughly 10-11 minutes after the policy lands. HOW FAST UNDONE: + # the REVERT recipe below is a one-line deletion; from merge it is + # live within ~11 minutes (GitRepository interval 1m0s, + # monitoring-rules Kustomization interval 10m), or immediately with + # `flux reconcile kustomization monitoring-rules --with-source`. # # /ping on both: unauthenticated, and already the pods' own kubelet # probe path (navidrome/deployment.yaml:61-74, @@ -142,16 +167,41 @@ spec: # probe_http_status_code on the first scrapes; a target that settles # on a 3xx gets REVERTED, not widened to accept 3xx. # - # REVERT (recipe tested by review at 0a82d61f): delete everything - # from the "APP-HEALTH probes for the two gated navidrome-namespace - # apps" line above through the lidarr target below. Do NOT use - # `git revert -m 1 16238d39` — it conflicts, and resolving the - # conflict the natural way also drops the paperless target below, - # silently leaving no app-health signal at all. The paperless block - # below reverts on its own with `git revert -m 1 0a82d61f`, but only - # against a tree where this comment block is absent — this commit - # edits that block, so expect one conflict covering the whole - # paperless block, and resolve it by taking the deletion. + # REVERT — an executed recipe, not a claim. Every command below was + # actually run against master 2b71dda1 on 2026-09-25; the results are + # what is recorded here. + # + # NO `git revert` works. All four candidates conflict in THIS file at + # 2b71dda1, because the comment blocks around these targets were + # rewritten after the probes landed (#979): + # + # git revert -m 1 0a82d61f merge of #977, paperless CONFLICT + # git revert -m 1 16238d39 merge of #976, nav + lidarr CONFLICT + # git revert cb9e73bd the paperless commit CONFLICT + # git revert 2219c928 the nav + lidarr commit CONFLICT + # + # Hand-resolving any of them is the trap: the natural --theirs + # resolution of the 16238d39 conflict also drops the paperless target + # below, leaving NO app-health probes and no signal that it happened. + # + # Delete the target lines forward instead. Only the target lines are + # part of the object; the comments are inert. Both of these were run + # and verified: + # + # # all three app-health targets — blackbox-http 20 -> 17 + # sed -i '/svc\.cluster\.local:80\/ping$/d;/paperless\.paperless\.svc\.cluster\.local:80\/accounts\/login\/$/d' \ + # kubernetes/apps/monitoring-rules/probes-blackbox.yaml + # + # # the paperless target alone — blackbox-http 20 -> 19 + # sed -i '/paperless\.paperless\.svc\.cluster\.local:80\/accounts\/login\/$/d' \ + # kubernetes/apps/monitoring-rules/probes-blackbox.yaml + # + # Verified after each run: the file still parses, blackbox-http drops + # to 17 and to 19 respectively, blackbox-edge-wan (3) and + # blackbox-tcp-relays (2) are untouched, and `git diff` contains + # nothing but the deleted target lines. Then drop whichever comment + # block is left orphaned, open a PR, and merge — timing as in the + # BLAST RADIUS paragraph above. - http://navidrome.navidrome.svc.cluster.local:80/ping - http://lidarr.navidrome.svc.cluster.local:80/ping # APP-HEALTH probe for paperless. Same rule as the block above @@ -193,7 +243,10 @@ spec: # # This is the target most likely to fail the STATUS CODE check in the # block above — a Django login view can answer 302 — so check it - # first, and revert rather than widen if it does. + # first, and revert rather than widen if it does. Use the REVERT + # recipe in that block: `git revert -m 1 0a82d61f` conflicts here too + # at 2b71dda1, and the tested one-line deletion for this target alone + # is the second command listed there (20 -> 19 targets). - http://paperless.paperless.svc.cluster.local:80/accounts/login/ # WAN-path probes of the public edge BY PROXY IP. The name-based # home.renner.uno probe above resolves in-cluster (CoreDNS/k8s_gateway)