docs(monitoring): egress dependency, blast radius, tested revert recipe (REN-21)
Comment-only. Rebuilt on top of master (2b71dda1) after #979 merged, so it adds to that text instead of reverting it. - EGRESS: names the two ingress-only Cilium policies in the monitoring namespace and where blackbox-exporter runs; counts re-verified at2b71dda1(635 yaml files, 73 Cilium objects, 38 with egress/egressDeny, 0 in monitoring, 1 CCNP scoped to reserved.health, 17 namespaces with an egress block). - BLAST RADIUS: what breaks if that dependency is ever broken (BlackboxProbeFailing x3, BlackboxProbesAbsent stays silent), who notices (admin@renner.uno, ~10-11 min), and how fast it is undone. - REVERT: replaced the claim with an executed recipe. All four git revert candidates CONFLICT at 2b71dda1; the two forward sed deletions were run and verified (blackbox-http 20 -> 17 and 20 -> 19). No object changes: 3 documents deep-equal before/after under two independent parsers, blackbox-http still 20 targets, and `git diff -U0` has zero non-comment non-blank changed lines. apps/gatus/configmap.yaml is not touched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
atlas-bot
co-authored by
Claude Opus 5
Claude Opus 5
parent
2b71dda105
commit
49bd927147
@@ -116,16 +116,41 @@ spec:
|
||||
# destination ingress allow above is worthless if the prober itself
|
||||
# is in egress enforcement without a matching rule. It is not: an
|
||||
# endpoint only enters egress enforcement once some policy selecting
|
||||
# it declares an `egress:` block, and none does for monitoring.
|
||||
# Verified at 0a82d61f across the whole repo — 73 Cilium policies, 38
|
||||
# with an egress block, none of them in the monitoring namespace, and
|
||||
# the single CiliumClusterwideNetworkPolicy is scoped to
|
||||
# `reserved.health` so it cannot select a pod. 17 other namespaces
|
||||
# here already have an egress block. If a routine hardening pass ever
|
||||
# adds one to monitoring, it MUST list the navidrome and paperless
|
||||
# namespaces or these three cross-namespace targets go red while the
|
||||
# *.monitoring.svc ones above keep working — a failure mode nobody
|
||||
# will connect back to the hardening commit.
|
||||
# it declares an `egress:` block, and none does for monitoring. The
|
||||
# namespace has exactly two Cilium policies and both are ingress-only:
|
||||
# infrastructure/network-policies/monitoring-ns.yaml
|
||||
# (default-deny-ingress) and monitoring-alloy-syslog.yaml
|
||||
# (allow-syslog-world-ingress). blackbox-exporter runs in that same
|
||||
# namespace (apps/monitoring/helmrelease-blackbox-exporter.yaml:5).
|
||||
# Re-verified at 2b71dda1 across all 635 yaml files in the repo — 73
|
||||
# Cilium objects, 38 carrying an egress or egressDeny block, none of
|
||||
# them in the monitoring namespace, and the single
|
||||
# CiliumClusterwideNetworkPolicy is scoped to `reserved.health` so it
|
||||
# cannot select a pod. 17 other namespaces here already have an egress
|
||||
# block. If a routine hardening pass ever adds one to monitoring, it
|
||||
# MUST list the navidrome and paperless namespaces or these three
|
||||
# cross-namespace targets go red while the *.monitoring.svc ones above
|
||||
# keep working — a failure mode nobody will connect back to the
|
||||
# hardening commit. The cnp-cross-namespace-* notes document only the
|
||||
# ingress half; that silence is what makes this the trap.
|
||||
#
|
||||
# BLAST RADIUS if that egress dependency is ever broken. Nothing
|
||||
# user-facing breaks: these are Prometheus-side Probe targets, not
|
||||
# gatus checks, so status.renner.uno is unaffected and no workload is
|
||||
# restarted. What breaks is the signal, and it breaks misleadingly.
|
||||
# All three targets start failing BlackboxProbeFailing
|
||||
# (prometheusrule-blackbox.yaml:34 — severity warning, for: 10m); none
|
||||
# of the three is in that rule's exclusion list. BlackboxProbesAbsent
|
||||
# does NOT fire, because the other 17 targets keep producing
|
||||
# probe_success — so it reads as three apps going down, not as a
|
||||
# policy change. WHO NOTICES: warning routes to the default `email`
|
||||
# receiver, admin@renner.uno (helmrelease-prometheus.yaml alertmanager
|
||||
# route, group_wait 30s, grouped by alertname), so the owner gets one
|
||||
# email roughly 10-11 minutes after the policy lands. HOW FAST UNDONE:
|
||||
# the REVERT recipe below is a one-line deletion; from merge it is
|
||||
# live within ~11 minutes (GitRepository interval 1m0s,
|
||||
# monitoring-rules Kustomization interval 10m), or immediately with
|
||||
# `flux reconcile kustomization monitoring-rules --with-source`.
|
||||
#
|
||||
# /ping on both: unauthenticated, and already the pods' own kubelet
|
||||
# probe path (navidrome/deployment.yaml:61-74,
|
||||
@@ -142,16 +167,41 @@ spec:
|
||||
# probe_http_status_code on the first scrapes; a target that settles
|
||||
# on a 3xx gets REVERTED, not widened to accept 3xx.
|
||||
#
|
||||
# REVERT (recipe tested by review at 0a82d61f): delete everything
|
||||
# from the "APP-HEALTH probes for the two gated navidrome-namespace
|
||||
# apps" line above through the lidarr target below. Do NOT use
|
||||
# `git revert -m 1 16238d39` — it conflicts, and resolving the
|
||||
# conflict the natural way also drops the paperless target below,
|
||||
# silently leaving no app-health signal at all. The paperless block
|
||||
# below reverts on its own with `git revert -m 1 0a82d61f`, but only
|
||||
# against a tree where this comment block is absent — this commit
|
||||
# edits that block, so expect one conflict covering the whole
|
||||
# paperless block, and resolve it by taking the deletion.
|
||||
# REVERT — an executed recipe, not a claim. Every command below was
|
||||
# actually run against master 2b71dda1 on 2026-09-25; the results are
|
||||
# what is recorded here.
|
||||
#
|
||||
# NO `git revert` works. All four candidates conflict in THIS file at
|
||||
# 2b71dda1, because the comment blocks around these targets were
|
||||
# rewritten after the probes landed (#979):
|
||||
#
|
||||
# git revert -m 1 0a82d61f merge of #977, paperless CONFLICT
|
||||
# git revert -m 1 16238d39 merge of #976, nav + lidarr CONFLICT
|
||||
# git revert cb9e73bd the paperless commit CONFLICT
|
||||
# git revert 2219c928 the nav + lidarr commit CONFLICT
|
||||
#
|
||||
# Hand-resolving any of them is the trap: the natural --theirs
|
||||
# resolution of the 16238d39 conflict also drops the paperless target
|
||||
# below, leaving NO app-health probes and no signal that it happened.
|
||||
#
|
||||
# Delete the target lines forward instead. Only the target lines are
|
||||
# part of the object; the comments are inert. Both of these were run
|
||||
# and verified:
|
||||
#
|
||||
# # all three app-health targets — blackbox-http 20 -> 17
|
||||
# sed -i '/svc\.cluster\.local:80\/ping$/d;/paperless\.paperless\.svc\.cluster\.local:80\/accounts\/login\/$/d' \
|
||||
# kubernetes/apps/monitoring-rules/probes-blackbox.yaml
|
||||
#
|
||||
# # the paperless target alone — blackbox-http 20 -> 19
|
||||
# sed -i '/paperless\.paperless\.svc\.cluster\.local:80\/accounts\/login\/$/d' \
|
||||
# kubernetes/apps/monitoring-rules/probes-blackbox.yaml
|
||||
#
|
||||
# Verified after each run: the file still parses, blackbox-http drops
|
||||
# to 17 and to 19 respectively, blackbox-edge-wan (3) and
|
||||
# blackbox-tcp-relays (2) are untouched, and `git diff` contains
|
||||
# nothing but the deleted target lines. Then drop whichever comment
|
||||
# block is left orphaned, open a PR, and merge — timing as in the
|
||||
# BLAST RADIUS paragraph above.
|
||||
- http://navidrome.navidrome.svc.cluster.local:80/ping
|
||||
- http://lidarr.navidrome.svc.cluster.local:80/ping
|
||||
# APP-HEALTH probe for paperless. Same rule as the block above
|
||||
@@ -193,7 +243,10 @@ spec:
|
||||
#
|
||||
# This is the target most likely to fail the STATUS CODE check in the
|
||||
# block above — a Django login view can answer 302 — so check it
|
||||
# first, and revert rather than widen if it does.
|
||||
# first, and revert rather than widen if it does. Use the REVERT
|
||||
# recipe in that block: `git revert -m 1 0a82d61f` conflicts here too
|
||||
# at 2b71dda1, and the tested one-line deletion for this target alone
|
||||
# is the second command listed there (20 -> 19 targets).
|
||||
- http://paperless.paperless.svc.cluster.local:80/accounts/login/
|
||||
# WAN-path probes of the public edge BY PROXY IP. The name-based
|
||||
# home.renner.uno probe above resolves in-cluster (CoreDNS/k8s_gateway)
|
||||
|
||||
Reference in New Issue
Block a user