From d28b0d112d4b7d18b209c96c19f7d47923b22be7 Mon Sep 17 00:00:00 2001 From: Konstantin Renner Date: Fri, 10 Jul 2026 19:26:24 +0200 Subject: [PATCH] monitoring: add in-cluster /-/healthy probes for the gated UIs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Follow-up to the Alertmanager (#146) and Prometheus (#147) gates. Both alerts.renner.uno and prometheus.renner.uno are now Authentik-gated, and the existing blackbox http_2xx probes target those external hostnames. A gated endpoint returns 302; blackbox follows it to the Authentik login (200), so the probe stays GREEN — but it now measures the gate path, not the app. A dead Prometheus/Alertmanager behind a live gate would show up. Verified empirically before this change: probe_success{instance= "https://alerts.renner.uno"}=1 with final_status=200 (3 redirect hops landing on auth.renner.uno). No probe goes red, but the signal is a false green for app health. Fix per monitoring-vantage-point-rule: an in-cluster service must be probed from an in-cluster prober at its in-cluster address. Add two INTERNAL targets to the blackbox-http Probe: http://kube-prometheus-stack-alertmanager.monitoring.svc:9093/-/healthy http://kube-prometheus-stack-prometheus.monitoring.svc:9090/-/healthy The gate is only on the ingress, so these ClusterIP /-/healthy endpoints are reachable and unauthenticated in-cluster (verified: alertmanager -> 200 "OK", prometheus -> 200 "Prometheus Server is Healthy"). blackbox- exporter runs in the monitoring namespace, so intra-namespace reach to :9093/:9090 is allowed. These probes go red on actual app death, independent of the gate — closing the blind spot. The external hostname probes are KEPT: they still catch DNS/TLS/Traefik/ gate-outpost failures that an internal probe cannot. So both signals now exist — app health (internal) and gated-path reachability (external) — instead of one conflated one. Scope note: the legacy srv-host Prometheus on VLAN 40 (ansible/roles/prometheus-server, ansible/roles/blackbox-exporter) also probes https://prometheus.renner.uno, but VLAN 40 has no route to cluster ClusterIPs, so it structurally cannot adopt the internal-health probe. Per the vantage-point rule the in-cluster prober is the correct owner of app health; the srv-host probe stays a pure external-reachability check and is left untouched (its cleanup belongs with the eventual retirement of the legacy srv-host monitoring). No alert depends on these probe instances (only the Hetzner object-storage probe is alerted), so no alert breaks. Verified: kubectl kustomize builds; rendered Probe carries both internal targets. Post-merge: probe_success for the two /-/healthy instances == 1, and killing the app path (or confirming via a scaled test) drops it. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../monitoring-rules/probes-blackbox.yaml | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/kubernetes/apps/monitoring-rules/probes-blackbox.yaml b/kubernetes/apps/monitoring-rules/probes-blackbox.yaml index 26a93c5d..4c496607 100644 --- a/kubernetes/apps/monitoring-rules/probes-blackbox.yaml +++ b/kubernetes/apps/monitoring-rules/probes-blackbox.yaml @@ -21,6 +21,13 @@ spec: - https://immich.renner.uno - https://auth.renner.uno - https://home.renner.uno + # prometheus.renner.uno and alerts.renner.uno are Authentik-gated + # (PR #147 / #146). These EXTERNAL probes now measure the full + # gated path — DNS, TLS, Traefik, the gate outpost — following the + # 302 to the Authentik login (200). They are REACHABILITY checks, + # not app health: with the gate up, they stay green even if the app + # behind it is dead (the probe never reaches the app). Real app + # health comes from the two INTERNAL /-/healthy probes below. - https://prometheus.renner.uno - https://alerts.renner.uno - https://dash.renner.uno @@ -28,6 +35,18 @@ spec: # returns 200. CNPG WAL archiving + Velero both depend on it; # this probe replaces the old GarageDown signal on nas-1. - https://fsn1.your-objectstorage.com + # APP-HEALTH probes for the gated monitoring UIs, from the correct + # in-cluster vantage (monitoring-vantage-point-rule): the gate is + # only on the ingress, so the ClusterIP /-/healthy endpoints are + # reachable and unauthenticated in-cluster. These go red when the + # app is actually down, regardless of the gate — closing the blind + # spot where a dead app behind a live gate showed green. The + # blackbox-exporter runs in this namespace, so intra-namespace + # reach to :9093/:9090 is allowed. (The legacy srv-host Prometheus + # on VLAN 40 cannot reach cluster ClusterIPs, so it keeps only the + # external reachability probe above; it is not the health vantage.) + - http://kube-prometheus-stack-alertmanager.monitoring.svc.cluster.local:9093/-/healthy + - http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090/-/healthy # blackbox-tcp probe deleted 2026-05-20 — only target was the legacy # postgres-ha keepalived VIP at 10.2.40.135:5432, decommissioned after # the authentik DB moved to in-cluster CNPG (ak-pg). Re-introduce a