Follow-up to the Alertmanager (#146) and Prometheus (#147) gates. Both
alerts.renner.uno and prometheus.renner.uno are now Authentik-gated, and
the existing blackbox http_2xx probes target those external hostnames. A
gated endpoint returns 302; blackbox follows it to the Authentik login
(200), so the probe stays GREEN — but it now measures the gate path, not
the app. A dead Prometheus/Alertmanager behind a live gate would show up.
Verified empirically before this change: probe_success{instance=
"https://alerts.renner.uno"}=1 with final_status=200 (3 redirect hops
landing on auth.renner.uno). No probe goes red, but the signal is a false
green for app health.
Fix per monitoring-vantage-point-rule: an in-cluster service must be
probed from an in-cluster prober at its in-cluster address. Add two
INTERNAL targets to the blackbox-http Probe:
http://kube-prometheus-stack-alertmanager.monitoring.svc:9093/-/healthyhttp://kube-prometheus-stack-prometheus.monitoring.svc:9090/-/healthy
The gate is only on the ingress, so these ClusterIP /-/healthy endpoints
are reachable and unauthenticated in-cluster (verified: alertmanager ->
200 "OK", prometheus -> 200 "Prometheus Server is Healthy"). blackbox-
exporter runs in the monitoring namespace, so intra-namespace reach to
:9093/:9090 is allowed. These probes go red on actual app death,
independent of the gate — closing the blind spot.
The external hostname probes are KEPT: they still catch DNS/TLS/Traefik/
gate-outpost failures that an internal probe cannot. So both signals now
exist — app health (internal) and gated-path reachability (external) —
instead of one conflated one.
Scope note: the legacy srv-host Prometheus on VLAN 40
(ansible/roles/prometheus-server, ansible/roles/blackbox-exporter) also
probes https://prometheus.renner.uno, but VLAN 40 has no route to cluster
ClusterIPs, so it structurally cannot adopt the internal-health probe.
Per the vantage-point rule the in-cluster prober is the correct owner of
app health; the srv-host probe stays a pure external-reachability check
and is left untouched (its cleanup belongs with the eventual retirement of
the legacy srv-host monitoring). No alert depends on these probe instances
(only the Hetzner object-storage probe is alerted), so no alert breaks.
Verified: kubectl kustomize builds; rendered Probe carries both internal
targets. Post-merge: probe_success for the two /-/healthy instances == 1,
and killing the app path (or confirming via a scaled test) drops it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
k.rennerandClaude Opus 4.8
d28b0d112d