monitoring: add in-cluster /-/healthy probes for the gated UIs #148

2026-07-10T17:27:13Z

k.renner commented

2026-07-10 17:27:13 +00:00

The blind spot

Follow-up to the Alertmanager (#146) and Prometheus (#147) gates. Both hostnames are now Authentik-gated, and the blackbox http_2xx probes target those external URLs. A gated endpoint returns 302; blackbox follows it to the Authentik login (200), so the probe stays green — but now measures the gate path, not the app. A dead Prometheus or Alertmanager behind a live gate would show up as healthy.

Verified empirically before this change: probe_success{instance="https://alerts.renner.uno"}=1, final_status=200, 3 redirect hops landing on auth.renner.uno. No probe goes red — it's a false green for app health.

The fix (per `monitoring-vantage-point-rule`)

An in-cluster service must be probed from an in-cluster prober at its in-cluster address. Add two internal targets to the blackbox-http Probe:

http://kube-prometheus-stack-alertmanager.monitoring.svc:9093/-/healthy
http://kube-prometheus-stack-prometheus.monitoring.svc:9090/-/healthy

The gate is only on the ingress, so these ClusterIP /-/healthy endpoints are reachable and unauthenticated in-cluster — verified: alertmanager → 200 OK, prometheus → 200 Prometheus Server is Healthy. blackbox-exporter runs in the monitoring namespace, so intra-namespace reach is allowed. These go red on actual app death, regardless of the gate.

Both signals now exist

The external hostname probes are kept — they still catch DNS / TLS / Traefik / gate-outpost failures an internal probe can't see. So instead of one conflated signal, there are now two honest ones:

app health → internal /-/healthy (new)
gated-path reachability → external hostname (existing)

Scope: the legacy srv-host is deliberately left alone

The srv-host Prometheus on VLAN 40 (ansible/roles/prometheus-server, ansible/roles/blackbox-exporter) also probes https://prometheus.renner.uno — but VLAN 40 has no route to cluster ClusterIPs, so it structurally cannot adopt the internal probe. Per the vantage-point rule the in-cluster prober is the correct owner of app health; the srv-host probe stays a pure external-reachability check, untouched, to be cleaned up when the legacy srv-host monitoring is retired. This is a principled scope boundary, not an omission — the blind spot is closed by the in-cluster probe regardless.

No alert depends on these probe instances (only the Hetzner object-storage probe is alerted), so nothing breaks.

Verified

kubectl kustomize builds; the rendered Probe carries both internal targets.

Post-merge check

probe_success{instance="http://…alertmanager…:9093/-/healthy"} and the prometheus one both == 1 (the in-cluster prober can reach them). That confirms the real app-health signal is live.

## The blind spot Follow-up to the Alertmanager (#146) and Prometheus (#147) gates. Both hostnames are now Authentik-gated, and the blackbox `http_2xx` probes target those external URLs. A gated endpoint returns `302`; blackbox **follows it to the Authentik login (200)**, so the probe stays **green** — but now measures the gate path, not the app. A dead Prometheus or Alertmanager behind a live gate would show up as healthy. Verified empirically before this change: `probe_success{instance="https://alerts.renner.uno"}=1`, `final_status=200`, 3 redirect hops landing on `auth.renner.uno`. No probe goes red — it's a false green for app health. ## The fix (per `monitoring-vantage-point-rule`) An in-cluster service must be probed from an **in-cluster prober at its in-cluster address**. Add two internal targets to the `blackbox-http` Probe: ``` http://kube-prometheus-stack-alertmanager.monitoring.svc:9093/-/healthy http://kube-prometheus-stack-prometheus.monitoring.svc:9090/-/healthy ``` The gate is only on the ingress, so these ClusterIP `/-/healthy` endpoints are reachable and unauthenticated in-cluster — verified: alertmanager → `200 OK`, prometheus → `200 Prometheus Server is Healthy`. blackbox-exporter runs in the monitoring namespace, so intra-namespace reach is allowed. These go **red on actual app death, regardless of the gate**. ## Both signals now exist The external hostname probes are **kept** — they still catch DNS / TLS / Traefik / gate-outpost failures an internal probe can't see. So instead of one conflated signal, there are now two honest ones: - **app health** → internal `/-/healthy` (new) - **gated-path reachability** → external hostname (existing) ## Scope: the legacy srv-host is deliberately left alone The srv-host Prometheus on VLAN 40 (`ansible/roles/prometheus-server`, `ansible/roles/blackbox-exporter`) also probes `https://prometheus.renner.uno` — but VLAN 40 has **no route to cluster ClusterIPs**, so it structurally cannot adopt the internal probe. Per the vantage-point rule the in-cluster prober is the correct owner of app health; the srv-host probe stays a pure external-reachability check, untouched, to be cleaned up when the legacy srv-host monitoring is retired. This is a principled scope boundary, not an omission — the blind spot is closed by the in-cluster probe regardless. No alert depends on these probe instances (only the Hetzner object-storage probe is alerted), so nothing breaks. ## Verified `kubectl kustomize` builds; the rendered Probe carries both internal targets. ## Post-merge check `probe_success{instance="http://…alertmanager…:9093/-/healthy"}` and the prometheus one both == 1 (the in-cluster prober can reach them). That confirms the real app-health signal is live.

k.renner added 1 commit 2026-07-10 17:27:13 +00:00

monitoring: add in-cluster /-/healthy probes for the gated UIs d28b0d112d

Follow-up to the Alertmanager (#146) and Prometheus (#147) gates. Both
alerts.renner.uno and prometheus.renner.uno are now Authentik-gated, and
the existing blackbox http_2xx probes target those external hostnames. A
gated endpoint returns 302; blackbox follows it to the Authentik login
(200), so the probe stays GREEN — but it now measures the gate path, not
the app. A dead Prometheus/Alertmanager behind a live gate would show up.

Verified empirically before this change: probe_success{instance=
"https://alerts.renner.uno"}=1 with final_status=200 (3 redirect hops
landing on auth.renner.uno). No probe goes red, but the signal is a false
green for app health.

Fix per monitoring-vantage-point-rule: an in-cluster service must be
probed from an in-cluster prober at its in-cluster address. Add two
INTERNAL targets to the blackbox-http Probe:

  http://kube-prometheus-stack-alertmanager.monitoring.svc:9093/-/healthy
  http://kube-prometheus-stack-prometheus.monitoring.svc:9090/-/healthy

The gate is only on the ingress, so these ClusterIP /-/healthy endpoints
are reachable and unauthenticated in-cluster (verified: alertmanager ->
200 "OK", prometheus -> 200 "Prometheus Server is Healthy"). blackbox-
exporter runs in the monitoring namespace, so intra-namespace reach to
:9093/:9090 is allowed. These probes go red on actual app death,
independent of the gate — closing the blind spot.

The external hostname probes are KEPT: they still catch DNS/TLS/Traefik/
gate-outpost failures that an internal probe cannot. So both signals now
exist — app health (internal) and gated-path reachability (external) —
instead of one conflated one.

Scope note: the legacy srv-host Prometheus on VLAN 40
(ansible/roles/prometheus-server, ansible/roles/blackbox-exporter) also
probes https://prometheus.renner.uno, but VLAN 40 has no route to cluster
ClusterIPs, so it structurally cannot adopt the internal-health probe.
Per the vantage-point rule the in-cluster prober is the correct owner of
app health; the srv-host probe stays a pure external-reachability check
and is left untouched (its cleanup belongs with the eventual retirement of
the legacy srv-host monitoring). No alert depends on these probe instances
(only the Hetzner object-storage probe is alerted), so no alert breaks.

Verified: kubectl kustomize builds; rendered Probe carries both internal
targets. Post-merge: probe_success for the two /-/healthy instances == 1,
and killing the app path (or confirming via a scaled test) drops it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

k.renner merged commit 2fb32dbbc3 into master

2026-07-10 17:29:03 +00:00

k.renner referenced this issue from a commit

2026-07-10 17:29:04 +00:00

Merge pull request 'monitoring: add in-cluster /-/healthy probes for the gated UIs' (#148) from fix/blackbox-internal-health-gated-apps into master

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#148