Gate Prometheus behind Authentik #147

2026-07-10T16:43:19Z

k.renner commented

2026-07-10 16:43:19 +00:00

Why

Prometheus (prometheus.renner.uno, monitoring ns, :9090) has no auth of its own. Its admin API is disabled (read-only), but the full 90-day metrics DB is an infra-recon oracle — every hostname, IP, and service in the cluster. Last of the deferred unauthenticated monitoring findings from the 2026-07-10 audit.

Gated the same way as alertmanager-gate (#146).

What

Proxy provider + application + outpost (ak-outpost-prometheus-gate) → ClusterIP Service kube-prometheus-stack-prometheus.monitoring.svc:9090.
Reuses the existing "Monitoring Access" group — created generically in #146 for exactly this. No new group.
implicit-consent flow (same reasoning as alertmanager/paperless: a frequently-checked dashboard; an authenticated session already closes the exposure).
No skip_path_regex — verified the Prometheus UI's index.html has no serviceWorker/manifest/webmanifest.
Helm-generated Prometheus Ingress disabled (prometheus.ingress.enabled: false); standalone gate ingress backends the outpost ExternalName. Mirrors the Alertmanager gate.
Scoped CNPs: monitoring ingress rule allowing only the outpost → :9090; matching egress in the authentik namespace.

Grafana is unaffected

Its Prometheus datasource targets the ClusterIP Service (kube-prometheus-stack-prometheus.monitoring:9090) — verified in the live datasource provisioning — not the ingress hostname. Dashboards keep querying Prometheus directly.

Documented side effect (not silently shipped)

Three blackbox http_2xx uptime probes target https://prometheus.renner.uno: kubernetes/apps/monitoring-rules/probes-blackbox.yaml, the srv-host prometheus.yml.j2, and the ansible blackbox-exporter role. After gating they follow the 302 to the Authentik login (200), so they stay GREEN — but now measure Authentik reachability, not Prometheus health.

This is already the case for alerts.renner.uno since #146 — verified empirically: probe_success=1, final_status=200 via redirect-follow (3 hops, lands on auth.renner.uno). No probe goes red; the check just degrades to a gate-liveness signal. Repointing those probes to an internal endpoint (so they measure the actual app again) is a separate follow-up, not done here — it touches the blackbox target lists for multiple services and is its own decision. Flagging it because it affects both this PR and the already-merged #146.

Verified so far

kubectl kustomize builds for authentik, monitoring, network-policies; the embedded outpost survived the blueprint insert; rendered ingress → ak-outpost-prometheus-gate:9000; both prometheus and alertmanager Helm ingresses now disabled; egress + CNP scoped to prometheus:9090.

⚠️ Same rollout blip as #146

helm-controller (removing the Helm ingress) and kustomize-controller (applying the standalone one) reconcile on independent timers — a few seconds where both/neither ingress exists for prometheus.renner.uno. I'll wait for the Helm ingress to disappear before testing.

Post-merge checks

Only the gate Ingress serves prometheus.renner.uno (Helm one gone).
Unauthenticated GET / → 302 to auth.renner.uno.
Unauthenticated GET /api/v1/query → 302 (was 200).
Outpost forwards to :9090 after login.
Grafana dashboards still render (ClusterIP datasource intact).

## Why Prometheus (`prometheus.renner.uno`, monitoring ns, `:9090`) has no auth of its own. Its admin API is disabled (read-only), but the full 90-day metrics DB is an infra-recon oracle — every hostname, IP, and service in the cluster. Last of the deferred unauthenticated monitoring findings from the 2026-07-10 audit. Gated the same way as `alertmanager-gate` (#146). ## What - Proxy provider + application + outpost (`ak-outpost-prometheus-gate`) → ClusterIP Service `kube-prometheus-stack-prometheus.monitoring.svc:9090`. - **Reuses the existing "Monitoring Access" group** — created generically in #146 for exactly this. No new group. - implicit-consent flow (same reasoning as alertmanager/paperless: a frequently-checked dashboard; an authenticated session already closes the exposure). - No `skip_path_regex` — verified the Prometheus UI's `index.html` has no serviceWorker/manifest/webmanifest. - Helm-generated Prometheus Ingress disabled (`prometheus.ingress.enabled: false`); standalone gate ingress backends the outpost ExternalName. Mirrors the Alertmanager gate. - Scoped CNPs: monitoring ingress rule allowing only the outpost → `:9090`; matching egress in the authentik namespace. ## Grafana is unaffected Its Prometheus datasource targets the **ClusterIP Service** (`kube-prometheus-stack-prometheus.monitoring:9090`) — verified in the live datasource provisioning — not the ingress hostname. Dashboards keep querying Prometheus directly. ## Documented side effect (not silently shipped) Three blackbox `http_2xx` **uptime probes** target `https://prometheus.renner.uno`: `kubernetes/apps/monitoring-rules/probes-blackbox.yaml`, the srv-host `prometheus.yml.j2`, and the ansible blackbox-exporter role. After gating they **follow the 302 to the Authentik login (200), so they stay GREEN** — but now measure Authentik reachability, not Prometheus health. This is already the case for `alerts.renner.uno` since #146 — verified empirically: `probe_success=1, final_status=200` via redirect-follow (3 hops, lands on `auth.renner.uno`). **No probe goes red**; the check just degrades to a gate-liveness signal. Repointing those probes to an internal endpoint (so they measure the actual app again) is a **separate follow-up**, not done here — it touches the blackbox target lists for multiple services and is its own decision. Flagging it because it affects both this PR and the already-merged #146. ## Verified so far `kubectl kustomize` builds for authentik, monitoring, network-policies; the embedded outpost survived the blueprint insert; rendered ingress → `ak-outpost-prometheus-gate:9000`; both `prometheus` and `alertmanager` Helm ingresses now disabled; egress + CNP scoped to `prometheus:9090`. ## ⚠️ Same rollout blip as #146 helm-controller (removing the Helm ingress) and kustomize-controller (applying the standalone one) reconcile on independent timers — a few seconds where both/neither ingress exists for `prometheus.renner.uno`. I'll wait for the Helm ingress to disappear before testing. ## Post-merge checks 1. Only the gate Ingress serves `prometheus.renner.uno` (Helm one gone). 2. Unauthenticated `GET /` → `302` to `auth.renner.uno`. 3. Unauthenticated `GET /api/v1/query` → `302` (was `200`). 4. Outpost forwards to `:9090` after login. 5. Grafana dashboards still render (ClusterIP datasource intact).

k.renner added 1 commit 2026-07-10 16:43:19 +00:00

Gate Prometheus behind Authentik ce0c78b503

Prometheus (prometheus.renner.uno, monitoring ns, :9090) had no auth of its
own. Its admin API is disabled (read-only), but the full 90d metrics DB is
an infra-recon oracle — every hostname, IP and service in the cluster
(2026-07-10 audit). Gate it the same way as alertmanager-gate (#146).

- Blueprint: prometheus-gate proxy provider (internal_host
  http://kube-prometheus-stack-prometheus.monitoring.svc:9090, external_host
  https://prometheus.renner.uno), application, outpost. REUSES the existing
  git-owned "Monitoring Access" group — created generically in #146 for
  exactly this, so no new group.
- implicit-consent flow (same rationale as alertmanager/paperless: a
  frequently-checked monitoring UI; an authenticated session already closes
  the exposure).
- No skip_path_regex: the Prometheus UI is a plain React SPA — verified its
  index.html has no serviceWorker/manifest/webmanifest.
- Helm-generated Prometheus Ingress disabled (prometheus.ingress.enabled=
  false); standalone prometheus-gate-ingress.yaml backends the outpost
  ExternalName. Mirrors the alertmanager gate.
- Scoped CNPs: monitoring ingress rule allowing only the outpost -> :9090;
  matching egress in the authentik namespace.

Grafana is UNAFFECTED: its Prometheus datasource targets the ClusterIP
Service (kube-prometheus-stack-prometheus.monitoring:9090), verified in the
live datasource provisioning, not the ingress hostname.

Known side effect, documented not silently shipped: three blackbox
`http_2xx` uptime probes target https://prometheus.renner.uno
(kubernetes/apps/monitoring-rules/probes-blackbox.yaml, the srv-host
ansible prometheus.yml.j2, and the ansible blackbox-exporter role). After
gating they follow the 302 to the Authentik login (200), so they stay GREEN
— but now measure Authentik reachability, not Prometheus. This is already
true for alerts.renner.uno since #146 (verified: probe_success=1,
final_status=200 via redirect-follow). No probe goes red; the monitoring
just becomes a gate-liveness check. Repointing those probes to an internal
endpoint is a separate follow-up, not done here.

Verified: kubectl kustomize builds for authentik, monitoring,
network-policies; embedded outpost survived the blueprint insert; rendered
ingress backends ak-outpost-prometheus-gate:9000; both prometheus and
alertmanager helm ingresses now disabled; egress + CNP scoped to
prometheus:9090.

Post-merge checks: only the gate Ingress serves prometheus.renner.uno (helm
one gone); unauthenticated GET -> 302 to auth.renner.uno; unauthenticated
GET /api/v1/query -> 302; outpost forwards to :9090 after login; Grafana
dashboards still render (ClusterIP datasource intact).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>