Closes the H2 gap from the rotation PR (#297) adversarial review: the rotation Job is fire-and-forget, so a downstream failure (ESO/Reloader/pod-reschedule) yields a green Job with no signal. ESO metrics aren't scraped here, so these are kube-state-metrics based:
- WoodpeckerServerNotReady (critical): server StatefulSet 0 ready for 20m.
- WoodpeckerAgentNotReady (warning): agent Deployment 0 available for 20m.
- WoodpeckerSecretRotationJobFailed (warning): named signal for a failed rotation Job (e.g. missing OpenBao bootstrap).
- WoodpeckerSecretRotationStale (warning): quarterly CronJob hasn't succeeded in >100d; absent-until-first-success so it can't false-fire early.
Independent of #297: readiness alerts are useful immediately; the Job/CronJob series stay empty until that PR merges.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
The dashboard was entirely dead: every panel queried postgres_exporter
metrics (pg_up, pg_stat_activity_count, ...) that have never existed in
this Prometheus. Postgres runs as four CloudNativePG clusters (ak-pg,
im-pg, gf-pg, pprl-pg) whose built-in exporter on :9187 was not scraped
at all (enablePodMonitor is false and no PodMonitor existed).
- add PodMonitors for the four CNPG clusters in monitoring-rules,
labelled release=kube-prometheus-stack so the Prometheus
podMonitorSelector actually picks them up (CNPG's own
enablePodMonitor would generate unlabelled PodMonitors that this
Prometheus ignores)
- rewrite all dashboard queries to the cnpg_* metrics the exporter
actually exposes (verified against the pods' /metrics endpoints)
- dedupe multi-instance clusters (max by namespace/datname for sizes,
primary-only filter via cnpg_pg_replication_in_recovery == 0 for
per-database counters)
- replace the Active Locks stat (no lock metric in the CNPG default
queries) with Max Replication Lag
- instant queries on all stat panels so dead instances show No data
instead of a stale last value
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-up to the Alertmanager (#146) and Prometheus (#147) gates. Both
alerts.renner.uno and prometheus.renner.uno are now Authentik-gated, and
the existing blackbox http_2xx probes target those external hostnames. A
gated endpoint returns 302; blackbox follows it to the Authentik login
(200), so the probe stays GREEN — but it now measures the gate path, not
the app. A dead Prometheus/Alertmanager behind a live gate would show up.
Verified empirically before this change: probe_success{instance=
"https://alerts.renner.uno"}=1 with final_status=200 (3 redirect hops
landing on auth.renner.uno). No probe goes red, but the signal is a false
green for app health.
Fix per monitoring-vantage-point-rule: an in-cluster service must be
probed from an in-cluster prober at its in-cluster address. Add two
INTERNAL targets to the blackbox-http Probe:
http://kube-prometheus-stack-alertmanager.monitoring.svc:9093/-/healthyhttp://kube-prometheus-stack-prometheus.monitoring.svc:9090/-/healthy
The gate is only on the ingress, so these ClusterIP /-/healthy endpoints
are reachable and unauthenticated in-cluster (verified: alertmanager ->
200 "OK", prometheus -> 200 "Prometheus Server is Healthy"). blackbox-
exporter runs in the monitoring namespace, so intra-namespace reach to
:9093/:9090 is allowed. These probes go red on actual app death,
independent of the gate — closing the blind spot.
The external hostname probes are KEPT: they still catch DNS/TLS/Traefik/
gate-outpost failures that an internal probe cannot. So both signals now
exist — app health (internal) and gated-path reachability (external) —
instead of one conflated one.
Scope note: the legacy srv-host Prometheus on VLAN 40
(ansible/roles/prometheus-server, ansible/roles/blackbox-exporter) also
probes https://prometheus.renner.uno, but VLAN 40 has no route to cluster
ClusterIPs, so it structurally cannot adopt the internal-health probe.
Per the vantage-point rule the in-cluster prober is the correct owner of
app health; the srv-host probe stays a pure external-reachability check
and is left untouched (its cleanup belongs with the eventual retirement of
the legacy srv-host monitoring). No alert depends on these probe instances
(only the Hetzner object-storage probe is alerted), so no alert breaks.
Verified: kubectl kustomize builds; rendered Probe carries both internal
targets. Post-merge: probe_success for the two /-/healthy instances == 1,
and killing the app path (or confirming via a scaled test) drops it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Finish line of the backup migration — nas-1's Garage is no longer a
cluster-backup dependency, so:
- drop the prometheus `garage` scrape job (s3.nas-1.renner.uno:39033) +
the garage-metrics-token secret (now orphaned)
- delete prometheusrule-garage.yaml (GarageDown / GarageScrapeFailing) —
it would fire every night once nas-1 powers off
Replace it with outage visibility for the new offsite target:
- add https://fsn1.your-objectstorage.com to the blackbox-http probe
(bare endpoint returns 200)
- new HetznerBackupEndpointDown alert on probe_success == 0 for 10m
Velero outages are already covered by VeleroBackupValidationFailure /
VeleroBackupStale. A granular CNPG "WAL archiving behind" alert needs
CNPG PodMonitors (enablePodMonitor=false today, metrics not scraped) —
left as a follow-up rather than shipping an alert on an absent metric.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Today's Garage outage at 02:00-04:00 UTC silently lost 3 of 6 daily
backups (paperless, audiobookshelf, authentik all FailedValidation
because BackupStorageLocation was Unavailable). GarageDown alert
fired, but nothing surfaced per-schedule backup loss. Add three
alerts covering validation failure, failed/partial backup, and
30h+ stale schedule.
ha.renner.uno and grafana.renner.uno don't exist as Ingress hosts —
Home Assistant lives at home.renner.uno, Grafana at dash.renner.uno
(already in list). NXDOMAIN every 30s was the cluster's top error
generator (721 lines/hr in Loki). Rename ha → home, drop grafana
duplicate.
Loki had no Prometheus scrape config — no loki_* metrics, no way to
alert on the log stack itself going sideways. ServiceMonitor covers
the single-binary + canary services, alerts cover Down / 5xx /
panics / canary-detected log loss.
Two Alertmanager alerts cleared at the source:
1. KubeDeploymentRolloutStuck / KubeContainerWaiting / KubePodNotReady
on apps/gatus. The deployment's RWO PVC (gatus-data, ceph-block)
can't be mounted twice. Default RollingUpdate strategy tries to
start the new pod BEFORE terminating the old one — surge=25% of
1 replica rounds up to 1 surge pod. The new pod hangs in
ContainerCreating indefinitely (no event, just a stuck volume
attachment) and ProgressDeadlineExceeded fires.
Recreate strategy guarantees the old pod releases the volume first.
Same fix the home-assistant Deployment already uses for the same
reason.
2. TargetDown blackbox-tcp / monitoring. The only target was the
keepalived VIP for the legacy authentik-HA Postgres LXC pair at
10.2.40.135:5432. That LXC stack was decommissioned when authentik
moved to in-cluster CNPG (ak-pg). Probe deleted; the comment
block in the file documents what's no longer there.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 2b of the dashboard triage. Deploys prometheus-community's
blackbox-exporter chart in the monitoring namespace and creates Probe
CRs covering the legacy Ansible blackbox_http_targets list plus the
TCP probe for the legacy postgres-ha host. The chart's ServiceMonitor
exposes the exporter's own metrics; the Probe CRs drive periodic
endpoint probing.
Feeds blackbox-overview dashboard.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 2a of the dashboard triage. Wires three metric sources whose
dashboards previously had no data path:
- alloy-servicemonitor.yaml: scrapes the alloy DaemonSet's
http-metrics:12345 endpoint. Feeds alloy-overview.
- immich-servicemonitor.yaml: scrapes immich-server's metrics:8081
endpoint (already exposed by the existing Service). Feeds
immich-overview.
- helmrelease-prometheus.yaml: adds a static scrape for
gitea.srv.renner.uno:3000/metrics (anonymous, no auth token
required). Feeds gitea-overview.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The in-cluster ServiceMonitor was missing the `release: kube-prometheus-stack`
selector label, so the Prometheus resource never discovered it (same trap
the garage PrometheusRule documents). Adding the label brings the scrape
online via the existing in-cluster Service + bearer-token Secret.
Also removes the obsolete `homeassistant` additionalScrapeConfig that
pointed at the external IoT-VLAN IP with an empty bearer_token TODO —
HA lives in the cluster now, the external path is dead.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
scrape_samples_scraped is a gauge, not a counter, so rate() of a
stable value is 0 — the alert fired whenever Garage was healthy and
producing a steady sample count. Switch to a direct gauge==0
comparison so the alert reflects the annotated intent (endpoint up,
zero samples returned).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
matter-server's seed initContainer mounts garage-migration-creds, owned
by the home-assistant Kustomization. Without an explicit dependsOn the
init container CreateContainerConfigError-loops on a fresh rebuild while
HA is still gated upstream — observed during the 2026-05-05 cold start.
home-assistant's ServiceMonitor consumes a CRD installed by the
kube-prometheus-stack chart (monitoring KS), so applying it from the HA
KS races the CRD on first reconcile. Move it under monitoring-rules,
which already exists for monitoring.coreos.com/* resources after the
chart lands its CRDs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The instrumented wait_kustomization helper from #731 finally let us see
*why* paperless / monitoring / home-assistant Kustomizations were stalling.
Three independent bugs:
1. monitoring stalled forever with
PrometheusRule/monitoring/garage: dry-run failed: no matches for
kind "PrometheusRule" in version "monitoring.coreos.com..."
Chicken-and-egg: the rule's CRD comes from kube-prometheus-stack's
HelmRelease in the same Kustomization, so Flux's server-side dry-run
on the rule fails before the chart can install. Move the rule into a
new `monitoring-rules` Kustomization that dependsOn `monitoring` —
chart installs (CRDs and all) first, then the rule applies cleanly.
2. home-assistant stalled with
Service/home-assistant-iot: namespace not specified
iot-service.yaml was missing `metadata.namespace: home-assistant`
while every other manifest in the dir has it. Added.
3. paperless stalled with
Deployment/paperless/paperless status: 'F[ailed early due to ...]
The truncated message was cut off at 120 chars in the per-iteration
progress line so we couldn't see the actual reason. Bump truncation
to 240 chars and dump full `flux get kustomization` + recent warning
events on TIMEOUT so the next run gives a definitive answer instead
of needing a second cluster trip.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
claude-botandClaude Opus 4.8
k.rennerandClaude Opus 4.8