Both wrong assessments (#249 openbao 'sealed', #250 three untested
hypotheses while one 'kubectl logs' showed the 401) share one root
cause: the agents diagnose blind, so they narrate plausible fiction.
Labeling unverified claims (previous commit) treats the symptom; this
gives them the means to verify:
- kubernetes/apps/monitoring/ci-diagnostics-rbac.yaml: ServiceAccount
ci-diagnostics with a read-only ClusterRole (workloads, events,
pods/log, Flux/ESO/cert-manager/monitoring/ceph CRDs — NO secrets)
plus a namespaced services/proxy Role for the Prometheus and
Alertmanager HTTP APIs.
- scripts/ci/make-diag-kubeconfig.sh: mints a 1h token per run from
the admin kubeconfig (no new long-lived secret), builds
.ci-diag/kubeconfig, copies the static kubectl binary for the
jq-only ci-claude image, smoke-tests with the restricted identity,
and degrades to snapshot-only mode when the SA isn't reconciled.
- alert-monitor.yml / issue-fix.yml: agents get exactly one extra tool,
Bash(./.ci-diag/kubectl:*) — prefix match forbids pipes and any other
binary; RBAC forbids mutation and secret reads.
- prompts: verification is now mandatory, not aspirational — 'a
hypothesis you could have settled with one read-only command but
didn't is a defect in your output'; fix agent must test the issue's
claims before building on them; minimal-quote rule against dumping
logs into public issues.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
evcc, otbr, libation, external-dns (apps), monitoring (4), woodpecker:
values in OpenBao KV (homelab/apps/*, loaded 2026-07-13), manifests are
now ExternalSecrets keeping the old Secret names. libation's payload is
a binary SQLite db — stored base64 in KV with decodingStrategy: Base64
on the ES. Decryption blocks dropped from apps/evcc/otbr/libation/
monitoring KS (paths sops-free) plus the dormant one on grafana-db.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CRgBFn9m8a2ncFMLusikH6
Prometheus (prometheus.renner.uno, monitoring ns, :9090) had no auth of its
own. Its admin API is disabled (read-only), but the full 90d metrics DB is
an infra-recon oracle — every hostname, IP and service in the cluster
(2026-07-10 audit). Gate it the same way as alertmanager-gate (#146).
- Blueprint: prometheus-gate proxy provider (internal_host
http://kube-prometheus-stack-prometheus.monitoring.svc:9090, external_host
https://prometheus.renner.uno), application, outpost. REUSES the existing
git-owned "Monitoring Access" group — created generically in #146 for
exactly this, so no new group.
- implicit-consent flow (same rationale as alertmanager/paperless: a
frequently-checked monitoring UI; an authenticated session already closes
the exposure).
- No skip_path_regex: the Prometheus UI is a plain React SPA — verified its
index.html has no serviceWorker/manifest/webmanifest.
- Helm-generated Prometheus Ingress disabled (prometheus.ingress.enabled=
false); standalone prometheus-gate-ingress.yaml backends the outpost
ExternalName. Mirrors the alertmanager gate.
- Scoped CNPs: monitoring ingress rule allowing only the outpost -> :9090;
matching egress in the authentik namespace.
Grafana is UNAFFECTED: its Prometheus datasource targets the ClusterIP
Service (kube-prometheus-stack-prometheus.monitoring:9090), verified in the
live datasource provisioning, not the ingress hostname.
Known side effect, documented not silently shipped: three blackbox
`http_2xx` uptime probes target https://prometheus.renner.uno
(kubernetes/apps/monitoring-rules/probes-blackbox.yaml, the srv-host
ansible prometheus.yml.j2, and the ansible blackbox-exporter role). After
gating they follow the 302 to the Authentik login (200), so they stay GREEN
— but now measure Authentik reachability, not Prometheus. This is already
true for alerts.renner.uno since #146 (verified: probe_success=1,
final_status=200 via redirect-follow). No probe goes red; the monitoring
just becomes a gate-liveness check. Repointing those probes to an internal
endpoint is a separate follow-up, not done here.
Verified: kubectl kustomize builds for authentik, monitoring,
network-policies; embedded outpost survived the blueprint insert; rendered
ingress backends ak-outpost-prometheus-gate:9000; both prometheus and
alertmanager helm ingresses now disabled; egress + CNP scoped to
prometheus:9090.
Post-merge checks: only the gate Ingress serves prometheus.renner.uno (helm
one gone); unauthenticated GET -> 302 to auth.renner.uno; unauthenticated
GET /api/v1/query -> 302; outpost forwards to :9090 after login; Grafana
dashboards still render (ClusterIP datasource intact).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Alertmanager (alerts.renner.uno, monitoring ns, :9093) had no auth of its
own, and its v2 API is writable: an unauthenticated LAN client could POST
/api/v2/silences and suppress ALL alerting — a tamper capability, not just
disclosure (2026-07-10 audit, highest-teeth of the deferred unauth
findings).
Gate it with the same Authentik proxy-outpost pattern as
paperless/vscode/shelly-manager:
- New proxy provider + application + dedicated outpost
(ak-outpost-alertmanager-gate), mode proxy, forwarding to the ClusterIP
Service (kube-prometheus-stack-alertmanager.monitoring.svc:9093).
- New git-owned "Monitoring Access" group (k.renner) bound to the app.
Named generically so it can also gate the still-deferred Prometheus UI.
- The Helm-chart-generated Alertmanager Ingress is disabled
(alertmanager.ingress.enabled=false); a standalone gate Ingress
(alertmanager-gate-ingress.yaml) backends the outpost ExternalName —
two Ingresses for the same host would conflict. Mirrors loki-ingress.
- Scoped CNPs: monitoring ingress rule allowing only the outpost -> :9093;
matching egress rule in the authentik namespace.
implicit-consent flow, NOT passkey step-up: Alertmanager is a
frequently-checked monitoring UI, and requiring an authenticated Authentik
session already closes the unauthenticated-LAN exposure. Passkey re-auth on
every visit is disproportionate friction for a dashboard (paperless-gate
uses implicit-consent for the same reason). Bump to
passkey-stepup-authorization if the write API warrants it.
No skip_path_regex: the Alertmanager UI is a plain Elm SPA with no service
worker and no manifest (verified 2026-07-10 — index.html has no
serviceWorker/manifest/webmanifest), so none of the credentials:'omit' PWA
fetches that forced skip carve-outs elsewhere.
Prometheus's alerting path is UNAFFECTED: it posts to the ClusterIP Service
(verified: spec.alerting.alertmanagers targets
monitoring/kube-prometheus-stack-alertmanager:http-web), not this ingress.
Prometheus's own ingress is untouched (still enabled) — its unauth exposure
is a separate deferred finding, and it is read-only.
Verified: kubectl kustomize builds for authentik, monitoring and
network-policies; the embedded outpost survived the blueprint insert; the
rendered ingress backends ak-outpost-alertmanager-gate:9000, egress is
scoped to alertmanager:9093, and alertmanager.ingress.enabled=false while
prometheus.ingress.enabled=true.
Post-merge checks to run: only the gate Ingress exists for alerts.renner.uno
(the Helm one is gone); unauthenticated GET -> 302 to auth.renner.uno;
POST /api/v2/silences unauthenticated is refused; the outpost forwards to
:9093 after login; Prometheus is still delivering alerts to Alertmanager.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Finish line of the backup migration — nas-1's Garage is no longer a
cluster-backup dependency, so:
- drop the prometheus `garage` scrape job (s3.nas-1.renner.uno:39033) +
the garage-metrics-token secret (now orphaned)
- delete prometheusrule-garage.yaml (GarageDown / GarageScrapeFailing) —
it would fire every night once nas-1 powers off
Replace it with outage visibility for the new offsite target:
- add https://fsn1.your-objectstorage.com to the blackbox-http probe
(bare endpoint returns 200)
- new HetznerBackupEndpointDown alert on probe_success == 0 for 10m
Velero outages are already covered by VeleroBackupValidationFailure /
VeleroBackupStale. A granular CNPG "WAL archiving behind" alert needs
CNPG PodMonitors (enablePodMonitor=false today, metrics not scraped) —
left as a follow-up rather than shipping an alert on an absent metric.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ingress-nginx auth-type/auth-secret/auth-realm trio becomes a Traefik
BasicAuth Middleware. Re-encrypts the SOPS Secret with both `auth`
(nginx convention, kept for safety during transition) and `users`
(Traefik convention) pointing at the same htpasswd content; SOPS file
now uses encrypted_regex so only stringData is encrypted (the rest of
the manifest is plaintext, matching the convention used elsewhere).
This is the last hostname on ingress-nginx. After this lands and
verifies, the controller can be decommissioned.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
VM 108 (the legacy InfluxDB at influxdb.srv.renner.uno:10.2.40.16) is
unreachable — no ICMP, no SSH, no port 8086. K8s Grafana's InfluxDB
datasource has been silently broken since the in-cluster InfluxDB
came online (commit 1daaa6f, 2026-05-05). The heatpump-roi dashboard
and any historical HA panel that queries InfluxDB are dead until this
repoint lands.
Datasource URL: http://influxdb-influxdb2.influxdb.svc.cluster.local:8086.
The legacy SOPS-stored token (`grafana-influxdb-token.sops.yaml`) is
useless against the in-cluster bolt store. Replaced with a seed Job
that mirrors the existing `influxdb-ha-token` pattern: ServiceAccount
in influxdb ns + cross-namespace Role on the `grafana-influxdb-token`
Secret in monitoring ns + Job that creates a read-only auth on the
suedstr28 bucket and writes the token to that Secret. The key name
inside the Secret stays `INFLUXDB_TOKEN` so the existing
`envFromSecrets` reference in the helmrelease keeps working.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 2b of the dashboard triage. Deploys prometheus-community's
blackbox-exporter chart in the monitoring namespace and creates Probe
CRs covering the legacy Ansible blackbox_http_targets list plus the
TCP probe for the legacy postgres-ha host. The chart's ServiceMonitor
exposes the exporter's own metrics; the Probe CRs drive periodic
endpoint probing.
Feeds blackbox-overview dashboard.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pointing unpoller v2.39.0 at the UDM caused the controller to crash.
Removing the deployment, secret, image-automation policy, and the
helmrelease pointer entirely. UniFi metrics stay offline until a
known-good poller/UDM combination is identified.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous deploys (Ansible roles install-prometheus.yml against
hosts: prometheus.srv.renner.uno on LXC 227) died with that host's
retirement. Re-deploy both as K8s Deployments in the monitoring
namespace, reusing the existing Ansible-vault credentials for the
Proxmox root@pam!ansible token and the UniFi `Poller` read-only user
(now stored in SOPS-encrypted Secrets per the K8s convention).
pve-exporter (prompve/prometheus-pve-exporter:3.8.3) listens on 9221
and is queried as a blackbox-style exporter: the existing pve scrape
config keeps the three 10.2.40.20x targets and uses relabel_configs
to direct the actual HTTP call to pve-exporter.monitoring.svc.
unpoller (ghcr.io/unpoller/unpoller:v2.1.5) polls https://udm.renner.uno
and exposes :9130. Scraped via a ServiceMonitor; the dead static
unifi scrape config is removed from the helmrelease.
Both images are enrolled in image-automation: pve-exporter on semver
3.x with plain 3-component tags; unpoller on semver 2.x with v-prefix
extract that rejects -rc and arch-suffixed variants.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Alertmanager (kube-prometheus-stack) now sends to admin@renner.uno via
Proton SMTP submission (smtp.protonmail.ch:587 STARTTLS) using a per-
account token in the alertmanager-smtp SOPS-encrypted Secret.
A new infrastructure subtree, flux-notifications/, adds a Provider of
type alertmanager pointing at the in-cluster Alertmanager and two
Alert CRs: one info-severity for ImageUpdateAutomation bumps (with the
30m "no change" heartbeat excluded), and one error-severity catch-all
across every Flux source/sync kind. Flux events and Prometheus alerts
now share a single email funnel.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ETP=Local broke cross-VLAN reach because workstations on VLAN 50 traverse
the UDM gateway L3, which lands traffic on whichever node it has ARP for —
with Local that node may not host the backend pod, producing TCP RST and
silent UDP drop. Match the default services-pool house pattern (ingress-nginx,
kube-dns-lb) which both run ETP=Cluster.
Drop the source_ip label promotion since under ETP=Cluster it would carry
a cluster node IP rather than the sender's IP.
Add an ingress-nginx Ingress for the in-cluster Loki at loki.k.renner.uno
gated by HTTP basic auth (htpasswd secret in sops) and TLS via
letsencrypt-prod, so the workstation can point logcli at the K8s Loki via
LOKI_ADDR/LOKI_USERNAME/LOKI_PASSWORD.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Stand up a dedicated alloy-syslog Deployment in monitoring that listens
on UDP/514, UDP/1514 (legacy parity), TCP/514 and TCP/6514+TLS and
writes into the in-cluster Loki SingleBinary.
Exposed as a Cilium L2 LoadBalancer pinned to 10.2.40.84 and published
as syslog.k.renner.uno via external-dns; that meant adding `service` to
external-dns-unifi sources (was ingress-only). TLS cert issued by
letsencrypt-prod through cert-manager.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The instrumented wait_kustomization helper from #731 finally let us see
*why* paperless / monitoring / home-assistant Kustomizations were stalling.
Three independent bugs:
1. monitoring stalled forever with
PrometheusRule/monitoring/garage: dry-run failed: no matches for
kind "PrometheusRule" in version "monitoring.coreos.com..."
Chicken-and-egg: the rule's CRD comes from kube-prometheus-stack's
HelmRelease in the same Kustomization, so Flux's server-side dry-run
on the rule fails before the chart can install. Move the rule into a
new `monitoring-rules` Kustomization that dependsOn `monitoring` —
chart installs (CRDs and all) first, then the rule applies cleanly.
2. home-assistant stalled with
Service/home-assistant-iot: namespace not specified
iot-service.yaml was missing `metadata.namespace: home-assistant`
while every other manifest in the dir has it. Added.
3. paperless stalled with
Deployment/paperless/paperless status: 'F[ailed early due to ...]
The truncated message was cut off at 120 chars in the per-iteration
progress line so we couldn't see the actual reason. Bump truncation
to 240 chars and dump full `flux get kustomization` + recent warning
events on TIMEOUT so the next run gives a definitive answer instead
of needing a second cluster trip.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Prometheus now scrapes Garage's admin endpoint on nas-1
(s3.nas-1.renner.uno:39033 via DSM reverse proxy, plain HTTP) with
bearer-token auth from a SOPS-encrypted Secret mounted into the
kube-prometheus-stack pod. Adds GarageDown / GarageScrapeFailing
alerts; richer rules to follow once the metric surface is wiki-documented.
install-garage-synology.yml: secret writes are now compare-then-write
and the force-recreate task fires on any rotated token, not just
garage.toml changes. Without this, a new metrics_token would never
reach the running Garage process.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Pin Loki datasource UID to `loki` so dashboards referencing that UID resolve
- Add `prometheus-legacy` datasource pointing at LXC 227 so host-level
dashboards (Alloy, pve host node_exporter, docker) keep working until
the legacy Prometheus retires at global cut-over
- Add `influxdb` datasource pointing at VM 108 (Stage A); the
suedstr28/renner bucket is the heatpump-roi dashboard's source.
Token from grafana-influxdb-token SOPS secret (same read-only token
the legacy Grafana role uses). Stage B flip to the in-cluster
InfluxDB is deferred to the HA migration PR.
- Enable kube-prometheus-stack's Grafana sidecar on the
`grafana_dashboard: "1"` label so per-dashboard ConfigMaps from
kubernetes/apps/monitoring/dashboards/ get auto-loaded
DASHBOARD-MIGRATION.md tracks the 27-dashboard port queue (bucketed
k8s-native / host-legacy / mixed). Empty dashboards/ kustomization is
ready to accept per-dashboard PRs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Configure Grafana to use the K8s-internal Authentik instance for
OAuth2 login. Infrastructure Admins group gets Admin role, others Viewer.
Client credentials stored in SOPS-encrypted secret.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
