Add a blackbox-http probe of kuma.renner.uno with a dedicated critical
UptimeKumaUnreachable rule (excluded from BlackboxProbeFailing), scrape
Kuma /metrics with the API key from OpenBao, and alert on sustained slow
monitor response times. No egress policy exists in the monitoring
namespace, so no CNP change is needed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The flux-image-automation Alert forwarded every info event from
ImageUpdateAutomation (i.e. each successful bump commit) to Alertmanager,
which emailed it as FluxImageUpdateAutomationSucceeded. A success is not
a problem and must not be an alert; the commits are in git history.
Image-automation failures still alert: flux-failures covers
ImageUpdateAutomation at error severity. The Alertmanager source="flux"
route stays for those; its comment now says so.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The chart's KubeJobFailed fires while a failed Job object exists, so it
clears only on Job GC (TTL, history limit, Velero's keep-latest), not on
recovery. On 2026-10-18 this held 10 alerts open for runs that had already
been superseded: renovate recovered 2026-10-13 but alerted until its 7d
TTL; registry-bot-secret-rotate-init2 succeeded 9 min after init failed,
the alert started 7 min after that and stayed 11h.
Replace it with a rule that drops a failed Job once a later Job of the
same series completes: series = owning CronJob, or velero.io/repo-name
for kopia maintenance Jobs (exposed via KSM metricLabelsAllowlist).
Failed Jobs without a series alert as before.
KubeContainerWaiting fired for the never-started main container of a pod
whose init container failed. Exclude Failed/Succeeded pods; KubeJobFailed
covers those.
Verified against live Prometheus: the new expression returns nothing now
(chart rule: 2), returns the 30 Velero maintenance failures at 16:40Z
2026-10-18, and returns nothing at the renovate/registry-bot timestamps
after their successful re-runs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
paperless, shelly-manager, zone-configurator, esphome, zigbee2mqtt,
alertmanager, prometheus, wiki, paperclip, navidrome, lidarr and wordpress
now gate through authentik-gate-sync: each source Ingress names its real
backend and carries the old provider's settings as annotations (group,
login-flow, authorization-flow where it was implicit consent, paperless'
skip-path).
Removed per gate: the ExternalName Service, the egress CNP in authentik,
the namespace CNP rule admitting the old outpost, and the blueprint
provider/application/binding/outpost (state: absent, outpost ->
application -> provider). Groups stay declared without users lists;
membership is managed in Authentik.
Per-gate specifics:
- paperless: the docs.renner.uno alias backends the generated
paperless-gate Service (its middleware redirects before the backend); the
auth.renner.uno/outpost.goauthentik.io route is removed, proxy outposts
take their callback on the app host.
- alertmanager / prometheus: source Ingresses renamed to alertmanager /
prometheus; the controller generates the *-gate route Ingresses.
- wordpress-admin: gains a tls block for www.renner.uno (wordpress-tls,
certificate still owned by the public Ingress), so the gate's external
host stays https.
- wiki, paperclip: only the gate peer/rule leaves their shared CNPs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner decision on REN-107: option B, 30 days on the existing disk.
retention 90d -> 30d, retentionSize 30GB -> 40GiB, PVC unchanged (50Gi).
At ~1 GB/day the 30GiB cap bound first and the real window was a
20.5-27.2 day saw-tooth; 30d peaks near 38 GB, under the 40GiB backstop.
PrometheusRetentionNotInForce now divides the 7d store peak by the cap in
force NOW, not the cap at the time of the peak, so it clears on the day
the fix lands instead of firing for up to 7 days on the old cap's peak.
Adds tests/prometheus-retention_test.yaml (5 scenarios) and 5
`retention-*` mutations, all caught.
Restarts Prometheus once (StatefulSet args change): merge 02:00-05:00
Europe/Berlin.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The version range >=72.0.0 let Flux upgrade the chart unattended
(91.5.2 -> 91.7.0 at 19:43 UTC on 2026-09-26, no git commit). That
restart moved Prometheus onto k8s-1 while its Ceph RBD mounts were
broken, causing a ~48 min alerting outage (REN-198).
Pinned to the version already deployed, so reconcile is a no-op.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Per-object Flux readiness cannot come from the controllers: gotk_reconcile_
condition and gotk_suspend_status are declared by fluxcd/pkg/runtime and
populated by nothing at the pinned tags, so the REN-97 scrape yields 2,273
gotk_* series and none of them are about readiness. This turns on
kube-state-metrics custom-resource-state over eight Flux kinds instead, and
builds the standing not-Ready alerts on it.
The not-Ready half is complete. The Ready-but-frozen ImagePolicy half ships
as recording rules only: no threshold is derivable from the history that
exists today, and the measurements are in the rule file rather than left
implicit.
- kube-state-metrics customResourceState + rbac.extraRules (list, watch only)
- prometheusrule-flux.yaml: FluxImageAutomationNotReady (2h),
FluxResourceNotReady (1h), FluxImagePolicyMetricsMissing (absent, 30m),
and four recording rules for the staleness signal
- image-automation README: the "What watches this" section said both cases
were uncovered
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Our hand-written Alertmanager config replaced the chart's default and
dropped two pieces kube-prometheus relies on:
- a route sending InfoInhibitor to the null receiver
- the inhibit rules that let InfoInhibitor (and warning/critical)
suppress severity=info alerts
So both the helper alert and the info alerts it exists to silence were
emailed, daily. InfoInhibitor fired over the last 7 days in crowdsec,
kube-system, woodpecker-ci and without a namespace, driven by
CPUThrottlingHigh (crowdsec-agent, whereabouts-reconciler),
KubeletTooManyPods and KubeQuota* in woodpecker-ci.
Route and the three inhibit rules are copied from kube-prometheus
manifests/alertmanager-secret.yaml. Info alerts now reach email only
while a warning or critical alert fires in the same namespace, which is
the upstream design. amtool check-config passes; amtool config routes
test sends InfoInhibitor to null.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- kubernetes/apps/gitea: every object in namespace gitea; applied by a
new gitea Flux Kustomization (clusters/homelab/gitea.yaml) instead of
apps. gitea-allow-metrics-scrape dropped: the gitea-ns baseline
already admits monitoring.
- Rotators (flux-git, review-bot) and the Prometheus gitea job use
gitea.gitea.svc.
- bootstrap.sh waits for the gitea KS.
Data: data-gitea-0 is not copied. The PV (reclaimPolicy Retain) is
rebound to a PVC gitea/data-gitea-0 pre-created with volumeName, which
the new StatefulSet adopts. gitea-tls is pre-issued in the gitea
namespace so HTTPS has no certificate gap.
Refs #878
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Chasing a stale comment turned up a live duplicate. Since serviceMonitorSelector
was loosened to {}, Prometheus discovers the goauthentik.io operator's own
ServiceMonitors — which is why it was loosened. The hand-written copies for ldap
and paperless-gate, added back when the strict selector was dropping the
operator's, now scrape the same endpoints a second time.
Measured, not assumed: 10.244.4.113:9300 and 10.244.4.76:9300 were each
producing 40 samples per 10 minutes against 20 for every other outpost, and
ldap picked up a third target because the hand-written selector matched the
plain Service as well as the -metrics one. No duplicate-timestamp errors,
because the two pools scrape at different offsets — the cost is doubled
ingestion and a target list that lies about how many outposts there are.
Radius stays, and the comment now says why it is not the same case: the
operator's ServiceMonitor for it selects on managed-by=goauthentik.io together
with service-type=metrics, and no Service carries both — the operator's has the
first, the one defined here has the second. It is the only thing scraping
radius, so deleting it would have been a silent loss.
Also corrects the comments that claimed the release label is required for
discovery. It is not, for ServiceMonitors and PodMonitors: those two selectors
are empty. It very much is for Probes, PrometheusRules and ScrapeConfigs, whose
selectors still match on it, so probes-blackbox.yaml now says so explicitly —
otherwise the corrected comments elsewhere read as license to drop the label
there too.
Live selector state, for the record:
serviceMonitorSelector {} podMonitorSelector {}
probeSelector / ruleSelector / scrapeConfigSelector release=kube-prometheus-stack
No Proxmox remains. jupiter, saturn and neptune all run Talos on bare metal,
so everything that scraped, proxied, alerted on or rotated credentials for the
PVE API now points at hosts that answer nothing on :8006.
Two of these were not merely dead but actively failing on a schedule:
gatus an Infrastructure check on https://pve.renner.uno, every
60s, which can now only ever fail
openbao rotation a Class-B CronJob rolling the pve-exporter@pve API token
against pve.apps.svc.cluster.local:8006
Removed:
apps/pve/ Service, EndpointSlice (10.2.40.201-203),
IngressRoute for the PVE web UIs
monitoring/pve-exporter-* Deployment and its config ExternalSecret
prometheus scrape configs job_name pve and pve-node
dashboards/proxmox-cluster.json and its ConfigMap generator entry
image-automation/pve-exporter image policy for an image nothing runs
openbao/rotation/pve-exporter the rotation CronJob
openbao/policies/rotator-proxmox.hcl
crowdsec pve-ui-polling-whitelist, which suppressed false bans on PVE UI
API polling that no longer happens
Kept, with comments corrected: the insecure-backend ServersTransport, still
used by apps/file for the NAS web UI.
The gatus check leaves a numbering gap at "Infrastructure 2" rather than
renumbering 3 and 4. The labels are deliberately opaque and renumbering would
relabel the surviving checks, discarding their history for cosmetics.
Not touched: the opentofu Proxmox provider, VM and LXC modules,
vm_152_k8s_3.tf, sync_state_from_pve.py, and the ansible PVE inventory. Those
carry state implications and want a deliberate pass rather than folding into a
manifest cleanup.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Prometheus held 298,236 active series. ~108,000 of them -- over a third -- were
apiserver/etcd `_bucket` histograms. Each fans out to one series per bucket x
verb x resource x scope, so a single metric was 23,810 series on its own. Every
Home Assistant metric combined is ~4,100 for comparison.
That cardinality is the dominant driver of Prometheus write volume (~11 GB/day of
TSDB + WAL, doubled again by Ceph replication) and none of it is observable value
on three nodes: per-bucket apiserver latency quantiles are for clusters with SLAs
and dozens of nodes.
Dropped, all verified to have ZERO rule dependencies by checking each metric
against the live alert and recording rules:
apiserver_request_duration_seconds_bucket rules=0 23,810 series
apiserver_request_body_size_bytes_bucket rules=0 23,200
etcd_request_duration_seconds_bucket rules=0 15,840
apiserver_watch_list_duration_seconds_bucket rules=0 11,538
apiserver_watch_cache_read_wait_seconds_bucket rules=0 8,232
apiserver_response_sizes_bucket rules=0 7,592
apiserver_watch_events_sizes_bucket rules=0 5,220
-------
95,432
KEPT deliberately: apiserver_request_sli_duration_seconds_bucket (12,144 series).
23 recording rules depend on it, including the apiserver availability SLO and the
KubeAPIErrorBudgetBurn burn-rate series. Dropping it would silently break that
alerting -- the manifest says so inline, so a later "tidy up" of the regex does
not remove it by accident.
Only `_bucket` series are dropped. The matching `_count` and `_sum` remain, so
request rates and mean latencies still work; what is lost is per-bucket
quantiles.
etcd_* is included even though kubeEtcd is disabled: those are the apiserver's
own etcd-client metrics on the apiserver endpoint, not a separate etcd scrape.
Expected: ~32% fewer active series, so roughly 11 -> 7 GB/day of Prometheus
writes, ~8 GB/day off the drives once replication is counted. Will verify against
prometheus_tsdb_head_series rather than assume.
Refs #434
Co-Authored-By: Claude Bot <noreply@anthropic.com>
There are zero ceph_* series in Prometheus, and there have been for all 79
days of the cluster's life:
count of ceph_ metrics in Prometheus: 0
rook-ceph active scrape targets: 0
Not a Rook misconfiguration — `monitoring.enabled: true` is set and both
ServiceMonitors exist. They are simply never selected:
Prometheus serviceMonitorSelector : matchLabels {release: kube-prometheus-stack}
rook-ceph-mgr / rook-ceph-exporter: labels {team: rook}
The Rook operator creates those ServiceMonitors at runtime with its own
hardcoded label, so no static manifest or Kustomize patch in this repo can
relabel them.
Cluster-wide this was dropping 12 of 30 ServiceMonitors:
rook-ceph/rook-ceph-mgr, rook-ceph/rook-ceph-exporter (team: rook)
authentik/ak-outpost-* x10 (goauthentik.io/...)
Consequences: the entire storage layer had no telemetry and no history — so
Ceph trend analysis is impossible for anything before this lands, and any Ceph
PrometheusRule added today would have silently had nothing to fire on.
This is the ServiceMonitor twin of the trap already documented for rules in
monitoring-rules/prometheusrule-velero.yaml ("without this label the rule is
created but never loaded into Prometheus").
ruleSelector is deliberately NOT loosened — this repo's PrometheusRules all
carry the release label on purpose, and keeping that selector strict stops a
stray third-party rule from firing here.
Blast radius: 12 new scrape targets. The 10 authentik outposts come along with
the 2 rook ones; they are legitimate targets, but they are a side effect of
this change rather than its goal.
Found while investigating the Ceph BLUESTORE_SLOW_OP_ALERT warning, which
itself turned out to be a benign self-limiting event.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Step 2 of the InfluxDB seed->apps handoff. #388 already landed
prune: disabled on the live HelmRelease/HelmRepository/ExternalSecret, so
retiring seed-influxdb here cannot trigger a helm uninstall — GC skips the
annotated objects and the new influxdb KS adopts them in place.
- git mv kubernetes/seed/influxdb/ -> kubernetes/apps/influxdb/ (annotations
from #388 travel with the files).
- Add kubernetes/clusters/homelab/influxdb.yaml (KS 'influxdb', path
apps/influxdb, deps infrastructure/crds/ceph). Delete seed-influxdb.yaml.
Review fixes from the closed atomic PR (#387):
- bootstrap.sh: wait on 'influxdb' instead of the deleted 'seed-influxdb'
(would otherwise spin the full 600s timeout on cold bootstrap).
- e2e-test.sh: swap the seed-influxdb readiness check for 'influxdb'.
- Fix stale comment paths kubernetes/seed/influxdb/... -> apps/influxdb/... in
network-policies/home-assistant-ns.yaml and apps/monitoring/
helmrelease-prometheus.yaml.
- Trim the helmrelease.yaml comment that described the removed seed-job.yaml,
and the ha-token-job.yaml 'at cutover moves to apps/influxdb' note (this is
that cutover).
Left intentionally: externalsecret.yaml key 'seed/influxdb/bootstrap' is an
OpenBao KV path, not a repo path — moving it needs a vault-side secret
migration, out of scope here.
Verified: kubectl kustomize apps/influxdb builds all objects with the right
prune/force annotations; bash -n passes on bootstrap.sh + e2e-test.sh; no
remaining seed-influxdb references in the repo (only the OpenBao KV path).
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
*.k.renner.uno was a temporary migration namespace and is banned (owner
directive 2026-07-14, documented in openbao/ingress.yaml) — it was never
the live naming. Removes the .k. hosts/SANs from ical-proxy, loki,
grafana and the mosquitto cert, renames syslog.k.renner.uno to
syslog.renner.uno (cert + external-dns annotation; any syslog sender
still configured with the k. name must be repointed), fixes the stale
comments, updates e2e-test.sh probes to the real names, and drops
k.renner.uno from the external-dns domainFilters so leftover k. records
get garbage-collected by policy: sync.
The migration script's historical docstring reference stays — it
documents what the migration did.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S5CLfizEtzHS51kxJFQgc
No hardcoded IP remains in the pve scrape config: the node job now
targets jupiter/neptune/saturn.renner.uno. instance label reads the
hostname, and a renumbered node follows DNS without a config edit.
Verified: exporter returns node-local metrics when targeted at
jupiter.renner.uno from inside the cluster.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
The pve cluster-collector job hardcoded 10.2.40.201, so cluster-wide
metrics (pve_up, guest info, storage) paused whenever jupiter
specifically was down, even with the cluster healthy. Point it at the
pve Service DNS (pve.apps.svc.cluster.local) whose EndpointSlice
load-balances the PVE API across all three nodes, so the scrape
follows whichever node is up. Node-local job keeps the three explicit
targets — it needs per-node identity.
Verified: exporter returns pve_up for cluster + all nodes when
targeted at the Service DNS.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
The single pve scrape job hit all three node IPs with cluster=1, and
pve-exporter returns the full cluster-wide resource set regardless of
which node answers - every node/guest series landed in the TSDB three
times (same id, differing instance). Dashboards papered over it with
max by (id); the cardinality cost stayed.
Split per upstream prometheus-pve-exporter guidance: cluster collectors
scrape one node only, node-local collectors keep scraping all three
under a new pve-node job. No dashboard queries filter on job=, and all
already aggregate by id, so this is query-compatible.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner directive: PVE runs ONLY the three k8s VMs. The proxmox-dns trio
(lxc 118/218/318), rustfs trio (lxc 800/801/802), fortress (vm 102)
and paperless (vm 111) are decommissioned; guest definitions are being
destroyed on PVE out-of-band.
Removed:
- opentofu: lxc_118_proxmox_dns.tf, vm_102_fortress.tf,
vm_111_paperless.tf, backup_jobs.tf (the vzdump job covered ONLY
destroyed guests — next tofu apply removes the job from PVE; k8s
data protection is Velero/CNPG)
- ansible: roles/proxmox-dns, install-proxmox-dns{,-ha}.yml,
check-chaos-testing.yml (tested failover of decommissioned DNS-HA
and postgres LXCs), proxmox_dns_vault.yml, rustfs_vault.yml,
[proxmox_dns_ha] inventory group
- monitoring: dns-servers scrape job in both the k8s HelmRelease and
the legacy prometheus-server template — resolves TargetDown
(dns-servers)
Closes#259
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
Gitea moved in-cluster (apps ns StatefulSet); LXC 114 is stopped, so the static
`gitea` scrape target gitea.srv.renner.uno:3000 is down -> firing TargetDown.
Point it at the ClusterIP Service (gitea.apps.svc.cluster.local:3000), which
serves /metrics anonymously (verified HTTP 200 with gitea_* metrics).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HQhvu1mrQ7EgwLhrzdNyY7
Prometheus (prometheus.renner.uno, monitoring ns, :9090) had no auth of its
own. Its admin API is disabled (read-only), but the full 90d metrics DB is
an infra-recon oracle — every hostname, IP and service in the cluster
(2026-07-10 audit). Gate it the same way as alertmanager-gate (#146).
- Blueprint: prometheus-gate proxy provider (internal_host
http://kube-prometheus-stack-prometheus.monitoring.svc:9090, external_host
https://prometheus.renner.uno), application, outpost. REUSES the existing
git-owned "Monitoring Access" group — created generically in #146 for
exactly this, so no new group.
- implicit-consent flow (same rationale as alertmanager/paperless: a
frequently-checked monitoring UI; an authenticated session already closes
the exposure).
- No skip_path_regex: the Prometheus UI is a plain React SPA — verified its
index.html has no serviceWorker/manifest/webmanifest.
- Helm-generated Prometheus Ingress disabled (prometheus.ingress.enabled=
false); standalone prometheus-gate-ingress.yaml backends the outpost
ExternalName. Mirrors the alertmanager gate.
- Scoped CNPs: monitoring ingress rule allowing only the outpost -> :9090;
matching egress in the authentik namespace.
Grafana is UNAFFECTED: its Prometheus datasource targets the ClusterIP
Service (kube-prometheus-stack-prometheus.monitoring:9090), verified in the
live datasource provisioning, not the ingress hostname.
Known side effect, documented not silently shipped: three blackbox
`http_2xx` uptime probes target https://prometheus.renner.uno
(kubernetes/apps/monitoring-rules/probes-blackbox.yaml, the srv-host
ansible prometheus.yml.j2, and the ansible blackbox-exporter role). After
gating they follow the 302 to the Authentik login (200), so they stay GREEN
— but now measure Authentik reachability, not Prometheus. This is already
true for alerts.renner.uno since #146 (verified: probe_success=1,
final_status=200 via redirect-follow). No probe goes red; the monitoring
just becomes a gate-liveness check. Repointing those probes to an internal
endpoint is a separate follow-up, not done here.
Verified: kubectl kustomize builds for authentik, monitoring,
network-policies; embedded outpost survived the blueprint insert; rendered
ingress backends ak-outpost-prometheus-gate:9000; both prometheus and
alertmanager helm ingresses now disabled; egress + CNP scoped to
prometheus:9090.
Post-merge checks: only the gate Ingress serves prometheus.renner.uno (helm
one gone); unauthenticated GET -> 302 to auth.renner.uno; unauthenticated
GET /api/v1/query -> 302; outpost forwards to :9090 after login; Grafana
dashboards still render (ClusterIP datasource intact).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Alertmanager (alerts.renner.uno, monitoring ns, :9093) had no auth of its
own, and its v2 API is writable: an unauthenticated LAN client could POST
/api/v2/silences and suppress ALL alerting — a tamper capability, not just
disclosure (2026-07-10 audit, highest-teeth of the deferred unauth
findings).
Gate it with the same Authentik proxy-outpost pattern as
paperless/vscode/shelly-manager:
- New proxy provider + application + dedicated outpost
(ak-outpost-alertmanager-gate), mode proxy, forwarding to the ClusterIP
Service (kube-prometheus-stack-alertmanager.monitoring.svc:9093).
- New git-owned "Monitoring Access" group (k.renner) bound to the app.
Named generically so it can also gate the still-deferred Prometheus UI.
- The Helm-chart-generated Alertmanager Ingress is disabled
(alertmanager.ingress.enabled=false); a standalone gate Ingress
(alertmanager-gate-ingress.yaml) backends the outpost ExternalName —
two Ingresses for the same host would conflict. Mirrors loki-ingress.
- Scoped CNPs: monitoring ingress rule allowing only the outpost -> :9093;
matching egress rule in the authentik namespace.
implicit-consent flow, NOT passkey step-up: Alertmanager is a
frequently-checked monitoring UI, and requiring an authenticated Authentik
session already closes the unauthenticated-LAN exposure. Passkey re-auth on
every visit is disproportionate friction for a dashboard (paperless-gate
uses implicit-consent for the same reason). Bump to
passkey-stepup-authorization if the write API warrants it.
No skip_path_regex: the Alertmanager UI is a plain Elm SPA with no service
worker and no manifest (verified 2026-07-10 — index.html has no
serviceWorker/manifest/webmanifest), so none of the credentials:'omit' PWA
fetches that forced skip carve-outs elsewhere.
Prometheus's alerting path is UNAFFECTED: it posts to the ClusterIP Service
(verified: spec.alerting.alertmanagers targets
monitoring/kube-prometheus-stack-alertmanager:http-web), not this ingress.
Prometheus's own ingress is untouched (still enabled) — its unauth exposure
is a separate deferred finding, and it is read-only.
Verified: kubectl kustomize builds for authentik, monitoring and
network-policies; the embedded outpost survived the blueprint insert; the
rendered ingress backends ak-outpost-alertmanager-gate:9000, egress is
scoped to alertmanager:9093, and alertmanager.ingress.enabled=false while
prometheus.ingress.enabled=true.
Post-merge checks to run: only the gate Ingress exists for alerts.renner.uno
(the Helm one is gone); unauthenticated GET -> 302 to auth.renner.uno;
POST /api/v2/silences unauthenticated is refused; the outpost forwards to
:9093 after login; Prometheus is still delivering alerts to Alertmanager.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Finish line of the backup migration — nas-1's Garage is no longer a
cluster-backup dependency, so:
- drop the prometheus `garage` scrape job (s3.nas-1.renner.uno:39033) +
the garage-metrics-token secret (now orphaned)
- delete prometheusrule-garage.yaml (GarageDown / GarageScrapeFailing) —
it would fire every night once nas-1 powers off
Replace it with outage visibility for the new offsite target:
- add https://fsn1.your-objectstorage.com to the blackbox-http probe
(bare endpoint returns 200)
- new HetznerBackupEndpointDown alert on probe_success == 0 for 10m
Velero outages are already covered by VeleroBackupValidationFailure /
VeleroBackupStale. A granular CNPG "WAL archiving behind" alert needs
CNPG PodMonitors (enablePodMonitor=false today, metrics not scraped) —
left as a follow-up rather than shipping an alert on an absent metric.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
gatus (status), ical-proxy, kube-prometheus-stack (alerts/dash/prometheus)
have no nginx-only annotations, so the migration is pure
ingressClassName: nginx → traefik plus apps/allow-from-ingress now
permitting the traefik label as a source. cloudflared retargets
status.renner.uno (the only WAN-exposed name in this batch); the rest
are LAN-only via external-dns-unifi.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous commit added a second `grafana.sidecar:` block at the
end of the grafana values, which collided with the existing one that
already configured `sidecar.dashboards`. YAML duplicate-key resolution
silently dropped one of them; the rendered Deployment showed no
resources on either sidecar.
Move the resources block inside the original sidecar map so both
keys coexist.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both had no resources block set. Sized to observed steady-state load
plus headroom:
- grafana main: 100m/768Mi req, 1Gi limit (was 28m/498Mi live).
- grafana sidecars (sc-dashboard, sc-datasources): 10m/128Mi req,
256Mi limit.
- kube-state-metrics: 50m/64Mi req, 256Mi limit (was 4m/28Mi live;
ksm scales with cluster object count, so generous headroom).
Note on ksm: kube-prometheus-stack's top-level `kubeStateMetrics`
section only toggles enable/disable. Resources must be passed via
the subchart name `kube-state-metrics:` (with hyphens).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
VM 108 (the legacy InfluxDB at influxdb.srv.renner.uno:10.2.40.16) is
unreachable — no ICMP, no SSH, no port 8086. K8s Grafana's InfluxDB
datasource has been silently broken since the in-cluster InfluxDB
came online (commit 1daaa6f, 2026-05-05). The heatpump-roi dashboard
and any historical HA panel that queries InfluxDB are dead until this
repoint lands.
Datasource URL: http://influxdb-influxdb2.influxdb.svc.cluster.local:8086.
The legacy SOPS-stored token (`grafana-influxdb-token.sops.yaml`) is
useless against the in-cluster bolt store. Replaced with a seed Job
that mirrors the existing `influxdb-ha-token` pattern: ServiceAccount
in influxdb ns + cross-namespace Role on the `grafana-influxdb-token`
Secret in monitoring ns + Job that creates a read-only auth on the
suedstr28 bucket and writes the token to that Secret. The key name
inside the Secret stays `INFLUXDB_TOKEN` so the existing
`envFromSecrets` reference in the helmrelease keeps working.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 2a of the dashboard triage. Wires three metric sources whose
dashboards previously had no data path:
- alloy-servicemonitor.yaml: scrapes the alloy DaemonSet's
http-metrics:12345 endpoint. Feeds alloy-overview.
- immich-servicemonitor.yaml: scrapes immich-server's metrics:8081
endpoint (already exposed by the existing Service). Feeds
immich-overview.
- helmrelease-prometheus.yaml: adds a static scrape for
gitea.srv.renner.uno:3000/metrics (anonymous, no auth token
required). Feeds gitea-overview.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ingress-nginx + letsencrypt-cluster cert at prometheus.renner.uno
and alerts.renner.uno. No external-dns/cloudflare label, so only
external-dns-unifi publishes the hostnames — they resolve on LAN
and NXDOMAIN from the public internet.
externalUrl set on both so the click-through links embedded in
Alertmanager emails resolve to the external hostname instead of the
in-cluster service name.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ingress-nginx + letsencrypt-cluster cert at prometheus.k.renner.uno
and alerts.k.renner.uno. The .k.renner.uno zone is published by
external-dns-unifi only, so neither hostname is reachable from the
public internet.
externalUrl set on both so the click-through links embedded in
Alertmanager emails resolve to the external hostname instead of the
in-cluster service name.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The kube-prom-stack default alert fires whenever the cluster can't
tolerate the failure of one node. On a small homelab with no spare
node headroom this is structurally unsatisfiable, not actionable.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pointing unpoller v2.39.0 at the UDM caused the controller to crash.
Removing the deployment, secret, image-automation policy, and the
helmrelease pointer entirely. UniFi metrics stay offline until a
known-good poller/UDM combination is identified.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous deploys (Ansible roles install-prometheus.yml against
hosts: prometheus.srv.renner.uno on LXC 227) died with that host's
retirement. Re-deploy both as K8s Deployments in the monitoring
namespace, reusing the existing Ansible-vault credentials for the
Proxmox root@pam!ansible token and the UniFi `Poller` read-only user
(now stored in SOPS-encrypted Secrets per the K8s convention).
pve-exporter (prompve/prometheus-pve-exporter:3.8.3) listens on 9221
and is queried as a blackbox-style exporter: the existing pve scrape
config keeps the three 10.2.40.20x targets and uses relabel_configs
to direct the actual HTTP call to pve-exporter.monitoring.svc.
unpoller (ghcr.io/unpoller/unpoller:v2.1.5) polls https://udm.renner.uno
and exposes :9130. Scraped via a ServiceMonitor; the dead static
unifi scrape config is removed from the helmrelease.
Both images are enrolled in image-automation: pve-exporter on semver
3.x with plain 3-component tags; unpoller on semver 2.x with v-prefix
extract that rejects -rc and arch-suffixed variants.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The in-cluster ServiceMonitor was missing the `release: kube-prometheus-stack`
selector label, so the Prometheus resource never discovered it (same trap
the garage PrometheusRule documents). Adding the label brings the scrape
online via the existing in-cluster Service + bearer-token Secret.
Also removes the obsolete `homeassistant` additionalScrapeConfig that
pointed at the external IoT-VLAN IP with an empty bearer_token TODO —
HA lives in the cluster now, the external path is dead.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Watchdog is an always-firing alert designed to feed an external
dead-man's-switch (PagerDuty DMS, Healthchecks.io, etc.); we don't
run one, and a 12h-repeat email of a continually-firing alert is
pure noise. Route alertname="Watchdog" to a null receiver.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Alertmanager (kube-prometheus-stack) now sends to admin@renner.uno via
Proton SMTP submission (smtp.protonmail.ch:587 STARTTLS) using a per-
account token in the alertmanager-smtp SOPS-encrypted Secret.
A new infrastructure subtree, flux-notifications/, adds a Provider of
type alertmanager pointing at the in-cluster Alertmanager and two
Alert CRs: one info-severity for ImageUpdateAutomation bumps (with the
30m "no change" heartbeat excluded), and one error-severity catch-all
across every Flux source/sync kind. Flux events and Prometheus alerts
now share a single email funnel.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Authentik DB carried over a `Local Docker connection`
ServiceConnection from the Proxmox deployment, and all four outposts
(Caddy proxy, LDAP, Local Radius, embedded) still pointed at it. The
worker's `outpost_service_connection_monitor` task crashed every cycle
in-pod with `DockerException: ('Connection aborted.',
FileNotFoundError(...))` because the pod has no /var/run/docker.sock.
Ship a Blueprint via `blueprints.configMaps` that reassigns every
Outpost to the auto-created `Local Kubernetes Cluster` connection and
deletes the Docker connection. Authentik's chart-rendered Role already
grants the deploy/svc/ingress/secret/configmap perms needed to spawn
outposts via the K8s API.
While here:
- Grafana OIDC token_url/api_url were still hitting
`http://authentik-server.authentik.svc.cluster.local`, violating the
public-hostname-everywhere rule (split iss claim). Point them at
https://auth.renner.uno/ alongside auth_url.
- seed-authentik comment still described the dropped pg_dump→pg_restore
Job; replace with the current initdb + Garage CNPG recovery model.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a global cluster-settings ConfigMap (CLUSTER_IN_DEVELOPMENT,
ACME_SERVER) consumed via Flux postBuild substitution on the
infrastructure-crds Kustomization. A new letsencrypt-cluster
ClusterIssuer reads ACME_SERVER from the ConfigMap at reconcile time;
every Certificate / Ingress in the repo now references letsencrypt-
cluster instead of letsencrypt-{prod,staging} directly.
While CLUSTER_IN_DEVELOPMENT=true, ACME_SERVER points at acme-staging-
v02, so the only certs the cluster can issue are LE staging — sidesteps
the recurring 5-cert/168h prod rate-limit hit during back-to-back
rebuilds (latest occurrence: 2026-05-05 alloy-syslog + loki-ingress).
Flip both keys together when leaving development.
Per-app KSes intentionally do NOT enable postBuild — Grafana dashboards
and HA's deployment shell scripts contain unrelated \${VAR} markers
that would be clobbered. Substitution stays scoped to post-crds/ which
has no other variable expansions.
The standalone letsencrypt-prod / letsencrypt-staging ClusterIssuers
remain defined as escape hatches but are no longer referenced.
e2e-test.sh now also asserts letsencrypt-cluster Ready.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Prometheus now scrapes Garage's admin endpoint on nas-1
(s3.nas-1.renner.uno:39033 via DSM reverse proxy, plain HTTP) with
bearer-token auth from a SOPS-encrypted Secret mounted into the
kube-prometheus-stack pod. Adds GarageDown / GarageScrapeFailing
alerts; richer rules to follow once the metric surface is wiki-documented.
install-garage-synology.yml: secret writes are now compare-then-write
and the force-recreate task fires on any rotated token, not just
garage.toml changes. Without this, a new metrics_token would never
reach the running Garage process.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SQLite on Ceph RBD does per-row fsync, each round-tripping to OSD,
making cold-bootstrap schema migrations exceed the 30s liveness probe
budget and flooding Grafana into CrashLoopBackOff. Postgres batches WAL
fsyncs server-side; migrations finish in seconds.
- New `grafana-db` Kustomization (apps/grafana-db) with a CNPG Cluster
(gf-pg, single instance, 5Gi). No Barman backup yet — Grafana data
is mostly reproducible from git (dashboards, datasources). Pattern
matches authentik-db / paperless-db / immich-db.
- New top-level `monitoring` Kustomization with `dependsOn: grafana-db`
and a Cluster healthCheckExpr that gates on `readyInstances >= 1`,
so the HelmRelease only fires after gf-pg-app secret exists.
- Removed `monitoring/` from the apps umbrella so it's gated independently.
- HelmRelease values now point Grafana at gf-pg-rw.monitoring.svc with
user/password sourced from the CNPG-managed gf-pg-app secret.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
kube-prometheus-stack default RollingUpdate deadlocks on Multi-Attach
when Grafana is a single replica on RWO ceph-block — new pod can't
mount the volume while the old pod is still Running. Use Recreate so
the old pod terminates first.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Pin Loki datasource UID to `loki` so dashboards referencing that UID resolve
- Add `prometheus-legacy` datasource pointing at LXC 227 so host-level
dashboards (Alloy, pve host node_exporter, docker) keep working until
the legacy Prometheus retires at global cut-over
- Add `influxdb` datasource pointing at VM 108 (Stage A); the
suedstr28/renner bucket is the heatpump-roi dashboard's source.
Token from grafana-influxdb-token SOPS secret (same read-only token
the legacy Grafana role uses). Stage B flip to the in-cluster
InfluxDB is deferred to the HA migration PR.
- Enable kube-prometheus-stack's Grafana sidecar on the
`grafana_dashboard: "1"` label so per-dashboard ConfigMaps from
kubernetes/apps/monitoring/dashboards/ get auto-loaded
DASHBOARD-MIGRATION.md tracks the 27-dashboard port queue (bucketed
k8s-native / host-legacy / mixed). Empty dashboards/ kustomization is
ready to accept per-dashboard PRs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Enable Velero ServiceMonitor and node-agent PodMonitor with
`release: kube-prometheus-stack` labels so kube-prom-stack's
selector picks them up, and import community dashboard 11055 via
Grafana's gnetId download path so backup inventory/status is
visible at dash.renner.uno.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds <app>.renner.uno as a second host rule and TLS SAN on each test
Ingress (authentik, paperless/docs, audiobookshelf, ical-proxy, grafana).
With the external-dns domain filter now scoped to k.renner.uno, these
prod hostnames cannot be written back into UniFi by external-dns — the
prod CNAMEs managed by OpenTofu stay pointed at the Caddy VIP, while
the K8s ingress now accepts traffic arriving with a prod Host header
(e.g. from a Kasm session whose extra_hosts maps *.renner.uno to the
cluster LB 10.2.40.80).
Also removes *.k.renner.uno from app-level config where it was baked
in as cruft: PAPERLESS_URL, Grafana root_url, Grafana OIDC auth_url
are now prod hostnames — matching the architectural intent that the
K8s cluster IS the future prod. PAPERLESS_CSRF_TRUSTED_ORIGINS keeps
both variants during transition so existing *.k.renner.uno browsing
doesn't immediately break on form POSTs.
cert-manager will issue fresh letsencrypt-staging certs with the
expanded SAN set via DNS-01; no public DNS changes needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Flip all ingress cert-manager.io/cluster-issuer annotations from
letsencrypt-prod to letsencrypt-staging. Five rebuilds in a week
tripped LE's per-identifier rate limit (5 certs / 168h) and left 3
hostnames with the ingress-nginx fake cert. Staging has ~30k/week,
so rebuilds stop being rate-limited. ClusterIssuer definitions for
both remain in post-crds; flip back per-ingress when ready for prod.
- Rework e2e-test.sh TLS check to validate the cert *issuer* contains
"Let's Encrypt" rather than chain-validating against the public CA
bundle. Same check works for both staging and prod, and still
catches the real failure mode (fake cert = cert-manager broken).
- HTTP checks now use -k since staging certs aren't publicly trusted;
the separate TLS check is what asserts cert-manager worked.
- Chain the e2e step onto .woodpecker/rebuild-k8s.yml so every rebuild
validates the resulting cluster end-to-end before reporting success.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
auth_url stays external (browser redirect), but token_url and api_url
use the cluster-internal authentik-server service for server-to-server
OAuth calls from the monitoring namespace.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The chart template sets rollingUpdate params that conflict with Recreate.
RollingUpdate works fine with replicas=1 since the PVC detach race
was a one-time issue during the initial config change.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
