fix(ci): dispatch security-scan by cron name and deploy target

Two separate bugs in one `when` block, only one of which is #582.

The cron gate has never matched. It asked for `PIPELINE == "security-scan"` on
`event: cron`, and crons cannot carry variables — `repo cron add` takes name,
branch, schedule and enabled, and nothing else. So the variable is undefined on
every cron run and the workflow is filtered out. The `daily-security-scan` cron
does fire, and pipeline 6877 shows what actually ran under it: the `e2e`
workflow alone, which opts into that cron by name. The Trivy scan and the
security-fix agent have not been running on schedule. This predates 3.18.0 and
would have survived any fix to it.

The manual gate is the #582 case. Woodpecker 3.18 stopped merging manual-run
variables into the environment the workflow-root filter evaluates against
(builder.go:102-111 builds it from metadata plus global envs; AdditionalEnvs is
passed to the compiler at line 223 but never into this map), so
`PIPELINE == "security-scan"` cannot be true there either.

Both now use selectors the root filter can actually see:

  cron       -> matched by the cron's NAME, the way alert-monitor and e2e
                already do it. First-class filter, no variable involved.
  deployment -> CI_PIPELINE_DEPLOY_TARGET, which comes from pipeline metadata
                (metadata/environment.go:74) and therefore reaches the root
                filter on 3.18.

Dispatch becomes `woodpecker-cli pipeline deploy k.renner/homelab <n>
security-scan` — a named deployment target rather than a magic variable that
exists only to pick which of fourteen files is allowed to run.

Pilot for the remaining thirteen. Nothing else is converted until this one is
confirmed working end to end.

.woodpecker/security-scan.yml

@@ -1,8 +1,20 @@
+# Trivy scan of the IaC tree and every image referenced from ansible/, with an
+# agent pass that opens a PR for CRITICAL/HIGH findings.
+#
+# Triggers:
+#   - cron       : the named `daily-security-scan` cron, matched by NAME.
+#                  Crons cannot carry variables — `woodpecker-cli repo cron add`
+#                  has no --var — so a variable gate can never match a cron.
+#   - deployment : woodpecker-cli pipeline deploy k.renner/homelab <n> security-scan
+#                  The target rides in pipeline metadata as
+#                  CI_PIPELINE_DEPLOY_TARGET, which the workflow-root filter
+#                  does see, unlike manual-run variables on Woodpecker 3.18
+#                  (#582).
 when:
   - event: cron
-    evaluate: 'PIPELINE == "security-scan"'
-  - event: manual
-    evaluate: 'PIPELINE == "security-scan"'
+    cron: daily-security-scan
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "security-scan"'
 
 steps:
   - name: iac-scan

kubernetes/apps/authentik/blueprints-configmap.yaml

@@ -25,20 +25,13 @@ data:
         attrs:
           local: true
 
-      # Retired OIDC layer. Paperless's URL-level access is gated by the
-      # paperless-gate ProxyProvider (below); user identity is forwarded
-      # from the gate to paperless via the X-authentik-* request headers
-      # and consumed by paperless via PAPERLESS_ENABLE_HTTP_REMOTE_USER
-      # (see kubernetes/apps/paperless/paperless.yaml). One Authentik
-      # Application, one Provider — the dual-OAuth2-flow setup with a
-      # separate paperless OIDC client + django-allauth callback was
-      # structurally fine but added a second consent screen on first
-      # login for no real defense-in-depth gain.
+      # Retired OIDC layer. Access is gated by the paperless-gate
+      # ProxyProvider below, which forwards identity as X-authentik-* headers
+      # consumed via PAPERLESS_ENABLE_HTTP_REMOTE_USER.
       #
-      # Order matters: delete the Application BEFORE the OAuth2Provider
-      # because Application.provider is on_delete=PROTECT, so the
-      # OAuth2Provider can't be removed while the Application still
-      # references it.
+      # ORDER MATTERS: delete the Application BEFORE the OAuth2Provider.
+      # Application.provider is on_delete=PROTECT, so the provider cannot be
+      # removed while an Application still references it.
       - model: authentik_core.application
         identifiers:
           slug: paperless
@@ -49,22 +42,15 @@ data:
           name: Paperless
         state: absent
 
-      # Also clean up the very-early orphan `paperless` ProxyProvider
-      # left behind from this session's first attempts. It's not bound
-      # to anything any more, just leftover state in the DB.
+      # Orphan ProxyProvider, unbound — leftover DB state.
       - model: authentik_providers_proxy.proxyprovider
         identifiers:
           name: paperless
         state: absent
 
-      # Dedicated forward-auth gate for paperless. This is a SEPARATE
-      # outpost (its own Deployment + Service `ak-outpost-paperless-
-      # gate`) running in front of paperless so every request is
-      # authenticated and authorized by Authentik before it reaches
-      # paperless's own attack surface. The gate sits on top of, and
-      # separate from, the OIDC OAuth2Provider above (which is what
-      # paperless's django-allauth uses for SSO once a request makes
-      # it through the gate).
+      # Dedicated forward-auth gate for paperless: its own outpost
+      # Deployment + Service, so every request is authenticated before it
+      # reaches paperless.
       - id: paperless-gate-provider
         model: authentik_providers_proxy.proxyprovider
         identifiers:
@@ -73,16 +59,13 @@ data:
         attrs:
           authorization_flow: !Find [authentik_flows.flow, [slug, default-provider-authorization-implicit-consent]]
           invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
-          # Proxy (not forward_domain) so the outpost itself terminates
-          # the request and forwards to paperless — moving the trust
-          # boundary off ingress-nginx and onto paperless-gate. ingress-
-          # nginx now only routes paperless.renner.uno to the outpost
-          # service; it never talks to paperless directly. The outpost
-          # is what populates X-Authentik-Username on the upstream
-          # request, and the NetworkPolicy in the paperless namespace
-          # only allows traffic to paperless from the gate outpost pod
-          # itself, so ingress-nginx (or any other in-cluster workload)
-          # cannot bypass the gate to inject headers.
+          # Proxy, NOT forward_domain: the outpost terminates the request
+          # itself, so the trust boundary sits here rather than at the ingress
+          # controller, which only routes to the outpost Service and never
+          # reaches paperless directly. The outpost populates
+          # X-Authentik-Username, and the paperless NetworkPolicy admits
+          # traffic only from this outpost — so nothing else in the cluster
+          # can bypass the gate and inject those headers.
           mode: proxy
           internal_host: http://paperless.paperless.svc.cluster.local
           internal_host_ssl_validation: false
@@ -91,19 +74,12 @@ data:
           access_token_validity: hours=24
           refresh_token_validity: days=30
           intercept_header_auth: true
-          # Pass paperless's entire static-asset tree through anonymously.
-          # Two reasons:
-          # 1. <link rel="manifest"> is fetched with `credentials: 'omit'`
-          #    (W3C spec, no crossorigin attr) so the cookie is never sent
-          #    — every page load and every SPA navigation turned into a
-          #    302 → auth.renner.uno round-trip with an orphan session JWT.
-          #    Same root cause as the retired music-assistant-gate (see
-          #    comment further below); paperless can't retire the gate so
-          #    it carves out the offending paths instead.
-          # 2. Every CSS/JS/icon fetch otherwise validates the cookie at
-          #    the outpost — pure overhead on assets that carry no
-          #    authentication value. Django's STATIC tree is public by
-          #    design; MEDIA (under /api/documents/*) stays gated.
+          # The static-asset tree passes anonymously, and must. A
+          # <link rel="manifest"> is fetched with `credentials: 'omit'` per
+          # the W3C spec, so the session cookie is never sent and every page
+          # load turns into a redirect loop through the login origin. Django's
+          # STATIC tree is public by design; MEDIA under /api/documents/*
+          # stays gated.
           skip_path_regex: ^/static/.*
 
       - model: authentik_core.application
@@ -130,18 +106,14 @@ data:
           negate: false
           failure_result: false
 
-      # Retired: the music-assistant-gate ProxyProvider, Application,
-      # and dedicated Outpost. The gate fundamentally conflicted with
-      # MA's PWA model — Chrome fetches manifest.webmanifest / sw.js /
-      # icons with `credentials: 'omit'` (per W3C spec, no crossorigin
-      # attribute on MA's <link rel="manifest">), and the cached
-      # service worker bypasses fresh navigations on PWA cold-start so
-      # the Authentik cookie never lands in the WebAPK's cookie jar.
-      # MA has its own auth (delegated to Home Assistant); the gate's
-      # marginal value didn't justify the recurring breakage.
-      # Order matters: Application before Provider before Outpost.
-      # Application.provider is on_delete=PROTECT, so the Provider
-      # can't be removed while the Application still references it.
+      # Retired: the music-assistant gate. A forward-auth gate cannot work in
+      # front of a PWA — the manifest and service worker are fetched with
+      # `credentials: 'omit'`, and a cached service worker bypasses fresh
+      # navigations, so the session cookie never reaches the installed app.
+      # Do not reintroduce one there; MA delegates auth to Home Assistant.
+      #
+      # ORDER MATTERS: Application before Provider before Outpost, because
+      # Application.provider is on_delete=PROTECT.
       - model: authentik_core.application
         identifiers:
           slug: music-assistant-gate
@@ -157,18 +129,13 @@ data:
           name: music-assistant-gate
         state: absent
 
-      # Dedicated outpost: its ONLY job is to serve the in-cluster
-      # /outpost.goauthentik.io/auth/nginx endpoint that nginx
-      # auth_request hits for every request to paperless. The
-      # browser-facing legs of the OAuth flow (/start, /callback, the
-      # `/if/...` login UI, /application/o/authorize, /api, etc.) all
-      # need to happen on auth.renner.uno because that's the origin
-      # WebAuthn/FIDO/TOTP credentials are bound to. So we attach the
-      # ProxyProvider to BOTH outposts (see embedded outpost below) and
-      # set authentik_host_browser to auth.renner.uno here too — the
-      # nginx auth_request happens server-to-server, but if the
-      # dedicated outpost ever does emit a redirect URL, it points at
-      # the right host.
+      # Dedicated outpost, serving the in-cluster auth endpoint only.
+      #
+      # The browser-facing legs of the OAuth flow must happen on
+      # auth.renner.uno, because that is the origin WebAuthn credentials are
+      # bound to. Hence the ProxyProvider is attached to BOTH this outpost and
+      # the embedded one, and authentik_host_browser points at auth.renner.uno
+      # here too — so any redirect this outpost emits names the right host.
       - model: authentik_outposts.outpost
         identifiers:
           name: paperless-gate
@@ -196,12 +163,7 @@ data:
             kubernetes_ingress_secret_name: "authentik-outpost-tls"
             kubernetes_json_patches: null
 
-      # Tear down the leftover paperless-proxy Application from an
-      # earlier experiment. The orphan `paperless` ProxyProvider that
-      # used to back paperless.app is left in place — it's PROTECTed
-      # by Application.provider as long as anything references it, and
-      # leftover orphans don't break anything. Delete via the Authentik
-      # UI once everything's confirmed working.
+      # Leftover Application from an earlier experiment.
       - model: authentik_policies.policybinding
         identifiers:
           target: !Find [authentik_core.application, [slug, paperless-proxy]]
@@ -217,15 +179,11 @@ data:
       # ------------------------------------------------------------------
       # vscode config-editor gate
       #
-      # code-server (the HA /config editor sidecar) runs with --auth=none,
-      # as root, mounting the whole HA /config incl. plaintext secrets.yaml
-      # and .storage. This gate moves it behind Authentik with a passkey
-      # step-up, mirroring the paperless-gate proxy-outpost pattern above:
-      # the dedicated ak-outpost-vscode-gate terminates the request,
-      # enforces the session, and forwards to the code-server Service.
-      # Traefik only ever routes vscode.home.renner.uno to the outpost; the
-      # home-assistant CNP is tightened so ONLY the gate outpost may reach
-      # :8080 (see home-assistant-ns.yaml + vscode-gate-egress.yaml).
+      # code-server runs with --auth=none, as root, mounting the whole HA
+      # /config including plaintext secrets.yaml and .storage. This gate is
+      # the only thing in front of it: the outpost terminates the request and
+      # enforces a passkey step-up, Traefik routes only to the outpost, and
+      # the home-assistant CNP admits :8080 from the gate alone.
       # ------------------------------------------------------------------
       - id: vscode-config-editors-group
         model: authentik_core.group
@@ -265,17 +223,14 @@ data:
           access_token_validity: hours=24
           refresh_token_validity: days=30
           intercept_header_auth: true
-          # code-server is a service-worker app: <link rel="manifest"> and
-          # the service worker are fetched with credentials:'omit', which
-          # turned into a /authorize redirect loop and forced the RETIREMENT
-          # of the music-assistant gate (see comment above). Carve its
-          # static/manifest/SW tree out so those cookieless fetches pass
-          # through instead of 302-looping; the editor's actual content
-          # (the /config tree) is served over authenticated websocket/API
-          # paths, which stay gated.
-          # NOTE: the exact skip set is code-server-version-specific and
-          # MUST be confirmed post-deploy against the real redirect-loop
-          # behaviour (see the PR validation checklist).
+          # code-server is a service-worker app, so its manifest and SW are
+          # fetched with credentials:'omit' and 302-loop unless carved out.
+          # The editor's actual content rides authenticated websocket/API
+          # paths and stays gated.
+          #
+          # This skip set is version-specific: re-check it after a
+          # code-server bump, because a renamed asset path reintroduces the
+          # loop silently.
           skip_path_regex: ^/(_static/|manifest\.json|.*\.webmanifest|.*serviceWorker\.js).*
 
       - model: authentik_core.application
@@ -331,16 +286,11 @@ data:
       # ------------------------------------------------------------------
       # shelly-manager gate
       #
-      # shelly-manager has no authentication of its own: anything that can
-      # reach :8080 can rewrite Shelly device config, push scripts, and read
-      # the stored per-device AdminPassword values. Until now the only
-      # control was reachability of shelly-manager.home.renner.uno.
-      #
-      # Same shape as the vscode-gate above: a dedicated outpost terminates
-      # the request, enforces a passkey step-up, and forwards to the
-      # shelly-manager Service. Traefik only ever routes the hostname to the
-      # outpost; the home-assistant CNP allows ONLY this outpost to reach
-      # :8080 (see home-assistant-ns.yaml + shelly-manager-gate-egress.yaml).
+      # shelly-manager has no authentication of its own: anything that reaches
+      # :8080 can rewrite device config, push scripts and read the stored
+      # per-device AdminPassword values. Same shape as the vscode gate — the
+      # outpost is the only control, Traefik routes only to it, and the CNP
+      # admits :8080 from this outpost alone.
       # ------------------------------------------------------------------
       - id: shelly-managers-group
         model: authentik_core.group

kubernetes/apps/authentik/outpost-servicemonitors.yaml

@@ -1,50 +1,10 @@
 ---
-# The goauthentik.io operator (inside authentik-worker) auto-creates a
-# ServiceMonitor per outpost, but it does not stamp the
-# `release: kube-prometheus-stack` label that this cluster's Prometheus
-# requires on serviceMonitorSelector. Those operator-owned SMs are
-# silently ignored. Declare standalone SMs here with the correct label
-# so the outposts actually get scraped.
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: authentik-outpost-ldap
-  namespace: authentik
-  labels:
-    release: kube-prometheus-stack
-spec:
-  namespaceSelector:
-    matchNames:
-      - authentik
-  selector:
-    matchLabels:
-      goauthentik.io/outpost-name: ldap-outpost
-      goauthentik.io/outpost-type: ldap
-  endpoints:
-    - port: http-metrics
-      path: /metrics
-      interval: 30s
----
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: authentik-outpost-paperless-gate
-  namespace: authentik
-  labels:
-    release: kube-prometheus-stack
-spec:
-  namespaceSelector:
-    matchNames:
-      - authentik
-  selector:
-    matchLabels:
-      goauthentik.io/outpost-name: paperless-gate
-      goauthentik.io/outpost-type: proxy
-  endpoints:
-    - port: http-metrics
-      path: /metrics
-      interval: 30s
----
+# Only radius needs a hand-written ServiceMonitor. The goauthentik.io operator
+# creates one per outpost and Prometheus discovers those directly, because
+# serviceMonitorSelector here is empty (see monitoring/helmrelease-prometheus.yaml).
+# Standalone copies for ldap and paperless-gate used to live here as well; they
+# duplicated the operator's and were removed.
+#
 # Radius requires extra plumbing. The radius binary listens on :9300 just
 # like ldap/proxy (hardcoded in authentik/lib/default.yml), but authentik's
 # Python operator (`providers/radius/controllers/kubernetes.py`) omits the
@@ -53,6 +13,11 @@ spec:
 # it, radius does not. Result: the operator-managed `ak-outpost-local-radius`
 # Service exposes only UDP 1812. Add a sibling Service here that selects the
 # same pod labels and exposes 9300/TCP, then a ServiceMonitor for it.
+#
+# The operator's own `ak-outpost-local-radius` ServiceMonitor cannot cover
+# this: it selects on `app.kubernetes.io/managed-by: goauthentik.io` together
+# with `goauthentik.io/service-type: metrics`, and no Service carries both —
+# the operator's has the first, the one below has the second.
 apiVersion: v1
 kind: Service
 metadata:
@@ -86,6 +51,8 @@ spec:
     matchNames:
       - authentik
   selector:
+    # Pinned to the Service above. Without managed-by this would also match the
+    # operator's Service and scrape the same endpoint twice.
     matchLabels:
       app.kubernetes.io/managed-by: homelab
       goauthentik.io/outpost-name: local-radius

kubernetes/apps/monitoring-rules/alloy-servicemonitor.yaml

@@ -5,9 +5,8 @@ metadata:
   namespace: monitoring
   labels:
     app.kubernetes.io/name: alloy
-    # kube-prometheus-stack's Prometheus resource selects ServiceMonitors
-    # by this label; without it the resource exists but is never
-    # discovered.
+    # Not required for discovery — serviceMonitorSelector is empty — but kept
+    # for consistency with the other monitors.
     release: kube-prometheus-stack
 spec:
   selector:

kubernetes/apps/monitoring-rules/cnpg-podmonitors.yaml

@@ -1,7 +1,8 @@
 # Scrape the CNPG Postgres instances (port 9187, built-in exporter).
-# CNPG's own enablePodMonitor would generate PodMonitors without the
-# `release: kube-prometheus-stack` label, which this Prometheus requires
-# (podMonitorSelector), so we define them here instead.
+# Defined here rather than via CNPG's own enablePodMonitor so the scrape
+# config lives with the other monitors. The `release` label below is not
+# required for discovery — podMonitorSelector is empty — but kept for
+# consistency.
 apiVersion: monitoring.coreos.com/v1
 kind: PodMonitor
 metadata:

kubernetes/apps/monitoring-rules/home-assistant-servicemonitor.yaml

@@ -5,9 +5,8 @@ metadata:
   namespace: home-assistant
   labels:
     app.kubernetes.io/name: home-assistant
-    # kube-prometheus-stack's Prometheus resource selects ServiceMonitors
-    # by this label; without it the resource exists but is never
-    # discovered. Same trap the garage PrometheusRule documents.
+    # Not required for discovery — serviceMonitorSelector is empty — but kept
+    # for consistency with the other monitors.
     release: kube-prometheus-stack
 spec:
   selector:

kubernetes/apps/monitoring-rules/loki-servicemonitor.yaml

@@ -5,9 +5,8 @@ metadata:
   namespace: monitoring
   labels:
     app.kubernetes.io/name: loki
-    # kube-prometheus-stack's Prometheus resource selects ServiceMonitors
-    # by this label; without it the resource exists but is never
-    # discovered.
+    # Not required for discovery — serviceMonitorSelector is empty — but kept
+    # for consistency with the other monitors.
     release: kube-prometheus-stack
 spec:
   # Matches the `loki` (single-binary) and `loki-canary` services, but

kubernetes/apps/monitoring-rules/probes-blackbox.yaml

@@ -1,5 +1,9 @@
 # HTTP probes — fan out to blackbox-exporter. Probe targets mirror the
 # legacy Ansible role's blackbox_http_targets list.
+#
+# The `release` label on every Probe here IS load-bearing: probeSelector still
+# matches on it, unlike serviceMonitorSelector and podMonitorSelector, which
+# are empty. Removing it silently stops the probe from being discovered.
 apiVersion: monitoring.coreos.com/v1
 kind: Probe
 metadata:

kubernetes/apps/monitoring-rules/prometheusrule-cnpg-backup.yaml

@@ -11,25 +11,22 @@ spec:
   groups:
     - name: cnpg-backup.rules
       rules:
-        # WHY THIS FILE EXISTS
-        # Seven Postgres clusters back up to Hetzner and NOTHING alerted if that
-        # stopped. The only related rule was HetznerBackupEndpointDown, which
-        # probes the S3 endpoint — it proves the bucket is reachable, not that
-        # anything was written to it. Same failure shape as the WordPress backup
-        # that failed nightly for three days against a bucket that was declared
-        # in tofu but never applied: the status surface looked fine because
-        # nothing checked the thing that mattered.
+        # The Postgres clusters back up to Hetzner. HetznerBackupEndpointDown
+        # probes that S3 endpoint, which proves the bucket is reachable and
+        # nothing more — these two rules check that something is actually being
+        # written to it.
 
         - alert: CNPGWalArchiveBacklog
           # `ready` counts WAL segments sitting in pg_wal/archive_status waiting
-          # to be archived. Steady state is 0 across every cluster (verified
-          # 2026-08-19). A sustained backlog means the archiver is failing:
-          # point-in-time recovery is silently broken from that moment, and
-          # pg_wal keeps growing until it fills the PVC and takes Postgres down.
+          # to be archived. Steady state is 0. A sustained backlog means the
+          # archiver is failing: point-in-time recovery is silently broken from
+          # that moment, and pg_wal keeps growing until it fills the PVC and
+          # takes Postgres down.
           #
-          # This is the load-bearing alert of the two. It depends only on the
-          # instance-level collector, so unlike the backup-age metrics below it
-          # is unaffected by the missing Backup CRs (#522).
+          # This is the load-bearing alert of the two. It reads the
+          # instance-level collector directly, so unlike the backup-age metric
+          # discussed below it does not depend on Cluster status being
+          # populated.
           #
           # Threshold: >5 segments sustained for 30m. Brief non-zero values are
           # normal under write bursts — a segment is 16 MiB, so 5 is roughly
@@ -63,16 +60,17 @@ spec:
         #
         # The natural expression is
         #   (time() - cnpg_collector_last_available_backup_timestamp) > 48*3600
-        # and it CANNOT be used today: that metric reads exactly 0 on every
-        # instance in the cluster (verified 2026-08-19 across gf-pg, ak-pg-1/2,
-        # pprl-pg-2/3, im-pg, wp-pg, ha-pg). CNPG has no record of any
-        # successful backup anywhere, because the Backup CRs it derives that
-        # from are being deleted within a minute of completing — see #522.
+        # and it cannot be used, because that metric reads exactly 0 on every
+        # instance. The collector derives it from `Cluster.status
+        # .lastSuccessfulBackup`, and that field is absent on every Cluster
+        # here: all of them back up with `method: plugin` (barman-cloud), and
+        # the plugin path does not write it back.
         #
-        # Backups themselves DO work; that was verified independently by
-        # listing the bucket with barman-cloud-backup-list and restoring a WAL
-        # segment back out of S3. The gap is in the record, not the data.
+        # The backups themselves run and are retained — completed Backup CRs
+        # going back weeks exist for every scheduled cluster. The gap is in the
+        # status field the metric reads, not in the data.
         #
-        # Adding the rule now would fire permanently for all seven clusters and
-        # be silenced within a day, which is worse than not having it. Add it as
-        # part of resolving #522, once the metric reflects reality.
+        # Adding the rule against that metric would fire permanently for every
+        # cluster, which is worse than not having it. Either the plugin starts
+        # populating the field, or the alert has to be built on the Backup CRs
+        # directly, which Prometheus does not see today.

kubernetes/apps/monitoring-rules/prometheusrule-disk-wear.yaml

@@ -12,21 +12,54 @@ spec:
     - name: disk-wear.rules
       rules:
         # Wear percentages are self-reported: 100 means the rated endurance is
-        # spent, not that the drive stops. Planning thresholds, not predictions.
-        - alert: NvmeDriveWearHigh
-          # Long `for` because wear is monotonic and cannot flap.
-          expr: smartctl_device_percentage_used >= 80
-          for: 1h
+        # spent, not that the drive stops.
+        #
+        # Wear alone is not a signal. It only ever rises, so a plain threshold
+        # starts firing the day a drive crosses it and never stops — a status
+        # dressed as an alert (#673, #675, #682, #686 were all one drive set
+        # crossing 80%). What matters is wear that is high AND still climbing,
+        # so the alert below is a projection: at the write rate of the last
+        # week, when does this drive reach 95%?
+        #
+        # The rate is self-calibrating. Each drive reports both its lifetime
+        # bytes written and its own wear percentage, so bytes-per-percent comes
+        # from the drive itself and no rated-endurance table has to be kept in
+        # sync with the hardware. percentage_used is a whole-number percent and
+        # advances only every few weeks here, which is far too coarse to
+        # differentiate; bytes_written moves on every scrape.
+        - alert: NvmeDriveWearProjectedExhaustion
+          # The >= 80 gate is numerical, not editorial: bytes-per-percent is
+          # derived by dividing by an integer percentage, so on a lightly worn
+          # drive a half-percent rounding error swings the projection by tens
+          # of percent. Above 80 the same rounding is under one percent.
+          # Idle drives divide by a zero rate and yield +Inf, which is not < the
+          # threshold, so they stay silent on their own.
+          expr: >-
+            (
+              (95 - smartctl_device_percentage_used)
+              /
+              (
+                rate(smartctl_device_bytes_written[7d]) * 86400
+                  * smartctl_device_percentage_used
+                  / smartctl_device_bytes_written
+              )
+            ) < 180
+            and smartctl_device_percentage_used >= 80
+          # Long, because a projection off a 7d rate should not react to a day.
+          for: 6h
           labels:
             severity: warning
           annotations:
-            summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} is at {{ $value }}% rated wear"
+            summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} projects to 95% wear in {{ $value | printf \"%.0f\" }} days"
             description: >-
-              {{ $value }}% of rated write endurance used. Nothing is failing
-              yet. Urgency depends on what the drive carries — the Disk Health
-              dashboard shows the role. Losing an OSD costs a rebalance;
-              EPHEMERAL is /var and rebuilds, apart from the node's etcd member
-              data, which has to be removed and re-added.
+              The drive is past 80% of its rated endurance and the write rate
+              of the last week would spend the rest within six months. This is
+              a lead-time warning, not a fault: decide whether to replace the
+              drive or move what is writing to it. The Disk Health dashboard
+              shows which drives carry which role.
+        # Kept as a flat threshold on purpose: past rated endurance the drive
+        # runs outside what the manufacturer stands behind regardless of how
+        # fast it got there, and the alert clears by replacing it.
         - alert: NvmeDriveWearCritical
           expr: smartctl_device_percentage_used >= 95
           for: 1h

kubernetes/apps/monitoring/helmrelease-blackbox-exporter.yaml

@@ -24,8 +24,10 @@ spec:
       enabled: true
       defaults:
         labels:
-          # Required for kube-prometheus-stack's Prometheus to pick up
-          # this ServiceMonitor (its selector matches release).
+          # Not required for discovery — serviceMonitorSelector is empty — but
+          # kept for consistency with the other monitors. The Probes in
+          # monitoring-rules/probes-blackbox.yaml do need it; probeSelector
+          # still matches on release.
           release: kube-prometheus-stack
         interval: 30s
     # Helm DOES deep-merge this map with the chart's default (that is why

kubernetes/apps/monitoring/helmrelease-prometheus.yaml

@@ -56,26 +56,21 @@ spec:
         #
         # By default kube-prometheus-stack sets these selectors to
         # `matchLabels: {release: kube-prometheus-stack}`. ServiceMonitors
-        # created at RUNTIME by third-party operators carry that operator's own
-        # labels instead and can never be made to match — no static manifest or
-        # Kustomize patch can relabel an object the operator creates itself.
+        # created at RUNTIME by third-party operators — Rook, goauthentik.io —
+        # carry that operator's own labels instead and can never be made to
+        # match: no static manifest or Kustomize patch can relabel an object
+        # the operator creates itself. Left strict, those targets are dropped
+        # silently, with no failed resource and no gap visible anywhere except
+        # in the missing series.
         #
-        # This was silently dropping 12 of 30 ServiceMonitors (2026-07-27):
-        #   rook-ceph/rook-ceph-mgr, rook-ceph/rook-ceph-exporter  (team: rook)
-        #   authentik/ak-outpost-*  x10                            (goauthentik.io/...)
-        # Consequence: ZERO `ceph_*` series existed in Prometheus for all 79
-        # days of the cluster's life — the entire storage layer had no
-        # telemetry, no history, and any Ceph alerting rule would have had
-        # nothing to fire on. The Rook side is the reason this was found; the
-        # authentik outpost targets come along with it.
+        # Because these two are empty, the `release` label on this repo's own
+        # ServiceMonitors and PodMonitors is decorative. It is still set for
+        # consistency, and the comments there say so.
         #
-        # This is the ServiceMonitor twin of the trap already documented for
-        # rules in monitoring-rules/prometheusrule-velero.yaml ("without this
-        # label the rule is created but never loaded").
-        #
-        # ruleSelector is deliberately NOT loosened: this repo's PrometheusRules
-        # all carry `release: kube-prometheus-stack` on purpose, and keeping
-        # that selector strict stops a stray third-party rule from firing here.
+        # ruleSelector, probeSelector and scrapeConfigSelector are deliberately
+        # NOT loosened: nothing outside this repo creates those, so keeping the
+        # selectors strict stops a stray third-party rule from firing here. For
+        # PrometheusRules and Probes the label IS required.
         serviceMonitorSelectorNilUsesHelmValues: false
         podMonitorSelectorNilUsesHelmValues: false
         # Rewrites the in-cluster URL embedded in alert generatorURLs and