fix(ci): dispatch the remaining pipelines by deploy target

Rolls the security-scan pilot out to the other thirteen. Every workflow-root
gate of the form

    - event: manual
      evaluate: 'PIPELINE == "x"'

becomes

    - event: deployment
      evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "x"'

The target rides in pipeline metadata rather than in the manual-run variable
map, so it reaches the workflow-root filter on Woodpecker 3.18 where PIPELINE
does not. Verified on the pilot: pipeline 7004 selected the security-scan
workflow and nothing else.

Two files also carry step-level manual gates, and those had to move with the
root or the pipelines would have converted into something that runs but cannot
do its job:

  deploy-infrastructure  the apply step gates on TOFU_APPLY. Left on
                         `event: manual` it would never fire again, so the
                         pipeline would plan and silently never apply.
                         TOFU_APPLY now arrives via `--param`, which lands in
                         the same non-plugin environment step-level `when`
                         evaluates against. No step here is a plugin — nothing
                         in .woodpecker uses `settings:` — so that path holds.

  build-ci-images        eight per-image steps gate on manual-or-push to decide
                         whether to rebuild. All eight follow the root.

Cron gates are untouched. They already matched by name everywhere except
security-scan, which the pilot fixed.

Also corrects the trigger documentation, including a claim in update-audit.yml
that the other pipelines gate on "an injected PIPELINE var set at cron-create
time in the UI". Crons cannot carry variables at all — `repo cron add` takes
name, branch, schedule and enabled. That sentence described a mechanism that
never existed and was the same misunderstanding that left the daily security
scan silently not running.

The `timeout: 30` lint warning on the four deploy-* files predates this and is
left alone.

.woodpecker/alert-monitor.yml

@@ -32,13 +32,13 @@
 #              mechanism as update-audit.yml):
 #                woodpecker repo cron add --repo k.renner/homelab \
 #                  --name alert-monitor --branch master --schedule '*/15 * * * *'
-#   - manual : woodpecker pipeline create --branch master \
-#                --var PIPELINE=alert-monitor k.renner/homelab
+#   - deploy : woodpecker-cli pipeline deploy k.renner/homelab <n> alert-monitor
+#              where <n> is an existing pipeline to promote.
 when:
   - event: cron
     cron: alert-monitor
-  - event: manual
-    evaluate: 'PIPELINE == "alert-monitor"'
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "alert-monitor"'
 
 steps:
   - name: fetch-alerts

.woodpecker/build-ci-images.yml

@@ -31,8 +31,8 @@ when:
       - "ci/images/**"
       - "secret-rotator/**"
       - ".woodpecker/build-ci-images.yml"
-  - event: manual
-    evaluate: 'PIPELINE == "build-ci-images"'
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "build-ci-images"'
 
 steps:
   - name: build-ansible
@@ -44,7 +44,7 @@ steps:
     # secret-rotator commit needlessly rebuilt all four ~1.3G ci-* images and
     # orphaned ~5G of registry layers per run — a main driver of #336.
     when:
-      - event: manual
+      - event: deployment
       - event: push
         path: ["ci/images/ansible/**", ".woodpecker/build-ci-images.yml"]
     environment:
@@ -68,7 +68,7 @@ steps:
     directory: ci/images/k8s
     depends_on: []
     when:
-      - event: manual
+      - event: deployment
       - event: push
         path: ["ci/images/k8s/**", ".woodpecker/build-ci-images.yml"]
     environment:
@@ -92,7 +92,7 @@ steps:
     directory: ci/images/tofu
     depends_on: []
     when:
-      - event: manual
+      - event: deployment
       - event: push
         path: ["ci/images/tofu/**", ".woodpecker/build-ci-images.yml"]
     environment:
@@ -123,7 +123,7 @@ steps:
     directory: secret-rotator
     depends_on: []
     when:
-      - event: manual
+      - event: deployment
       - event: push
         path: ["secret-rotator/**", ".woodpecker/build-ci-images.yml"]
     environment:
@@ -152,7 +152,7 @@ steps:
     directory: ci/images/alpine-jq
     depends_on: []
     when:
-      - event: manual
+      - event: deployment
       - event: push
         path: ["ci/images/alpine-jq/**", ".woodpecker/build-ci-images.yml"]
     environment:
@@ -181,7 +181,7 @@ steps:
     directory: ci/images/cni-plugins
     depends_on: []
     when:
-      - event: manual
+      - event: deployment
       - event: push
         path: ["ci/images/cni-plugins/**", ".woodpecker/build-ci-images.yml"]
     environment:
@@ -210,7 +210,7 @@ steps:
     directory: ci/images/alpine-openssl
     depends_on: []
     when:
-      - event: manual
+      - event: deployment
       - event: push
         path: ["ci/images/alpine-openssl/**", ".woodpecker/build-ci-images.yml"]
     environment:
@@ -239,7 +239,7 @@ steps:
     directory: ci/images/heating-can
     depends_on: []
     when:
-      - event: manual
+      - event: deployment
       - event: push
         path: ["ci/images/heating-can/**", ".woodpecker/build-ci-images.yml"]
     environment:

.woodpecker/ci-claude-refresh.yml

@@ -54,8 +54,8 @@
 #
 # Triggering:
 #   - cron   : the cron NAMED exactly "ci-claude-refresh" (see above).
-#   - manual : woodpecker pipeline create --branch master \
-#                --var PIPELINE=ci-claude-refresh k.renner/homelab
+#   - deploy : woodpecker-cli pipeline deploy k.renner/homelab <n> ci-claude-refresh
+#              where <n> is an existing pipeline to promote.
 #   - push   : a change to the Containerfile (or this file) ON master forces a
 #              rebuild of the current latest version so Containerfile fixes ship
 #              promptly. Restricted to master ON PURPOSE — a feature branch must
@@ -65,8 +65,8 @@
 when:
   - event: cron
     cron: ci-claude-refresh
-  - event: manual
-    evaluate: 'PIPELINE == "ci-claude-refresh"'
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "ci-claude-refresh"'
   - event: push
     branch: master
     path:

.woodpecker/deploy-claude-workstation.yml

@@ -6,8 +6,8 @@ when:
       - "ansible/roles/claude-workstation/**"
       - "ansible/plays/install-claude-workstation.yml"
       - ".woodpecker/deploy-claude-workstation.yml"
-  - event: manual
-    evaluate: 'PIPELINE == "deploy-claude-workstation"'
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "deploy-claude-workstation"'
 
 timeout: 30
 

.woodpecker/deploy-infrastructure.yml

@@ -8,8 +8,8 @@ when:
     path:
       - "opentofu/**"
       - ".woodpecker/deploy-infrastructure.yml"
-  - event: manual
-    evaluate: 'PIPELINE == "deploy-infrastructure"'
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "deploy-infrastructure"'
 
 steps:
   - name: validate-secrets
@@ -84,7 +84,7 @@ steps:
     image: gitea.renner.uno/gitea_admin/ci-tofu:2026-04-29
     directory: opentofu
     when:
-      - event: manual
+      - event: deployment
         evaluate: 'TOFU_APPLY == "true"'
     environment:
       TF_IN_AUTOMATION: "true"

.woodpecker/deploy-netbird-client.yml

@@ -7,8 +7,8 @@ when:
       - "ansible/plays/install-netbird-client.yml"
       - "ansible/inventory/group_vars/claude_workstation/**"
       - ".woodpecker/deploy-netbird-client.yml"
-  - event: manual
-    evaluate: 'PIPELINE == "deploy-netbird-client"'
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "deploy-netbird-client"'
 
 timeout: 30
 

.woodpecker/deploy-netbird-proxy.yml

@@ -9,8 +9,8 @@ when:
       - "ansible/roles/netbird-proxy-watchdog/**"
       - "ansible/plays/install-netbird-proxy.yml"
       - ".woodpecker/deploy-netbird-proxy.yml"
-  - event: manual
-    evaluate: 'PIPELINE == "deploy-netbird-proxy"'
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "deploy-netbird-proxy"'
 
 timeout: 30
 

.woodpecker/deploy-netbird-server.yml

@@ -9,8 +9,8 @@ when:
       - "ansible/roles/netbird-proxy-watchdog/**"
       - "ansible/plays/install-netbird-server.yml"
       - ".woodpecker/deploy-netbird-server.yml"
-  - event: manual
-    evaluate: 'PIPELINE == "deploy-netbird-server"'
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "deploy-netbird-server"'
 
 timeout: 30
 

.woodpecker/e2e-test.yml

@@ -1,6 +1,6 @@
 when:
-  - event: manual
-    evaluate: 'PIPELINE == "e2e-test"'
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "e2e-test"'
 
 steps:
   - name: e2e

.woodpecker/e2e.yml

@@ -7,7 +7,8 @@
 #                was harmless at daily/weekly cadence — but alert-monitor.yml
 #                fires every 15 minutes, so piggybacking must be opt-in now.
 #                A new cron that SHOULD also run e2e gets its name added here.
-#   - manual   : `woodpecker pipeline create --branch master --var PIPELINE=e2e k.renner/homelab`
+#   - deploy   : `woodpecker-cli pipeline deploy k.renner/homelab <n> e2e`, where
+#                <n> is an existing pipeline to promote.
 
 when:
   - event: cron
@@ -16,8 +17,8 @@ when:
     cron: update-audit
   - event: cron
     cron: weekly-update
-  - event: manual
-    evaluate: 'PIPELINE == "e2e"'
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "e2e"'
 
 steps:
   - name: model-selection-tests

.woodpecker/install-garage-synology.yml

@@ -1,6 +1,6 @@
 when:
-  - event: manual
-    evaluate: 'PIPELINE == "install-garage-synology"'
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "install-garage-synology"'
 
 steps:
   - name: run-playbook

.woodpecker/issue-fix.yml

@@ -34,13 +34,13 @@
 #   - cron   : a cron NAMED exactly "issue-fix":
 #                woodpecker repo cron add --repo k.renner/homelab \
 #                  --name issue-fix --branch master --schedule '41 * * * *'
-#   - manual : woodpecker pipeline create --branch master \
-#                --var PIPELINE=issue-fix k.renner/homelab
+#   - deploy : woodpecker-cli pipeline deploy k.renner/homelab <n> issue-fix
+#              where <n> is an existing pipeline to promote.
 when:
   - event: cron
     cron: issue-fix
-  - event: manual
-    evaluate: 'PIPELINE == "issue-fix"'
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "issue-fix"'
 
 steps:
   - name: fetch-diag

.woodpecker/pr-review.yml

@@ -53,9 +53,9 @@
 #
 # Triggering: pull_request only — automatic on PR open + push-to-open-PR. To
 # re-review a PR, push a commit (the new head sha clears the dedup marker below
-# and earns a fresh review). There is deliberately no manual PIPELINE trigger:
-# Woodpecker exposes a --var only to YAML variable substitution and `when`
-# filters, NOT to a step's shell environment, so a --var PR=<n> re-review would
+# and earns a fresh review). There is deliberately no on-demand trigger for a
+# specific PR: Woodpecker exposes injected variables to YAML substitution and
+# `when` filters, NOT to a step's shell environment, so a PR=<n> re-review would
 # silently no-op (the step would find no PR number and exit green).
 # (Note: keep literal dollar-brace out of THIS file entirely, even in comments —
 # Woodpecker runs its variable substitution over the whole YAML text, comments

.woodpecker/security-scan.yml

@@ -1,8 +1,20 @@
+# Trivy scan of the IaC tree and every image referenced from ansible/, with an
+# agent pass that opens a PR for CRITICAL/HIGH findings.
+#
+# Triggers:
+#   - cron       : the named `daily-security-scan` cron, matched by NAME.
+#                  Crons cannot carry variables — `woodpecker-cli repo cron add`
+#                  has no --var — so a variable gate can never match a cron.
+#   - deployment : woodpecker-cli pipeline deploy k.renner/homelab <n> security-scan
+#                  The target rides in pipeline metadata as
+#                  CI_PIPELINE_DEPLOY_TARGET, which the workflow-root filter
+#                  does see, unlike manual-run variables on Woodpecker 3.18
+#                  (#582).
 when:
   - event: cron
-    evaluate: 'PIPELINE == "security-scan"'
-  - event: manual
-    evaluate: 'PIPELINE == "security-scan"'
+    cron: daily-security-scan
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "security-scan"'
 
 steps:
   - name: iac-scan

.woodpecker/update-audit.yml

@@ -10,14 +10,14 @@
 # report; the pipeline posts it to a single persistent Gitea issue.
 #
 # Triggering:
-#   - cron   : a cron NAMED exactly "update-audit". Unlike the other pipelines
-#              here (which gate on an injected `PIPELINE` var set at cron-create
-#              time in the UI), this uses Woodpecker's documented `cron:` name
-#              filter so the schedule can be created entirely from the CLI:
+#   - cron   : a cron NAMED exactly "update-audit". Crons carry no variables —
+#              `repo cron add` takes name, branch, schedule and enabled — so the
+#              name filter is the only thing that can select a workflow for a
+#              scheduled run:
 #                woodpecker repo cron add --repo k.renner/homelab \
 #                  --name update-audit --branch master --schedule '@weekly'
-#   - manual : woodpecker pipeline create --branch master \
-#                --var PIPELINE=update-audit k.renner/homelab
+#   - deploy : woodpecker-cli pipeline deploy k.renner/homelab <n> update-audit
+#              where <n> is an existing pipeline to promote.
 #
 # NOTE: e2e.yml runs on an explicit list of cron names that includes this one,
 # so a weekly e2e run piggybacks on the audit (it was a bare `event: cron`
@@ -25,8 +25,8 @@
 when:
   - event: cron
     cron: update-audit
-  - event: manual
-    evaluate: 'PIPELINE == "update-audit"'
+  - event: deployment
+    evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "update-audit"'
 
 steps:
   - name: update-audit-agent

kubernetes/apps/authentik/outpost-servicemonitors.yaml

@@ -1,50 +1,10 @@
 ---
-# The goauthentik.io operator (inside authentik-worker) auto-creates a
-# ServiceMonitor per outpost, but it does not stamp the
-# `release: kube-prometheus-stack` label that this cluster's Prometheus
-# requires on serviceMonitorSelector. Those operator-owned SMs are
-# silently ignored. Declare standalone SMs here with the correct label
-# so the outposts actually get scraped.
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: authentik-outpost-ldap
-  namespace: authentik
-  labels:
-    release: kube-prometheus-stack
-spec:
-  namespaceSelector:
-    matchNames:
-      - authentik
-  selector:
-    matchLabels:
-      goauthentik.io/outpost-name: ldap-outpost
-      goauthentik.io/outpost-type: ldap
-  endpoints:
-    - port: http-metrics
-      path: /metrics
-      interval: 30s
----
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: authentik-outpost-paperless-gate
-  namespace: authentik
-  labels:
-    release: kube-prometheus-stack
-spec:
-  namespaceSelector:
-    matchNames:
-      - authentik
-  selector:
-    matchLabels:
-      goauthentik.io/outpost-name: paperless-gate
-      goauthentik.io/outpost-type: proxy
-  endpoints:
-    - port: http-metrics
-      path: /metrics
-      interval: 30s
----
+# Only radius needs a hand-written ServiceMonitor. The goauthentik.io operator
+# creates one per outpost and Prometheus discovers those directly, because
+# serviceMonitorSelector here is empty (see monitoring/helmrelease-prometheus.yaml).
+# Standalone copies for ldap and paperless-gate used to live here as well; they
+# duplicated the operator's and were removed.
+#
 # Radius requires extra plumbing. The radius binary listens on :9300 just
 # like ldap/proxy (hardcoded in authentik/lib/default.yml), but authentik's
 # Python operator (`providers/radius/controllers/kubernetes.py`) omits the
@@ -53,6 +13,11 @@ spec:
 # it, radius does not. Result: the operator-managed `ak-outpost-local-radius`
 # Service exposes only UDP 1812. Add a sibling Service here that selects the
 # same pod labels and exposes 9300/TCP, then a ServiceMonitor for it.
+#
+# The operator's own `ak-outpost-local-radius` ServiceMonitor cannot cover
+# this: it selects on `app.kubernetes.io/managed-by: goauthentik.io` together
+# with `goauthentik.io/service-type: metrics`, and no Service carries both —
+# the operator's has the first, the one below has the second.
 apiVersion: v1
 kind: Service
 metadata:
@@ -86,6 +51,8 @@ spec:
     matchNames:
       - authentik
   selector:
+    # Pinned to the Service above. Without managed-by this would also match the
+    # operator's Service and scrape the same endpoint twice.
     matchLabels:
       app.kubernetes.io/managed-by: homelab
       goauthentik.io/outpost-name: local-radius

kubernetes/apps/monitoring-rules/alloy-servicemonitor.yaml

@@ -5,9 +5,8 @@ metadata:
   namespace: monitoring
   labels:
     app.kubernetes.io/name: alloy
-    # kube-prometheus-stack's Prometheus resource selects ServiceMonitors
-    # by this label; without it the resource exists but is never
-    # discovered.
+    # Not required for discovery — serviceMonitorSelector is empty — but kept
+    # for consistency with the other monitors.
     release: kube-prometheus-stack
 spec:
   selector:

kubernetes/apps/monitoring-rules/cnpg-podmonitors.yaml

@@ -1,7 +1,8 @@
 # Scrape the CNPG Postgres instances (port 9187, built-in exporter).
-# CNPG's own enablePodMonitor would generate PodMonitors without the
-# `release: kube-prometheus-stack` label, which this Prometheus requires
-# (podMonitorSelector), so we define them here instead.
+# Defined here rather than via CNPG's own enablePodMonitor so the scrape
+# config lives with the other monitors. The `release` label below is not
+# required for discovery — podMonitorSelector is empty — but kept for
+# consistency.
 apiVersion: monitoring.coreos.com/v1
 kind: PodMonitor
 metadata:

kubernetes/apps/monitoring-rules/home-assistant-servicemonitor.yaml

@@ -5,9 +5,8 @@ metadata:
   namespace: home-assistant
   labels:
     app.kubernetes.io/name: home-assistant
-    # kube-prometheus-stack's Prometheus resource selects ServiceMonitors
-    # by this label; without it the resource exists but is never
-    # discovered. Same trap the garage PrometheusRule documents.
+    # Not required for discovery — serviceMonitorSelector is empty — but kept
+    # for consistency with the other monitors.
     release: kube-prometheus-stack
 spec:
   selector:

kubernetes/apps/monitoring-rules/loki-servicemonitor.yaml

@@ -5,9 +5,8 @@ metadata:
   namespace: monitoring
   labels:
     app.kubernetes.io/name: loki
-    # kube-prometheus-stack's Prometheus resource selects ServiceMonitors
-    # by this label; without it the resource exists but is never
-    # discovered.
+    # Not required for discovery — serviceMonitorSelector is empty — but kept
+    # for consistency with the other monitors.
     release: kube-prometheus-stack
 spec:
   # Matches the `loki` (single-binary) and `loki-canary` services, but

kubernetes/apps/monitoring-rules/probes-blackbox.yaml

@@ -1,5 +1,9 @@
 # HTTP probes — fan out to blackbox-exporter. Probe targets mirror the
 # legacy Ansible role's blackbox_http_targets list.
+#
+# The `release` label on every Probe here IS load-bearing: probeSelector still
+# matches on it, unlike serviceMonitorSelector and podMonitorSelector, which
+# are empty. Removing it silently stops the probe from being discovered.
 apiVersion: monitoring.coreos.com/v1
 kind: Probe
 metadata:

kubernetes/apps/monitoring-rules/prometheusrule-cnpg-backup.yaml

@@ -11,25 +11,22 @@ spec:
   groups:
     - name: cnpg-backup.rules
       rules:
-        # WHY THIS FILE EXISTS
-        # Seven Postgres clusters back up to Hetzner and NOTHING alerted if that
-        # stopped. The only related rule was HetznerBackupEndpointDown, which
-        # probes the S3 endpoint — it proves the bucket is reachable, not that
-        # anything was written to it. Same failure shape as the WordPress backup
-        # that failed nightly for three days against a bucket that was declared
-        # in tofu but never applied: the status surface looked fine because
-        # nothing checked the thing that mattered.
+        # The Postgres clusters back up to Hetzner. HetznerBackupEndpointDown
+        # probes that S3 endpoint, which proves the bucket is reachable and
+        # nothing more — these two rules check that something is actually being
+        # written to it.
 
         - alert: CNPGWalArchiveBacklog
           # `ready` counts WAL segments sitting in pg_wal/archive_status waiting
-          # to be archived. Steady state is 0 across every cluster (verified
-          # 2026-08-19). A sustained backlog means the archiver is failing:
-          # point-in-time recovery is silently broken from that moment, and
-          # pg_wal keeps growing until it fills the PVC and takes Postgres down.
+          # to be archived. Steady state is 0. A sustained backlog means the
+          # archiver is failing: point-in-time recovery is silently broken from
+          # that moment, and pg_wal keeps growing until it fills the PVC and
+          # takes Postgres down.
           #
-          # This is the load-bearing alert of the two. It depends only on the
-          # instance-level collector, so unlike the backup-age metrics below it
-          # is unaffected by the missing Backup CRs (#522).
+          # This is the load-bearing alert of the two. It reads the
+          # instance-level collector directly, so unlike the backup-age metric
+          # discussed below it does not depend on Cluster status being
+          # populated.
           #
           # Threshold: >5 segments sustained for 30m. Brief non-zero values are
           # normal under write bursts — a segment is 16 MiB, so 5 is roughly
@@ -63,16 +60,17 @@ spec:
         #
         # The natural expression is
         #   (time() - cnpg_collector_last_available_backup_timestamp) > 48*3600
-        # and it CANNOT be used today: that metric reads exactly 0 on every
-        # instance in the cluster (verified 2026-08-19 across gf-pg, ak-pg-1/2,
-        # pprl-pg-2/3, im-pg, wp-pg, ha-pg). CNPG has no record of any
-        # successful backup anywhere, because the Backup CRs it derives that
-        # from are being deleted within a minute of completing — see #522.
+        # and it cannot be used, because that metric reads exactly 0 on every
+        # instance. The collector derives it from `Cluster.status
+        # .lastSuccessfulBackup`, and that field is absent on every Cluster
+        # here: all of them back up with `method: plugin` (barman-cloud), and
+        # the plugin path does not write it back.
         #
-        # Backups themselves DO work; that was verified independently by
-        # listing the bucket with barman-cloud-backup-list and restoring a WAL
-        # segment back out of S3. The gap is in the record, not the data.
+        # The backups themselves run and are retained — completed Backup CRs
+        # going back weeks exist for every scheduled cluster. The gap is in the
+        # status field the metric reads, not in the data.
         #
-        # Adding the rule now would fire permanently for all seven clusters and
-        # be silenced within a day, which is worse than not having it. Add it as
-        # part of resolving #522, once the metric reflects reality.
+        # Adding the rule against that metric would fire permanently for every
+        # cluster, which is worse than not having it. Either the plugin starts
+        # populating the field, or the alert has to be built on the Backup CRs
+        # directly, which Prometheus does not see today.

kubernetes/apps/monitoring-rules/prometheusrule-disk-wear.yaml

@@ -12,21 +12,54 @@ spec:
     - name: disk-wear.rules
       rules:
         # Wear percentages are self-reported: 100 means the rated endurance is
-        # spent, not that the drive stops. Planning thresholds, not predictions.
-        - alert: NvmeDriveWearHigh
-          # Long `for` because wear is monotonic and cannot flap.
-          expr: smartctl_device_percentage_used >= 80
-          for: 1h
+        # spent, not that the drive stops.
+        #
+        # Wear alone is not a signal. It only ever rises, so a plain threshold
+        # starts firing the day a drive crosses it and never stops — a status
+        # dressed as an alert (#673, #675, #682, #686 were all one drive set
+        # crossing 80%). What matters is wear that is high AND still climbing,
+        # so the alert below is a projection: at the write rate of the last
+        # week, when does this drive reach 95%?
+        #
+        # The rate is self-calibrating. Each drive reports both its lifetime
+        # bytes written and its own wear percentage, so bytes-per-percent comes
+        # from the drive itself and no rated-endurance table has to be kept in
+        # sync with the hardware. percentage_used is a whole-number percent and
+        # advances only every few weeks here, which is far too coarse to
+        # differentiate; bytes_written moves on every scrape.
+        - alert: NvmeDriveWearProjectedExhaustion
+          # The >= 80 gate is numerical, not editorial: bytes-per-percent is
+          # derived by dividing by an integer percentage, so on a lightly worn
+          # drive a half-percent rounding error swings the projection by tens
+          # of percent. Above 80 the same rounding is under one percent.
+          # Idle drives divide by a zero rate and yield +Inf, which is not < the
+          # threshold, so they stay silent on their own.
+          expr: >-
+            (
+              (95 - smartctl_device_percentage_used)
+              /
+              (
+                rate(smartctl_device_bytes_written[7d]) * 86400
+                  * smartctl_device_percentage_used
+                  / smartctl_device_bytes_written
+              )
+            ) < 180
+            and smartctl_device_percentage_used >= 80
+          # Long, because a projection off a 7d rate should not react to a day.
+          for: 6h
           labels:
             severity: warning
           annotations:
-            summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} is at {{ $value }}% rated wear"
+            summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} projects to 95% wear in {{ $value | printf \"%.0f\" }} days"
             description: >-
-              {{ $value }}% of rated write endurance used. Nothing is failing
-              yet. Urgency depends on what the drive carries — the Disk Health
-              dashboard shows the role. Losing an OSD costs a rebalance;
-              EPHEMERAL is /var and rebuilds, apart from the node's etcd member
-              data, which has to be removed and re-added.
+              The drive is past 80% of its rated endurance and the write rate
+              of the last week would spend the rest within six months. This is
+              a lead-time warning, not a fault: decide whether to replace the
+              drive or move what is writing to it. The Disk Health dashboard
+              shows which drives carry which role.
+        # Kept as a flat threshold on purpose: past rated endurance the drive
+        # runs outside what the manufacturer stands behind regardless of how
+        # fast it got there, and the alert clears by replacing it.
         - alert: NvmeDriveWearCritical
           expr: smartctl_device_percentage_used >= 95
           for: 1h

kubernetes/apps/monitoring/helmrelease-blackbox-exporter.yaml

@@ -24,8 +24,10 @@ spec:
       enabled: true
       defaults:
         labels:
-          # Required for kube-prometheus-stack's Prometheus to pick up
-          # this ServiceMonitor (its selector matches release).
+          # Not required for discovery — serviceMonitorSelector is empty — but
+          # kept for consistency with the other monitors. The Probes in
+          # monitoring-rules/probes-blackbox.yaml do need it; probeSelector
+          # still matches on release.
           release: kube-prometheus-stack
         interval: 30s
     # Helm DOES deep-merge this map with the chart's default (that is why

kubernetes/apps/monitoring/helmrelease-prometheus.yaml

@@ -56,26 +56,21 @@ spec:
         #
         # By default kube-prometheus-stack sets these selectors to
         # `matchLabels: {release: kube-prometheus-stack}`. ServiceMonitors
-        # created at RUNTIME by third-party operators carry that operator's own
-        # labels instead and can never be made to match — no static manifest or
-        # Kustomize patch can relabel an object the operator creates itself.
+        # created at RUNTIME by third-party operators — Rook, goauthentik.io —
+        # carry that operator's own labels instead and can never be made to
+        # match: no static manifest or Kustomize patch can relabel an object
+        # the operator creates itself. Left strict, those targets are dropped
+        # silently, with no failed resource and no gap visible anywhere except
+        # in the missing series.
         #
-        # This was silently dropping 12 of 30 ServiceMonitors (2026-07-27):
-        #   rook-ceph/rook-ceph-mgr, rook-ceph/rook-ceph-exporter  (team: rook)
-        #   authentik/ak-outpost-*  x10                            (goauthentik.io/...)
-        # Consequence: ZERO `ceph_*` series existed in Prometheus for all 79
-        # days of the cluster's life — the entire storage layer had no
-        # telemetry, no history, and any Ceph alerting rule would have had
-        # nothing to fire on. The Rook side is the reason this was found; the
-        # authentik outpost targets come along with it.
+        # Because these two are empty, the `release` label on this repo's own
+        # ServiceMonitors and PodMonitors is decorative. It is still set for
+        # consistency, and the comments there say so.
         #
-        # This is the ServiceMonitor twin of the trap already documented for
-        # rules in monitoring-rules/prometheusrule-velero.yaml ("without this
-        # label the rule is created but never loaded").
-        #
-        # ruleSelector is deliberately NOT loosened: this repo's PrometheusRules
-        # all carry `release: kube-prometheus-stack` on purpose, and keeping
-        # that selector strict stops a stray third-party rule from firing here.
+        # ruleSelector, probeSelector and scrapeConfigSelector are deliberately
+        # NOT loosened: nothing outside this repo creates those, so keeping the
+        # selectors strict stops a stray third-party rule from firing here. For
+        # PrometheusRules and Probes the label IS required.
         serviceMonitorSelectorNilUsesHelmValues: false
         podMonitorSelectorNilUsesHelmValues: false
         # Rewrites the in-cluster URL embedded in alert generatorURLs and

kubernetes/apps/woodpecker/MIGRATION-namespace.md

@@ -305,7 +305,7 @@ server → agent → step pod → server end to end:
 
 ```sh
 curl -sS -o /dev/null -w '%{http_code}\n' https://ci.renner.uno/
-woodpecker pipeline create --branch master --var PIPELINE=e2e-test k.renner/homelab
+woodpecker-cli pipeline deploy k.renner/homelab <pipeline-number> e2e-test
 ```
 
 Step pods must still reach the server on `:8000` while still being unable to