ci(monitoring-rules): machine-check PromQL with promtool (REN-161)

Nothing in this repo checked an alert expression. ci-manifests-test.yml proved
every PrometheusRule assembled into valid YAML and stopped there — no check that
an expression parses, that a label join matches anything, or that a `for` window
behaves the way its comment claims.

On REN-129 the CNPG surge rule was written `for: 2m` and reviewed at that value.
Against Iris's seven measured nights it fires on six of them: two owner emails a
night, 04:16 and 04:22 local, for a WAL backlog that clears itself in under ten
minutes. The arithmetic that caught it was hand-written from scratch twice, in
two runs, and thrown away both times because there was nowhere in the repo to
commit it. It is now a negative control.

Adds:

* scripts/ci/promtool-rules.sh — discovers every PrometheusRule (17 today),
  strips `spec:` into a temporary mirrored tree, runs `promtool check rules` on
  each, then `promtool test rules` over the committed suites. Discovery floors
  on both, as in the sibling steps.
* a `promtool` step in ci-manifests-test.yml, `depends_on: []` like the other
  four, on prom/prometheus:v3.15.0. The 3.x pin is load-bearing: the cluster
  runs kube-prometheus-stack >= 72.0.0, and alert `for` evaluation is
  version-sensitive, so a 2.x promtool would check a different evaluator than
  production. The script asserts the major version it is handed.
* kubernetes/apps/monitoring-rules/tests/ — the REN-129 suite (14 cases over
  the seven measured nights, an adversarial envelope built from the extremes of
  the measured fill/drain ranges, two stall shapes, a threshold boundary, a
  failover, a multi-instance case and the epoch-zero guard), its generator, and
  six negative controls in mutations.txt.
* a mutation phase. Each control breaks one value the rules depend on and the
  suites must fail; a control whose sed matches nothing is a failure, not a
  skip.
* scripts/ci/test-promtool-rules.sh, run from ci-scripts-test.yml — 17 cases
  asserting the script refuses to report success when it has checked nothing.

The mutation phase is not belt-and-braces, it is the point. Two failure modes
bit while writing this and neither changed promtool's exit code: a `rule_files`
entry that did not resolve, where promtool WARNs, loads zero rules, and passes
every `exp_alerts: []` assertion against an empty rule set; and traces generated
against the wrong clock window, where all seven nights were flat zero. Both were
green. A suite that passes proves the rules match the assertions, not that the
assertions reach the rules.

Production: nothing applied to the cluster. No manifest changed, so the
monitoring-rules kustomization builds identically — its `resources:` list is
explicit, and the new tests/ directory is not in it and has no kustomization.yaml
of its own, so Flux does not see it. Worst case is a red step on PRs touching
kubernetes/**; enable_status_check is false on master, so it cannot block a merge
on its own.

Verified with real promtool 3.15.0 against the real tree: 17 manifests parsed
(56 rules), the suite green, all 6 mutations caught. Run again under BusyBox
1.35.0 sh/awk/sed/find/xargs, the toolchain prom/prometheus actually ships.
Guard tests 17/17 under both bash and BusyBox sh. shellcheck 0.11.0 clean at
every severity. woodpecker-cli 3.18.0 lint clean on all 17 pipeline files.
Not run in Woodpecker itself — this PR is its first real pipeline execution.

Revert: revert this commit. It adds files and one step; nothing else reads them.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

.woodpecker/build-ci-images.yml

@@ -377,37 +377,6 @@ steps:
           --destination gitea.renner.uno/gitea_admin/netbird-service-sync:$TAG \
           --destination gitea.renner.uno/gitea_admin/netbird-service-sync:latest
 
-  # wiki-index: embedding index over the Notes repo's wiki/, fed from Gitea and
-  # queried by ~/Notes/bin/wiki_rank.py. Runtime artifact like netbird-service-sync.
-  - name: build-wiki-index
-    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
-    directory: ci/images/wiki-index
-    depends_on: []
-    when:
-      - event: deployment
-      - event: push
-        path: ["ci/images/wiki-index/**", ".woodpecker/build-ci-images.yml"]
-    environment:
-      GITEA_USER: gitea_admin
-      GITEA_TOKEN:
-        from_secret: ci_images_token
-    commands:
-      - mkdir -p /kaniko/.docker
-      - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
-      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
-      - TAG=$(date +%Y-%m-%d)
-      # Also a per-commit tag: several builds a day (branch pushes and the
-      # merge) all rewrite the dated tag, and the deployment pin raced them
-      # on 2026-09-20. The commit tag is immutable and needs no digest lookup.
-      - SHORT=$(printf '%s' "$CI_COMMIT_SHA" | cut -c1-8)
-      - |
-        /kaniko/executor \
-          --context . \
-          --dockerfile Containerfile \
-          --destination gitea.renner.uno/gitea_admin/wiki-index:$TAG \
-          --destination gitea.renner.uno/gitea_admin/wiki-index:$TAG-$SHORT \
-          --destination gitea.renner.uno/gitea_admin/wiki-index:latest
-
   # bird: BGP daemon for the pod-network mesh over pod0 (#576). Runtime
   # artifact like heating-can; same registry so the audit tracks it.
   - name: build-bird

.woodpecker/ci-manifests-test.yml

@@ -17,30 +17,43 @@
 # here is a static check that either passes or fails on the checkout alone, so it
 # can gate PRs without becoming a bottleneck or a flake source.
 when:
-  # Path-filtered on purpose, unlike ci-scripts-test.yml. That pipeline is one
-  # step on one already-cached image, so running it everywhere is free; this one
-  # pulls three images and builds every kustomization in the tree, which is real
+  # Path-filtered on purpose, unlike ci-scripts-test.yml. That pipeline is two
+  # steps on already-cached images, so running it everywhere is free; this one
+  # pulls four images and builds every kustomization in the tree, which is real
   # cost to put on a docs-only PR. The filter covers everything the steps actually read:
-  # kubernetes/** for the manifests and scripts, .woodpecker/** for the lint (and
-  # so an edit to this file re-runs its own checks).
+  # kubernetes/** for the manifests, scripts and promtool suites, .woodpecker/**
+  # for the lint (and so an edit to this file re-runs its own checks), and the
+  # promtool driver script by name.
+  #
+  # That last entry is NOT covered by kubernetes/**: promtool-rules.sh lives
+  # under scripts/, so without it a PR that only edits the script would never run
+  # the check it drives. It is listed as one file rather than scripts/ci/**
+  # because the rest of that directory is CI-agent helpers with nothing to do
+  # with this pipeline, and pulling four images for them is the cost this filter
+  # exists to avoid. Their own tests run in ci-scripts-test.yml on every PR.
   - event: pull_request
     path:
       - "kubernetes/**"
       - ".woodpecker/**"
+      - "scripts/ci/promtool-rules.sh"
 
-# ADDING A THIRD DISCOVERY CHECK: kustomize-build and shellcheck both hand-roll
-# the same shape — `find`, a floor guard against empty discovery, then a
-# file-fed `while read`. Two copies is deliberate: the two traps being commented
-# around (BusyBox find lacking `-printf`, and a piped loop running in a subshell
-# where `fail` cannot propagate) are different, and both are worth spelling out
-# where they bite. A third instance is the point to stop copying and factor it
-# into a shared script under `scripts/ci/` — which is already covered by
-# ci-scripts-test.yml, so it would gain unit tests in the process.
+# THE THIRD DISCOVERY CHECK WENT TO A SCRIPT. kustomize-build and shellcheck
+# both hand-roll the same shape — `find`, a floor guard against empty discovery,
+# then a file-fed `while read`. Two copies was deliberate: the two traps being
+# commented around (BusyBox find lacking `-printf`, and a piped loop running in
+# a subshell where `fail` cannot propagate) are different, and both are worth
+# spelling out where they bite. An earlier revision of this comment said a third
+# instance would be the point to stop copying and factor it into `scripts/ci/`,
+# where it would gain unit tests — and that is what `promtool` below does. Its
+# discovery, floors and guards live in scripts/ci/promtool-rules.sh and are
+# tested by scripts/ci/test-promtool-rules.sh on every PR via
+# ci-scripts-test.yml. A FOURTH inline copy should go the same way; do not add
+# one here.
 #
-# Every step carries `depends_on: []` so all four run in PARALLEL and every
+# Every step carries `depends_on: []` so all five run in PARALLEL and every
 # failure shows in one run. Woodpecker's default is sequential, which would stop
 # at the first failure and make a contributor fix-and-repush to discover the
-# next — and these four are independent, cheap, offline checks with no ordering
+# next — and these five are independent, cheap, offline checks with no ordering
 # between them. Keep it on every step: `depends_on` is per-step, so dropping it
 # from one silently re-serialises that step behind the others.
 steps:
@@ -258,3 +271,52 @@ steps:
           exit 1
         fi
         echo "ok    no digest pins under openbao/rotation/"
+
+  # Machine-check the alert rules: parse every PrometheusRule, run the committed
+  # promtool unit suites, then damage the rules on purpose and require the suites
+  # to notice. Everything it does lives in scripts/ci/promtool-rules.sh; read
+  # that file and kubernetes/apps/monitoring-rules/tests/README.md for the why.
+  #
+  # The gap it closes: the other four steps prove a PrometheusRule assembles into
+  # valid YAML and nothing more — not that an expression parses, not that a label
+  # join matches anything, not that a `for` window does what its comment claims.
+  # On REN-129 the CNPG surge rule was written `for: 2m`, which fires on six of
+  # seven measured nights and would have mailed the owner twice a night at 04:16
+  # and 04:22 local for a backlog that clears itself. Nothing here would have
+  # caught it, and the arithmetic that did catch it was hand-written from scratch
+  # twice, in two runs, and thrown away both times because there was nowhere in
+  # the repo to commit it. It is now mutations.txt:surge-for-2m.
+  #
+  # Offline, like the rest of this pipeline: no cluster, no registry, no secrets,
+  # no network. promtool reads files and exits.
+  #
+  # IMAGE — prom/prometheus, not a registry-local ci-* image, and pinned to an
+  # exact upstream version rather than a dated tag. The dated-tag convention
+  # guarded by no-digest-pins above is about OUR images, whose digests are
+  # garbage-collected out from under a pin; this is upstream and immutable.
+  #
+  # The version must be 3.x, and that is not a preference. The cluster runs
+  # whatever kube-prometheus-stack >= 72.0.0 ships, which is Prometheus 3, and
+  # alert `for` evaluation is version-sensitive — a 2.x promtool would be
+  # checking a different evaluator than production, so the margins the suites
+  # assert would be claims about the wrong program. promtool-rules.sh asserts the
+  # major version it was handed and fails on 2.x, so a careless bump here is a
+  # red step rather than a quietly wrong check. Keep this at the latest stable
+  # 3.x, per the standing rule that any software uses the latest stable version.
+  #
+  # This image is NOT covered by image-automation (same reason as
+  # woodpecker-lint above: a Woodpecker pipeline file has no apiVersion/kind, so
+  # an $imagepolicy marker here is silently ignored — fluxcd/flux2#2826). The
+  # bump is manual.
+  #
+  # `prom/prometheus` is busybox-based and ships /bin/sh, so it can run a
+  # `commands:` block — unlike the plain woodpecker-cli tag, which cannot. Its
+  # find/grep/awk/sed are BusyBox, which is why promtool-rules.sh avoids
+  # `find -printf` and `grep -r --include`; it was verified against BusyBox
+  # 1.35.0 as well as GNU. The image also declares `USER nobody`, so the script
+  # writes its stripped mirror under mktemp and never into the workspace.
+  - name: promtool
+    depends_on: [] # see note above `steps:`
+    image: prom/prometheus:v3.15.0
+    commands:
+      - sh scripts/ci/promtool-rules.sh

.woodpecker/ci-scripts-test.yml

@@ -8,15 +8,39 @@
 # against adversarial PR input. Running it here fails the offending PR's checks
 # before it can merge.
 #
+# scripts/ci/test-promtool-rules.sh is the same idea for the alert-rule gate.
+# What it tests is not whether promtool evaluates PromQL — ci-manifests-test.yml
+# does that with the real binary against the real rules — but whether
+# promtool-rules.sh REFUSES TO REPORT SUCCESS when it has checked nothing: zero
+# manifests discovered, zero suites discovered, a `rule_files` path that does not
+# resolve (promtool warns, loads zero rules, and passes every assertion), a
+# mutation the suites no longer catch. Those are the states a green pipeline
+# cannot tell apart from real coverage, so they need a test that asserts the
+# failure. It stubs promtool, so it needs neither the binary nor the heavy image.
+#
+# Both are unfiltered by path on purpose: this pipeline is two steps on one
+# already-cached image, so running it everywhere is free, and a guard whose test
+# only runs when someone remembers to touch the right directory is not a guard.
+#
 # Only in homelab (scripts/ci/ lives here); the file is not deployed to the
-# mirrored repos, so it does not run on their PRs. The test is offline (no token,
-# no network, no cluster) — the ci-claude image just provides sh + jq, and it is
-# already pulled by pr-review.yml on the same event, so the runner has it cached.
+# mirrored repos, so it does not run on their PRs. Both tests are offline (no
+# token, no network, no cluster) — the ci-claude image just provides sh + jq, and
+# it is already pulled by pr-review.yml on the same event, so the runner has it
+# cached.
 when:
   - event: pull_request
 
+# `depends_on: []` on both so they run in parallel and one failure does not hide
+# the other. Same reasoning as the note above `steps:` in ci-manifests-test.yml.
 steps:
   - name: model-selection-tests
+    depends_on: []
     image: gitea.renner.uno/gitea_admin/ci-claude:2.1.280
     commands:
       - sh scripts/ci/test-model-selection.sh
+
+  - name: promtool-rules-guard-tests
+    depends_on: []
+    image: gitea.renner.uno/gitea_admin/ci-claude:2.1.280
+    commands:
+      - sh scripts/ci/test-promtool-rules.sh

.woodpecker/deploy-netbird-server.yml

@@ -4,9 +4,6 @@ when:
     branch: master
     path:
       - "ansible/roles/netbird-server/**"
-      # install-netbird-server.yml also applies the netbird-proxy-watchdog role,
-      # so changes to it must redeploy this host too.
-      - "ansible/roles/netbird-proxy-watchdog/**"
       - "ansible/plays/install-netbird-server.yml"
       - ".woodpecker/deploy-netbird-server.yml"
   - event: deployment

.woodpecker/wiki-maintenance.yml

@@ -4,7 +4,7 @@
 # script (structural checks, no model), then consolidate (wiki-consolidate
 # agent: verifies a cohort of notes against sources, repos and the live
 # cluster and edits them). The pipeline commits and pushes as Claude Bot; the
-# push webhook then refreshes the wiki-index service.
+# push webhook then refreshes the wiki service.
 #
 # The agents address the vault as ~/Notes and the repos as ~/source/<repo>, so
 # each agent step links those into the workspace instead of rewriting them.

ansible/plays/install-netbird-proxy.yml

@@ -1,5 +1,5 @@
 ---
-# Deploy a NetBird reverse-proxy HA cluster member (a second proxy host).
+# Deploy the NetBird reverse-proxy HA cluster members (the public edge).
 # Usage: ansible-playbook -i inventory/ plays/install-netbird-proxy.yml
 #
 # Prerequisites:
@@ -7,12 +7,11 @@
 #   listed under [netbird_proxy] in inventory/pve_hosts.yml. Docker is
 #   installed by the (Arch-aware) docker-install role.
 # - DNS A/AAAA for the host's own name (e.g. proxy-1.renner.uno) -> its IP.
-# - Uses the existing vault_netbird_proxy_token (shared with VPS-1) — no new
-#   token to generate; a shared token + same NB_PROXY_DOMAIN forms the cluster.
-# - AFTER deploy + verify: add this host's IP to the proxy domain so traffic
-#   round-robins across the cluster. Repoint proxy.renner.uno away from its
-#   CNAME to netbird.renner.uno (management host) to A/AAAA records listing
-#   BOTH proxy IPs (VPS-1 + this host).
+# - Uses the existing vault_netbird_proxy_token (shared by every member) — no
+#   new token to generate; a shared token + same NB_PROXY_DOMAIN forms the
+#   cluster.
+# - AFTER deploy + verify: add this host's A/AAAA to proxy.renner.uno so
+#   traffic round-robins across the cluster.
 
 - name: Deploy NetBird reverse-proxy cluster member
   hosts: netbird_proxy

ansible/plays/install-netbird-server.yml

@@ -18,4 +18,3 @@
     - common
     - docker-install
     - netbird-server
-    - netbird-proxy-watchdog

ansible/plays/rotate-netbird-secrets.yml

@@ -43,10 +43,10 @@
 #
 # WHAT IS NOT, and must not be added:
 #
-#   vault_netbird_proxy_token    LOOKS symmetric — it is in proxy.env on both
-#                                the server and the proxies, all Ansible
-#                                templates — but the management server does not
-#                                validate it from any of them. It is registered
+#   vault_netbird_proxy_token    LOOKS symmetric — it is in proxy.env on the
+#                                proxies, an Ansible template — but the
+#                                management server does not validate it from
+#                                any file. It is registered
 #                                in the management DATABASE. Rotating it changes
 #                                only what the proxies present, so every proxy
 #                                is locked out: "proxy auth failed: too many

ansible/roles/netbird-proxy-watchdog/defaults/main.yml

@@ -1,5 +1,5 @@
 ---
-# Self-healing watchdog for the NetBird reverse proxy (VPS-1 and proxy-1).
+# Self-healing watchdog for the NetBird reverse proxy on the proxy hosts.
 #
 # Why it exists: the proxy's embedded NetBird client pins each proxied
 # domain's route to a specific routing-peer identity. The netbird-ingress
@@ -20,9 +20,7 @@ netbird_watchdog_probe_hosts:
   - home.renner.uno
   - immich.renner.uno
 
-# Address the probe connects to. 127.0.0.1 reaches the local proxy on both
-# members: VPS-1 routes it via Traefik's HostSNI(*) TCP passthrough, proxy-1
-# binds :443 directly.
+# Address the probe connects to. The proxy binds :443 on the host directly.
 netbird_watchdog_probe_address: 127.0.0.1
 
 netbird_watchdog_container: netbird-proxy

ansible/roles/netbird-proxy-watchdog/tasks/main.yml

@@ -1,7 +1,7 @@
 ---
 # Deploys the reverse-proxy watchdog (see defaults/main.yml for why).
 
-- name: Ensure curl is installed (probe dependency; proxy-1 is Arch, VPS-1 Debian)
+- name: Ensure curl is installed (probe dependency)
   ansible.builtin.package:
     name: curl
     state: present

ansible/roles/netbird-proxy/defaults/main.yml

@@ -1,8 +1,8 @@
 ---
 # NetBird reverse-proxy HA cluster member (proxy-only host, e.g. proxy-1).
-# Joins the same proxy cluster as the VPS-1 proxy: identical NB_PROXY_DOMAIN
-# + the same token. NetBird load-balances/fails the cluster over via DNS
-# round-robin on the proxy domain (see plays/install-netbird-proxy.yml).
+# Every member uses the same NB_PROXY_DOMAIN + token. NetBird load-balances/
+# fails the cluster over via DNS round-robin on the proxy domain (see
+# plays/install-netbird-proxy.yml).
 
 # MUST match the netbird-server role's netbird_version — the reverse-proxy
 # release line has to track the management server it enrolls against.
@@ -11,11 +11,13 @@ netbird_version: "0.73.2"
 # Management server this proxy enrolls against (public API, TLS).
 netbird_server_host: netbird.renner.uno
 
-# Proxy domain — MUST be identical across every cluster member.
+# Proxy domain — MUST be identical across every cluster member, and must NOT
+# be a subdomain of the management domain (netbird.renner.uno) per NetBird
+# docs (TLS/routing conflicts).
 netbird_proxy_domain: "proxy.renner.uno"
 
-# Reuse VPS-1's proxy token — a shared NB_PROXY_TOKEN across members with the
-# same NB_PROXY_DOMAIN is a valid HA cluster, so no separate token is needed.
+# A shared NB_PROXY_TOKEN across members with the same NB_PROXY_DOMAIN is a
+# valid HA cluster, so one token serves every member.
 netbird_proxy_token: "{{ vault_netbird_proxy_token }}"
 
 # Relay auth secret — MUST be identical across every relay instance and the

ansible/roles/netbird-proxy/templates/docker-compose.yml.j2

@@ -1,8 +1,7 @@
 # NetBird reverse-proxy — HA cluster member on a proxy-only host.
-# Just the reverse-proxy container, bound directly to :443. No Traefik:
-# unlike VPS-1 (where the proxy shares :443 with management/signal/relay via
-# SNI routing), this host runs nothing else, so the proxy owns the port and
-# terminates TLS + answers its own tls-alpn-01 ACME challenges directly.
+# Just the reverse-proxy container, bound directly to :443. No Traefik: the
+# proxy owns the port and terminates TLS + answers its own tls-alpn-01 ACME
+# challenges directly.
 services:
   proxy:
     image: netbirdio/reverse-proxy:{{ netbird_version }}

ansible/roles/netbird-server/defaults/main.yml

@@ -23,9 +23,3 @@ netbird_datastore_key: "{{ vault_netbird_datastore_key }}"
 
 # Relay secret for WebSocket relay authentication
 netbird_relay_secret: "{{ vault_netbird_relay_secret }}"
-
-# Reverse proxy configuration
-# IMPORTANT: Proxy domain must NOT be a subdomain of the management domain
-# (netbird.renner.uno) per NetBird docs - causes TLS/routing conflicts
-netbird_proxy_domain: "proxy.renner.uno"
-netbird_proxy_token: "{{ vault_netbird_proxy_token }}"

ansible/roles/netbird-server/handlers/main.yml

@@ -16,7 +16,3 @@
 - name: Restart netbird coturn
   ansible.builtin.shell: |
     cd /opt/netbird && docker compose up -d --force-recreate coturn
-
-- name: Restart netbird proxy
-  ansible.builtin.shell: |
-    cd /opt/netbird && docker compose up -d --force-recreate proxy

ansible/roles/netbird-server/tasks/main.yml

@@ -36,7 +36,6 @@
     - /opt/netbird/management
     - /opt/netbird/signal
     - /opt/netbird/letsencrypt
-    - /opt/netbird/proxy/certs
 
 - name: Deploy docker-compose.yml
   ansible.builtin.template:
@@ -59,23 +58,83 @@
     mode: "0644"
   notify: Restart netbird coturn
 
-- name: Deploy proxy.env
-  ansible.builtin.template:
-    src: proxy.env.j2
-    dest: /opt/netbird/proxy.env
-    mode: "0644"
-  notify: Restart netbird proxy
+# The public edge is the proxy hosts (plays/install-netbird-proxy.yml); this
+# host runs no reverse proxy. The tasks below remove the one it used to run,
+# its watchdogs first so they cannot restart it mid-removal, and can go once
+# the role has run against every host that ever carried it.
+- name: Check for reverse-proxy watchdog units
+  ansible.builtin.stat:
+    path: "/etc/systemd/system/{{ item }}"
+  loop:
+    - netbird-proxy-watchdog.timer
+    - netbird-proxy-flood-watchdog.service
+  register: _proxy_watchdog_units
+
+- name: Stop and disable reverse-proxy watchdogs
+  ansible.builtin.systemd:
+    name: "{{ item.item }}"
+    state: stopped
+    enabled: false
+  loop: "{{ _proxy_watchdog_units.results }}"
+  loop_control:
+    label: "{{ item.item }}"
+  when: item.stat.exists
+
+- name: Remove reverse-proxy watchdog units and scripts
+  ansible.builtin.file:
+    path: "{{ item }}"
+    state: absent
+  loop:
+    - /etc/systemd/system/netbird-proxy-watchdog.timer
+    - /etc/systemd/system/netbird-proxy-watchdog.service
+    - /etc/systemd/system/netbird-proxy-flood-watchdog.service
+    - /usr/local/bin/netbird-proxy-watchdog
+    - /usr/local/bin/netbird-proxy-flood-watchdog
+    - /run/netbird-proxy-watchdog
+  register: _proxy_watchdog_files
+
+- name: Reload systemd after removing the watchdog units
+  ansible.builtin.systemd:
+    daemon_reload: true
+  when: _proxy_watchdog_files.changed
 
 - name: Pull NetBird container images
   ansible.builtin.shell: |
     cd /opt/netbird && docker compose pull
   changed_when: false
 
+# --remove-orphans removes containers of services no longer in the compose
+# file (the reverse proxy).
 - name: Start NetBird containers
   ansible.builtin.shell: |
-    cd /opt/netbird && docker compose up -d
+    cd /opt/netbird && docker compose up -d --remove-orphans
   register: docker_compose_result
-  changed_when: "'Started' in docker_compose_result.stdout or 'Creating' in docker_compose_result.stdout"
+  changed_when: >-
+    'Started' in docker_compose_result.stdout or
+    'Creating' in docker_compose_result.stdout or
+    'Removed' in docker_compose_result.stdout or
+    'Removed' in docker_compose_result.stderr
+
+- name: Remove reverse-proxy volumes
+  ansible.builtin.shell: |
+    set -eu
+    for v in netbird_netbird_proxy_certs netbird_netbird_proxy_data netbird_netbird_proxy_state; do
+      if [ -n "$(docker volume ls -q --filter "name=^${v}$")" ]; then
+        docker volume rm "$v"
+        echo "removed $v"
+      fi
+    done
+  register: _proxy_volumes
+  changed_when: "'removed' in _proxy_volumes.stdout"
+
+- name: Remove reverse-proxy config and state
+  ansible.builtin.file:
+    path: "{{ item }}"
+    state: absent
+  loop:
+    - /opt/netbird/proxy.env
+    - /opt/netbird/proxy
+    - /opt/netbird/proxy-state
 
 # The management store has no cluster-side backup. Pulling one put a root
 # key on this host that the cluster held, and this server's own state is

ansible/roles/netbird-server/templates/docker-compose.yml.j2

@@ -187,10 +187,9 @@ services:
       - "traefik.http.routers.netbird-management.service=netbird-management"
       - "traefik.http.services.netbird-management.loadbalancer.server.port=33073"
       - "traefik.http.services.netbird-management.loadbalancer.server.scheme=h2c"
-      # gRPC reverse-proxy service — remote reverse-proxy instances (e.g.
-      # proxy-1) connect here via NB_PROXY_MANAGEMENT_ADDRESS=https://host:443.
-      # The local proxy uses management:33073 directly, so it never needed
-      # this route; remote ones get a 404 without it. Same backend (h2c 33073).
+      # gRPC reverse-proxy service — the proxy hosts (proxy-1, proxy-2)
+      # connect here via NB_PROXY_MANAGEMENT_ADDRESS=https://host:443 and get
+      # a 404 without it. Same backend (h2c 33073).
       - "traefik.http.routers.netbird-proxyservice.rule=Host(`{{ netbird_server_host }}`) && PathPrefix(`/management.ProxyService/`)"
       - "traefik.http.routers.netbird-proxyservice.entrypoints=websecure"
       - "traefik.http.routers.netbird-proxyservice.tls=true"
@@ -221,37 +220,6 @@ services:
         max-size: "500m"
         max-file: "2"
 
-  # Reverse Proxy Service (per docs.netbird.io/selfhosted/migration/enable-reverse-proxy)
-  proxy:
-    image: netbirdio/reverse-proxy:{{ netbird_version }}
-    container_name: netbird-proxy
-    restart: unless-stopped
-    networks:
-      - netbird
-    depends_on:
-      - management
-    env_file:
-      - ./proxy.env
-    volumes:
-      - netbird_proxy_certs:/certs
-      - /opt/netbird/proxy-state:/var/lib/netbird
-    labels:
-      - "traefik.enable=true"
-      - "traefik.tcp.routers.proxy-passthrough.entrypoints=websecure"
-      - "traefik.tcp.routers.proxy-passthrough.rule=HostSNI(`*`)"
-      - "traefik.tcp.routers.proxy-passthrough.tls.passthrough=true"
-      - "traefik.tcp.routers.proxy-passthrough.service=proxy-tls"
-      - "traefik.tcp.routers.proxy-passthrough.priority=1"
-      - "traefik.tcp.services.proxy-tls.loadbalancer.server.port=8443"
-    logging:
-      driver: "json-file"
-      options:
-        max-size: "500m"
-        max-file: "2"
-
 networks:
   netbird:
     driver: bridge
-
-volumes:
-  netbird_proxy_certs:

ansible/roles/netbird-server/templates/proxy.env.j2

@@ -1,8 +0,0 @@
-NB_PROXY_DOMAIN={{ netbird_proxy_domain }}
-NB_PROXY_TOKEN={{ netbird_proxy_token }}
-NB_PROXY_MANAGEMENT_ADDRESS=http://management:33073
-NB_PROXY_ALLOW_INSECURE=true
-NB_PROXY_ADDRESS=:8443
-NB_PROXY_ACME_CERTIFICATES=true
-NB_PROXY_ACME_CHALLENGE_TYPE=tls-alpn-01
-NB_PROXY_CERTIFICATE_DIRECTORY=/certs

ci/images/wiki-index/Containerfile

@@ -1,10 +0,0 @@
-# wiki-index: embedding index over the Notes repo's wiki/, fed from Gitea,
-# queried by ~/Notes/bin/wiki_rank.py. Standard library only; runs unprivileged.
-FROM python:3.13-slim
-
-RUN useradd --system --no-create-home --shell /usr/sbin/nologin wikiindex
-COPY wiki_index.py /opt/wiki_index.py
-
-ENV PYTHONUNBUFFERED=1
-USER wikiindex
-ENTRYPOINT ["python3", "/opt/wiki_index.py"]

ci/images/wiki-index/test_wiki_index.py

@@ -1,178 +0,0 @@
-#!/usr/bin/env python3
-"""Unit tests for wiki_index. Run: python3 -m unittest test_wiki_index"""
-
-import hashlib
-import hmac
-import json
-import tempfile
-import unittest
-from pathlib import Path
-
-import wiki_index as wi
-
-NOTE = """---
-title: 'UDM intercepts DNS'
-type: wiki
-tags: [dns, unifi]
-aliases: [udm dns interception, dig 1.1.1.1 wrong]
----
-
-# UDM intercepts DNS
-
-The UDM answers every UDP/53 query itself.
-
-## Details
-
-More.
-"""
-
-
-class ParseTests(unittest.TestCase):
-    def test_note_text(self):
-        title, text = wi.note_text("udm-dns", NOTE)
-        self.assertEqual(title, "UDM intercepts DNS")
-        self.assertIn("udm dns UDM intercepts DNS udm dns interception", text)
-        self.assertIn("dns unifi", text)
-        self.assertTrue(text.endswith("The UDM answers every UDP/53 query itself."))
-        self.assertLessEqual(len(text), wi.EMBED_CHARS)
-
-    def test_credentials_are_stripped(self):
-        original = wi.http
-        wi.http = lambda method, url, body=None, headers=None, timeout=60.0: (
-            {"auth": {"client_token": "t"}} if url.endswith("/login")
-            else {"data": {"data": {"gitea_token": "abc\n", "webhook_secret": " s3cret\n"}}})
-        original_sa = wi.SA_TOKEN
-        try:
-            with tempfile.TemporaryDirectory() as d:
-                wi.SA_TOKEN = Path(d) / "token"
-                wi.SA_TOKEN.write_text("jwt\n")
-                self.assertEqual(wi.Credentials().fetch(), {"gitea_token": "abc", "webhook_secret": "s3cret"})
-        finally:
-            wi.http, wi.SA_TOKEN = original, original_sa
-
-    def test_tree_paging_stops_on_short_last_page(self):
-        # Gitea: truncated=true on every page including the last, "tree": null past it.
-        pages = {
-            1: {"tree": [{"path": f"wiki/n{i}.md", "type": "blob", "sha": f"s{i}"} for i in range(1000)], "truncated": True, "total_count": 1192},
-            2: {"tree": [{"path": f"wiki/n{i}.md", "type": "blob", "sha": f"s{i}"} for i in range(1000, 1192)] + [{"path": "raw/x.md", "type": "blob", "sha": "r"}, {"path": "wiki/sub/y.md", "type": "blob", "sha": "y"}], "truncated": True, "total_count": 1192},
-            3: {"tree": None, "truncated": False, "total_count": 1192},
-        }
-        calls = []
-        original = wi.http
-
-        def fake_http(method, url, body=None, headers=None, timeout=60.0):
-            page = int(url.rsplit("page=", 1)[1])
-            calls.append(page)
-            return pages[page]
-
-        wi.http = fake_http
-        try:
-            tree = wi.Gitea("t").tree("abc")
-        finally:
-            wi.http = original
-        self.assertEqual(calls, [1, 2])
-        self.assertEqual(len(tree), 1192)
-        self.assertNotIn("raw/x.md", tree)
-        self.assertNotIn("wiki/sub/y.md", tree)
-        self.assertEqual(tree["wiki/n1191.md"], "s1191")
-
-    def test_signature(self):
-        body = b'{"ref":"refs/heads/master"}'
-        sig = hmac.new(b"s3cret", body, hashlib.sha256).hexdigest()
-        self.assertTrue(wi.verify_signature("s3cret", body, sig))
-        self.assertFalse(wi.verify_signature("other", body, sig))
-        self.assertFalse(wi.verify_signature("s3cret", body, ""))
-
-
-class FakeGitea:
-    def __init__(self, head, tree, files):
-        self._head, self._tree, self.files = head, tree, files
-        self.raw_calls = []
-
-    def head(self):
-        return self._head
-
-    def tree(self, commit):
-        return self._tree
-
-    def raw(self, path, ref):
-        self.raw_calls.append(path)
-        return self.files[path]
-
-
-class IndexTests(unittest.TestCase):
-    def setUp(self):
-        self.tmp = tempfile.TemporaryDirectory()
-        self.path = Path(self.tmp.name) / "index.json"
-        self.original_embed = wi.embed
-        self.embed_calls = []
-
-        def fake_embed(texts, batch=2):
-            self.embed_calls.append(list(texts))
-            return [wi.unit([1.0, float(len(t) % 7)]) for t in texts]
-
-        wi.embed = fake_embed
-
-    def tearDown(self):
-        wi.embed = self.original_embed
-        self.tmp.cleanup()
-
-    def test_sync_is_incremental_and_persists(self):
-        files = {"wiki/udm-dns.md": NOTE, "wiki/ceph.md": "---\ntitle: Ceph\n---\n# Ceph\n\nOSD.\n", "wiki/_index.md": "---\ntitle: Wiki\n---\n"}
-        tree = {"wiki/udm-dns.md": "aaa", "wiki/ceph.md": "bbb", "wiki/_index.md": "ccc"}
-        g = FakeGitea("c1", tree, files)
-        idx = wi.Index(self.path)
-        s = idx.sync(g)
-        self.assertEqual((s["changed"], s["removed"], s["notes"]), (2, 0, 2))
-        self.assertEqual(set(idx.notes), {"udm-dns", "ceph"})
-        self.assertEqual(idx.notes["udm-dns"]["title"], "UDM intercepts DNS")
-        self.assertTrue(self.path.exists())
-
-        # head unchanged: nothing fetched
-        s = idx.sync(g)
-        self.assertEqual(s.get("skipped"), "head unchanged")
-        self.assertEqual(len(g.raw_calls), 2)
-
-        # one note changed, one removed, new head
-        g2 = FakeGitea("c2", {"wiki/udm-dns.md": "aaa2"}, {"wiki/udm-dns.md": NOTE.replace("itself", "itself!")})
-        s = idx.sync(g2)
-        self.assertEqual((s["changed"], s["removed"], s["notes"]), (1, 1, 1))
-        self.assertEqual(g2.raw_calls, ["wiki/udm-dns.md"])
-
-        # restart loads the persisted index
-        idx2 = wi.Index(self.path)
-        self.assertEqual(idx2.commit, "c2")
-        self.assertEqual(set(idx2.notes), {"udm-dns"})
-
-    def test_rank_orders_by_cosine(self):
-        idx = wi.Index(self.path)
-        idx.notes = {"a": {"sha": "1", "title": "A", "vector": wi.unit([1.0, 0.0])},
-                     "b": {"sha": "2", "title": "B", "vector": wi.unit([0.0, 1.0])}}
-        wi.embed = lambda texts, batch=2: [wi.unit([0.2, 1.0])]
-        rows = idx.rank("q", k=5)
-        self.assertEqual([r["slug"] for r in rows], ["b", "a"])
-        self.assertGreater(rows[0]["score"], rows[1]["score"])
-
-    def test_status_reports_age_and_errors(self):
-        idx = wi.Index(self.path)
-        self.assertFalse(idx.status()["ready"])
-        g = FakeGitea("c1", {"wiki/a.md": "x"}, {"wiki/a.md": "---\ntitle: A\n---\n# A\n\nt\n"})
-        idx.sync(g)
-        st = idx.status()
-        self.assertTrue(st["ready"])
-        self.assertEqual(st["notes"], 1)
-        self.assertEqual(st["commit"], "c1")
-
-        class Broken(FakeGitea):
-            def head(self):
-                raise wi.ApiError("gitea down")
-
-        with self.assertRaises(wi.ApiError):
-            idx.sync(Broken("c1", {}, {}))
-        self.assertIn("gitea down", idx.status()["last_error"])
-        self.assertEqual(idx.status()["notes"], 1)  # last good index kept
-        self.assertTrue(idx.status()["ready"])  # and still served
-
-
-if __name__ == "__main__":
-    unittest.main()

ci/images/wiki-index/wiki_index.py

@@ -1,408 +0,0 @@
-#!/usr/bin/env python3
-"""wiki-index: embedding index over the Notes repo's wiki/, fed from Gitea.
-
-Source of truth is the notes repository on Gitea. The index holds, per wiki
-note, the git blob sha, title and a bge-m3 vector of slug/title/aliases/tags/
-first paragraph. A sync fetches the repo tree, embeds every note whose blob sha
-changed (via wiki-embed, in-cluster) and drops notes that left the tree; it
-runs at start, on every push webhook from Gitea, and every RESYNC_SECONDS.
-
-HTTP:
-  POST /rank     {"query": str, "k": int}   -> {"rows": [{"slug","title","score"}], "index": {...}}
-  GET  /health   -> 200 always (liveness) with {"ready", "notes", "commit", "synced_at", "age_s", "syncing", "last_error"}
-  GET  /ready    -> 200 when an index exists, else 503 (readiness)
-  POST /webhook  Gitea push event, HMAC-SHA256 in X-Gitea-Signature -> {"queued": true}
-
-Credentials (Gitea token, webhook secret) are read from OpenBao through the
-Kubernetes auth role on every use; they never land in a Secret or a log line.
-Standard library only.
-"""
-
-from __future__ import annotations
-
-import hashlib
-import hmac
-import json
-import logging
-import os
-import re
-import sys
-import threading
-import time
-import urllib.error
-import urllib.parse
-import urllib.request
-from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
-from pathlib import Path
-
-LOG = logging.getLogger("wiki-index")
-
-GITEA_URL = os.environ.get("GITEA_URL", "https://gitea.renner.uno").rstrip("/")
-NOTES_REPO = os.environ.get("NOTES_REPO", "k.renner/notes")
-NOTES_BRANCH = os.environ.get("NOTES_BRANCH", "master")
-WIKI_DIR = os.environ.get("WIKI_DIR", "wiki").strip("/")
-EMBED_URL = os.environ.get("EMBED_URL", "http://wiki-embed.ai.svc.cluster.local:8080").rstrip("/")
-EMBED_MODEL = os.environ.get("EMBED_MODEL", "bge-m3")
-EMBED_CHARS = int(os.environ.get("EMBED_CHARS", "1500"))
-INDEX_PATH = Path(os.environ.get("INDEX_PATH", "/data/index.json"))
-RESYNC_SECONDS = int(os.environ.get("RESYNC_SECONDS", "1800"))
-LISTEN_PORT = int(os.environ.get("LISTEN_PORT", "8080"))
-BAO_ADDR = os.environ.get("BAO_ADDR", "http://openbao.openbao.svc.cluster.local:8200").rstrip("/")
-BAO_ROLE = os.environ.get("BAO_ROLE", "wiki-index")
-BAO_SECRET_PATH = os.environ.get("BAO_SECRET_PATH", "homelab/data/apps/ai/wiki-index")
-SA_TOKEN = Path(os.environ.get("SA_TOKEN", "/var/run/secrets/kubernetes.io/serviceaccount/token"))
-
-
-class ApiError(Exception):
-    pass
-
-
-# ---------------------------------------------------------------- http helpers
-
-
-def http(method: str, url: str, body=None, headers=None, timeout: float = 60.0):
-    data = None
-    hdrs = dict(headers or {})
-    if body is not None:
-        data = json.dumps(body).encode("utf-8")
-        hdrs.setdefault("Content-Type", "application/json")
-    req = urllib.request.Request(url, data=data, method=method, headers=hdrs)
-    try:
-        with urllib.request.urlopen(req, timeout=timeout) as resp:
-            raw = resp.read()
-            ctype = resp.headers.get("Content-Type", "")
-    except urllib.error.HTTPError as e:
-        raise ApiError(f"{method} {url}: HTTP {e.code}: {e.read().decode('utf-8', 'replace')[:300]}") from e
-    except urllib.error.URLError as e:
-        raise ApiError(f"{method} {url}: {e.reason}") from e
-    if "json" in ctype:
-        return json.loads(raw.decode("utf-8"))
-    return raw
-
-
-# ---------------------------------------------------------------- credentials
-
-
-class Credentials:
-    """Gitea token and webhook secret from OpenBao, fetched per use."""
-
-    def __init__(self, static: dict | None = None):
-        self.static = static  # tests / local runs
-
-    def fetch(self) -> dict:
-        if self.static is not None:
-            return self.static
-        jwt = SA_TOKEN.read_text(encoding="utf-8").strip()
-        login = http("POST", f"{BAO_ADDR}/v1/auth/kubernetes/login", {"role": BAO_ROLE, "jwt": jwt})
-        token = login["auth"]["client_token"]
-        kv = http("GET", f"{BAO_ADDR}/v1/{BAO_SECRET_PATH}", headers={"X-Vault-Token": token})
-        data = {k: str(v).strip() for k, v in kv["data"]["data"].items()}  # `bao kv put key=@file` keeps the file's newline
-        for key in ("gitea_token", "webhook_secret"):
-            if not data.get(key):
-                raise ApiError(f"OpenBao {BAO_SECRET_PATH} has no key {key}")
-        return data
-
-
-# ---------------------------------------------------------------- gitea
-
-
-class Gitea:
-    def __init__(self, token: str, base: str = GITEA_URL, repo: str = NOTES_REPO, branch: str = NOTES_BRANCH):
-        self.base, self.repo, self.branch = base, repo, branch
-        self.headers = {"Authorization": f"token {token}"}
-
-    def api(self, path: str, **params):
-        q = ("?" + urllib.parse.urlencode(params)) if params else ""
-        return http("GET", f"{self.base}/api/v1/repos/{self.repo}{path}{q}", headers=self.headers)
-
-    def head(self) -> str:
-        return self.api(f"/branches/{urllib.parse.quote(self.branch, safe='')}")["commit"]["id"]
-
-    def tree(self, commit: str) -> dict[str, str]:
-        """path -> blob sha for every wiki note at that commit."""
-        out: dict[str, str] = {}
-        page, per_page, seen = 1, 1000, 0
-        while True:
-            t = self.api(f"/git/trees/{commit}", recursive="true", per_page=per_page, page=page)
-            entries = t.get("tree") or []  # Gitea answers "tree": null past the last page
-            for e in entries:
-                p = e.get("path", "")
-                if e.get("type") == "blob" and p.startswith(WIKI_DIR + "/") and p.endswith(".md") and "/" not in p[len(WIKI_DIR) + 1:]:
-                    out[p] = e["sha"]
-            seen += len(entries)
-            total = int(t.get("total_count") or 0)
-            if not entries or len(entries) < per_page or (total and seen >= total):
-                break
-            page += 1
-            if page > 50:
-                raise ApiError("tree paging runaway")
-        return out
-
-    def raw(self, path: str, ref: str) -> str:
-        data = http("GET", f"{self.base}/api/v1/repos/{self.repo}/raw/{urllib.parse.quote(path)}?ref={urllib.parse.quote(ref, safe='')}",
-                    headers=self.headers)
-        return data.decode("utf-8") if isinstance(data, bytes) else json.dumps(data)
-
-
-# ---------------------------------------------------------------- notes
-
-
-def parse_frontmatter(text: str) -> tuple[dict, str]:
-    m = re.match(r"^---[ \t]*\n(.*?)\n---[ \t]*(?:\n|$)", text, re.DOTALL)
-    if not m:
-        return {}, text
-    fm: dict = {}
-    for line in m.group(1).splitlines():
-        mm = re.match(r"^([A-Za-z_][\w-]*):\s*(.*)$", line)
-        if mm:
-            fm[mm.group(1)] = mm.group(2).strip()
-    return fm, text[m.end():]
-
-
-def as_list(value: str) -> list[str]:
-    s = (value or "").strip()
-    if s.startswith("[") and s.endswith("]"):
-        s = s[1:-1]
-    return [p.strip().strip("'\"") for p in s.split(",") if p.strip()]
-
-
-def first_paragraph(body: str) -> str:
-    para: list[str] = []
-    seen_h1 = False
-    for line in body.splitlines():
-        if line.startswith("# "):
-            seen_h1 = True
-            continue
-        if not seen_h1:
-            continue
-        if line.strip() == "" or line.startswith("#"):
-            if para:
-                break
-            continue
-        para.append(line.strip())
-    return " ".join(para)
-
-
-def note_text(slug: str, text: str) -> tuple[str, str]:
-    """(title, embedding text) for one note file."""
-    fm, body = parse_frontmatter(text)
-    title = (fm.get("title") or slug).strip().strip("'\"")
-    parts = [slug.replace("-", " "), title, " ".join(as_list(fm.get("aliases", ""))),
-             " ".join(as_list(fm.get("tags", ""))), first_paragraph(body)]
-    return title, " ".join(parts)[:EMBED_CHARS]
-
-
-# ---------------------------------------------------------------- embeddings
-
-
-def embed(texts: list[str], batch: int = 2) -> list[list[float]]:
-    out: list[list[float]] = []
-    for i in range(0, len(texts), batch):
-        resp = http("POST", f"{EMBED_URL}/v1/embeddings", {"model": EMBED_MODEL, "input": texts[i:i + batch]}, timeout=120)
-        rows = sorted(resp["data"], key=lambda d: d["index"])
-        out.extend(unit(d["embedding"]) for d in rows)
-    return out
-
-
-def unit(v: list[float]) -> list[float]:
-    n = sum(x * x for x in v) ** 0.5 or 1.0
-    return [round(x / n, 5) for x in v]
-
-
-# ---------------------------------------------------------------- index
-
-
-class Index:
-    def __init__(self, path: Path):
-        self.path = path
-        self.lock = threading.Lock()
-        self.notes: dict[str, dict] = {}  # slug -> {sha, title, vector}
-        self.commit = ""
-        self.synced_at = 0.0
-        self.syncing = False
-        self.last_error = ""
-        if path.exists():
-            try:
-                d = json.loads(path.read_text(encoding="utf-8"))
-                self.notes, self.commit, self.synced_at = d.get("notes", {}), d.get("commit", ""), d.get("synced_at", 0.0)
-                LOG.info("loaded index: %d notes at %s", len(self.notes), self.commit[:8])
-            except (OSError, ValueError) as e:
-                LOG.warning("index unreadable (%s); starting empty", e)
-
-    def save(self) -> None:
-        self.path.parent.mkdir(parents=True, exist_ok=True)
-        tmp = self.path.with_suffix(".tmp")
-        tmp.write_text(json.dumps({"notes": self.notes, "commit": self.commit, "synced_at": self.synced_at}), encoding="utf-8")
-        tmp.replace(self.path)
-
-    def sync(self, gitea: Gitea) -> dict:
-        """Bring the index to the branch head; returns a summary."""
-        with self.lock:
-            self.syncing = True
-        try:
-            head = gitea.head()
-            if head == self.commit and self.notes:
-                summary = {"commit": head, "changed": 0, "removed": 0, "notes": len(self.notes), "skipped": "head unchanged"}
-                self.synced_at = time.time()
-                self.save()
-                return summary
-            tree = gitea.tree(head)
-            slugs = {p[len(WIKI_DIR) + 1:-3]: (p, sha) for p, sha in tree.items() if not p.endswith("/_index.md")}
-            todo = [s for s, (p, sha) in slugs.items() if self.notes.get(s, {}).get("sha") != sha]
-            removed = [s for s in list(self.notes) if s not in slugs]
-            LOG.info("sync %s: %d notes, %d to embed, %d removed", head[:8], len(slugs), len(todo), len(removed))
-            fresh: dict[str, dict] = {}
-            for i in range(0, len(todo), 20):
-                chunk = todo[i:i + 20]
-                texts, titles = [], {}
-                for s in chunk:
-                    title, text = note_text(s, gitea.raw(slugs[s][0], head))
-                    titles[s] = title
-                    texts.append(text)
-                for s, vec in zip(chunk, embed(texts)):
-                    fresh[s] = {"sha": slugs[s][1], "title": titles[s], "vector": vec}
-            with self.lock:
-                for s in removed:
-                    self.notes.pop(s, None)
-                self.notes.update(fresh)
-                self.commit = head
-                self.synced_at = time.time()
-                self.last_error = ""
-                self.save()
-            return {"commit": head, "changed": len(todo), "removed": len(removed), "notes": len(self.notes)}
-        except Exception as e:
-            with self.lock:
-                self.last_error = str(e)[:300]
-            raise
-        finally:
-            with self.lock:
-                self.syncing = False
-
-    def rank(self, query: str, k: int = 60) -> list[dict]:
-        q = embed([query])[0]
-        with self.lock:
-            rows = [(s, n["title"], sum(a * b for a, b in zip(q, n["vector"]))) for s, n in self.notes.items()]
-        rows.sort(key=lambda r: -r[2])
-        return [{"slug": s, "title": t, "score": round(sc, 4)} for s, t, sc in rows[:k]]
-
-    def status(self) -> dict:
-        """`ready` = an index exists and can answer /rank; a failed sync keeps the last good index."""
-        with self.lock:
-            return {"ready": bool(self.notes), "notes": len(self.notes), "commit": self.commit,
-                    "synced_at": self.synced_at, "age_s": round(time.time() - self.synced_at) if self.synced_at else None,
-                    "syncing": self.syncing, "last_error": self.last_error or None}
-
-
-# ---------------------------------------------------------------- syncer
-
-
-class Syncer(threading.Thread):
-    """One worker: syncs on demand (webhook) and every RESYNC_SECONDS."""
-
-    def __init__(self, index: Index, creds: Credentials, interval: int):
-        super().__init__(daemon=True, name="syncer")
-        self.index, self.creds, self.interval = index, creds, interval
-        self.wake = threading.Event()
-
-    def trigger(self) -> None:
-        self.wake.set()
-
-    RETRY_SECONDS = 60
-
-    def run(self) -> None:
-        while True:
-            try:
-                data = self.creds.fetch()
-                summary = self.index.sync(Gitea(data["gitea_token"]))
-                LOG.info("sync done: %s", summary)
-                wait = self.interval
-            except Exception as e:  # keep serving the last good index; retry soon, not next interval
-                LOG.error("sync failed: %s", e)
-                wait = self.RETRY_SECONDS
-            self.wake.wait(wait)
-            self.wake.clear()
-
-
-def verify_signature(secret: str, body: bytes, signature: str) -> bool:
-    digest = hmac.new(secret.encode("utf-8"), body, hashlib.sha256).hexdigest()
-    return hmac.compare_digest(digest, (signature or "").strip())
-
-
-# ---------------------------------------------------------------- server
-
-
-def make_handler(index: Index, syncer: Syncer, creds: Credentials):
-    class Handler(BaseHTTPRequestHandler):
-        def log_message(self, fmt, *args):
-            LOG.info("%s %s", self.address_string(), fmt % args)
-
-        def send_json(self, code: int, obj) -> None:
-            data = json.dumps(obj).encode("utf-8")
-            self.send_response(code)
-            self.send_header("Content-Type", "application/json")
-            self.send_header("Content-Length", str(len(data)))
-            self.end_headers()
-            self.wfile.write(data)
-
-        def body(self) -> bytes:
-            n = int(self.headers.get("Content-Length") or 0)
-            return self.rfile.read(n) if n else b""
-
-        def do_GET(self):
-            if self.path == "/health":  # liveness: the process serves
-                return self.send_json(200, index.status())
-            if self.path == "/ready":  # readiness: an index exists
-                st = index.status()
-                return self.send_json(200 if st["ready"] else 503, st)
-            self.send_json(404, {"error": "not found"})
-
-        def do_POST(self):
-            if self.path == "/rank":
-                try:
-                    req = json.loads(self.body() or b"{}")
-                    query = str(req.get("query", "")).strip()
-                    k = max(1, min(int(req.get("k", 60)), 500))
-                    if not query:
-                        return self.send_json(400, {"error": "query required"})
-                    rows = index.rank(query, k)
-                    return self.send_json(200, {"rows": rows, "index": index.status()})
-                except ApiError as e:
-                    return self.send_json(502, {"error": str(e)})
-                except (ValueError, TypeError) as e:
-                    return self.send_json(400, {"error": str(e)})
-            if self.path == "/webhook":
-                raw = self.body()
-                try:
-                    secret = creds.fetch()["webhook_secret"]
-                except ApiError as e:
-                    return self.send_json(503, {"error": str(e)})
-                if not verify_signature(secret, raw, self.headers.get("X-Gitea-Signature", "")):
-                    return self.send_json(403, {"error": "bad signature"})
-                try:
-                    event = json.loads(raw or b"{}")
-                except ValueError:
-                    return self.send_json(400, {"error": "bad json"})
-                ref = event.get("ref", "")
-                if ref and ref != f"refs/heads/{NOTES_BRANCH}":
-                    return self.send_json(200, {"queued": False, "ref": ref})
-                syncer.trigger()
-                return self.send_json(200, {"queued": True})
-            self.send_json(404, {"error": "not found"})
-
-    return Handler
-
-
-def main() -> int:
-    logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s", stream=sys.stdout)
-    index = Index(INDEX_PATH)
-    creds = Credentials()
-    syncer = Syncer(index, creds, RESYNC_SECONDS)
-    syncer.start()
-    server = ThreadingHTTPServer(("", LISTEN_PORT), make_handler(index, syncer, creds))
-    LOG.info("listening on :%d, repo %s@%s, embed %s", LISTEN_PORT, NOTES_REPO, NOTES_BRANCH, EMBED_URL)
-    server.serve_forever()
-    return 0
-
-
-if __name__ == "__main__":
-    sys.exit(main())

docs/superpowers/plans/2026-09-25-paperclip.md

@@ -0,0 +1,976 @@
+# Paperclip Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Deploy Paperclip (server-local agent runs, option B) into the homelab cluster behind an Authentik gate, with CNPG Postgres and OpenBao-sourced secrets.
+
+**Architecture:** One Flux `Kustomization` (`paperclip`) applies `kubernetes/apps/paperclip/`: a single-replica Deployment on a ceph-block PVC, CNPG `pclp-pg` (PG17) with barman backups to Hetzner, and a Traefik Ingress that backends only the `paperclip-gate` Authentik outpost. Namespace, baseline CNP, ClusterSecretStore allowlist and image automation live in the existing `infrastructure` tree.
+
+**Tech Stack:** Flux v2 (kustomize-controller, image-reflector v1.2.5), CNPG + barman-cloud plugin, External Secrets → OpenBao, Cilium CNP, Traefik, Authentik blueprints.
+
+**Spec:** `docs/superpowers/specs/2026-09-25-paperclip-design.md`
+
+## Global Constraints
+
+- Worktree: `/home/krenner/source/homelab-worktrees/paperclip`, branch `feat/paperclip`. Never edit the main checkout.
+- Commits as claude-bot: `git -c user.name='Claude Bot' -c user.email='claude-bot@renner.uno' commit …`, message ends with `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`.
+- Push: `git -c credential.helper='store --file=/home/krenner/.config/claude-bot/creds' push …`. PR via `tea pr create --login claude-bot`. Never self-merge.
+- One Bash command per call: no `&&`, `;`, `|` chaining, no `2>/dev/null`, no `python -c`.
+- Namespace `paperclip`; hostname `paperclip.renner.uno`; LAN-only (no `dns.renner.uno/public` label).
+- Image `ghcr.io/paperclipai/paperclip`, tag `nightly`, digest-pinned, `digestReflectionPolicy: Always`.
+- Postgres: CNPG `pclp-pg`, `ghcr.io/cloudnative-pg/postgresql:17`, 2 instances, `ceph-block`.
+- PVC `paperclip-data`, `ceph-block`, 20Gi, mounted at `/paperclip`.
+- OpenBao key `apps/paperclip` with fields `BETTER_AUTH_SECRET`, `PAPERCLIP_TOOL_ACTION_SIGNING_SECRET`, `PAPERCLIP_SECRETS_MASTER_KEY` (64-char hex), `CLAUDE_CODE_OAUTH_TOKEN`.
+- Env: `PAPERCLIP_DEPLOYMENT_MODE=authenticated`, `PAPERCLIP_DEPLOYMENT_EXPOSURE=private`, `PAPERCLIP_PUBLIC_URL=https://paperclip.renner.uno`, `HOST=0.0.0.0`, `PORT=3100`.
+- Manifest comments: one terse factual line; rationale goes in commit messages/PR.
+- No `kubectl apply`/`exec`/`rollout restart` against the live cluster; Flux applies after merge. Server-side `--dry-run=server` is allowed.
+
+## Review Focus
+
+1. **Pod replaced mid agent run** (image bump, secret rotation via Reloader, node drain) — `Recreate` kills in-flight runs; expected: Paperclip marks them failed/recoverable on restart, not wedged. Covered in Task 9 Step 6.
+2. **Shared OAuth token expired or missing** — expected: agent runs fail with a visible auth error in Paperclip, server stays healthy. Covered in Task 9 Step 7.
+3. **Direct access bypassing the gate** from another namespace pod — expected: connection refused/dropped by CNP. Covered in Task 9 Step 4.
+4. **Unauthenticated LAN request** to `paperclip.renner.uno` — expected: redirect to Authentik, never Paperclip's UI. Covered in Task 9 Step 3.
+5. **Nightly image with a breaking DB migration** — expected: a base backup exists before the first bump lands. Covered in Task 9 Step 5.
+
+---
+
+### Task 1: Upstream facts that decide two manifest values
+
+No repo changes; produces two decisions consumed by Task 6.
+
+**Interfaces:**
+- Produces: `SIGNUP_DISABLED` (`"true"` or `"false"`), `CONFIG_PARTIAL_OK` (yes/no), `ENTRYPOINT_NEEDS_ROOT` (yes/no).
+
+- [ ] **Step 1: Sign-up vs instance claim.** Read upstream `server/src/auth/better-auth.ts`, `server/src/routes/auth.ts`, and the claim flow for `authenticated`+`private` (search `claim` under `server/src`):
+
+Run: `gh api repos/paperclipai/paperclip/contents/server/src/routes/auth.ts -H "Accept: application/vnd.github.raw"`
+
+Decide: if the first-admin claim creates the account through a path that ignores `disableSignUp` (e.g. invite/claim token handler calling the adapter directly), `SIGNUP_DISABLED="true"`. Otherwise `SIGNUP_DISABLED="false"` — the Authentik gate is the access control, and a follow-up PR flips it to `"true"` after the admin is claimed.
+
+- [ ] **Step 2: Partial config.json.** Read `server/src/config.ts` and the config-file loader it calls (search `PAPERCLIP_CONFIG` / `readConfigFile`).
+
+Run: `gh api repos/paperclipai/paperclip/contents/server/src/config.ts -H "Accept: application/vnd.github.raw"`
+
+Decide `CONFIG_PARTIAL_OK=yes` if a file containing only `{"telemetry":{"enabled":false}}` parses (every other section optional/defaulted). If not, stop and report to the user — do not invent config fields.
+
+- [ ] **Step 3: Entrypoint privilege.**
+
+Run: `gh api repos/paperclipai/paperclip/contents/docker-entrypoint.sh -H "Accept: application/vnd.github.raw"`
+
+`ENTRYPOINT_NEEDS_ROOT=yes` if it calls `usermod`/`groupmod`/`chown`/`gosu`. Record the result for Task 6.
+
+- [ ] **Step 4: Anonymous health response.**
+
+Run: `gh api repos/paperclipai/paperclip/contents/server/src/routes/health.ts -H "Accept: application/vnd.github.raw"`
+
+Confirm `GET /api/health` returns HTTP 200 to an anonymous caller when healthy (probes use it). If it returns 401 anonymously, use a `tcpSocket` probe on 3100 in Task 6 instead.
+
+---
+
+### Task 2: Namespace, baseline CNP, ClusterSecretStore allowlist
+
+**Files:**
+- Modify: `kubernetes/infrastructure/namespaces/namespaces.yaml` (append)
+- Create: `kubernetes/infrastructure/network-policies/paperclip-ns.yaml`
+- Modify: `kubernetes/infrastructure/network-policies/kustomization.yaml`
+- Modify: `kubernetes/infrastructure/post-crds/openbao-clustersecretstore.yaml` (sorted list)
+
+**Interfaces:**
+- Produces: namespace `paperclip`; ESO may reference `openbao` from it.
+
+- [ ] **Step 1: Append the namespace** to the end of `namespaces.yaml`:
+
+```yaml
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: paperclip
+```
+
+- [ ] **Step 2: Create `paperclip-ns.yaml`** (copy of `paperless-ns.yaml`'s `default-deny-ingress`, ns renamed, paperless-only CNPs dropped):
+
+```yaml
+# Traefik is not admitted: the only entry is the paperclip-gate outpost (apps/paperclip/network-policy.yaml).
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: default-deny-ingress
+  namespace: paperclip
+spec:
+  endpointSelector: {}
+  ingress:
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": paperclip
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": database
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": monitoring
+    - fromEntities:
+        - host
+        - remote-node
+    - fromEndpoints:
+        - matchExpressions:
+            - key: io.kubernetes.pod.namespace
+              operator: Exists
+      icmps:
+        - fields:
+            - { type: 0,   family: IPv4 }
+            - { type: 3,   family: IPv4 }
+            - { type: 8,   family: IPv4 }
+            - { type: 11,  family: IPv4 }
+            - { type: 12,  family: IPv4 }
+            - { type: 1,   family: IPv6 }
+            - { type: 2,   family: IPv6 }
+            - { type: 3,   family: IPv6 }
+            - { type: 4,   family: IPv6 }
+            - { type: 128, family: IPv6 }
+            - { type: 129, family: IPv6 }
+            - { type: 133, family: IPv6 }
+            - { type: 134, family: IPv6 }
+            - { type: 135, family: IPv6 }
+            - { type: 136, family: IPv6 }
+            - { type: 143, family: IPv6 }
+  egress:
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": paperclip
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": authentik
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": ingress
+            app.kubernetes.io/name: traefik
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": rook-ceph
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": kube-system
+            k8s-app: kube-dns
+      toPorts:
+        - ports:
+            - port: "53"
+              protocol: UDP
+    - toEntities:
+        - host
+        - remote-node
+        - world
+    - toEndpoints:
+        - matchExpressions:
+            - key: io.kubernetes.pod.namespace
+              operator: Exists
+      icmps:
+        - fields:
+            - { type: 0,   family: IPv4 }
+            - { type: 3,   family: IPv4 }
+            - { type: 8,   family: IPv4 }
+            - { type: 11,  family: IPv4 }
+            - { type: 12,  family: IPv4 }
+            - { type: 1,   family: IPv6 }
+            - { type: 2,   family: IPv6 }
+            - { type: 3,   family: IPv6 }
+            - { type: 4,   family: IPv6 }
+            - { type: 128, family: IPv6 }
+            - { type: 129, family: IPv6 }
+            - { type: 133, family: IPv6 }
+            - { type: 134, family: IPv6 }
+            - { type: 135, family: IPv6 }
+            - { type: 136, family: IPv6 }
+            - { type: 143, family: IPv6 }
+```
+
+- [ ] **Step 3: Register it** — in `network-policies/kustomization.yaml` add `  - paperclip-ns.yaml` directly after `  - paperless-ns.yaml`.
+
+- [ ] **Step 4: Allowlist the namespace** — in `openbao-clustersecretstore.yaml` insert `        - paperclip` between `        - monitoring` and `        - paperless` (list is sorted).
+
+- [ ] **Step 5: Render check.**
+
+Run: `kubectl kustomize kubernetes/infrastructure/network-policies`
+Expected: output contains `namespace: paperclip`, exit 0.
+
+Run: `kubectl apply --dry-run=server -f kubernetes/infrastructure/post-crds/openbao-clustersecretstore.yaml`
+Expected: `clustersecretstore.external-secrets.io/openbao configured (server dry run)`.
+
+- [ ] **Step 6: Commit** `feat(paperclip): namespace, baseline CNP and OpenBao allowlist`.
+
+---
+
+### Task 3: OpenBao secret `apps/paperclip`
+
+Out-of-repo; must exist before merge or the ExternalSecret never becomes Ready.
+
+**Interfaces:**
+- Produces: OpenBao KV `apps/paperclip` with the four fields in Global Constraints.
+
+- [ ] **Step 1:** Ask wiki-synth for the claude-bot OpenBao write procedure (note `claude-bot-openbao-write-only-identity`): login method, KV mount, exact `bao kv put`/`patch` syntax for an `apps/<name>` key. Use exactly what it returns.
+
+- [ ] **Step 2: Write the generated fields** as claude-bot (write-only identity — never read values back). Values generated inline by the command:
+
+`BETTER_AUTH_SECRET` = `openssl rand -hex 32`, `PAPERCLIP_TOOL_ACTION_SIGNING_SECRET` = `openssl rand -hex 32`, `PAPERCLIP_SECRETS_MASTER_KEY` = `openssl rand -hex 32` (64 hex chars = 32 bytes, accepted format).
+
+Generate each with a separate `openssl rand -hex 32` call and pass it directly into the `bao kv put` from Step 1; do not echo or log them.
+
+- [ ] **Step 3: Ask the user** to add the shared token themselves (never handle it): "Run `claude setup-token`, then `bao kv patch <mount>/apps/paperclip CLAUDE_CODE_OAUTH_TOKEN=<token>`." Wait for confirmation.
+
+- [ ] **Step 4: Verify field names only** (no values): `bao kv metadata get <mount>/apps/paperclip` shows a current version ≥2. If claude-bot cannot read metadata, ask the user to confirm the four keys exist.
+
+---
+
+### Task 4: Image automation + initial digest
+
+**Files:**
+- Create: `kubernetes/infrastructure/image-automation/paperclip.yaml`
+- Modify: `kubernetes/infrastructure/image-automation/kustomization.yaml`
+
+**Interfaces:**
+- Produces: ImagePolicy `flux-system:paperclip`; `INITIAL_DIGEST` (`sha256:…`) for Task 6.
+
+- [ ] **Step 1: Create `paperclip.yaml`:**
+
+```yaml
+apiVersion: image.toolkit.fluxcd.io/v1
+kind: ImageRepository
+metadata:
+  name: paperclip
+  namespace: flux-system
+spec:
+  image: ghcr.io/paperclipai/paperclip
+  interval: 1h
+---
+# Upstream publishes no semver tags; track the nightly tag by digest.
+apiVersion: image.toolkit.fluxcd.io/v1
+kind: ImagePolicy
+metadata:
+  name: paperclip
+  namespace: flux-system
+spec:
+  imageRepositoryRef:
+    name: paperclip
+  digestReflectionPolicy: Always
+  filterTags:
+    pattern: '^nightly$'
+  policy:
+    alphabetical:
+      order: asc
+```
+
+- [ ] **Step 2: Register** — add `  - paperclip.yaml` after `  - paperless-ngx.yaml` in `image-automation/kustomization.yaml`.
+
+- [ ] **Step 3: Server dry-run.**
+
+Run: `kubectl apply --dry-run=server -f kubernetes/infrastructure/image-automation/paperclip.yaml`
+Expected: both objects `created (server dry run)`; no `unknown field "digestReflectionPolicy"` error.
+
+- [ ] **Step 4: Resolve the current nightly digest.**
+
+The workstation has `docker` only (no skopeo/crane/buildx, checked 2026-09-25); `docker manifest inspect` lists per-platform digests but not the index digest, so pull once:
+
+Run: `docker pull ghcr.io/paperclipai/paperclip:nightly`
+Run: `docker image inspect --format '{{index .RepoDigests 0}}' ghcr.io/paperclipai/paperclip:nightly`
+Expected: `ghcr.io/paperclipai/paperclip@sha256:<64 hex>`. Record the `sha256:…` part as `INITIAL_DIGEST`.
+Run: `docker image rm ghcr.io/paperclipai/paperclip:nightly`
+
+- [ ] **Step 5: Commit** `feat(paperclip): Flux image automation tracks the nightly digest`.
+
+---
+
+### Task 5: Postgres (CNPG) and secrets in the app directory
+
+**Files:**
+- Create: `kubernetes/apps/paperclip/cluster.yaml`, `objectstore.yaml`, `scheduled-backup.yaml`, `hetzner-creds-externalsecret.yaml`, `externalsecret.yaml`
+
+**Interfaces:**
+- Produces: Secret `pclp-pg-app` (CNPG, key `uri`), Service `pclp-pg-rw`, Secret `paperclip-secrets`.
+
+- [ ] **Step 1: `cluster.yaml`:**
+
+```yaml
+apiVersion: postgresql.cnpg.io/v1
+kind: Cluster
+metadata:
+  name: pclp-pg
+  namespace: paperclip
+  annotations:
+    kustomize.toolkit.fluxcd.io/prune: disabled # data guard
+spec:
+  instances: 2
+  smartShutdownTimeout: 10
+  stopDelay: 60
+  # PG17: upstream's external Postgres target (docker/docker-compose.yml).
+  imageName: ghcr.io/cloudnative-pg/postgresql:17
+  storage:
+    storageClass: ceph-block
+    size: 10Gi
+  plugins:
+    - name: barman-cloud.cloudnative-pg.io
+      isWALArchiver: true
+      parameters:
+        barmanObjectName: pclp-pg-store
+  bootstrap:
+    initdb:
+      database: paperclip
+      owner: paperclip
+  postgresql:
+    parameters:
+      shared_buffers: "256MB"
+      max_connections: "200"
+      checkpoint_timeout: "15min"
+      max_wal_size: "2GB"
+      wal_compression: "zstd"
+      archive_timeout: "30min"
+    synchronous:
+      method: any
+      number: 1
+      dataDurability: preferred
+  resources:
+    requests:
+      cpu: 200m
+      memory: 512Mi
+    limits:
+      memory: 1Gi
+```
+
+- [ ] **Step 2: `objectstore.yaml`:**
+
+```yaml
+apiVersion: barmancloud.cnpg.io/v1
+kind: ObjectStore
+metadata:
+  name: pclp-pg-store
+  namespace: paperclip
+  annotations:
+    kustomize.toolkit.fluxcd.io/prune: disabled # backup target
+spec:
+  configuration:
+    destinationPath: s3://renner-uno-k8s-cnpg/pclp-pg
+    endpointURL: https://fsn1.your-objectstorage.com
+    s3Credentials:
+      accessKeyId:
+        name: hetzner-s3-creds
+        key: ACCESS_KEY_ID
+      secretAccessKey:
+        name: hetzner-s3-creds
+        key: ACCESS_SECRET_KEY
+      region:
+        name: hetzner-s3-creds
+        key: REGION
+    wal:
+      compression: gzip
+    data:
+      compression: gzip
+  retentionPolicy: "30d"
+```
+
+- [ ] **Step 3: `scheduled-backup.yaml`:**
+
+```yaml
+apiVersion: postgresql.cnpg.io/v1
+kind: ScheduledBackup
+metadata:
+  name: pclp-pg-daily
+  namespace: paperclip
+  annotations:
+    kustomize.toolkit.fluxcd.io/prune: disabled # backup schedule
+spec:
+  schedule: "0 45 2 * * *"
+  immediate: true
+  backupOwnerReference: self
+  cluster:
+    name: pclp-pg
+  method: plugin
+  pluginConfiguration:
+    name: barman-cloud.cloudnative-pg.io
+```
+
+- [ ] **Step 4: `hetzner-creds-externalsecret.yaml`:**
+
+```yaml
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: hetzner-s3-creds
+  namespace: paperclip
+  annotations:
+    kustomize.toolkit.fluxcd.io/prune: disabled # barman S3 creds
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: openbao
+  target:
+    name: hetzner-s3-creds
+  dataFrom:
+    - extract:
+        key: infrastructure/hetzner-s3
+```
+
+- [ ] **Step 5: `externalsecret.yaml`:**
+
+```yaml
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: paperclip-secrets
+  namespace: paperclip
+spec:
+  refreshInterval: 10m
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: openbao
+  target:
+    name: paperclip-secrets
+  dataFrom:
+    - extract:
+        key: apps/paperclip
+```
+
+- [ ] **Step 6: Commit** `feat(paperclip): CNPG pclp-pg with barman backups and OpenBao secrets`. (Rendered together with Task 6's kustomization.)
+
+---
+
+### Task 6: Server workload, storage, ingress, network policy
+
+**Files:**
+- Create: `kubernetes/apps/paperclip/pvc.yaml`, `paperclip.yaml`, `gate-service.yaml`, `ingress.yaml`, `network-policy.yaml`, `kustomization.yaml`
+
+**Interfaces:**
+- Consumes: `pclp-pg-app` (`uri`), `paperclip-secrets`, ImagePolicy `flux-system:paperclip`, `INITIAL_DIGEST` (Task 4), `SIGNUP_DISABLED` / `ENTRYPOINT_NEEDS_ROOT` / probe choice (Task 1).
+- Produces: pods labelled `app.kubernetes.io/name: paperclip` listening on 3100; Service `paperclip:3100`; ExternalName `ak-outpost-paperclip-gate:9000`.
+
+- [ ] **Step 1: `pvc.yaml`:**
+
+```yaml
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: paperclip-data
+  namespace: paperclip
+  annotations:
+    kustomize.toolkit.fluxcd.io/prune: disabled # data guard
+spec:
+  accessModes: [ReadWriteOnce]
+  storageClassName: ceph-block
+  resources:
+    requests:
+      storage: 20Gi
+```
+
+- [ ] **Step 2: `paperclip.yaml`** — replace `<INITIAL_DIGEST>` with the Task 4 value and `<SIGNUP_DISABLED>` with the Task 1 value. If Task 1 found `ENTRYPOINT_NEEDS_ROOT=no`, replace the container `securityContext` with `runAsNonRoot: true`, `runAsUser: 1000`, `runAsGroup: 1000`, `allowPrivilegeEscalation: false`, `capabilities: {drop: ["ALL"]}`. If Task 1 chose `tcpSocket`, replace each `httpGet` block with `tcpSocket: {port: http}`.
+
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: paperclip
+  namespace: paperclip
+  labels:
+    app.kubernetes.io/name: paperclip
+  annotations:
+    secret.reloader.stakater.com/reload: paperclip-secrets
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: paperclip
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: paperclip
+    spec:
+      enableServiceLinks: false
+      securityContext:
+        fsGroup: 1000
+        fsGroupChangePolicy: OnRootMismatch
+        seccompProfile:
+          type: RuntimeDefault
+      initContainers:
+        # Telemetry is only configurable in config.json.
+        - name: disable-telemetry
+          image: ghcr.io/paperclipai/paperclip:nightly@<INITIAL_DIGEST> # {"$imagepolicy": "flux-system:paperclip"}
+          command:
+            - sh
+            - -c
+            - |
+              set -eu
+              f=/paperclip/instances/default/config.json
+              mkdir -p "$(dirname "$f")"
+              if [ -s "$f" ]; then
+                jq '.telemetry.enabled = false' "$f" > "$f.tmp"
+              else
+                printf '%s\n' '{"telemetry":{"enabled":false}}' > "$f.tmp"
+              fi
+              mv "$f.tmp" "$f"
+          securityContext:
+            runAsNonRoot: true
+            runAsUser: 1000
+            runAsGroup: 1000
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop: ["ALL"]
+          volumeMounts:
+            - name: data
+              mountPath: /paperclip
+      containers:
+        - name: paperclip
+          image: ghcr.io/paperclipai/paperclip:nightly@<INITIAL_DIGEST> # {"$imagepolicy": "flux-system:paperclip"}
+          ports:
+            - name: http
+              containerPort: 3100
+          env:
+            - name: DATABASE_URL
+              valueFrom:
+                secretKeyRef:
+                  name: pclp-pg-app
+                  key: uri
+            - name: HOST
+              value: "0.0.0.0"
+            - name: PORT
+              value: "3100"
+            - name: PAPERCLIP_DEPLOYMENT_MODE
+              value: authenticated
+            - name: PAPERCLIP_DEPLOYMENT_EXPOSURE
+              value: private
+            - name: PAPERCLIP_PUBLIC_URL
+              value: https://paperclip.renner.uno
+            - name: PAPERCLIP_AUTH_DISABLE_SIGN_UP
+              value: "<SIGNUP_DISABLED>"
+          envFrom:
+            - secretRef:
+                name: paperclip-secrets
+          volumeMounts:
+            - name: data
+              mountPath: /paperclip
+          startupProbe:
+            httpGet:
+              path: /api/health
+              port: http
+            periodSeconds: 10
+            timeoutSeconds: 5
+            failureThreshold: 60
+          livenessProbe:
+            httpGet:
+              path: /api/health
+              port: http
+            periodSeconds: 30
+            timeoutSeconds: 10
+          readinessProbe:
+            httpGet:
+              path: /api/health
+              port: http
+            periodSeconds: 10
+            timeoutSeconds: 5
+          resources:
+            requests:
+              cpu: 250m
+              memory: 1Gi
+            limits:
+              memory: 4Gi
+          securityContext:
+            runAsNonRoot: false
+            allowPrivilegeEscalation: true
+            capabilities:
+              drop: ["ALL"]
+              add: ["CHOWN", "DAC_OVERRIDE", "FOWNER", "SETGID", "SETUID"]
+      volumes:
+        - name: data
+          persistentVolumeClaim:
+            claimName: paperclip-data
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: paperclip
+  namespace: paperclip
+spec:
+  selector:
+    app.kubernetes.io/name: paperclip
+  ports:
+    - name: http
+      port: 3100
+      targetPort: http
+```
+
+- [ ] **Step 3: `gate-service.yaml`:**
+
+```yaml
+# ExternalName mirror so the Ingress can backend the paperclip-gate outpost in the authentik namespace.
+apiVersion: v1
+kind: Service
+metadata:
+  name: ak-outpost-paperclip-gate
+  namespace: paperclip
+spec:
+  type: ExternalName
+  externalName: ak-outpost-paperclip-gate.authentik.svc.cluster.local
+  ports:
+    - port: 9000
+      targetPort: 9000
+```
+
+- [ ] **Step 4: `ingress.yaml`:**
+
+```yaml
+# LAN-only (no dns.renner.uno/public label); backends the gate outpost, never the Service.
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: paperclip
+  namespace: paperclip
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-cluster
+spec:
+  ingressClassName: traefik
+  tls:
+    - hosts:
+        - paperclip.renner.uno
+      secretName: paperclip-tls
+  rules:
+    - host: paperclip.renner.uno
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: ak-outpost-paperclip-gate
+                port:
+                  number: 9000
+```
+
+- [ ] **Step 5: `network-policy.yaml`** (Traefik pod-side port is 8443, verified via `kubectl get pods -n ingress -l app.kubernetes.io/name=traefik`):
+
+```yaml
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: paperclip-gate-ingress
+  namespace: paperclip
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: paperclip
+  ingress:
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": authentik
+            goauthentik.io/outpost-name: paperclip-gate
+      toPorts:
+        - ports:
+            - port: "3100"
+              protocol: TCP
+---
+# Covers the server and every agent it spawns (same pod).
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: paperclip-egress
+  namespace: paperclip
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: paperclip
+  egress:
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": paperclip
+            cnpg.io/cluster: pclp-pg
+      toPorts:
+        - ports:
+            - port: "5432"
+              protocol: TCP
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": kube-system
+            k8s-app: kube-dns
+      toPorts:
+        - ports:
+            - port: "53"
+              protocol: UDP
+            - port: "53"
+              protocol: TCP
+    # Gitea, Woodpecker, HA, Loki via their *.renner.uno names.
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": ingress
+            app.kubernetes.io/name: traefik
+      toPorts:
+        - ports:
+            - port: "8443"
+              protocol: TCP
+    - toEntities:
+        - kube-apiserver
+      toPorts:
+        - ports:
+            - port: "6443"
+              protocol: TCP
+    - toEntities:
+        - world
+      toPorts:
+        - ports:
+            - port: "443"
+              protocol: TCP
+```
+
+- [ ] **Step 6: `kustomization.yaml`:**
+
+```yaml
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - externalsecret.yaml
+  - hetzner-creds-externalsecret.yaml
+  - cluster.yaml
+  - objectstore.yaml
+  - scheduled-backup.yaml
+  - pvc.yaml
+  - paperclip.yaml
+  - gate-service.yaml
+  - ingress.yaml
+  - network-policy.yaml
+```
+
+- [ ] **Step 7: Render check.**
+
+Run: `kubectl kustomize kubernetes/apps/paperclip`
+Expected: exit 0; output contains `kind: Cluster`, `kind: Deployment`, `claimName: paperclip-data`, and no literal `<INITIAL_DIGEST>` or `<SIGNUP_DISABLED>`.
+
+Run: `grep -rn "<INITIAL_DIGEST>\|<SIGNUP_DISABLED>" kubernetes/apps/paperclip`
+Expected: no output, exit 1.
+
+- [ ] **Step 8: Commit** `feat(paperclip): server Deployment, PVC, gated Ingress and CNPs`.
+
+---
+
+### Task 7: Authentik gate
+
+**Files:**
+- Modify: `kubernetes/apps/authentik/blueprints-configmap.yaml` (insert before the line `      # navidrome-gate: the Navidrome web UI.`)
+- Create: `kubernetes/apps/authentik/paperclip-gate-egress.yaml`
+- Modify: `kubernetes/apps/authentik/kustomization.yaml`
+
+**Interfaces:**
+- Produces: outpost pods labelled `goauthentik.io/outpost-name: paperclip-gate`, Service `ak-outpost-paperclip-gate.authentik:9000`.
+
+- [ ] **Step 1: Insert blueprint entries** (same indentation as the neighbouring `wiki-gate` block):
+
+```yaml
+      # paperclip-gate: Paperclip UI; the CNP admits only this outpost. Paperclip keeps its own login behind it.
+      - id: paperclip-users-group
+        model: authentik_core.group
+        identifiers:
+          name: Paperclip Users
+        state: present
+        attrs:
+          is_superuser: false
+          users:
+            - !Find [authentik_core.user, [username, k.renner]]
+
+      - id: paperclip-gate-provider
+        model: authentik_providers_proxy.proxyprovider
+        identifiers:
+          name: paperclip-gate
+        state: present
+        attrs:
+          authentication_flow: !Find [authentik_flows.flow, [slug, passkey-passwordless-authentication]]
+          authorization_flow: !Find [authentik_flows.flow, [slug, passkey-stepup-authorization]]
+          invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
+          mode: proxy
+          internal_host: http://paperclip.paperclip.svc.cluster.local:3100
+          internal_host_ssl_validation: false
+          external_host: https://paperclip.renner.uno
+          cookie_domain: renner.uno
+          access_token_validity: hours=24
+          refresh_token_validity: days=30
+          intercept_header_auth: true
+
+      - model: authentik_core.application
+        identifiers:
+          slug: paperclip-gate
+        state: present
+        attrs:
+          name: Paperclip
+          slug: paperclip-gate
+          provider: !KeyOf paperclip-gate-provider
+          meta_launch_url: https://paperclip.renner.uno
+          meta_description: "Paperclip agent orchestration"
+
+      - model: authentik_policies.policybinding
+        identifiers:
+          target: !Find [authentik_core.application, [slug, paperclip-gate]]
+          group: !KeyOf paperclip-users-group
+          order: 0
+        state: present
+        attrs:
+          enabled: true
+          negate: false
+
+      - model: authentik_outposts.outpost
+        identifiers:
+          name: paperclip-gate
+        state: present
+        attrs:
+          type: proxy
+          service_connection: !KeyOf k8s-conn
+          providers:
+            - !KeyOf paperclip-gate-provider
+          config:
+            log_level: info
+            authentik_host: "http://authentik-server.authentik.svc.cluster.local"
+            authentik_host_insecure: false
+            authentik_host_browser: "https://auth.renner.uno/"
+            refresh_interval: "minutes=5"
+            container_image: null
+            object_naming_template: "ak-outpost-%(name)s"
+            kubernetes_namespace: authentik
+            kubernetes_replicas: 1
+            kubernetes_service_type: ClusterIP
+            kubernetes_disabled_components: ["ingress"]
+            kubernetes_image_pull_secrets: []
+            kubernetes_ingress_annotations: {}
+            kubernetes_ingress_class_name: null
+            kubernetes_ingress_secret_name: "authentik-outpost-tls"
+            kubernetes_json_patches: null
+
+```
+
+- [ ] **Step 2: `paperclip-gate-egress.yaml`:**
+
+```yaml
+# Lets the paperclip-gate outpost reach the Paperclip server; scoped to this outpost's pods.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: paperclip-gate-egress
+  namespace: authentik
+spec:
+  endpointSelector:
+    matchLabels:
+      goauthentik.io/outpost-name: paperclip-gate
+  egress:
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": paperclip
+            app.kubernetes.io/name: paperclip
+      toPorts:
+        - ports:
+            - port: "3100"
+              protocol: TCP
+```
+
+- [ ] **Step 3: Register** — add `  - paperclip-gate-egress.yaml` after `  - paperless-gate-egress.yaml` in `apps/authentik/kustomization.yaml`.
+
+- [ ] **Step 4: Render + dry-run.**
+
+Run: `kubectl kustomize kubernetes/apps/authentik`
+Expected: exit 0 (valid YAML incl. the blueprint ConfigMap).
+
+Run: `kubectl apply --dry-run=server -f kubernetes/apps/authentik/paperclip-gate-egress.yaml`
+Expected: `ciliumnetworkpolicy.cilium.io/paperclip-gate-egress created (server dry run)`.
+
+- [ ] **Step 5: Commit** `feat(authentik): paperclip-gate outpost and Paperclip Users group`.
+
+---
+
+### Task 8: Flux Kustomization
+
+**Files:**
+- Create: `kubernetes/clusters/homelab/paperclip.yaml`
+
+- [ ] **Step 1:**
+
+```yaml
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: paperclip
+  namespace: flux-system
+spec:
+  interval: 10m
+  retryInterval: 1m
+  timeout: 20m
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./kubernetes/apps/paperclip
+  prune: true
+  wait: true
+  dependsOn:
+    - name: infrastructure
+    - name: infrastructure-crds
+    - name: infrastructure-ceph-cluster
+    - name: infrastructure-cnpg-barman
+  healthCheckExprs:
+    - apiVersion: postgresql.cnpg.io/v1
+      kind: Cluster
+      failed: "has(status.phase) && status.phase.startsWith('Failed')"
+      current: "has(status.phase) && (status.phase == 'Cluster in healthy state' || status.readyInstances > 0)"
+```
+
+- [ ] **Step 2: Server dry-run.**
+
+Run: `kubectl apply --dry-run=server -f kubernetes/clusters/homelab/paperclip.yaml`
+Expected: `kustomization.kustomize.toolkit.fluxcd.io/paperclip created (server dry run)`.
+
+- [ ] **Step 3: Commit** `feat(paperclip): Flux Kustomization`.
+
+---
+
+### Task 9: PR, merge by user, live verification
+
+- [ ] **Step 1: Push and open the PR** (Task 3 must be complete).
+
+Run: `git -c credential.helper='store --file=/home/krenner/.config/claude-bot/creds' push -u origin feat/paperclip`
+Run: `tea pr create --login claude-bot --base master --head feat/paperclip --title "feat(paperclip): deploy Paperclip (server-local runs, trial)" --description "<summary of spec decisions + link to spec + bootstrap steps>\n\n🤖 Generated with [Claude Code](https://claude.com/claude-code)"`
+Then wait for the user to merge. Do not merge.
+
+- [ ] **Step 2: Reconciliation.** After merge:
+
+Run: `flux get kustomizations paperclip`
+Expected: `Ready True`. If False, read `kubectl describe kustomization paperclip -n flux-system` and the failing object's events; diagnose, don't poke.
+
+Run: `kubectl get cluster -n paperclip pclp-pg`
+Expected: `Cluster in healthy state`, 2 ready instances.
+
+Run: `kubectl get pods -n paperclip -l app.kubernetes.io/name=paperclip`
+Expected: `1/1 Running`.
+
+- [ ] **Step 3: Gate enforced (Review Focus 4).**
+
+Run: `curl -sS -o /dev/null -w '%{http_code} %{redirect_url}\n' https://paperclip.renner.uno/`
+Expected: `302` redirecting to `https://paperclip.renner.uno/outpost.goauthentik.io/…` or `auth.renner.uno`, never `200`.
+
+- [ ] **Step 4: Gate not bypassable (Review Focus 3).**
+
+`default` has no CNPs (verified 2026-09-25), so its egress is open and only paperclip's ingress policy can block the probe.
+
+Run: `kubectl get ciliumnetworkpolicies -n default`
+Expected: `No resources found` (else pick another CNP-free namespace).
+
+Run: `kubectl run cnp-probe -n default --rm -i --restart=Never --image=curlimages/curl:8.10.1 -- curl -sS -m 5 http://paperclip.paperclip.svc.cluster.local:3100/api/health`
+Expected: timeout (exit 28), no HTTP response.
+
+- [ ] **Step 5: First backup (Review Focus 5).**
+
+Run: `kubectl get backup -n paperclip`
+Expected: one `pclp-pg-daily-…` with phase `completed`.
+
+- [ ] **Step 6: Image policy.**
+
+Run: `kubectl get imagepolicy -n flux-system paperclip -o jsonpath='{.status.latestRef}'`
+Expected: `nightly` tag with a `sha256:` digest.
+
+- [ ] **Step 7: Hand-over to the user (browser steps, theirs):**
+  1. Open `https://paperclip.renner.uno`, pass Authentik, claim the instance as admin.
+  2. Create a test agent with the `claude-local` adapter and a trivial task ("reply with OK"); it must complete with the shared token (Review Focus 2: if it fails, report the run's error text — an expired token must show as an auth error, not a hang).
+  3. If `SIGNUP_DISABLED` was `"false"`, open the follow-up PR flipping `PAPERCLIP_AUTH_DISABLE_SIGN_UP` to `"true"` once the admin is claimed.
+  4. Review Focus 1 is observed on the first nightly bump: check the interrupted-run status in the UI.
+
+- [ ] **Step 8: Wiki.** Route the verified Paperclip facts (execution topology, env vars, image tags, digest-reflection usage) through wiki-ingest → wiki-distill once the deployment has proven out.

docs/superpowers/specs/2026-09-25-paperclip-design.md

@@ -0,0 +1,206 @@
+# Paperclip in the homelab cluster — design
+
+Date: 2026-09-25
+Status: draft, awaiting review
+
+## Goal
+
+Run [Paperclip](https://github.com/paperclipai/paperclip) (MIT, agent-orchestration
+server + UI) in the homelab cluster for real use: its agents do actual work against
+homelab systems. This spec covers the **platform** only. Bot identities and their
+credentials are out of scope (see "Deferred").
+
+## Decisions
+
+| Topic | Decision |
+|---|---|
+| Execution model | **B — server-local runs** (upstream default): agent CLIs are child processes of the server, in the server container. Trial only. |
+| Upgrade path | If Paperclip is kept: **C** — `@paperclipai/plugin-kubernetes` sandbox provider + kubernetes-sigs agent-sandbox controller (`agents.x-k8s.io`), per-run Pods and Secrets. Separate spec. |
+| Claude auth | **One shared Claude subscription AI connection**, created once via Paperclip's local sign-in (`claude auth login` into `ai-local-logins/<id>`); Paperclip stores and refreshes it. Server-env `CLAUDE_CODE_OAUTH_TOKEN` is not used: `stripAiAuthBindings` blanks it for managed runs, and upstream accepts no pasted setup-token (the device-code flow needs a Daytona sandbox). |
+| Bot credentials | Out of scope. Future "AIR-Manager" Paperclip role handles permission requests. |
+| Exposure | LAN-only, behind an Authentik forward-auth gate, then Paperclip's own login. |
+
+### Known trade-off of B (accepted for the trial)
+
+Upstream `runChildProcess()` spawns every local adapter with
+`{...sanitizeInheritedPaperclipEnv(process.env), ...opts.env}`; the sanitizer strips only
+`PAPERCLIP_*` keys. Every agent therefore sees `DATABASE_URL`, `BETTER_AUTH_SECRET`,
+the shared filesystem (including every other agent's workspace and the
+`ai-local-logins` credentials) and every agent-scoped secret it can read from disk. Under B, no bot may hold
+a credential we would not tolerate any other bot using. Per-bot isolation is C's job.
+
+## Architecture
+
+```
+LAN client ─▶ Traefik ─▶ ExternalName svc ─▶ ak-outpost-paperclip-gate (authentik ns)
+                                                   │ forward-auth OK
+                                                   ▼
+                                  paperclip Deployment (ns paperclip, 1 replica)
+                                   ├─ server :3100  (+ agent CLIs as child processes)
+                                   ├─ PVC /paperclip  (ceph-block)
+                                   └─▶ pclp-pg-rw (CNPG, PG17, 2 instances) ─▶ barman ▶ Hetzner S3
+```
+
+## Components
+
+All modeled on `kubernetes/apps/paperless/` unless stated.
+
+### Namespace and baseline policy
+- `Namespace paperclip` in `kubernetes/infrastructure/namespaces/namespaces.yaml`.
+- Baseline default-deny CNP `kubernetes/infrastructure/network-policies/paperclip-ns.yaml`
+  (copy of `paperless-ns.yaml`, renamed).
+- Add `paperclip` to the `openbao` ClusterSecretStore namespace allowlist.
+
+### Flux
+- `kubernetes/clusters/homelab/paperclip.yaml`: Flux `Kustomization`,
+  `path: ./kubernetes/apps/paperclip`, `prune: true`, `wait: true`, same `dependsOn` and
+  CNPG `healthCheckExprs` as `paperless.yaml`.
+- `kubernetes/apps/paperclip/kustomization.yaml`: plain resource list.
+
+### Server Deployment (`paperclip.yaml`)
+- Image `ghcr.io/paperclipai/paperclip:nightly@sha256:<digest>` with the Flux
+  image-automation marker for digest updates.
+- `replicas: 1`, `strategy: Recreate` (single writer on an RWO PVC).
+- Starts as root: upstream `docker-entrypoint.sh` remaps the `node` user, chowns
+  `/paperclip`, then drops to uid 1000 via `gosu`. Caps limited to CHOWN,
+  DAC_OVERRIDE, FOWNER, SETGID, SETUID; `fsGroup: 1000`.
+- `serviceAccountName: paperclip-prometheus-read` with
+  `automountServiceAccountToken: true`. All agents share the pod filesystem, so all
+  of them can read that token — it is mounted anyway because the Role behind it
+  grants exactly `get` on `services/proxy` for
+  `kube-prometheus-stack-prometheus:9090` and
+  `kube-prometheus-stack-alertmanager:9093` in `monitoring`, and nothing else
+  (`kubernetes/apps/paperclip/prometheus-read-rbac.yaml`, REN-36 then REN-95).
+  It was `false` until then.
+  Three things about that grant are worth knowing before you use or debug it:
+  - **It scopes the HTTP method, not the endpoint.** The apiserver maps GET→`get`,
+    POST→`create`, PUT→`update` on a service-proxy request, so `get` alone refuses
+    `POST /-/reload` and `POST /-/quit` in the authorizer. It does *not* narrow
+    which GET endpoints are reachable: the whole read surface of both services is,
+    including `/api/v1/status/config` (scrape credentials there render as
+    `<secret>`) and the web UI.
+  - **Force GET in the caller.** Prometheus accepts both GET and POST on
+    `/api/v1/query`; under this Role only the GET form works, so a client library
+    that switches to POST for long queries will 403 for a reason that has nothing
+    to do with the query.
+  - **If reads start 403ing, check the Service name against `resourceNames`
+    first.** `resourceNames` is matched verbatim, and
+    `kubernetes/apps/monitoring/helmrelease-prometheus.yaml` floats the chart at
+    `>=72.0.0` — a release rename breaks this path silently,
+    with everything still looking correct in git. The port-qualified form is the
+    only one granted: `svc`, `svc:9090` and `http:svc:9090` are three different
+    strings to the authorizer.
+- Port 3100; `Service paperclip` ClusterIP :3100.
+- Probes: startup and readiness on `GET /api/health` with `Host: localhost` (the
+  private-hostname guard rejects the pod IP); liveness is a TCP check, since
+  `/api/health` runs `SELECT 1` and a DB outage must not restart the server.
+- Env:
+  - `DATABASE_URL` assembled from CNPG `pclp-pg-app` Secret (`uri` key).
+  - `HOST=0.0.0.0`, `PORT=3100`
+  - `PAPERCLIP_DEPLOYMENT_MODE=authenticated`
+  - `PAPERCLIP_DEPLOYMENT_EXPOSURE=private`
+  - `PAPERCLIP_PUBLIC_URL=https://paperclip.renner.uno`
+  - `PAPERCLIP_AUTH_DISABLE_SIGN_UP=false` until the first admin is claimed: the
+    claim (`/bootstrap/claim`) requires an already signed-in Better Auth account.
+    A follow-up flips it to `true`.
+  - `PAPERCLIP_TELEMETRY_DISABLED=1` (read by `resolveTelemetryConfig`).
+  - `PAPERCLIP_API_URL=http://paperclip.paperclip.svc.cluster.local:3100` plus
+    `PAPERCLIP_ALLOWED_HOSTNAMES=paperclip.paperclip.svc.cluster.local`: upstream
+    otherwise hands agents the public URL, and the gate (`intercept_header_auth`)
+    rejects their bearer tokens. The Service name also works for sandbox pods under
+    option C; agent JWTs still authenticate each call.
+  - `envFrom: secretRef paperclip-secrets`
+- Reloader: `secret.reloader.stakater.com/reload: paperclip-secrets`.
+
+### Storage
+- `PVC paperclip-data`, `ceph-block`, 20Gi, mounted at `/paperclip` (`PAPERCLIP_HOME`
+  and `HOME`): instance config, workspaces, repo checkouts, per-agent `claude-runtime`
+  dirs.
+
+### Postgres
+- CNPG `Cluster pclp-pg`: 2 instances, `imageName: ghcr.io/cloudnative-pg/postgresql:17`,
+  `ceph-block` 10Gi, `bootstrap.initdb {database: paperclip, owner: paperclip}`,
+  barman-cloud plugin as WAL archiver.
+- `ObjectStore pclp-pg-store` → `s3://renner-uno-k8s-cnpg/pclp-pg`, creds via the
+  existing `hetzner-s3-creds` ExternalSecret pattern.
+- `ScheduledBackup` nightly, `method: plugin`, `backupOwnerReference: self`.
+- `kustomize.toolkit.fluxcd.io/prune: disabled` on Cluster, ObjectStore, ScheduledBackup.
+- PG17 deviates from the cluster-wide PG16 convention because upstream's external
+  Postgres target is `postgres:17` (`docker/docker-compose.yml`) and its embedded default
+  is PG18.
+
+### Secrets
+One ExternalSecret `paperclip-secrets` ← OpenBao `apps/paperclip` (whole-object
+extract):
+- `BETTER_AUTH_SECRET`
+- `PAPERCLIP_TOOL_ACTION_SIGNING_SECRET`
+- `PAPERCLIP_SECRETS_MASTER_KEY` (64-char hex). Held in OpenBao, not on the PVC, so
+  encrypted agent secrets in Postgres survive PVC loss.
+
+### Ingress and SSO
+- Hostname `paperclip.renner.uno`, Traefik `Ingress`, `cert-manager.io/cluster-issuer:
+  letsencrypt-cluster`, **no** `dns.renner.uno/public` label (LAN-only).
+- Authentik gate, per the paperless pattern:
+  - blueprint entries in `kubernetes/apps/authentik/blueprints-configmap.yaml`:
+    proxy provider `paperclip-gate` (`internal_host:
+    http://paperclip.paperclip.svc.cluster.local:3100`, `external_host:
+    https://paperclip.renner.uno`), application, group "Paperclip Users", policy binding;
+  - outpost `ak-outpost-paperclip-gate`;
+  - `kubernetes/apps/authentik/paperclip-gate-egress.yaml` (no `auth.renner.uno`
+    outpost Ingress: like wiki/shelly gates, the outpost serves its own callback on
+    `paperclip.renner.uno`);
+  - `ExternalName` Service in `paperclip` ns pointing at the outpost; the app Ingress
+    targets it on :9000.
+- Paperclip's own email/password login stays on (no header/OIDC login upstream).
+
+### Network policy (`kubernetes/apps/paperclip/network-policy.yaml`)
+Covers the server **and every agent** (same pod under B).
+- Ingress to the app pod: only from the `paperclip-gate` outpost (by label) on 3100.
+- Egress from the app pod:
+  - `pclp-pg` pods :5432
+  - kube-dns :53
+  - `ingress` namespace (Traefik) :8443 (pod-side websecure port) — reaches Gitea, Woodpecker, HA, Loki by their
+    `*.renner.uno` names
+  - `kube-apiserver` entity :6443
+  - `world` TCP :443 (Anthropic API, GitHub, npm)
+- No SSH egress; git over HTTPS only. The namespace baseline
+  (`paperclip-ns.yaml`) also restricts `world` to TCP 443 — Cilium unions allows
+  across CNPs, so an unrestricted baseline would override this list.
+- Known gap (house baseline): pods in `monitoring`/`database` and other pods in
+  `paperclip` can reach :3100 without the gate; Paperclip's own login still applies.
+
+### Image automation
+`kubernetes/infrastructure/image-automation/paperclip.yaml`: `ImageRepository`
+(`ghcr.io/paperclipai/paperclip`) + `ImagePolicy` filtering tag `^nightly$` with
+`digestReflectionPolicy: Always` (image-reflector-controller v1.2.5 in cluster; field
+available since v0.35.0). First use of digest reflection in this repo.
+
+## Guardrails
+- No bot receives cluster-admin or any binding in `flux-system`, `authentik`, `openbao`.
+- Git work goes through PRs; bots get no merge rights on protected branches.
+- Nothing is granted under B that we would not tolerate any agent misusing.
+
+## Bootstrap (one-time)
+1. Place `apps/paperclip` values in OpenBao (the three generated secrets).
+2. Merge; Flux deploys with sign-up enabled.
+3. Sign up through the gate and claim the instance as admin; then set
+   `PAPERCLIP_AUTH_DISABLE_SIGN_UP=true`.
+4. In onboarding "Connect a model", choose Claude subscription and run the shown login
+   **as the node user** (the entrypoint makes exec sessions root, and root-owned 0600
+   credentials are unreadable to the server):
+   `kubectl exec -it -n paperclip deploy/paperclip -- gosu node sh -c 'export CLAUDE_CONFIG_DIR=<shown path> && mkdir -p "$CLAUDE_CONFIG_DIR" && claude auth login'`
+   Repeat after loss of the PVC or database.
+
+## Verification
+- Flux Kustomization `paperclip` Ready; CNPG `pclp-pg` healthy; first ScheduledBackup
+  completes.
+- `curl https://paperclip.renner.uno/api/health` through the gate returns healthy after
+  Authentik login; unauthenticated request redirects to `auth.renner.uno`.
+- Direct pod access from another namespace is denied (CNP).
+- A test agent (claude-local) completes a trivial task using the shared Claude connection.
+- Flux ImagePolicy reports a `nightly@sha256:` digest.
+
+## Deferred
+- **Option C** (sandboxed runs) — required before any per-bot credential isolation.
+- **AIR-Manager** role and all bot identities/credentials.
+- Public exposure.

kubernetes/PROPOSAL-wiki-web-api.md

@@ -0,0 +1,315 @@
+# Wiki: local web interface and agent API, as a modular .NET solution
+
+## Goal
+
+A web interface for the Notes wiki that runs entirely in the cluster, and an
+HTTP API through which agents ask the wiki typed questions and deposit facts.
+Built as one C# solution with a MudBlazor front end, and built so that every
+external dependency is replaceable: where the notes live, which model embeds
+them, what judges a typed question, what runs an agent, how the vector index
+is stored. The first adapters are what the homelab uses today. The point of
+the design is that the next one is a project and a config line, not a
+rewrite.
+
+## Where we are
+
+- `wiki-index` (Python, ns `ai`) syncs `wiki/` from `k.renner/notes` by blob
+  SHA, embeds through `wiki-embed` (bge-m3 on llama-server), answers
+  `POST /rank`. It keeps slug, title and vector, not the text. LAN-only behind
+  Traefik BasicAuth; the Gitea webhook hits the ClusterIP.
+- `~/Notes/bin/wiki_rank.py` and `wiki_ask.py` rank through the index, build a
+  document context from a local vault checkout, send it with typed questions
+  to TypeSafe, gate on `covered`. The wiki-synth agent composes answers from
+  those judgements.
+- New knowledge enters as a file in `raw/` (facts with provenance, never
+  hypotheses), distilled nightly in CI.
+- The house C# project, shelly-manager, sets the conventions: net10.0, an
+  `.slnx` with `src/` and `test/`, xUnit, MudBlazor 9, Blazor Web App with
+  interactive server rendering, minimal APIs, an Authentik proxy-header
+  authentication handler for humans and an API-key handler for machines, a
+  multi-stage Dockerfile, `dotnet test` on every push and a kaniko release
+  pipeline to the Gitea registry. Human-facing apps sit behind an Authentik
+  outpost (`<app>-gate`) with a CNP that admits only that outpost.
+
+## Architecture: ports in the core, adapters around it
+
+```
+                 ┌──────────────────────────────────────────┐
+                 │  Wiki.Host  (ASP.NET Core + Blazor/Mud)  │  composition root,
+                 │  /api/v1, /rank, /webhook, UI pages      │  config, auth
+                 └───────────────┬──────────────────────────┘
+                                 │ uses
+                 ┌───────────────▼──────────────────────────┐
+                 │  Wiki.Application  (use cases)           │  Sync, Rank, Ask,
+                 │                                          │  DepositFact, Queue-
+                 └───────────────┬──────────────────────────┘  Question, Evaluate
+                                 │ depends only on
+                 ┌───────────────▼──────────────────────────┐
+                 │  Wiki.Core  (domain + ports)             │  Note, RawSource,
+                 │  no package references                   │  Report, LinkGraph,
+                 └──────────────────────────────────────────┘  the interfaces
+   implemented by ▲            ▲            ▲            ▲            ▲
+   Adapters.Gitea  Adapters.LlamaCpp  Adapters.TypeSafe  Adapters.FileIndex  Adapters.Anthropic
+   (later: Database, Ollama/Onnx, local Jev, pgvector/Qdrant, OpenAI-compatible answer models)
+```
+
+Rules that keep it modular, enforced by project references and a test:
+
+- `Wiki.Core` references nothing. It holds the domain types and the port
+  interfaces. Adapters reference Core only; they never reference each other
+  or the host. `Wiki.Host` is the only project that knows which adapters
+  exist.
+- Every port has a **contract test suite** in `test/Wiki.Contracts` that
+  runs against every adapter of that port. A new adapter is done when the
+  suite is green. That is what makes "swap the storage" a safe operation.
+- Adapters are selected by configuration and registered by name (keyed
+  services), so several can be active at once. The UI's lab page and the
+  evaluator use that to compare, the API uses the one marked active.
+- Nothing in Core or Application knows about HTTP, Gitea, TypeSafe, files or
+  Kubernetes.
+
+### Ports
+
+| port | what it abstracts | first adapter | examples of the next ones |
+|---|---|---|---|
+| `INoteStore` | read notes, raw sources and reports at a head; create a new document (a fact becomes a raw note in the application layer, the store only moves text) | Gitea repository API (`k.renner/notes`, blob SHA) | git checkout on a volume; a database (CNPG Postgres) with the vault imported; an Obsidian sync folder |
+| `IEmbeddingModel` | text → vector, with a model id and dimension | llama-server `/v1/embeddings` (bge-m3 via wiki-embed, the model that won the German-document comparison on accuracy per cost) | the qwen3 embedding variants from that comparison; ONNX in-process; any OpenAI-compatible endpoint |
+| `IVectorIndex` | store vectors per (model id, document), query top-k | in-memory cosine with a file snapshot per model, as today | pgvector in the database; Qdrant |
+| `IChunker` | split a document into addressable units for typed questions | paragraph splitter (headings, fences), as in `wiki_ask.py` | sentence windows; semantic chunks |
+| `IJudge` | typed questions over a state: Choice, Noul, Score, with probabilities | TypeSafe System One (`jev-latest`, cloud; the wiki records no locally hosted Jev today) | a locally hosted Jev with the same request shape, should TypeSafe offer one; a constrained-output local LLM adapter, flagged as uncalibrated |
+| `IAnswerModel` | prose from a question plus evidence: the selected paragraphs with their notes and the judgements; must cite by slug; streaming | Anthropic Messages API (Claude, model from config, Sonnet by default) | any OpenAI-compatible endpoint, including the local llama-server; a different Claude model per question |
+| `IEvaluator` | measure a retrieval or judge configuration against the labelled set (questions and cited sources in `output/`) | recall@k, hit rate, MRR, as `wiki_rank_eval.py` | judge accuracy against hand-labelled paragraphs |
+| `IIdentity` | who is calling, for attribution of facts and saved answers | Authentik proxy headers (humans), API key (machines), as in shelly-manager | OIDC bearer tokens |
+
+Two ports deserve a sentence of scope each. The judge returns judgements
+with probabilities, never prose, and the UI never presents its output as an
+answer, only as ranked, quoted evidence. The answer model turns that
+evidence into words and is allowed nothing else: it sees the selected
+paragraphs and the judgements, not the whole context, and its output is
+shown next to the evidence it was given. Both are single model calls inside
+the request. The service runs no agent: no tool loop, no vault edits. The
+wiki-synth agent in a Claude session remains the path for artifacts and for
+multi-question synthesis, and it can call `/questions` itself.
+
+### Solution layout
+
+```
+Wiki.slnx
+src/
+  Wiki.Core/               domain types, ports, frontmatter and link parsing
+  Wiki.Application/        use cases; depends on Core only
+  Wiki.Adapters.Gitea/     INoteStore (read at a head, create)
+  Wiki.Adapters.LlamaCpp/  IEmbeddingModel (OpenAI-compatible embeddings)
+  Wiki.Adapters.TypeSafe/  IJudge (System One client)
+  Wiki.Adapters.FileIndex/ IVectorIndex (memory + snapshot per model)
+  Wiki.Adapters.Anthropic/ IAnswerModel (Messages API, streaming)
+  Wiki.Host/               ASP.NET Core: API, webhook, MudBlazor UI, auth,
+                           composition root, appsettings
+test/
+  Wiki.Core.Tests/         parsing, link graph, chunker
+  Wiki.Contracts/          one suite per port, parameterised over adapters
+  Wiki.Application.Tests/  use cases with fake ports
+  Wiki.Host.Tests/         API surface, auth handlers
+```
+
+Own repository `k.renner/wiki`, same shape as shelly-manager: `Dockerfile`,
+`.woodpecker/test.yml` on pull requests, `.woodpecker/release.yml` building
+the image with kaniko into the Gitea registry on push to `main`, tagged
+`build.<unix-ts>` plus the commit SHA so Flux image automation can follow
+it. The homelab repo holds only the manifests. `claude-bot` needs write
+access to the new repository.
+
+Two things the shelly-manager history settles in advance: the TypeSafe
+client is written from the documented field contract (endpoint, bearer key,
+`jev-latest`, Noul/Choice/Score, 255-option cap, 32k-token state budget),
+there is no prior C# client to copy; and the UI needs no login page, because
+identity comes from the outpost's headers, which avoids the Blazor-circuit
+sign-in constraint shelly-manager ran into.
+
+### Configuration
+
+Adapters are declared in `appsettings` and selected by name; secrets come
+through environment variables, filled by ExternalSecret, never from the file.
+
+```json
+"Wiki": {
+  "NoteStore": { "Active": "gitea", "gitea": { "Repository": "k.renner/notes", "Branch": "main" } },
+  "Embeddings": {
+    "Active": "bge-m3",
+    "Models": {
+      "bge-m3":   { "Adapter": "LlamaCpp", "Url": "http://wiki-embed.ai.svc:8080", "Model": "bge-m3" },
+      "e5-large": { "Adapter": "Ollama",   "Url": "http://ollama.ai.svc:11434",    "Model": "multilingual-e5-large" }
+    }
+  },
+  "Judges": {
+    "Active": "typesafe",
+    "Models": {
+      "typesafe":  { "Adapter": "TypeSafe", "Url": "https://api.typesafe.ai/v1/systemone", "Model": "jev-latest" },
+      "jev-local": { "Adapter": "TypeSafe", "Url": "http://jev.ai.svc:8080/v1/systemone",  "Model": "jev" }   // illustration: same adapter, other endpoint
+    }
+  },
+  "Index": { "Active": "file", "file": { "Path": "/data/index" } },
+  "AnswerModels": {
+    "Active": "claude-sonnet",
+    "Models": {
+      "claude-sonnet": { "Adapter": "Anthropic", "Model": "claude-sonnet-5" },
+      "qwen-local":    { "Adapter": "OpenAI",    "Url": "http://llama-server.ai.svc:8080/v1", "Model": "qwen3.6-35b-a3b" }
+    }
+  }
+}
+```
+
+Every embedding model gets its own index namespace. Switching the active
+model is a config change after the new index has been built in the
+background; until then both exist and can be compared. Nothing is
+re-embedded in place.
+
+## The web interface (MudBlazor)
+
+Blazor Web App, interactive server rendering, hosted in `Wiki.Host`. Static
+assets vendored into the image, no CDN. Pages:
+
+1. **Search**: a query box, results ranked by the active index with score and
+   tags; for a phrased question, optionally the paragraph-level hits from the
+   judge with their probabilities, shown as quoted evidence.
+2. **Note**: frontmatter, rendered body with `[[wikilinks]]` resolved,
+   backlinks, sources linking to their raw file.
+3. **Browse**: by tag, by `updated`, recently changed, orphans.
+4. **Answers**: a question box. The answer streams in within seconds: rank,
+   judge, compose, all in the request. Below it the evidence: the paragraphs
+   the judge selected with their notes and probabilities, and the `covered`
+   gate; a low gate is shown as "the wiki does not cover this" instead of an
+   answer. A save button writes the answer to `output/` with the wiki-synth
+   frontmatter (`question`, `sources`, `confidence`), so it joins the
+   labelled set. Underneath, the saved answers, newest first, including the
+   ones the wiki-synth agent wrote from a session.
+5. **Maintenance**: the raw queue with ages, the last consolidation and audit
+   reports, index and sync status.
+6. **Lab**: pick any registered embedding model and judge, run the same query
+   through them side by side, and run the evaluator on the labelled set to
+   get recall@k per configuration. This is the page for "play around with
+   different models".
+7. **Settings**: read-only view of the active adapters and their health.
+
+No editing of notes in the browser. The vault stays a git repository edited
+by the consolidate agent and by hand.
+
+## API v1
+
+Reads are authenticated by the hostname's gate (humans) or API key
+(machines); writes always need an API key so a fact carries its author.
+
+| method and path | purpose |
+|---|---|
+| `GET /api/v1/status` | index commit and age, counts, active adapters |
+| `GET /api/v1/notes?q=&tag=&limit=` | list; `q` ranks through the active index |
+| `GET /api/v1/notes/{slug}` | frontmatter, body, links, backlinks, sources |
+| `GET /api/v1/tags` | tags with counts |
+| `POST /api/v1/rank` | `{question, k, model?}` → ranked slugs; also served at `/rank` for the existing scripts |
+| `POST /api/v1/ask` | `{queries, questions, k, judge?, model?}` → context, typed judgements with the answering paragraphs verbatim, `covered`. For agents and code. |
+| `POST /api/v1/questions` | `{question, k?, judge?, model?, answerModel?, save?}` → the answer text with `[[slug]]` citations, the evidence (paragraphs, probabilities), `covered`, model and token usage; streamed as server-sent events when requested; with `save`, the `output/` path and commit |
+| `POST /api/v1/facts` | deposit a raw source: title, source, `kind: fact`, observed_at, content, tags → path and commit |
+| `GET /api/v1/raw` | the distill queue with ages |
+| `GET /api/v1/reports` | `output/` documents by type, newest first |
+| `POST /api/v1/lab/eval` | run the evaluator for a (model, judge) pair |
+| `POST /webhook` | Gitea push, HMAC as today |
+
+`POST /api/v1/facts` is rejected without `source`, with a `kind` other than
+`fact`, or with content under a minimum length. The raw rule is enforced by
+shape: there is no field for a hypothesis. The file gets the vault's
+provenance frontmatter with `captured_by` set from the API key's identity.
+
+`POST /api/v1/ask` is `wiki_ask.py` moved server-side, so the scripts become
+thin clients and agents no longer need a vault checkout. Its result is
+judgements and quotations, never prose.
+
+`POST /api/v1/questions` is `/ask` plus one answer-model call, in process:
+rank the question, build the context, run the default typed questions
+(answered, where, covered), hand the selected paragraphs and judgements to
+the answer model, stream the prose back. Below the `covered` threshold it
+returns the gap instead of an answer. Time is the sum of the parts, about
+0.2 s rank, 1 to 4 s judge, a few seconds of generation. Cost is one model
+call over a few thousand tokens, a few cents on Sonnet, against a quarter
+dollar for a full agent run.
+
+## Auth and network
+
+- Humans: `wiki.renner.uno`, LAN-only (no `dns.renner.uno/public` label),
+  Ingress backend `ak-outpost-wiki-gate:9000`, a `wiki-gate-egress` CNP in
+  `authentik`, the proxy-header handler trusts `X-Authentik-*` only from that
+  outpost.
+- Machines: `wiki-index.renner.uno` with the API-key handler; the workstation
+  key stays where it is. CI reaches the ClusterIP.
+- Secrets (Gitea token, webhook secret, API keys, TypeSafe key, Anthropic
+  key) reach the
+  pod as environment variables from Kubernetes Secrets filled by
+  ExternalSecret from the `openbao` ClusterSecretStore. The pod itself has
+  no OpenBao access. The Python service fetched its secrets from OpenBao
+  directly through its own Kubernetes-auth role; that role, its policy and
+  the `ai` → `openbao` CNP entry are retired with it.
+- Egress from the pod: Gitea, wiki-embed, `api.typesafe.ai:443`,
+  `api.anthropic.com:443`. Every
+  further adapter that talks to a new endpoint gets its own CNP line, which
+  is the one place adding an adapter touches the cluster.
+
+## Health endpoints
+
+Three endpoints with three different meanings, wired to the .NET health
+checks with tags so that the probes and the humans see the right thing:
+
+| endpoint | meaning | probe |
+|---|---|---|
+| `GET /health` | the process serves HTTP. Always 200 once Kestrel listens; no dependency checks. | liveness |
+| `GET /ready` | the active note store has synced at least once and the active index is loaded; 503 while the first sync or index build runs. A failed later sync keeps the last good state and stays ready. | readiness, plus a startup probe with enough budget for the first sync (the Python service was killed mid-build when its `/health` returned 503 during the initial sync) |
+| `GET /api/v1/status` | the full picture for humans and the Settings page: index commit and age, counts, queue lengths, and one entry per registered adapter with its last check result and latency. | none |
+
+Adapter checks (Gitea reachable, embeddings endpoint answering, judge
+endpoint answering, answer model answering) are reported in `/status` and
+on the Settings page and are never part of `/ready`: a judge outage must not
+take the reader down. Their failures show up as red entries, not as
+restarts.
+
+## Phases
+
+One PR each, verified before the next. The Python `wiki-index` keeps running
+until phase 3 proves parity.
+
+0. **Skeleton.** Repository, solution, `Wiki.Core` with the ports and domain
+   types, `Wiki.Contracts` with the suites, `Wiki.Host` serving `/health`,
+   `/ready` and `/api/v1/status`. CI: test and release pipelines. Verify:
+   image in the registry, pipeline green, the Deployment's three probes
+   wired and the pod Ready.
+1. **Parity with the Python service.** Gitea, LlamaCpp and FileIndex
+   adapters, the sync use case, `/rank` and `/webhook`. Verify: the 40-query
+   recall eval gives the same numbers as the Python service on the same
+   commit; `wiki_rank.py --probe-index` against the new service passes.
+2. **Read API and UI.** Notes, tags, raw, reports; Search, Note, Browse,
+   Maintenance pages; `wiki-gate`. Verify: a note renders with resolved
+   links; an unauthenticated request is redirected to Authentik.
+3. **Ask.** Chunker, TypeSafe adapter, `/api/v1/ask`, scripts as clients.
+   Verify: `test_wiki_ask.py` passes against the API; a wiki-synth run with
+   the definition unchanged. Then the Python service is retired and the
+   `wiki-index` hostname points at the new one.
+4. **Facts.** API keys as an ExternalSecret, the deposit-fact use case, `/api/v1/facts`. Verify:
+   a fact posted from the workstation shows as a `claude-bot` commit and in
+   the next distill.
+5. **Questions.** Anthropic adapter, `/api/v1/questions` with streaming,
+   the Answers page with save. Verify: a question answered in under ten
+   seconds with citations that resolve to the shown evidence; a question the
+   wiki does not cover returns the gap; a saved answer appears in `output/`
+   with the wiki-synth frontmatter and is picked up by the evaluator.
+6. **Lab.** Evaluator, a second embedding model and a second answer model
+   registered, the Lab page. Verify: two models compared on the labelled set
+   from the UI.
+
+## Not in scope
+
+- Editing notes in the browser.
+- Replacing Obsidian; the vault stays a git repository and `notes-push.path`
+  keeps pushing.
+- Public exposure; LAN and VPN only.
+- Running or triggering agents. The service makes single model calls inside
+  a request; it never runs a tool loop or edits the vault. Distill,
+  consolidate and multi-question synthesis stay Claude processes in CI or in
+  a session; the service shows their inputs and outputs and serves their API
+  calls.

kubernetes/apps/audiobookshelf/deployment.yaml

@@ -122,7 +122,7 @@ spec:
             claimName: audiobookshelf-metadata
         - name: books
           persistentVolumeClaim:
-            claimName: audiobookshelf-books
+            claimName: audiobookshelf-books-bulk
         - name: le-ca
           emptyDir:
             medium: Memory

kubernetes/apps/audiobookshelf/pvcs.yaml

@@ -35,14 +35,14 @@ spec:
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
-  name: audiobookshelf-books
+  name: audiobookshelf-books-bulk
   namespace: audiobookshelf
   annotations:
     kustomize.toolkit.fluxcd.io/prune: disabled
 spec:
   accessModes:
     - ReadWriteMany
-  storageClassName: ceph-filesystem
+  storageClassName: ceph-filesystem-bulk
   resources:
     requests:
       storage: 300Gi

kubernetes/apps/authentik/blueprints-configmap.yaml

@@ -976,6 +976,280 @@ data:
             kubernetes_ingress_secret_name: "authentik-outpost-tls"
             kubernetes_json_patches: null
 
+      # wiki-gate: the wiki service (ns ai) behind the same passkey flows as
+      # llama-gate. The service trusts X-Authentik-Username only from pod
+      # addresses, and its CNP admits only this outpost, so the gate is the
+      # whole access control for wiki.renner.uno.
+      - id: wiki-gate-provider
+        model: authentik_providers_proxy.proxyprovider
+        identifiers:
+          name: wiki-gate
+        state: present
+        attrs:
+          authentication_flow: !Find [authentik_flows.flow, [slug, passkey-passwordless-authentication]]
+          authorization_flow: !Find [authentik_flows.flow, [slug, passkey-stepup-authorization]]
+          invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
+          mode: proxy
+          internal_host: http://wiki.ai.svc.cluster.local:8080
+          internal_host_ssl_validation: false
+          external_host: https://wiki.renner.uno
+          cookie_domain: renner.uno
+          access_token_validity: hours=24
+          refresh_token_validity: days=30
+          intercept_header_auth: true
+
+      - model: authentik_core.application
+        identifiers:
+          slug: wiki-gate
+        state: present
+        attrs:
+          name: Wiki
+          slug: wiki-gate
+          provider: !KeyOf wiki-gate-provider
+          meta_launch_url: https://wiki.renner.uno
+          meta_description: "Notes wiki: search, notes, answers, maintenance"
+
+      - model: authentik_outposts.outpost
+        identifiers:
+          name: wiki-gate
+        state: present
+        attrs:
+          type: proxy
+          service_connection: !KeyOf k8s-conn
+          providers:
+            - !KeyOf wiki-gate-provider
+          config:
+            log_level: info
+            authentik_host: "http://authentik-server.authentik.svc.cluster.local"
+            authentik_host_insecure: false
+            authentik_host_browser: "https://auth.renner.uno/"
+            refresh_interval: "minutes=5"
+            container_image: null
+            object_naming_template: "ak-outpost-%(name)s"
+            kubernetes_namespace: authentik
+            kubernetes_replicas: 1
+            kubernetes_service_type: ClusterIP
+            kubernetes_disabled_components: ["ingress"]
+            kubernetes_image_pull_secrets: []
+            kubernetes_ingress_annotations: {}
+            kubernetes_ingress_class_name: null
+            kubernetes_ingress_secret_name: "authentik-outpost-tls"
+            kubernetes_json_patches: null
+
+      # paperclip-gate: Paperclip UI; the CNP admits only this outpost. Paperclip keeps its own login behind it.
+      - id: paperclip-users-group
+        model: authentik_core.group
+        identifiers:
+          name: Paperclip Users
+        state: present
+        attrs:
+          is_superuser: false
+          users:
+            - !Find [authentik_core.user, [username, k.renner]]
+
+      - id: paperclip-gate-provider
+        model: authentik_providers_proxy.proxyprovider
+        identifiers:
+          name: paperclip-gate
+        state: present
+        attrs:
+          authentication_flow: !Find [authentik_flows.flow, [slug, passkey-passwordless-authentication]]
+          authorization_flow: !Find [authentik_flows.flow, [slug, passkey-stepup-authorization]]
+          invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
+          mode: proxy
+          internal_host: http://paperclip.paperclip.svc.cluster.local:3100
+          internal_host_ssl_validation: false
+          external_host: https://paperclip.renner.uno
+          cookie_domain: renner.uno
+          access_token_validity: hours=24
+          refresh_token_validity: days=30
+          intercept_header_auth: true
+
+      - model: authentik_core.application
+        identifiers:
+          slug: paperclip-gate
+        state: present
+        attrs:
+          name: Paperclip
+          slug: paperclip-gate
+          provider: !KeyOf paperclip-gate-provider
+          meta_launch_url: https://paperclip.renner.uno
+          meta_description: "Paperclip agent orchestration"
+
+      - model: authentik_policies.policybinding
+        identifiers:
+          target: !Find [authentik_core.application, [slug, paperclip-gate]]
+          group: !KeyOf paperclip-users-group
+          order: 0
+        state: present
+        attrs:
+          enabled: true
+          negate: false
+
+      - model: authentik_outposts.outpost
+        identifiers:
+          name: paperclip-gate
+        state: present
+        attrs:
+          type: proxy
+          service_connection: !KeyOf k8s-conn
+          providers:
+            - !KeyOf paperclip-gate-provider
+          config:
+            log_level: info
+            authentik_host: "http://authentik-server.authentik.svc.cluster.local"
+            authentik_host_insecure: false
+            authentik_host_browser: "https://auth.renner.uno/"
+            refresh_interval: "minutes=5"
+            container_image: null
+            object_naming_template: "ak-outpost-%(name)s"
+            kubernetes_namespace: authentik
+            kubernetes_replicas: 1
+            kubernetes_service_type: ClusterIP
+            kubernetes_disabled_components: ["ingress"]
+            kubernetes_image_pull_secrets: []
+            kubernetes_ingress_annotations: {}
+            kubernetes_ingress_class_name: null
+            kubernetes_ingress_secret_name: "authentik-outpost-tls"
+            kubernetes_json_patches: null
+
+      # navidrome-gate: the Navidrome web UI. Navidrome auto-creates the
+      # X-Authentik-Username user on first login; the first user is admin.
+      # /rest and /share bypass the gate at the Traefik router.
+      - id: navidrome-gate-provider
+        model: authentik_providers_proxy.proxyprovider
+        identifiers:
+          name: navidrome-gate
+        state: present
+        attrs:
+          authorization_flow: !Find [authentik_flows.flow, [slug, default-provider-authorization-implicit-consent]]
+          invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
+          mode: proxy
+          internal_host: http://navidrome.navidrome.svc.cluster.local
+          internal_host_ssl_validation: false
+          external_host: https://navidrome.renner.uno
+          cookie_domain: renner.uno
+          access_token_validity: hours=24
+          refresh_token_validity: days=30
+          intercept_header_auth: true
+
+      - model: authentik_core.application
+        identifiers:
+          slug: navidrome-gate
+        state: present
+        attrs:
+          name: Navidrome
+          slug: navidrome-gate
+          provider: !KeyOf navidrome-gate-provider
+          meta_launch_url: https://navidrome.renner.uno
+          meta_description: "Music streaming (Subsonic)"
+
+      - model: authentik_outposts.outpost
+        identifiers:
+          name: navidrome-gate
+        state: present
+        attrs:
+          type: proxy
+          service_connection: !KeyOf k8s-conn
+          providers:
+            - !KeyOf navidrome-gate-provider
+          config:
+            log_level: info
+            authentik_host: "http://authentik-server.authentik.svc.cluster.local"
+            authentik_host_insecure: false
+            authentik_host_browser: "https://auth.renner.uno/"
+            refresh_interval: "minutes=5"
+            container_image: null
+            object_naming_template: "ak-outpost-%(name)s"
+            kubernetes_namespace: authentik
+            kubernetes_replicas: 1
+            kubernetes_service_type: ClusterIP
+            kubernetes_disabled_components: ["ingress"]
+            kubernetes_image_pull_secrets: []
+            kubernetes_ingress_annotations: {}
+            kubernetes_ingress_class_name: null
+            kubernetes_ingress_secret_name: "authentik-outpost-tls"
+            kubernetes_json_patches: null
+
+      # lidarr-gate: Lidarr runs with AuthenticationMethod=External, so the gate
+      # is its only access control.
+      - id: music-librarians-group
+        model: authentik_core.group
+        identifiers:
+          name: Music Librarians
+        state: present
+        attrs:
+          is_superuser: false
+          # Membership is owned here; blueprint apply reverts UI edits.
+          users:
+            - !Find [authentik_core.user, [username, k.renner]]
+
+      - id: lidarr-gate-provider
+        model: authentik_providers_proxy.proxyprovider
+        identifiers:
+          name: lidarr-gate
+        state: present
+        attrs:
+          authorization_flow: !Find [authentik_flows.flow, [slug, default-provider-authorization-implicit-consent]]
+          invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
+          mode: proxy
+          internal_host: http://lidarr.navidrome.svc.cluster.local
+          internal_host_ssl_validation: false
+          external_host: https://lidarr.renner.uno
+          cookie_domain: renner.uno
+          access_token_validity: hours=24
+          refresh_token_validity: days=30
+          intercept_header_auth: true
+
+      - model: authentik_core.application
+        identifiers:
+          slug: lidarr-gate
+        state: present
+        attrs:
+          name: Lidarr
+          slug: lidarr-gate
+          provider: !KeyOf lidarr-gate-provider
+          meta_launch_url: https://lidarr.renner.uno
+          meta_description: "Music library gaps: discographies, Spotify artists"
+
+      - model: authentik_policies.policybinding
+        identifiers:
+          target: !Find [authentik_core.application, [slug, lidarr-gate]]
+          group: !KeyOf music-librarians-group
+          order: 0
+        state: present
+        attrs:
+          enabled: true
+          negate: false
+          failure_result: false
+
+      - model: authentik_outposts.outpost
+        identifiers:
+          name: lidarr-gate
+        state: present
+        attrs:
+          type: proxy
+          service_connection: !KeyOf k8s-conn
+          providers:
+            - !KeyOf lidarr-gate-provider
+          config:
+            log_level: info
+            authentik_host: "http://authentik-server.authentik.svc.cluster.local"
+            authentik_host_insecure: false
+            authentik_host_browser: "https://auth.renner.uno/"
+            refresh_interval: "minutes=5"
+            container_image: null
+            object_naming_template: "ak-outpost-%(name)s"
+            kubernetes_namespace: authentik
+            kubernetes_replicas: 1
+            kubernetes_service_type: ClusterIP
+            kubernetes_disabled_components: ["ingress"]
+            kubernetes_image_pull_secrets: []
+            kubernetes_ingress_annotations: {}
+            kubernetes_ingress_class_name: null
+            kubernetes_ingress_secret_name: "authentik-outpost-tls"
+            kubernetes_json_patches: null
+
       - model: authentik_outposts.outpost
         identifiers:
           name: authentik Embedded Outpost

kubernetes/apps/authentik/kustomization.yaml

@@ -7,6 +7,7 @@ resources:
   - helmrelease.yaml
   - paperless-gate-ingress.yaml
   - paperless-gate-egress.yaml
+  - paperclip-gate-egress.yaml
   - vscode-gate-egress.yaml
   - shelly-manager-gate-egress.yaml
   - esphome-gate-egress.yaml
@@ -16,6 +17,9 @@ resources:
   - prometheus-gate-egress.yaml
   - wordpress-gate-egress.yaml
   - llama-gate-egress.yaml
+  - wiki-gate-egress.yaml
+  - navidrome-gate-egress.yaml
+  - lidarr-gate-egress.yaml
   - ldap-outpost-cert.yaml
   - ldap-outpost-lb.yaml
   - ldap-outpost-cnp.yaml

kubernetes/apps/authentik/lidarr-gate-egress.yaml

@@ -0,0 +1,20 @@
+# Lets the lidarr-gate outpost reach Lidarr; the matching ingress rule is in
+# network-policies/navidrome-ns.yaml.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: lidarr-gate-egress
+  namespace: authentik
+spec:
+  endpointSelector:
+    matchLabels:
+      goauthentik.io/outpost-name: lidarr-gate
+  egress:
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": navidrome
+            app.kubernetes.io/name: lidarr
+      toPorts:
+        - ports:
+            - port: "8686"
+              protocol: TCP

kubernetes/apps/authentik/navidrome-gate-egress.yaml

@@ -0,0 +1,20 @@
+# Lets the navidrome-gate outpost reach Navidrome; the matching ingress rule
+# is in network-policies/navidrome-ns.yaml.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: navidrome-gate-egress
+  namespace: authentik
+spec:
+  endpointSelector:
+    matchLabels:
+      goauthentik.io/outpost-name: navidrome-gate
+  egress:
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": navidrome
+            app.kubernetes.io/name: navidrome
+      toPorts:
+        - ports:
+            - port: "4533"
+              protocol: TCP

kubernetes/apps/authentik/paperclip-gate-egress.yaml

@@ -0,0 +1,19 @@
+# Lets the paperclip-gate outpost reach the Paperclip server; scoped to this outpost's pods.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: paperclip-gate-egress
+  namespace: authentik
+spec:
+  endpointSelector:
+    matchLabels:
+      goauthentik.io/outpost-name: paperclip-gate
+  egress:
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": paperclip
+            app.kubernetes.io/name: paperclip
+      toPorts:
+        - ports:
+            - port: "3100"
+              protocol: TCP

kubernetes/apps/authentik/wiki-gate-egress.yaml

@@ -0,0 +1,21 @@
+# Lets the wiki-gate outpost reach the wiki service. The authentik namespace
+# baseline CNP otherwise stops any authentik pod from entering ai. Scoped to
+# this outpost's pod label; the matching ingress rule is in ai-ns.yaml.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: wiki-gate-egress
+  namespace: authentik
+spec:
+  endpointSelector:
+    matchLabels:
+      goauthentik.io/outpost-name: wiki-gate
+  egress:
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": ai
+            app.kubernetes.io/name: wiki
+      toPorts:
+        - ports:
+            - port: "8080"
+              protocol: TCP

kubernetes/apps/gatus/configmap.yaml

@@ -30,6 +30,35 @@ metadata:
 #     Web Service 9 = Music Assistant
 #     Web Service 10 = TRMNL
 #     Web Service 11 = Paperless (gate-level only, see its entry)
+#
+# CORRECTION (REN-21, 2026-09-25) to the Web Service 11 comment further down,
+# which claims that an in-cluster probe of the paperless ClusterIP "needs a
+# CiliumNetworkPolicy first — the monitoring probes get away without one only
+# because their targets share a namespace with the prober".
+#
+# That is wrong, and it was wrong when written. It is stated here rather than
+# fixed in place because the sentence lives inside the `config.yaml: |` block
+# scalar below: any edit to it changes ConfigMap DATA, and
+# `configmap.reloader.stakater.com/reload: gatus-config`
+# (gatus/deployment.yaml:13) would then roll the gatus Deployment — a
+# production restart for a comment. This header is outside `data:`, so it is
+# inert; the served config is byte-identical.
+#
+#   * infrastructure/network-policies/paperless-ns.yaml:34-36 ALREADY admits
+#     the whole monitoring namespace on every port, and monitoring is where
+#     blackbox-exporter runs. Cilium unions the allow rules of every CNP
+#     selecting an endpoint, so no extra policy is required.
+#   * The in-cluster paperless probe therefore exists, added with no change
+#     to the paperless namespace:
+#     monitoring-rules/probes-blackbox.yaml, target
+#     http://paperless.paperless.svc.cluster.local:80/accounts/login/
+#   * The claim does NOT generalise to gatus itself. Gatus runs in `apps`,
+#     which is a different source namespace; a gatus-sourced in-cluster check
+#     would need its own allow, exactly as
+#     cnp-cross-namespace-allow-apps-gatus-to-monitored-services describes.
+#
+# The Web Service 11 entry below stays a reachability-only check on purpose.
+# The app-health half of the pair is the blackbox probe named above.
 data:
   config.yaml: |
     storage:

kubernetes/apps/gatus/deployment.yaml

@@ -46,7 +46,7 @@ spec:
           type: RuntimeDefault
       containers:
         - name: gatus
-          image: twinproduction/gatus:v5.36.0 # {"$imagepolicy": "flux-system:gatus"}
+          image: twinproduction/gatus:v5.37.0 # {"$imagepolicy": "flux-system:gatus"}
           securityContext:
             allowPrivilegeEscalation: false
             capabilities:

kubernetes/apps/home-assistant/ha-pg-cluster.yaml

@@ -36,12 +36,32 @@ spec:
   imageName: ghcr.io/cloudnative-pg/postgresql:16
   storage:
     storageClass: ceph-block
-    # Sized for the migrated statistics plus 7 days of states at the current
-    # rate: statistics 10.8M rows + statistics_short_term 1.9M (~1.8 GB with
-    # indexes in Postgres) and ~18M states rows for the purge_keep_days:7 window
-    # (~4 GB). 20Gi leaves room for autovacuum bloat and index churn.
-    # ceph-block has allowVolumeExpansion: true, so this can grow in place.
-    size: 20Gi
+    # Sized for WAL headroom during an archiving stall, not for table growth.
+    # The data itself needs far less: migrated statistics 10.8M rows +
+    # statistics_short_term 1.9M (~1.8 GB with indexes in Postgres) and ~18M
+    # states rows for the purge_keep_days:7 window (~4 GB).
+    #
+    # 20Gi left only 6.63 GiB free, and that is the number that matters here.
+    # WAL archiving stalled for 10 minutes and the un-archived backlog reached
+    # 1.84 GiB — 28% of that free space (REN-129).
+    #
+    # That stall was first blamed on Velero repository maintenance. It is not
+    # the cause: REN-136 measured seven nights and the archiver's sustained
+    # drain (~14 segments/min, its serial ceiling) is flat whether Kopia is
+    # running or not, and Kopia's nearest batch drifts across 02:03-02:24Z
+    # while the burst stays pinned to 02:13Z. What is real is a nightly
+    # producer burst — 56-119 segments queued at 02:13-02:15 UTC on all seven
+    # nights, against a steady-state queue of 0. Postgres cannot recycle a WAL
+    # segment until the archiver has shipped it, so a burst that arrives while
+    # archiving is stalled walks the volume toward full, and a full PGDATA
+    # stops the recorder outright.
+    #
+    # 32Gi takes free space to ~18.6 GiB, which puts the same 1.84 GiB backlog
+    # at 10% of headroom rather than 28%. In REN-136's terms: at 6.36 GiB free
+    # one stalled night survives and two fill the volume; ~18.6 GiB carries
+    # about four. Approved by the owner on REN-135.
+    # ceph-block has allowVolumeExpansion: true, so this grows in place.
+    size: 32Gi
   plugins:
     - name: barman-cloud.cloudnative-pg.io
       isWALArchiver: true

kubernetes/apps/home-assistant/kustomization.yaml

@@ -16,6 +16,9 @@ resources:
   # health broker credential: OpenBao-backed + Class-A rotated (first of the
   # eight users to migrate off mosquitto-bootstrap.yaml).
   - mosquitto-health-credentials-externalsecret.yaml
+  # open3e-int broker credential: open3e-esp32 gateway on the Vitocal internal
+  # CAN; OpenBao-backed + Class-A rotated.
+  - mosquitto-open3e-int-credentials-externalsecret.yaml
   - mosquitto-config.yaml
   - mosquitto-certificate.yaml
   - mosquitto-service.yaml

kubernetes/apps/home-assistant/mosquitto-deployment.yaml

@@ -12,7 +12,7 @@ metadata:
     #
     # Listed per-Secret rather than with the blanket auto annotation, so that
     # enrolling another broker user is a visible edit to this line.
-    secret.reloader.stakater.com/reload: mosquitto-health-credentials
+    secret.reloader.stakater.com/reload: mosquitto-health-credentials,mosquitto-open3e-int-credentials
 spec:
   replicas: 1
   strategy:
@@ -78,6 +78,7 @@ spec:
               mosquitto_passwd -b /mosquitto/auth/passwd admin "$ADMIN_PASSWORD"
               mosquitto_passwd -b /mosquitto/auth/passwd sml2mqtt "$SML2MQTT_PASSWORD"
               mosquitto_passwd -b /mosquitto/auth/passwd open3e "$OPEN3E_PASSWORD"
+              mosquitto_passwd -b /mosquitto/auth/passwd open3e-int "$OPEN3E_INT_PASSWORD"
               mosquitto_passwd -b /mosquitto/auth/passwd evcc "$EVCC_PASSWORD"
               mosquitto_passwd -b /mosquitto/auth/passwd health "$HEALTH_PASSWORD"
               mosquitto_passwd -b /mosquitto/auth/passwd zigbee2mqtt "$ZIGBEE2MQTT_PASSWORD"
@@ -124,6 +125,12 @@ spec:
               user open3e
               topic readwrite open3e/#
 
+              # open3e-esp32 gateway on the Vitocal internal CAN: own subtree
+              # under open3e/ so edge-1's listener on open3e/cmnd never sees
+              # its commands.
+              user open3e-int
+              topic readwrite open3e/int/#
+
               user evcc
               topic read open3e/#
               topic readwrite evcc/#
@@ -187,6 +194,13 @@ spec:
                 secretKeyRef:
                   name: mosquitto-open3e-credentials
                   key: password
+            # open3e-int: OpenBao-backed + Class-A rotated (see
+            # mosquitto-open3e-int-credentials-externalsecret.yaml).
+            - name: OPEN3E_INT_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: mosquitto-open3e-int-credentials
+                  key: password
             - name: EVCC_PASSWORD
               valueFrom:
                 secretKeyRef:

kubernetes/apps/home-assistant/mosquitto-open3e-int-credentials-externalsecret.yaml

@@ -0,0 +1,21 @@
+# Projects the mosquitto `open3e-int` broker credential (open3e-esp32 gateway
+# on the Vitocal internal CAN) from OpenBao into the Secret the broker
+# consumes. Rotator: kubernetes/infrastructure/openbao/rotation/mosquitto-open3e-int.yaml.
+#
+# creationPolicy is implicitly Owner: deleting this ExternalSecret deletes the
+# live Secret. Set creationPolicy: Orphan first to detach without destroying.
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: mosquitto-open3e-int-credentials
+  namespace: home-assistant
+spec:
+  refreshInterval: 10m
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: openbao
+  target:
+    name: mosquitto-open3e-int-credentials
+  dataFrom:
+    - extract:
+        key: apps/home-assistant/mosquitto-open3e-int-credentials

kubernetes/apps/immich/pvcs.yaml

@@ -28,6 +28,38 @@ spec:
     requests:
       storage: 300Gi
 ---
+# Media root on the EC 2+1 bulk pool; thumbs mount separately below.
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: immich-library-bulk
+  namespace: immich
+  annotations:
+    kustomize.toolkit.fluxcd.io/prune: disabled
+spec:
+  accessModes:
+    - ReadWriteMany
+  storageClassName: ceph-filesystem-bulk
+  resources:
+    requests:
+      storage: 300Gi
+---
+# Thumbnails stay replicated: read affinity serves them from the local OSD.
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: immich-thumbs
+  namespace: immich
+  annotations:
+    kustomize.toolkit.fluxcd.io/prune: disabled
+spec:
+  accessModes:
+    - ReadWriteMany
+  storageClassName: ceph-filesystem
+  resources:
+    requests:
+      storage: 30Gi
+---
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:

kubernetes/apps/immich/server.yaml

@@ -110,6 +110,8 @@ spec:
           volumeMounts:
             - name: library
               mountPath: /data
+            - name: thumbs
+              mountPath: /data/thumbs
             - name: immich-config-file
               mountPath: /config
               readOnly: true
@@ -145,7 +147,10 @@ spec:
       volumes:
         - name: library
           persistentVolumeClaim:
-            claimName: immich-library
+            claimName: immich-library-bulk
+        - name: thumbs
+          persistentVolumeClaim:
+            claimName: immich-thumbs
         - name: immich-config-file
           secret:
             secretName: immich-secrets

kubernetes/apps/libation/cronjob.yaml

@@ -4,7 +4,7 @@
 # the entrypoint exit after one pass instead of looping internally —
 # the CronJob handles scheduling.
 #
-# Books land in /audiobooks (the audiobookshelf-books PVC, shared
+# Books land in /audiobooks (the audiobookshelf-books-bulk PVC, shared
 # with the audiobookshelf Deployment). Config + DB live on
 # libation-config (RWO ceph-block, seeded once from
 # libation-bootstrap secret).
@@ -107,6 +107,9 @@ spec:
                 - |
                   trap 'cp /config-internal/AccountsSettings.json /libation-state/AccountsSettings.json' EXIT
                   set -e
+                  # Strip the Audible intro/outro; re-liberated books replace their file instead of gaining a " (1)" copy.
+                  jq '.StripAudibleBrandAudio = true | .OverwriteExisting = true' /config-internal/Settings.json > /config-internal/Settings.json.tmp
+                  mv /config-internal/Settings.json.tmp /config-internal/Settings.json
                   /libation/LibationCli scan
                   /libation/LibationCli liberate
               env:
@@ -135,7 +138,7 @@ spec:
                 claimName: libation-config
             - name: books
               persistentVolumeClaim:
-                claimName: audiobookshelf-books
+                claimName: audiobookshelf-books-bulk
             - name: bootstrap
               secret:
                 secretName: libation-bootstrap

kubernetes/apps/libation/pvc.yaml

@@ -5,7 +5,7 @@
 # SOPS-encrypted Secret on first boot (sentinel file gate); after that
 # Libation reads and writes it directly so token rotation persists.
 #
-# Books are written to the audiobookshelf-books PVC (already provisioned
+# Books are written to the audiobookshelf-books-bulk PVC (already provisioned
 # by the audiobookshelf seed Kustomization, RWX ceph-filesystem). We
 # share the same namespace so the PVC reference is direct — k8s PVCs
 # are namespace-scoped.

kubernetes/apps/monitoring-rules/cilium-servicemonitor.yaml

@@ -1,8 +1,75 @@
+---
+# The agent metrics Service, declared here because the chart does not render
+# one under our values (REN-147).
+#
+# cilium's templates/cilium-agent/service.yaml is gated twice:
+#
+#   {{- if and .Values.agent (not .Values.preflight.enabled) .Values.prometheus.enabled }}
+#   {{- if (or .Values.prometheus.serviceMonitor.enabled .Values.prometheus.metricsService) }}
+#
+# infrastructure/cilium/helmrelease.yaml sets `prometheus.enabled: true` and
+# `prometheus.serviceMonitor.enabled: false`, and leaves `metricsService` at its
+# default of false. First gate passes, second does not. The chart's fallback
+# branch is envoy-only and needs envoy in the agent pod, but envoy runs as its
+# own DaemonSet here — so no `cilium-agent` Service was rendered at all.
+#
+# The consequence was silent: agents exported metrics on :9962 all along, the
+# ServiceMonitor below selected a Service that did not exist, and the scrape
+# pool sat at 0 active / 46 dropped targets. `cilium_*` had zero series, so
+# CiliumBPFMapPressureHigh and CiliumBPFMapFull in prometheusrule-cilium.yaml
+# evaluated an empty vector forever and could never fire.
+#
+# Why a Service here and not `prometheus.metricsService: true` in the
+# HelmRelease: that flag is the chart-native fix and its rendered diff is
+# exactly this one object (it appears in no other template — not the
+# ConfigMap, not the DaemonSet). But setting it changes the HelmRelease values,
+# which makes helm-controller run `helm upgrade` on the CNI release, and that
+# three-way-merges against live objects. PROPOSAL-cilium-single-owner.md
+# records that `cilium-config` has carried a `kubectl-patch` field manager; if
+# any of that drift is still live, the upgrade reconciles it, and the
+# `configmap.reloader.stakater.com/reload: cilium-config` annotation on the
+# HelmRelease then rolls every agent node by node. A Service applied by the
+# apps Kustomization cannot do that. If the drift is ever proven gone, delete
+# this object and set `prometheus.metricsService: true` instead.
+#
+# Name and selector follow authentik-outpost-servicemonitors.yaml: a distinct
+# name plus `app.kubernetes.io/managed-by: homelab`, so flipping either chart
+# flag later renders the chart's own `cilium-agent` Service without an
+# ownership conflict and without double-scraping the same endpoints.
+apiVersion: v1
+kind: Service
+metadata:
+  name: cilium-agent-metrics
+  namespace: kube-system
+  labels:
+    app.kubernetes.io/managed-by: homelab
+    app.kubernetes.io/name: cilium-agent
+    app.kubernetes.io/part-of: cilium
+    # Copied onto every series by the ServiceMonitor's targetLabels below.
+    # The dashboard the chart ships queries
+    # cilium_bpf_map_pressure{k8s_app="cilium"}, so this label is load-bearing.
+    k8s-app: cilium
+spec:
+  type: ClusterIP
+  # Headless on purpose: agent metrics are per-node and every endpoint has to
+  # be scraped. A ClusterIP would hand Prometheus one random agent per scrape.
+  clusterIP: None
+  # Agent pods only. cilium-envoy carries k8s-app: cilium-envoy and exposes
+  # 9964, so it cannot match.
+  selector:
+    k8s-app: cilium
+  ports:
+    # Named and numbered exactly as the chart's own Service, so the
+    # ServiceMonitor below stays a field-for-field copy of the chart's.
+    - name: metrics
+      port: 9962
+      targetPort: prometheus
+      protocol: TCP
+---
 # Cilium's agent ServiceMonitor, declared here rather than by the chart.
 #
-# Reproduces what cilium 1.20.1 renders with prometheus.serviceMonitor.enabled,
-# taken field for field off the live object so scraping is unchanged: same name,
-# same namespace, same selector, same 10s interval and node relabeling.
+# Reproduces what cilium renders with prometheus.serviceMonitor.enabled: same
+# name, same namespace, same 10s interval and node relabeling.
 #
 # It lives here because the chart's template is gated on monitoring.coreos.com/v1
 # existing at install time, and on an empty cluster it does not — those CRDs
@@ -22,13 +89,14 @@ metadata:
     app.kubernetes.io/part-of: cilium
 spec:
   selector:
+    # Pinned to the Service above. Without managed-by this would also match a
+    # chart-rendered `cilium-agent` Service and scrape the same endpoints twice.
     matchLabels:
+      app.kubernetes.io/managed-by: homelab
       app.kubernetes.io/name: cilium-agent
   namespaceSelector:
     matchNames:
       - kube-system
-  # The chart's Service keeps rendering (prometheus.enabled), so the target and
-  # its `metrics` port are there regardless of who declares the ServiceMonitor.
   targetLabels:
     - k8s-app
   endpoints:

kubernetes/apps/monitoring-rules/external-secrets-podmonitor.yaml

@@ -0,0 +1,71 @@
+# Scrape the external-secrets (ESO) controller's own metrics. Until this
+# existed, nothing in this cluster could see a failed ExternalSecret sync:
+# `externalsecret_status_condition` held 0 series, so no rule could be written
+# about a key that silently stops syncing (REN-100).
+#
+# This is the scrape only. The alert rule is deliberately NOT in this change —
+# its `for:` window has to be tuned against real data, and that data does not
+# exist until this has been live for a while. See REN-103.
+#
+# A PodMonitor, NOT the chart's `serviceMonitor.enabled`, and NOT a hand-written
+# ServiceMonitor. Both alternatives were checked against chart 2.11.0:
+#
+#   * A hand-written ServiceMonitor has nothing to select. `metrics.service.enabled`
+#     defaults to false and infrastructure/external-secrets/helmrelease.yaml
+#     overrides nothing, so templates/service.yaml renders no Service for the
+#     controller. The container port exists (`name: metrics`, containerPort 8080)
+#     but nothing fronts it — the same trap that forced a PodMonitor for the Flux
+#     controllers.
+#   * Setting `serviceMonitor.enabled: true` would work, but it triggers a Helm
+#     UPGRADE of ESO, which re-renders the Deployment and re-applies its CRDs
+#     (`crds: CreateReplace` in the HelmRelease). ESO is what all 54 ExternalSecret
+#     manifests in this repo depend on. A PodMonitor gets the same series while
+#     touching the release not at all, which is the trade this repo already made
+#     for Velero's node-agent (velero-podmonitor.yaml) and for Flux.
+#
+# THIS NEEDS A NETWORK POLICY CHANGE, unlike the Flux PodMonitor: the cluster is
+# default-deny and external-secrets-ns.yaml admitted only intra-namespace traffic,
+# host/remote-node and ICMP — no `monitoring` grant, so this scrape would have
+# been dropped and the metric would have stayed absent. The matching ingress rule
+# is added to infrastructure/network-policies/external-secrets-ns.yaml in the
+# same pull request. Without it this file is decoration.
+apiVersion: monitoring.coreos.com/v1
+kind: PodMonitor
+metadata:
+  name: external-secrets
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/name: external-secrets
+    # Not required for discovery — podMonitorSelector is empty, see
+    # apps/monitoring/helmrelease-prometheus.yaml — but set for consistency with
+    # the other monitors in this directory.
+    release: kube-prometheus-stack
+spec:
+  # Lives in `monitoring` rather than in `external-secrets`, matching
+  # external-dns-servicemonitor.yaml and the Flux PodMonitor. Every object in
+  # the external-secrets namespace is owned by the ESO HelmRelease; keeping this
+  # out of that namespace means this change cannot interact with the release.
+  namespaceSelector:
+    matchNames:
+      - external-secrets
+  selector:
+    matchLabels:
+      # The CONTROLLER only. The webhook and cert-controller carry
+      # `app.kubernetes.io/name: external-secrets-webhook` and
+      # `-cert-controller` respectively (see the chart's
+      # external-secrets-webhook.selectorLabels / -cert-controller.selectorLabels
+      # helpers), so an exact match on the unsuffixed name excludes both. That is
+      # intended: the `externalsecret_*` series are emitted by the controller, and
+      # the other two would only add idle targets.
+      app.kubernetes.io/name: external-secrets
+      app.kubernetes.io/instance: external-secrets
+  podMetricsEndpoints:
+    # Named container port, not a hardcoded 8080, so this follows
+    # `metrics.listen.port` if it is ever overridden in the HelmRelease.
+    - port: metrics
+      # 60s, not the chart's 30s default: `externalsecret_status_condition` is a
+      # slow-moving gauge over ~54 ExternalSecrets, and the alert built on it in
+      # REN-103 will use a multi-minute `for:` window. 30s would double the
+      # samples for no gain in detection time.
+      interval: 60s
+      scrapeTimeout: 10s

kubernetes/apps/monitoring-rules/flux-podmonitor.yaml

@@ -0,0 +1,80 @@
+# Scrape the Flux controllers' own metrics. Until this existed Prometheus held
+# no `gotk_*` series at all, so nothing about Flux's own behaviour was
+# queryable and no rule could be written about it (REN-97).
+#
+# THIS IS THE SCRAPE ONLY, and the readiness alert is not merely deferred —
+# it cannot be written from this endpoint. `gotk_reconcile_condition` and
+# `gotk_suspend_status` are still DECLARED by fluxcd/pkg/runtime (v0.110.3,
+# runtime/metrics/recorder.go), but at the tags pinned in gotk-components.yaml
+# no controller calls the recorder that populates them: grepping the six
+# tagged source trees for RecordReadiness / RecordCondition / RecordSuspend
+# returns zero call sites, and an unpopulated GaugeVec exports no series. So
+# the readiness metrics would stay empty however this file is written. Flux
+# moved per-object readiness to kube-state-metrics custom-resource-state
+# (`gotk_resource_info{ready=...}`); that is REN-102, and it is a prerequisite
+# for the not-Ready alert rather than a follow-up to it.
+#
+# What this scrape does yield, and why it is still worth merging on its own:
+#   * `gotk_reconcile_duration_seconds` per kind/name/namespace (RecordDuration
+#     IS called, by all six controllers) — reconcile latency and, by its
+#     absence, an object that has stopped being reconciled at all.
+#   * controller-runtime's own series: reconcile totals and errors per
+#     controller, workqueue depth and retry counts, leader election.
+#   * the target itself, so `up{job="monitoring/flux-controllers"}` and the
+#     chart's default TargetDown rule start covering the Flux control plane.
+#   * the scrape REN-102 needs in place anyway.
+#
+# A PodMonitor, NOT a ServiceMonitor, and that is forced rather than preferred:
+# gotk-components.yaml creates Services for only two of the six controllers
+# (source-controller, notification-controller), and both publish port 80 ->
+# targetPort `http`, which is the artifact/webhook server on :9090 — not the
+# metrics port. kustomize-, helm-, image-reflector- and image-automation-
+# controller have no Service at all, so there is nothing for a ServiceMonitor to
+# select. All six pods do carry the same two labels and the same `http-prom`
+# (:8080) container port, so one PodMonitor covers the set.
+#
+# NO NETWORK POLICY CHANGE IS NEEDED, verified rather than assumed, and unlike
+# external-secrets-podmonitor.yaml which did need one: flux-system has no
+# CiliumNetworkPolicy of its own (infrastructure/network-policies/ has no
+# flux-system file), and gotk-components.yaml ships an `allow-scraping`
+# NetworkPolicy admitting every namespace to :8080 (`from: [namespaceSelector:
+# {}]`, port 8080). So `monitoring` is already admitted, as it was for REN-21.
+# Nothing in the Flux install is touched.
+apiVersion: monitoring.coreos.com/v1
+kind: PodMonitor
+metadata:
+  name: flux-controllers
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/part-of: flux
+    # Not required for discovery — podMonitorSelector is empty, see
+    # apps/monitoring/helmrelease-prometheus.yaml — but set for consistency
+    # with the other monitors in this directory.
+    release: kube-prometheus-stack
+spec:
+  # Lives in `monitoring` with a namespaceSelector rather than in flux-system,
+  # matching external-dns-servicemonitor.yaml and external-secrets-podmonitor.yaml.
+  # Flux owns every object in flux-system via gotk-components.yaml; keeping this
+  # out of that namespace means this change cannot interact with the Flux install
+  # itself.
+  namespaceSelector:
+    matchNames:
+      - flux-system
+  selector:
+    matchLabels:
+      app.kubernetes.io/part-of: flux
+      app.kubernetes.io/instance: flux-system
+  podMetricsEndpoints:
+    # Matches the controllers' `http-prom` port (containerPort 8080). Present on
+    # all six Deployments in gotk-components.yaml; a future controller without
+    # it simply yields no target rather than a broken scrape config.
+    - port: http-prom
+      path: /metrics
+      # 60s, not the 30s used elsewhere here, matching the ESO PodMonitor: the
+      # useful series are slow-moving per-object gauges and histograms over a
+      # few hundred Flux objects, and any rule built on them (REN-102) will use
+      # a multi-minute `for:`. 30s would double the samples for no gain in
+      # detection time — and Prometheus is already near its retention cap
+      # (REN-107).
+      interval: 60s
+      scrapeTimeout: 10s

kubernetes/apps/monitoring-rules/kustomization.yaml

@@ -30,6 +30,9 @@ resources:
   - prometheusrule-dns-resolution.yaml
   - probes-dns.yaml
   - alloy-servicemonitor.yaml
+  # Agent metrics. Ships the metrics Service as well as the ServiceMonitor:
+  # the chart renders neither under our values, so this scraped nothing and
+  # the two CiliumBPFMapPressure alerts could never fire. See the file.
   - cilium-servicemonitor.yaml
   - cnpg-podmonitors.yaml
   - crowdsec-servicemonitor.yaml
@@ -48,4 +51,22 @@ resources:
   - loki-servicemonitor.yaml
   - smartctl-exporter-servicemonitor.yaml
   - intel-gpu-exporter-servicemonitor.yaml
+  # ESO's controller metrics. Needs the `monitoring` ingress grant added to
+  # infrastructure/network-policies/external-secrets-ns.yaml in the same change,
+  # or it scrapes nothing. See the file.
+  - external-secrets-podmonitor.yaml
+  # Flux's own control plane. The controllers were never scraped, so nothing
+  # about Flux's behaviour was queryable. Scrape only: per-object readiness is
+  # not on this endpoint at the pinned controller versions and needs
+  # kube-state-metrics instead. See the file.
+  - flux-podmonitor.yaml
+  # The readiness half the PodMonitor above cannot provide. Reads
+  # `gotk_resource_info`, which comes from the kube-state-metrics
+  # customResourceState block in apps/monitoring/helmrelease-prometheus.yaml —
+  # these two files must land and be reverted together.
+  - prometheusrule-flux.yaml
   - probes-blackbox.yaml
+  # TEMPORARY, REN-140. Negative-test probes for the REN-28 §7 bar, taking the
+  # "reachable before" reading that stops being takeable once the paperless PR
+  # merges. Delete this line and the file once the "after" reading is recorded.
+  - probes-ren28-negative.yaml

kubernetes/apps/monitoring-rules/probes-blackbox.yaml

@@ -88,6 +88,169 @@ spec:
         # /ready answers 200 unauthenticated and is in this namespace, so the
         # same intra-namespace allowance the two above rely on covers it.
         - http://loki.monitoring.svc.cluster.local:3100/ready
+        # APP-HEALTH probes for the two gated navidrome-namespace apps. Same
+        # rule as the block above (monitoring-vantage-point-rule), but these
+        # are the first targets here that leave the monitoring namespace.
+        #
+        # Both sit entirely behind an Authentik outpost — navidrome.renner.uno
+        # on ak-outpost-navidrome-gate (navidrome/ingress.yaml:21),
+        # lidarr.renner.uno on ak-outpost-lidarr-gate
+        # (navidrome/lidarr.yaml:218) — so an EXTERNAL probe of either name
+        # measures the gate, not the app, and stays green while the app behind
+        # a live gate is dead. That external signal is still wanted, and REN-2
+        # adds it from UptimeRobot; these are the other half of the pair.
+        #
+        # NO CiliumNetworkPolicy change is needed — but NOT because
+        # cnp-cross-namespace-allow-apps-gatus-to-monitored-services fails to
+        # apply here. It applies: its Generalization section names
+        # blackbox-exporter by hand. The reason we are safe is that the allow
+        # that note demands is PRESENT AND DEDICATED — the navidrome baseline
+        # CNP (infrastructure/network-policies/navidrome-ns.yaml) admits
+        # blackbox-exporter by pod label on exactly the two listeners these
+        # two probes reach, :4533 and :8686. It used to admit the whole
+        # monitoring namespace on every port; REN-28 narrowed it, and the
+        # ports in that rule are the ones these probes depend on, so a probe
+        # added here against a THIRD listener in that namespace would need
+        # that rule widened first. Run that check for the next namespace you
+        # add; do not skip it on the grounds that the note "is about gatus".
+        #
+        # EGRESS — the half the wiki note is silent on, and the half that is
+        # actually load-bearing here. Cilium enforces at BOTH ends, so the
+        # destination ingress allow above is worthless if the prober itself
+        # is in egress enforcement without a matching rule. It is not: an
+        # endpoint only enters egress enforcement once some policy selecting
+        # it declares an `egress:` block, and none does for monitoring. The
+        # namespace has exactly two Cilium policies and both are ingress-only:
+        # infrastructure/network-policies/monitoring-ns.yaml
+        # (default-deny-ingress) and monitoring-alloy-syslog.yaml
+        # (allow-syslog-world-ingress). blackbox-exporter runs in that same
+        # namespace (apps/monitoring/helmrelease-blackbox-exporter.yaml:5).
+        # Re-verified at 2b71dda1 across all 635 yaml files in the repo — 73
+        # Cilium objects, 38 carrying an egress or egressDeny block, none of
+        # them in the monitoring namespace, and the single
+        # CiliumClusterwideNetworkPolicy is scoped to `reserved.health` so it
+        # cannot select a pod. 17 other namespaces here already have an egress
+        # block. If a routine hardening pass ever adds one to monitoring, it
+        # MUST list the navidrome and paperless namespaces or these three
+        # cross-namespace targets go red while the *.monitoring.svc ones above
+        # keep working — a failure mode nobody will connect back to the
+        # hardening commit. The cnp-cross-namespace-* notes document only the
+        # ingress half; that silence is what makes this the trap.
+        #
+        # BLAST RADIUS if that egress dependency is ever broken. Nothing
+        # user-facing breaks: these are Prometheus-side Probe targets, not
+        # gatus checks, so status.renner.uno is unaffected and no workload is
+        # restarted. What breaks is the signal, and it breaks misleadingly.
+        # All three targets start failing BlackboxProbeFailing
+        # (prometheusrule-blackbox.yaml:34 — severity warning, for: 10m); none
+        # of the three is in that rule's exclusion list. BlackboxProbesAbsent
+        # does NOT fire, because the other 17 targets keep producing
+        # probe_success — so it reads as three apps going down, not as a
+        # policy change. WHO NOTICES: warning routes to the default `email`
+        # receiver, admin@renner.uno (helmrelease-prometheus.yaml alertmanager
+        # route, group_wait 30s, grouped by alertname), so the owner gets one
+        # email roughly 10-11 minutes after the policy lands. HOW FAST UNDONE:
+        # the REVERT recipe below is a one-line deletion; from merge it is
+        # live within ~11 minutes (GitRepository interval 1m0s,
+        # monitoring-rules Kustomization interval 10m), or immediately with
+        # `flux reconcile kustomization monitoring-rules --with-source`.
+        #
+        # /ping on both: unauthenticated, and already the pods' own kubelet
+        # probe path (navidrome/deployment.yaml:80-93,
+        # navidrome/lidarr.yaml:114-127). For lidarr that it is ungated is
+        # proven independently of the gate — lidarr.yaml:29 curls /ping in a
+        # loop BEFORE it has read the API key out of config.xml.
+        #
+        # STATUS CODE, unproven for all three of the new targets: module
+        # http_2xx follows redirects and passes only on a FINAL 2xx, while
+        # every piece of repo evidence behind these paths establishes only
+        # `< 400` — a kubelet httpGet accepts 200-399, and lidarr.yaml:29's
+        # `curl -f` without -L exits 0 on a 302. That evidence proves the
+        # paths are UNGATED, not that they answer 200. Check
+        # probe_http_status_code on the first scrapes; a target that settles
+        # on a 3xx gets REVERTED, not widened to accept 3xx.
+        #
+        # REVERT — an executed recipe, not a claim. Every command below was
+        # actually run against master 2b71dda1 on 2026-09-25; the results are
+        # what is recorded here.
+        #
+        # NO `git revert` works. All four candidates conflict in THIS file at
+        # 2b71dda1, because the comment blocks around these targets were
+        # rewritten after the probes landed (#979):
+        #
+        #   git revert -m 1 0a82d61f   merge of #977, paperless     CONFLICT
+        #   git revert -m 1 16238d39   merge of #976, nav + lidarr  CONFLICT
+        #   git revert cb9e73bd        the paperless commit         CONFLICT
+        #   git revert 2219c928        the nav + lidarr commit      CONFLICT
+        #
+        # Hand-resolving any of them is the trap: the natural --theirs
+        # resolution of the 16238d39 conflict also drops the paperless target
+        # below, leaving NO app-health probes and no signal that it happened.
+        #
+        # Delete the target lines forward instead. Only the target lines are
+        # part of the object; the comments are inert. Both of these were run
+        # and verified:
+        #
+        #   # all three app-health targets — blackbox-http 20 -> 17
+        #   sed -i '/svc\.cluster\.local:80\/ping$/d;/paperless\.paperless\.svc\.cluster\.local:80\/accounts\/login\/$/d' \
+        #     kubernetes/apps/monitoring-rules/probes-blackbox.yaml
+        #
+        #   # the paperless target alone — blackbox-http 20 -> 19
+        #   sed -i '/paperless\.paperless\.svc\.cluster\.local:80\/accounts\/login\/$/d' \
+        #     kubernetes/apps/monitoring-rules/probes-blackbox.yaml
+        #
+        # Verified after each run: the file still parses, blackbox-http drops
+        # to 17 and to 19 respectively, blackbox-edge-wan (3) and
+        # blackbox-tcp-relays (2) are untouched, and `git diff` contains
+        # nothing but the deleted target lines. Then drop whichever comment
+        # block is left orphaned, open a PR, and merge — timing as in the
+        # BLAST RADIUS paragraph above.
+        - http://navidrome.navidrome.svc.cluster.local:80/ping
+        - http://lidarr.navidrome.svc.cluster.local:80/ping
+        # APP-HEALTH probe for paperless. Same rule as the block above
+        # (monitoring-vantage-point-rule); the blind spot it closes is the one
+        # gatus already documents at apps/gatus/configmap.yaml:205-214 —
+        # through the paperless-gate outpost EVERY path, /api/ included,
+        # answers 302 to an unauthenticated caller, so no external probe of
+        # paperless.renner.uno can tell a live app from a live gate. That
+        # external signal is still wanted, and REN-2 adds it from UptimeRobot;
+        # this is the other half of the pair.
+        #
+        # That gatus comment says the in-cluster probe "needs a
+        # CiliumNetworkPolicy first". That sentence is simply WRONG, not
+        # merely inapplicable: paperless-ns.yaml:34-36 already admits the
+        # whole monitoring namespace, which is where blackbox-exporter runs,
+        # and Cilium unions the allow rules of every CNP selecting an
+        # endpoint (the mechanism apps/paperless/network-policy.yaml:22-26
+        # relies on). The EGRESS paragraph in the block above applies here
+        # identically. Nothing in the paperless namespace is edited by this
+        # change.
+        #
+        # The stale sentence is deliberately left in place in
+        # apps/gatus/configmap.yaml: it sits INSIDE the `config.yaml: |`
+        # block scalar, so editing it changes ConfigMap data and
+        # `configmap.reloader.stakater.com/reload` (gatus/deployment.yaml:13)
+        # rolls the gatus Deployment. The correction is written at the top of
+        # that file instead, outside `data:`, where it costs no restart.
+        #
+        # /accounts/login/ because paperless-ngx has no unauthenticated health
+        # endpoint at all. It is the app's own established "the webserver
+        # answers" signal — the same path its startup, liveness and readiness
+        # probes use (paperless.yaml:252-272) — and it renders a Django view
+        # rather than a static file. It does NOT cover the consumer/celery
+        # workers: see process-liveness-vs-protocol-liveness.
+        #
+        # The cluster-local Host header is accepted only because
+        # PAPERLESS_ALLOWED_HOSTS is "*" (paperless.yaml:27). Narrow that and
+        # Django answers 400 here and this probe goes permanently red.
+        #
+        # This is the target most likely to fail the STATUS CODE check in the
+        # block above — a Django login view can answer 302 — so check it
+        # first, and revert rather than widen if it does. Use the REVERT
+        # recipe in that block: `git revert -m 1 0a82d61f` conflicts here too
+        # at 2b71dda1, and the tested one-line deletion for this target alone
+        # is the second command listed there (20 -> 19 targets).
+        - http://paperless.paperless.svc.cluster.local:80/accounts/login/
 # WAN-path probes of the public edge BY PROXY IP. The name-based
 # home.renner.uno probe above resolves in-cluster (CoreDNS/k8s_gateway)
 # and never leaves the LAN — the vantage-point blind spot that let the
@@ -150,3 +313,76 @@ spec:
       static:
         - proxy-1.renner.uno:33080
         - proxy-2.renner.uno:33080
+---
+# NEGATIVE CONTROLS for the paperless ingress scoping (REN-28 PR 2a).
+#
+# These four targets are here TO BE REFUSED. They are not availability
+# monitoring: every one of them is a workload in the `paperless` namespace that
+# blackbox-exporter has no business reaching, and each must read
+# `probe_success == 1` today and `0` once REN-28's PR 2b lands. That flip is the
+# test. Without them, "the whole-namespace admit is gone" is an assertion about
+# a CNP diff rather than a measured property of the cluster, and a later
+# re-widening of the grant would be silent.
+#
+# Why this can only be the ingress policy under test: blackbox-exporter runs in
+# `monitoring`, whose baseline CNP (infrastructure/network-policies/
+# monitoring-ns.yaml) is ingress-only — it declares no `egress:` at all — and
+# the cluster's default-deny.yaml is scoped `namespace: apps`, which covers
+# neither `monitoring` nor `paperless`. So the only thing that can turn these
+# red is paperless's own ingress policy, which is exactly the variable. Nothing
+# else can make the test pass or fail.
+#
+# WHY A SEPARATE jobName, and why these must NEVER be folded into the
+# blackbox-http Probe above: after PR 2b these targets fail permanently and BY
+# DESIGN. BlackboxProbeFailing and BlackboxProbesAbsent
+# (prometheusrule-blackbox.yaml:35,50) are both scoped to job="blackbox-http",
+# so a distinct job is outside both and stays silent — and the hand-maintained
+# instance!~ exclusion regex at :35, whose fragility that file warns about, does
+# not have to grow a fifth entry. Adding these to blackbox-http instead would
+# page the on-call forever.
+#
+# The inverted alert that gives these targets teeth — CNPNegativeControlReachable,
+# `probe_success{job="blackbox-negative-control"} == 1` — ships in PR 2b, NOT
+# here. Between 2a and 2b these controls are SUPPOSED to read 1, so the alert
+# would fire on the correct state. Do not add it before the policy change.
+#
+# A bare TCP connect is the right probe: it completes or it times out, with no
+# application-layer ambiguity about whether a 403 meant the policy or the app.
+# Known, accepted side effect: the pprl-pg-rw connect opens and closes without a
+# startup packet, so Postgres logs "incomplete startup packet" at LOG level once
+# a minute until PR 2b lands and the connect stops arriving.
+apiVersion: monitoring.coreos.com/v1
+kind: Probe
+metadata:
+  name: blackbox-negative-control-paperless
+  namespace: monitoring
+  labels:
+    # Load-bearing, as noted at the top of this file: probeSelector matches on
+    # it. Without it this Probe is never discovered and the control reads
+    # `absent`, not `1` — which would look like a passing negative test.
+    release: kube-prometheus-stack
+spec:
+  jobName: blackbox-negative-control
+  prober:
+    url: blackbox-exporter-prometheus-blackbox-exporter.monitoring.svc.cluster.local:9115
+  module: tcp_connect
+  interval: 60s
+  scrapeTimeout: 10s
+  targets:
+    staticConfig:
+      static:
+        # Proves BOTH halves of the pprl-pg scoping PR 2b will introduce: the
+        # source (blackbox is not Prometheus, and the pprl-pg-allow-prometheus
+        # rule 2b adds admits only Prometheus) and the port (5432 is not the
+        # exporter's 9187). Neither that rule nor paperless-allow-blackbox
+        # exists yet at this commit — that is the point, and why this target
+        # must read 1 now. Service is CNPG-generated from Cluster `pprl-pg`
+        # (paperless/cluster.yaml:4) and is the name paperless itself dials
+        # (paperless.yaml:46).
+        - pprl-pg-rw.paperless.svc.cluster.local:5432
+        # Three witnesses that the whole-namespace `monitoring` admit is gone
+        # for workloads no new rule selects. Ports from the Services themselves:
+        # redis.yaml:76, tika.yaml:57, gotenberg.yaml:63.
+        - redis.paperless.svc.cluster.local:6379
+        - tika.paperless.svc.cluster.local:9998
+        - gotenberg.paperless.svc.cluster.local:3000

kubernetes/apps/monitoring-rules/probes-ren28-negative.yaml

@@ -0,0 +1,98 @@
+# TEMPORARY — negative-test probes for REN-28 §7. DELETE AFTER THE "AFTER"
+# READING IS TAKEN. This file exists to make two of the four §7 negative tests
+# measurable without granting `pods/exec` in `monitoring`, which the owner
+# declined on REN-140 as a wider grant than the change it would verify.
+#
+# WHAT IT IS. blackbox-exporter is already an HTTP/TCP prober running in the
+# `monitoring` namespace. Pointing it at targets that the REN-28 paperless PR
+# is supposed to CLOSE turns "is this path still open?" into a number in
+# Prometheus, readable through the existing read-only REN-36 grant. Every
+# target below answers today and MUST stop answering after the PR lands. A
+# `probe_success` that stays 1 after the merge means the rule does not do what
+# §5 claims.
+#
+# WHY IT DOES NOT PAGE ANYONE. jobName is `blackbox-ren28-negative`, not
+# `blackbox-http`. Every rule in prometheusrule-blackbox.yaml is scoped to
+# job="blackbox-http" (BlackboxProbeFailing) or absent(...{job="blackbox-http"})
+# (BlackboxProbesAbsent), so neither can match these series — before or after.
+# TargetDown (general.rules) cannot match either: the scrape target of a Probe
+# is blackbox-exporter itself, which answers 200 with probe_success 0 on a
+# failed probe, so `up` stays 1 regardless of the result. Verified against the
+# live rule set on 2026-09-26. This is deliberate: a monitor that pages for an
+# expected failure is worse than no monitor.
+#
+# NO POLICY CHANGE IS NEEDED TO MAKE THESE WORK TODAY. Ingress: the paperless
+# baseline still admits the whole `monitoring` namespace on every port
+# (infrastructure/network-policies/paperless-ns.yaml:34-36) — that admission is
+# exactly what REN-28 removes. Egress: the `monitoring` namespace is in no
+# egress enforcement at all; its only two Cilium policies, monitoring-ns.yaml
+# and monitoring-alloy-syslog.yaml, are ingress-only. Same dependency the three
+# cross-namespace targets in probes-blackbox.yaml already rest on.
+#
+# TWO POD-IP TARGETS. pprl-pg's :9187 metrics port is on the POD, not on any
+# Service — that is why it is scraped by a PodMonitor and not a ServiceMonitor
+# (cnpg-podmonitors.yaml). There is no stable name for it, so the literal pod
+# IPs are used. They were read live on 2026-09-26; if either pod is rescheduled
+# the probe goes red for the wrong reason. BEFORE TRUSTING AN "AFTER" READING,
+# re-confirm the IP still belongs to the same pod:
+#
+#   kube_pod_info{namespace="paperless", pod=~"pprl-pg-.*"}   -> pod_ip
+#
+# The pprl-pg-r:5432 target below is the stable-name version of the same
+# statement and survives a reschedule; it is on the database port rather than
+# the metrics port, so the two are kept together on purpose.
+#
+# REVERT. Delete this file and its line in kustomization.yaml. Nothing else
+# references it; no workload is touched; Prometheus drops the job on the next
+# operator reconcile.
+apiVersion: monitoring.coreos.com/v1
+kind: Probe
+metadata:
+  name: blackbox-ren28-negative
+  namespace: monitoring
+  labels:
+    # probeSelector is NOT loosened on this cluster (helmrelease-prometheus.yaml
+    # keeps ruleSelector/probeSelector/scrapeConfigSelector strict), so this
+    # label is load-bearing, not decorative. Without it the Probe is created and
+    # never discovered — and a test that silently does not run reads exactly
+    # like a test that passed.
+    release: kube-prometheus-stack
+spec:
+  jobName: blackbox-ren28-negative
+  prober:
+    url: blackbox-exporter-prometheus-blackbox-exporter.monitoring.svc.cluster.local:9115
+  module: tcp_connect
+  interval: 60s
+  # 10s matches blackbox-tcp-relays. It is also the discriminator: a Cilium
+  # deny DROPS the SYN, so a refused path completes in ~1ms and a denied path
+  # burns the full timeout. probe_duration_seconds distinguishes "nothing is
+  # listening" from "policy stopped it" — probe_success alone does not.
+  scrapeTimeout: 10s
+  targets:
+    staticConfig:
+      static:
+        # REN-28 §7 test 4 — "from a monitoring pod that is not Prometheus, a
+        # connection to pprl-pg :9187 must time out". blackbox-exporter is
+        # precisely that pod. Both CNPG instances, because the PR's
+        # pprl-pg-allow-prometheus rule selects on a label both carry: if the
+        # selector were wrong in a way that left one instance open, one target
+        # would tell us and the other would not.
+        - 10.244.4.166:9187 # pprl-pg-2, node k8s-2
+        - 10.244.2.220:9187 # pprl-pg-3, node k8s-3
+        # Same statement on a stable name and the port that actually matters:
+        # after the PR the only monitoring→pprl-pg allow is Prometheus on
+        # :9187, so a non-Prometheus monitoring pod must not reach Postgres.
+        - pprl-pg-r.paperless.svc.cluster.local:5432
+        # REN-28 §7 test 2, workload dimension — "from blackbox-exporter, a
+        # paperless target other than the one it is scoped to must time out".
+        # The test as written in §7 says "a paperless POD port other than
+        # 8000"; there is no such port. The paperless pod's only other listener
+        # is the bridge-tls-proxy socat, bound to 127.0.0.1:11143
+        # (apps/paperless/paperless.yaml:289), so every other port on that pod
+        # is closed today and answers RST — there is no "reachable before" to
+        # record and the port half of paperless-allow-blackbox (:8000) cannot
+        # be tested at all. gotenberg is the half that CAN be: it is one of the
+        # other eight workloads the endpointSelector:{} baseline currently
+        # exposes to all of `monitoring`, and the PR's per-workload selector is
+        # what closes it.
+        - gotenberg.paperless.svc.cluster.local:3000

kubernetes/apps/monitoring-rules/prometheusrule-ceph.yaml

@@ -52,19 +52,61 @@ spec:
         - alert: CephPoolLostRedundancy
           # Names the pool, which POOL_NO_REDUNDANCY does not.
           #
-          # There is no pool-size metric, but bytes_used/stored IS the
-          # replication factor. The >1e6 floor drops pools too small to measure
-          # meaningfully, empty ones included.
+          # There is no pool-size metric, but for a REPLICATED pool
+          # bytes_used/stored IS the replication factor. The >1e6 floor drops
+          # pools too small to measure meaningfully, empty ones included.
           #
-          # No pool is exempt.
+          # ERASURE-CODED POOLS ARE EXCLUDED, AND MUST STAY EXCLUDED (REN-88).
+          #
+          # That ratio is not a replication factor for an EC pool, it is the
+          # coding overhead: k+m shards over k data shards. For the two ec:2+1
+          # pools here that is 3/2 = exactly 1.5 — sitting precisely on this
+          # rule's own threshold, on the wrong side of it half the time.
+          # Measured on `cephfs-bulk` (pool 15): used 480042086400 / stored
+          # 320028049408 = 1.5000000384, a margin of 12 KiB in 480 GB, so which
+          # way the comparison lands is decided by the last bits of a float
+          # division. It paged critical for 199 minutes on 2026-09-25 (12:37 to
+          # 15:56 UTC, two emails) with the cluster HEALTH_OK, zero degraded
+          # PGs and every OSD up, then cleared on its own. 50 of the 210
+          # samples in the preceding 30h were below 1.5; only that run held the
+          # 30m `for`. `migration.rgw.buckets.data` (pool 12) is the same ec:2+1
+          # shape and was one drift away from doing the same thing.
+          #
+          # The premise fails for EC too, not just the arithmetic: the old
+          # description claimed a single OSD loss destroys data, but 2+1 carries
+          # a parity shard and survives one OSD loss, same as replica-2.
+          #
+          # Do NOT "fix" this by nudging the threshold to 1.4 or 1.49. That
+          # keeps a knife-edge, just a quieter one, and it also narrows the real
+          # replica-2 -> replica-1 signal this rule exists to catch (1.0).
+          #
+          # EC pools keep full redundancy coverage without this rule: a real
+          # loss raises POOL_NO_REDUNDANCY, which is a live series in
+          # `ceph_health_detail` and is deliberately in scope for
+          # CephHealthWarn above. That check reads Ceph's own verdict instead of
+          # inferring one from a byte ratio, which is the right instrument here
+          # — an EC pool that loses a shard does not change its stored/used
+          # ratio at all, because Ceph does not delete the surviving shards.
+          # So there is no gap to reopen, and no second EC-specific threshold
+          # worth inventing.
+          #
+          # `on(pool_id)` is deliberately the whole matcher. ceph_pool_metadata
+          # carries labels ceph_pool_bytes_used does not (name, type,
+          # description, application, compression_mode), so default all-label
+          # matching would never join. Adding `cluster` would be defensive
+          # against a second Ceph cluster that does not exist, at the cost of
+          # the rule silently matching nothing if that label is ever dropped —
+          # an alert that fails open invisibly is the worse trade.
           expr: |
-            ceph_pool_bytes_used / ceph_pool_stored < 1.5 and ceph_pool_stored > 1e6
+            ceph_pool_bytes_used / ceph_pool_stored < 1.5
+              and on(pool_id) ceph_pool_metadata{type="replicated"} == 1
+              and ceph_pool_stored > 1e6
           for: 30m
           labels:
             severity: critical
           annotations:
-            summary: "Ceph pool {{ $labels.pool_id }} is running unreplicated"
-            description: "Pool {{ $labels.pool_id }} has a replication factor below 1.5, meaning a single OSD loss destroys data. No pool is meant to be unreplicated — an unreplicated pool also blocks ALL OSD maintenance cluster-wide, since `ceph osd ok-to-stop` refuses while its PGs would go offline. Check with `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd pool ls detail`."
+            summary: "Ceph replicated pool {{ $labels.pool_id }} is running unreplicated"
+            description: "Replicated pool {{ $labels.pool_id }} has a replication factor below 1.5, meaning it is down to a single copy and one OSD loss destroys data. No replicated pool is meant to be unreplicated — it also blocks OSD maintenance for as long as it lasts, since `ceph osd ok-to-stop` refuses while its PGs would go offline. Erasure-coded pools are out of scope for this rule and are covered by POOL_NO_REDUNDANCY via CephHealthWarn (REN-88). Check with `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd pool ls detail`."
 
         - alert: CephHealthError
           # HEALTH_ERR means data availability or redundancy is at risk.

kubernetes/apps/monitoring-rules/prometheusrule-ci-openbao.yaml

@@ -34,9 +34,20 @@ spec:
           # query parameter, and an exact match would break silently if the
           # instance label ever encodes it differently.
           expr: probe_success{job="blackbox-http", instance=~"https://openbao\\.renner\\.uno/v1/sys/health.*"} == 0
-          # Two consecutive failed probes: enough to skip a single lost probe
-          # through Traefik, not enough to sit on a sealed store.
-          for: 1m
+          # Five minutes of continuously failing probes — about ten consecutive
+          # samples, since the probe and the rule evaluation both run at 30s
+          # (probes-blackbox.yaml, helmrelease-prometheus.yaml). Matches
+          # WoodpeckerDown above.
+          #
+          # This was 1m, the tightest probe window in the cluster by 5x, and it
+          # went pending on a single 5s HTTPS timeout with OpenBao healthy
+          # (REN-131). Isolated blips are ambient on this probe fleet at roughly
+          # one per two hours, so a window that clears on three samples pages on
+          # jitter. The widening costs nothing real: per the description below
+          # nothing breaks the moment OpenBao goes away, and OpenBaoQuorumLost
+          # (prometheusrule-openbao.yaml, for: 2m) still covers the fast
+          # failure, which is quorum loss rather than a lost probe.
+          for: 5m
           labels:
             severity: critical
           annotations:

kubernetes/apps/monitoring-rules/prometheusrule-cnpg-backup.yaml

@@ -13,7 +13,7 @@ spec:
       rules:
         # The Postgres clusters back up to Hetzner. HetznerBackupEndpointDown
         # probes that S3 endpoint, which proves the bucket is reachable and
-        # nothing more — these two rules check that something is actually being
+        # nothing more — these rules check that something is actually being
         # written to it.
 
         - alert: CNPGWalArchiveBacklog
@@ -23,22 +23,106 @@ spec:
           # that moment, and pg_wal keeps growing until it fills the PVC and
           # takes Postgres down.
           #
-          # This is the load-bearing alert of the two. It reads the
+          # This is the load-bearing alert of the group. It reads the
           # instance-level collector directly, so unlike the backup-age metric
           # discussed below it does not depend on Cluster status being
           # populated.
           #
-          # Threshold: >5 segments sustained for 30m. Brief non-zero values are
+          # Threshold: >5 segments sustained for 15m. Brief non-zero values are
           # normal under write bursts — a segment is 16 MiB, so 5 is roughly
           # 80 MiB of un-archived WAL, well below anything that threatens the
           # volume but far above idle noise.
-          expr: cnpg_collector_pg_wal_archive_status{value="ready"} > 5
-          for: 30m
+          #
+          # `for` was 30m and is now 15m (REN-129). To be clear about what that
+          # does and does not buy: this rule does NOT fire on the nightly write
+          # burst documented under the Surge rule below, at 30m or at 15m, and
+          # that is correct — across seven consecutive measured nights the
+          # longest contiguous excursion above 5 was 10 minutes and every one
+          # of them drained itself. What 15m is for is the case that actually
+          # costs something: an archiver that has stopped. Outside those
+          # nightly windows the metric is a hard 0 on all 12 CNPG instances in
+          # this cluster, so there was never a noise floor that needed 30m to
+          # wait out — the extra 15m only delayed that page.
+          #
+          # `max by (namespace, pod)` is not cosmetic. The raw series carries
+          # `role`, which flips on failover; a changed label is a new series,
+          # and a new series restarts the `for` timer. Aggregating `role` away
+          # means a backlog that persists across a failover still pages 15m
+          # after the backlog started rather than 15m after the failover.
+          expr: max by (namespace, pod) (cnpg_collector_pg_wal_archive_status{value="ready"}) > 5
+          for: 15m
           labels:
             severity: warning
           annotations:
             summary: "CNPG WAL archiving is falling behind on {{ $labels.namespace }}/{{ $labels.pod }}"
-            description: "{{ $value }} WAL segments have been waiting to archive for over 30 minutes. Point-in-time recovery is broken for anything after the last archived segment, and pg_wal will grow until the PVC fills. Check the barman-cloud sidecar logs and the ObjectStore credentials: `kubectl -n {{ $labels.namespace }} logs {{ $labels.pod }} -c plugin-barman-cloud`."
+            description: "{{ $value }} WAL segments have been waiting to archive for over 15 minutes. Point-in-time recovery is broken for anything after the last archived segment, and pg_wal will grow until the PVC fills. Check the barman-cloud sidecar logs and the ObjectStore credentials: `kubectl -n {{ $labels.namespace }} logs {{ $labels.pod }} -c plugin-barman-cloud`."
+
+        - alert: CNPGWalArchiveBacklogSurge
+          # Companion to CNPGWalArchiveBacklog above, for the opposite failure
+          # shape: not a stuck archiver, but a working archiver that cannot
+          # drain as fast as Postgres is producing.
+          #
+          # CNPGWalArchiveBacklog cannot see that shape at any `for` value that
+          # is also safe against a flapping archiver, because the event is over
+          # before the window closes.
+          #
+          # What is normal here, measured on ha-pg-1 at 1-minute steps over the
+          # seven nights 2026-09-19 to 2026-09-26 (REN-129). Every night is the
+          # same burst, and nothing in that week exceeds 5 at any other hour:
+          #
+          #   flat 0 until 02:12Z; fill from 02:13Z; peak 02:14-02:15Z;
+          #   drained by 02:22Z. Nightly peaks 56, 91, 100, 101, 106, 113, 119.
+          #
+          # This is a write burst, not IO contention. Drain was 13.0-14.9
+          # segments/min on all seven nights — including the night whose Kopia
+          # repository-maintenance batch did not start until after the burst had
+          # already drained, and the night when two such jobs were saturating
+          # the RBD devices. That batch drifts across 02:03-02:24Z while the
+          # peak stays pinned to 02:15Z, and a drifting cause cannot produce an
+          # effect fixed to the clock. What varies is the fill (18.7-39.7
+          # segments/min), so the height of the backlog is set by how much WAL
+          # is produced against an archiver with a flat ~14 segments/min
+          # ceiling. 02:12Z is 04:12 local, which is when Home Assistant's
+          # recorder purge runs.
+          #
+          # So the nightly burst is a benign, self-clearing fact of this
+          # cluster and this rule has to stay silent through it. Threshold:
+          # >64 segments (1 GiB of un-archived WAL) for 8m.
+          #
+          #   - 64, not something nearer the >5 above, because this rule is
+          #     deliberately not a faster copy of that one. 1 GiB is where the
+          #     backlog becomes a material fraction of a CNPG PVC's free space
+          #     (ha-pg's volume is 20Gi, holding steady at 6.4-7.7 GiB free),
+          #     and a rule that trips on every write spike is how the
+          #     load-bearing rule above gets ignored.
+          #   - 8m because the measured envelope spends at most 5 consecutive
+          #     minutes above 64 on any of the seven nights, so 8m clears the
+          #     worst observed night by 3 minutes. An earlier revision of this
+          #     rule used 2m; that fires at ~02:16Z on 6 of the 7 nights, which
+          #     is two owner emails a night at 04:16 and 04:22 local for a
+          #     condition that repairs itself.
+          #
+          # What 8m still catches: 1 GiB queued that is not moving. From 119,
+          # the highest peak in the week, a healthy archiver is back under 64
+          # inside 5 minutes. Sitting above it for 8 means the archiver has
+          # lost most of its throughput, and that is worth a 04:00 look.
+          #
+          # When it does fire, the fix is drain rate or headroom, never a
+          # higher threshold here. ObjectStore `wal.maxParallel`
+          # (kubernetes/apps/home-assistant/ha-pg-objectstore.yaml) is unset,
+          # which the barman-cloud plugin reads as one WAL segment at a time;
+          # that is where the ~14 segments/min ceiling comes from. Headroom is
+          # `spec.storage.size` in ha-pg-cluster.yaml. Both are REN-135.
+          #
+          # `max by (namespace, pod)` for the same reason as the rule above:
+          # `role` flips on failover and would restart the `for` timer.
+          expr: max by (namespace, pod) (cnpg_collector_pg_wal_archive_status{value="ready"}) > 64
+          for: 8m
+          labels:
+            severity: warning
+          annotations:
+            summary: "CNPG WAL archiving cannot keep up on {{ $labels.namespace }}/{{ $labels.pod }}"
+            description: "{{ $value }} WAL segments (>1 GiB) have been queued for archiving for 8 minutes, which is longer than a healthy archiver needs to clear that much — from a backlog of 119 it takes under 5. Point-in-time recovery is degraded for everything after the last archived segment, and pg_wal grows against the PVC's free space until it clears. Start with the archiver: `kubectl -n {{ $labels.namespace }} logs {{ $labels.pod }} -c plugin-barman-cloud`, and check whether its ObjectStore sets `wal.maxParallel` — unset means one segment at a time, a ceiling of about 14 segments/min (REN-129)."
 
         - alert: CNPGBackupFailed
           # cnpg_collector_last_failed_backup_timestamp is a unix timestamp, 0

kubernetes/apps/monitoring-rules/prometheusrule-flux.yaml

@@ -0,0 +1,274 @@
+# Per-object Flux readiness, and the groundwork for the Ready-but-frozen
+# ImagePolicy signal (REN-102, closing REN-97).
+#
+# EVERYTHING HERE READS `gotk_resource_info`, WHICH IS A kube-state-metrics
+# SERIES, NOT A FLUX CONTROLLER SERIES. apps/monitoring-rules/flux-podmonitor.yaml
+# scrapes the controllers and yields reconcile durations and controller-runtime
+# counters, but readiness is not on that endpoint at the pinned controller
+# versions: `gotk_reconcile_condition` and `gotk_suspend_status` are declared by
+# fluxcd/pkg/runtime and populated by nothing. The series these rules read are
+# created by the `customResourceState` block in
+# apps/monitoring/helmrelease-prometheus.yaml. If that block is removed, every
+# rule below goes blind — which is what FluxImagePolicyMetricsMissing exists to
+# say out loud.
+#
+# THE STALENESS HALF DELIBERATELY SHIPS WITHOUT AN ALERT. See the second group.
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: flux
+  namespace: monitoring
+  labels:
+    # ruleSelector matches on this label; without it the rule is created but
+    # never loaded into Prometheus.
+    release: kube-prometheus-stack
+spec:
+  groups:
+    - name: flux.rules
+      rules:
+        # Why `max by (...)` and not the bare selector: it collapses the KSM
+        # target labels (`pod`, `instance`, `endpoint`, ...). Without it, a
+        # kube-state-metrics rollout changes `pod`, which resolves the alert on
+        # the old series and re-fires it on the new one — a fresh mail for an
+        # unchanged failure. KSM rolled 11 times in the last 30 days, so this is
+        # routine, not hypothetical.
+        #
+        # Why `ready=~"False|Unknown"` and not `ready="False"`: Flux sets Ready
+        # False on failure but Unknown while a reconcile is in flight, and an
+        # object wedged in Unknown is stuck just as hard as one that is False —
+        # this repo has already had a Progressing CephCluster deadlock the whole
+        # apps layer (see rook-servicemonitors.yaml). Matching the two values
+        # explicitly, rather than `ready!="True"`, keeps an object whose `ready`
+        # label is absent entirely (freshly created, condition not yet written)
+        # out of the alert.
+        #
+        # Why `suspended!="true"` and not an `unless` join: `suspended` is a
+        # label on the same Info series, so one matcher does what the withdrawn
+        # draft needed a second vector for. The `!=` form also covers kinds where
+        # the label is absent. There is no deleted-object guard because KSM stops
+        # emitting the series when the object goes away — under `gotk_*` that
+        # guard had to be written, here it is structural.
+        #
+        # for: 2h — 31 of the 34 ImageRepositories scan at `interval: 1h`
+        # (2 at 30m: shelly-manager, vsftpd; 1 at 10m: wiki), so 91% of the
+        # population cannot recover from a transient registry failure faster
+        # than one hour whatever this value says. Anything under 1h mails about
+        # a single self-healing missed scan. 2h means two consecutive failed
+        # scans. Detection latency added by going through KSM is one 30s scrape.
+        - alert: FluxImageAutomationNotReady
+          expr: >-
+            max by (customresource_kind, name, exported_namespace) (
+              gotk_resource_info{
+                customresource_kind=~"ImageRepository|ImagePolicy|ImageUpdateAutomation",
+                ready=~"False|Unknown",
+                suspended!="true"
+              }
+            )
+          for: 2h
+          labels:
+            severity: warning
+          annotations:
+            summary: "Flux {{ $labels.customresource_kind }}/{{ $labels.name }} has not been Ready for 2h"
+            description: >-
+              An image-automation object has been not-Ready for two reconcile
+              intervals. Nothing is broken in the running cluster, but the image
+              it governs has stopped moving, security rebuilds included.
+              `flux get image all -n {{ $labels.exported_namespace }}` gives the
+              reason; the usual causes are registry credentials, a rate limit, or
+              a tag format the policy's `pattern`/`extract` no longer matches.
+              flux-notifications/alerts.yaml already mailed once at the moment of
+              transition — this is the "still broken two hours later" signal that
+              edge-triggered mail cannot give.
+
+        # The exact negation of the matcher above, so a kind added to
+        # helmrelease-prometheus.yaml's customResourceState config is covered by
+        # one of these two rules without anyone remembering to widen a list.
+        #
+        # for: 1h, derived from the intervals actually in this repo: all 33
+        # Kustomizations are at 10m (six retries inside the hour), all 25
+        # HelmReleases at 30m (two), and the flux-system GitRepository at 1m0s
+        # (sixty). The dominant non-failure source of ready="False" is a
+        # `dependsOn` wait, whose `reason` is not a label and so cannot be
+        # filtered out; an hour clears a whole-cluster reconcile chain.
+        #
+        # KNOWN TRADE, stated rather than hidden: this matcher also picks up the
+        # 17 HelmRepository and 4 OCIRepository objects, and every one of those is
+        # at `interval: 24h`. A failed index or manifest fetch there cannot
+        # self-heal inside an hour, so a transient failure yields up to two mails
+        # (repeat_interval is 12h) before the next scheduled reconcile. That is
+        # accepted as correct — a chart source stale for an hour is worth knowing
+        # — and it is real information rather than a blip. If it turns out to be
+        # the noisy one, the fix is a third rule for the 24h-interval kinds, not
+        # a global bump of this value, which would make Kustomization alerts
+        # useless.
+        - alert: FluxResourceNotReady
+          expr: >-
+            max by (customresource_kind, name, exported_namespace) (
+              gotk_resource_info{
+                customresource_kind!~"ImageRepository|ImagePolicy|ImageUpdateAutomation",
+                ready=~"False|Unknown",
+                suspended!="true"
+              }
+            )
+          for: 1h
+          labels:
+            severity: critical
+          annotations:
+            summary: "Flux {{ $labels.customresource_kind }}/{{ $labels.name }} has not been Ready for 1h"
+            description: >-
+              A Flux object outside image automation has been not-Ready for an
+              hour, so the desired state in git is not reaching the cluster.
+              `flux get all -A --status-selector ready=false` lists them with a
+              reason. For a Kustomization or HelmRelease this means a rollout is
+              stuck; for a HelmRepository or OCIRepository it means the chart or
+              artifact index is stale and dependent releases will not upgrade.
+
+        # THE DEAD MAN'S SWITCH, and the reason it is safe here and was not in
+        # #992. This asserts that the 34 ImagePolicy series exist — and they
+        # exist only because this change created them. Merged against `gotk_*`
+        # alone, as the withdrawn draft had it, this rule would have been
+        # permanently true and mailed admin@renner.uno every 12h forever.
+        #
+        # Its value is catching the removal of the customResourceState block, or
+        # a malformed edit to it, which would otherwise take all three rules
+        # above silently blind. `KubeStateMetricsDown` and `TargetDown` overlap it
+        # when the whole pod is down; neither notices a config that parses but
+        # emits nothing.
+        #
+        # for: 30m rides out the kube-state-metrics pod roll that applying this
+        # change itself causes, and any later roll.
+        - alert: FluxImagePolicyMetricsMissing
+          expr: absent(gotk_resource_info{customresource_kind="ImagePolicy"})
+          for: 30m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Flux per-object readiness metrics have stopped being exported"
+            description: >-
+              kube-state-metrics is exporting no ImagePolicy series, so
+              FluxImageAutomationNotReady and FluxResourceNotReady cannot fire and
+              Flux readiness is unmonitored. Check the `customResourceState` block
+              in apps/monitoring/helmrelease-prometheus.yaml, then the
+              kube-state-metrics pod log — a custom-resource-state config that
+              fails to parse is logged there and yields no series.
+
+    # The Ready-but-frozen ImagePolicy signal: a policy whose `filterTags` or
+    # semver range excludes everything new keeps resolving the tag it resolved
+    # months ago, holds Ready=True the whole time, and no readiness rule can see
+    # it. `tika.yaml`'s own comment names the case.
+    #
+    # THIS GROUP DELIBERATELY CONTAINS NO ALERT, AND THAT IS A MEASURED DECISION,
+    # NOT AN OMISSION. Every candidate threshold was tested against the real
+    # release history and the real registries on 2026-09-26:
+    #
+    #   * A flat "frozen for N days" duration does not exist. Over the 191
+    #     automated bump commits in the 147 days to 2026-09-25, gatus went 127
+    #     days between bumps and paperless-ngx 126, both provably healthy — they
+    #     bumped. Six policies have never bumped at all. Any N below 127 fires on
+    #     the healthy pair; any N above 147 is longer than the whole observation
+    #     window.
+    #
+    #   * The "tag count grew >= 10%" comparator from this card's first design is
+    #     wrong, and the registries say so. Of the 18 policies frozen 14 days or
+    #     more, a 10% threshold fires on exactly three — gotenberg (+87 tags/14d
+    #     on 526), valkey (+68 on 301), eclipse-mosquitto (+13 on 90) — and all
+    #     three churn variant tags their filters reject on purpose. It does NOT
+    #     fire on tika (+4 on 136 = 3.0%), which is the canonical case this card
+    #     was opened for.
+    #
+    #   * Relaxing to "grew by >= 1 tag" does catch tika, but then also fires on
+    #     those same three plus velero-plugin-for-aws, whose `1.13.x` pin is
+    #     deliberate. Alertmanager routes every severity to the same email
+    #     receiver with repeat_interval: 12h, so that is four standing mails
+    #     every twelve hours, indefinitely, for four non-faults.
+    #
+    #   * `.status.lastScanResult.latestTags` looks like the way out and is not:
+    #     the CRD defines it as "the first 10 tags when sorting all the tags in
+    #     descending ALPHABETICAL order", which is not newest-first for any tag
+    #     scheme in this repo.
+    #
+    # So the recorded series below materialise the candidate set and, more to the
+    # point, START THE HISTORY that a threshold has to be derived from — a real
+    # time series of tagCount per repository, which has never existed here. The
+    # alert is deferred to a follow-up card that reads 14+ days of it. Recording
+    # rules cannot mail anyone, so this costs nothing while it warms up.
+    #
+    # interval: 15m is LOAD-BEARING and is declared here rather than inherited.
+    # Global evaluationInterval is 30s and no other rule group in this directory
+    # sets one (45 of the 46 loaded groups run at 30s). The `>= 1250` warm-up
+    # guard below counts samples, so at 30s it would be satisfied in 10.4 hours
+    # instead of 13 days and the frozen series would read true for policies that
+    # had not been observed frozen at all.
+    - name: flux-image-policy-staleness.rules
+      interval: 15m
+      rules:
+        # Stage one exists only to strip the KSM target labels. Counting distinct
+        # tags means counting distinct series, and `gotk_resource_info` carries
+        # `pod` and `instance` — there were 11 distinct kube-state-metrics pod
+        # identities in the last 30 days, so counting the raw series would report
+        # "several tags seen" after every roll, including the roll this change
+        # itself causes, and would keep reporting it for the whole window.
+        - record: flux:image_policy_tag:info
+          expr: >-
+            max by (name, exported_namespace, source_name, tag) (
+              gotk_resource_info{customresource_kind="ImagePolicy"}
+            )
+
+        # Distinct resolved tags seen in a trailing day. A tag change retires one
+        # series and starts another, so `count by (...)` over `count_over_time`
+        # is a count of tag values, and `== 1` means the tag did not move.
+        - record: flux:image_policy_tags_seen:1d
+          expr: >-
+            count by (name, exported_namespace, source_name) (
+              count_over_time(flux:image_policy_tag:info[1d])
+            )
+
+        # The repository side, keyed by `source_name` so it joins on the actual
+        # `spec.imageRepositoryRef.name` reference rather than on the convention
+        # that a policy is named after its repository. All 34 pairs match by name
+        # today, so this is drift insurance, not a bug fix.
+        - record: flux:image_repository_tag_count:1d
+          expr: >-
+            max by (source_name, exported_namespace) (
+              label_replace(
+                max_over_time(gotk_image_repository_tag_count[1d]),
+                "source_name", "$1", "name", "(.+)"
+              )
+            )
+
+        # The candidate set: resolved tag unmoved for 14 days while the
+        # repository's tag count is genuinely higher than it was.
+        #
+        # 14d, not the 30d this card was first designed around, because 30d does
+        # not fit. `retentionSize: 30GB` binds long before `retention: 90d`:
+        # (time() - prometheus_tsdb_lowest_timestamp/1000)/86400 measured 26.63
+        # days on 2026-09-26, at 84.5% of the size cap and growing ~1.09 GB/day
+        # (REN-107). A 30d window can never be satisfied. 14d uses 53% of that
+        # floor.
+        #
+        # >= 1250 is the warm-up guard and its arithmetic is: at interval 15m a
+        # full 14d window holds 14*24*4 = 1344 samples, so 1250 is 93% of a full
+        # window = 13.02 days of history, leaving ~23h of slack for Prometheus
+        # restarts and KSM rolls. Without it the series would read true on day
+        # one, when "unmoved for 14 days" is simply not yet knowable. 13.02 days
+        # against a 26.63-day floor is 2.0x margin; re-check it with the
+        # lowest_timestamp query above if retention is ever cut.
+        #
+        # `last_over_time - min_over_time >= 1` and not a max/min ratio: a ratio
+        # is direction-agnostic and is satisfied by a tag count that DROPPED
+        # (registry pruning, an edited exclusionList, a partial scan) exactly as
+        # by one that grew. Comparing the current value against the window
+        # minimum only counts net growth.
+        - record: flux:image_policy_tag_frozen:14d
+          expr: >-
+            (
+              max_over_time(flux:image_policy_tags_seen:1d[14d]) == 1
+              and
+              count_over_time(flux:image_policy_tags_seen:1d[14d]) >= 1250
+            )
+            and on (source_name, exported_namespace) (
+              (
+                last_over_time(flux:image_repository_tag_count:1d[1h])
+                - min_over_time(flux:image_repository_tag_count:1d[14d])
+              ) >= 1
+            )

kubernetes/apps/monitoring-rules/tests/README.md

@@ -0,0 +1,125 @@
+# promtool unit suites for the PrometheusRules in this directory
+
+`promtool test rules` suites, run on every pull request that touches
+`kubernetes/**` by the `promtool` step in
+[`.woodpecker/ci-manifests-test.yml`](../../../../.woodpecker/ci-manifests-test.yml)
+via [`scripts/ci/promtool-rules.sh`](../../../../scripts/ci/promtool-rules.sh).
+
+## Why this exists
+
+Before this, CI proved that every `PrometheusRule` in the tree assembled into
+valid YAML and nothing else. It did not check that an expression parses, that a
+label join matches anything, or that a `for` window behaves the way its comment
+claims.
+
+That was not theoretical. On [REN-129](/REN/issues/REN-129) the surge rule in
+`prometheusrule-cnpg-backup.yaml` was written `for: 2m` and reviewed at that
+value. Against Iris's seven measured nights it fires on six of them — two owner
+emails a night, 04:16 and 04:22 local, for a WAL backlog that clears itself in
+under ten minutes. It was caught by a human writing the arithmetic out by hand,
+twice, in two separate runs, and throwing the work away both times because
+there was nowhere in the repo to put it. That rule is now
+`mutations.txt:surge-for-2m`.
+
+## Layout
+
+| File | What it is |
+| --- | --- |
+| `cnpg-backup_test.yaml` | The suite. **Generated** — see below. |
+| `generate-cnpg-backup-test.py` | Regenerates it from the REN-129 measurement table. |
+| `mutations.txt` | Negative controls. Each must make the suite fail. |
+
+## Adding a suite
+
+Name it `<something>_test.yaml` in this directory. Discovery is by that suffix,
+and `promtool-rules.sh` fails if it finds none, so a suite in the wrong place
+is a red pipeline rather than a silent gap.
+
+**`rule_files` entries are relative to this directory, not the repo root.**
+promtool joins `dirname(suite)` onto each entry, so a manifest in the parent
+directory is `../name.yaml`. Getting this wrong is not a test failure: promtool
+prints `WARNING: no file match pattern …`, loads **zero rules**, and every
+`exp_alerts: []` assertion then passes against an empty rule set. The first
+draft of the suite here shipped exactly that way — ninety silence assertions,
+green, testing nothing. `promtool-rules.sh` resolves every entry itself before
+running promtool and fails on a miss, because promtool has no flag that makes
+it fatal.
+
+`promtool` cannot read a `PrometheusRule` directly: it wants a bare `groups:`
+document, and a `PrometheusRule` carries that under `spec:`. The script strips
+`spec:` into a temporary mirrored tree with the same relative layout and runs
+there, so `../name.yaml` resolves in the mirror exactly as it reads here.
+Nothing is written back into the checkout.
+
+## Regenerating `cnpg-backup_test.yaml`
+
+```sh
+python3 kubernetes/apps/monitoring-rules/tests/generate-cnpg-backup-test.py \
+  > kubernetes/apps/monitoring-rules/tests/cnpg-backup_test.yaml
+```
+
+and commit the result. `--report` prints the reconstruction self-check table
+instead of the suite.
+
+The generator exists so the numbers in the suite are auditable: the seven
+measured nights are rebuilt from the `(peak, peak minute, fill, drain)` tuples
+Iris published on REN-129, not hand-typed, so a reader can check them against
+the measurement rather than taking them on trust. It is not a hermetic
+artifact, and **CI does not regenerate and diff it** — the promtool image has
+no Python, and byte-equality with a generator is not what stops a suite from
+being weakened. The mutation phase is.
+
+Two things the generator refuses to emit, both because they happened:
+
+- a window that does not cover the bursts. An earlier revision had `T0` at
+  `8 * 60 + 8` (08:08Z) instead of `2 * 60 + 8`, seven hours after every burst,
+  so all seven nights rebuilt as forty zeroes and their silence assertions
+  passed without touching a threshold. `assert_window_covers_bursts()` now
+  fails instead.
+- a night that clears the threshold *sooner* than the measured one, which would
+  be a weaker silence test than the night it claims to stand for. Longer is
+  fine and expected — a linear decay at the published mean drain rate holds the
+  tail above 5 a minute or two longer than the real trace on four of the seven
+  nights, which makes the suite harder to pass, not easier.
+
+## Negative controls
+
+A suite that passes proves the rules match the assertions. It does not prove
+the assertions reach the rules — both failure modes above were green. So
+`promtool-rules.sh` runs a second phase: for each entry in `mutations.txt` it
+breaks one value the rules depend on and requires the suites to **fail**. A
+mutation nothing notices is reported as a hole in the suite.
+
+This is also the answer to "why is that case in there". `surge-for-5m` is
+silent on all seven real nights and fires only on the adversarial envelope
+case, so deleting the envelope case makes `for: 8m` reducible to `5m` with the
+suite still green. The control is what makes the case load-bearing.
+
+Mutations are `name|manifest-path|sed-program`, applied to the stripped copy in
+the mirror (two spaces shallower than the manifest on disk — anchor on
+`^[[:space:]]*`). A program that matches nothing is a failure, not a skip:
+otherwise renaming a rule quietly turns its control into a no-op.
+
+## Reading `for` margins
+
+Margin is **pending** time, which is one minute less than the run length. An
+alert goes pending at the first evaluation where the expression is true and
+fires when `now - activeAt >= for`, so N consecutive one-minute samples above a
+threshold accumulate N-1 minutes. A trace with 5 samples above 64 does not fire
+`for: 5m`; it fires `for: 4m`. The `>5 pend` and `>64 pend` columns in
+`--report` are the numbers to compare a `for` against — not the sample counts
+next to them.
+
+## Running it locally
+
+```sh
+# promtool must be 3.x: the cluster runs kube-prometheus-stack >= 72.0.0, and
+# alert `for` evaluation is version-sensitive, so a 2.x promtool would be
+# checking a different evaluator than production. The script enforces this.
+sh scripts/ci/promtool-rules.sh
+```
+
+The script's own guards — the discovery floors, the `rule_files` resolution, the
+mutation phase — are tested offline and without promtool by
+`scripts/ci/test-promtool-rules.sh`, which runs on every pull request via
+`.woodpecker/ci-scripts-test.yml`.

kubernetes/apps/monitoring-rules/tests/cnpg-backup_test.yaml

@@ -0,0 +1,592 @@
+# GENERATED by generate-cnpg-backup-test.py — see README.md in this
+# directory before editing by hand.
+#
+# Unit tests for kubernetes/apps/monitoring-rules/prometheusrule-cnpg-backup.yaml.
+# Every assertion below is either one of Iris's seven measured nights
+# (REN-129) rebuilt from its published (peak, fill, drain) numbers, an
+# adversarial combination of the extremes of those numbers, or a failure
+# this cluster has not had but the rules exist to catch.
+#
+# `rule_files` entries are resolved RELATIVE TO THIS FILE'S DIRECTORY, not
+# to the working directory promtool was invoked from — promtool joins
+# dirname(testfile) onto each entry. So this is `../`, and it resolves in
+# the mirrored tree scripts/ci/promtool-rules.sh builds, where the
+# `spec:`-stripped manifests sit in the same layout as the real ones.
+#
+# A path that does NOT resolve is only a WARNING to promtool: it loads zero
+# rules, every exp_alerts: [] assertion passes against nothing, and the
+# suite exits 0. promtool-rules.sh resolves every entry itself before
+# running and fails on a miss; that check is the only thing standing
+# between a typo here and a green pipeline that tests nothing.
+rule_files:
+  - ../prometheusrule-cnpg-backup.yaml
+evaluation_interval: 1m
+
+tests:
+  # ---------------------------------------------------------------------
+  # 1-7. THE SEVEN MEASURED NIGHTS. Both rules must stay silent through
+  # every one of them. This is the half of the suite that has already
+  # caught something: the surge rule shipped for review at `for: 2m` and
+  # fires here on 6 of these 7 nights (mutations.txt:surge-for-2m).
+  # ---------------------------------------------------------------------
+  # 2026-09-20: the nightly Home Assistant recorder purge at 04:12 local.
+  # peak 106 at 02:15Z, fill 35.3/min, drain 13.2/min -> 10 samples above 5 (9m pending vs for: 15m), 5 above 64 (4m pending vs for: 8m).
+  - name: nightly burst 2026-09-20 — recorder purge, benign, self-clearing
+    interval: 1m
+    input_series:
+      - series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="ready"}'
+        # 02:08Z .. 02:47Z, one sample per minute
+        values: '0 0 0 0 0 35 71 106 93 80 66 53 40 27 14 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0'
+      - series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="done"}'
+        # The cumulative archived counter on the same metric name. The
+        # rules must never read it; see mutations.txt:no-value-selector.
+        values: '148000 148014 148028 148042 148056 148070 148084 148098 148112 148126 148140 148154 148168 148182 148196 148210 148224 148238 148252 148266 148280 148294 148308 148322 148336 148350 148364 148378 148392 148406 148420 148434 148448 148462 148476 148490 148504 148518 148532 148546'
+    alert_rule_test:
+      # 02:15Z, backlog 106
+      - eval_time: 7m
+        alertname: CNPGWalArchiveBacklog
+        exp_alerts: []
+      # 02:15Z, backlog 106
+      - eval_time: 7m
+        alertname: CNPGWalArchiveBacklogSurge
+        exp_alerts: []
+      # 02:18Z, backlog 66
+      - eval_time: 10m
+        alertname: CNPGWalArchiveBacklog
+        exp_alerts: []
+      # 02:18Z, backlog 66
+      - eval_time: 10m
+        alertname: CNPGWalArchiveBacklogSurge
+        exp_alerts: []
+      # 02:22Z, backlog 14
+      - eval_time: 14m
+        alertname: CNPGWalArchiveBacklog
+        exp_alerts: []
+      # 02:22Z, backlog 14
+      - eval_time: 14m
+        alertname: CNPGWalArchiveBacklogSurge
+        exp_alerts: []
+      # 02:24Z, backlog 0
+      - eval_time: 16m
+        alertname: CNPGWalArchiveBacklog
+        exp_alerts: []
+      # 02:24Z, backlog 0
+      - eval_time: 16m
+        alertname: CNPGWalArchiveBacklogSurge
+        exp_alerts: []
+
+  # 2026-09-21: the nightly Home Assistant recorder purge at 04:12 local.
+  # peak 100 at 02:15Z, fill 33.3/min, drain 14.3/min -> 9 samples above 5 (8m pending vs for: 15m), 4 above 64 (3m pending vs for: 8m).
+  - name: nightly burst 2026-09-21 — recorder purge, benign, self-clearing
+    interval: 1m
+    input_series:
+      - series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="ready"}'
+        # 02:08Z .. 02:47Z, one sample per minute
+        values: '0 0 0 0 0 33 67 100 86 71 57 43 28 14 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0'
+      - series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="done"}'
+        # The cumulative archived counter on the same metric name. The
+        # rules must never read it; see mutations.txt:no-value-selector.
+        values: '148000 148014 148028 148042 148056 148070 148084 148098 148112 148126 148140 148154 148168 148182 148196 148210 148224 148238 148252 148266 148280 148294 148308 148322 148336 148350 148364 148378 148392 148406 148420 148434 148448 148462 148476 148490 148504 148518 148532 148546'
+    alert_rule_test:
+      # 02:15Z, backlog 100
+      - eval_time: 7m
+        alertname: CNPGWalArchiveBacklog
+        exp_alerts: []
+      # 02:15Z, backlog 100
+      - eval_time: 7m
+        alertname: CNPGWalArchiveBacklogSurge
+        exp_alerts: []
+      # 02:17Z, backlog 71
+      - eval_time: 9m
+        alertname: CNPGWalArchiveBacklog
+        exp_alerts: []
+      # 02:17Z, backlog 71
+      - eval_time: 9m
+        alertname: CNPGWalArchiveBacklogSurge
+        exp_alerts: []
+      # 02:21Z, backlog 14
+      - eval_time: 13m
+        alertname: CNPGWalArchiveBacklog
+        exp_alerts: []
+      # 02:21Z, backlog 14
+      - eval_time: 13m
+        alertname: CNPGWalArchiveBacklogSurge
+        exp_alerts: []
+      # 02:23Z, backlog 0
+      - eval_time: 15m
+        alertname: CNPGWalArchiveBacklog
+        exp_alerts: []
+      # 02:23Z, backlog 0
+      - eval_time: 15m
+        alertname: CNPGWalArchiveBacklogSurge
+        exp_alerts: []
+
+  # 2026-09-22: the nightly Home Assistant recorder purge at 04:12 local.
+  # peak 101 at 02:15Z, fill 33.7/min, drain 14.4/min -> 9 samples above 5 (8m pending vs for: 15m), 4 above 64 (3m pending vs for: 8m).
+  - name: nightly burst 2026-09-22 — recorder purge, benign, self-clearing
+    interval: 1m
+    input_series:
+      - series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="ready"}'
+        # 02:08Z .. 02:47Z, one sample per minute
+        values: '0 0 0 0 0 34 67 101 87 72 58 43 29 15 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0'
+      - series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="done"}'
+        # The cumulative archived counter on the same metric name. The
+        # rules must never read it; see mutations.txt:no-value-selector.
+        values: '148000 148014 148028 148042 148056 148070 148084 148098 148112 148126 148140 148154 148168 148182 148196 148210 148224 148238 148252 148266 148280 148294 148308 148322 148336 148350 148364 148378 148392 148406 148420 148434 148448 148462 148476 148490 148504 148518 148532 148546'
+    alert_rule_test:
+      # 02:15Z, backlog 101
+      - eval_time: 7m
+        alertname: CNPGWalArchiveBacklog
+        exp_alerts: []
+      # 02:15Z, backlog 101
+      - eval_time: 7m
+        alertname: CNPGWalArchiveBacklogSurge
+        exp_alerts: []
+      # 02:17Z, backlog 72
+      - eval_time: 9m
+        alertname: CNPGWalArchiveBacklog
+        exp_alerts: []
+      # 02:17Z, backlog 72
+      - eval_time: 9m
+        alertname: CNPGWalArchiveBacklogSurge
+        exp_alerts: []
+      # 02:21Z, backlog 15
+      - eval_time: 13m
+        alertname: CNPGWalArchiveBacklog
+        exp_alerts: []
+      # 02:21Z, backlog 15
+      - eval_time: 13m
+        alertname: CNPGWalArchiveBacklogSurge
+        exp_alerts: []
+      # 02:23Z, backlog 0
+      - eval_time: 15m
+        alertname: CNPGWalArchiveBacklog
+        exp_alerts: []
+      # 02:23Z, backlog 0
+      - eval_time: 15m
+        alertname: CNPGWalArchiveBacklogSurge
+        exp_alerts: []
+
+  # 2026-09-23: the nightly Home Assistant recorder purge at 04:12 local.
+  # peak 119 at 02:15Z, fill 39.7/min, drain 14.9/min -> 10 samples above 5 (9m pending vs for: 15m), 5 above 64 (4m pending vs for: 8m).
+  - name: nightly burst 2026-09-23 — recorder purge, benign, self-clearing
+    interval: 1m
+    input_series:
+      - series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="ready"}'
+        # 02:08Z .. 02:47Z, one sample per minute
+        values: '0 0 0 0 0 40 79 119 104 89 74 59 44 30 15 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0'
+      - series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="done"}'
+        # The cumulative archived counter on the same metric name. The
+        # rules must never read it; see mutations.txt:no-value-selector.
+        values: '148000 148014 148028 148042 148056 148070 148084 148098 148112 148126 148140 148154 148168 148182 148196 148210 148224 148238 148252 148266 148280 148294 148308 148322 148336 148350 148364 148378 148392 148406 148420 148434 148448 148462 148476 148490 148504 148518 148532 148546'
+    alert_rule_test:
+      # 02:15Z, backlog 119
+      - eval_time: 7m
+        alertname: CNPGWalArchiveBacklog
+        exp_alerts: []
+      # 02:15Z, backlog 119
+      - eval_time: 7m
+        alertname: CNPGWalArchiveBacklogSurge
+        exp_alerts: []
+      # 02:18Z, backlog 74
+      - eval_time: 10m
+        alertname: CNPGWalArchiveBacklog
+        exp_alerts: []
+      # 02:18Z, backlog 74
+      - eval_time: 10m
+        alertname: CNPGWalArchiveBacklogSurge
+        exp_alerts: []
+      # 02:22Z, backlog 15
+      - eval_time: 14m
+        alertname: CNPGWalArchiveBacklog
+        exp_alerts: []
+      # 02:22Z, backlog 15
+      - eval_time: 14m
+        alertname: CNPGWalArchiveBacklogSurge
+        exp_alerts: []
+      # 02:24Z, backlog 0
+      - eval_time: 16m
+        alertname: CNPGWalArchiveBacklog
+        exp_alerts: []
+      # 02:24Z, backlog 0
+      - eval_time: 16m
+        alertname: CNPGWalArchiveBacklogSurge
+        exp_alerts: []
+
+  # 2026-09-24: the nightly Home Assistant recorder purge at 04:12 local.
+  # peak 56 at 02:14Z, fill 18.7/min, drain 14.0/min -> 6 samples above 5 (5m pending vs for: 15m), 0 above 64 (0m pending vs for: 8m).
+  - name: nightly burst 2026-09-24 — recorder purge, benign, self-clearing
+    interval: 1m
+    input_series:
+      - series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="ready"}'
+        # 02:08Z .. 02:47Z, one sample per minute
+        values: '0 0 0 0 19 37 56 42 28 14 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0'
+      - series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="done"}'
+        # The cumulative archived counter on the same metric name. The
+        # rules must never read it; see mutations.txt:no-value-selector.
+        values: '148000 148014 148028 148042 148056 148070 148084 148098 148112 148126 148140 148154 148168 148182 148196 148210 148224 148238 148252 148266 148280 148294 148308 148322 148336 148350 148364 148378 148392 148406 148420 148434 148448 148462 148476 148490 148504 148518 148532 148546'
+    alert_rule_test:
+      # 02:14Z, backlog 56
+      - eval_time: 6m
+        alertname: CNPGWalArchiveBacklog
+        exp_alerts: []
+      # 02:14Z, backlog 56
+      - eval_time: 6m
+        alertname: CNPGWalArchiveBacklogSurge
+        exp_alerts: []
+      # 02:17Z, backlog 14
+      - eval_time: 9m
+        alertname: CNPGWalArchiveBacklog
+        exp_alerts: []
+      # 02:17Z, backlog 14
+      - eval_time: 9m
+        alertname: CNPGWalArchiveBacklogSurge
+        exp_alerts: []
+      # 02:19Z, backlog 0
+      - eval_time: 11m
+        alertname: CNPGWalArchiveBacklog
+        exp_alerts: []
+      # 02:19Z, backlog 0
+      - eval_time: 11m
+        alertname: CNPGWalArchiveBacklogSurge
+        exp_alerts: []
+
+  # 2026-09-25: the nightly Home Assistant recorder purge at 04:12 local.
+  # peak 91 at 02:15Z, fill 22.8/min, drain 13.0/min -> 10 samples above 5 (9m pending vs for: 15m), 4 above 64 (3m pending vs for: 8m).
+  - name: nightly burst 2026-09-25 — recorder purge, benign, self-clearing
+    interval: 1m
+    input_series:
+      - series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="ready"}'
+        # 02:08Z .. 02:47Z, one sample per minute
+        values: '0 0 0 0 23 46 68 91 78 65 52 39 26 13 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0'
+      - series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="done"}'
+        # The cumulative archived counter on the same metric name. The
+        # rules must never read it; see mutations.txt:no-value-selector.
+        values: '148000 148014 148028 148042 148056 148070 148084 148098 148112 148126 148140 148154 148168 148182 148196 148210 148224 148238 148252 148266 148280 148294 148308 148322 148336 148350 148364 148378 148392 148406 148420 148434 148448 148462 148476 148490 148504 148518 148532 148546'
+    alert_rule_test:
+      # 02:15Z, backlog 91
+      - eval_time: 7m
+        alertname: CNPGWalArchiveBacklog
+        exp_alerts: []
+      # 02:15Z, backlog 91
+      - eval_time: 7m
+        alertname: CNPGWalArchiveBacklogSurge
+        exp_alerts: []
+      # 02:17Z, backlog 65
+      - eval_time: 9m
+        alertname: CNPGWalArchiveBacklog
+        exp_alerts: []
+      # 02:17Z, backlog 65
+      - eval_time: 9m
+        alertname: CNPGWalArchiveBacklogSurge
+        exp_alerts: []
+      # 02:21Z, backlog 13
+      - eval_time: 13m
+        alertname: CNPGWalArchiveBacklog
+        exp_alerts: []
+      # 02:21Z, backlog 13
+      - eval_time: 13m
+        alertname: CNPGWalArchiveBacklogSurge
+        exp_alerts: []
+      # 02:23Z, backlog 0
+      - eval_time: 15m
+        alertname: CNPGWalArchiveBacklog
+        exp_alerts: []
+      # 02:23Z, backlog 0
+      - eval_time: 15m
+        alertname: CNPGWalArchiveBacklogSurge
+        exp_alerts: []
+
+  # 2026-09-26: the nightly Home Assistant recorder purge at 04:12 local.
+  # peak 113 at 02:15Z, fill 28.5/min, drain 14.1/min -> 11 samples above 5 (10m pending vs for: 15m), 5 above 64 (4m pending vs for: 8m).
+  - name: nightly burst 2026-09-26 — recorder purge, benign, self-clearing
+    interval: 1m
+    input_series:
+      - series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="ready"}'
+        # 02:08Z .. 02:47Z, one sample per minute
+        values: '0 0 0 0 28 57 86 113 99 85 71 57 42 28 14 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0'
+      - series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="done"}'
+        # The cumulative archived counter on the same metric name. The
+        # rules must never read it; see mutations.txt:no-value-selector.
+        values: '148000 148014 148028 148042 148056 148070 148084 148098 148112 148126 148140 148154 148168 148182 148196 148210 148224 148238 148252 148266 148280 148294 148308 148322 148336 148350 148364 148378 148392 148406 148420 148434 148448 148462 148476 148490 148504 148518 148532 148546'
+    alert_rule_test:
+      # 02:15Z, backlog 113
+      - eval_time: 7m
+        alertname: CNPGWalArchiveBacklog
+        exp_alerts: []
+      # 02:15Z, backlog 113
+      - eval_time: 7m
+        alertname: CNPGWalArchiveBacklogSurge
+        exp_alerts: []
+      # 02:18Z, backlog 71
+      - eval_time: 10m
+        alertname: CNPGWalArchiveBacklog
+        exp_alerts: []
+      # 02:18Z, backlog 71
+      - eval_time: 10m
+        alertname: CNPGWalArchiveBacklogSurge
+        exp_alerts: []
+      # 02:22Z, backlog 14
+      - eval_time: 14m
+        alertname: CNPGWalArchiveBacklog
+        exp_alerts: []
+      # 02:22Z, backlog 14
+      - eval_time: 14m
+        alertname: CNPGWalArchiveBacklogSurge
+        exp_alerts: []
+      # 02:24Z, backlog 0
+      - eval_time: 16m
+        alertname: CNPGWalArchiveBacklog
+        exp_alerts: []
+      # 02:24Z, backlog 0
+      - eval_time: 16m
+        alertname: CNPGWalArchiveBacklogSurge
+        exp_alerts: []
+
+  # ---------------------------------------------------------------------
+  # 8. THE ADVERSARIAL ENVELOPE. A night that did not happen, built from
+  # the worst fill (39.7/min, 09-23), the highest peak (119, 09-23) and
+  # the slowest drain (13.0/min, 09-25) in the week. Inside what the
+  # cluster has demonstrated, worse than anything it actually did.
+  #
+  # This is where the two `for` windows get measured for margin rather
+  # than asserted about. Margin is PENDING time, which is one less than
+  # the run length — an alert goes pending on the first true evaluation
+  # and fires at activeAt + for, so 6 consecutive samples above 64
+  # accumulate 5 minutes, not 6:
+  #
+  #   above 5:   11 samples ->  10m pending  vs `for: 15m`  -> 5m margin
+  #   above 64:   6 samples ->   5m pending  vs `for: 8m`   -> 3m margin
+  #
+  # The worst MEASURED night accumulates 4m above 64 (5 samples, 09-20
+  # and 09-23), so the envelope is one minute tighter than anything that
+  # actually happened. 3 minutes is the number to defend if anyone
+  # proposes lowering `for: 8m` — and mutations.txt:surge-for-5m is the
+  # control that makes this case load-bearing rather than decorative: at
+  # `for: 5m` the surge rule is still silent on all seven real nights and
+  # fires only here. Delete this case and that mutation stops being
+  # caught.
+  # ---------------------------------------------------------------------
+  # Synthetic: max fill 39.7 + max peak 119 + min drain 13.0.
+  # peak 119 at 02:15Z, fill 39.7/min, drain 13.0/min -> 11 samples above 5 (10m pending vs for: 15m), 6 above 64 (5m pending vs for: 8m).
+  - name: adversarial envelope — worst measured fill and peak, slowest drain
+    interval: 1m
+    input_series:
+      - series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="ready"}'
+        # 02:08Z .. 02:47Z, one sample per minute
+        values: '0 0 0 0 0 40 79 119 106 93 80 67 54 41 28 15 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0'
+      - series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="done"}'
+        # The cumulative archived counter on the same metric name. The
+        # rules must never read it; see mutations.txt:no-value-selector.
+        values: '148000 148014 148028 148042 148056 148070 148084 148098 148112 148126 148140 148154 148168 148182 148196 148210 148224 148238 148252 148266 148280 148294 148308 148322 148336 148350 148364 148378 148392 148406 148420 148434 148448 148462 148476 148490 148504 148518 148532 148546'
+    alert_rule_test:
+      # 02:15Z, backlog 119
+      - eval_time: 7m
+        alertname: CNPGWalArchiveBacklog
+        exp_alerts: []
+      # 02:15Z, backlog 119
+      - eval_time: 7m
+        alertname: CNPGWalArchiveBacklogSurge
+        exp_alerts: []
+      # 02:19Z, backlog 67
+      - eval_time: 11m
+        alertname: CNPGWalArchiveBacklog
+        exp_alerts: []
+      # 02:19Z, backlog 67
+      - eval_time: 11m
+        alertname: CNPGWalArchiveBacklogSurge
+        exp_alerts: []
+      # 02:23Z, backlog 15
+      - eval_time: 15m
+        alertname: CNPGWalArchiveBacklog
+        exp_alerts: []
+      # 02:23Z, backlog 15
+      - eval_time: 15m
+        alertname: CNPGWalArchiveBacklogSurge
+        exp_alerts: []
+      # 02:26Z, backlog 0
+      - eval_time: 18m
+        alertname: CNPGWalArchiveBacklog
+        exp_alerts: []
+      # 02:26Z, backlog 0
+      - eval_time: 18m
+        alertname: CNPGWalArchiveBacklogSurge
+        exp_alerts: []
+
+  # ---------------------------------------------------------------------
+  # 9. THE FAILURE BOTH RULES EXIST FOR: an archiver that stopped with
+  # 70 segments (1.1 GiB) queued. Not a burst — it does not drain.
+  # Surge fires first (8m after the backlog crosses 64), the
+  # load-bearing rule 7 minutes later (15m after it crosses 5). Both
+  # assertions bracket the transition: one minute early nothing is
+  # firing, on the minute it is.
+  # ---------------------------------------------------------------------
+  - name: stalled archiver at 70 segments — both rules must fire
+    interval: 1m
+    input_series:
+      - series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="ready"}'
+        values: '0 0 0 0 0 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70'
+      - series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="done"}'
+        # Flat, because a stalled archiver archives nothing. The rules do
+        # not read this; it is here so the input stays realistic.
+        values: '148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000 148000'
+    alert_rule_test:
+      # Backlog crosses 64 at t=5m, so `for: 8m` elapses at t=13m.
+      - eval_time: 12m
+        alertname: CNPGWalArchiveBacklogSurge
+        exp_alerts: []
+      - eval_time: 13m
+        alertname: CNPGWalArchiveBacklogSurge
+        exp_alerts:
+          # No `role` label: proof that `max by (namespace, pod)`
+          # aggregated it away. See mutations.txt:no-max-by.
+          - exp_labels:
+              severity: warning
+              namespace: home-assistant
+              pod: ha-pg-1
+            exp_annotations:
+              summary: "CNPG WAL archiving cannot keep up on home-assistant/ha-pg-1"
+              description: "70 WAL segments (>1 GiB) have been queued for archiving for 8 minutes, which is longer than a healthy archiver needs to clear that much — from a backlog of 119 it takes under 5. Point-in-time recovery is degraded for everything after the last archived segment, and pg_wal grows against the PVC's free space until it clears. Start with the archiver: `kubectl -n home-assistant logs ha-pg-1 -c plugin-barman-cloud`, and check whether its ObjectStore sets `wal.maxParallel` — unset means one segment at a time, a ceiling of about 14 segments/min (REN-129)."
+      # Backlog crosses 5 at t=5m, so `for: 15m` elapses at t=20m.
+      - eval_time: 19m
+        alertname: CNPGWalArchiveBacklog
+        exp_alerts: []
+      - eval_time: 20m
+        alertname: CNPGWalArchiveBacklog
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              namespace: home-assistant
+              pod: ha-pg-1
+            exp_annotations:
+              summary: "CNPG WAL archiving is falling behind on home-assistant/ha-pg-1"
+              description: "70 WAL segments have been waiting to archive for over 15 minutes. Point-in-time recovery is broken for anything after the last archived segment, and pg_wal will grow until the PVC fills. Check the barman-cloud sidecar logs and the ObjectStore credentials: `kubectl -n home-assistant logs ha-pg-1 -c plugin-barman-cloud`."
+
+  # ---------------------------------------------------------------------
+  # 10. THE STALL THE SURGE RULE CANNOT SEE: 12 segments, stuck. Under
+  # 1 GiB forever, so the surge rule is right to stay silent — and this
+  # is exactly the shape that makes the >5 rule load-bearing rather than
+  # a slower copy of it. Point-in-time recovery is just as broken at 12
+  # segments as at 70.
+  # ---------------------------------------------------------------------
+  - name: low stall at 12 segments — only the >5 rule may fire
+    interval: 1m
+    input_series:
+      - series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="ready"}'
+        values: '0 0 0 0 0 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12'
+    alert_rule_test:
+      - eval_time: 20m
+        alertname: CNPGWalArchiveBacklog
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              namespace: home-assistant
+              pod: ha-pg-1
+            exp_annotations:
+              summary: "CNPG WAL archiving is falling behind on home-assistant/ha-pg-1"
+              description: "12 WAL segments have been waiting to archive for over 15 minutes. Point-in-time recovery is broken for anything after the last archived segment, and pg_wal will grow until the PVC fills. Check the barman-cloud sidecar logs and the ObjectStore credentials: `kubectl -n home-assistant logs ha-pg-1 -c plugin-barman-cloud`."
+      # 34 minutes at 12 segments and the surge rule is still quiet.
+      - eval_time: 39m
+        alertname: CNPGWalArchiveBacklogSurge
+        exp_alerts: []
+
+  # ---------------------------------------------------------------------
+  # 11. PINNED AT EXACTLY 5 for the whole window. `> 5` is strict, so
+  # this is silence. A rule written `>= 5` would page on it, and a rule
+  # that had lost its `value="ready"` selector would page on the `done`
+  # counter regardless of what `ready` does — which is what the second
+  # series here is for.
+  # ---------------------------------------------------------------------
+  - name: pinned at exactly 5 — strict > means silence
+    interval: 1m
+    input_series:
+      - series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="ready"}'
+        values: '5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5'
+      - series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="done"}'
+        values: '148000 148014 148028 148042 148056 148070 148084 148098 148112 148126 148140 148154 148168 148182 148196 148210 148224 148238 148252 148266 148280 148294 148308 148322 148336 148350 148364 148378 148392 148406 148420 148434 148448 148462 148476 148490 148504 148518 148532 148546'
+    alert_rule_test:
+      - eval_time: 39m
+        alertname: CNPGWalArchiveBacklog
+        exp_alerts: []
+      - eval_time: 39m
+        alertname: CNPGWalArchiveBacklogSurge
+        exp_alerts: []
+
+  # ---------------------------------------------------------------------
+  # 12. FAILOVER DURING A STALL — the case `max by (namespace, pod)` was
+  # added for. The backlog starts at t=5m on the primary; at t=11m the
+  # instance is promoted and `role` changes, which is a NEW SERIES to
+  # Prometheus and restarts the `for` timer on the raw expression.
+  # Aggregating `role` away keeps one continuous series, so the page
+  # still lands 15m after the backlog started (t=20m) rather than 15m
+  # after the failover. Without `max by` this case is silent at t=20m —
+  # see mutations.txt:no-max-by.
+  # ---------------------------------------------------------------------
+  - name: role flip mid-stall — for timer must not restart
+    interval: 1m
+    input_series:
+      - series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="ready"}'
+        # t=0..10m, then this series ends.
+        values: '0 0 0 0 0 70 70 70 70 70 70'
+      - series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="replica",value="ready"}'
+        # Starts at t=11m. `_` is a gap, not a zero.
+        values: '_ _ _ _ _ _ _ _ _ _ _ 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70'
+    alert_rule_test:
+      - eval_time: 19m
+        alertname: CNPGWalArchiveBacklog
+        exp_alerts: []
+      - eval_time: 20m
+        alertname: CNPGWalArchiveBacklog
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              namespace: home-assistant
+              pod: ha-pg-1
+            exp_annotations:
+              summary: "CNPG WAL archiving is falling behind on home-assistant/ha-pg-1"
+              description: "70 WAL segments have been waiting to archive for over 15 minutes. Point-in-time recovery is broken for anything after the last archived segment, and pg_wal will grow until the PVC fills. Check the barman-cloud sidecar logs and the ObjectStore credentials: `kubectl -n home-assistant logs ha-pg-1 -c plugin-barman-cloud`."
+
+  # ---------------------------------------------------------------------
+  # 13. ONE STALLED INSTANCE AMONG HEALTHY ONES. There are 12 CNPG
+  # instances in this cluster. `by (namespace, pod)` has to keep them
+  # apart: exactly one alert, naming the pod that is actually stuck, so
+  # the 04:00 reader is not sent to look at the wrong Postgres.
+  # ---------------------------------------------------------------------
+  - name: one stalled pod among healthy ones — exactly one alert
+    interval: 1m
+    input_series:
+      - series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-1",container="postgres",role="primary",value="ready"}'
+        values: '0 0 0 0 0 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70 70'
+      - series: 'cnpg_collector_pg_wal_archive_status{namespace="home-assistant",pod="ha-pg-2",container="postgres",role="replica",value="ready"}'
+        values: '0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0'
+      - series: 'cnpg_collector_pg_wal_archive_status{namespace="immich",pod="immich-pg-1",container="postgres",role="primary",value="ready"}'
+        values: '0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0'
+    alert_rule_test:
+      - eval_time: 20m
+        alertname: CNPGWalArchiveBacklog
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              namespace: home-assistant
+              pod: ha-pg-1
+            exp_annotations:
+              summary: "CNPG WAL archiving is falling behind on home-assistant/ha-pg-1"
+              description: "70 WAL segments have been waiting to archive for over 15 minutes. Point-in-time recovery is broken for anything after the last archived segment, and pg_wal will grow until the PVC fills. Check the barman-cloud sidecar logs and the ObjectStore credentials: `kubectl -n home-assistant logs ha-pg-1 -c plugin-barman-cloud`."
+
+  # ---------------------------------------------------------------------
+  # 14. THE EPOCH-ZERO GUARD on CNPGBackupFailed. The metric is a unix
+  # timestamp and reads 0 when no failure was ever recorded, so the rule
+  # carries a `> 0` guard to stop that default being read as "failed in
+  # 1970". Held at 0 for the whole window: silence. Nothing else in this
+  # suite exercises that rule, and a dropped guard would page on every
+  # instance in the cluster forever.
+  # ---------------------------------------------------------------------
+  - name: never-failed backup reads timestamp 0 — must not fire
+    interval: 1m
+    input_series:
+      - series: 'cnpg_collector_last_failed_backup_timestamp{namespace="home-assistant",pod="ha-pg-1"}'
+        values: '0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0'
+    alert_rule_test:
+      - eval_time: 39m
+        alertname: CNPGBackupFailed
+        exp_alerts: []

kubernetes/apps/monitoring-rules/tests/generate-cnpg-backup-test.py

@@ -0,0 +1,563 @@
+#!/usr/bin/env python3
+"""Regenerate cnpg-backup_test.yaml from the REN-129 measurements.
+
+The committed YAML is what CI runs; this script exists so the numbers in it are
+auditable and reproducible from the measurement table rather than being a
+hand-typed artifact nobody can re-derive. Run it from the repo root:
+
+    python3 kubernetes/apps/monitoring-rules/tests/generate-cnpg-backup-test.py \
+      > kubernetes/apps/monitoring-rules/tests/cnpg-backup_test.yaml
+
+Nothing in CI regenerates and diffs this — the promtool image has no Python, and
+the defence against a hand-weakened suite is the mutation phase of
+scripts/ci/promtool-rules.sh, not byte-equality with a generator. See README.md.
+
+THE TRACE MODEL. Iris measured seven consecutive nights of
+`cnpg_collector_pg_wal_archive_status{value="ready",pod="ha-pg-1"}` at 1-minute
+steps and published, per night, a (peak, peak minute, fill seg/min, drain
+seg/min) tuple. Those four numbers are consistent with one shape, which is the
+shape every night has: flat 0, a linear ramp at `fill` per minute to `peak`,
+then a linear decay at `drain` per minute back to 0. So a night is rebuilt as
+
+    ramp length  = round(peak / fill)          minutes
+    ramp start   = peak minute - ramp length   (value 0 at that minute)
+    during ramp  = round(fill * minutes since ramp start)
+    at peak      = peak                        (exact, not rounded from fill)
+    during decay = max(0, round(peak - drain * minutes since peak))
+
+FIDELITY. The reconstruction reproduces Iris's published `Duration >5` column
+exactly on 2026-09-20, -21 and -22, and runs one minute long on -23 and -24 and
+two minutes long on -25 and -26. Every discrepancy is in the same direction:
+the rebuild is never SHORTER above the threshold than the measurement. A linear
+decay at the published mean drain rate holds the tail above 5 for slightly
+longer than the real trace did, so each night is a marginally harder silence
+test than the night it stands for. That is the safe direction for this suite,
+whose whole job is to catch a rule that fires on a benign night. `--report`
+prints the comparison; do not "fix" it toward the published column without
+checking which way each night moves.
+
+These are reconstructions, not the raw samples. They are faithful in the only
+dimensions the rules read: how high the backlog goes, and how many consecutive
+minutes it spends above each threshold. Do not read them as a recording.
+"""
+
+import argparse
+import sys
+
+# Iris's 7-night measurement, REN-129. `dur_gt5` is her published "Duration >5"
+# column and is an INPUT TO THE SELF-CHECK ONLY — the traces are built from
+# peak/fill/drain/peak_min, and dur_gt5 is what the result is compared against.
+NIGHTS = [
+    # date        peak  peak_min  fill  drain  dur_gt5
+    ("2026-09-20", 106, "02:15", 35.3, 13.2, 10),
+    ("2026-09-21", 100, "02:15", 33.3, 14.3, 9),
+    ("2026-09-22", 101, "02:15", 33.7, 14.4, 9),
+    ("2026-09-23", 119, "02:15", 39.7, 14.9, 9),
+    ("2026-09-24", 56, "02:14", 18.7, 14.0, 5),
+    ("2026-09-25", 91, "02:15", 22.8, 13.0, 8),
+    ("2026-09-26", 113, "02:15", 28.5, 14.1, 9),
+]
+
+# The adversarial envelope: worst fill, worst peak, slowest drain observed in the
+# week, combined into a night that did not happen but is inside what the cluster
+# has demonstrated it can do. Nothing here is extrapolated past the measurements.
+ENVELOPE = ("envelope", 119, "02:15", 39.7, 13.0, None)
+
+# Series labels. `container`, `role` and `instance` are on the real series and
+# are carried here on purpose: they are what `max by (namespace, pod)` has to
+# aggregate away, and a test whose input is already aggregated cannot see whether
+# the rule does that. `value="done"` is the cumulative counter that sits on the
+# same metric name — it is in the input so that dropping the `value="ready"`
+# selector from the rules is a detectable mutation and not a silent no-op.
+NS = "home-assistant"
+POD = "ha-pg-1"
+
+
+def labels(pod=POD, value="ready", role="primary", ns=NS):
+    return (
+        "cnpg_collector_pg_wal_archive_status{"
+        f'namespace="{ns}",pod="{pod}",container="postgres",'
+        f'role="{role}",value="{value}"'
+        "}"
+    )
+
+
+# Test window: t=0 is 02:08Z, 40 one-minute samples, so every night's burst sits
+# inside it with lead-in and tail. promtool eval_times are relative to t=0, so
+# every assertion below carries its UTC clock time in a comment.
+#
+# This MUST stay aligned with the peak minutes in NIGHTS. An earlier revision
+# had `8 * 60 + 8` here (08:08Z), which put the whole window seven hours after
+# every burst: all seven nights rebuilt as 40 zeroes and their ~90 silence
+# assertions passed without exercising a single threshold. The
+# assert_window_covers_bursts() call in main() exists so that cannot recur
+# silently.
+T0 = 2 * 60 + 8  # 02:08Z in minutes since midnight
+SAMPLES = 40
+
+
+def hhmm(minute):
+    return f"{minute // 60:02d}:{minute % 60:02d}"
+
+
+def parse_hhmm(s):
+    h, m = s.split(":")
+    return int(h) * 60 + int(m)
+
+
+def trace(peak, peak_min, fill, drain, n=SAMPLES, t0=T0):
+    """Rebuild one night. See THE TRACE MODEL in the module docstring."""
+    ramp_len = max(1, round(peak / fill))
+    ramp_start = peak_min - ramp_len
+    out = []
+    for m in range(t0, t0 + n):
+        if m <= ramp_start:
+            v = 0
+        elif m < peak_min:
+            v = round(fill * (m - ramp_start))
+        elif m == peak_min:
+            v = peak
+        else:
+            v = max(0, round(peak - drain * (m - peak_min)))
+        out.append(v)
+    return out
+
+
+def longest_run_above(vals, thr):
+    """(length, index of last sample) of the longest run of vals > thr."""
+    best = (0, None)
+    run = 0
+    for i, v in enumerate(vals):
+        if v > thr:
+            run += 1
+            if run > best[0]:
+                best = (run, i)
+        else:
+            run = 0
+    return best
+
+
+def pending_minutes(vals, thr):
+    """Max minutes a `for` timer accumulates on `> thr` over this trace.
+
+    This is NOT the run length, and the difference is the whole margin
+    argument. Prometheus makes an alert pending at the first evaluation where
+    the expression is true and fires it when `now - activeAt >= for`, so a run
+    of N consecutive 1-minute samples above the threshold accumulates N-1
+    minutes of pending time, not N. A trace that spends 5 samples above 64
+    therefore does NOT fire `for: 5m`; it fires `for: 4m`. Quote this number,
+    not the run length, when arguing about a `for` window.
+    """
+    n, _ = longest_run_above(vals, thr)
+    return max(0, n - 1)
+
+
+def assert_window_covers_bursts():
+    """Refuse to emit a suite whose trace window misses the bursts.
+
+    Guards the T0 bug described above T0: if the window and the peak minutes
+    ever drift apart again, every measured night rebuilds as flat zero and the
+    suite passes while testing nothing. A night that is flat 0 cannot be a
+    silence test for a burst, so treat it as a generator fault, not a result.
+    """
+    for date, peak, pm, fill, drain, _dur in NIGHTS + [ENVELOPE]:
+        vals = trace(peak, parse_hhmm(pm), fill, drain)
+        if max(vals) != peak:
+            raise SystemExit(
+                f"generator fault: {date} rebuilt with max {max(vals)}, expected "
+                f"peak {peak}. The T0={T0} window ({hhmm(T0)}Z.."
+                f"{hhmm(T0 + SAMPLES - 1)}Z) does not cover peak minute {pm}Z. "
+                "Fix T0 or SAMPLES; do not emit a suite of flat-zero nights."
+            )
+
+
+def counter(start, step, n=SAMPLES):
+    return [start + step * i for i in range(n)]
+
+
+def fmt(vals):
+    return " ".join(str(v) for v in vals)
+
+
+def eval_minutes(vals):
+    """The minutes where a `for` window is closest to elapsing, so the tightest
+    places to assert silence. For each threshold: the last sample of the longest
+    run above it (the moment maximum pending time has accumulated). Plus the
+    peak, plus two minutes after the backlog has settled back to 0."""
+    picks = set()
+    for thr in (5, 64):
+        _, last = longest_run_above(vals, thr)
+        if last is not None:
+            picks.add(last)
+    picks.add(max(range(len(vals)), key=lambda i: vals[i]))
+    settled = next(
+        (i for i in range(len(vals) - 1, -1, -1) if vals[i] > 0),
+        len(vals) - 1,
+    )
+    picks.add(min(settled + 2, len(vals) - 1))
+    return sorted(picks)
+
+
+L = []
+
+
+def w(line=""):
+    L.append(line)
+
+
+def burst_case(name, date, peak, peak_min_s, fill, drain, why):
+    peak_min = parse_hhmm(peak_min_s)
+    vals = trace(peak, peak_min, fill, drain)
+    n5, _ = longest_run_above(vals, 5)
+    n64, _ = longest_run_above(vals, 64)
+    w(f"  # {why}")
+    w(
+        f"  # peak {peak} at {peak_min_s}Z, fill {fill}/min, drain {drain}/min"
+        f" -> {n5} samples above 5 ({pending_minutes(vals, 5)}m pending vs"
+        f" for: 15m), {n64} above 64 ({pending_minutes(vals, 64)}m pending vs"
+        f" for: 8m)."
+    )
+    w(f"  - name: {name}")
+    w("    interval: 1m")
+    w("    input_series:")
+    w(f"      - series: '{labels()}'")
+    w(f"        # {hhmm(T0)}Z .. {hhmm(T0 + SAMPLES - 1)}Z, one sample per minute")
+    w(f"        values: '{fmt(vals)}'")
+    w(f"      - series: '{labels(value='done')}'")
+    w("        # The cumulative archived counter on the same metric name. The")
+    w("        # rules must never read it; see mutations.txt:no-value-selector.")
+    w(f"        values: '{fmt(counter(148_000, 14))}'")
+    w("    alert_rule_test:")
+    for i in eval_minutes(vals):
+        clock = hhmm(T0 + i)
+        for alert in ("CNPGWalArchiveBacklog", "CNPGWalArchiveBacklogSurge"):
+            w(f"      # {clock}Z, backlog {vals[i]}")
+            w(f"      - eval_time: {i}m")
+            w(f"        alertname: {alert}")
+            w("        exp_alerts: []")
+    w("")
+    return vals, n5, n64
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument(
+        "--report",
+        action="store_true",
+        help="print the reconstruction self-check table instead of the suite",
+    )
+    args = ap.parse_args()
+
+    assert_window_covers_bursts()
+
+    if args.report:
+        # `pend` columns are what the `for` windows are actually compared
+        # against; see pending_minutes(). `>5 pub` is Iris's published column
+        # and is the only number here that is not derived by this script.
+        print(f"{'night':12} {'peak':>5} {'fill':>6} {'drain':>6} "
+              f"{'>5 pub':>7} {'>5 n':>6} {'>5 pend':>8} "
+              f"{'>64 n':>6} {'>64 pend':>9}")
+        for date, peak, pm, fill, drain, dur in NIGHTS + [ENVELOPE]:
+            vals = trace(peak, parse_hhmm(pm), fill, drain)
+            n5, _ = longest_run_above(vals, 5)
+            n64, _ = longest_run_above(vals, 64)
+            print(f"{date:12} {peak:5} {fill:6} {drain:6} "
+                  f"{('-' if dur is None else dur):>7} {n5:>6} "
+                  f"{pending_minutes(vals, 5):>8} "
+                  f"{n64:>6} {pending_minutes(vals, 64):>9}")
+        print()
+        print("for: 15m on >5  must exceed every '>5 pend'  above.")
+        print("for: 8m  on >64 must exceed every '>64 pend' above.")
+        return 0
+
+    w("# GENERATED by generate-cnpg-backup-test.py — see README.md in this")
+    w("# directory before editing by hand.")
+    w("#")
+    w("# Unit tests for kubernetes/apps/monitoring-rules/prometheusrule-cnpg-backup.yaml.")
+    w("# Every assertion below is either one of Iris's seven measured nights")
+    w("# (REN-129) rebuilt from its published (peak, fill, drain) numbers, an")
+    w("# adversarial combination of the extremes of those numbers, or a failure")
+    w("# this cluster has not had but the rules exist to catch.")
+    w("#")
+    w("# `rule_files` entries are resolved RELATIVE TO THIS FILE'S DIRECTORY, not")
+    w("# to the working directory promtool was invoked from — promtool joins")
+    w("# dirname(testfile) onto each entry. So this is `../`, and it resolves in")
+    w("# the mirrored tree scripts/ci/promtool-rules.sh builds, where the")
+    w("# `spec:`-stripped manifests sit in the same layout as the real ones.")
+    w("#")
+    w("# A path that does NOT resolve is only a WARNING to promtool: it loads zero")
+    w("# rules, every exp_alerts: [] assertion passes against nothing, and the")
+    w("# suite exits 0. promtool-rules.sh resolves every entry itself before")
+    w("# running and fails on a miss; that check is the only thing standing")
+    w("# between a typo here and a green pipeline that tests nothing.")
+    w("rule_files:")
+    w("  - ../prometheusrule-cnpg-backup.yaml")
+    w("evaluation_interval: 1m")
+    w("")
+    w("tests:")
+    w("  # ---------------------------------------------------------------------")
+    w("  # 1-7. THE SEVEN MEASURED NIGHTS. Both rules must stay silent through")
+    w("  # every one of them. This is the half of the suite that has already")
+    w("  # caught something: the surge rule shipped for review at `for: 2m` and")
+    w("  # fires here on 6 of these 7 nights (mutations.txt:surge-for-2m).")
+    w("  # ---------------------------------------------------------------------")
+    checks = []
+    for date, peak, pm, fill, drain, dur in NIGHTS:
+        vals, n5, n64 = burst_case(
+            f"nightly burst {date} — recorder purge, benign, self-clearing",
+            date, peak, pm, fill, drain,
+            f"{date}: the nightly Home Assistant recorder purge at 04:12 local.",
+        )
+        checks.append((date, dur, n5))
+
+    w("  # ---------------------------------------------------------------------")
+    w("  # 8. THE ADVERSARIAL ENVELOPE. A night that did not happen, built from")
+    w("  # the worst fill (39.7/min, 09-23), the highest peak (119, 09-23) and")
+    w("  # the slowest drain (13.0/min, 09-25) in the week. Inside what the")
+    w("  # cluster has demonstrated, worse than anything it actually did.")
+    w("  #")
+    w("  # This is where the two `for` windows get measured for margin rather")
+    w("  # than asserted about. Margin is PENDING time, which is one less than")
+    w("  # the run length — an alert goes pending on the first true evaluation")
+    w("  # and fires at activeAt + for, so 6 consecutive samples above 64")
+    w("  # accumulate 5 minutes, not 6:")
+    w("  #")
+    w("  #   above 5:   11 samples ->  10m pending  vs `for: 15m`  -> 5m margin")
+    w("  #   above 64:   6 samples ->   5m pending  vs `for: 8m`   -> 3m margin")
+    w("  #")
+    w("  # The worst MEASURED night accumulates 4m above 64 (5 samples, 09-20")
+    w("  # and 09-23), so the envelope is one minute tighter than anything that")
+    w("  # actually happened. 3 minutes is the number to defend if anyone")
+    w("  # proposes lowering `for: 8m` — and mutations.txt:surge-for-5m is the")
+    w("  # control that makes this case load-bearing rather than decorative: at")
+    w("  # `for: 5m` the surge rule is still silent on all seven real nights and")
+    w("  # fires only here. Delete this case and that mutation stops being")
+    w("  # caught.")
+    w("  # ---------------------------------------------------------------------")
+    burst_case(
+        "adversarial envelope — worst measured fill and peak, slowest drain",
+        *ENVELOPE[:5],
+        why="Synthetic: max fill 39.7 + max peak 119 + min drain 13.0.",
+    )
+
+    # ---- 9. stalled archiver, above the surge threshold -------------------
+    stalled = [0] * 5 + [70] * 35
+    w("  # ---------------------------------------------------------------------")
+    w("  # 9. THE FAILURE BOTH RULES EXIST FOR: an archiver that stopped with")
+    w("  # 70 segments (1.1 GiB) queued. Not a burst — it does not drain.")
+    w("  # Surge fires first (8m after the backlog crosses 64), the")
+    w("  # load-bearing rule 7 minutes later (15m after it crosses 5). Both")
+    w("  # assertions bracket the transition: one minute early nothing is")
+    w("  # firing, on the minute it is.")
+    w("  # ---------------------------------------------------------------------")
+    w("  - name: stalled archiver at 70 segments — both rules must fire")
+    w("    interval: 1m")
+    w("    input_series:")
+    w(f"      - series: '{labels()}'")
+    w(f"        values: '{fmt(stalled)}'")
+    w(f"      - series: '{labels(value='done')}'")
+    w("        # Flat, because a stalled archiver archives nothing. The rules do")
+    w("        # not read this; it is here so the input stays realistic.")
+    w(f"        values: '{fmt([148_000] * SAMPLES)}'")
+    w("    alert_rule_test:")
+    w("      # Backlog crosses 64 at t=5m, so `for: 8m` elapses at t=13m.")
+    w("      - eval_time: 12m")
+    w("        alertname: CNPGWalArchiveBacklogSurge")
+    w("        exp_alerts: []")
+    w("      - eval_time: 13m")
+    w("        alertname: CNPGWalArchiveBacklogSurge")
+    w("        exp_alerts:")
+    w("          # No `role` label: proof that `max by (namespace, pod)`")
+    w("          # aggregated it away. See mutations.txt:no-max-by.")
+    w("          - exp_labels:")
+    w("              severity: warning")
+    w(f"              namespace: {NS}")
+    w(f"              pod: {POD}")
+    w("            exp_annotations:")
+    w(f'              summary: "CNPG WAL archiving cannot keep up on {NS}/{POD}"')
+    w("              description: \"70 WAL segments (>1 GiB) have been queued for archiving for 8 minutes, which is longer than a healthy archiver needs to clear that much — from a backlog of 119 it takes under 5. Point-in-time recovery is degraded for everything after the last archived segment, and pg_wal grows against the PVC's free space until it clears. Start with the archiver: `kubectl -n home-assistant logs ha-pg-1 -c plugin-barman-cloud`, and check whether its ObjectStore sets `wal.maxParallel` — unset means one segment at a time, a ceiling of about 14 segments/min (REN-129).\"")
+    w("      # Backlog crosses 5 at t=5m, so `for: 15m` elapses at t=20m.")
+    w("      - eval_time: 19m")
+    w("        alertname: CNPGWalArchiveBacklog")
+    w("        exp_alerts: []")
+    w("      - eval_time: 20m")
+    w("        alertname: CNPGWalArchiveBacklog")
+    w("        exp_alerts:")
+    w("          - exp_labels:")
+    w("              severity: warning")
+    w(f"              namespace: {NS}")
+    w(f"              pod: {POD}")
+    w("            exp_annotations:")
+    w(f'              summary: "CNPG WAL archiving is falling behind on {NS}/{POD}"')
+    w("              description: \"70 WAL segments have been waiting to archive for over 15 minutes. Point-in-time recovery is broken for anything after the last archived segment, and pg_wal will grow until the PVC fills. Check the barman-cloud sidecar logs and the ObjectStore credentials: `kubectl -n home-assistant logs ha-pg-1 -c plugin-barman-cloud`.\"")
+    w("")
+
+    # ---- 10. low stall, below the surge threshold -------------------------
+    low = [0] * 5 + [12] * 35
+    w("  # ---------------------------------------------------------------------")
+    w("  # 10. THE STALL THE SURGE RULE CANNOT SEE: 12 segments, stuck. Under")
+    w("  # 1 GiB forever, so the surge rule is right to stay silent — and this")
+    w("  # is exactly the shape that makes the >5 rule load-bearing rather than")
+    w("  # a slower copy of it. Point-in-time recovery is just as broken at 12")
+    w("  # segments as at 70.")
+    w("  # ---------------------------------------------------------------------")
+    w("  - name: low stall at 12 segments — only the >5 rule may fire")
+    w("    interval: 1m")
+    w("    input_series:")
+    w(f"      - series: '{labels()}'")
+    w(f"        values: '{fmt(low)}'")
+    w("    alert_rule_test:")
+    w("      - eval_time: 20m")
+    w("        alertname: CNPGWalArchiveBacklog")
+    w("        exp_alerts:")
+    w("          - exp_labels:")
+    w("              severity: warning")
+    w(f"              namespace: {NS}")
+    w(f"              pod: {POD}")
+    w("            exp_annotations:")
+    w(f'              summary: "CNPG WAL archiving is falling behind on {NS}/{POD}"')
+    w("              description: \"12 WAL segments have been waiting to archive for over 15 minutes. Point-in-time recovery is broken for anything after the last archived segment, and pg_wal will grow until the PVC fills. Check the barman-cloud sidecar logs and the ObjectStore credentials: `kubectl -n home-assistant logs ha-pg-1 -c plugin-barman-cloud`.\"")
+    w("      # 34 minutes at 12 segments and the surge rule is still quiet.")
+    w("      - eval_time: 39m")
+    w("        alertname: CNPGWalArchiveBacklogSurge")
+    w("        exp_alerts: []")
+    w("")
+
+    # ---- 11. exactly at the threshold ------------------------------------
+    w("  # ---------------------------------------------------------------------")
+    w("  # 11. PINNED AT EXACTLY 5 for the whole window. `> 5` is strict, so")
+    w("  # this is silence. A rule written `>= 5` would page on it, and a rule")
+    w("  # that had lost its `value=\"ready\"` selector would page on the `done`")
+    w("  # counter regardless of what `ready` does — which is what the second")
+    w("  # series here is for.")
+    w("  # ---------------------------------------------------------------------")
+    w("  - name: pinned at exactly 5 — strict > means silence")
+    w("    interval: 1m")
+    w("    input_series:")
+    w(f"      - series: '{labels()}'")
+    w(f"        values: '{fmt([5] * SAMPLES)}'")
+    w(f"      - series: '{labels(value='done')}'")
+    w(f"        values: '{fmt(counter(148_000, 14))}'")
+    w("    alert_rule_test:")
+    w("      - eval_time: 39m")
+    w("        alertname: CNPGWalArchiveBacklog")
+    w("        exp_alerts: []")
+    w("      - eval_time: 39m")
+    w("        alertname: CNPGWalArchiveBacklogSurge")
+    w("        exp_alerts: []")
+    w("")
+
+    # ---- 12. role flip ---------------------------------------------------
+    w("  # ---------------------------------------------------------------------")
+    w("  # 12. FAILOVER DURING A STALL — the case `max by (namespace, pod)` was")
+    w("  # added for. The backlog starts at t=5m on the primary; at t=11m the")
+    w("  # instance is promoted and `role` changes, which is a NEW SERIES to")
+    w("  # Prometheus and restarts the `for` timer on the raw expression.")
+    w("  # Aggregating `role` away keeps one continuous series, so the page")
+    w("  # still lands 15m after the backlog started (t=20m) rather than 15m")
+    w("  # after the failover. Without `max by` this case is silent at t=20m —")
+    w("  # see mutations.txt:no-max-by.")
+    w("  # ---------------------------------------------------------------------")
+    w("  - name: role flip mid-stall — for timer must not restart")
+    w("    interval: 1m")
+    w("    input_series:")
+    w(f"      - series: '{labels(role='primary')}'")
+    w("        # t=0..10m, then this series ends.")
+    w(f"        values: '{fmt([0] * 5 + [70] * 6)}'")
+    w(f"      - series: '{labels(role='replica')}'")
+    w("        # Starts at t=11m. `_` is a gap, not a zero.")
+    w(f"        values: '{'_ ' * 11}{fmt([70] * 29)}'")
+    w("    alert_rule_test:")
+    w("      - eval_time: 19m")
+    w("        alertname: CNPGWalArchiveBacklog")
+    w("        exp_alerts: []")
+    w("      - eval_time: 20m")
+    w("        alertname: CNPGWalArchiveBacklog")
+    w("        exp_alerts:")
+    w("          - exp_labels:")
+    w("              severity: warning")
+    w(f"              namespace: {NS}")
+    w(f"              pod: {POD}")
+    w("            exp_annotations:")
+    w(f'              summary: "CNPG WAL archiving is falling behind on {NS}/{POD}"')
+    w("              description: \"70 WAL segments have been waiting to archive for over 15 minutes. Point-in-time recovery is broken for anything after the last archived segment, and pg_wal will grow until the PVC fills. Check the barman-cloud sidecar logs and the ObjectStore credentials: `kubectl -n home-assistant logs ha-pg-1 -c plugin-barman-cloud`.\"")
+    w("")
+
+    # ---- 13. one stalled pod among healthy ones --------------------------
+    w("  # ---------------------------------------------------------------------")
+    w("  # 13. ONE STALLED INSTANCE AMONG HEALTHY ONES. There are 12 CNPG")
+    w("  # instances in this cluster. `by (namespace, pod)` has to keep them")
+    w("  # apart: exactly one alert, naming the pod that is actually stuck, so")
+    w("  # the 04:00 reader is not sent to look at the wrong Postgres.")
+    w("  # ---------------------------------------------------------------------")
+    w("  - name: one stalled pod among healthy ones — exactly one alert")
+    w("    interval: 1m")
+    w("    input_series:")
+    w(f"      - series: '{labels(pod='ha-pg-1')}'")
+    w(f"        values: '{fmt([0] * 5 + [70] * 35)}'")
+    w(f"      - series: '{labels(pod='ha-pg-2', role='replica')}'")
+    w(f"        values: '{fmt([0] * SAMPLES)}'")
+    w(f"      - series: '{labels(pod='immich-pg-1', ns='immich')}'")
+    w(f"        values: '{fmt([0] * SAMPLES)}'")
+    w("    alert_rule_test:")
+    w("      - eval_time: 20m")
+    w("        alertname: CNPGWalArchiveBacklog")
+    w("        exp_alerts:")
+    w("          - exp_labels:")
+    w("              severity: warning")
+    w(f"              namespace: {NS}")
+    w("              pod: ha-pg-1")
+    w("            exp_annotations:")
+    w(f'              summary: "CNPG WAL archiving is falling behind on {NS}/ha-pg-1"')
+    w("              description: \"70 WAL segments have been waiting to archive for over 15 minutes. Point-in-time recovery is broken for anything after the last archived segment, and pg_wal will grow until the PVC fills. Check the barman-cloud sidecar logs and the ObjectStore credentials: `kubectl -n home-assistant logs ha-pg-1 -c plugin-barman-cloud`.\"")
+    w("")
+
+    # ---- 14. CNPGBackupFailed --------------------------------------------
+    w("  # ---------------------------------------------------------------------")
+    w("  # 14. THE EPOCH-ZERO GUARD on CNPGBackupFailed. The metric is a unix")
+    w("  # timestamp and reads 0 when no failure was ever recorded, so the rule")
+    w("  # carries a `> 0` guard to stop that default being read as \"failed in")
+    w("  # 1970\". Held at 0 for the whole window: silence. Nothing else in this")
+    w("  # suite exercises that rule, and a dropped guard would page on every")
+    w("  # instance in the cluster forever.")
+    w("  # ---------------------------------------------------------------------")
+    w("  - name: never-failed backup reads timestamp 0 — must not fire")
+    w("    interval: 1m")
+    w("    input_series:")
+    w(f"      - series: 'cnpg_collector_last_failed_backup_timestamp{{namespace=\"{NS}\",pod=\"{POD}\"}}'")
+    w(f"        values: '{fmt([0] * SAMPLES)}'")
+    w("    alert_rule_test:")
+    w("      - eval_time: 39m")
+    w("        alertname: CNPGBackupFailed")
+    w("        exp_alerts: []")
+
+    print("\n".join(L))
+
+    # Fidelity self-check on stderr so it is visible when regenerating but does
+    # not contaminate the suite on stdout. Longer than measured is tolerated and
+    # expected (see FIDELITY in the module docstring); SHORTER is not, because a
+    # night that clears the threshold sooner than the real one is a weaker
+    # silence test than the night it claims to stand for.
+    short = [(d, m, r) for d, m, r in checks if r < m]
+    if short:
+        print(
+            "FAIL: reconstruction is SHORTER above 5 than the measurement on "
+            + ", ".join(f"{d} measured {m} rebuilt {r}" for d, m, r in short)
+            + " — these nights are weaker silence tests than the real ones. "
+            "Refusing to vouch for the suite.",
+            file=sys.stderr,
+        )
+        return 1
+    long_ = [(d, m, r) for d, m, r in checks if r > m]
+    if long_:
+        print(
+            f"NOTE: {len(long_)} of {len(checks)} nights rebuild LONGER above 5 "
+            "than the published column (conservative direction): "
+            + ", ".join(f"{d} {m}->{r}" for d, m, r in long_),
+            file=sys.stderr,
+        )
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

kubernetes/apps/monitoring-rules/tests/mutations.txt

@@ -0,0 +1,72 @@
+# Negative controls for the promtool suites in this directory.
+#
+# A unit-test suite that passes proves the rules behave as asserted. It does not
+# prove the assertions are worth anything: a suite whose `rule_files` path does
+# not resolve, or whose traces are all zero, passes just as green. Both of those
+# happened while this suite was being written, and neither was visible from the
+# exit code.
+#
+# So scripts/ci/promtool-rules.sh runs a second phase. For each mutation below
+# it damages ONE value the rules depend on, re-runs every suite, and requires the
+# run to FAIL. A mutation the suite does not notice is a hole in the suite, and
+# CI reports it as such.
+#
+# FORMAT — three `|`-separated fields, no `|` inside any of them:
+#
+#   name | manifest path relative to the repo root | sed program
+#
+# The sed program is applied to the `spec:`-stripped copy in the mirrored tree,
+# so its indentation is two spaces shallower than the manifest on disk. Anchor
+# on `^[[:space:]]*` rather than a fixed indent. `#` in the first column is a
+# comment; blank lines are ignored.
+#
+# The script also requires each sed program to CHANGE the file. A mutation that
+# matches nothing is reported as a failure, not skipped — otherwise renaming a
+# rule or rewording an expression silently turns its control into a no-op and
+# the suite goes back to vouching for itself.
+#
+# `detected by` notes below are for the reader. The machine check is only "the
+# suite must fail"; nothing parses them.
+
+# The defect this whole exercise exists because of. The surge rule was written
+# `for: 2m` and reviewed at that value. Six of Iris's seven measured nights
+# accumulate 3-4 minutes above 64, so it fires on all six — two owner emails a
+# night, 04:16 and 04:22 local, for a backlog that clears itself.
+# Detected by: cases 1-7, the measured nights.
+surge-for-2m|kubernetes/apps/monitoring-rules/prometheusrule-cnpg-backup.yaml|s/^\([[:space:]]*\)for: 8m$/\1for: 2m/
+
+# The margin control, and the only reason case 8 is in the suite. At `for: 5m`
+# the surge rule is still silent on all seven REAL nights (worst is 4m pending)
+# and fires only on the adversarial envelope (5m pending). Without the envelope
+# case, `for: 8m` could be cut to 5m with the suite staying green.
+# Detected by: case 8, the adversarial envelope.
+surge-for-5m|kubernetes/apps/monitoring-rules/prometheusrule-cnpg-backup.yaml|s/^\([[:space:]]*\)for: 8m$/\1for: 5m/
+
+# The value REN-129 changed, in the direction it changed it from. At 30m the
+# load-bearing rule does not page until 35 minutes into a stall that started at
+# t=5m, so the assertion at t=20m sees nothing. This is the control that stops
+# the 15m being quietly walked back.
+# Detected by: cases 9, 10, 12, 13 — every case that expects a page.
+backlog-for-30m|kubernetes/apps/monitoring-rules/prometheusrule-cnpg-backup.yaml|s/^\([[:space:]]*\)for: 15m$/\1for: 30m/
+
+# Drop the aggregation, keeping the selector. `role` flips on failover; a
+# changed label is a new series to Prometheus and restarts the `for` timer, so
+# a backlog that persists across a promotion pages 15m after the promotion
+# instead of 15m after the backlog. Scoped to `expr:` lines because
+# `max by (namespace, pod)` also appears in this manifest's comments, where
+# rewriting it would mutate the documentation and nothing else.
+# Detected by: case 12, the role flip.
+no-max-by|kubernetes/apps/monitoring-rules/prometheusrule-cnpg-backup.yaml|/^[[:space:]]*expr:/s/max by (namespace, pod) (\(cnpg_collector_pg_wal_archive_status{value="ready"}\))/\1/
+
+# Drop the `value="ready"` selector. The same metric name also carries
+# `value="done"`, a monotonic counter in the hundreds of thousands, so both
+# rules would fire permanently on every CNPG instance in the cluster. The
+# measured-night cases carry that counter series in their input specifically so
+# this is detectable rather than a silent no-op.
+# Detected by: cases 1-8 and 11 — every case with a `done` series.
+no-value-selector|kubernetes/apps/monitoring-rules/prometheusrule-cnpg-backup.yaml|/^[[:space:]]*expr:/s/{value="ready"}//
+
+# Loosen the comparison on the load-bearing rule. `> 5` is strict; `>= 5` pages
+# on a backlog sitting at exactly five segments, which is 80 MiB and normal.
+# Detected by: case 11, pinned at exactly 5.
+backlog-ge-5|kubernetes/apps/monitoring-rules/prometheusrule-cnpg-backup.yaml|/^[[:space:]]*expr:/s/> 5$/>= 5/

kubernetes/apps/monitoring/helmrelease-prometheus.yaml

@@ -304,6 +304,10 @@ spec:
             - matchers:
                 - alertname = "KubeCPUOvercommit"
               receiver: 'null'
+            # Helper alert that only drives the info inhibit rule below.
+            - matchers:
+                - alertname = "InfoInhibitor"
+              receiver: 'null'
             # Flux notification-controller posts info-severity events
             # here (e.g. image-automation commits). Keep them visible
             # but don't repeat-spam every 12h on the same event.
@@ -313,6 +317,23 @@ spec:
               group_wait: 1m
               group_interval: 10m
               repeat_interval: 24h
+        # From kube-prometheus manifests/alertmanager-secret.yaml.
+        inhibit_rules:
+          - source_matchers:
+              - severity = "critical"
+            target_matchers:
+              - severity =~ "warning|info"
+            equal: ['namespace', 'alertname']
+          - source_matchers:
+              - severity = "warning"
+            target_matchers:
+              - severity = "info"
+            equal: ['namespace', 'alertname']
+          - source_matchers:
+              - alertname = "InfoInhibitor"
+            target_matchers:
+              - severity = "info"
+            equal: ['namespace']
         receivers:
           - name: email
             email_configs:
@@ -339,6 +360,285 @@ spec:
         limits:
           memory: 256Mi
 
+      # --- Flux per-object readiness (REN-102) ---------------------------------
+      #
+      # WHY THIS IS HERE AND NOT ON THE FLUX CONTROLLERS. `gotk_reconcile_condition`
+      # and `gotk_suspend_status` are still declared by fluxcd/pkg/runtime, but no
+      # controller populates them at the tags pinned in
+      # clusters/homelab/flux-system/gotk-components.yaml, and an unpopulated
+      # GaugeVec exports no series. Verified after the REN-97 scrape landed: the
+      # PodMonitor now yields 2,273 `gotk_*` series and
+      # `count({__name__=~"gotk_reconcile_condition|gotk_suspend_status"})` is still
+      # empty. Upstream moved per-object readiness to kube-state-metrics
+      # custom-resource-state, which is what this block turns on.
+      #
+      # DO NOT ADD `collectors: []` OR `--custom-resource-state-only=true`.
+      # Upstream's reference config (fluxcd/flux2-monitoring-example,
+      # monitoring/controllers/kube-prometheus-stack/kube-state-metrics-config.yaml)
+      # opens with exactly those two lines. Copying them replaces every built-in
+      # collector with the Flux ones: measured 2026-09-26, that would take
+      # `count({job="kube-state-metrics"})` from 27,792 series to ~183 and blind
+      # every `Kube*` alert in the chart. Both keys are deliberately absent below.
+      #
+      # ROLLS THE kube-state-metrics POD. Enabling customResourceState adds an arg
+      # and a volume to the Deployment, so the pod is replaced once on apply. That
+      # is the whole blast radius of this block — KSM only watches, it writes
+      # nothing, and no Flux object or workload is touched. During the roll (a few
+      # seconds) `kube_*` has a gap; `KubeStateMetricsDown` and `TargetDown` cover
+      # it if it does not come back.
+      #
+      # NOTE FOR LATER EDITS: the subchart puts no config checksum on the
+      # Deployment's pod template, so editing `config` below changes the ConfigMap
+      # WITHOUT rolling the pod, and KSM keeps serving the old config until it
+      # restarts for another reason. Any future change to this config needs a
+      # deliberate pod restart to take effect.
+      #
+      # RBAC: list + watch, nothing else. `get` is NOT needed — KSM drives
+      # reflectors, never point-reads. Also note the chart adds
+      # `customresourcedefinitions: [list, watch]` on its own via
+      # rbac.customResourceState.createClusterRoleRules (default true); KSM needs it
+      # to resolve the GVKs below. Both land in the existing
+      # kube-prometheus-stack-kube-state-metrics ClusterRole rather than a second
+      # binding to maintain. No write verb, no wildcard, no cluster-admin.
+      rbac:
+        extraRules:
+          - apiGroups:
+              - source.toolkit.fluxcd.io
+              - kustomize.toolkit.fluxcd.io
+              - helm.toolkit.fluxcd.io
+              - image.toolkit.fluxcd.io
+            # Trimmed to the kinds this cluster actually has. Omitted from
+            # upstream's list, each for a checked reason:
+            #   buckets, receivers        — zero objects exist in this repo.
+            #   helmcharts                — created by source-controller at
+            #     runtime, one per HelmRelease; a not-Ready HelmChart is always
+            #     mirrored by the HelmRelease that owns it, so alerting on both
+            #     is one failure reported twice.
+            #   alerts, providers         — notification.toolkit.fluxcd.io/v1beta3
+            #     Alert and Provider have NO status.conditions in the CRD at all
+            #     (checked in gotk-components.yaml), so there is no readiness to
+            #     report; upstream omits `ready` for them for the same reason. A
+            #     series carrying only `suspended` is not alertable.
+            resources:
+              - gitrepositories
+              - helmrepositories
+              - ocirepositories
+              - kustomizations
+              - helmreleases
+              - imagerepositories
+              - imagepolicies
+              - imageupdateautomations
+            verbs: ["list", "watch"]
+
+      customResourceState:
+        enabled: true
+        config:
+          spec:
+            resources:
+              # Label set is upstream's, minus the churning ones. Dropped
+              # deliberately: `revision` (status.artifact.revision /
+              # lastAppliedRevision — a git SHA, so every commit that reaches a
+              # Kustomization mints a NEW SERIES), and `url` / `endpoint`.
+              # Prometheus is at 84.5% of its 30GB retentionSize cap (REN-107);
+              # a label that changes several times a day across 33 Kustomizations
+              # is exactly what must not be added. Kept: `ready` and `suspended`
+              # (what the alerts read), `name` / `exported_namespace` (identity),
+              # `source_name` (the join key, see prometheusrule-flux.yaml), plus
+              # `image` and `tag` on the two image kinds, which are stable enough
+              # to ride the existing series.
+              #
+              # `exported_namespace` is upstream's own label name for this, not a
+              # Prometheus collision rename: the chart's KSM ServiceMonitor runs
+              # honorLabels, so `namespace` on a KSM series is already the
+              # OBJECT's namespace and nothing gets renamed. The rules use the
+              # same string because the config declares it.
+              #
+              # KSM adds `customresource_group`, `customresource_version` and
+              # `customresource_kind` to every custom-resource-state series by
+              # itself. The rules key off `customresource_kind`.
+              - groupVersionKind:
+                  group: kustomize.toolkit.fluxcd.io
+                  version: v1
+                  kind: Kustomization
+                metricNamePrefix: gotk
+                errorLogV: 10
+                metrics:
+                  - name: "resource_info"
+                    help: "The current state of a Flux Kustomization resource."
+                    each:
+                      type: Info
+                      info:
+                        labelsFromPath:
+                          name: [metadata, name]
+                    labelsFromPath:
+                      exported_namespace: [metadata, namespace]
+                      ready: [status, conditions, "[type=Ready]", status]
+                      suspended: [spec, suspend]
+                      source_name: [spec, sourceRef, name]
+              - groupVersionKind:
+                  group: helm.toolkit.fluxcd.io
+                  version: v2
+                  kind: HelmRelease
+                metricNamePrefix: gotk
+                errorLogV: 10
+                metrics:
+                  - name: "resource_info"
+                    help: "The current state of a Flux HelmRelease resource."
+                    each:
+                      type: Info
+                      info:
+                        labelsFromPath:
+                          name: [metadata, name]
+                    labelsFromPath:
+                      exported_namespace: [metadata, namespace]
+                      ready: [status, conditions, "[type=Ready]", status]
+                      suspended: [spec, suspend]
+              - groupVersionKind:
+                  group: source.toolkit.fluxcd.io
+                  version: v1
+                  kind: GitRepository
+                metricNamePrefix: gotk
+                errorLogV: 10
+                metrics:
+                  - name: "resource_info"
+                    help: "The current state of a Flux GitRepository resource."
+                    each:
+                      type: Info
+                      info:
+                        labelsFromPath:
+                          name: [metadata, name]
+                    labelsFromPath:
+                      exported_namespace: [metadata, namespace]
+                      ready: [status, conditions, "[type=Ready]", status]
+                      suspended: [spec, suspend]
+              - groupVersionKind:
+                  group: source.toolkit.fluxcd.io
+                  version: v1
+                  kind: HelmRepository
+                metricNamePrefix: gotk
+                errorLogV: 10
+                metrics:
+                  - name: "resource_info"
+                    help: "The current state of a Flux HelmRepository resource."
+                    each:
+                      type: Info
+                      info:
+                        labelsFromPath:
+                          name: [metadata, name]
+                    labelsFromPath:
+                      exported_namespace: [metadata, namespace]
+                      ready: [status, conditions, "[type=Ready]", status]
+                      suspended: [spec, suspend]
+              - groupVersionKind:
+                  group: source.toolkit.fluxcd.io
+                  version: v1
+                  kind: OCIRepository
+                metricNamePrefix: gotk
+                errorLogV: 10
+                metrics:
+                  - name: "resource_info"
+                    help: "The current state of a Flux OCIRepository resource."
+                    each:
+                      type: Info
+                      info:
+                        labelsFromPath:
+                          name: [metadata, name]
+                    labelsFromPath:
+                      exported_namespace: [metadata, namespace]
+                      ready: [status, conditions, "[type=Ready]", status]
+                      suspended: [spec, suspend]
+              - groupVersionKind:
+                  group: image.toolkit.fluxcd.io
+                  version: v1
+                  kind: ImageRepository
+                metricNamePrefix: gotk
+                errorLogV: 10
+                metrics:
+                  - name: "resource_info"
+                    help: "The current state of a Flux ImageRepository resource."
+                    each:
+                      type: Info
+                      info:
+                        labelsFromPath:
+                          name: [metadata, name]
+                    labelsFromPath:
+                      exported_namespace: [metadata, namespace]
+                      ready: [status, conditions, "[type=Ready]", status]
+                      suspended: [spec, suspend]
+                      image: [spec, image]
+                  # The repository-side half of the Ready-but-frozen signal. A
+                  # policy can hold Ready=True forever while its filter excludes
+                  # every new release, so readiness cannot see it; what can is
+                  # "the repository kept finding tags and the policy's answer
+                  # never moved". .status.lastScanResult.tagCount is a required
+                  # field on this CRD, so the series exists for every object.
+                  - name: "image_repository_tag_count"
+                    help: "Number of tags found by the ImageRepository's last scan."
+                    each:
+                      type: Gauge
+                      gauge:
+                        path: [status, lastScanResult]
+                        valueFrom: [tagCount]
+                    # `name` belongs HERE and not under `gauge`. KSM resolves
+                    # `gauge.labelsFromPath` RELATIVE TO `gauge.path`, so
+                    # [metadata, name] under there would be looked up inside
+                    # status.lastScanResult and silently yield no label. The
+                    # metric-level labelsFromPath resolves from the object root.
+                    labelsFromPath:
+                      name: [metadata, name]
+                      exported_namespace: [metadata, namespace]
+              - groupVersionKind:
+                  group: image.toolkit.fluxcd.io
+                  version: v1
+                  kind: ImagePolicy
+                metricNamePrefix: gotk
+                errorLogV: 10
+                metrics:
+                  - name: "resource_info"
+                    help: "The current state of a Flux ImagePolicy resource."
+                    each:
+                      type: Info
+                      info:
+                        labelsFromPath:
+                          name: [metadata, name]
+                    labelsFromPath:
+                      exported_namespace: [metadata, namespace]
+                      ready: [status, conditions, "[type=Ready]", status]
+                      suspended: [spec, suspend]
+                      # The resolved tag, as a label on the Info series that
+                      # already exists — so the policy-side half of the frozen
+                      # signal costs zero extra series at rest. It retires one
+                      # series and starts another only when the tag actually
+                      # changes: 191 automated bumps in 147 days across 34
+                      # policies is ~1.3 new series/day, which ages out with
+                      # retention.
+                      tag: [status, latestRef, tag]
+                      # The join key. `metadata.name` happens to equal
+                      # `spec.imageRepositoryRef.name` for all 34 pairs in this
+                      # repo today, but the REFERENCE is this field, and a policy
+                      # named differently from its repository would silently drop
+                      # out of a name-based join.
+                      source_name: [spec, imageRepositoryRef, name]
+              - groupVersionKind:
+                  group: image.toolkit.fluxcd.io
+                  version: v1
+                  kind: ImageUpdateAutomation
+                metricNamePrefix: gotk
+                errorLogV: 10
+                metrics:
+                  - name: "resource_info"
+                    help: "The current state of a Flux ImageUpdateAutomation resource."
+                    each:
+                      type: Info
+                      info:
+                        labelsFromPath:
+                          name: [metadata, name]
+                    labelsFromPath:
+                      exported_namespace: [metadata, namespace]
+                      ready: [status, conditions, "[type=Ready]", status]
+                      suspended: [spec, suspend]
+                      source_name: [spec, sourceRef, name]
+
     # Drop the apiserver's high-cardinality latency histograms.
     #
     # Measured 2026-08-20: Prometheus held 298,236 active series, of which

kubernetes/apps/navidrome/deployment.yaml

@@ -0,0 +1,129 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: navidrome-config
+  namespace: navidrome
+data:
+  TZ: "Europe/Berlin"
+  ND_LOGLEVEL: "info"
+  ND_SCANNER_SCHEDULE: "0 * * * *"
+  ND_ENABLEINSIGHTSCOLLECTOR: "false"
+  # Identity from the navidrome-gate outpost. Traefik strips this header on
+  # the ungated /rest and /share routes.
+  #
+  # ND_EXTAUTH_TRUSTEDSOURCES BELOW IS NOT A NARROWING. fd10:244::/48 plus
+  # 10.244.0.0/16 is the entire cluster pod CIDR, so navidrome's own
+  # reverse-proxy trust check accepts X-Authentik-Username from ANY pod.
+  # The CiliumNetworkPolicy is the only real boundary, and it is wider than
+  # "the outpost and Traefik" — default-deny-ingress in
+  # kubernetes/infrastructure/network-policies/navidrome-ns.yaml uses
+  # `endpointSelector: {}` and admits, on every port:
+  #   - the navidrome namespace, every port
+  #   - traefik, every port
+  #   - the navidrome-gate outpost, every port
+  #   - the lidarr-gate outpost, every port - reaches navidrome too
+  #   - host / remote-node, every port - the kubelet sources the HTTP probes
+  #     from the host netns, so this one can never be removed while the pod
+  #     has HTTP probes
+  #   - blackbox-exporter on :4533 and :8686 only
+  # The whole-monitoring-namespace admit that used to be in this list was
+  # narrowed to blackbox-exporter on those two ports in REN-28. Rules are
+  # cited by what they admit, not by line number, so this list cannot go
+  # stale the next time that file is edited.
+  # The value stays at the pod CIDR ON PURPOSE - see REN-28. The address
+  # navidrome sees is the outpost's POD IP, which is not stable across
+  # reschedules, so a narrower literal here fails open-ended and silently
+  # the next time that pod moves. Tightening the CNP is the control that
+  # would actually do work.
+  ND_EXTAUTH_USERHEADER: "X-Authentik-Username"
+  ND_EXTAUTH_TRUSTEDSOURCES: "fd10:244::/48,10.244.0.0/16"
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: navidrome
+  namespace: navidrome
+  labels:
+    app.kubernetes.io/name: navidrome
+  annotations:
+    configmap.reloader.stakater.com/reload: navidrome-config
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: navidrome
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: navidrome
+    spec:
+      enableServiceLinks: false
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1000
+        runAsGroup: 1000
+        fsGroup: 1000
+        fsGroupChangePolicy: OnRootMismatch
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: navidrome
+          image: deluan/navidrome:0.64.2 # {"$imagepolicy": "flux-system:navidrome"}
+          ports:
+            - name: http
+              containerPort: 4533
+          envFrom:
+            - configMapRef:
+                name: navidrome-config
+          volumeMounts:
+            - name: data
+              mountPath: /data
+            - name: music
+              mountPath: /music
+              readOnly: true
+          livenessProbe:
+            httpGet:
+              path: /ping
+              port: http
+            initialDelaySeconds: 30
+            periodSeconds: 30
+            timeoutSeconds: 10
+          readinessProbe:
+            httpGet:
+              path: /ping
+              port: http
+            initialDelaySeconds: 10
+            periodSeconds: 10
+            timeoutSeconds: 5
+          resources:
+            requests:
+              cpu: 100m
+              memory: 256Mi
+            limits:
+              memory: 1Gi
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop: ["ALL"]
+      volumes:
+        - name: data
+          persistentVolumeClaim:
+            claimName: navidrome-data
+        - name: music
+          persistentVolumeClaim:
+            claimName: navidrome-music-bulk
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: navidrome
+  namespace: navidrome
+spec:
+  selector:
+    app.kubernetes.io/name: navidrome
+  ports:
+    - name: http
+      port: 80
+      targetPort: http

kubernetes/apps/navidrome/gate-service.yaml

@@ -0,0 +1,13 @@
+# ExternalName mirror so the navidrome-namespace Ingress can backend to the
+# navidrome-gate outpost in the authentik namespace.
+apiVersion: v1
+kind: Service
+metadata:
+  name: ak-outpost-navidrome-gate
+  namespace: navidrome
+spec:
+  type: ExternalName
+  externalName: ak-outpost-navidrome-gate.authentik.svc.cluster.local
+  ports:
+    - port: 9000
+      targetPort: 9000

kubernetes/apps/navidrome/ingress.yaml

@@ -0,0 +1,69 @@
+# Web UI behind the navidrome-gate outpost.
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: navidrome
+  namespace: navidrome
+  labels:
+    dns.renner.uno/public: "true"
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-cluster
+spec:
+  ingressClassName: traefik
+  rules:
+    - host: navidrome.renner.uno
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: ak-outpost-navidrome-gate
+                port:
+                  number: 9000
+  tls:
+    - secretName: navidrome-tls
+      hosts:
+        - navidrome.renner.uno
+---
+apiVersion: traefik.io/v1alpha1
+kind: Middleware
+metadata:
+  name: navidrome-strip-identity
+  namespace: navidrome
+spec:
+  headers:
+    customRequestHeaders:
+      X-Authentik-Username: ""
+---
+# Subsonic API and public shares bypass the gate; clients authenticate with
+# their Navidrome password.
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: navidrome-subsonic
+  namespace: navidrome
+  labels:
+    dns.renner.uno/public: "true"
+  annotations:
+    traefik.ingress.kubernetes.io/router.middlewares: navidrome-navidrome-strip-identity@kubernetescrd
+spec:
+  ingressClassName: traefik
+  rules:
+    - host: navidrome.renner.uno
+      http:
+        paths:
+          - path: /rest
+            pathType: Prefix
+            backend:
+              service:
+                name: navidrome
+                port:
+                  number: 80
+          - path: /share
+            pathType: Prefix
+            backend:
+              service:
+                name: navidrome
+                port:
+                  number: 80

kubernetes/apps/navidrome/kustomization.yaml

@@ -0,0 +1,8 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - pvcs.yaml
+  - deployment.yaml
+  - gate-service.yaml
+  - ingress.yaml
+  - lidarr.yaml

kubernetes/apps/navidrome/lidarr.yaml

@@ -0,0 +1,224 @@
+# Lidarr as a read-mostly "what is missing" view over the Navidrome library.
+# No download client or indexer is configured.
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: lidarr-config
+  namespace: navidrome
+  annotations:
+    kustomize.toolkit.fluxcd.io/prune: disabled
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: ceph-block
+  resources:
+    requests:
+      storage: 5Gi
+---
+# Applied on every pod start, before the root folder exists: no renames, no tag
+# writes, no deletes. Adding /music as root folder starts Lidarr's library import.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: lidarr-reconcile
+  namespace: navidrome
+data:
+  reconcile.sh: |
+    set -eu
+    API=http://localhost:8686/api/v1
+    until curl -sf http://localhost:8686/ping >/dev/null; do sleep 5; done
+    KEY=$(sed -n 's:.*<ApiKey>\(.*\)</ApiKey>.*:\1:p' /config/config.xml)
+    if [ -z "$KEY" ]; then
+      echo "FATAL: no ApiKey in /config/config.xml" >&2
+      exit 1
+    fi
+    H="X-Api-Key: $KEY"
+
+    put() {
+      cur=$(curl -sf -H "$H" "$API/$1")
+      id=$(printf '%s' "$cur" | jq -r '.id')
+      printf '%s' "$cur" | jq -c "$2" \
+        | curl -sf -X PUT -H "$H" -H 'Content-Type: application/json' -d @- "$API/$1/$id" >/dev/null
+      echo "reconciled $1"
+    }
+
+    put config/naming '.renameTracks = false'
+    put config/metadataprovider '.writeAudioTags = "no" | .scrubAudioTags = false | .embedCoverArt = false'
+    put config/mediamanagement '.deleteEmptyFolders = false | .createEmptyArtistFolders = false
+      | .recycleBin = "" | .setPermissionsLinux = false | .importExtraFiles = false
+      | .skipFreeSpaceCheckWhenImporting = true'
+
+    roots=$(curl -sf -H "$H" "$API/rootfolder" | jq '[.[] | select(.path == "/music")] | length')
+    if [ "$roots" = "0" ]; then
+      qp=$(curl -sf -H "$H" "$API/qualityprofile" | jq '.[0].id')
+      mp=$(curl -sf -H "$H" "$API/metadataprofile" | jq '([.[] | select(.name == "Standard")][0].id) // .[0].id')
+      jq -nc --argjson qp "$qp" --argjson mp "$mp" '{
+          name: "Music", path: "/music",
+          defaultQualityProfileId: $qp, defaultMetadataProfileId: $mp,
+          defaultMonitorOption: "all", defaultNewItemMonitorOption: "all",
+          defaultTags: []
+        }' \
+        | curl -sf -X POST -H "$H" -H 'Content-Type: application/json' -d @- "$API/rootfolder" >/dev/null
+      echo "added root folder /music"
+    fi
+
+    echo "reconcile done"
+    while true; do sleep 3600; done
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: lidarr
+  namespace: navidrome
+  labels:
+    app.kubernetes.io/name: lidarr
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: lidarr
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: lidarr
+    spec:
+      enableServiceLinks: false
+      securityContext:
+        fsGroup: 1000
+        fsGroupChangePolicy: OnRootMismatch
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: lidarr
+          image: lscr.io/linuxserver/lidarr:3.1.0.4875-ls42 # {"$imagepolicy": "flux-system:lidarr"}
+          env:
+            - name: PUID
+              value: "1000"
+            - name: PGID
+              value: "1000"
+            - name: TZ
+              value: Europe/Berlin
+            # Identity is enforced by the lidarr-gate outpost.
+            - name: LIDARR__AUTH__METHOD
+              value: External
+          ports:
+            - name: http
+              containerPort: 8686
+          volumeMounts:
+            - name: config
+              mountPath: /config
+            - name: music
+              mountPath: /music
+          livenessProbe:
+            httpGet:
+              path: /ping
+              port: http
+            initialDelaySeconds: 60
+            periodSeconds: 30
+            timeoutSeconds: 10
+          readinessProbe:
+            httpGet:
+              path: /ping
+              port: http
+            initialDelaySeconds: 20
+            periodSeconds: 10
+            timeoutSeconds: 5
+          resources:
+            requests:
+              cpu: 100m
+              memory: 512Mi
+            limits:
+              memory: 4Gi
+          securityContext:
+            runAsNonRoot: false
+            allowPrivilegeEscalation: true
+            capabilities:
+              drop: ["ALL"]
+              add: ["CHOWN", "DAC_OVERRIDE", "FOWNER", "SETGID", "SETUID"]
+        - name: reconcile
+          image: lscr.io/linuxserver/lidarr:3.1.0.4875-ls42 # {"$imagepolicy": "flux-system:lidarr"}
+          command: ["sh", "/scripts/reconcile.sh"]
+          volumeMounts:
+            - name: config
+              mountPath: /config
+              readOnly: true
+            - name: reconcile
+              mountPath: /scripts
+              readOnly: true
+          resources:
+            requests:
+              cpu: 10m
+              memory: 16Mi
+            limits:
+              memory: 64Mi
+          securityContext:
+            runAsNonRoot: true
+            runAsUser: 1000
+            runAsGroup: 1000
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop: ["ALL"]
+      volumes:
+        - name: config
+          persistentVolumeClaim:
+            claimName: lidarr-config
+        - name: music
+          persistentVolumeClaim:
+            claimName: navidrome-music-bulk
+        - name: reconcile
+          configMap:
+            name: lidarr-reconcile
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: lidarr
+  namespace: navidrome
+spec:
+  selector:
+    app.kubernetes.io/name: lidarr
+  ports:
+    - name: http
+      port: 80
+      targetPort: http
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: ak-outpost-lidarr-gate
+  namespace: navidrome
+spec:
+  type: ExternalName
+  externalName: ak-outpost-lidarr-gate.authentik.svc.cluster.local
+  ports:
+    - port: 9000
+      targetPort: 9000
+---
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: lidarr
+  namespace: navidrome
+  labels:
+    dns.renner.uno/public: "true"
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-cluster
+spec:
+  ingressClassName: traefik
+  rules:
+    - host: lidarr.renner.uno
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: ak-outpost-lidarr-gate
+                port:
+                  number: 9000
+  tls:
+    - secretName: lidarr-tls
+      hosts:
+        - lidarr.renner.uno

kubernetes/apps/navidrome/pvcs.yaml

@@ -0,0 +1,31 @@
+# SQLite database + artwork/transcode cache; single writer.
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: navidrome-data
+  namespace: navidrome
+  annotations:
+    kustomize.toolkit.fluxcd.io/prune: disabled
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: ceph-block
+  resources:
+    requests:
+      storage: 10Gi
+---
+# Music library on the EC bulk pool.
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: navidrome-music-bulk
+  namespace: navidrome
+  annotations:
+    kustomize.toolkit.fluxcd.io/prune: disabled
+spec:
+  accessModes:
+    - ReadWriteMany
+  storageClassName: ceph-filesystem-bulk
+  resources:
+    requests:
+      storage: 300Gi

kubernetes/apps/paperclip/cluster-view-rbac.yaml

@@ -0,0 +1,97 @@
+---
+# Cluster-wide READ for the agents in the paperclip pod (REN-111, decided on
+# REN-52 2026-09-25 by the owner, option B).
+#
+# WHAT THIS IS
+#
+# One ClusterRoleBinding. It binds the Kubernetes built-in `view` ClusterRole to
+# the ServiceAccount that REN-36 created in `../paperclip/prometheus-read-rbac.yaml`.
+# No new ServiceAccount, no new ClusterRole, no write verb anywhere. `view` is
+# maintained upstream and holds get/list/watch only; revocation is deleting this
+# one object.
+#
+# WHY IT IS A SEPARATE FILE
+#
+# `prometheus-read-rbac.yaml` is deliberately narrow and its header argues for
+# staying that way. This grant is the opposite in character — cluster-wide,
+# every namespace — so it is kept in its own file and its own kustomization
+# entry. Reverting it is deleting this file and one line from
+# `kustomization.yaml`; nothing in the REN-36 path is touched, and the token
+# falls back to `get services/proxy` on two named Services.
+#
+# WHY THE OWNER WANTED IT
+#
+# REN-108 part B had to validate a CiliumNetworkPolicy allowlist whose selector
+# was `app.kubernetes.io/name: prometheus-blackbox-exporter`, and could not:
+# kube-state-metrics here does not export `kube_pod_labels`, and
+# blackbox-exporter is not a scrape target so Prometheus has no
+# `__meta_kubernetes_pod_label_*` for it either. A CNP selector that matches no
+# pod allows nothing and reports no error, so the proposal shipped unverified.
+# `list pods` closes that, and the same gap recurs on every netpol or
+# scrape-selector change.
+#
+# WHAT IT COSTS — ACCEPTED ON REN-52, NOT DISCOVERED HERE
+#
+#   * EVERY process in the paperclip pod can read the mounted token, so this is
+#     a grant to every agent, not to one. REN-36 was approved on the premise
+#     that "the token is only worth what this Role grants"; this file changes
+#     that arithmetic, and the owner priced it knowingly.
+#   * `view` reads ConfigMaps in EVERY namespace. This repo keeps application
+#     configuration in ConfigMaps. Chosen over Hermes' narrower alternative.
+#   * `view` grants `pods/log`, which surfaces anything an application prints,
+#     including credentials it prints by mistake.
+#
+# WHAT IT DOES NOT GRANT
+#
+#   * Secrets. `view` excludes them. "Secrets, metadata only" was the original
+#     ask and is NOT expressible in Kubernetes RBAC — the authorizer has no
+#     field-level verb — so that part of the ask is dead. Do not re-add Secrets
+#     here to approximate it.
+#   * Any write verb, on anything.
+#   * ClusterRoles and RoleBindings: `view` excludes the whole
+#     `rbac.authorization.k8s.io` group, on purpose, so a holder cannot read the
+#     shape of its own cage or find a Role to escalate through.
+#
+# FLUX CRDs — COVERED BY AGGREGATION, NO EXTRA ClusterRole NEEDED
+#
+# REN-52 pre-approved adding a read-only ClusterRole for the Flux CRDs *if*
+# aggregation did not already cover them. It does, so none is added:
+# `flux-view-flux-system` carries `rbac.authorization.k8s.io/aggregate-to-view:
+# "true"` and lists get/list/watch on `*` across all six API groups —
+# `../../clusters/homelab/flux-system/gotk-components.yaml:232-254`. The
+# aggregation controller merges those rules into `view` itself, so this one
+# binding picks up kustomizations, helmreleases, gitrepositories, imagepolicies,
+# imagerepositories and imageupdateautomations. Read from git, not from a live
+# ClusterRole — `view` cannot read ClusterRoles, so it is unverifiable from
+# inside the pod. The post-merge SelfSubjectAccessReview on REN-111 proves
+# whether aggregation actually fired in the cluster; if `list kustomizations`
+# comes back `allowed: false` there, the pre-approved ClusterRole is the fix.
+#
+# PRODUCTION
+#
+# Adding a ClusterRoleBinding mutates no workload, restarts no Pod and moves no
+# traffic. RBAC is evaluated per request, so the grant takes effect on the
+# already-mounted token with no remount and no rollout. That sequencing was the
+# point of REN-52 putting this after REN-36: the paperclip Deployment restart
+# was paid once, by REN-36, and is already paid.
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: paperclip-cluster-view
+  labels:
+    app.kubernetes.io/name: paperclip
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  # Built-in, shipped by kube-controller-manager's default RBAC bootstrap. Not
+  # defined in this repo and must not be: it is auto-reconciled, so a local copy
+  # would be overwritten on every apiserver start.
+  name: view
+subjects:
+  - kind: ServiceAccount
+    name: paperclip-prometheus-read
+    # A ClusterRoleBinding has no namespace of its own, so this is the only
+    # place the subject's namespace can be stated. Omit it and the binding
+    # resolves to no subject and fails closed, looking identical to a missing
+    # grant.
+    namespace: paperclip

kubernetes/apps/paperclip/cluster.yaml

@@ -0,0 +1,43 @@
+apiVersion: postgresql.cnpg.io/v1
+kind: Cluster
+metadata:
+  name: pclp-pg
+  namespace: paperclip
+  annotations:
+    kustomize.toolkit.fluxcd.io/prune: disabled # data guard
+spec:
+  instances: 2
+  smartShutdownTimeout: 10
+  stopDelay: 60
+  # PG17: upstream's external Postgres target (docker/docker-compose.yml).
+  imageName: ghcr.io/cloudnative-pg/postgresql:17
+  storage:
+    storageClass: ceph-block
+    size: 10Gi
+  plugins:
+    - name: barman-cloud.cloudnative-pg.io
+      isWALArchiver: true
+      parameters:
+        barmanObjectName: pclp-pg-store
+  bootstrap:
+    initdb:
+      database: paperclip
+      owner: paperclip
+  postgresql:
+    parameters:
+      shared_buffers: "256MB"
+      max_connections: "200"
+      checkpoint_timeout: "15min"
+      max_wal_size: "2GB"
+      wal_compression: "zstd"
+      archive_timeout: "30min"
+    synchronous:
+      method: any
+      number: 1
+      dataDurability: preferred
+  resources:
+    requests:
+      cpu: 200m
+      memory: 512Mi
+    limits:
+      memory: 1Gi

kubernetes/apps/paperclip/externalsecret.yaml

@@ -0,0 +1,15 @@
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: paperclip-secrets
+  namespace: paperclip
+spec:
+  refreshInterval: 10m
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: openbao
+  target:
+    name: paperclip-secrets
+  dataFrom:
+    - extract:
+        key: apps/paperclip

kubernetes/apps/paperclip/gate-service.yaml

@@ -0,0 +1,12 @@
+# ExternalName mirror so the Ingress can backend the paperclip-gate outpost in the authentik namespace.
+apiVersion: v1
+kind: Service
+metadata:
+  name: ak-outpost-paperclip-gate
+  namespace: paperclip
+spec:
+  type: ExternalName
+  externalName: ak-outpost-paperclip-gate.authentik.svc.cluster.local
+  ports:
+    - port: 9000
+      targetPort: 9000

kubernetes/apps/paperclip/hetzner-creds-externalsecret.yaml

@@ -0,0 +1,17 @@
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: hetzner-s3-creds
+  namespace: paperclip
+  annotations:
+    kustomize.toolkit.fluxcd.io/prune: disabled # barman S3 creds
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: openbao
+  target:
+    name: hetzner-s3-creds
+  dataFrom:
+    - extract:
+        key: infrastructure/hetzner-s3

kubernetes/apps/paperclip/ingress.yaml

@@ -0,0 +1,27 @@
+# LAN-only (no dns.renner.uno/public label); backends the gate outpost, never the Service.
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: paperclip
+  namespace: paperclip
+  labels:
+    dns.renner.uno/public: "true"
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-cluster
+spec:
+  ingressClassName: traefik
+  tls:
+    - hosts:
+        - paperclip.renner.uno
+      secretName: paperclip-tls
+  rules:
+    - host: paperclip.renner.uno
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: ak-outpost-paperclip-gate
+                port:
+                  number: 9000

kubernetes/apps/paperclip/kustomization.yaml

@@ -0,0 +1,15 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - externalsecret.yaml
+  - hetzner-creds-externalsecret.yaml
+  - cluster.yaml
+  - objectstore.yaml
+  - scheduled-backup.yaml
+  - pvc.yaml
+  - prometheus-read-rbac.yaml
+  - cluster-view-rbac.yaml
+  - paperclip.yaml
+  - gate-service.yaml
+  - ingress.yaml
+  - network-policy.yaml

kubernetes/apps/paperclip/network-policy.yaml

@@ -0,0 +1,78 @@
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: paperclip-gate-ingress
+  namespace: paperclip
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: paperclip
+  ingress:
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": authentik
+            goauthentik.io/outpost-name: paperclip-gate
+      toPorts:
+        - ports:
+            - port: "3100"
+              protocol: TCP
+---
+# Covers the server and every agent it spawns (same pod).
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: paperclip-egress
+  namespace: paperclip
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: paperclip
+  egress:
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": paperclip
+            cnpg.io/cluster: pclp-pg
+      toPorts:
+        - ports:
+            - port: "5432"
+              protocol: TCP
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": kube-system
+            k8s-app: kube-dns
+      toPorts:
+        - ports:
+            - port: "53"
+              protocol: UDP
+            - port: "53"
+              protocol: TCP
+    # Gitea, Woodpecker, HA, Loki via their *.renner.uno names.
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": ingress
+            app.kubernetes.io/name: traefik
+      toPorts:
+        - ports:
+            - port: "8443"
+              protocol: TCP
+    # Notes wiki API, authenticated by the paperclip API key.
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": ai
+            app.kubernetes.io/name: wiki
+      toPorts:
+        - ports:
+            - port: "8080"
+              protocol: TCP
+    - toEntities:
+        - kube-apiserver
+      toPorts:
+        - ports:
+            - port: "6443"
+              protocol: TCP
+    - toEntities:
+        - world
+      toPorts:
+        - ports:
+            - port: "443"
+              protocol: TCP

kubernetes/apps/paperclip/objectstore.yaml

@@ -0,0 +1,26 @@
+apiVersion: barmancloud.cnpg.io/v1
+kind: ObjectStore
+metadata:
+  name: pclp-pg-store
+  namespace: paperclip
+  annotations:
+    kustomize.toolkit.fluxcd.io/prune: disabled # backup target
+spec:
+  configuration:
+    destinationPath: s3://renner-uno-k8s-cnpg/pclp-pg
+    endpointURL: https://fsn1.your-objectstorage.com
+    s3Credentials:
+      accessKeyId:
+        name: hetzner-s3-creds
+        key: ACCESS_KEY_ID
+      secretAccessKey:
+        name: hetzner-s3-creds
+        key: ACCESS_SECRET_KEY
+      region:
+        name: hetzner-s3-creds
+        key: REGION
+    wal:
+      compression: gzip
+    data:
+      compression: gzip
+  retentionPolicy: "30d"

kubernetes/apps/paperclip/paperclip.yaml

@@ -0,0 +1,128 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: paperclip
+  namespace: paperclip
+  labels:
+    app.kubernetes.io/name: paperclip
+  annotations:
+    secret.reloader.stakater.com/reload: paperclip-secrets
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: paperclip
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: paperclip
+    spec:
+      enableServiceLinks: false
+      # Every agent shares this filesystem, so every agent can read this token.
+      # It is mounted anyway because it is worth exactly one thing: `get` on the
+      # Prometheus Service proxy, for monitoring reads (REN-36). See
+      # prometheus-read-rbac.yaml. Bots still bring their own Gitea and cluster
+      # credentials; this is not a general-purpose cluster credential.
+      serviceAccountName: paperclip-prometheus-read
+      automountServiceAccountToken: true
+      securityContext:
+        fsGroup: 1000
+        fsGroupChangePolicy: OnRootMismatch
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: paperclip
+          image: ghcr.io/paperclipai/paperclip:nightly@sha256:9222efbf11863160669e13d5559e6a1be881886dd1758650b74c3993dc3ef2a6 # {"$imagepolicy": "flux-system:paperclip"}
+          ports:
+            - name: http
+              containerPort: 3100
+          env:
+            - name: DATABASE_URL
+              valueFrom:
+                secretKeyRef:
+                  name: pclp-pg-app
+                  key: uri
+            - name: HOST
+              value: "0.0.0.0"
+            - name: PORT
+              value: "3100"
+            - name: PAPERCLIP_DEPLOYMENT_MODE
+              value: authenticated
+            - name: PAPERCLIP_DEPLOYMENT_EXPOSURE
+              value: private
+            - name: PAPERCLIP_PUBLIC_URL
+              value: https://paperclip.renner.uno
+            # Agents reach the API via the Service, not through the Authentik gate.
+            - name: PAPERCLIP_API_URL
+              value: http://paperclip.paperclip.svc.cluster.local:3100
+            - name: PAPERCLIP_ALLOWED_HOSTNAMES
+              value: paperclip.paperclip.svc.cluster.local
+            - name: PAPERCLIP_AUTH_DISABLE_SIGN_UP
+              value: "true"
+            - name: PAPERCLIP_TELEMETRY_DISABLED
+              value: "1"
+          envFrom:
+            - secretRef:
+                name: paperclip-secrets
+          volumeMounts:
+            - name: data
+              mountPath: /paperclip
+          # The private-hostname guard 403s any Host but loopback and the public URL; the kubelet sends the pod IP.
+          startupProbe:
+            httpGet:
+              path: /api/health
+              port: http
+              httpHeaders:
+                - name: Host
+                  value: localhost
+            periodSeconds: 10
+            timeoutSeconds: 5
+            failureThreshold: 60
+          # /api/health runs SELECT 1; liveness stays DB-independent so a Postgres outage doesn't restart the server.
+          livenessProbe:
+            tcpSocket:
+              port: http
+            periodSeconds: 30
+            timeoutSeconds: 10
+          readinessProbe:
+            httpGet:
+              path: /api/health
+              port: http
+              httpHeaders:
+                - name: Host
+                  value: localhost
+            periodSeconds: 10
+            timeoutSeconds: 5
+          # Every concurrent agent (Claude Code + MCP helpers) runs inside this container.
+          resources:
+            requests:
+              cpu: 250m
+              memory: 2Gi
+            limits:
+              memory: 32Gi
+          # Entrypoint remaps the node user and chowns /paperclip as root, then drops to it via gosu.
+          securityContext:
+            runAsNonRoot: false
+            allowPrivilegeEscalation: true
+            capabilities:
+              drop: ["ALL"]
+              add: ["CHOWN", "DAC_OVERRIDE", "FOWNER", "SETGID", "SETUID"]
+      volumes:
+        - name: data
+          persistentVolumeClaim:
+            claimName: paperclip-data
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: paperclip
+  namespace: paperclip
+spec:
+  selector:
+    app.kubernetes.io/name: paperclip
+  ports:
+    - name: http
+      port: 3100
+      targetPort: http

kubernetes/apps/paperclip/prometheus-read-rbac.yaml

@@ -0,0 +1,101 @@
+---
+# A read-only Prometheus path for the agents in the paperclip pod (REN-36).
+#
+# WHY THE APISERVER PROXY AND NOT A NETWORK RULE
+#
+# Prometheus runs with `--web.enable-lifecycle` (the chart leaves
+# `reloadStrategy` unset, so the operator defaults it to HTTP and appends the
+# flag), and :9090 carries no auth of its own. That puts `POST /-/quit` and
+# `POST /-/reload` on the same TCP port as `GET /api/v1/query`. An L4 allow on
+# 9090 cannot tell them apart, so it would hand this pod a kill switch for
+# cluster monitoring.
+#
+# The apiserver can tell them apart: it maps the HTTP method of a service-proxy
+# request onto an RBAC verb — GET->get, POST->create, PUT->update. A Role
+# holding only `get` therefore refuses `/-/quit` and `/-/reload` in the
+# authorizer, before the request is ever forwarded to Prometheus. Read-only by
+# construction, and `--web.enable-lifecycle` stops mattering.
+#
+# Shape copied from `../monitoring/ci-diagnostics-rbac.yaml` (the
+# `ci-diagnostics-service-proxy` Role), which has run this same path for the
+# alert-monitor since 2026-07-16.
+#
+# ONE VERB, NAMED SERVICES ONLY — ON PURPOSE
+#
+# Every process in the paperclip pod can read the mounted ServiceAccount token,
+# so the token is worth the sum of everything bound to this ServiceAccount. This
+# Role's own share of that is `get` on the two Services named below and nothing
+# else. Widening it is a new card and a new review, not an edit here — the
+# Alertmanager name was added that way (REN-95, for read-only alert-delivery
+# checks; parent REN-94).
+#
+# THIS IS NO LONGER THE ONLY GRANT ON THIS ServiceAccount
+#
+# `cluster-view-rbac.yaml` in this directory binds the built-in `view`
+# ClusterRole to the same ServiceAccount, cluster-wide (REN-111, owner-approved
+# on REN-52). So reading this file alone now understates what the token can do:
+# it also carries get/list/watch on most namespaced resources in every
+# namespace, including ConfigMaps and `pods/log`. Still no Secrets and still no
+# write verb. Read both files.
+#
+# `resourceNames` is matched verbatim by the authorizer, not as a selector:
+# `svc`, `svc:9090` and `http:svc:9090` are three different strings. Only the
+# port-qualified forms are granted, so the only URLs that work are
+#   /api/v1/namespaces/monitoring/services/kube-prometheus-stack-prometheus:9090/proxy/...
+#   /api/v1/namespaces/monitoring/services/kube-prometheus-stack-alertmanager:9093/proxy/...
+# A 403 on some other form is the grant working. Do not fix it by adding a
+# string.
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: paperclip-prometheus-read
+  namespace: paperclip
+  labels:
+    app.kubernetes.io/name: paperclip
+# Named for the only thing it is worth. If a future change gives this account a
+# second grant, the name is a lie and the review should say so. Whether the
+# token is mounted is decided on the pod (`paperclip.yaml`), not here.
+#
+# REN-95 stretched the name: it now also proxies to Alertmanager. REN-111 broke
+# it outright — this account now also holds cluster-wide `view`, which is not
+# Prometheus and not read of one thing. Kept as-is deliberately, and REN-52
+# ruled it stays: `serviceAccountName` is in the pod template, so renaming it
+# would restart the paperclip Deployment and evict running agents, which is a
+# larger production cost than a misleading name. Read the name as "the paperclip
+# pod's read-only cluster identity", and read this file plus
+# `cluster-view-rbac.yaml` for the truth.
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: paperclip-prometheus-read
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/name: paperclip
+rules:
+  - apiGroups: [""]
+    resources: ["services/proxy"]
+    verbs: ["get"]
+    resourceNames:
+      - "kube-prometheus-stack-prometheus:9090"
+      - "kube-prometheus-stack-alertmanager:9093"
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: paperclip-prometheus-read
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/name: paperclip
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: paperclip-prometheus-read
+# The Role and this binding live in `monitoring`; the subject lives in
+# `paperclip`, so `namespace:` here is load-bearing. Omit it and the binding
+# silently resolves to a nonexistent `monitoring/paperclip-prometheus-read`,
+# which fails closed with a 403 indistinguishable from a wrong resourceName.
+subjects:
+  - kind: ServiceAccount
+    name: paperclip-prometheus-read
+    namespace: paperclip

kubernetes/apps/paperclip/pvc.yaml

@@ -0,0 +1,13 @@
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: paperclip-data
+  namespace: paperclip
+  annotations:
+    kustomize.toolkit.fluxcd.io/prune: disabled # data guard
+spec:
+  accessModes: [ReadWriteOnce]
+  storageClassName: ceph-block
+  resources:
+    requests:
+      storage: 20Gi

kubernetes/apps/paperclip/scheduled-backup.yaml

@@ -0,0 +1,16 @@
+apiVersion: postgresql.cnpg.io/v1
+kind: ScheduledBackup
+metadata:
+  name: pclp-pg-daily
+  namespace: paperclip
+  annotations:
+    kustomize.toolkit.fluxcd.io/prune: disabled # backup schedule
+spec:
+  schedule: "0 45 2 * * *"
+  immediate: true
+  backupOwnerReference: self
+  cluster:
+    name: pclp-pg
+  method: plugin
+  pluginConfiguration:
+    name: barman-cloud.cloudnative-pg.io

kubernetes/apps/paperless/network-policy.yaml

@@ -1,9 +1,23 @@
 ---
-# Trust boundary: paperless-gate (the dedicated Authentik proxy outpost
-# in the authentik namespace). Header-based remote-user auth means
-# paperless trusts whatever fills X-Authentik-Username unconditionally,
-# so any pod that can talk directly to paperless's HTTP listener can
-# claim any username. Lock the ingress side to the gate outpost only.
+# Intended trust boundary: paperless-gate (the dedicated Authentik proxy
+# outpost in the authentik namespace). Header-based remote-user auth means
+# paperless trusts whatever fills X-Authentik-Username unconditionally, so
+# any pod that can talk directly to paperless's HTTP listener can claim any
+# username.
+#
+# WHAT IS ENFORCED IS WIDER THAN THIS RULE. default-deny-ingress in
+# kubernetes/infrastructure/network-policies/paperless-ns.yaml uses
+# `endpointSelector: {}`, so it selects this pod as well, and Cilium unions
+# the allow rules of every CNP selecting an endpoint (the same mechanism
+# described below). The effective ingress set is this outpost PLUS, on
+# every port:
+#   - the paperless namespace   (paperless-ns.yaml:9-11)
+#   - the database namespace    (paperless-ns.yaml:31-33)
+#   - the monitoring namespace  (paperless-ns.yaml:34-36)
+#   - host / remote-node        (paperless-ns.yaml:37-39)
+# So any pod in database or monitoring can reach this HTTP listener
+# directly and set X-Authentik-Username to any value. Do not read this
+# rule as the whole boundary. Narrowing is tracked in REN-28.
 #
 # This is a CiliumNetworkPolicy (not a standard NetworkPolicy) so it
 # composes additively with the cluster's baseline CNP

kubernetes/apps/trmnl/terminus.yaml

@@ -134,7 +134,7 @@ spec:
         # is a device-bricking class of risk. Bumps are a reviewed, manual
         # act — see kubernetes/infrastructure/image-automation/README.md.
         - name: terminus
-          image: ghcr.io/usetrmnl/terminus:0.74.0 # {"$imagepolicy": "flux-system:terminus"}
+          image: ghcr.io/usetrmnl/terminus:0.75.0 # {"$imagepolicy": "flux-system:terminus"}
           ports:
             - name: http
               containerPort: 2345
@@ -197,7 +197,7 @@ spec:
             capabilities:
               drop: ["ALL"]
         - name: sidekiq
-          image: ghcr.io/usetrmnl/terminus:0.74.0 # {"$imagepolicy": "flux-system:terminus"}
+          image: ghcr.io/usetrmnl/terminus:0.75.0 # {"$imagepolicy": "flux-system:terminus"}
           command: ["bundle", "exec", "sidekiq", "-r", "./config/sidekiq.rb"]
           envFrom:
             - configMapRef:

kubernetes/apps/uptime-monitor/crd-endpointmonitors.yaml

@@ -0,0 +1,473 @@
+# VENDORED, DO NOT EDIT BY HAND.
+#
+# Source: stakater/IngressMonitorController at tag v2.2.13 — the tag the image in
+#   deployment.yaml is pinned to — commit 2deeba188fbd2860d0b3a7c3590a954a1a709924,
+#   charts/ingressmonitorcontroller/crds/endpointmonitor.stakater.com_endpointmonitors.yaml
+# Everything below this header is byte-identical to that file: md5
+#   f753baa9c9684002ad01d309e98755ba over `apiVersion:` to EOF. The only edits are
+#   this header and the dropped leading `---`.
+#
+# An earlier revision of this header cited aaffd8a866aa09a77030979d938e766895bde5d4
+# instead. That is a master merge commit dated 2026-09-25, four months after the
+# v2.2.13 tag (2026-05-06). The file content was not affected — the chart CRD is
+# unchanged between the tag and that commit — but a citation pointing at a moving
+# branch cannot be used to check this file against the version actually deployed.
+#
+# Vendored rather than installed via HelmRelease because this Kustomization
+# ships hand-written RBAC (see rbac.yaml) — the chart's own Role grants pods,
+# services and `secrets: list`, none of which this controller uses. Pulling the
+# chart for the CRD alone would drag those back in.
+#
+# To update: bump the image tag in deployment.yaml and re-copy this file from
+# the matching chart tag. Nothing here is Renovate-managed, so the tag and this
+# file move together, by hand, in one reviewable commit.
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.16.1
+  name: endpointmonitors.endpointmonitor.stakater.com
+spec:
+  group: endpointmonitor.stakater.com
+  names:
+    kind: EndpointMonitor
+    listKind: EndpointMonitorList
+    plural: endpointmonitors
+    singular: endpointmonitor
+  scope: Namespaced
+  versions:
+  - name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: EndpointMonitor is the Schema for the endpointmonitors API
+        properties:
+          apiVersion:
+            description: |-
+              APIVersion defines the versioned schema of this representation of an object.
+              Servers should convert recognized schemas to the latest internal value, and
+              may reject unrecognized values.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
+            type: string
+          kind:
+            description: |-
+              Kind is a string value representing the REST resource this object represents.
+              Servers may infer this from the endpoint the client submits requests to.
+              Cannot be updated.
+              In CamelCase.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: EndpointMonitorSpec defines the desired state of EndpointMonitor
+            properties:
+              appInsightsConfig:
+                description: Configuration for AppInsights Monitor Provider
+                properties:
+                  frequency:
+                    description: 'Sets how often the test should run from each test
+                      location. Possible values: `300,600,900` seconds'
+                    type: integer
+                  retryEnable:
+                    description: 'If its `true`, falied test will be retry after a
+                      short interval. Possible values: `true, false`'
+                    type: boolean
+                  statusCode:
+                    description: Returned status code that is counted as a success
+                    type: integer
+                type: object
+              forceHttps:
+                description: Force monitor endpoint to use HTTPS
+                type: boolean
+              gcloudConfig:
+                description: Configuration for Google Cloud Monitor Provider
+                properties:
+                  projectId:
+                    description: Google Cloud Project ID
+                    type: string
+                type: object
+              grafanaConfig:
+                description: Configuration for Grafana Cloud Monitor Provider
+                properties:
+                  alertSensitivity:
+                    default: none
+                    description: |-
+                      The alertSensitivity value defaults to none if there are no alerts or can be set to low, medium,
+                      or high to correspond to the check alert levels.
+                    enum:
+                    - none
+                    - low
+                    - medium
+                    - high
+                    type: string
+                  frequency:
+                    description: The frequency value specifies how often the check
+                      runs in milliseconds
+                    format: int64
+                    type: integer
+                  probes:
+                    description: |-
+                      Probes are the monitoring agents responsible for simulating user interactions with your web applications
+                      or services. These agents periodically send requests to predefined URLs and record the responses,
+                      checking for expected outcomes and measuring performance.
+                    items:
+                      type: string
+                    type: array
+                  tenantId:
+                    format: int64
+                    type: integer
+                type: object
+              healthEndpoint:
+                type: string
+              pingdomConfig:
+                description: Configuration for Pingdom Monitor Provider
+                properties:
+                  alertContacts:
+                    description: '`-` separated contact id''s (e.g. "1234567_8_9-9876543_2_1")'
+                    type: string
+                  alertIntegrations:
+                    description: '`-` separated set list of integrations ids (e.g.
+                      "91166-12168")'
+                    type: string
+                  basicAuthUser:
+                    description: Required for basic-authentication
+                    type: string
+                  notifyWhenBackUp:
+                    description: Set to "false" to disable recovery notifications
+                    type: boolean
+                  paused:
+                    description: Set to "true" to pause checks
+                    type: boolean
+                  postDataEnvVar:
+                    description: |-
+                      Data that should be posted to the web page, for example submission data for a sign-up or login form.
+                      The data needs to be formatted in the same way as a web browser would send it to the web server.
+                      Because post data contains sensitive secret this field is only a reference to an environment variable.
+                    type: string
+                  requestHeaders:
+                    description: Custom request headers
+                    type: string
+                  requestHeadersEnvVar:
+                    description: |-
+                      Custom request headers that should be read from an environment variable as it possibly contains sensitive data.
+                      An example would be an API token.
+                    type: string
+                  resolution:
+                    description: The pingdom check interval in minutes
+                    type: integer
+                  sendNotificationWhenDown:
+                    description: How many failed check attempts before notifying
+                    type: integer
+                  shouldContain:
+                    description: Set to text string that has to be present in the
+                      HTML code of the page
+                    type: string
+                  sslDownDaysBefore:
+                    description: |-
+                      Consider down prior to certificate expiring
+                      Select the number of days prior to your certificate expiry date that you want to consider the check down.
+                      At this day your check will be considered down and if applicable a down alert will be sent.
+                    type: integer
+                  tags:
+                    description: Comma separated set of tags to apply to check (e.g.
+                      "testing,aws")
+                    type: string
+                  teamAlertContacts:
+                    description: '`-` separated team id''s (e.g. "1234567_8_9-9876543_2_1")'
+                    type: string
+                  verifyCertificate:
+                    description: |-
+                      Monitor SSL/TLS certificate
+                      Monitor the validity of your SSL/TLS certificate. With this enabled Uptime checks will be considered DOWN when
+                      the certificate becomes invalid or expires.
+                      SSL/TLS certificate monitoring is available for HTTP checks.
+                    type: boolean
+                type: object
+              pingdomTransactionConfig:
+                description: Configuration for Pingdom Transaction Monitor Provider
+                properties:
+                  alertContacts:
+                    description: '`-` separated contact id''s (e.g. "1234567_8_9-9876543_2_1")'
+                    type: string
+                  alertIntegrations:
+                    description: '`-` separated set list of integrations ids (e.g.
+                      "91166-12168")'
+                    type: string
+                  custom_message:
+                    description: Custom message that is part of the email and webhook
+                      alerts
+                    type: string
+                  interval:
+                    description: 'TMS test intervals in minutes. Allowed intervals:
+                      5,10,20,60,720,1440. The interval you''re allowed to set may
+                      vary depending on your current plan.'
+                    enum:
+                    - 5
+                    - 10
+                    - 20
+                    - 60
+                    - 720
+                    - 1440
+                    type: integer
+                  paused:
+                    description: 'Check status: active or inactive'
+                    type: boolean
+                  region:
+                    description: 'Name of the region where the check is executed.
+                      Supported regions: us-east, us-west, eu, au'
+                    enum:
+                    - us-east
+                    - us-west
+                    - eu
+                    - au
+                    type: string
+                  send_notification_when_down:
+                    description: Send notification when down X times
+                    format: int64
+                    type: integer
+                  severity_level:
+                    description: 'Check importance- how important are the alerts when
+                      the check fails. Allowed values: low, high'
+                    enum:
+                    - low
+                    - high
+                    type: string
+                  steps:
+                    description: steps to be executed as part of the check
+                    items:
+                      description: PingdomStep respresents a step of the script to
+                        run a transcaction check
+                      properties:
+                        args:
+                          additionalProperties:
+                            type: string
+                          description: |-
+                            contains the html element with assigned value
+                            the key element is always lowercase for example {"url": "https://www.pingdom.com"}
+                            see available values at https://pkg.go.dev/github.com/karlderkaefer/pingdom-golang-client@latest/pkg/pingdom/client/tmschecks#StepArg
+                          type: object
+                        function:
+                          description: |-
+                            contains the function that is executed as part of the step
+                            commands: go_to, click, fill, check, uncheck, sleep, select_radio, basic_auth, submit, wait_for_element, wait_for_contains
+                            validations: url, exists, not_exists, contains, not_contains, field_contains, field_not_contains, is_checked, is_not_checked, radio_selected, dropdown_selected, dropdown_not_selected
+                            see updated list https://docs.pingdom.com/api/#section/TMS-Steps-Vocabulary/Script-transaction-checks
+                          type: string
+                      required:
+                      - args
+                      - function
+                      type: object
+                    type: array
+                  tags:
+                    description: List of tags for a check. The tag name may contain
+                      the characters 'A-Z', 'a-z', '0-9', '_' and '-'. The maximum
+                      length of a tag is 64 characters.
+                    items:
+                      type: string
+                    type: array
+                  teamAlertContacts:
+                    description: '`-` separated team id''s (e.g. "1234567_8_9-9876543_2_1")'
+                    type: string
+                required:
+                - steps
+                type: object
+              providers:
+                description: Comma separated list of providers
+                type: string
+              statusCakeConfig:
+                description: Configuration for StatusCake Monitor Provider
+                properties:
+                  basicAuthSecret:
+                    description: Basic Auth Secret Name
+                    type: string
+                  basicAuthUser:
+                    description: Basic Auth User
+                    type: string
+                  checkRate:
+                    default: 300
+                    description: Set Check Rate for the monitor.
+                    type: integer
+                  confirmation:
+                    description: Confirmation value ranges from (0,10)
+                    maximum: 10
+                    minimum: 0
+                    type: integer
+                  contactGroup:
+                    description: Contact Group to be alerted.
+                    type: string
+                  enableSslAlert:
+                    description: Enable SSL Alert
+                    type: boolean
+                  findString:
+                    description: String to look for within the response. Considered
+                      down if not found
+                    type: string
+                  followRedirect:
+                    description: Enable ingress redirects
+                    type: boolean
+                  paused:
+                    description: Pause the service
+                    type: boolean
+                  pingUrl:
+                    description: Webhook for alerts
+                    type: string
+                  port:
+                    description: TCP Port
+                    type: integer
+                  rawPostData:
+                    description: RawPostData can be used to send parameters within
+                      the URL. Changes the request from a GET to a POST
+                    type: string
+                  realBrowser:
+                    description: Enable Real Browser
+                    type: boolean
+                  regions:
+                    description: Comma separated list of Node Location IDs
+                    type: string
+                  statusCodes:
+                    description: Comma separated list of HTTP codes to trigger error
+                      on
+                    type: string
+                  testTags:
+                    description: Comma separated list of tags
+                    type: string
+                  testType:
+                    description: Set Test type - HTTP, TCP, PING, or Heartbeat
+                    enum:
+                    - HTTP
+                    - TCP
+                    - PING
+                    - Heartbeat
+                    type: string
+                  timeout:
+                    description: Timeout is used to set a user agent string.
+                    maximum: 75
+                    minimum: 5
+                    type: integer
+                  triggerRate:
+                    description: Minutes to wait before sending an alert
+                    type: integer
+                  userAgent:
+                    description: UserAgent is used to set a user agent string.
+                    type: string
+                type: object
+                x-kubernetes-validations:
+                - message: checkRate must be between 30 and 172800 seconds for Heartbeat
+                    monitors
+                  rule: self.testType != 'Heartbeat' || (self.checkRate >= 30 && self.checkRate
+                    <= 172800)
+                - message: 'checkRate for uptime monitors must be one of: 0, 30, 60,
+                    300, 900, 1800, 3600, 86400'
+                  rule: self.testType == 'Heartbeat' || self.checkRate in [0, 30,
+                    60, 300, 900, 1800, 3600, 86400]
+              updownConfig:
+                description: Configuration for Updown Monitor Provider
+                properties:
+                  enable:
+                    description: Enable or disable checks
+                    type: boolean
+                  period:
+                    description: The pingdom check interval in seconds
+                    type: integer
+                  publishPage:
+                    description: Make status page public or not
+                    type: boolean
+                  requestHeaders:
+                    description: Additional request headers for API calls
+                    type: string
+                type: object
+              uptimeConfig:
+                description: Configuration for Uptime Monitor Provider
+                properties:
+                  checkType:
+                    description: The uptime check type that can be HTTP/DNS/ICMP etc.
+                    type: string
+                  contacts:
+                    description: Add one or more contact groups separated by `,`
+                    type: string
+                  interval:
+                    description: The uptime check interval in seconds
+                    type: integer
+                  locations:
+                    description: Add different locations for the check
+                    type: string
+                  tags:
+                    description: Add one or more tags for the check separated by `,`
+                    type: string
+                type: object
+              uptimeRobotConfig:
+                description: Configuration for UptimeRobot Monitor Provider
+                properties:
+                  alertContacts:
+                    description: The uptimerobot alertContacts to be associated with
+                      this monitor
+                    type: string
+                  customHTTPStatuses:
+                    description: |-
+                      Defines which http status codes are treated as up or down
+                      For ex: 200:0_401:1_503:1 (to accept 200 as down and 401 and 503 as up)
+                    type: string
+                  interval:
+                    description: The uptimerobot check interval in seconds
+                    minimum: 60
+                    type: integer
+                  keywordExists:
+                    description: Alert if value exist (yes) or doesn't exist (no)
+                      (Only if monitor-type is keyword)
+                    enum:
+                    - "yes"
+                    - "no"
+                    type: string
+                  keywordValue:
+                    description: keyword to check on URL (e.g.'search' or '404') (Only
+                      if monitor-type is keyword)
+                    type: string
+                  maintenanceWindows:
+                    description: Specify maintenanceWindows i.e. once or recurring
+                      “do-not-monitor periods”
+                    type: string
+                  monitorType:
+                    description: The uptimerobot monitor type (http or keyword)
+                    enum:
+                    - http
+                    - keyword
+                    type: string
+                  statusPages:
+                    description: The uptimerobot public status page ID to add this
+                      monitor to
+                    type: string
+                type: object
+              url:
+                description: URL to monitor
+                type: string
+              urlFrom:
+                description: URL to monitor from either an ingress or route reference
+                properties:
+                  ingressRef:
+                    description: IngressURLSource selects an Ingress to populate the
+                      URL with
+                    properties:
+                      name:
+                        type: string
+                    required:
+                    - name
+                    type: object
+                  routeRef:
+                    description: RouteURLSource selects a Route to populate the URL
+                      with
+                    properties:
+                      name:
+                        type: string
+                    required:
+                    - name
+                    type: object
+                type: object
+            type: object
+          status:
+            description: EndpointMonitorStatus defines the observed state of EndpointMonitor
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}

kubernetes/apps/uptime-monitor/deployment.yaml

@@ -0,0 +1,118 @@
+---
+# Stakater IngressMonitorController (REN-6).
+#
+# It ships with zero EndpointMonitor resources, which is the whole point of this
+# first deployment: the reconciler is keyed on one custom resource per event
+# (endpointmonitor_controller.go:62-107) and never enumerates the UptimeRobot
+# account, so with no EndpointMonitors it reconciles nothing and makes no API
+# call to UptimeRobot at all. The 9 monitors already in that account are not
+# read, not updated and not deleted.
+#
+# Plain manifests, not the upstream HelmRelease: the chart's RBAC grants pods,
+# services and `secrets: list`, and it also creates a Service and optional
+# cert-manager objects for a metrics endpoint that is disabled below. The
+# maintained thing we depend on is the upstream image, which is pinned here.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: ingressmonitorcontroller
+  namespace: uptime-monitor
+  labels:
+    app.kubernetes.io/name: ingressmonitorcontroller
+  annotations:
+    # The API key is read ONCE, at startup (pkg/config/config.go
+    # `LoadControllerConfig` runs before the manager starts), so a rotated
+    # Secret is not picked up until the pod restarts. Reloader — which this
+    # cluster already runs — does that restart. Same idiom as
+    # apps/external-dns/helmrelease.yaml.
+    secret.reloader.stakater.com/reload: uptime-robot-api-key
+spec:
+  replicas: 1
+  strategy:
+    # Recreate, not RollingUpdate, and no --leader-elect. A rolling update would
+    # briefly run two reconcilers against the same UptimeRobot account; Recreate
+    # makes "at most one controller pod, ever" a property of the Deployment
+    # instead of something a Lease has to arbitrate. It costs a few seconds of
+    # controller downtime per upgrade, which for a monitor-syncing controller is
+    # nothing, and it removes the leases/events RBAC entirely.
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: ingressmonitorcontroller
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: ingressmonitorcontroller
+    spec:
+      serviceAccountName: ingressmonitorcontroller
+      automountServiceAccountToken: true
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 65532
+        runAsGroup: 65532
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: manager
+          # Pinned, and deliberately NOT under Flux image automation. rbac.yaml
+          # is derived from what this exact version calls; an automatic bump
+          # could add a call the Role does not cover, silently.
+          image: ghcr.io/stakater/ingressmonitorcontroller:v2.2.13
+          command: ["/manager"]
+          args:
+            - --health-probe-bind-address=:8081
+            # Metrics off. Enabling them means either TLS certs or
+            # `--metrics-secure=false`, and the authn/authz filter
+            # (cmd/main.go, `filters.WithAuthenticationAndAuthorization`) wants a
+            # ClusterRole for TokenReview/SubjectAccessReview — a cluster-scoped
+            # grant this namespace otherwise does not need. A ServiceMonitor for
+            # this controller is worth having, but it is its own change.
+            - --metrics-bind-address=0
+          env:
+            # Scopes the manager's cache to this namespace
+            # (cmd/main.go `buildDefaultNamespaces`), which is what makes the
+            # namespaced Role in rbac.yaml sufficient — there is no ClusterRole.
+            - name: WATCH_NAMESPACE
+              value: uptime-monitor
+            # The namespace the config Secret is looked up in. Read from the
+            # ServiceAccount token file when unset; set explicitly so the lookup
+            # is visible in the manifest.
+            - name: OPERATOR_NAMESPACE
+              valueFrom:
+                fieldRef:
+                  fieldPath: metadata.namespace
+            # IMC reads its whole configuration from key `config.yaml` of this
+            # Secret — providers, API key and enableMonitorDeletion together.
+            # ESO renders that key; see externalsecret.yaml for the contents and
+            # for why the config lives in git while the key does not.
+            - name: CONFIG_SECRET_NAME
+              value: uptime-robot-api-key
+          # Both probes are served by the manager on :8081 and are dialled by
+          # the kubelet, which Cilium identifies as the `host` entity — the only
+          # ingress the namespace policy admits
+          # (infrastructure/network-policies/uptime-monitor-ns.yaml). The port is
+          # not exposed as a containerPort because nothing else may reach it.
+          livenessProbe:
+            httpGet:
+              path: /healthz
+              port: 8081
+            initialDelaySeconds: 15
+            periodSeconds: 20
+          readinessProbe:
+            httpGet:
+              path: /readyz
+              port: 8081
+            initialDelaySeconds: 5
+            periodSeconds: 10
+          resources:
+            requests:
+              cpu: 10m
+              memory: 64Mi
+            limits:
+              memory: 128Mi
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop: ["ALL"]
+      terminationGracePeriodSeconds: 10

kubernetes/apps/uptime-monitor/externalsecret.yaml

@@ -0,0 +1,79 @@
+---
+# UptimeRobot API key for the IngressMonitorController (REN-2).
+#
+# OpenBao key infrastructure/uptime-robot, field `api_key`. `data` + `property`
+# rather than `dataFrom: extract` (the shape most ExternalSecrets in this repo
+# use) on purpose: extract copies EVERY field at the KV path into the cluster
+# Secret, so any field added there later — an account id, a second token —
+# would land in this namespace without anyone editing this file. Naming the one
+# field keeps the Secret to exactly what the controller needs. Same shape as
+# kubernetes/apps/wiki/externalsecrets.yaml.
+#
+# refreshInterval 1h matches the un-rotated ExternalSecrets here (velero,
+# wiki-*). The rotated ones use 10m so an automated roll propagates fast — if
+# kubernetes/infrastructure/openbao/rotation/ ever gains an uptime-robot
+# rotator, drop this to 10m. The controller Deployment already carries the
+# stakater/reloader annotation, so a new value restarts the pod that has the
+# old one cached.
+#
+# The `eso` role already reads this path: eso-read grants `homelab/data/*`
+# (kubernetes/infrastructure/openbao/policies/eso-read.hcl:31). No OpenBao
+# policy change is needed for this key.
+#
+# ---
+# Why this Secret holds a rendered `config.yaml` rather than a bare key (REN-6):
+# IngressMonitorController does not read an API key from an env var. It reads
+# its ENTIRE configuration — provider list, API key, enableMonitorDeletion —
+# from one Secret key named `config.yaml` (pkg/config/config.go,
+# `IngressMonitorControllerSecretConfigKey`), at startup, through the manager's
+# APIReader. So the choice is not "key or config"; it is which Secret the config
+# lives in. Templating it here keeps every non-secret decision in git and in
+# review, with only the key itself injected at sync time.
+#
+# The template replaces the Secret's data rather than merging, so `apiKey` is a
+# template input and the rendered `config.yaml` is the single output key.
+#
+# Two values are load-bearing:
+#
+#   apiURL — REQUIRED. `Setup()` assigns `monitor.url = p.ApiURL` with no
+#   default (pkg/monitors/uptimerobot/uptime-monitor.go:39-44); omit it and
+#   every request goes to a relative URL.
+#
+#   enableMonitorDeletion — true, so that removing an EndpointMonitor removes
+#   its monitor, which is what the mission asks for. It is bounded: deletion is
+#   reachable only via `GetByName`, which filters `m.FriendlyName == name` —
+#   exact equality — against the name the controller itself computed from the
+#   default template `{{.Name}}-{{.Namespace}}`. Every name it can compute ends
+#   in `-uptime-monitor`; none of the 9 monitors already in the account does.
+#   Set it to false for a create/update-only kill switch.
+#
+# monitorNameTemplate is deliberately absent: an empty value makes
+# GetNameTemplateFormat fall back to that same default (pkg/util/template-utils.go:9),
+# and writing it out here would mean escaping Go template syntax inside a Go
+# template for no gain.
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: uptime-robot-api-key
+  namespace: uptime-monitor
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: openbao
+  target:
+    name: uptime-robot-api-key
+    template:
+      engineVersion: v2
+      data:
+        config.yaml: |
+          providers:
+            - name: UptimeRobot
+              apiKey: "{{ .apiKey }}"
+              apiURL: https://api.uptimerobot.com/v2/
+          enableMonitorDeletion: true
+  data:
+    - secretKey: apiKey
+      remoteRef:
+        key: infrastructure/uptime-robot
+        property: api_key

kubernetes/apps/uptime-monitor/kustomization.yaml

@@ -0,0 +1,11 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - crd-endpointmonitors.yaml
+  - externalsecret.yaml
+  - rbac.yaml
+  - deployment.yaml
+  # Last, and after the CRD it depends on. The monitors themselves — the only
+  # file in this directory that causes an UptimeRobot write. Everything above it
+  # is inert until an EndpointMonitor exists.
+  - pilot-endpointmonitors.yaml

kubernetes/apps/uptime-monitor/pilot-endpointmonitors.yaml

@@ -0,0 +1,280 @@
+---
+# ============================================================================
+# REN-7 CRITERION 3 PASSED. THIS COMMIT PUTS THE `audiobooks` ENDPOINTMONITOR
+# BACK — IT IS THE DELIBERATE REVERT OF THE REMOVAL TEST, NOT A NEW FEATURE.
+#
+# The card requires proof in both directions, and both are now on record:
+#
+#   CREATE  2026-09-26T03:26:27Z  `Monitor Added: audiobooks.renner.uno-
+#           uptime-monitor`, UptimeRobot id 804092841, feed
+#           `psp.totalMonitors` 9 → 10.
+#   DELETE  2026-09-26T03:57:58Z  object pruned by Flux after #1018 merged as
+#           101c11f4; controller log `Removing monitor audiobooks.renner.uno-
+#           uptime-monitor from provider: UptimeRobot` then `Monitor Removed`;
+#           feed 10 → 9 with id 804092841 absent, confirmed independently.
+#
+# THE LOG STRING IS `Removing monitor … from provider` / `Monitor Removed`.
+# An earlier revision of this header predicted `Deleting Monitor: …`, which
+# appears nowhere in v2.2.13. Anyone grepping the predicted string finds
+# nothing and wrongly concludes the delete failed. Grep `Monitor Removed`.
+#
+# Re-creating the object is idempotent by design: the friendly name binds
+# object to monitor, not the id, so the new monitor carries a NEW id.
+#
+# PRECONDITION RE-CHECKED BEFORE THIS COMMIT WAS WRITTEN, and it is the one the
+# no-finalizer caveat below demands: pod `ingressmonitorcontroller-5d577cc58-
+# pgqwg` read from the Kubernetes API at 2026-09-26T08:01Z — `Ready=True`,
+# `ContainersReady=True`, `restarts=0`, image v2.2.13, up since
+# 2026-09-26T00:43:45Z, the same pod that served the whole create/delete
+# window. That rule binds DELETES, so it does not gate this commit, but the
+# reviewer should re-read it anyway before merging.
+#
+# EXPECT THIS CREATE TO BE REFUSED, AND THAT IS NOT A FAULT. The account cap is
+# 10 monitors and it is full again (see the capacity section below). Restoring
+# the object is what Zeus directed after the cap reading, on the reasoning that
+# the retry loop then creates the monitor by itself the moment the tier is
+# raised — no further PR needed. Until then this object logs one refused POST
+# every 300s and nothing else: nothing pages, nothing degrades, and
+# `audiobooks.renner.uno` stays covered by its 60s Gatus check
+# (apps/gatus/configmap.yaml:163) throughout.
+#
+# REVERT: revert this commit. Flux prunes the object again and, because the
+# monitor does not exist to begin with, nothing in UptimeRobot changes.
+# ============================================================================
+#
+# REN-7 — the pilot batch. Two of the 14 monitors from the survey's §A0 page
+# list, chosen because they are the two lowest-risk rows on it:
+#
+#   * `www.renner.uno`      — public by design, no Authentik gate in front,
+#                             so the check reaches the application itself.
+#   * `audiobooks.renner.uno` — ungated `/healthcheck` endpoint, so likewise
+#                             real app health rather than reachability.
+#
+# Both are already on the owner's approved page list (§A0 rows 2 and 6), so if
+# the status page turns out to include new monitors automatically, these two
+# appearing on it is authorised rather than a disclosure. That was a deliberate
+# selection criterion, not a coincidence — see the status-page note below.
+#
+# Neither host had a monitor in the account when the pilot was chosen; today
+# `www.renner.uno-uptime-monitor` (id 804092931) exists because this pilot
+# created it. The nine pre-existing ones are
+# `renner.uno`, `home.`, `immich.`, `auth.`, `docs.renner.uno` and the four
+# third-party ones; creating an EndpointMonitor for any of those would produce
+# a SECOND monitor beside the hand-made one rather than adopt it, because IMC
+# only ever matches the name it computes itself. Picking two hosts with no
+# existing monitor keeps the pilot free of that question, which belongs to the
+# REN-8 rollout.
+#
+# ---
+# WHY `spec.url` AND NOT `spec.urlFrom.ingressRef`
+#
+# REN-7 was written asking for `urlFrom.ingressRef`. That cannot work from this
+# namespace, and the reason is structural rather than a permissions gap:
+#
+#   * `IngressURLSource` has exactly one field, `name` — there is no namespace
+#     field on it (crd-endpointmonitors.yaml:443-459).
+#   * `discoverURLFromRefs` resolves that name against
+#     `ingressMonitor.Namespace` — the EndpointMonitor's OWN namespace
+#     (v2.2.13 pkg/kube/util/url.go:71 → :38-40).
+#
+# So an EndpointMonitor in `uptime-monitor` can only reference an Ingress in
+# `uptime-monitor`, and there are none — every app Ingress lives in its app's
+# namespace. `spec.url` short-circuits ref discovery entirely
+# (pkg/kube/util/url.go:26-34), so no Ingress is read and the Role's
+# `networking.k8s.io/ingresses` read grant stays unused here.
+#
+# The alternative — one EndpointMonitor per app namespace — was not taken: it
+# would need a Role, a RoleBinding and a widened WATCH_NAMESPACE in every
+# production application namespace, which is a far larger blast radius than the
+# mission's "own namespace, own ServiceAccount, minimal RBAC" constraint allows.
+# Writing the URL out here keeps all of this work inside `uptime-monitor`.
+#
+# The mission property that mattered is unaffected: these are still declarative
+# opt-in objects in git, and deleting one still deletes its monitor.
+#
+# ---
+# MONITOR NAMES
+#
+# `monitorNameTemplate` is left at the default `{{.Name}}-{{.Namespace}}`
+# (pkg/util/template-utils.go:9), so these two objects produce UptimeRobot
+# friendly names:
+#
+#   www.renner.uno-uptime-monitor
+#   audiobooks.renner.uno-uptime-monitor
+#
+# The `-uptime-monitor` suffix is load-bearing and must not be templated away:
+# the delete path matches `FriendlyName == <computed name>` exactly
+# (pkg/monitors/uptimerobot/uptime-monitor.go:70), so the suffix is what keeps
+# the controller structurally unable to touch the nine monitors already in the
+# account (REN-34). The cost is that the public page shows the suffixed name;
+# that is REN-9's to resolve and must not be "fixed" here by changing the
+# template.
+#
+# ---
+# NO `alertContacts` YET — AND THIS IS A GAP, NOT AN OVERSIGHT
+#
+# `processProviderConfig` ALWAYS appends `&alert_contacts=`, falling back to the
+# provider-level value from config.yaml when the CR sets none
+# (uptime-monitor.go:240-244). Neither is set today, so these two monitors are
+# created with an EMPTY alert-contact list: UptimeRobot will record the outage
+# and show it on the page, and will notify nobody.
+#
+# That is the safe direction to be wrong in for a pilot, but it means REN-7's
+# "an alert reaches the owner's existing alert channel" is NOT satisfied by this
+# file. Closing it needs the numeric alert-contact id from `getAlertContacts`,
+# which needs the API key. Tracked separately; do not read a green pilot as
+# proof the alert path works.
+#
+# ---
+# NO `statusPages` — AND THE EXPERIMENT THAT JUSTIFIED LEAVING IT UNSET IS NOW
+# ANSWERED: THE PAGE IS IN ALL-MONITORS MODE. KEEP THE FIELD UNSET.
+#
+# IMC only adds a monitor to a public status page when
+# `uptimeRobotConfig.statusPages` is set; with the field absent
+# `handleStatusPagesConfig` returns without calling the API at all
+# (uptime-monitor.go:318-329). It was left unset to ask the page itself REN-9's
+# question, and the pilot answered it in BOTH directions with no API key:
+#
+#   9 → 10   on create, monitor appeared on the feed unbidden
+#   10 → 9   on prune, monitor disappeared from it unbidden
+#
+# So `uptime.renner.uno` includes every monitor in the account automatically and
+# nothing needs adding to it. Caveat for REN-9, and it is the fragile part:
+# all-monitors mode is a setting in the UptimeRobot dashboard, NOT in git. If
+# anyone switches that page to a fixed list, this stops being true silently and
+# no manifest, review or alert here would catch it.
+#
+# Baseline that comparison was made against, read keyless at 2026-09-26 from
+# `https://uptime.renner.uno/api/getMonitorList/UgMXmP1iRV`: 9 monitors, ids
+# 801710132/37/43/45/47/52/97 and 801714845/79. Parser notes for whoever reads
+# this feed next: there is no top-level `monitors` key — the list is at `data`
+# (and again at `psp.monitors`), the count is `psp.totalMonitors`, and an id is
+# `monitorId`, not `id`.
+#
+# ---
+# BLAST RADIUS
+#
+# Two new cluster objects in `uptime-monitor`, and two HTTP GETs every 300s from
+# UptimeRobot's probers to two hosts that already serve the public internet.
+# Nothing in an application namespace is read, written or restarted; no Ingress
+# is edited; no DNS record changes. The 300s interval sits far inside the
+# Traefik rate-limit middleware on the `websecure` routers, and the CrowdSec
+# bouncer is detached today — but see the survey §C: if it is ever re-attached,
+# UptimeRobot's source addresses must be allowlisted first.
+#
+# REVERT: delete this file and remove it from kustomization.yaml, or revert the
+# merge commit. Flux prunes both objects, and because `enableMonitorDeletion` is
+# true the controller then deletes both monitors from UptimeRobot. Reverting
+# leaves the account exactly as the baseline above.
+#
+# ---
+# ONE CAVEAT ON THAT REVERT, AND IT IS THE "A MONITOR THAT CANNOT BE REMOVED IS
+# A LEAK" CASE: v2.2.13 USES NO FINALIZER.
+#
+# `handleDelete` runs from the reconcile path only when the Get returns
+# NotFound (endpointmonitor_controller.go:80-87) — there is no finalizer
+# anywhere in the source, so nothing holds the object in the API server until
+# the monitor has actually been deleted. Deletion is therefore observed as an
+# EVENT, and an event needs a running watcher:
+#
+#   * controller running when the object is deleted → the monitor is deleted.
+#     This is the normal case and the one the pilot proves.
+#   * controller down when the object is deleted → the delete is never
+#     reconciled. On restart the informer lists what EXISTS, so a
+#     no-longer-existing object produces no request, and nothing else ever
+#     reconciles it: uptimerobot's `GetAll()` has no caller, so the account is
+#     never enumerated and never garbage-collected (REN-34). The monitor is
+#     then orphaned in UptimeRobot with no source in git.
+#
+# Recovering from that is manual and its blast radius is the account, so the
+# operational rule is: do not delete an EndpointMonitor while the controller is
+# unavailable. Check `kube_deployment_status_replicas_available{namespace=
+# "uptime-monitor"}` first, and if a monitor is orphaned anyway, delete it by id
+# through the API rather than hoping a later reconcile finds it.
+#
+# ---
+# CAPACITY: THE ACCOUNT CAP IS EXACTLY 10 MONITORS, PROVEN BY EXPERIMENT AND
+# WITHOUT AN API KEY. THIS SUPERSEDES THIS FILE'S EARLIER READING OF THE 403.
+#
+# An earlier revision of this header asserted the opposite — that "a duplicate
+# URL, a monitor-count cap and a bad parameter would all arrive as 200 +
+# `stat: fail`, not as 403", and therefore that the cap was EXCLUDED. That
+# inference was wrong, and the correction is the useful finding here:
+#
+#   UPTIMEROBOT RETURNS HTTP 403 WHEN THE ACCOUNT MONITOR LIMIT IS REACHED.
+#   It does not use the 200 + `stat: fail` validation path for this. Anyone
+#   reasoning about an IMC 403 should treat "account is full" as the FIRST
+#   hypothesis, not a discarded one.
+#
+# The proof is a natural experiment the removal test handed us for free, read
+# from the controller log of the single pod that spans the whole window:
+#
+#   03:26:27 → 03:56:27   account at 10/10. `Creating Monitor:
+#                         www.renner.uno-uptime-monitor` then `AddMonitor
+#                         Request failed. Status Code: 403`, SEVEN consecutive
+#                         300s cycles. On cycles 2-7 that create is the only
+#                         POST in the cycle, so it is not a burst/rate effect.
+#   03:57:58              removal test deletes monitor 804092841 → account 9.
+#   04:01:27              next ordinary reconcile, nothing else changed:
+#                         `Monitor Added: www.renner.uno-uptime-monitor`,
+#                         new id 804092931, feed back to 10.
+#
+# The only variable between the last 403 and the success is one free slot. A
+# duplicate-URL or bad-parameter cause cannot explain a create that starts
+# succeeding when an UNRELATED monitor is deleted. Capacity is settled on fact;
+# the tier decision is REN-158's, and REN-8's §A0 page needs 18 monitors
+# against this cap of 10, so no rollout batch may proceed until it is resolved.
+#
+# Why the object below stayed in git through those seven refusals: `Add()` is
+# fire-and-forget (`handleCreate` returns nil regardless,
+# endpointmonitor_created.go:12-31) and the failure is logged at INFO, not
+# ERROR — so there is no error, no status and no alert, and the retrying object
+# was the only evidence that the create was being refused at all. That property
+# is worth remembering for REN-8: a rollout that exceeds the cap fails SILENTLY,
+# monitor by monitor, with nothing to notice it but this one INFO line.
+apiVersion: endpointmonitor.stakater.com/v1alpha1
+kind: EndpointMonitor
+metadata:
+  name: www.renner.uno
+  namespace: uptime-monitor
+spec:
+  # `https://www.renner.uno` and not `/wp-admin` or `/wp-login.php`: those two
+  # paths are served by a SECOND Ingress that sits behind the Authentik outpost
+  # (apps/wordpress/ingress.yaml:50,57). `/` is the ungated one.
+  url: https://www.renner.uno/
+  uptimeRobotConfig:
+    # `http`, not `keyword`. The survey recommends a keyword check here
+    # eventually, and that is right — a WordPress instance can serve 200 with a
+    # broken page. It is deliberately not in the pilot: a keyword check needs a
+    # string chosen against the live rendered page, and getting it wrong
+    # produces a monitor that is red while the site is fine. Tune it in REN-8,
+    # once this file has proven the create/delete mechanism.
+    monitorType: http
+    # 300s is also IMC's own default (DefaultInterval, uptime-monitor.go:28) and
+    # matches the interval of the nine monitors already in the account. Written
+    # out rather than left implicit so the value is reviewable here.
+    interval: 300
+---
+# Restored by this commit. This object already proved criterion 3 end to end:
+# created as id 804092841 at 03:26:27Z, removed as `Monitor Removed` at
+# 03:57:58Z when it was pruned from git. See the header for that evidence.
+#
+# Its create will now be refused with HTTP 403 until the account tier is raised
+# (REN-158), because `www.renner.uno` took the slot this object's deletion
+# freed. That is expected and documented, not a regression to chase.
+apiVersion: endpointmonitor.stakater.com/v1alpha1
+kind: EndpointMonitor
+metadata:
+  name: audiobooks.renner.uno
+  namespace: uptime-monitor
+spec:
+  # The path belongs in `spec.url` and NOT in `spec.healthEndpoint`, because
+  # `healthEndpoint` is only consulted on the ref-discovery path and is
+  # explicitly ignored when `spec.url` is set (pkg/kube/util/url.go:31-33).
+  # `/healthcheck` is ungated, so the check reaches application health rather
+  # than mere reachability — it is the same endpoint the Gatus check uses
+  # (apps/gatus/configmap.yaml:163).
+  url: https://audiobooks.renner.uno/healthcheck
+  uptimeRobotConfig:
+    monitorType: http
+    interval: 300

kubernetes/apps/uptime-monitor/rbac.yaml

@@ -0,0 +1,96 @@
+---
+# ServiceAccount + namespaced RBAC for IngressMonitorController (REN-6).
+#
+# There is NO ClusterRole here, on purpose. The Deployment sets
+# WATCH_NAMESPACE=uptime-monitor, which makes the manager build a
+# namespace-scoped cache (cmd/main.go `buildDefaultNamespaces`), so every read
+# the controller performs is confined to this namespace. A ClusterRole would
+# grant reach the process cannot use.
+#
+# The rules below were derived from what the pinned image actually calls, not
+# from the chart's rules or the upstream kubebuilder markers. Deliberately NOT
+# granted, and why:
+#
+#   - `""/pods`, `""/services` — read from exactly one place,
+#     `tryGetHealthEndpointFromIngress` (pkg/kube/wrappers/ingress-wrapper.go:126),
+#     which is reached only from the `urlFrom.ingressRef` path and only when the
+#     EndpointMonitor sets no explicit `healthEndpoint`. It is a best-effort
+#     guess at a readiness-probe path. Omitted because that path cannot reach
+#     anything here: ingressRef resolves against the EndpointMonitor's own
+#     namespace (see the ingresses rule below) and nothing in `uptime-monitor`
+#     serves HTTP, so the grant would be inert.
+#     It does NOT fail gracefully, so grant it BEFORE an Ingress lands in this
+#     namespace, not after the first EndpointMonitor points at one. The function's
+#     own error branches do log-and-continue, but an RBAC denial never reaches
+#     them: the reconciler's client is `mgr.GetClient()` (cmd/main.go:168), the
+#     cache-backed client, so the first Service Get starts an informer and waits
+#     for it to sync on a `context.TODO()` that nothing cancels. A forbidden
+#     list/watch means that sync never completes, the Get never returns, and the
+#     single reconcile worker (`--max-concurrent-reconciles`, default 1,
+#     cmd/main.go:84) stops processing anything at all. `/readyz` is
+#     `healthz.Ping` (cmd/main.go:182) and keeps answering, so the pod stays
+#     Ready while reconciliation is wedged.
+#   - `route.openshift.io/routes` — OpenShift only; `kube.IsOpenshift` is false
+#     here, so the Route scheme is never even registered (cmd/main.go init()).
+#   - `endpointmonitors/finalizers` — IMC uses no finalizers. Deletion is driven
+#     by the reconciler's IsNotFound branch, not by a finalizer
+#     (endpointmonitor_controller.go:74-81); `grep -rn Finalizer internal/controller/`
+#     has no hits.
+#   - leases / events — no leader election (see the Recreate strategy comment in
+#     deployment.yaml) and the reconciler holds no EventRecorder.
+#
+# Re-check this list whenever the image tag in deployment.yaml moves: it is
+# pinned and not Renovate-managed precisely so that never happens silently.
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: ingressmonitorcontroller
+  namespace: uptime-monitor
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: ingressmonitorcontroller
+  namespace: uptime-monitor
+rules:
+  # Its own custom resource — the only thing the reconciler is keyed on.
+  - apiGroups: ["endpointmonitor.stakater.com"]
+    resources: ["endpointmonitors"]
+    verbs: ["get", "list", "watch"]
+  # Not exercised by v2.2.13 (`grep -rn "Status()" internal/controller/` is
+  # empty), but granted: it is this CRD's own subresource, so it confers no
+  # reach outside the resource already listed above, and it is the one write an
+  # operator of this shape is expected to gain.
+  - apiGroups: ["endpointmonitor.stakater.com"]
+    resources: ["endpointmonitors/status"]
+    verbs: ["get", "update", "patch"]
+  # Read-only. Used to resolve an EndpointMonitor's URL from an Ingress via
+  # `spec.urlFrom.ingressRef`. Note that pkg/kube/util/url.go:71 resolves that
+  # ref against the EndpointMonitor's OWN namespace, so this grant can only ever
+  # reach Ingresses in `uptime-monitor` — it cannot read an app's Ingress.
+  - apiGroups: ["networking.k8s.io"]
+    resources: ["ingresses"]
+    verbs: ["get", "list", "watch"]
+  # The controller loads its entire configuration — including the UptimeRobot
+  # API key — from one Secret key at startup (pkg/config/config.go
+  # `LoadControllerConfig`). It reads it through the manager's APIReader, i.e. a
+  # direct GET by name and not a cached list, which is why `get` alone is enough
+  # and why resourceNames can pin it to the single Secret ESO renders.
+  - apiGroups: [""]
+    resources: ["secrets"]
+    resourceNames: ["uptime-robot-api-key"]
+    verbs: ["get"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: ingressmonitorcontroller
+  namespace: uptime-monitor
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: ingressmonitorcontroller
+subjects:
+  - kind: ServiceAccount
+    name: ingressmonitorcontroller
+    namespace: uptime-monitor

kubernetes/apps/wiki-index/deployment.yaml

@@ -1,137 +0,0 @@
----
-# ServiceAccount is the OpenBao identity (auth/kubernetes/role/wiki-index); no
-# Kubernetes API access needed.
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: wiki-index
-  namespace: ai
-  labels:
-    app.kubernetes.io/name: wiki-index
----
-# Index (~17 MB JSON for 1700 notes). Survives restarts so a pod start does not
-# re-embed the wiki; a lost volume only costs one full sync (~5 min).
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: wiki-index-data
-  namespace: ai
-spec:
-  accessModes:
-    - ReadWriteOnce
-  storageClassName: ceph-block
-  resources:
-    requests:
-      storage: 1Gi
----
-# Embedding index over the Notes repo's wiki/ (ci/images/wiki-index). Syncs from
-# Gitea on push webhook and every RESYNC_SECONDS, embeds via wiki-embed, serves
-# POST /rank to ~/Notes/bin/wiki_rank.py through wiki-index.renner.uno.
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: wiki-index
-  namespace: ai
-  labels:
-    app.kubernetes.io/name: wiki-index
-spec:
-  replicas: 1
-  strategy:
-    type: Recreate
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: wiki-index
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/name: wiki-index
-    spec:
-      serviceAccountName: wiki-index
-      securityContext:
-        runAsNonRoot: true
-        runAsUser: 65534
-        runAsGroup: 65534
-        fsGroup: 65534
-        seccompProfile:
-          type: RuntimeDefault
-      containers:
-        - name: wiki-index
-          # Per-commit tag (build-ci-images.yml): immutable, no same-day overwrite.
-          image: gitea.renner.uno/gitea_admin/wiki-index:2026-09-20-4b529cd1
-          securityContext:
-            allowPrivilegeEscalation: false
-            readOnlyRootFilesystem: true
-            capabilities:
-              drop: ["ALL"]
-          env:
-            - name: GITEA_URL
-              value: https://gitea.renner.uno
-            - name: NOTES_REPO
-              value: k.renner/notes
-            - name: NOTES_BRANCH
-              value: main
-            - name: EMBED_URL
-              value: http://wiki-embed.ai.svc.cluster.local:8080
-            - name: BAO_ADDR
-              value: http://openbao.openbao.svc.cluster.local:8200
-            - name: BAO_ROLE
-              value: wiki-index
-            - name: BAO_SECRET_PATH
-              value: homelab/data/apps/ai/wiki-index
-            - name: INDEX_PATH
-              value: /data/index.json
-            - name: RESYNC_SECONDS
-              value: "1800"
-          ports:
-            - name: http
-              containerPort: 8080
-          # Pure-Python cosine over 1700 x 1024 floats per query; the index itself is ~60 MB in memory.
-          resources:
-            requests:
-              cpu: 100m
-              memory: 256Mi
-            limits:
-              cpu: "1"
-              memory: 512Mi
-          # /health: the process serves (liveness). /ready: an index exists; a cold
-          # start on an empty PVC embeds every note (~5 min) before traffic arrives.
-          startupProbe:
-            httpGet:
-              path: /health
-              port: http
-            periodSeconds: 5
-            failureThreshold: 12
-          readinessProbe:
-            httpGet:
-              path: /ready
-              port: http
-            periodSeconds: 10
-          livenessProbe:
-            httpGet:
-              path: /health
-              port: http
-            periodSeconds: 60
-            timeoutSeconds: 10
-            failureThreshold: 3
-          volumeMounts:
-            - name: data
-              mountPath: /data
-      volumes:
-        - name: data
-          persistentVolumeClaim:
-            claimName: wiki-index-data
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: wiki-index
-  namespace: ai
-  labels:
-    app.kubernetes.io/name: wiki-index
-spec:
-  selector:
-    app.kubernetes.io/name: wiki-index
-  ports:
-    - name: http
-      port: 8080
-      targetPort: http

kubernetes/apps/wiki-index/ingress.yaml

@@ -1,56 +0,0 @@
----
-# wiki-index.renner.uno — LAN-only (no dns.renner.uno/public label), machine
-# client (bin/wiki_rank.py), so Traefik BasicAuth like wiki-embed.renner.uno.
-# Gitea's webhook does not pass here; it targets the ClusterIP Service directly.
-apiVersion: traefik.io/v1alpha1
-kind: Middleware
-metadata:
-  name: wiki-index-basic-auth
-  namespace: ai
-spec:
-  basicAuth:
-    secret: wiki-index-basic-auth
-    realm: wiki-index
----
-# OpenBao key apps/ai/wiki-index-basic-auth, field `users` = htpasswd line.
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: wiki-index-basic-auth
-  namespace: ai
-spec:
-  refreshInterval: 1h
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: wiki-index-basic-auth
-  dataFrom:
-    - extract:
-        key: apps/ai/wiki-index-basic-auth
----
-apiVersion: networking.k8s.io/v1
-kind: Ingress
-metadata:
-  name: wiki-index
-  namespace: ai
-  annotations:
-    cert-manager.io/cluster-issuer: letsencrypt-cluster
-    traefik.ingress.kubernetes.io/router.middlewares: ai-wiki-index-basic-auth@kubernetescrd
-spec:
-  ingressClassName: traefik
-  tls:
-    - hosts:
-        - wiki-index.renner.uno
-      secretName: wiki-index-tls
-  rules:
-    - host: wiki-index.renner.uno
-      http:
-        paths:
-          - path: /
-            pathType: Prefix
-            backend:
-              service:
-                name: wiki-index
-                port:
-                  number: 8080

kubernetes/apps/wiki/deployment.yaml

@@ -0,0 +1,141 @@
+# Index snapshots per embedding model and the document catalog. A lost volume
+# costs one full sync from Gitea (~10 min).
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: wiki-data
+  namespace: ai
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: ceph-block
+  resources:
+    requests:
+      storage: 1Gi
+---
+# Wiki service (konairius/Wiki): mirrors the Notes repo from Gitea, embeds
+# wiki/ via wiki-embed, serves /rank, /api/v1 and the MudBlazor UI. Adapters
+# are selected in appsettings (Wiki__* env overrides); secrets arrive as env
+# from ExternalSecrets, the pod has no OpenBao access.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: wiki
+  namespace: ai
+  annotations:
+    # Keys, tokens and the Anthropic key arrive via envFrom; a rotated Secret rolls the pod.
+    secret.reloader.stakater.com/reload: wiki-gitea,wiki-api-keys,wiki-typesafe,wiki-anthropic
+  labels:
+    app.kubernetes.io/name: wiki
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: wiki
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: wiki
+    spec:
+      automountServiceAccountToken: false
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 65534
+        runAsGroup: 65534
+        fsGroup: 65534
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: wiki
+          image: gitea.renner.uno/konairius/wiki:build.1790322994 # {"$imagepolicy": "flux-system:wiki"}
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop: ["ALL"]
+          env:
+            - name: HOME
+              value: /tmp
+            - name: Wiki__Index__File__Directory
+              value: /data
+            - name: Wiki__NoteStore__Gitea__BaseUrl
+              value: https://gitea.renner.uno
+            - name: Wiki__NoteStore__Gitea__Repository
+              value: k.renner/notes
+            - name: Wiki__NoteStore__Gitea__Branch
+              value: main
+            - name: Wiki__Embeddings__Models__bge-m3__Url
+              value: http://wiki-embed.ai.svc.cluster.local:8080
+            # Pod networks: the CNP admits only the wiki-gate outpost and Gitea.
+            - name: Auth__ReverseProxy__TrustedProxies__0
+              value: fd10:244::/48
+            - name: Auth__ReverseProxy__TrustedProxies__1
+              value: 10.244.0.0/16
+          envFrom:
+            - secretRef:
+                name: wiki-gitea
+            - secretRef:
+                name: wiki-api-keys
+            - secretRef:
+                name: wiki-typesafe
+            - secretRef:
+                name: wiki-anthropic
+                optional: true
+          ports:
+            - name: http
+              containerPort: 8080
+          resources:
+            requests:
+              cpu: 200m
+              memory: 256Mi
+            limits:
+              cpu: "1"
+              memory: 1Gi
+          # /health: the process serves (liveness). /ready: catalog synced once and the
+          # active index loaded; a cold start syncs the whole vault first (~10 min).
+          startupProbe:
+            httpGet:
+              path: /ready
+              port: http
+            periodSeconds: 10
+            failureThreshold: 120
+          readinessProbe:
+            httpGet:
+              path: /ready
+              port: http
+            periodSeconds: 10
+          livenessProbe:
+            httpGet:
+              path: /health
+              port: http
+            periodSeconds: 60
+            timeoutSeconds: 10
+            failureThreshold: 3
+          volumeMounts:
+            - name: data
+              mountPath: /data
+            - name: tmp
+              mountPath: /tmp
+      volumes:
+        - name: data
+          persistentVolumeClaim:
+            claimName: wiki-data
+        - name: tmp
+          emptyDir: {}
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: wiki
+  namespace: ai
+  labels:
+    app.kubernetes.io/name: wiki
+spec:
+  selector:
+    app.kubernetes.io/name: wiki
+  ports:
+    - name: http
+      port: 8080
+      targetPort: http

kubernetes/apps/wiki/externalsecrets.yaml

@@ -0,0 +1,104 @@
+---
+# Gitea token (claude-bot, write:repository: the service creates raw/ files for
+# deposited facts) and the notes webhook secret. Secret keys are the
+# configuration paths the host reads.
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: wiki-gitea
+  namespace: ai
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: openbao
+  target:
+    name: wiki-gitea
+  data:
+    - secretKey: Wiki__NoteStore__Gitea__Token
+      remoteRef:
+        key: apps/ai/wiki-service
+        property: gitea_token
+    - secretKey: Wiki__Webhook__Secret
+      remoteRef:
+        key: apps/ai/wiki-index
+        property: webhook_secret
+---
+# One API key per machine identity: OpenBao apps/ai/wiki-api-keys, field = name.
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: wiki-api-keys
+  namespace: ai
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: openbao
+  target:
+    name: wiki-api-keys
+  data:
+    - secretKey: Auth__ApiKeys__workstation
+      remoteRef:
+        key: apps/ai/wiki-api-keys
+        property: workstation
+    - secretKey: Auth__ApiKeys__paperclip
+      remoteRef:
+        key: apps/ai/wiki-api-keys
+        property: paperclip
+---
+# TypeSafe System One key for the judge adapter (Wiki:Judges:Models:typesafe).
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: wiki-typesafe
+  namespace: ai
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: openbao
+  target:
+    name: wiki-typesafe
+  data:
+    - secretKey: Wiki__Judges__Models__typesafe__ApiKey
+      remoteRef:
+        key: apps/ai/wiki-typesafe
+        property: api_key
+---
+# Answer model: Wiki:AnswerModels:Models:claude-sonnet. The pod starts without it (optional
+# secretRef); the questions endpoint answers 503 until the key is placed.
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: wiki-anthropic
+  namespace: ai
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: openbao
+  target:
+    name: wiki-anthropic
+  data:
+    - secretKey: Wiki__AnswerModels__Models__claude-sonnet__ApiKey
+      remoteRef:
+        key: apps/ai/wiki-anthropic
+        property: api_key
+---
+# BasicAuth users for the machine door; same OpenBao key as the Python service used.
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: wiki-api-basic-auth
+  namespace: ai
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: openbao
+  target:
+    name: wiki-api-basic-auth
+  dataFrom:
+    - extract:
+        key: apps/ai/wiki-index-basic-auth

kubernetes/apps/wiki/gate-service.yaml

@@ -0,0 +1,20 @@
+# ExternalName mirror so the ai-namespace Ingress can backend to the wiki-gate
+# outpost in the authentik namespace.
+#
+#   browser → traefik (wiki.renner.uno)
+#           → ak-outpost-wiki-gate (ExternalName → authentik ns)
+#           → outpost validates the session + passkey step-up, sets X-Authentik-Username
+#           → http://wiki.ai.svc.cluster.local:8080
+#
+# In-cluster clients (CI agents, the Gitea webhook) address the Service directly.
+apiVersion: v1
+kind: Service
+metadata:
+  name: ak-outpost-wiki-gate
+  namespace: ai
+spec:
+  type: ExternalName
+  externalName: ak-outpost-wiki-gate.authentik.svc.cluster.local
+  ports:
+    - port: 9000
+      targetPort: 9000

kubernetes/apps/wiki/ingress.yaml

@@ -0,0 +1,75 @@
+# wiki.renner.uno — LAN-only (no dns.renner.uno/public label). Traefik backends
+# the wiki-gate outpost, never the Service; the ai CNP admits only that outpost.
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: wiki
+  namespace: ai
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-cluster
+spec:
+  ingressClassName: traefik
+  tls:
+    - hosts:
+        - wiki.renner.uno
+      secretName: wiki-tls
+  rules:
+    - host: wiki.renner.uno
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: ak-outpost-wiki-gate
+                port:
+                  number: 9000
+---
+# wiki-index.renner.uno — the machine door (bin/wiki_rank.py, wiki_ask.py):
+# Traefik BasicAuth, and the identity header stripped so only the outpost path
+# can carry X-Authentik-Username to the service.
+apiVersion: traefik.io/v1alpha1
+kind: Middleware
+metadata:
+  name: wiki-api-basic-auth
+  namespace: ai
+spec:
+  basicAuth:
+    secret: wiki-api-basic-auth
+    realm: wiki
+---
+apiVersion: traefik.io/v1alpha1
+kind: Middleware
+metadata:
+  name: wiki-api-strip-identity
+  namespace: ai
+spec:
+  headers:
+    customRequestHeaders:
+      X-Authentik-Username: ""
+---
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: wiki-api
+  namespace: ai
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-cluster
+    traefik.ingress.kubernetes.io/router.middlewares: ai-wiki-api-strip-identity@kubernetescrd,ai-wiki-api-basic-auth@kubernetescrd
+spec:
+  ingressClassName: traefik
+  tls:
+    - hosts:
+        - wiki-index.renner.uno
+      secretName: wiki-api-tls
+  rules:
+    - host: wiki-index.renner.uno
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: wiki
+                port:
+                  number: 8080

kubernetes/apps/wiki/kustomization.yaml

@@ -1,5 +1,7 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
+  - externalsecrets.yaml
   - deployment.yaml
+  - gate-service.yaml
   - ingress.yaml

kubernetes/bootstrap.sh

@@ -1955,6 +1955,7 @@ if phase_runs wait-apps; then
     wait_kustomization paperless           "Paperless (incl. pprl-pg CNPG + data PVCs)"          600
     wait_kustomization home-assistant      "Home Assistant (HAOS backup → native restore)"       180
     wait_kustomization audiobookshelf      "Audiobookshelf (incl. library/metadata PVCs)"        900
+    wait_kustomization navidrome           "Navidrome (incl. data + music library PVCs)"         600
     wait_kustomization immich              "Immich (incl. im-pg CNPG + library PVC)"             1800
 
     # Wait for cert-manager to finish issuing every Certificate. Kustomization

kubernetes/clusters/homelab/libation.yaml

@@ -5,7 +5,7 @@ metadata:
   namespace: flux-system
 spec:
   # Headless Libation CronJob that downloads new Audible books into
-  # the audiobookshelf-books PVC. Lives in the audiobookshelf namespace
+  # the audiobookshelf-books-bulk PVC. Lives in the audiobookshelf namespace
   # because k8s PVCs are namespace-scoped and we share the books volume
   # with the audiobookshelf Deployment. Depends on audiobookshelf so
   # the books PVC exists before the libation-bootstrap secret/CronJob

kubernetes/clusters/homelab/navidrome.yaml

@@ -0,0 +1,19 @@
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: navidrome
+  namespace: flux-system
+spec:
+  interval: 10m
+  retryInterval: 1m
+  timeout: 15m
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./kubernetes/apps/navidrome
+  prune: true
+  wait: true
+  dependsOn:
+    - name: infrastructure
+    - name: infrastructure-crds
+    - name: infrastructure-ceph-cluster

kubernetes/clusters/homelab/paperclip.yaml

@@ -0,0 +1,44 @@
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: paperclip
+  namespace: flux-system
+spec:
+  interval: 10m
+  retryInterval: 1m
+  timeout: 20m
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./kubernetes/apps/paperclip
+  prune: true
+  wait: true
+  dependsOn:
+    # Deliberately NOT dependsOn: monitoring, even though
+    # apps/paperclip/prometheus-read-rbac.yaml puts a Role and a RoleBinding in
+    # namespace `monitoring` (REN-86).
+    #
+    # The only thing those two objects need first is the namespace, and the
+    # namespace does not come from the `monitoring` Kustomization — it is
+    # declared in infrastructure/namespaces/namespaces.yaml and applied by
+    # `infrastructure`, listed right below. `infrastructure` runs wait: false, so
+    # it applies the namespaces on its first pass and only its healthChecks hold
+    # its readiness, which is strictly later; by the time this Kustomization
+    # starts, `monitoring` exists. Role and RoleBinding are core RBAC kinds, and
+    # a RoleBinding is allowed to name a Role that does not exist yet, so
+    # nothing on this path waits on kube-prometheus-stack's CRDs or workloads.
+    #
+    # The edge would not be free either. `monitoring` dependsOn `grafana-db` and
+    # itself waits (wait: true, 30m) on kube-prometheus-stack, Loki, Alloy and
+    # blackbox-exporter. Adding it would stop the Kustomization that carries the
+    # paperclip pod from reconciling whenever any of that is unhealthy — a real
+    # coupling bought for a bootstrap-ordering problem that does not exist.
+    - name: infrastructure
+    - name: infrastructure-crds
+    - name: infrastructure-ceph-cluster
+    - name: infrastructure-cnpg-barman
+  healthCheckExprs:
+    - apiVersion: postgresql.cnpg.io/v1
+      kind: Cluster
+      failed: "has(status.phase) && status.phase.startsWith('Failed')"
+      current: "has(status.phase) && (status.phase == 'Cluster in healthy state' || status.readyInstances > 0)"

kubernetes/clusters/homelab/uptime-monitor.yaml

@@ -0,0 +1,194 @@
+# Automated uptime monitoring (REN-2): Stakater IngressMonitorController keeps
+# UptimeRobot monitors in sync with `EndpointMonitor` custom resources. (Not
+# Ingress annotations — `monitor.stakater.com/enabled` does nothing in IMC v2;
+# there is no Ingress reconciler in the source, and docs/migration-guide.md
+# calls the annotation mechanism obsolete. The opt-in property survives intact:
+# creating an EndpointMonitor per host is more explicit than an annotation, and
+# the default state is still nothing-monitored.) This Kustomization holds the
+# API key, the controller Deployment, its ServiceAccount and its RBAC (REN-6),
+# and since REN-7 the `EndpointMonitor` objects themselves.
+#
+# THE ENDPOINTMONITORS ARE THE ONLY OBJECTS HERE THAT CAUSE AN EXTERNAL WRITE,
+# and `wait: true` cannot see whether that write succeeded. This is the same
+# trap as the ExternalSecret one documented further down, one layer out:
+# `EndpointMonitorStatus` is an empty struct in v2.2.13 and IMC never writes to
+# it, so an EndpointMonitor has no `observedGeneration` and no `Reconciling` /
+# `Stalled` condition — kstatus's generic rule returns Current for it
+# immediately. A Ready Kustomization therefore proves the objects were APPLIED,
+# and says nothing at all about whether UptimeRobot accepted the call. No
+# `healthCheckExprs` rule can fix that, because there is no status field to
+# read; the only evidence is the account itself or the controller's log. Do not
+# let a green Kustomization stand in for either.
+#
+# A DEDICATED Kustomization rather than adding the ExternalSecret to
+# ./kubernetes/infrastructure/post-crds, which is where the other
+# infrastructure-layer ExternalSecrets live. That directory is applied by the
+# `infrastructure-crds` Kustomization, which runs `wait: true` with no explicit
+# healthChecks, so EVERY object under it must go healthy or the whole
+# Kustomization reports NotReady — and 19 other Kustomizations in this directory
+# dependsOn it. An ExternalSecret whose OpenBao path is not yet populated never
+# turns Ready, so putting this key there would put `homelab/infrastructure/uptime-robot`
+# in the critical path of most of the cluster. Here, the only thing a missing or
+# misnamed key can stall is this Kustomization, which nothing depends on.
+#
+# `wait: true` is kept below rather than dropped, but ON ITS OWN IT DOES NOT
+# MAKE A FAILED SYNC VISIBLE — an earlier revision of this comment claimed it
+# did, and that was wrong (REN-100). Flux ships no status reader for
+# `ExternalSecret`, so kstatus falls back to its generic rule, which looks only
+# at `status.observedGeneration` and at conditions of type `Reconciling` /
+# `Stalled`. ESO sets none of those three: a failed sync is reported as
+# `Ready=False, reason=SecretSyncedError`. kstatus therefore returns Current and
+# the Kustomization goes Ready with no Secret behind it.
+#
+# In-repo proof rather than a doc citation: `llama-server` is also `wait: true`
+# and carries the `ai/wiki-embed-basic-auth` ExternalSecret
+# (apps/llama-server/wiki-embed-ingress.yaml). That key sat in ESO retry backoff
+# through ~10 failures on 2026-09-20 and needed a manual `force-sync`. Its
+# Kustomization stayed Ready throughout, which is why a human had to notice.
+#
+# So `wait: true` is paired with the `healthCheckExprs` below, which is the
+# mechanism this repo already uses for CRs kstatus cannot read — CNPG `Cluster`
+# (paperless, immich, paperclip, authentik, woodpecker, monitoring), MariaDB
+# (wordpress) and the Ceph CRs (infrastructure). Now a key that fails to sync
+# holds this Kustomization NotReady, and it is safe to fail loudly here
+# precisely because the blast radius is this Kustomization alone: nothing
+# `dependsOn` it.
+#
+# That gate earns its keep now that the controller is here (REN-6) rather than
+# only the key. IMC GETs its whole configuration from the Secret once, at
+# startup, through the manager's APIReader, and never revalidates. Which failure
+# it papers over depends on which one happens:
+#
+#   * Secret absent (first sync never succeeded — `creationPolicy` is the default
+#     Owner, so no successful sync means no Secret at all): IMC panics in
+#     `LoadControllerConfig` and the pod CrashLoopBackOffs. That one is loud.
+#   * Secret present but stale, because a LATER refresh failed: the pod starts
+#     against the old value, reports Available, and the first sign of trouble is
+#     a rejected UptimeRobot call on some future reconcile. Nothing about
+#     Deployment health says the key is current.
+#
+# The second case is the live one — the Secret exists today — and the CEL rule
+# below is what turns it into a NotReady Kustomization instead of a green one.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: uptime-monitor
+  namespace: flux-system
+spec:
+  interval: 10m
+  # 2m, not the 1m most app Kustomizations use: what stalls this Kustomization
+  # is an ExternalSecret against a key a human may still be placing in OpenBao,
+  # and a retry every minute against a 404 is just noise in the ESO and Flux
+  # logs.
+  retryInterval: 2m
+  timeout: 5m
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./kubernetes/apps/uptime-monitor
+  prune: true
+  wait: true
+  # No name/namespace qualifier, so this is evaluated against every
+  # ExternalSecret in this Kustomization's inventory. That is exactly one
+  # (apps/uptime-monitor/externalsecret.yaml) — the IMC Deployment added in
+  # REN-6 brought no second one — and any ExternalSecret added here later should
+  # be held to the same bar anyway.
+  #
+  # `external-secrets.io/v1` is the version every one of the 54 ExternalSecret
+  # manifests in this repo is written against, so it is the GVK Flux records in
+  # the inventory and the one that has to be matched here.
+  #
+  # Both expressions are guarded with `has(status.conditions)`, copying
+  # wordpress.yaml. Without the guard, CEL errors instead of evaluating while
+  # `status` is still empty — which is the state a freshly applied ExternalSecret
+  # is in, i.e. exactly the case this check exists for. Unguarded, the
+  # Kustomization would report a CEL evaluation error rather than a clean
+  # in-progress.
+  #
+  # `failed` NAMES THE TWO REASONS ESO STOPS RETRYING ON, AND DELIBERATELY LEAVES
+  # OUT `SecretSyncedError` (REN-106). The first cut of this block fired on any
+  # `Ready=False` at all, and that path is undampened: the `flux-failures` Alert
+  # (infrastructure/flux-notifications/alerts.yaml) forwards error events from
+  # every Kustomization to Alertmanager, and a Flux health check has no `for:`
+  # window, so a single bad poll would have mailed the owner.
+  #
+  # Read off the ESO the cluster actually runs, v2.11.0 (the chart range
+  # `>=2.7.0 <3.0.0` in infrastructure/external-secrets/helmrelease.yaml resolves
+  # there today). On the Secret target path ESO sets `Ready=False` with exactly
+  # three reasons, and what separates them is whether it intends to try again
+  # (pkg/controllers/externalsecret/externalsecret_controller.go):
+  #   SecretSyncedError  — provider read, template or update failed. Almost every
+  #                        site returns the error, so controller-runtime requeues
+  #                        with backoff. This is the reason an OpenBao blip, a
+  #                        token renewal AND a not-yet-populated key all produce —
+  #                        the transient case and the REN-100 case share one
+  #                        string, so the string cannot tell them apart.
+  #   SecretOwnedByOther — the target Secret is owned by something else.
+  #   SecretImmutable    — the target Secret is immutable and needs an update.
+  # The last two return `nil`, commented in ESO's own source as "cant be fixed by
+  # retrying": ESO has given up, nothing is in flight, so failing at once is
+  # correct and cannot be a flap. Narrowing to `reason == 'SecretSyncedError'`
+  # instead — REN-106's first suggestion — would have inverted that, keeping the
+  # one flap-prone reason and dropping the two that cannot flap.
+  #
+  # Two details that make `failed` an allowlist of reasons rather than the
+  # complement of `SecretSyncedError`: one `SecretSyncedError` site
+  # (`ErrSecretSetCtrlRef`) also returns `nil`, so that string is not a reliable
+  # "still trying" signal either way; and the generic-target path (`spec.target`
+  # pointing at something other than a Secret) has its own reason set,
+  # `ResourceSyncedError`, which is not used here — this ExternalSecret targets a
+  # Secret — and is left to the in-progress path if it ever is.
+  #
+  # A `SecretSyncedError` is dampened, not dropped: it matches neither
+  # expression, so Flux reports in-progress and keeps polling until `timeout: 5m`
+  # above expires and the Kustomization fails. That timeout is the `for:` window
+  # this path otherwise has none of — a blip that clears inside it never reaches
+  # the owner, a sync that is genuinely stuck still does, about five minutes
+  # later.
+  #
+  # EVERY SHAPE AN ExternalSecret STATUS CAN TAKE GETS A VERDICT FROM THE TWO
+  # EXPRESSIONS BELOW. There are three, not two, and the third one — no `Ready`
+  # condition at all — is the one a reader is most likely to miss:
+  #
+  #   1. no conditions yet (applied, first reconcile unfinished): the `has()`
+  #      guard makes both expressions false -> in-progress, bounded by
+  #      `timeout: 5m`.
+  #   2. `Ready=True`: `current` -> Current. Note this includes reasons
+  #      `SecretDeleted` and `SecretMissing`, which ESO reports as Ready=True
+  #      (markAsDone) even though no target Secret exists. Reaching them needs
+  #      `deletionPolicy: Delete`/`Merge` or `creationPolicy: Merge`; this
+  #      ExternalSecret takes the defaults (Retain/Owner), where a vanished
+  #      provider key is a `SecretSyncedError` instead. Anyone who later sets
+  #      either policy here reopens REN-100 through this expression.
+  #   3. `Ready=False`: `failed` for the two non-retrying reasons above,
+  #      in-progress for the rest.
+  #
+  # No `inProgress` expression: only `apiVersion` and `current` are required by
+  # the CRD (flux-system/gotk-components.yaml, kustomize-controller v1.9.5), and
+  # Flux evaluates inProgress -> failed -> current and returns in-progress when
+  # nothing matches (fluxcd/pkg runtime/cel, StatusEvaluator.Evaluate). An
+  # `inProgress` naming shapes 1 and 3 would therefore be a no-op on all three.
+  #
+  # "conditions present but no `Ready` condition" is NOT a fourth shape (REN-106).
+  # On a live object ESO only ever writes conditions of type `Ready`; the
+  # `Deleted` type is never persisted — it is built on a throwaway object in the
+  # `IsNotFound` branch of Reconcile purely to retire that object's metric series
+  # ("this does not actually set the condition on the ExternalSecret, because it
+  # does not exist"). So `externalsecret_status_condition{condition="Deleted"}`
+  # outlives the object it names, until the ESO pod restarts.
+  # `ai/wiki-index-basic-auth` is that: retired from git in 419cbed3, target
+  # Secret gone from the namespace, sync counter frozen, still a series in an ESO
+  # pod older than the prune. Any alert built on this metric (REN-103) has to
+  # exclude that shape; this health check never sees it.
+  healthCheckExprs:
+    - apiVersion: external-secrets.io/v1
+      kind: ExternalSecret
+      failed: "has(status.conditions) && status.conditions.exists(c, c.type == 'Ready' && c.status == 'False' && (c.reason == 'SecretOwnedByOther' || c.reason == 'SecretImmutable'))"
+      current: "has(status.conditions) && status.conditions.exists(c, c.type == 'Ready' && c.status == 'True')"
+  dependsOn:
+    # ESO itself, and the `uptime-monitor` namespace (declared centrally in
+    # kubernetes/infrastructure/namespaces/namespaces.yaml).
+    - name: infrastructure
+    # The `openbao` ClusterSecretStore, including the conditions.namespaces
+    # allowlist entry this namespace needs.
+    - name: infrastructure-crds

kubernetes/clusters/homelab/wiki.yaml

@@ -1,17 +1,17 @@
 apiVersion: kustomize.toolkit.fluxcd.io/v1
 kind: Kustomization
 metadata:
-  name: wiki-index
+  name: wiki
   namespace: flux-system
 spec:
   interval: 10m
   retryInterval: 1m
-  # First start on an empty PVC embeds every note (~5 min) before /health is 200.
-  timeout: 20m
+  # First start on an empty PVC syncs and embeds the whole vault before /ready is 200.
+  timeout: 25m
   sourceRef:
     kind: GitRepository
     name: flux-system
-  path: ./kubernetes/apps/wiki-index
+  path: ./kubernetes/apps/wiki
   prune: true
   wait: true
   dependsOn:

kubernetes/infrastructure/cilium/helmrelease.yaml

@@ -72,10 +72,29 @@ spec:
       # alertable beforehand.
       #
       # That reasoning is about the metrics, which this line provides. The
-      # ServiceMonitor is only the scrape wiring and is declared standalone in
-      # apps/monitoring-rules instead — the same treatment velero/helmrelease
-      # already gives its own, and for the same reason: the chart's template is
-      # gated on monitoring.coreos.com/v1 existing at install time.
+      # scrape wiring — both the ServiceMonitor and the metrics Service — is
+      # declared standalone in apps/monitoring-rules/cilium-servicemonitor.yaml
+      # instead. For the ServiceMonitor that is the same treatment
+      # velero/helmrelease already gives its own, and for the same reason: the
+      # chart's template is gated on monitoring.coreos.com/v1 existing at
+      # install time.
+      #
+      # The Service is there for a different reason, and it is the one that
+      # bit us (REN-147). templates/cilium-agent/service.yaml renders only when
+      # `prometheus.enabled` AND (`serviceMonitor.enabled` OR `metricsService`).
+      # With serviceMonitor.enabled false and metricsService at its default,
+      # no Service was rendered, so the standalone ServiceMonitor selected
+      # nothing and cilium_* had zero series for as long as this file has
+      # looked like this. Turning either flag on here is the chart-native fix
+      # and would render exactly that one object — but it is a values change,
+      # so helm-controller would upgrade the CNI release and three-way-merge
+      # against live objects, and the reloader annotation above turns any
+      # resulting cilium-config change into a rolling restart of every agent.
+      # Hence the Service is applied by the apps Kustomization, which cannot.
+      #
+      # If you ever do set `serviceMonitor.enabled` or `metricsService` here,
+      # delete the matching object from cilium-servicemonitor.yaml in the same
+      # change.
       #
       # Here it is worse than for velero. Cilium's chart does not skip the
       # template, it aborts rendering — "Service Monitor requires

kubernetes/infrastructure/image-automation/README.md

@@ -49,7 +49,8 @@ existing `flux-system` GitRepository whose Secret was created by
 | redis-immich | `7.4.x` | `-alpine` suffix extract; tight to match LXC 225 compose |
 | redis-paperless | `8.x` | `-alpine` suffix extract |
 | ceph | `19.x` | v-prefix filter; rejects date-suffixed builds |
-| tika | `3.x` | 4-component → 3-component extract; theoretical loss of 4th-component precision (always `0` in history) |
+| tika | `3.x` | 4-component tags; `.0` image-revision pinned in the filter so the extracted key is unique |
+| lidarr | Lidarr `3.x` | LinuxServer `-ls<n>` rebuilds; synthetic `<build>.<ls>.0` semver key — major bounded in the filter, not the range |
 | protonmail-bridge | `3.x` | `-1` build-suffix extract |
 | firefox | `1.x` | `-rolling-daily` suffix extract; rejects arch-prefixed duplicates |
 | mmwave-configurator | `2.x` | Plain 3-component filter (rejects `addon-`/`-armv7`/`-arm64`/`-amd64` variants) |
@@ -120,6 +121,69 @@ unaffected.
   releases land in master immediately. Mitigation: ImagePolicy ranges
   must be tight (`semver: range: "2.x"` not `"*"`) so a major bump
   requires manual policy widening.
+- **`extract` must produce a total order over the tags `pattern`
+  matches.** Any part of the tag that varies has to be either in the
+  extracted key or excluded by the filter. A component that is matched
+  but not extracted makes two different tags share one key, and the
+  winner is then an arbitrary tie-break that can flap and downgrade a
+  running app with no human in the loop. This is not hypothetical: it
+  auto-committed a lidarr downgrade (ls42 → ls11) to master in
+  `b858b4aa` and rolled the pod. See `lidarr.yaml` for the two-field
+  case (synthetic semver key) and `tika.yaml` / `protonmail-bridge.yaml`
+  for the narrow-the-filter case.
 - Image findings still surface in the security-scan agent's report
   (Skipped / "Flux image-automation owns this") so they remain visible
   even when an image isn't yet enrolled in automation here.
+
+## What watches this
+
+Worth stating explicitly, because for a while nothing did and each
+plausible reporter deferred to another: the update-audit agent skips
+`$imagepolicy`-marked images (`ci/agents/update-audit/prompt.md`), the
+security scan records them as "Flux image-automation owns this" (above),
+and both are correct to — an enrolled image is supposed to be this
+directory's problem.
+
+What exists today:
+
+- `flux-notifications/alerts.yaml` emails error-severity events from
+  every image kind. Immediate, but edge-triggered: one mail at the
+  moment of transition, then silence. Nothing says "still broken twelve
+  hours later" and nothing answers "is anything not-Ready right now".
+- `apps/monitoring-rules/flux-podmonitor.yaml` scrapes the controllers,
+  so reconcile latency and controller-level error counts are queryable.
+  Readiness is **not** on that endpoint: at the controller versions
+  pinned in `clusters/homelab/flux-system/gotk-components.yaml`,
+  `gotk_reconcile_condition` and `gotk_suspend_status` are declared by
+  the shared runtime library and recorded by nothing, so they export no
+  series however the scrape is written.
+- `apps/monitoring/helmrelease-prometheus.yaml` therefore turns on
+  kube-state-metrics `customResourceState` over eight Flux kinds, which
+  is where per-object readiness now comes from: `gotk_resource_info`
+  with `ready` and `suspended` labels. For ImagePolicy it also carries
+  the resolved `tag`, and each ImageRepository gets
+  `gotk_image_repository_tag_count`.
+- `apps/monitoring-rules/prometheusrule-flux.yaml` builds the standing
+  alerts on those series. `FluxImageAutomationNotReady` fires after an
+  ImageRepository, ImagePolicy or ImageUpdateAutomation has been
+  not-Ready for 2h (two scan intervals for the 31 of 34 repositories at
+  `interval: 1h`); `FluxResourceNotReady` covers every other Flux kind
+  at 1h; `FluxImagePolicyMetricsMissing` is the dead man's switch that
+  says so if the metrics themselves stop. This is the "still broken two
+  hours later" signal the edge-triggered mail above cannot give.
+
+What still does **not** exist: an *alert* for a policy that is perfectly
+Ready while its own filter has frozen it — `tika` if upstream ships a
+3.x. The series for it are now recorded
+(`flux:image_policy_tag_frozen:14d` and the two rules under it), but no
+threshold is attached yet, because none is derivable from the history
+available today. Measured 2026-09-26 against the 191 automated bump
+commits and the upstream registries: a flat duration cannot work (gatus
+went 127 days between bumps while healthy), a 10% tag-count growth
+comparator fires on three variant-churning repositories and misses
+`tika` at 3.0%, relaxing it to one new tag adds `velero-plugin-for-aws`,
+whose pin is deliberate, and `.status.lastScanResult.latestTags` is
+sorted alphabetically rather than newest-first so it cannot substitute.
+The recording rules exist to accumulate the per-repository `tagCount`
+history that a defensible threshold has to be derived from; the alert is
+tracked separately.

kubernetes/infrastructure/image-automation/kustomization.yaml

@@ -9,11 +9,14 @@ resources:
   - gotenberg.yaml
   - home-assistant.yaml
   - immich.yaml
+  - lidarr.yaml
   - matter-server.yaml
   - mmwave-configurator.yaml
   - music-assistant.yaml
+  - navidrome.yaml
   - netbird-ingress.yaml
   - paperless-ngx.yaml
+  - paperclip.yaml
   - protonmail-bridge.yaml
   - mosquitto.yaml
   - redis.yaml
@@ -24,5 +27,6 @@ resources:
   - valkey.yaml
   - velero-plugin-for-aws.yaml
   - vsftpd.yaml
+  - wiki.yaml
   - woodpecker.yaml
   - wyoming.yaml

kubernetes/infrastructure/image-automation/lidarr.yaml

@@ -0,0 +1,57 @@
+apiVersion: image.toolkit.fluxcd.io/v1
+kind: ImageRepository
+metadata:
+  name: lidarr
+  namespace: flux-system
+spec:
+  image: lscr.io/linuxserver/lidarr
+  interval: 1h
+---
+# Tags are <maj>.<min>.<patch>.<build>-ls<n>. Two numbers move, and BOTH
+# must be in the sort key:
+#   <build>  upstream Lidarr's CI build counter, monotonic (4658 -> 4711
+#            -> 4836 -> 4866 -> 4875 over the visible history).
+#   ls<n>    LinuxServer's rebuild counter. Bumped weekly for security
+#            rebuilds of the same upstream build. Not globally unique and
+#            NOT monotonic on its own: it resets on some upstream bumps
+#            (2.12.4.4658-ls50 -> 2.13.3.4711-ls1) and has been reused
+#            across builds (3.0.1.4866-ls11 and 3.1.0.4875-ls11).
+#
+# Extracting <build> alone is not a total order: all 32 published ls
+# rebuilds of build 4875 collapse to the same key and the winner is an
+# arbitrary tie-break. That fired in production on 2026-09-25 (b858b4aa),
+# auto-committing ls42 -> ls11 and rolling the pod.
+#
+# `numerical` cannot order two fields, and `alphabetical` over a
+# concatenation would mis-sort on width (ls9 vs ls11). So the key is a
+# synthetic semver <build>.<ls>.0 and the policy is semver. Comparison is
+# per-component and numeric, so it is width-safe and orders exactly as
+# (build, ls) lexicographic -- build dominates, which is what makes the
+# ls reset above harmless.
+#
+# The Lidarr major is bounded in filterTags, not in the range, because
+# the range operates on the synthetic key where "3" would mean build 3.
+# A Lidarr 4.x bump therefore needs a deliberate edit here, per the
+# "ranges must be tight" rule in README.md.
+#
+# The range floor is the build we are on. It never needs bumping (it only
+# rejects sorting backwards past 4875) and is belt-and-braces against a
+# future key mistake, not a version bound.
+#
+# Residual risk: if LinuxServer ever reset ls within one build number,
+# that would still read as a downgrade. It has not done so in the visible
+# history -- every reset coincided with a build change.
+apiVersion: image.toolkit.fluxcd.io/v1
+kind: ImagePolicy
+metadata:
+  name: lidarr
+  namespace: flux-system
+spec:
+  imageRepositoryRef:
+    name: lidarr
+  filterTags:
+    pattern: '^3\.\d+\.\d+\.(?P<build>\d+)-ls(?P<ls>\d+)$'
+    extract: '${build}.${ls}.0'
+  policy:
+    semver:
+      range: ">=4875.0.0"