fix(paperless-eval): score a right answer the matcher threw away

The AI engine reports an unresolved guess as a name in
`suggested_correspondents` rather than an id in `correspondents`, and
paperless_ai.matching only resolves it when difflib.get_close_matches
clears MATCH_THRESHOLD = 0.8. A correct but more verbose answer does not:
"Generali Deutschland Krankenversicherung AG" fails against the existing
"Generali AG", leaving `correspondents` empty.

Scored as it was, that counts as a miss, which would have blamed the
model for a matcher limit. Score both instead — the id as offered, and
the name matched wordwise against the true label — so the two causes stay
separable. The classic engine emits no `suggested_*` fields, so its two
columns coincide and the comparison stays fair either way.

Names now come out of postgres alongside the ids, tab-separated because
'|' occurs in correspondent names.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

.woodpecker/build-ci-images.yml

@@ -19,13 +19,12 @@
 # the admin account could (#330).
 #
 # The token is still OWNED by gitea_admin, because Gitea scopes packages to
-# a user or org namespace and most of these images live under `gitea_admin/*`.
-# The rebuild-critical ones additionally publish to `konairius/` — an org
-# created for the ghcr mirror path, with gitea_admin as a member so this same
-# token can push there. The full gitea_admin/* -> org migration is a
-# 38-reference change tracked separately in #330 rather than bundled here.
-# Note this pipeline runs no LLM agent, so the exfil path that motivated the
-# other cutovers does not apply to it.
+# a user or org namespace and these images live under `gitea_admin/*` — a
+# different account cannot publish there. Moving them to a dedicated org is
+# the remaining cleanup; it is a 38-reference migration across pipelines and
+# Flux-deployed CronJobs, so it is tracked separately in #330 rather than
+# bundled here. Note this pipeline runs no LLM agent, so the exfil path that
+# motivated the other cutovers does not apply to it.
 when:
   - event: push
     path:
@@ -88,30 +87,6 @@ steps:
           --destination gitea.renner.uno/gitea_admin/ci-k8s:$TAG \
           --destination gitea.renner.uno/gitea_admin/ci-k8s:latest
 
-  - name: build-vmtest
-    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
-    directory: ci/images/vmtest
-    depends_on: []
-    when:
-      - event: deployment
-      - event: push
-        path: ["ci/images/vmtest/**", ".woodpecker/build-ci-images.yml"]
-    environment:
-      GITEA_USER: gitea_admin
-      GITEA_TOKEN:
-        from_secret: ci_images_token
-    commands:
-      - mkdir -p /kaniko/.docker
-      - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
-      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
-      - TAG=$(date +%Y-%m-%d)
-      - |
-        /kaniko/executor \
-          --context . \
-          --dockerfile Containerfile \
-          --destination gitea.renner.uno/gitea_admin/ci-vmtest:$TAG \
-          --destination gitea.renner.uno/gitea_admin/ci-vmtest:latest
-
   - name: build-tofu
     image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: ci/images/tofu
@@ -172,28 +147,6 @@ steps:
   # not for who uses it — its first consumer is the zigbee2mqtt seed-data init
   # container, but nothing about it is zigbee-specific. Exists so tools come from
   # the image rather than an `apk add` on every container start.
-  #
-  # Published THREE times, and the third one is not a typo. Its consumers — the
-  # zigbee2mqtt seed-data init container and the trmnl renderer — start while
-  # the cluster is still being rebuilt, and at that moment gitea.renner.uno does
-  # not exist, because Gitea runs inside the cluster being built. A cold-rebuild
-  # rehearsal caught both in ImagePullBackOff.
-  #
-  # So there is a copy on ghcr.io. But the manifests must NOT point at it:
-  # normal operation pulls from the local registry, and an external reference
-  # would make every steady-state pull leave the network. The recovery source
-  # and the everyday source are different things, exactly as they are for Flux's
-  # git source.
-  #
-  # Talos resolves that with machine.registries.mirrors, which takes an ordered
-  # endpoint list — gitea first, ghcr second (bootstrap.sh). Containerd keeps
-  # the repository path when it falls through, so the two registries have to
-  # agree on it: `gitea_admin/alpine-jq` cannot be served by ghcr, where the
-  # namespace is fixed to the account name. Hence the second Gitea destination
-  # under konairius/, which the manifests reference and which matches the ghcr
-  # path exactly.
-  #
-  # The gitea_admin/ copy stays until nothing references it any more.
   - name: build-alpine-jq
     image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: ci/images/alpine-jq
@@ -206,47 +159,23 @@ steps:
       GITEA_USER: gitea_admin
       GITEA_TOKEN:
         from_secret: ci_images_token
-      GHCR_USER: konairius
-      GHCR_TOKEN:
-        from_secret: ghcr_mirror_token
     commands:
       - mkdir -p /kaniko/.docker
       - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
-      - GH_AUTH=$(printf '%s:%s' "$GHCR_USER" "$GHCR_TOKEN" | base64 | tr -d '\n')
-      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"},"ghcr.io":{"auth":"%s"}}}' "$AUTH" "$GH_AUTH" > /kaniko/.docker/config.json
+      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
       - TAG=$(date +%Y-%m-%d)
       - |
         /kaniko/executor \
           --context . \
           --dockerfile Containerfile \
           --destination gitea.renner.uno/gitea_admin/alpine-jq:$TAG \
-          --destination gitea.renner.uno/gitea_admin/alpine-jq:latest \
-          --destination gitea.renner.uno/konairius/alpine-jq:$TAG \
-          --destination gitea.renner.uno/konairius/alpine-jq:latest \
-          --destination ghcr.io/konairius/alpine-jq:$TAG \
-          --destination ghcr.io/konairius/alpine-jq:latest
+          --destination gitea.renner.uno/gitea_admin/alpine-jq:latest
 
   # Runtime image, not CI tooling: upstream containernetworking/plugins binaries
   # baked in, consumed by the multus cni-plugins-installer DaemonSet. Replaces a
   # node-side fetch from GitHub that had NO checksum verification and sat on the
   # critical path of every node rebuild. The download now happens here, once,
   # where the sha256 is checked — see the Containerfile for the full rationale.
-  #
-  # This one ALSO publishes to ghcr.io. A cold-rebuild rehearsal proved why: the
-  # DaemonSet could not pull from gitea.renner.uno, because Gitea runs inside the
-  # very cluster being rebuilt. Without the plugins there is no macvlan, so
-  # Multus cannot attach ceph-san, so Rook's network canary never starts, so the
-  # CephCluster sits at "Configuring Ceph Mons" — and OpenBao waits forever for a
-  # PVC that will never bind. Moving the download here fixed the integrity gap
-  # but quietly swapped github.com on the critical path for something strictly
-  # worse: a registry that by definition does not exist during a recovery. The
-  # GHCR copy is public, so a bare node can pull it with no credential and no
-  # Gitea.
-  #
-  # The test that proved this also proved the rule is not specific to this
-  # image. alpine-jq and alpine-openssl mirror for the same reason — see the
-  # note on build-alpine-jq. Anything an app needs while the cluster is still
-  # being built cannot live only in that cluster's own registry.
   - name: build-cni-plugins
     image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: ci/images/cni-plugins
@@ -259,36 +188,23 @@ steps:
       GITEA_USER: gitea_admin
       GITEA_TOKEN:
         from_secret: ci_images_token
-      GHCR_USER: konairius
-      GHCR_TOKEN:
-        from_secret: ghcr_mirror_token
     commands:
       - mkdir -p /kaniko/.docker
       - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
-      - GH_AUTH=$(printf '%s:%s' "$GHCR_USER" "$GHCR_TOKEN" | base64 | tr -d '\n')
-      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"},"ghcr.io":{"auth":"%s"}}}' "$AUTH" "$GH_AUTH" > /kaniko/.docker/config.json
+      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
       - TAG=$(date +%Y-%m-%d)
       - |
         /kaniko/executor \
           --context . \
           --dockerfile Containerfile \
           --destination gitea.renner.uno/gitea_admin/cni-plugins:$TAG \
-          --destination gitea.renner.uno/gitea_admin/cni-plugins:latest \
-          --destination gitea.renner.uno/konairius/cni-plugins:$TAG \
-          --destination gitea.renner.uno/konairius/cni-plugins:latest \
-          --destination ghcr.io/konairius/cni-plugins:$TAG \
-          --destination ghcr.io/konairius/cni-plugins:latest
+          --destination gitea.renner.uno/gitea_admin/cni-plugins:latest
 
   # Runtime image, not CI tooling: alpine + the openssl CLI, for shell scripts
   # needing s_client/x509. Named for what it is; first consumer is the paperless
   # fetch-bridge-cert init container. Replaces an `apk add openssl` that ran on
   # every paperless pod start. The openssl CLI is genuinely absent from the alpine
   # base, so this one needs an image rather than a busybox substitute.
-  #
-  # Mirrored to ghcr.io for the reason given on build-alpine-jq: it is an init
-  # container, so paperless does not start without it, and during a rebuild
-  # gitea.renner.uno is not there to serve it. A cold-rebuild rehearsal left
-  # paperless in Init:ErrImagePull on exactly this image.
   - name: build-alpine-openssl
     image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: ci/images/alpine-openssl
@@ -301,25 +217,17 @@ steps:
       GITEA_USER: gitea_admin
       GITEA_TOKEN:
         from_secret: ci_images_token
-      GHCR_USER: konairius
-      GHCR_TOKEN:
-        from_secret: ghcr_mirror_token
     commands:
       - mkdir -p /kaniko/.docker
       - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
-      - GH_AUTH=$(printf '%s:%s' "$GHCR_USER" "$GHCR_TOKEN" | base64 | tr -d '\n')
-      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"},"ghcr.io":{"auth":"%s"}}}' "$AUTH" "$GH_AUTH" > /kaniko/.docker/config.json
+      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
       - TAG=$(date +%Y-%m-%d)
       - |
         /kaniko/executor \
           --context . \
           --dockerfile Containerfile \
           --destination gitea.renner.uno/gitea_admin/alpine-openssl:$TAG \
-          --destination gitea.renner.uno/gitea_admin/alpine-openssl:latest \
-          --destination gitea.renner.uno/konairius/alpine-openssl:$TAG \
-          --destination gitea.renner.uno/konairius/alpine-openssl:latest \
-          --destination ghcr.io/konairius/alpine-openssl:$TAG \
-          --destination ghcr.io/konairius/alpine-openssl:latest
+          --destination gitea.renner.uno/gitea_admin/alpine-openssl:latest
 
   # Heating CAN readers: open3e (Vitocal/VX3) + E3onCAN (E380 meter), one image
   # run as two services on edge-1. Replaces pip-into-venv on the host, where an
@@ -349,30 +257,3 @@ steps:
           --dockerfile Containerfile \
           --destination gitea.renner.uno/gitea_admin/heating-can:$TAG \
           --destination gitea.renner.uno/gitea_admin/heating-can:latest
-
-  # netbird-proxy-services: the controller that reconciles NetBird reverse-proxy
-  # services (public Ingress hosts -> TLS passthrough -> Traefik ClusterIP).
-  # Runtime artifact like heating-can; same registry so the audit tracks it.
-  - name: build-netbird-proxy-services
-    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
-    directory: ci/images/netbird-proxy-services
-    depends_on: []
-    when:
-      - event: deployment
-      - event: push
-        path: ["ci/images/netbird-proxy-services/**", ".woodpecker/build-ci-images.yml"]
-    environment:
-      GITEA_USER: gitea_admin
-      GITEA_TOKEN:
-        from_secret: ci_images_token
-    commands:
-      - mkdir -p /kaniko/.docker
-      - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
-      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
-      - TAG=$(date +%Y-%m-%d)
-      - |
-        /kaniko/executor \
-          --context . \
-          --dockerfile Containerfile \
-          --destination gitea.renner.uno/gitea_admin/netbird-proxy-services:$TAG \
-          --destination gitea.renner.uno/gitea_admin/netbird-proxy-services:latest

.woodpecker/deploy-netbird-proxy.yml

@@ -10,18 +10,6 @@ when:
       # install-netbird-proxy.yml also applies the edge bouncer.
       - "ansible/roles/crowdsec-edge-bouncer/**"
       - "ansible/plays/install-netbird-proxy.yml"
-      # The variables that steer those roles, not just the roles themselves.
-      # Without these two, a change that only touches inventory never deploys
-      # and nothing says so: arming the bouncer lives in host_vars, and its
-      # LAPI key lives in group_vars, so both would sit in git looking applied
-      # while the hosts kept running the previous configuration.
-      #
-      # pve_hosts.yml is deliberately NOT listed. It is the single inventory
-      # file for the whole fleet, so listing it would redeploy the proxies for
-      # any unrelated host edit. The cost is that a change to [netbird_proxy]
-      # group membership needs a manual deployment run.
-      - "ansible/inventory/host_vars/proxy-*.yml"
-      - "ansible/inventory/group_vars/netbird_proxy/**"
       - ".woodpecker/deploy-netbird-proxy.yml"
   - event: deployment
     evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "deploy-netbird-proxy"'

ansible/roles/crowdsec-edge-bouncer/defaults/main.yml

@@ -28,23 +28,14 @@ crowdsec_bouncer_sha256: f86e4b72693549d99f40a9402abefb894108f047a3fbe6e72fada25
 crowdsec_bouncer_api_url: https://crowdsec.renner.uno/
 crowdsec_bouncer_api_key: "{{ vault_crowdsec_edge_bouncer_key }}"
 
-# Enforcing. The bouncer creates two nftables tables (ip crowdsec, ip6
-# crowdsec6) holding the decisions it pulled, touches no existing chain and
-# sets no chain policy — SSH and :443 stay open unless the source is banned.
+# STARTS INERT, and that is the point.
 #
-# `dry-run` is the other value: it pulls and logs what it would block without
-# creating a single rule. That is what a new host should be given before it
-# enforces, and what an existing one can be dropped back to, per host through
-# inventory/host_vars, without disarming the rest.
-#
-# Worth knowing before arming anything new: the first bouncer on a path
-# enforces the whole CAPI set at once — around 15k addresses today — so the
-# false-positive window belongs before the switch, not after it. And because
-# this chain sits on the INPUT hook while these hosts run no other firewall,
-# there is no conntrack rule accepting established flows ahead of it: a banned
-# address is dropped even when it is answering our own outbound connection.
-# That is why capi_whitelists.yaml carries the proxies and netbird.renner.uno.
-crowdsec_bouncer_mode: nftables
+# dry-run pulls decisions and logs what it would block without creating a
+# single nftables rule. The first bouncer to arm anywhere enforces ~24k CAPI
+# decisions at once on the only external entry path, so the false-positive
+# window has to come before the enforcement, not after it. Flip to `nftables`
+# one host at a time once the log looks right.
+crowdsec_bouncer_mode: dry-run
 
 crowdsec_bouncer_update_frequency: 10s
 crowdsec_bouncer_log_level: info

ansible/roles/crowdsec-edge-bouncer/tasks/main.yml

@@ -19,40 +19,13 @@
     state: directory
     mode: "0755"
 
-# Fetched on the CONTROLLER and pushed, not pulled by the host.
-#
-# proxy-1 is IPv6-only, and neither github.com nor objects.githubusercontent.com
-# publishes an AAAA record — a get_url on the host fails with "Network is
-# unreachable" and no amount of retrying fixes it. Mirroring into
-# gitea.renner.uno is no help either: its public records are the internal LB
-# VIP (10.2.99.86 / fd99::86), so it is unreachable from outside the LAN. And
-# upstream ships no container image for this bouncer, which is expected —
-# it manipulates host netfilter.
-#
-# The CI runner has v4, so it does the fetching. The checksum is verified
-# there, before anything is copied to a host.
-- name: Fetch the pinned bouncer release on the controller
+- name: Fetch the pinned bouncer release
   ansible.builtin.get_url:
     url: >-
       https://github.com/crowdsecurity/cs-firewall-bouncer/releases/download/{{
       crowdsec_bouncer_version }}/crowdsec-firewall-bouncer-linux-amd64.tgz
-    dest: /tmp/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}.tgz
-    checksum: sha256:{{ crowdsec_bouncer_sha256 }}
-    mode: "0644"
-  delegate_to: localhost
-  become: false
-  run_once: true
-
-# Only the download had to move to the controller — the extraction stays on
-# the host. The ci-ansible image ships a tar that ansible rejects ("detected as
-# tar type None. GNU tar required"), while both host distributions have GNU
-# tar. Shipping the archive rather than the unpacked binary also keeps the
-# artefact that the checksum was verified against intact all the way to the
-# host.
-- name: Copy the bouncer archive to the host
-  ansible.builtin.copy:
-    src: /tmp/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}.tgz
     dest: /opt/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}.tgz
+    checksum: sha256:{{ crowdsec_bouncer_sha256 }}
     mode: "0644"
 
 # Only the binary. The tarball also carries install.sh / upgrade.sh /

ansible/roles/netbird-proxy-watchdog/defaults/main.yml

@@ -32,13 +32,8 @@ netbird_watchdog_fail_threshold: 3
 
 # Minimum seconds between watchdog-issued restarts, so a genuinely broken
 # backend can't put the proxy into a restart loop. Shared by both the URL-probe
-# watchdog and the flood watchdog below. One minute: the route-manager wedge
-# recurs on every routing-peer loss, and rolling node reboots lose one peer
-# every few minutes; a suppressed restart there leaves the wedge standing
-# until the URL probe's three rounds fire (3.5 min of 000 on every domain,
-# measured 2026-09-07). A minute still caps a loop at one restart per
-# probe interval.
-netbird_watchdog_cooldown_seconds: 60
+# watchdog and the flood watchdog below.
+netbird_watchdog_cooldown_seconds: 600
 
 # Log fingerprint of the reverse-proxy route-manager wedge, matched as a plain
 # substring in the proxy's docker logs by the flood watchdog. On this build the

ci/images/netbird-proxy-services/Containerfile

@@ -1,12 +0,0 @@
-# netbird-proxy-services: reconciles NetBird reverse-proxy services (public
-# domains -> TLS passthrough -> Traefik ClusterIP) from Ingresses carrying the
-# public label. Standard library only, so the image is the interpreter plus one
-# file; runs unprivileged as its own user.
-FROM python:3.13-slim
-
-RUN useradd --system --no-create-home --shell /usr/sbin/nologin controller
-COPY controller.py /opt/controller.py
-
-ENV PYTHONUNBUFFERED=1
-USER controller
-ENTRYPOINT ["python3", "/opt/controller.py"]

ci/images/netbird-proxy-services/controller.py

@@ -1,275 +0,0 @@
-#!/usr/bin/env python3
-"""netbird-proxy-services: reconcile NetBird reverse-proxy services from Ingresses.
-
-Source of truth: every Ingress carrying the public label (dns.renner.uno/public).
-For each host on such an Ingress the NetBird management gets one reverse-proxy
-service in TLS-passthrough mode on :443, targeting Traefik's ClusterIP through a
-subnet resource that covers the cluster's service CIDR. SNI and Host reach
-Traefik untouched, the proxy terminates nothing, and no domain resource is
-involved -- the routing peers stay ordinary IP routers. The target sends a
-PROXY protocol v2 header so Traefik (and CrowdSec behind its access log) sees
-the WAN client address instead of the routing peer's pod IP.
-
-Ownership is derived from NetBird itself, not from local state: a service is
-managed iff its mode is "tls" and its target is this controller's subnet
-resource. Services that do not match (the hand-made HTTP/domain ones) are never
-touched; a desired host held by such a service is reported and skipped until
-that service is removed by hand.
-
-Credentials: the NetBird PAT is read from OpenBao on every pass through the
-Kubernetes auth role, exactly like the setup-key rotator; it never lands in a
-Kubernetes Secret or in a log line.
-"""
-
-import json
-import logging
-import os
-import sys
-import time
-import urllib.error
-import urllib.parse
-import urllib.request
-
-LOG = logging.getLogger("netbird-proxy-services")
-
-NB = os.environ.get("NB_MANAGEMENT_URL", "https://netbird.renner.uno").rstrip("/")
-BAO_ADDR = os.environ.get("BAO_ADDR", "http://openbao.openbao.svc.cluster.local:8200").rstrip("/")
-BAO_ROLE = os.environ.get("BAO_ROLE", "netbird-proxy-services")
-BAO_PAT_PATH = os.environ.get("BAO_PAT_PATH", "homelab/data/infrastructure/secret-rotators-bootstrap/netbird")
-BAO_PAT_KEY = os.environ.get("BAO_PAT_KEY", "pat")
-SERVICE_CIDR = os.environ.get("SERVICE_CIDR", "10.96.0.0/12")
-NETWORK_NAME = os.environ.get("NETWORK_NAME", "k8s public ingress")
-ROUTER_GROUP = os.environ.get("ROUTER_GROUP_NAME", "k8s-ingress-router")
-SOURCE_GROUP = os.environ.get("SOURCE_GROUP_NAME", "All")
-TRAEFIK_NS, TRAEFIK_SVC = os.environ.get("TRAEFIK_SERVICE", "ingress/traefik").split("/", 1)
-PUBLIC_LABEL = os.environ.get("PUBLIC_LABEL", "dns.renner.uno/public")
-LISTEN_PORT = int(os.environ.get("LISTEN_PORT", "443"))
-INTERVAL = int(os.environ.get("INTERVAL_SECONDS", "60"))
-HTTP_TIMEOUT = 20
-
-K8S_API = "https://kubernetes.default.svc"
-K8S_TOKEN_FILE = "/var/run/secrets/kubernetes.io/serviceaccount/token"
-K8S_CA_FILE = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
-
-
-class ApiError(Exception):
-    pass
-
-
-def http(method, url, body=None, headers=None, cafile=None):
-    data = None if body is None else json.dumps(body).encode()
-    req = urllib.request.Request(url, data=data, method=method)
-    req.add_header("Accept", "application/json")
-    if data is not None:
-        req.add_header("Content-Type", "application/json")
-    for k, v in (headers or {}).items():
-        req.add_header(k, v)
-    ctx = None
-    if cafile:
-        import ssl
-        ctx = ssl.create_default_context(cafile=cafile)
-    try:
-        with urllib.request.urlopen(req, timeout=HTTP_TIMEOUT, context=ctx) as resp:
-            raw = resp.read()
-            return json.loads(raw) if raw else None
-    except urllib.error.HTTPError as e:
-        detail = e.read().decode(errors="replace")[:300]
-        raise ApiError(f"{method} {url} -> {e.code}: {detail}") from None
-
-
-# --- Kubernetes -------------------------------------------------------------
-
-def k8s(path):
-    with open(K8S_TOKEN_FILE) as f:
-        token = f.read().strip()
-    return http("GET", K8S_API + path, headers={"Authorization": f"Bearer {token}"}, cafile=K8S_CA_FILE)
-
-
-def desired_hosts():
-    sel = urllib.parse.quote(f"{PUBLIC_LABEL}=true")
-    ingresses = k8s(f"/apis/networking.k8s.io/v1/ingresses?labelSelector={sel}")
-    hosts = {}
-    for ing in ingresses.get("items", []):
-        ref = f'{ing["metadata"]["namespace"]}/{ing["metadata"]["name"]}'
-        for rule in ing.get("spec", {}).get("rules", []):
-            host = rule.get("host")
-            if not host or host.startswith("*"):
-                continue
-            hosts.setdefault(host, ref)
-    return hosts
-
-
-def traefik_cluster_ip():
-    svc = k8s(f"/api/v1/namespaces/{TRAEFIK_NS}/services/{TRAEFIK_SVC}")
-    for ip in svc["spec"].get("clusterIPs", []):
-        if ":" not in ip:
-            return ip
-    raise ApiError(f"{TRAEFIK_NS}/{TRAEFIK_SVC} has no IPv4 ClusterIP")
-
-
-# --- OpenBao ----------------------------------------------------------------
-
-def netbird_pat():
-    with open(K8S_TOKEN_FILE) as f:
-        jwt = f.read().strip()
-    login = http("POST", f"{BAO_ADDR}/v1/auth/kubernetes/login", {"role": BAO_ROLE, "jwt": jwt})
-    bao_token = login["auth"]["client_token"]
-    kv = http("GET", f"{BAO_ADDR}/v1/{BAO_PAT_PATH}", headers={"X-Vault-Token": bao_token})
-    pat = kv["data"]["data"].get(BAO_PAT_KEY)
-    if not pat:
-        raise ApiError(f"OpenBao {BAO_PAT_PATH} has no key {BAO_PAT_KEY}")
-    return pat
-
-
-# --- NetBird management -----------------------------------------------------
-
-class NetBird:
-    def __init__(self, pat):
-        self.h = {"Authorization": f"Token {pat}"}
-
-    def get(self, path):
-        return http("GET", f"{NB}/api/{path}", headers=self.h)
-
-    def list(self, path):
-        # An empty collection comes back as an empty body / null, not [].
-        return self.get(path) or []
-
-    def post(self, path, body):
-        return http("POST", f"{NB}/api/{path}", body, headers=self.h)
-
-    def put(self, path, body):
-        return http("PUT", f"{NB}/api/{path}", body, headers=self.h)
-
-    def delete(self, path):
-        return http("DELETE", f"{NB}/api/{path}", headers=self.h)
-
-
-def group_id(nb, name):
-    for g in nb.list("groups"):
-        if g["name"] == name:
-            return g["id"]
-    raise ApiError(f"NetBird group {name!r} not found")
-
-
-def ensure_network(nb):
-    for n in nb.list("networks"):
-        if n["name"] == NETWORK_NAME:
-            return n["id"]
-    n = nb.post("networks", {"name": NETWORK_NAME,
-                             "description": "managed by netbird-proxy-services; public ingress via TLS passthrough"})
-    LOG.info("created network %s (%s)", NETWORK_NAME, n["id"])
-    return n["id"]
-
-
-def ensure_resource(nb, net):
-    for r in nb.list(f"networks/{net}/resources"):
-        if r.get("address") == SERVICE_CIDR:
-            return r["id"]
-    r = nb.post(f"networks/{net}/resources", {"name": "cluster service CIDR", "address": SERVICE_CIDR,
-                                              "description": "managed by netbird-proxy-services",
-                                              "enabled": True, "groups": []})
-    LOG.info("created subnet resource %s (%s)", SERVICE_CIDR, r["id"])
-    return r["id"]
-
-
-def ensure_router(nb, net, router_group):
-    for r in nb.list(f"networks/{net}/routers"):
-        if router_group in (r.get("peer_groups") or []):
-            return r["id"]
-    r = nb.post(f"networks/{net}/routers", {"peer_groups": [router_group], "metric": 9999,
-                                            "masquerade": True, "enabled": True})
-    LOG.info("created router on group %s (%s)", ROUTER_GROUP, r["id"])
-    return r["id"]
-
-
-def ensure_policy(nb, resource, source_group):
-    name = f"{NETWORK_NAME} access"
-    for p in nb.list("policies"):
-        if p["name"] == name:
-            return p["id"]
-    p = nb.post("policies", {"name": name, "enabled": True, "rules": [{
-        "name": name, "enabled": True, "action": "accept", "protocol": "tcp",
-        "ports": [str(LISTEN_PORT)], "bidirectional": False, "sources": [source_group],
-        "destinationResource": {"id": resource, "type": "subnet"}}]})
-    LOG.info("created policy %s (%s)", name, p["id"])
-    return p["id"]
-
-
-def service_body(host, resource, target_ip):
-    # proxy_protocol: PROXY v2 header carries the WAN client address to
-    # Traefik (proxyProtocol.trustedIPs on websecure); TCP/TLS modes only.
-    return {"name": host, "domain": host, "mode": "tls", "listen_port": LISTEN_PORT, "enabled": True,
-            "targets": [{"target_type": "subnet", "target_id": resource, "protocol": "tcp",
-                         "host": target_ip, "port": 443, "enabled": True,
-                         "options": {"proxy_protocol": True}}]}
-
-
-def is_managed(svc, resource):
-    return svc.get("mode") == "tls" and any(t.get("target_id") == resource for t in svc.get("targets") or [])
-
-
-def target_matches(svc, target_ip):
-    if svc.get("listen_port") != LISTEN_PORT or not svc.get("enabled"):
-        return False
-    ts = svc.get("targets") or []
-    return len(ts) == 1 and ts[0].get("host") == target_ip and ts[0].get("port") == 443 \
-        and ts[0].get("protocol") == "tcp" and ts[0].get("enabled") \
-        and (ts[0].get("options") or {}).get("proxy_protocol") is True
-
-
-def reconcile():
-    hosts = desired_hosts()
-    target_ip = traefik_cluster_ip()
-    nb = NetBird(netbird_pat())
-
-    router_group = group_id(nb, ROUTER_GROUP)
-    source_group = group_id(nb, SOURCE_GROUP)
-    net = ensure_network(nb)
-    resource = ensure_resource(nb, net)
-    ensure_router(nb, net, router_group)
-    ensure_policy(nb, resource, source_group)
-
-    by_domain = {s["domain"]: s for s in nb.list("reverse-proxies/services")}
-    created = updated = deleted = held = 0
-
-    for host, ref in sorted(hosts.items()):
-        svc = by_domain.get(host)
-        if svc is None:
-            nb.post("reverse-proxies/services", service_body(host, resource, target_ip))
-            LOG.info("created %s (from %s) -> %s:443", host, ref, target_ip)
-            created += 1
-        elif not is_managed(svc, resource):
-            LOG.warning("%s is held by unmanaged service %s (mode %s); remove it to migrate",
-                        host, svc["id"], svc.get("mode"))
-            held += 1
-        elif not target_matches(svc, target_ip):
-            nb.put(f"reverse-proxies/services/{svc['id']}", service_body(host, resource, target_ip))
-            LOG.info("updated %s -> %s:443", host, target_ip)
-            updated += 1
-
-    for domain, svc in by_domain.items():
-        if is_managed(svc, resource) and domain not in hosts:
-            nb.delete(f"reverse-proxies/services/{svc['id']}")
-            LOG.info("deleted %s (no public Ingress carries it any more)", domain)
-            deleted += 1
-
-    LOG.info("reconciled: %d hosts desired, %d created, %d updated, %d deleted, %d held by unmanaged services",
-             len(hosts), created, updated, deleted, held)
-
-
-def main():
-    logging.basicConfig(level=os.environ.get("LOG_LEVEL", "INFO"), stream=sys.stdout,
-                        format="%(asctime)s %(levelname)s %(message)s")
-    if os.environ.get("ONCE") == "1":
-        reconcile()
-        return
-    while True:
-        try:
-            reconcile()
-        except Exception as e:  # keep running; the next pass retries
-            LOG.error("reconcile failed: %s", e)
-        time.sleep(INTERVAL)
-
-
-if __name__ == "__main__":
-    main()

ci/images/vmtest/Containerfile

@@ -1,84 +0,0 @@
-FROM alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d
-
-# Everything `bootstrap.sh vmtest` needs, so the rebuild test can run as a Job in
-# the cluster instead of on a workstation. The workstation has 31G and the test
-# wants ~14.5G for four VMs — a single run left under 1G headroom, and two
-# overlapping runs OOM-killed the session. The nodes have ~94G with 70G free.
-#
-# Separate from ci-k8s rather than built on it: that image pins talosctl v1.12.6,
-# and this test creates a cluster at the version talos/patches/cluster.yaml
-# installs. A talosctl older than the Talos it provisions is exactly the kind of
-# skew the test exists to catch, so it must not be baked in here.
-#
-# KEEP TALOS_VERSION IN STEP with the installer image tag in
-# kubernetes/talos/patches/cluster.yaml. The phase reads the version from that
-# file at run time and passes it to the provisioner; this pin only decides which
-# client talks to it.
-ARG TALOS_VERSION=v1.13.9
-ARG KUBECTL_VERSION=v1.36.2
-ARG FLUX_VERSION=2.9.2
-ARG KOPIA_VERSION=0.23.1
-
-# qemu-system-x86_64 and qemu-img are the provisioner's actual backend, and ovmf
-# supplies the UEFI firmware those VMs boot from — without it the provisioner
-# stops with "the required flash image was not found in any of the expected
-# paths", having already downloaded the ISO and generated the PKI. Note the
-# firmware comes from ovmf specifically: qemu-system-x86_64 ships an edk2 CODE
-# image but no VARS, which is the half the provisioner needs.
-#
-# iptables is shelled out to by the CNI plugins that build the VM network; the
-# provisioner fails with "error accessing iptables" without it. The rest is
-# what the phase shells out to: helm for the Cilium install, yq to read the
-# HelmRelease and the installer image, jq for OpenBao's seal status.
-#
-# Versions come from the digest-pinned base rather than being pinned one by one,
-# the same way ci-ansible does it.
-RUN apk add --no-cache \
-      bash \
-      ca-certificates \
-      curl \
-      git \
-      helm \
-      iptables \
-      jq \
-      openssl \
-      ovmf \
-      qemu-img \
-      qemu-system-x86_64 \
-      yq
-
-RUN curl -sfL -o /usr/local/bin/talosctl \
-      "https://github.com/siderolabs/talos/releases/download/${TALOS_VERSION}/talosctl-linux-amd64" \
- && echo "7e1d4b7d5846964bdcf63a794e3c8161bb6ef2983d5ace58ea5322f3bf32a27e  /usr/local/bin/talosctl" | sha256sum -c - \
- && chmod +x /usr/local/bin/talosctl
-
-RUN curl -sfL -o /usr/local/bin/kubectl \
-      "https://dl.k8s.io/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl" \
- && curl -sfL -o /tmp/kubectl.sha256 \
-      "https://dl.k8s.io/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl.sha256" \
- && echo "$(cat /tmp/kubectl.sha256)  /usr/local/bin/kubectl" | sha256sum -c - \
- && rm -f /tmp/kubectl.sha256 \
- && chmod +x /usr/local/bin/kubectl
-
-RUN curl -sfL -o /tmp/flux.tar.gz \
-      "https://github.com/fluxcd/flux2/releases/download/v${FLUX_VERSION}/flux_${FLUX_VERSION}_linux_amd64.tar.gz" \
- && echo "4092b367fd060097976fb7261601f79420ecf7266328417dfd8dfee27de0e6a3  /tmp/flux.tar.gz" | sha256sum -c - \
- && tar -xzf /tmp/flux.tar.gz -C /usr/local/bin flux \
- && chmod +x /usr/local/bin/flux \
- && rm -f /tmp/flux.tar.gz
-
-# kopia reads the Velero backup repository directly, which is what the OpenBao
-# restore does when there is no Velero yet to do it: the credentials Velero
-# needs live in OpenBao, so OpenBao cannot be restored by Velero. Its absence
-# was found by a rebuild rehearsal reaching the restore for the first time and
-# stopping at "kopia not found".
-#
-# Same checksum-verified download as the tools above; the tarball unpacks into
-# a versioned directory, so the binary is pulled out by name.
-RUN curl -sfL -o /tmp/kopia.tar.gz \
-      "https://github.com/kopia/kopia/releases/download/v${KOPIA_VERSION}/kopia-${KOPIA_VERSION}-linux-x64.tar.gz" \
- && echo "416d0f84a3dbb321a8b2d8f0997b1a0a6e915babe79ee76fa6e4d2bd1e1c5178  /tmp/kopia.tar.gz" | sha256sum -c - \
- && tar -xzf /tmp/kopia.tar.gz -C /tmp \
- && mv "/tmp/kopia-${KOPIA_VERSION}-linux-x64/kopia" /usr/local/bin/kopia \
- && chmod +x /usr/local/bin/kopia \
- && rm -rf /tmp/kopia.tar.gz "/tmp/kopia-${KOPIA_VERSION}-linux-x64"

kubernetes/apps/authentik/helmrelease.yaml

@@ -120,14 +120,7 @@ spec:
     server:
       resources:
         requests:
-          # 500m, not 200m: the HPA below targets 70% of *this* request, so at
-          # 200m the scale-up threshold sat at 140m — below the pod's own
-          # ordinary peak. Measured over 7d, a warm server pod p99s at ~108m
-          # and peaks at ~190m, and a freshly-started one burns ~300m booting
-          # Django. That made the HPA oscillate 2<->3<->4 all day and pin at
-          # maxReplicas (KubeHpaMaxedOut, issue #761). At 500m the threshold
-          # is 350m/pod, above the boot burst, so only real load scales out.
-          cpu: 500m
+          cpu: 200m
           memory: 1Gi
         limits:
           memory: 2Gi
@@ -161,11 +154,7 @@ spec:
     worker:
       resources:
         requests:
-          # Same reasoning as server above, and the worker is burstier still:
-          # over 7d its per-pod p99 reaches ~336m and peaks hit ~630m while
-          # running blueprint applies and scheduled tasks. 200m understated
-          # that badly enough that routine task batches read as saturation.
-          cpu: 500m
+          cpu: 200m
           memory: 1Gi
         limits:
           memory: 2Gi

kubernetes/apps/authentik/kustomization.yaml

@@ -16,6 +16,7 @@ resources:
   - prometheus-gate-egress.yaml
   - wordpress-gate-egress.yaml
   - llama-gate-egress.yaml
+  - outpost-servicemonitors.yaml
   - ldap-outpost-cert.yaml
   - ldap-outpost-lb.yaml
   - ldap-outpost-cnp.yaml

kubernetes/apps/home-assistant/deployment.yaml

@@ -95,7 +95,7 @@ spec:
               memory: 32Mi
       containers:
         - name: home-assistant
-          image: ghcr.io/home-assistant/home-assistant:2026.9.1 # {"$imagepolicy": "flux-system:home-assistant"}
+          image: ghcr.io/home-assistant/home-assistant:2026.8.3 # {"$imagepolicy": "flux-system:home-assistant"}
           env:
             - name: TZ
               value: Europe/Berlin

kubernetes/apps/home-assistant/mdns-relay-deployment.yaml

@@ -60,7 +60,7 @@ spec:
       # Two processes rather than one: socat carries a single address pair per
       # invocation, and the families need different receive and send addresses.
       # They are independent — one crashing does not blind the other — and each
-      # carries its own supervisor loop.
+      # is restarted by the kubelet on its own.
       containers:
         # -u is load-bearing on both: unidirectional, left to right. Without it
         # socat would also carry HA's replies back out, and this must stay a
@@ -77,29 +77,17 @@ spec:
         # UDP4-SENDTO takes its A record and UDP6-SENDTO its AAAA. See
         # mdns-relay-service.yaml for why that indirection is required and why
         # the Service is dual-stack.
-        #
-        # Supervisor loop: socat exits 1 on the first datagram it cannot
-        # deliver, and the ClusterIP has no backend while HA restarts. The
-        # trap forwards SIGTERM so the pod still terminates promptly.
         - name: socat-ipv4
           image: alpine/socat:1.8.0.3@sha256:beb4a68d9e4fe6b0f21ea774a0fde6c31f580dde6368939ed70100c5385b015e
           imagePullPolicy: IfNotPresent
-          command:
-            - /bin/sh
-            - -c
-            - |
-              trap 'kill "$pid"; exit 0' TERM INT
-              while :; do
-                socat -u \
-                  "UDP4-RECV:5353,ip-add-membership=224.0.0.251:net1,so-bindtodevice=net1,reuseaddr" \
-                  "UDP4-SENDTO:home-assistant-mdns.home-assistant.svc.cluster.local:5353" &
-                pid=$!
-                wait "$pid"
-                echo "mdns-relay: socat-ipv4 exited ($?); restarting in 2s" >&2
-                sleep 2
-              done
-          # No probes. The supervisor loop is the liveness story; a probe here
-          # could only re-assert that PID 1 is running.
+          args:
+            - -u
+            - "UDP4-RECV:5353,ip-add-membership=224.0.0.251:net1,so-bindtodevice=net1,reuseaddr"
+            - "UDP4-SENDTO:home-assistant-mdns.home-assistant.svc.cluster.local:5353"
+          # No probes. socat is a single process with no health surface: if it
+          # exits, the container exits and the kubelet restarts it, which is the
+          # entire liveness story. A probe here could only re-assert that PID 1
+          # is running.
           resources:
             requests:
               cpu: 10m
@@ -126,20 +114,10 @@ spec:
         - name: socat-ipv6
           image: alpine/socat:1.8.0.3@sha256:beb4a68d9e4fe6b0f21ea774a0fde6c31f580dde6368939ed70100c5385b015e
           imagePullPolicy: IfNotPresent
-          command:
-            - /bin/sh
-            - -c
-            - |
-              trap 'kill "$pid"; exit 0' TERM INT
-              while :; do
-                socat -u \
-                  "UDP6-RECV:5353,ipv6-join-group=[ff02::fb]:net1,so-bindtodevice=net1,ipv6-v6only=1,reuseaddr" \
-                  "UDP6-SENDTO:home-assistant-mdns.home-assistant.svc.cluster.local:5353" &
-                pid=$!
-                wait "$pid"
-                echo "mdns-relay: socat-ipv6 exited ($?); restarting in 2s" >&2
-                sleep 2
-              done
+          args:
+            - -u
+            - "UDP6-RECV:5353,ipv6-join-group=[ff02::fb]:net1,so-bindtodevice=net1,ipv6-v6only=1,reuseaddr"
+            - "UDP6-SENDTO:home-assistant-mdns.home-assistant.svc.cluster.local:5353"
           resources:
             requests:
               cpu: 10m

kubernetes/apps/home-assistant/zigbee2mqtt-deployment.yaml

@@ -57,14 +57,7 @@ spec:
           # and leaves this one untouched (tags are not deleted), so this keeps
           # resolving — but the image then came from the branch build rather than
           # master. Re-point it at a master-built tag when convenient.
-          #
-          # konairius/, not gitea_admin/: this init container runs during a cold
-          # rebuild, when Gitea does not exist because it runs inside the
-          # cluster being rebuilt. The node then falls back to a ghcr.io copy
-          # (machine.registries.mirrors, applied by bootstrap.sh), and containerd
-          # keeps the repository path when it falls through — so the path has to
-          # exist on both registries. Steady state still pulls from Gitea.
-          image: gitea.renner.uno/konairius/alpine-jq:2026-09-04
+          image: gitea.renner.uno/gitea_admin/alpine-jq:2026-07-30
           command:
             - sh
             - -c

kubernetes/apps/llama-server/deployment.yaml

@@ -25,13 +25,6 @@ metadata:
     app.kubernetes.io/name: llama-server
 spec:
   replicas: 1
-  # The startupProbe below sanctions a 40 min cold download, but the default
-  # progressDeadlineSeconds is 600. Left implicit the Deployment declares
-  # ProgressDeadlineExceeded at 10 minutes while the probe is still legitimately
-  # waiting, and KubeDeploymentRolloutStuck fires 15 min after that on a rollout
-  # that is fine. Kept above the startup budget, with room for the image pull
-  # and scheduling that happen before the first probe.
-  progressDeadlineSeconds: 3000
   # A rolling update would load a second copy of the weights onto the same
   # iGPU and into the same node's RAM before releasing the first.
   strategy:
@@ -82,62 +75,11 @@ spec:
             # many small per-expert matmuls that sets the pace, not bandwidth.
             - --threads
             - "4"
-            # 16k is room to spare, not a safety limit. The limit that matters
-            # is not set here at all.
-            #
-            # Isolated by bisection on a scratch pod: this stack breaks on
-            # PROMPT length, and the value of --ctx-size makes no difference to
-            # it. Measured at --ctx-size 65536 with a single slot, so with four
-            # times the headroom:
-            #
-            #     13588 prompt tokens   coherent answer
-            #     15086                 one token, immediate EOS, empty content
-            #     16088                 CJK characters and digit runs
-            #
-            # A single request past that boundary then poisons the slot: the
-            # same 13588-token prompt that answered cleanly a minute earlier
-            # returns one empty token afterwards, and stays broken until the
-            # pod restarts. That is why it looked intermittent, and why a
-            # 650-token classification once came back as 1025 tokens of "!" —
-            # it followed Home Assistant's ~19k prompts through the same slot.
-            #
-            # Nothing reports any of it: finish_reason "stop", a well-formed
-            # OpenAI response, HTTP 200, /health green.
-            #
-            # So the operating rule is about callers, not this flag. Paperless
-            # sits near 5k and is safe; Home Assistant's Assist prompt measured
-            # 18910 and never had a chance. An earlier version of this comment
-            # blamed --ctx-size 32768, which the bisection disproves — 32768
-            # and 65536 both answer correctly as long as the prompt stays under
-            # the boundary.
-            #
-            # The same stack also fails hard at --ubatch-size 2048, which is
-            # the loud version of a limit this one keeps quiet about.
-            #
+            # 16k rather than 8k because the measured prefill falloff is mild
+            # and this model's KV cache is unusually cheap — only 10 of its 40
+            # layers use full attention.
             - --ctx-size
             - "16384"
-            # One slot, so the 16384 belongs to one request at a time.
-            #
-            # Left on auto, llama.cpp opens four slots and — because auto is
-            # also what enables --kv-unified — has them share a single buffer
-            # of this size. Four Paperless requests at once wanted 6209, 6472,
-            # 4303 and 1271 tokens against those 16384, and all four died with
-            # "Context size has been exceeded".
-            #
-            # They were concurrent because a client timeout does not stop the
-            # generation behind it: each abandoned request keeps consuming the
-            # shared budget the next one needs, so one slow document is enough
-            # to start the pile-up. Queueing is the behaviour worth having
-            # here — nothing calls this server in parallel on purpose, and
-            # Paperless's celery worker runs at concurrency 1.
-            #
-            # No liveness probe change goes with this. The server recovered on
-            # its own once the batch drained, so a probe that restarted the pod
-            # would have made the incident worse; with a single slot, one that
-            # asked for a real completion would also queue behind live work and
-            # kill healthy pods during a long document.
-            - --parallel
-            - "1"
             # A micro-batch of 2048 or more makes this Vulkan/Mesa stack fail
             # with ErrorDeviceLost.
             - --ubatch-size

kubernetes/apps/llama-server/embed.yaml

@@ -19,11 +19,6 @@ metadata:
     app.kubernetes.io/name: llama-embed
 spec:
   replicas: 1
-  # Same reasoning as llama-server: the startupProbe budget (600s) must stay
-  # below progressDeadlineSeconds, whose default is also 600 — exactly equal, so
-  # any cold download that uses most of its budget trips
-  # KubeDeploymentRolloutStuck on a rollout that is still progressing normally.
-  progressDeadlineSeconds: 900
   strategy:
     type: Recreate
   selector:
@@ -46,7 +41,7 @@ spec:
           image: ghcr.io/ggml-org/llama.cpp:server-vulkan-b10731
           args:
             - --hf-repo
-            - gpustack/bge-m3-GGUF:Q8_0
+            - Qwen/Qwen3-Embedding-0.6B-GGUF:Q8_0
             # Serves POST /v1/embeddings.
             - --embeddings
             # A short, stable id for consumers. Without it the model is
@@ -55,41 +50,28 @@ spec:
             # One name only: llama-server keeps aliases in a sorted set and
             # reports the first as the id, so a second name can steal it.
             - --alias
-            - bge-m3
-            # bge-m3 is trained for CLS pooling — a different type from the
-            # Qwen family this replaces, which uses last-token. Left unspecified
-            # llama-server takes the GGUF default, which is not visible from
-            # outside and would silently produce vectors the model was not
-            # trained to emit. Read from the model's own 1_Pooling/config.json
-            # rather than assumed; of the candidates measured, three different
-            # pooling types were needed.
+            - qwen3-embedding-0.6b
+            # Qwen3-Embedding is trained for last-token pooling. Left
+            # unspecified llama-server takes the GGUF's default, which is not
+            # visible from outside and would silently produce vectors the model
+            # was not trained to emit. Set before anything is indexed: changing
+            # pooling later invalidates every stored embedding.
             - --pooling
-            - cls
+            - last
             - --n-gpu-layers
             - "99"
             - --threads
             - "2"
-            # Paperless chunks at 1024 tokens, but counts them with its own
-            # tokenizer while this server measures the batch in bge-m3's.
-            # Those disagree: XLM-RoBERTa splits German into more pieces than
-            # llama_index's counter does, so a chunk Paperless considers full
-            # at 1024 arrived here as 1069 and the request failed outright —
-            # "input (1069 tokens) is too large to process".
-            #
-            # The headroom therefore has to absorb a tokenizer ratio, not just
-            # a rounding error, and 1.5x covers far more than the 1.044 that
-            # broke it. The defaults are batch 2048 / ubatch 512, and with
-            # --embeddings llama-server clamps both to the smaller rather than
-            # run them mismatched, so both are set.
-            #
-            # 2048 stays avoided: that is where the chat model met
-            # ErrorDeviceLost on this Vulkan stack. That failure came under a
-            # 20 GB model's memory pressure and may not bind a 600 MB one, but
-            # this does not need to find out.
+            # Paperless chunks documents at 1024 tokens. The defaults are
+            # batch 2048 / ubatch 512, and with --embeddings llama-server
+            # clamps both down to 512 rather than run them mismatched — below
+            # the chunk size. Matching both to 1024 fits a whole chunk in one
+            # pass; 2048 is deliberately avoided, since that is where this
+            # Vulkan stack fails with ErrorDeviceLost.
             - --batch-size
-            - "1536"
+            - "1024"
             - --ubatch-size
-            - "1536"
+            - "1024"
             # Room for a full chunk plus overhead rather than exactly one.
             - --ctx-size
             - "4096"
@@ -109,14 +91,6 @@ spec:
             capabilities:
               drop:
                 - ALL
-          # Measured, not estimated: embedding 400 real documents peaked at
-          # 1.6 GiB (Prometheus max_over_time on the working set). The limit is
-          # well above that because the footprint is dominated by the four slots
-          # at 4096 context and the Vulkan working set rather than by the
-          # weights — the Qwen model this replaces peaked at 10.6 GiB on the
-          # same settings for the same 600 MiB of weights, and several
-          # candidates were OOMKilled at limits that looked generous against
-          # their model size.
           resources:
             requests:
               cpu: 250m
@@ -124,7 +98,7 @@ spec:
               gpu.intel.com/i915: "1"
             limits:
               cpu: "2"
-              memory: 8Gi
+              memory: 4Gi
               gpu.intel.com/i915: "1"
           # ~600 MiB rather than ~5 GiB, so a much shorter budget than the chat
           # model needs, but still sized for a cold download.

kubernetes/apps/monitoring-rules/cilium-servicemonitor.yaml

@@ -1,46 +0,0 @@
-# Cilium's agent ServiceMonitor, declared here rather than by the chart.
-#
-# Reproduces what cilium 1.20.1 renders with prometheus.serviceMonitor.enabled,
-# taken field for field off the live object so scraping is unchanged: same name,
-# same namespace, same selector, same 10s interval and node relabeling.
-#
-# It lives here because the chart's template is gated on monitoring.coreos.com/v1
-# existing at install time, and on an empty cluster it does not — those CRDs
-# arrive with kube-prometheus-stack, which reconciles after infrastructure.
-# Cilium is one of infrastructure's healthChecks, so the chart aborting on the
-# missing CRD held the whole cluster: CNI waiting on monitoring waiting on CNI.
-#
-# velero/helmrelease.yaml already does exactly this for its own SM/PM and says
-# why; this is the same move for the component where it mattered most.
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: cilium-agent
-  namespace: kube-system
-  labels:
-    app.kubernetes.io/name: cilium-agent
-    app.kubernetes.io/part-of: cilium
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: cilium-agent
-  namespaceSelector:
-    matchNames:
-      - kube-system
-  # The chart's Service keeps rendering (prometheus.enabled), so the target and
-  # its `metrics` port are there regardless of who declares the ServiceMonitor.
-  targetLabels:
-    - k8s-app
-  endpoints:
-    - port: metrics
-      path: /metrics
-      interval: 10s
-      honorLabels: true
-      relabelings:
-        # Agent metrics are per-node; without this every series lands without a
-        # node label and BPF map pressure cannot be attributed to a machine.
-        - action: replace
-          sourceLabels:
-            - __meta_kubernetes_pod_node_name
-          targetLabel: node
-          replacement: ${1}

kubernetes/apps/monitoring-rules/crowdsec-servicemonitor.yaml

@@ -1,36 +0,0 @@
-# Scrapes CrowdSec. Until this landed, nothing did: `up{namespace="crowdsec"}`
-# returned zero series and Prometheus held no cs_* metric at all, so the
-# detection side of the ingress path was invisible — including whether the
-# agents were reading logs and whether any bouncer was pulling decisions.
-#
-# One monitor for both services. They carry the same shape (a `metrics` port on
-# 6060) and differ only in `app`, so a matchExpressions selector keeps this to
-# a single object rather than two near-identical ones:
-#
-#   crowdsec-service        the LAPI      — decisions, bouncer + machine requests
-#   crowdsec-agent-service  the agents    — acquisition, parsers, scenarios
-#
-# The crowdsec-ns CiliumNetworkPolicy already admits the monitoring namespace
-# without a port restriction, so no policy change is needed.
-#
-# `release: kube-prometheus-stack` is what the chart's Prometheus selects on.
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: crowdsec
-  namespace: crowdsec
-  labels:
-    app.kubernetes.io/name: crowdsec
-    release: kube-prometheus-stack
-spec:
-  selector:
-    matchExpressions:
-      - key: app
-        operator: In
-        values:
-          - crowdsec-service
-          - crowdsec-agent-service
-  endpoints:
-    - port: metrics
-      path: /metrics
-      interval: 30s

kubernetes/apps/monitoring-rules/kustomization.yaml

@@ -22,19 +22,7 @@ resources:
   # endpoint, which proves reachability, not that anything was written.
   - prometheusrule-cnpg-backup.yaml
   - alloy-servicemonitor.yaml
-  - cilium-servicemonitor.yaml
   - cnpg-podmonitors.yaml
-  - crowdsec-servicemonitor.yaml
-  # Velero's own monitors: monitoring.coreos.com CRDs arrive with
-  # kube-prometheus-stack, which reconciles after infrastructure, so these
-  # cannot live beside the Velero HelmRelease.
-  - velero-podmonitor.yaml
-  - velero-servicemonitor.yaml
-  - authentik-outpost-servicemonitors.yaml
-  # Rook's exporter/mgr monitors: the operator aborts its node-daemon reconcile
-  # when it cannot create these, which held the CephCluster in Progressing and
-  # deadlocked the whole apps layer. See the file for the cycle.
-  - rook-servicemonitors.yaml
   - home-assistant-servicemonitor.yaml
   - immich-servicemonitor.yaml
   - loki-servicemonitor.yaml

kubernetes/apps/monitoring-rules/prometheusrule-velero.yaml

@@ -58,30 +58,10 @@ spec:
           # VeleroBackupFailed alert fires on the first failure regardless — so
           # nothing is hidden, only the escalation to critical is deferred
           # until a second cycle confirms it.
-          #
-          # The `and on(schedule) velero_backup_last_status` gate exists because
-          # velero_backup_last_successful_timestamp is an in-process gauge that
-          # velero does NOT retract when a Schedule is deleted: the series just
-          # stops being updated, ages past 60h and pages critical forever, for a
-          # schedule that no longer exists. That is what decommissioning
-          # netbird-backup-daily did on 2026-09-01.
-          #
-          # velero_backup_last_status is the same shape but IS retracted on
-          # Schedule deletion, so it works as a liveness set for "this schedule
-          # still exists". It hides nothing: velero repopulates it for every
-          # existing Schedule on its first scrape after a restart (verified
-          # across the three pods of the 2026-08-27 rollout), and it is written
-          # on failed runs too (0) as well as successful ones (1) — so a
-          # schedule that exists and is failing, or that exists and has stopped
-          # firing, still carries the series and still alerts.
           expr: |
-            (
-              time() - velero_backup_last_successful_timestamp{schedule!=""} > 60 * 3600
-              and
-              velero_backup_last_successful_timestamp{schedule!=""} > 0
-            )
-            and on(schedule)
-            velero_backup_last_status
+            time() - velero_backup_last_successful_timestamp{schedule!=""} > 60 * 3600
+            and
+            velero_backup_last_successful_timestamp{schedule!=""} > 0
           for: 10m
           labels:
             severity: critical

kubernetes/apps/monitoring-rules/rook-servicemonitors.yaml

@@ -1,93 +0,0 @@
-# Rook's two ServiceMonitors, declared here rather than by the operator.
-#
-# Reproduced field for field off the live objects in the production cluster, so
-# scraping is unchanged: same names, same namespace, same selectors, same 10s
-# interval and `cluster` relabeling. The only thing dropped is the CephCluster
-# ownerReference, which Rook set and Flux cannot.
-#
-# They live here because CephCluster.spec.monitoring.enabled is now false, and
-# that is not a monitoring decision — it breaks a deadlock. Rook's node-daemon
-# reconcile creates this ServiceMonitor and, unlike the exporter DaemonSet next
-# to it, treats a failure as fatal (pkg/operator/ceph/cluster/nodedaemon/
-# reconcile.go, v1.20.7):
-#
-#     if !cephCluster.Spec.Monitoring.MetricsDisabled {
-#         // exporter reconciled; its errors are logged and reconcile continues
-#         if cephCluster.Spec.Monitoring.Enabled {
-#             if err := EnableCephExporterServiceMonitor(...); err != nil {
-#                 return errors.Wrap(err, "failed to enable service monitor")
-#             }
-#         }
-#     }
-#
-# On an empty cluster monitoring.coreos.com/v1 does not exist yet, so that call
-# fails with "the server could not find the requested resource (post
-# servicemonitors.monitoring.coreos.com)", the whole node-daemon reconcile
-# aborts, and the CephCluster never leaves status.phase Progressing. That is
-# what infrastructure-ceph-cluster's healthCheckExpr waits on — and the chain
-# closes on itself:
-#
-#     infrastructure-ceph-cluster → CephCluster phase Ready
-#                                 → Rook reconcile settles
-#                                 → monitoring.coreos.com CRDs exist
-#                                 → kube-prometheus-stack (monitoring)
-#                                 → grafana-db
-#                                 → infrastructure-ceph-cluster
-#
-# Nothing outside that loop can break it, so a cold rebuild stopped here with
-# every one of the twenty app Kustomizations behind it.
-#
-# Same treatment velero/helmrelease.yaml and cilium/helmrelease.yaml already
-# get, and for the same reason: the metrics stay where they are, only the
-# scrape wiring moves to the layer where its CRDs exist.
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: rook-ceph-exporter
-  namespace: rook-ceph
-  labels:
-    team: rook
-spec:
-  selector:
-    matchLabels:
-      app: rook-ceph-exporter
-      rook_cluster: rook-ceph
-  namespaceSelector:
-    matchNames:
-      - rook-ceph
-  endpoints:
-    - port: ceph-exporter-http-metrics
-      path: /metrics
-      interval: 10s
-      honorLabels: true
-      # Rook stamps the cluster name so series from several Ceph clusters stay
-      # separable. One cluster here, but dropping it would rename every series.
-      relabelings:
-        - action: replace
-          replacement: rook-ceph
-          targetLabel: cluster
----
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: rook-ceph-mgr
-  namespace: rook-ceph
-  labels:
-    team: rook
-spec:
-  selector:
-    matchLabels:
-      app: rook-ceph-mgr
-      rook_cluster: rook-ceph
-  namespaceSelector:
-    matchNames:
-      - rook-ceph
-  endpoints:
-    - port: http-metrics
-      path: /metrics
-      interval: 10s
-      honorLabels: true
-      relabelings:
-        - action: replace
-          replacement: rook-ceph
-          targetLabel: cluster

kubernetes/apps/monitoring/dashboards/crowdsec-overview.json

@@ -1,163 +0,0 @@
-{
-  "annotations": {"list": []},
-  "description": "CrowdSec: Erkennung im Cluster, Durchsetzung am NetBird-Edge",
-  "editable": true,
-  "fiscalYearStartMonth": 0,
-  "graphTooltip": 1,
-  "id": null,
-  "links": [],
-  "panels": [
-    {
-      "datasource": {"type": "prometheus", "uid": "prometheus"},
-      "description": "Aktive Ban-Entscheidungen in der LAPI, ganz überwiegend aus den CAPI-Community-Blocklisten. Das ist die Menge, die ein scharfgeschalteter Bouncer durchsetzt.",
-      "fieldConfig": {
-        "defaults": {"color": {"mode": "thresholds"}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "text", "value": null}]}, "unit": "short"},
-        "overrides": []
-      },
-      "gridPos": {"h": 4, "w": 6, "x": 0, "y": 0},
-      "id": 1,
-      "options": {"colorMode": "value", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
-      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(cs_active_decisions)", "refId": "A"}],
-      "title": "Aktive Decisions",
-      "type": "stat"
-    },
-    {
-      "datasource": {"type": "prometheus", "uid": "prometheus"},
-      "description": "Bouncer, die in den letzten 10 Minuten Decisions abgeholt haben. Fällt das unter die erwartete Zahl, holt ein Bouncer nichts mehr — und setzt damit einen veralteten Stand durch, ohne dass irgendetwas ausfällt.",
-      "fieldConfig": {
-        "defaults": {"color": {"mode": "thresholds"}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "red", "value": null}, {"color": "green", "value": 1}]}, "unit": "short"},
-        "overrides": []
-      },
-      "gridPos": {"h": 4, "w": 6, "x": 6, "y": 0},
-      "id": 2,
-      "options": {"colorMode": "value", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
-      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "count(sum by (bouncer) (rate(cs_lapi_bouncer_requests_total[10m])) > 0)", "refId": "A"}],
-      "title": "Bouncer, die abholen",
-      "type": "stat"
-    },
-    {
-      "datasource": {"type": "prometheus", "uid": "prometheus"},
-      "description": "Vom Agent gelesene Logzeilen pro Sekunde. Steht das auf 0, detektiert CrowdSec nichts mehr — der häufigste stille Ausfall, weil alles andere weiterläuft.",
-      "fieldConfig": {
-        "defaults": {"color": {"mode": "thresholds"}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "red", "value": null}, {"color": "green", "value": 0.01}]}, "unit": "reqps"},
-        "overrides": []
-      },
-      "gridPos": {"h": 4, "w": 6, "x": 12, "y": 0},
-      "id": 3,
-      "options": {"colorMode": "value", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
-      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(rate(cs_filesource_hits_total[5m]))", "refId": "A"}],
-      "title": "Gelesene Logzeilen/s",
-      "type": "stat"
-    },
-    {
-      "datasource": {"type": "prometheus", "uid": "prometheus"},
-      "description": "Szenario-Overflows der letzten 24 h — also lokal erkannte Angriffe, im Gegensatz zu den von CAPI übernommenen.",
-      "fieldConfig": {
-        "defaults": {"color": {"mode": "thresholds"}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "text", "value": null}]}, "unit": "short"},
-        "overrides": []
-      },
-      "gridPos": {"h": 4, "w": 6, "x": 18, "y": 0},
-      "id": 4,
-      "options": {"colorMode": "value", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
-      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(increase(cs_bucket_overflowed_total[24h]))", "refId": "A"}],
-      "title": "Lokale Overflows (24 h)",
-      "type": "stat"
-    },
-    {
-      "datasource": {"type": "prometheus", "uid": "prometheus"},
-      "description": "Abholrate je Bouncer. Der Firewall-Bouncer auf den Proxies fragt alle 10 s. Die LAPI legt bei jeder neuen beobachteten Client-Adresse einen Eintrag `<name>@<ip>` an, und beobachtet wird die Pod-IP des netbird-ingress-Peers — deshalb erscheint ein logischer Bouncer hier mehrfach und nach Pod-Rollen unter neuen Namen.",
-      "fieldConfig": {
-        "defaults": {"color": {"mode": "palette-classic"}, "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]}, "unit": "reqps"},
-        "overrides": []
-      },
-      "gridPos": {"h": 8, "w": 12, "x": 0, "y": 4},
-      "id": 5,
-      "options": {"legend": {"calcs": ["lastNotNull"], "displayMode": "table", "placement": "right", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
-      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum by (bouncer) (rate(cs_lapi_bouncer_requests_total[5m]))", "legendFormat": "{{bouncer}}", "refId": "A"}],
-      "title": "Bouncer-Abholrate",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {"type": "prometheus", "uid": "prometheus"},
-      "description": "Aktive Decisions nach Grund. `http:scan` dominiert und stammt aus den CAPI-Blocklisten.",
-      "fieldConfig": {
-        "defaults": {"color": {"mode": "palette-classic"}, "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 20, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "normal"}, "thresholdsStyle": {"mode": "off"}}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]}, "unit": "short"},
-        "overrides": []
-      },
-      "gridPos": {"h": 8, "w": 12, "x": 12, "y": 4},
-      "id": 6,
-      "options": {"legend": {"calcs": ["lastNotNull"], "displayMode": "table", "placement": "right", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
-      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum by (reason) (cs_active_decisions)", "legendFormat": "{{reason}}", "refId": "A"}],
-      "title": "Aktive Decisions nach Grund",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {"type": "prometheus", "uid": "prometheus"},
-      "description": "Welche Szenarien lokal auslösen. Das ist das, was CrowdSec selbst aus den Traefik-Logs erkennt.",
-      "fieldConfig": {
-        "defaults": {"color": {"mode": "palette-classic"}, "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "bars", "fillOpacity": 70, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "normal"}, "thresholdsStyle": {"mode": "off"}}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]}, "unit": "short"},
-        "overrides": []
-      },
-      "gridPos": {"h": 8, "w": 12, "x": 0, "y": 12},
-      "id": 7,
-      "options": {"legend": {"calcs": ["sum"], "displayMode": "table", "placement": "right", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
-      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum by (name) (increase(cs_bucket_overflowed_total[1h]))", "legendFormat": "{{name}}", "refId": "A"}],
-      "title": "Szenario-Overflows (je Stunde)",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {"type": "prometheus", "uid": "prometheus"},
-      "description": "Geparste gegen nicht geparste Zeilen. Ein Anstieg bei `ko` heißt meist, dass sich ein Logformat geändert hat und der Parser nicht mehr greift — CrowdSec liest dann weiter, erkennt aber nichts mehr.",
-      "fieldConfig": {
-        "defaults": {"color": {"mode": "palette-classic"}, "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]}, "unit": "reqps"},
-        "overrides": []
-      },
-      "gridPos": {"h": 8, "w": 12, "x": 12, "y": 12},
-      "id": 8,
-      "options": {"legend": {"calcs": ["lastNotNull"], "displayMode": "list", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
-      "targets": [
-        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(rate(cs_parser_hits_ok_total[5m]))", "legendFormat": "geparst", "refId": "A"},
-        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(rate(cs_parser_hits_ko_total[5m]))", "legendFormat": "nicht geparst", "refId": "B"}
-      ],
-      "title": "Parser",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {"type": "prometheus", "uid": "prometheus"},
-      "description": "Whitelist-Treffer nach Grund. Nützlich beim Scharfschalten: was hier zählt, wäre sonst ein Kandidat für einen Ban gewesen.",
-      "fieldConfig": {
-        "defaults": {"color": {"mode": "palette-classic"}, "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]}, "unit": "reqps"},
-        "overrides": []
-      },
-      "gridPos": {"h": 8, "w": 12, "x": 0, "y": 20},
-      "id": 9,
-      "options": {"legend": {"calcs": ["lastNotNull"], "displayMode": "table", "placement": "right", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
-      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum by (reason) (rate(cs_node_wl_hits_ok_total[5m]))", "legendFormat": "{{reason}}", "refId": "A"}],
-      "title": "Whitelist-Treffer",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {"type": "prometheus", "uid": "prometheus"},
-      "gridPos": {"h": 8, "w": 12, "x": 12, "y": 20},
-      "id": 10,
-      "options": {
-        "code": {"language": "plaintext", "showLineNumbers": false, "showMiniMap": false},
-        "content": "## Was dieses Dashboard NICHT zeigt\n\nAlle Panels stammen aus der LAPI und den Agents im Cluster. Die **Bouncer auf proxy-1 und proxy-2 exportieren eigene Metriken**, die hier fehlen:\n\n- `fw_bouncer_active_decisions` — was tatsächlich in den nftables-Sets steht\n- `fw_bouncer_dropped_packets` / `_bytes` — was wirklich verworfen wurde\n- `lapi_requests_total` / `_failures_total` — aus Sicht des Bouncers\n\nSie liegen auf `127.0.0.1:60601` der Proxy-Hosts. Die Hosts sind öffentlich und haben bisher keinerlei Metrik-Pfad; den Port ins Internet zu öffnen, nur um ein Scrape-Ziel zu bekommen, wäre der falsche Tausch.\n\n**Konsequenz:** „Bouncer-Abholrate\" oben zeigt, dass ein Bouncer *fragt* — nicht, dass er die Antwort auch angewandt hat. Solange der Modus `dry-run` ist, wendet er ohnehin nichts an.\n\nEin Scrape-Pfad für die Hosts ist eigene Arbeit und Voraussetzung dafür, dass „was wird wirklich geblockt\" beantwortbar wird."
-      },
-      "title": "Blinder Fleck",
-      "type": "text"
-    }
-  ],
-  "refresh": "30s",
-  "schemaVersion": 38,
-  "style": "dark",
-  "tags": ["homelab", "crowdsec", "security"],
-  "templating": {"list": []},
-  "time": {"from": "now-6h", "to": "now"},
-  "timepicker": {},
-  "timezone": "",
-  "title": "CrowdSec Overview",
-  "uid": "crowdsec-overview",
-  "version": 1,
-  "weekStart": ""
-}

kubernetes/apps/monitoring/dashboards/kustomization.yaml

@@ -38,9 +38,6 @@ configMapGenerator:
   - name: dashboard-blackbox-overview
     files:
       - blackbox-overview.json
-  - name: dashboard-crowdsec-overview
-    files:
-      - crowdsec-overview.json
   - name: dashboard-disk-health
     files:
       - disk-health.json

kubernetes/apps/music-assistant/deployment.yaml

@@ -55,7 +55,7 @@ spec:
         # stdlib only; Flux keeps the tag in lockstep via the shared image
         # policy) — no extra dependency.
         - name: reconcile-publish-ip
-          image: ghcr.io/music-assistant/server:2.10.2 # {"$imagepolicy": "flux-system:music-assistant"}
+          image: ghcr.io/music-assistant/server:2.10.1 # {"$imagepolicy": "flux-system:music-assistant"}
           command: ["/app/venv/bin/python", "/scripts/reconcile.py"]
           volumeMounts:
             - name: data
@@ -65,7 +65,7 @@ spec:
               readOnly: true
       containers:
         - name: music-assistant
-          image: ghcr.io/music-assistant/server:2.10.2 # {"$imagepolicy": "flux-system:music-assistant"}
+          image: ghcr.io/music-assistant/server:2.10.1 # {"$imagepolicy": "flux-system:music-assistant"}
           env:
             - name: LOG_LEVEL
               value: info

kubernetes/apps/paperless/paperless.yaml

@@ -82,10 +82,7 @@ data:
   # primarily English archives — the better fit for a German corpus.
   PAPERLESS_AI_LLM_EMBEDDING_BACKEND: "openai-like"
   PAPERLESS_AI_LLM_EMBEDDING_ENDPOINT: "http://llama-embed.ai.svc.cluster.local:8080/v1"
-  # Must match --alias in kubernetes/apps/llama-server/embed.yaml. Changing the
-  # embedding model invalidates every stored vector, so a swap means the LLM
-  # index has to be rebuilt.
-  PAPERLESS_AI_LLM_EMBEDDING_MODEL: "bge-m3"
+  PAPERLESS_AI_LLM_EMBEDDING_MODEL: "qwen3-embedding-0.6b"
   PAPERLESS_AI_LLM_EMBEDDING_CHUNK_SIZE: "1024"
 ---
 apiVersion: apps/v1
@@ -143,20 +140,7 @@ spec:
           # on every paperless pod start: tools belong in the image, not
           # installed at runtime. Dated tag so a change is a reviewable diff
           # here, matching the ci-ansible / alpine-jq convention.
-          #
-          # konairius/, not gitea_admin/, and that is not cosmetic. This is an
-          # INIT container, so paperless does not start without it — and during
-          # a cold rebuild gitea.renner.uno does not exist, because Gitea runs
-          # inside the cluster being rebuilt. A rehearsal left this pod in
-          # Init:ErrImagePull on exactly this image.
-          #
-          # The node falls back to a ghcr.io copy in that window
-          # (machine.registries.mirrors, applied by bootstrap.sh). Containerd
-          # keeps the repository path when it falls through, so the path has to
-          # exist on both registries — hence konairius/, which matches the ghcr
-          # namespace. Steady state still pulls from Gitea and never leaves the
-          # network.
-          image: gitea.renner.uno/konairius/alpine-openssl:2026-09-04
+          image: gitea.renner.uno/gitea_admin/alpine-openssl:2026-07-30
           command:
             - sh
             - -c
@@ -183,7 +167,7 @@ spec:
               drop: ["ALL"]
       containers:
         - name: paperless
-          image: ghcr.io/paperless-ngx/paperless-ngx:3.1.3 # {"$imagepolicy": "flux-system:paperless-ngx"}
+          image: ghcr.io/paperless-ngx/paperless-ngx:3.1.2 # {"$imagepolicy": "flux-system:paperless-ngx"}
           ports:
             - name: http
               containerPort: 8000
@@ -338,6 +322,7 @@ spec:
     - port: 80
       targetPort: http
 ---
+---
 # ExternalName mirror so the paperless-namespace Ingress can backend to
 # the gate outpost in the authentik namespace (cross-namespace Service
 # refs aren't supported in standard Ingress).

kubernetes/apps/trmnl/README.md

@@ -1,36 +0,0 @@
-# trmnl
-
-Terminus (self-hosted TRMNL BYOS server) plus the headless renderer that
-screenshots the Home Assistant entrance dashboard and pushes it in. One
-device: a TRMNL X at the entrance, 1872x1404, 16 greys.
-
-## Device firmware
-
-Terminus mirrors one model-agnostic binary and it is the **ESP32-C3** (OG)
-build. This panel is a TRMNL X, i.e. ESP32-S3, so that file is not the
-artefact for a manual flash here. Use
-`https://trmnl-fw.s3.us-east-2.amazonaws.com/trmnl_x/flash/FW<version>.bin`
-— a merged image at offset `0x0` — and run `esptool image-info` on
-anything before writing it.
-
-The panel runs 1.8.16, flashed over USB on 2026-09-05. It shipped on
-1.8.6, where turning on `firmware_update` froze the screen for as long as
-an update was offered (fixed upstream in 1.8.7, commit `9b7a0f7`).
-
-## Image automation
-
-Both images are enrolled (`infrastructure/image-automation/terminus.yaml`,
-`trmnl-renderer.yaml`) and track the latest `0.x`. A bad bump on either
-surfaces as a **blank or corrupted panel**, not as a failed rollout — no
-readiness probe can tell that a rendered screen became unreadable. Look at
-the panel after a bump lands; revert the tag if it is wrong.
-
-## Device endpoint
-
-`trmnl-device.renner.uno` is the device-facing name, TLS via Traefik
-(`device-ingress.yaml`). `API_URI` must carry the same scheme the device
-polls with, because Terminus bakes it into the absolute `image_url` it
-returns. The LoadBalancer in `device-service.yaml` keeps `10.2.99.89`
-without a hostname as a cleartext fallback that depends on neither
-Traefik nor certificate renewal — the captive portal accepts a bare
-address.

kubernetes/apps/trmnl/device-ingress.yaml

@@ -1,52 +0,0 @@
-# Device-facing endpoint, over TLS.
-#
-# This replaces the reason the cleartext LoadBalancer in device-service.yaml
-# was given a hostname at all. That file claimed the firmware "would reject
-# the internal CA" — but nothing here is signed by an internal CA:
-# letsencrypt-cluster solves DNS-01 against the renner.uno Cloudflare zone,
-# so trmnl.renner.uno already serves a public Let's Encrypt certificate
-# (issuer CN=YR2) that any client with a normal CA bundle trusts. A
-# LAN-only name gets the same treatment — DNS-01 never needs the host to be
-# publicly reachable — so the device can speak HTTPS like everything else.
-#
-# The hostname moves here from the Service's external-dns annotation rather
-# than being duplicated: two sources claiming trmnl-device.renner.uno would
-# have external-dns flapping the record between the Traefik VIP and the
-# device LB. The LoadBalancer keeps its address and stays reachable at
-# http://10.2.99.89 as a nameless fallback if TLS ever turns out to be the
-# thing that breaks.
-#
-# CONSEQUENCE, and the reason API_URI in terminus.yaml changes with this:
-# Traefik's `web` entrypoint carries a global http -> websecure redirect
-# (helmrelease.yaml: `to: websecure, scheme: https, permanent: true`), so
-# once this name points at Traefik, http://trmnl-device.renner.uno answers
-# 308 rather than serving. Whether the firmware follows that is not
-# documented, so the device is provisioned with the https URL and Terminus
-# hands back https image_urls to match. No mixed-scheme fetch.
-apiVersion: networking.k8s.io/v1
-kind: Ingress
-metadata:
-  name: terminus-device
-  namespace: trmnl
-  # No external-dns hostname annotation: external-dns-unifi runs with
-  # --source=ingress, so it registers the hosts in spec.rules on its own.
-  # Same as the terminus Ingress next door.
-  annotations:
-    cert-manager.io/cluster-issuer: letsencrypt-cluster
-spec:
-  ingressClassName: traefik
-  tls:
-    - hosts:
-        - trmnl-device.renner.uno
-      secretName: trmnl-device-tls
-  rules:
-    - host: trmnl-device.renner.uno
-      http:
-        paths:
-          - path: /
-            pathType: Prefix
-            backend:
-              service:
-                name: terminus
-                port:
-                  number: 80

kubernetes/apps/trmnl/device-service.yaml

@@ -1,24 +1,15 @@
-# Cleartext fallback for the device, reachable by address only.
+# Device-facing endpoint. Cleartext HTTP, deliberately.
 #
-# This used to be THE device endpoint, on the premise that the firmware
-# "cannot be handed the internal CA". The premise was wrong: nothing here
-# hands out an internal CA for these names — letsencrypt-cluster solves
-# DNS-01 against the renner.uno Cloudflare zone and issues publicly-trusted
-# certificates for LAN-only hosts, which is why trmnl.renner.uno has always
-# served an issuer CN=YR2 certificate. The device speaks HTTPS via
-# device-ingress.yaml, which now owns the trmnl-device.renner.uno name.
-#
-# Kept, without a hostname, because it is the one path to Terminus that
-# depends on neither Traefik nor certificate renewal: http://10.2.99.89
-# still serves /api/setup and /api/display. If TLS is ever what breaks the
-# panel, the recovery is to type that address into the captive portal.
-#
-# Scope of the exposure is unchanged: LAN-only (no public DNS label, no
-# Cloudflare record), no admin session — the device authenticates with its
-# MAC in the `ID` header plus a per-device api_key.
+# The TRMNL firmware initiates every request and cannot be handed the
+# internal CA, so terminating TLS here would simply break polling. Scope of
+# the exposure: this Service is LAN-only (no public DNS label, no Cloudflare
+# record), it carries no admin session — the device authenticates with its
+# MAC in the `ID` header plus a per-device api_key — and the Terminus admin
+# UI stays on the TLS route at trmnl.renner.uno.
 #
 # Same shape as home-assistant-bgp: a dedicated address on the Cilium BGP
-# pool.
+# pool rather than a TLS-less Ingress, which would be a new pattern for this
+# repo. The device is provisioned with the HOSTNAME, not the address.
 #
 # The address is requested explicitly, matching every other LB service here.
 # Left to allocate freely, Cilium IPAM handed out the first free address in
@@ -39,10 +30,8 @@ metadata:
   labels:
     app.kubernetes.io/name: terminus
     lb-pool: bgp
-  # No external-dns hostname: device-ingress.yaml claims that name. Two
-  # sources claiming trmnl-device.renner.uno would flap the record between
-  # the Traefik VIP and this address (external-dns runs --policy=sync).
   annotations:
+    external-dns.alpha.kubernetes.io/hostname: trmnl-device.renner.uno
     lbipam.cilium.io/ips: "10.2.99.89,fd99::89"
 spec:
   type: LoadBalancer

kubernetes/apps/trmnl/kustomization.yaml

@@ -6,9 +6,6 @@ resources:
   - valkey.yaml
   - terminus.yaml
   - ingress.yaml
-  # TLS device endpoint (owns trmnl-device.renner.uno) plus the nameless
-  # cleartext LoadBalancer it falls back to.
-  - device-ingress.yaml
   - device-service.yaml
   # Headless HA screenshotter. Lives in this namespace because Terminus
   # FETCHES rendered screens back from it (uri delivery mode), so the

kubernetes/apps/trmnl/renderer.yaml

@@ -177,14 +177,7 @@ spec:
         # webhook start failing with "expired JWT access token" about
         # thirty minutes after seeding.
         - name: render-schedule
-          # konairius/, not gitea_admin/: this container starts during a cold
-          # rebuild, when Gitea does not exist because it runs inside the
-          # cluster being rebuilt — a rehearsal left it in Init:ImagePullBackOff.
-          # The node falls back to a ghcr.io copy then
-          # (machine.registries.mirrors, applied by bootstrap.sh), and containerd
-          # keeps the repository path, so it has to exist on both registries.
-          # Steady state still pulls from Gitea.
-          image: gitea.renner.uno/konairius/alpine-jq:2026-09-04
+          image: gitea.renner.uno/gitea_admin/alpine-jq:2026-07-30
           command:
             - sh
             - -c
@@ -234,12 +227,16 @@ spec:
             capabilities:
               drop: ["ALL"]
       containers:
-        # Enrolled in image automation (flux-system:trmnl-renderer). A bad
-        # bump shows up as a corrupted screen, not a failed rollout — check
-        # the panel after one lands. ARCH-SUFFIXED image: upstream publishes
-        # no multi-arch manifest.
+        # Pinned, not enrolled in image automation — same reasoning as
+        # Terminus itself. This drives a wall-mounted panel and upstream
+        # is pre-1.0; an unattended bump that changes the dithering
+        # pipeline or the webhook payload shows up as a corrupted
+        # screen, not as a failed rollout.
+        #
+        # Note the image is ARCH-SUFFIXED: upstream publishes no
+        # multi-arch manifest, only trmnl-ha-amd64 and trmnl-ha-aarch64.
         - name: renderer
-          image: ghcr.io/usetrmnl/trmnl-ha-amd64:0.10.3 # {"$imagepolicy": "flux-system:trmnl-renderer"}
+          image: ghcr.io/usetrmnl/trmnl-ha-amd64:0.9.5
           ports:
             - name: http
               containerPort: 10000

kubernetes/apps/trmnl/terminus.yaml

@@ -7,17 +7,11 @@ data:
   TZ: "Europe/Berlin"
   HANAMI_PORT: "2345"
   # What the device is told to fetch from. Terminus bakes this into the
-  # absolute `image_url` it hands back on /api/display, so it has to be a
-  # name the DEVICE can resolve and fetch — and it has to carry the same
-  # scheme the device polls with, or every screen fetch is a second,
-  # cleartext connection. No trailing slash: the same string is typed into
-  # the captive portal at provisioning time.
-  #
-  # https since device-ingress.yaml took over trmnl-device.renner.uno. The
-  # earlier value was http on the theory that the firmware would reject an
-  # internal CA; there is no internal CA here — letsencrypt-cluster issues
-  # publicly-trusted certs for LAN-only names via DNS-01.
-  API_URI: "https://trmnl-device.renner.uno"
+  # absolute `image_url` it hands back on /api/display, so it MUST be the
+  # cleartext LAN name, not trmnl.renner.uno — the firmware speaks plain
+  # HTTP and would reject the internal CA. No trailing slash: the same
+  # string is typed into the captive portal at provisioning time.
+  API_URI: "http://trmnl-device.renner.uno"
   KEYVALUE_URL: "redis://terminus-valkey:6379"
   # Rack::Attack rate-limit bypass. Beyond upstream's RFC1918 defaults this
   # adds fc00::/7, which covers the pod CIDR (fd10:244::/48), the Cilium LB
@@ -134,7 +128,7 @@ spec:
         # is a device-bricking class of risk. Bumps are a reviewed, manual
         # act — see kubernetes/infrastructure/image-automation/README.md.
         - name: terminus
-          image: ghcr.io/usetrmnl/terminus:0.72.0 # {"$imagepolicy": "flux-system:terminus"}
+          image: ghcr.io/usetrmnl/terminus:0.68.0
           ports:
             - name: http
               containerPort: 2345
@@ -197,7 +191,7 @@ spec:
             capabilities:
               drop: ["ALL"]
         - name: sidekiq
-          image: ghcr.io/usetrmnl/terminus:0.72.0 # {"$imagepolicy": "flux-system:terminus"}
+          image: ghcr.io/usetrmnl/terminus:0.68.0
           command: ["bundle", "exec", "sidekiq", "-r", "./config/sidekiq.rb"]
           envFrom:
             - configMapRef:

kubernetes/apps/woodpecker/rbac.yaml

@@ -1,3 +1,18 @@
+# Dedicated namespace where the kubernetes backend spawns pipeline step pods,
+# isolated from the apps namespace.
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: woodpecker-ci
+  labels:
+    app.kubernetes.io/name: woodpecker
+    # Image-build steps run a privileged docker/buildx plugin pod (see the
+    # server's WOODPECKER_PLUGINS_PRIVILEGED). Talos enforces Pod Security;
+    # without this the privileged step pods are rejected at admission. The
+    # namespace is dedicated to ephemeral CI step pods, so scoping the
+    # exemption here keeps it off apps/.
+    pod-security.kubernetes.io/enforce: privileged
+---
 apiVersion: v1
 kind: ServiceAccount
 metadata:

kubernetes/clusters/homelab/home-assistant.yaml

@@ -28,19 +28,6 @@ spec:
     - name: infrastructure
     - name: infrastructure-crds
     - name: infrastructure-ceph-cluster
-    # ha-pg-objectstore.yaml declares an ObjectStore, whose CRD comes from the
-    # CNPG Barman plugin in infrastructure-cnpg-barman. Without this edge the
-    # kind is unknown when this path is first applied, and a server-side
-    # dry-run rejects the ENTIRE set over the one object it cannot recognise:
-    #
-    #   ObjectStore/home-assistant/ha-pg-store dry-run failed:
-    #   no matches for kind "ObjectStore" in version ...
-    #
-    # Every other ObjectStore user — authentik, immich, paperless, trmnl,
-    # woodpecker — already declares it. This was the only one that did not, and
-    # it took speech-to-phrase, zone-configurator and matter-server down with
-    # it, since all three wait on home-assistant.
-    - name: infrastructure-cnpg-barman
 ---
 apiVersion: kustomize.toolkit.fluxcd.io/v1
 kind: Kustomization

kubernetes/clusters/homelab/infrastructure.yaml

@@ -122,20 +122,8 @@ spec:
   path: ./kubernetes/infrastructure/rook-ceph-cluster
   prune: true
   wait: true
-  # Deliberately NOT dependsOn: infrastructure. That edge closed a cycle no
-  # existing cluster could ever hit and no new one could ever escape:
-  # infrastructure is Ready only once its openbao healthCheck passes, openbao-0
-  # only runs once its PVC binds, and that PVC wants the ceph-block StorageClass
-  # defined in this very Kustomization. A cold rebuild stopped here for good,
-  # with openbao-0 Pending on "unbound immediate PersistentVolumeClaims".
-  #
-  # The edge was never needed. infrastructure runs wait: false, so it applies
-  # the rook-ceph namespace and the operator immediately and only its readiness
-  # waits on the healthChecks — everything this path needs exists long before
-  # infrastructure reports Ready. On an empty API server this Kustomization
-  # retries on retryInterval until the operator installs its CRDs, the same
-  # self-healing shape post-crds already relies on; on a live cluster the CRDs
-  # are always present and nothing changes.
+  dependsOn:
+    - name: infrastructure
   # kstatus doesn't know how to read Rook CR readiness, so without these
   # this Kustomization reports Ready as soon as the manifests apply — long
   # before OSDs/mons/MDS are actually serving I/O. Apps depending on this

kubernetes/infrastructure/cilium/helmrelease.yaml

@@ -70,23 +70,9 @@ spec:
       # map only shows up as "no space left on device" in the log while pod
       # sandbox creation fails (#452). cilium_bpf_map_pressure makes it
       # alertable beforehand.
-      #
-      # That reasoning is about the metrics, which this line provides. The
-      # ServiceMonitor is only the scrape wiring and is declared standalone in
-      # apps/monitoring-rules instead — the same treatment velero/helmrelease
-      # already gives its own, and for the same reason: the chart's template is
-      # gated on monitoring.coreos.com/v1 existing at install time.
-      #
-      # Here it is worse than for velero. Cilium's chart does not skip the
-      # template, it aborts rendering — "Service Monitor requires
-      # monitoring.coreos.com/v1 CRDs" — and cilium is one of the seven
-      # healthChecks the infrastructure Kustomization waits on. So on an empty
-      # cluster the CNI blocked infrastructure, which blocked grafana-db, which
-      # blocked monitoring, which is where those CRDs come from. The lowest
-      # layer in the cluster was waiting on the highest.
       enabled: true
       serviceMonitor:
-        enabled: false
+        enabled: true
     bpf:
       # Per-endpoint policy map size: one entry per (identity, direction,
       # protocol, port) allowed. When it fills, endpoint regeneration fails

kubernetes/infrastructure/cnpg-barman-plugin/manifest.yaml

@@ -937,7 +937,7 @@ subjects:
 # with no marker. Re-apply this block, or the skew silently returns.
 apiVersion: v1
 stringData:
-  SIDECAR_IMAGE: ghcr.io/cloudnative-pg/plugin-barman-cloud-sidecar:v0.15.0 # {"$imagepolicy": "flux-system:cnpg-barman-plugin-sidecar"}
+  SIDECAR_IMAGE: ghcr.io/cloudnative-pg/plugin-barman-cloud-sidecar:v0.14.0 # {"$imagepolicy": "flux-system:cnpg-barman-plugin-sidecar"}
 kind: Secret
 metadata:
   name: plugin-barman-cloud-kg5c2795f7
@@ -998,7 +998,7 @@ spec:
             secretKeyRef:
               key: SIDECAR_IMAGE
               name: plugin-barman-cloud-kg5c2795f7
-        image: ghcr.io/cloudnative-pg/plugin-barman-cloud:v0.15.0 # {"$imagepolicy": "flux-system:cnpg-barman-plugin"}
+        image: ghcr.io/cloudnative-pg/plugin-barman-cloud:v0.14.0 # {"$imagepolicy": "flux-system:cnpg-barman-plugin"}
         name: barman-cloud
         ports:
         - containerPort: 9090

kubernetes/infrastructure/crowdsec/helmrelease.yaml

@@ -65,21 +65,6 @@ spec:
           - 45.157.177.26/32
           - 2a03:4000:4b:c1e:4474:43ff:fedf:8f85/128
           - 2a03:4000:28:f6e:a4c7:59ff:fe1e:9a29/128
-          # netbird.renner.uno — management, signal, relay and TURN.
-          #
-          # Not optional once a bouncer is armed. Its chain sits on the INPUT
-          # hook, the proxy hosts run no other firewall, and there is therefore
-          # no conntrack rule accepting established flows ahead of it. A ban on
-          # this address would drop the RETURN traffic of the proxies' own
-          # outbound control connection and take them off the mesh — with no
-          # inbound attack involved at all.
-          #
-          # Checked when this was added: none of these addresses appear in the
-          # current decision set. That is the state today, not a guarantee —
-          # these are rented addresses on a hoster whose ranges do end up on
-          # community blocklists.
-          - 162.55.50.97/32
-          - 2a01:4f8:c2c:4579::1/128
 
     # mTLS between agents and the LAPI. Do not switch back to password auth:
     # the LAPI runs without persistence, so its machine-registration DB is
@@ -99,7 +84,7 @@ spec:
     # required.
     #
     # NOTE: enabling this makes the LAPI serve HTTPS-only. The Traefik ingress
-    # bouncer is currently INERT (detached, see post-crds/traefik-crowdsec-middleware.yaml)
+    # bouncer is currently INERT (detached, see traefik/crowdsec-middleware.yaml)
     # so there is no live impact, but re-enabling it will require pointing the
     # plugin at https + trusting this CA (or giving it a bouncer-ou client cert).
     tls:
@@ -123,24 +108,6 @@ spec:
 
     lapi:
       replicas: 1
-      service:
-        # Backend wiring for the public LAPI Ingress (lapi-ingress.yaml).
-        #
-        # These belong on the SERVICE, not on the Ingress: under the Kubernetes
-        # Ingress provider `traefik.ingress.kubernetes.io/service.*` are service
-        # annotations, and on an Ingress they are silently ignored. The symptom
-        # is Traefik speaking plaintext to a TLS-only backend, which the LAPI
-        # answers with "Client sent an HTTP request to an HTTPS server." and a
-        # 400 — an error that looks like a malformed request rather than a
-        # scheme mismatch.
-        #
-        # With tls.enabled the LAPI serves HTTPS on 8080 and offers no
-        # plaintext port, so https here is mandatory rather than a hardening
-        # choice. The ServersTransport carries the chart's own CA, since that
-        # certificate is self-signed.
-        annotations:
-          traefik.ingress.kubernetes.io/service.serversscheme: https
-          traefik.ingress.kubernetes.io/service.serverstransport: crowdsec-crowdsec-lapi@kubernetescrd
       # Far below the chart default, which reserves half a core per LAPI for a
       # workload that draws single-digit millicores — enough to leave the
       # cluster unschedulable with one node out. Only the reservation shrinks;
@@ -217,32 +184,6 @@ spec:
         - namespace: ingress
           podName: traefik-*
           program: traefik
-          # Read by polling, not inotify. Every path under
-          # /var/log/containers is a symlink into /var/log/pods, and kubelet
-          # rotates the target once it reaches containerLogMaxSize (10Mi,
-          # roughly every eight hours for Traefik here). An inotify tailer
-          # does not survive that rotation: it reopens the path, waits for a
-          # file that is in fact already present, and never reads another
-          # line. Nothing about that state looks broken from outside — the
-          # pod stays Running and Ready, the LAPI keeps answering, and the
-          # cumulative counters keep displaying whatever they reached, so the
-          # only symptom is that no scenario ever fires again.
-          #
-          # CrowdSec detects the hazard itself and logs it at startup, with
-          # this exact remedy in the message: "is a symlink, but inotify
-          # polling is enabled. Crowdsec will not be able to detect rotation.
-          # Consider setting poll_without_inotify to true". Upstream carries
-          # the same case as crowdsec#2449 (k3s + Traefik), and the chart
-          # gained this field in helm-charts#109 specifically to fix it.
-          #
-          # Orthogonal to the force_inotify: true that the chart hardcodes
-          # into acquis.yaml and offers no way to unset. That one governs the
-          # fsnotify watch on the directory, i.e. discovering the log of a
-          # replacement Traefik pod; this one governs how an already
-          # discovered file is read. Both true is the supported combination.
-          #
-          # Costs CPU per open file per upstream docs — two files per node.
-          poll_without_inotify: true
       env:
         - name: COLLECTIONS
           value: "crowdsecurity/traefik crowdsecurity/http-cve crowdsecurity/appsec-virtual-patching crowdsecurity/appsec-generic-rules"

kubernetes/infrastructure/crowdsec/kustomization.yaml

@@ -1,5 +1,7 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
+  - bouncer-key-externalsecret.yaml
+  - edge-bouncer-key-externalsecret.yaml
   - lapi-ingress.yaml
   - helmrelease.yaml

kubernetes/infrastructure/crowdsec/lapi-ingress.yaml

@@ -31,12 +31,22 @@
 # serves read-only decision streams to a valid bouncer key and nothing else.
 # The admin surface is cscli, in-cluster. If the source address ever becomes
 # deterministic, an ipAllowList can be added on top.
-#
+---
 # The LAPI serves HTTPS only once tls.enabled is set — there is no plaintext
 # fallback and no second port; 8080 is the TLS port. Its certificate comes
 # from the chart's own self-signed CA, so Traefik needs that CA to verify the
-# backend; the ServersTransport carrying it is a traefik.io CRD consumer and
-# lives in post-crds/crowdsec-lapi-serverstransport.yaml.
+# backend. SANs include crowdsec-service.crowdsec.svc.cluster.local, which is
+# what the Service resolves to.
+apiVersion: traefik.io/v1alpha1
+kind: ServersTransport
+metadata:
+  name: crowdsec-lapi
+  namespace: crowdsec
+spec:
+  serverName: crowdsec-service.crowdsec.svc.cluster.local
+  rootCAsSecrets:
+    - crowdsec-root-secret
+---
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
@@ -50,12 +60,8 @@ metadata:
     cert-manager.io/cluster-issuer: letsencrypt-cluster
     # NOT behind the Authentik gate: the client is a daemon with an API key,
     # not a browser that can complete an OIDC flow.
-    #
-    # The backend scheme and ServersTransport are NOT set here. Under the
-    # Kubernetes Ingress provider, `traefik.ingress.kubernetes.io/service.*`
-    # are SERVICE annotations — on an Ingress they are silently ignored, and
-    # Traefik then talks plaintext to a TLS-only backend. They live on the
-    # chart's Service instead, via lapi.service.annotations in helmrelease.yaml.
+    traefik.ingress.kubernetes.io/service.serversscheme: https
+    traefik.ingress.kubernetes.io/service.serverstransport: crowdsec-crowdsec-lapi@kubernetescrd
 spec:
   ingressClassName: traefik
   rules:

kubernetes/infrastructure/external-dns-unifi/kustomization.yaml

@@ -1,4 +1,5 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
+  - externalsecret.yaml
   - helmrelease.yaml

kubernetes/infrastructure/image-automation/README.md

@@ -96,13 +96,28 @@ The security-scan agent will continue to surface vuln findings on
 these as "Skipped / Flux image-automation owns this" even without
 a policy.
 
-Only a tag-format problem belongs on that list. The policy is to run the
-latest version and let image-automation hold us there wherever the tags
-allow it, so an image with a usable semver stream gets enrolled even when
-a bump carries risk. Where a bad bump is hard to detect, the answer is a
-note about what to check afterwards — see `apps/trmnl/README.md` for the
-panel, whose failure mode is a blank screen rather than a failed rollout
-— not an exclusion.
+- `ghcr.io/usetrmnl/terminus` — publishes a usable `0.x.0` tag stream, so
+  unlike the three above this is **not** a tag-format problem. It is left
+  unenrolled because the blast radius is hardware: Terminus is pre-1.0,
+  cuts no GitHub Releases and ships no changelog, and it serves the
+  firmware and `/api/display` contract that the TRMNL X panel polls. An
+  unattended bump that changes either can leave a wall-mounted device
+  dark, with no rollback path that doesn't involve the captive portal.
+  Bumps are a reviewed, manual act: read the commit range, then edit the
+  tag in `kubernetes/apps/trmnl/terminus.yaml` (both containers).
+
+  The Valkey sidecar alongside it *is* enrolled (`valkey.yaml`) — it holds
+  only the Sidekiq queue and touches no device-facing contract.
+
+- `ghcr.io/usetrmnl/trmnl-ha-amd64` — the TRMNL screenshot renderer, for
+  the same reason and one more. It owns the dithering pipeline and the
+  webhook payload shape, so a bad bump surfaces as a *corrupted or blank
+  panel*, not as a failed rollout — there is no readiness signal that
+  catches "the image is now unreadable". Separately, upstream publishes
+  **no multi-arch manifest**: only the arch-suffixed `trmnl-ha-amd64` and
+  `trmnl-ha-aarch64` repositories exist, so an ImagePolicy here would also
+  be pinning us to one architecture in a way the tag does not make
+  obvious. Bump manually in `kubernetes/apps/trmnl/renderer.yaml`.
 
 ## Re-import hazard: cnpg-barman-plugin
 

kubernetes/infrastructure/image-automation/ceph.yaml

@@ -9,26 +9,9 @@ spec:
 ---
 # v-prefix tags (v19.2.3). Upstream also publishes date-suffixed builds
 # (v19.2.3-20250717) and moving major/minor tags (v19, v19.2); the
-# filter pins us to the bare three-component release.
-#
-# The range was 19.x, to keep the 19 -> 20 (Tentacle) cutover a deliberate
-# upgrade tied to the Rook operator version. That coupling had already come
-# apart without anyone noticing: Rook's floating constraint pulled v1.20 in
-# June, and with it a ceph-csi image whose client libraries are Ceph 20.2.1.
-# The cluster stayed on 19.2.6, so every CSI pod has since been talking to
-# the mons with a client a whole major ahead of them.
-#
-# That is not academic. The 19.2.6 mons mint auth keys as aes256k, and the
-# 20.2.1 client cannot decode them — "auth: failed to decode key 'AgB...'".
-# Nothing broke only because the four CSI users predate the cipher change and
-# still carry legacy aes keys, which Ceph itself flags as insecure. The moment
-# one is recreated — ceph auth rotate, a deleted Secret, Rook resyncing — the
-# provisioner stops and no PVC binds. A rebuild rehearsal hits it immediately,
-# because a fresh cluster has no legacy keys at all.
-#
-# So the cutover happens now, with the version the client already is. Rook
-# v1.20 supports Tentacle from v20.2.1; v20.2.0 is excluded deliberately —
-# upstream reports data corruption there when read affinity is enabled.
+# filter pins us to the bare three-component release. Range stays at
+# 19.x — the 19 → 20 (Tentacle) cutover should be a deliberate
+# upgrade tied to the Rook operator version.
 apiVersion: image.toolkit.fluxcd.io/v1
 kind: ImagePolicy
 metadata:
@@ -42,4 +25,4 @@ spec:
     extract: '$ver'
   policy:
     semver:
-      range: ">=20.2.1 <21.0.0"
+      range: "19.x"

kubernetes/infrastructure/image-automation/kustomization.yaml

@@ -18,9 +18,7 @@ resources:
   - mosquitto.yaml
   - redis.yaml
   - shelly-manager.yaml
-  - terminus.yaml
   - tika.yaml
-  - trmnl-renderer.yaml
   - valkey.yaml
   - velero-plugin-for-aws.yaml
   - vsftpd.yaml

kubernetes/infrastructure/image-automation/terminus.yaml

@@ -1,30 +0,0 @@
-apiVersion: image.toolkit.fluxcd.io/v1
-kind: ImageRepository
-metadata:
-  name: terminus
-  namespace: flux-system
-spec:
-  image: ghcr.io/usetrmnl/terminus
-  interval: 1h
----
-# Pre-1.0, so the range is 0.x rather than a major pin: upstream's minors are
-# its release stream, and holding at a single minor is how this drifted four
-# releases behind while it was excluded from automation.
-#
-# A bad bump here surfaces as a blank or corrupted panel rather than a failed
-# rollout — no readiness probe can tell that a rendered screen became
-# unreadable. Check the panel after one lands; revert the tag if it is wrong.
-apiVersion: image.toolkit.fluxcd.io/v1
-kind: ImagePolicy
-metadata:
-  name: terminus
-  namespace: flux-system
-spec:
-  imageRepositoryRef:
-    name: terminus
-  filterTags:
-    pattern: '^(?P<ver>\d+\.\d+\.\d+)$'
-    extract: '$ver'
-  policy:
-    semver:
-      range: "0.x"

kubernetes/infrastructure/image-automation/trmnl-renderer.yaml

@@ -1,29 +0,0 @@
-apiVersion: image.toolkit.fluxcd.io/v1
-kind: ImageRepository
-metadata:
-  name: trmnl-renderer
-  namespace: flux-system
-spec:
-  image: ghcr.io/usetrmnl/trmnl-ha-amd64
-  interval: 1h
----
-# Upstream ships no multi-arch manifest, only the arch-suffixed repositories.
-# That does not obstruct a policy: the image reference already names the
-# architecture, so this pins nothing the tag did not.
-#
-# The floating `0.9` / `0.10` series tags are filtered out; only fully
-# qualified patch versions are candidates. Same panel caveat as terminus.yaml.
-apiVersion: image.toolkit.fluxcd.io/v1
-kind: ImagePolicy
-metadata:
-  name: trmnl-renderer
-  namespace: flux-system
-spec:
-  imageRepositoryRef:
-    name: trmnl-renderer
-  filterTags:
-    pattern: '^(?P<ver>\d+\.\d+\.\d+)$'
-    extract: '$ver'
-  policy:
-    semver:
-      range: "0.x"

kubernetes/infrastructure/kustomization.yaml

@@ -18,7 +18,6 @@ resources:
   - cloudnative-pg/
   - mariadb-operator/
   - netbird-ingress/
-  - netbird-proxy-services/
   - coredns-config/
   - k8s-gateway/
   - external-dns-unifi/

kubernetes/infrastructure/multus/cni-plugins-installer.yaml

@@ -30,11 +30,6 @@ spec:
       labels:
         app: cni-plugins-installer
     spec:
-      # Same reasoning as multus-daemonset.yaml. This one installs the macvlan
-      # binary the ceph-san attachment is made of, so a node that cannot
-      # schedule it has no plugin to delegate to — the failure the init
-      # container comment below already describes, reached by a different road.
-      priorityClassName: system-node-critical
       hostNetwork: true
       tolerations:
         - operator: Exists
@@ -49,32 +44,7 @@ spec:
           # at all and put github.com on the critical path of every node rebuild.
           # Dated tag so a version change is a reviewable diff here, matching the
           # ci-ansible / alpine-jq convention.
-          #
-          # konairius/, not gitea_admin/, and the difference is what makes a
-          # rebuild possible. Gitea runs inside this cluster, so during a rebuild
-          # it does not exist: a rehearsal sat here in ImagePullBackOff with "no
-          # route to host", and because the plugins never installed there was no
-          # macvlan, so Multus could not attach ceph-san, so Rook's network
-          # canary never started, so the CephCluster hung at "Configuring Ceph
-          # Mons" and OpenBao waited forever on a PVC that could not bind. The
-          # failure surfaced four layers from its cause.
-          #
-          # This used to point straight at ghcr.io, which fixed the rebuild and
-          # broke the normal case: every steady-state pull then left the network
-          # for an image the local registry already holds. The fallback belongs
-          # to the node, not to the manifest — machine.registries.mirrors names
-          # Gitea first and ghcr.io second (bootstrap.sh), so nothing here has to
-          # know about the recovery path.
-          #
-          # Containerd keeps the repository path when it falls through, so the
-          # path must resolve on both registries; the build pushes one digest to
-          # konairius/ on each, verified equal. Bumping the tag means building
-          # first — an unpublished tag is unpullable on both.
-          #
-          # No imagePullSecret: the ghcr namespace is private and its credential
-          # is in machine.registries too, because a Secret would have to come
-          # from OpenBao, which needs Ceph, which needs these binaries.
-          image: gitea.renner.uno/konairius/cni-plugins:2026-09-04
+          image: gitea.renner.uno/gitea_admin/cni-plugins:2026-07-30
           command:
             - sh
             - -c

kubernetes/infrastructure/multus/multus-daemonset.yaml

@@ -154,40 +154,6 @@ spec:
         app: multus
         name: multus
     spec:
-      # Upstream ships this DaemonSet with no priority class, so it runs at 0
-      # and can preempt nothing. Any event that makes the pod need a fresh
-      # scheduling decision — a rollout, a node restart, an eviction — then has
-      # to fit it onto a node that may since have filled up, and if it does not
-      # fit it simply stays Pending. The node has no multus after that, and
-      # nothing reports it: pods still start, they just never get their
-      # secondary attachment, and the annotation everything waits on never
-      # appears.
-      #
-      # Seen on a cold rebuild: two of four nodes sat at 99% memory requests
-      # when the DaemonSet rolled, both multus pods went Pending, and Rook's
-      # network canary hung in Init:0/1 waiting for a network-status annotation
-      # that could never come. The CephCluster stayed in Progressing, which is
-      # what infrastructure-ceph-cluster gates on, so the whole apps layer
-      # stopped behind a scheduling decision three layers away.
-      #
-      # node-critical, the same class cilium and cni-dhcp-daemon carry, because
-      # that is what this is: without multus a node cannot attach a secondary
-      # network, and only that node is affected. Nothing cluster-wide fails.
-      #
-      # This does outrank kube-apiserver, which Talos runs as a static pod at
-      # system-cluster-critical (2000000000, against 2000001000 here), and on a
-      # control plane that has run out of memory the kubelet will pick the API
-      # server as the pod to evict. That was observed twice, at 6GiB and at
-      # 8GiB.
-      #
-      # It is not a reason to demote this DaemonSet. Kubernetes puts the CNI
-      # above the control plane deliberately — a node without its CNI cannot run
-      # anything at all — and a class name is a statement about what the pod is,
-      # not a dial for picking a number. The eviction says the control plane is
-      # starved, and that is where it has to be fixed: enough allocatable
-      # memory, or kubeReserved/systemReserved so the static pods keep what they
-      # need. Demoting this to cluster-critical would only move the victim.
-      priorityClassName: system-node-critical
       hostNetwork: true
       hostPID: true
       tolerations:

kubernetes/infrastructure/multus/whereabouts-daemonset.yaml

@@ -146,11 +146,6 @@ spec:
         app: whereabouts
         name: whereabouts
     spec:
-      # Same reasoning as multus-daemonset.yaml: this is the IPAM half of the
-      # secondary-network path, so a node without it cannot hand out an address
-      # on ceph-san and the attachment fails just as completely as if multus
-      # itself were missing. Upstream leaves the priority class unset.
-      priorityClassName: system-node-critical
       hostNetwork: true
       serviceAccountName: whereabouts
       # Every node here is a control plane node, so the NoSchedule toleration

kubernetes/infrastructure/namespaces/namespaces.yaml

@@ -69,8 +69,6 @@ kind: Namespace
 metadata:
   name: paperless
 ---
-# A disposable copy of paperless's database, for measuring how the local LLM
-# classifies documents without experimenting on the live archive.
 apiVersion: v1
 kind: Namespace
 metadata:
@@ -190,31 +188,3 @@ apiVersion: v1
 kind: Namespace
 metadata:
   name: ai
----
-# Where the Woodpecker kubernetes backend spawns pipeline step pods, isolated
-# from `apps`. Distinct from `woodpecker`, which holds the service itself.
-#
-# Here rather than in apps/woodpecker for the reason spelled out above that
-# namespace: post-crds ships this namespace's CiliumNetworkPolicies via
-# ../network-policies/, and infrastructure-crds runs before every app KS. With
-# the namespace declared in the workload KS the order inverted, and a rebuild
-# stopped with
-#
-#   CiliumNetworkPolicy/woodpecker-ci/default-deny-ingress not found:
-#   namespaces "woodpecker-ci" not found
-#
-# which held infrastructure-crds, and with it all twenty app Kustomizations
-# that depend on it. The same reasoning was already written down for
-# `woodpecker` and simply not carried across to its sibling.
-#
-# Image-build steps run a privileged docker/buildx plugin pod (see the server's
-# WOODPECKER_PLUGINS_PRIVILEGED). Talos enforces Pod Security; without the
-# label the privileged step pods are rejected at admission. The namespace holds
-# nothing but ephemeral CI step pods, so the exemption stays off apps/.
-apiVersion: v1
-kind: Namespace
-metadata:
-  name: woodpecker-ci
-  labels:
-    app.kubernetes.io/name: woodpecker
-    pod-security.kubernetes.io/enforce: privileged

kubernetes/infrastructure/netbird-ingress/kustomization.yaml

@@ -1,4 +1,5 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
+  - externalsecret.yaml
   - daemonset.yaml

kubernetes/infrastructure/netbird-proxy-services/deployment.yaml

@@ -1,108 +0,0 @@
-# Reconciles NetBird reverse-proxy services from Ingresses that carry
-# dns.renner.uno/public: one TLS-passthrough service per host on :443,
-# targeting Traefik's ClusterIP through a subnet resource over the service
-# CIDR (ci/images/netbird-proxy-services/controller.py).
-#
-# openbao namespace, like the rotators: its default-deny already allows egress
-# to OpenBao, DNS, the apiserver and `world` (netbird.renner.uno), and the
-# NetBird PAT is read from OpenBao on every pass through the
-# netbird-proxy-services role — policies/netbird-proxy-services.hcl, applied
-# once by hand like the rotator roles. Until that role exists the pod logs an
-# auth error each minute and does nothing.
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: netbird-proxy-services
-  namespace: openbao
-  labels:
-    app.kubernetes.io/name: netbird-proxy-services
----
-# Read-only, cluster-wide: public Ingresses live in many namespaces, the
-# Traefik Service in ingress/.
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRole
-metadata:
-  name: netbird-proxy-services-read
-rules:
-  - apiGroups: ["networking.k8s.io"]
-    resources: ["ingresses"]
-    verbs: ["get", "list", "watch"]
-  - apiGroups: [""]
-    resources: ["services"]
-    verbs: ["get"]
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRoleBinding
-metadata:
-  name: netbird-proxy-services-read
-roleRef:
-  apiGroup: rbac.authorization.k8s.io
-  kind: ClusterRole
-  name: netbird-proxy-services-read
-subjects:
-  - kind: ServiceAccount
-    name: netbird-proxy-services
-    namespace: openbao
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: netbird-proxy-services
-  namespace: openbao
-  labels:
-    app.kubernetes.io/name: netbird-proxy-services
-spec:
-  replicas: 1
-  # One writer against the management API at a time.
-  strategy:
-    type: Recreate
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: netbird-proxy-services
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/name: netbird-proxy-services
-    spec:
-      serviceAccountName: netbird-proxy-services
-      securityContext:
-        runAsNonRoot: true
-        # Numeric, or the kubelet refuses the image's named user with
-        # "cannot verify user is non-root". The script needs no home and no
-        # writable path, so nobody/nogroup is enough.
-        runAsUser: 65534
-        runAsGroup: 65534
-        seccompProfile:
-          type: RuntimeDefault
-      containers:
-        - name: controller
-          # Pinned by digest: the date tag is rebuilt on every push of the
-          # image directory (branch and master alike) and the registry keeps
-          # only the newest manifest per tag, so the digest must come from
-          # the MASTER build of the code change.
-          image: gitea.renner.uno/gitea_admin/netbird-proxy-services:2026-09-07@sha256:caffa308d802c6fcc6aea919ac94657c08b993651baa6b8d18de395bcc40523e
-          securityContext:
-            allowPrivilegeEscalation: false
-            readOnlyRootFilesystem: true
-            capabilities:
-              drop: ["ALL"]
-          env:
-            - name: NB_MANAGEMENT_URL
-              value: https://netbird.renner.uno
-            - name: BAO_ADDR
-              value: http://openbao.openbao.svc.cluster.local:8200
-            - name: BAO_ROLE
-              value: netbird-proxy-services
-            # Must match cluster.network.serviceSubnets in talos/patches/cluster.yaml.
-            - name: SERVICE_CIDR
-              value: "10.96.0.0/12"
-            - name: TRAEFIK_SERVICE
-              value: ingress/traefik
-            - name: INTERVAL_SECONDS
-              value: "60"
-          resources:
-            requests:
-              cpu: 10m
-              memory: 32Mi
-            limits:
-              memory: 64Mi

kubernetes/infrastructure/netbird-proxy-services/kustomization.yaml

@@ -1,4 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-resources:
-  - deployment.yaml

kubernetes/infrastructure/network-policies/ai-ns.yaml

@@ -34,17 +34,6 @@ spec:
         - ports:
             - port: "8080"
               protocol: TCP
-    # Home Assistant's llama.cpp conversation agent calls the chat model only —
-    # it has no use for the embedding endpoint, so unlike paperless above this
-    # grant does not extend to llama-embed.
-    - fromEndpoints:
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": home-assistant
-            app.kubernetes.io/name: home-assistant
-      toPorts:
-        - ports:
-            - port: "8080"
-              protocol: TCP
     - fromEndpoints:
         - matchLabels:
             "k8s:io.kubernetes.pod.namespace": monitoring

kubernetes/infrastructure/network-policies/home-assistant-ns.yaml

@@ -167,18 +167,6 @@ spec:
     - toEndpoints:
         - matchLabels:
             "k8s:io.kubernetes.pod.namespace": influxdb
-    # The cluster's own chat model, for the llama.cpp conversation agent that
-    # backs Assist. Scoped to the one pod and port rather than the ai namespace
-    # as a whole, so this grant cannot widen if something else is placed there —
-    # the embedding endpoint next to it is deliberately not included.
-    - toEndpoints:
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": ai
-            app.kubernetes.io/name: llama-server
-      toPorts:
-        - ports:
-            - port: "8080"
-              protocol: TCP
     # traefik for east-west OIDC roundtrips that go through the
     # public hostname via k8s_gateway.
     - toEndpoints:

kubernetes/infrastructure/openbao/policies/netbird-proxy-services.hcl

@@ -1,23 +0,0 @@
-# OpenBao policy: netbird-proxy-services
-#
-# The reverse-proxy-services controller (Deployment netbird-proxy-services in
-# the openbao namespace) reads the NetBird PAT on every reconcile pass and
-# calls the management API with it. Read-only on that one path; it shares the
-# bootstrap credential with rotator-netbird but not that role's PATCH rights.
-#
-# NOT applied by Flux. Apply ONCE with an admin token:
-#
-#   bao policy write netbird-proxy-services kubernetes/infrastructure/openbao/policies/netbird-proxy-services.hcl
-#   bao write auth/kubernetes/role/netbird-proxy-services \
-#     bound_service_account_names=netbird-proxy-services \
-#     bound_service_account_namespaces=openbao \
-#     policies=netbird-proxy-services ttl=15m
-#
-# The PAT itself is the one rotator-netbird already uses
-# (homelab/infrastructure/secret-rotators-bootstrap/netbird, key `pat`).
-#
-# Do NOT set token_no_default_policy=true (kv-v2 preflight needs the default policy).
-
-path "homelab/data/infrastructure/secret-rotators-bootstrap/netbird" {
-  capabilities = ["read"]
-}

kubernetes/infrastructure/openbao/rotation/flux-git.yaml

@@ -10,8 +10,33 @@
 # name hairpins off the ingress LB from a pod. Needs BOTH netpol halves
 # (gitea-allow-flux-rotator + rotator-gitea-flux-egress).
 #
-# The ExternalSecret that publishes the rotated token to Flux lives in
-# ../../post-crds/flux-git-externalsecret.yaml (needs ESO's CRDs).
+# The ExternalSecret MERGES into Flux's existing flux-system secret (does not
+# own it) so a sync failure leaves the last-good value in place; Flux's
+# source-controller reads it fresh each fetch, no restart. OpenBao policy/role
+# rotator-gitea-flux + the gitea-flux bootstrap (flux account password) are
+# provisioned by hand (see policies/rotator-gitea-flux.hcl).
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: flux-system-git-auth
+  namespace: flux-system
+spec:
+  refreshInterval: 10m
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: openbao
+  target:
+    name: flux-system
+    creationPolicy: Merge
+  data:
+    - secretKey: username
+      remoteRef:
+        key: infrastructure/flux-git
+        property: username
+    - secretKey: password
+      remoteRef:
+        key: infrastructure/flux-git
+        property: token
 ---
 apiVersion: v1
 kind: ServiceAccount

kubernetes/infrastructure/post-crds/crowdsec-lapi-serverstransport.yaml

@@ -1,11 +0,0 @@
-# traefik.io CRD consumer for the public LAPI endpoint (crowdsec/lapi-ingress.yaml
-# holds the Ingress). The CA it names is the chart's self-signed LAPI CA.
-apiVersion: traefik.io/v1alpha1
-kind: ServersTransport
-metadata:
-  name: crowdsec-lapi
-  namespace: crowdsec
-spec:
-  serverName: crowdsec-service.crowdsec.svc.cluster.local
-  rootCAs:
-    - secret: crowdsec-root-secret

kubernetes/infrastructure/post-crds/flux-git-externalsecret.yaml

@@ -1,60 +0,0 @@
-# Keeps Flux's flux-system secret supplied with the rotated Gitea credential
-# WITHOUT owning it, so a sync failure leaves the last-good value in place;
-# Flux's source-controller reads it fresh each fetch, no restart. OpenBao
-# policy/role rotator-gitea-flux + the gitea-flux bootstrap (flux account
-# password) are provisioned by hand (see ../openbao/policies/rotator-gitea-flux.hcl).
-#
-# See creationPolicy below for why that is Orphan and not Merge — the
-# distinction is what makes a recovery from the break-glass source possible.
-#
-# Split out of ../openbao/rotation/flux-git.yaml, which still holds the
-# rotation CronJob: external-secrets.io CRDs are installed by the ESO
-# HelmRelease in the infrastructure Kustomization, so this manifest cannot
-# pass server-side dry-run there on a fresh cluster.
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: flux-system-git-auth
-  namespace: flux-system
-spec:
-  refreshInterval: 10m
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: flux-system
-    # Orphan, not Merge: create the Secret if it is absent, but never own it.
-    #
-    # Merge cannot create — it only writes into a Secret that already exists,
-    # on the assumption that `flux bootstrap` made one. That assumption breaks
-    # in exactly the situation this cluster is rebuilt from. The break-glass
-    # bootstrap pulls from the GitHub mirror and keeps its token in
-    # flux-system-breakglass (bootstrap.sh), so `flux-system` is never created,
-    # so this ExternalSecret has nothing to merge into — and reports Ready while
-    # producing no Secret at all. The operator then performs the documented
-    # handover back to Gitea and finds the GitRepository pointing at a Secret
-    # that does not exist.
-    #
-    # Owner would create it but attach an ownerReference, so deleting this
-    # object would take Flux's credential with it. Orphan creates and updates
-    # without ownership, which is what the paragraph at the top of this file
-    # actually wanted: a sync failure, or the loss of this object, leaves the
-    # last-good value in place.
-    #
-    # Safe on a live cluster: the Secret holds exactly the two keys written
-    # below (verified against production), so writing it whole is the same
-    # result as merging into it.
-    creationPolicy: Orphan
-    # Explicit rather than relying on the default, because it is the half that
-    # matters here: if the OpenBao entry ever disappears, keep serving the
-    # credential Flux already has rather than leaving it with none.
-    deletionPolicy: Retain
-  data:
-    - secretKey: username
-      remoteRef:
-        key: infrastructure/flux-git
-        property: username
-    - secretKey: password
-      remoteRef:
-        key: infrastructure/flux-git
-        property: token

kubernetes/infrastructure/post-crds/kustomization.yaml

@@ -10,26 +10,4 @@ resources:
   # configured (2026-07-13): a store that can't turn Ready would block this
   # KS's wait gate and freeze the dependsOn chain.
   - openbao-clustersecretstore.yaml
-  # ExternalSecrets. external-secrets.io CRDs come from the ESO HelmRelease in
-  # the infrastructure Kustomization, so these cannot pass its server-side
-  # dry-run on a fresh cluster — Flux rejects the whole apply set, including
-  # the ESO HelmRelease that would have installed the CRDs. They sat next to
-  # their consumers until a cold-rebuild test hit that deadlock (2026-09-01);
-  # nothing had ever applied this layer against an empty API server before.
-  - crowdsec-bouncer-key-externalsecret.yaml
-  - crowdsec-edge-bouncer-key-externalsecret.yaml
-  - external-dns-unifi-externalsecret.yaml
-  - flux-git-externalsecret.yaml
-  - netbird-ingress-externalsecret.yaml
-  - velero-externalsecret.yaml
-  - velero-hetzner-creds-externalsecret.yaml
-  # traefik.io CRDs, same reasoning — installed by the Traefik HelmRelease in
-  # infrastructure. Traefik is deliberately absent from that Kustomization's
-  # healthChecks, so this KS may dry-run before the chart lands; it retries on
-  # retryInterval rather than deadlocking, because nothing here installs the
-  # CRDs it is waiting for.
-  - traefik-crowdsec-middleware.yaml
-  - traefik-rate-limit-middleware.yaml
-  - traefik-security-headers-middleware.yaml
-  - crowdsec-lapi-serverstransport.yaml
   - ../network-policies/

kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml

@@ -5,7 +5,7 @@ metadata:
   namespace: rook-ceph
 spec:
   cephVersion:
-    image: quay.io/ceph/ceph:v20.2.4 # {"$imagepolicy": "flux-system:ceph"}
+    image: quay.io/ceph/ceph:v19.2.6 # {"$imagepolicy": "flux-system:ceph"}
     allowUnsupported: false
   dataDirHostPath: /var/lib/rook
   # Defaults to false. With three OSDs and size 3 there is no spare copy, so
@@ -78,35 +78,16 @@ spec:
     enabled: true
     ssl: false
   monitoring:
-    # false, and this is not a decision about metrics. `metricsDisabled` is the
-    # switch that turns metrics off; this one only decides whether Rook creates
-    # the ServiceMonitors — and creating them is fatal to the reconcile when it
-    # fails. On an empty cluster monitoring.coreos.com/v1 does not exist, so
-    # Rook's node-daemon reconcile aborted, the CephCluster never left
-    # status.phase Progressing, and infrastructure-ceph-cluster's healthCheckExpr
-    # waits on exactly that — with grafana-db → monitoring → those same CRDs
-    # behind it, closing the loop.
-    #
-    # The ceph-exporter DaemonSet and the mgr metrics endpoint are untouched;
-    # only the scrape wiring moved, to apps/monitoring-rules/
-    # rook-servicemonitors.yaml, where the CRDs exist by the time it applies.
-    # Both objects are reproduced there field for field off the live cluster.
-    enabled: false
+    enabled: true
   # Dedicated Ceph cluster network on VLAN 49, attached to the OSDs through
   # Multus. The attachment is san-network-attachment.yaml.
   #
-  # Only `cluster` is selected. Do not add `public`: under provider: host
-  # that puts every daemon including the mons on host networking. A mon's
-  # address then becomes the node's DHCP lease, and when a lease moves the mon
-  # crash-loops on "Cannot assign requested address" with no node free to
-  # fail over to, because all three already hold the mon host ports.
-  #
-  # addressRanges.cluster is stated, not discovered. Without it Rook runs a
-  # canary pod on an OSD node to read the CIDR off the attachment on EVERY
-  # reconcile; the attachment is host-device, so the OSD on that node already
-  # holds the NIC and the canary can never attach — the reconcile blocks for
-  # 15 minutes, fails, and repeats. The value must equal the whereabouts range
-  # in san-network-attachment.yaml.
+  # Only `cluster` is selected. Do not add `public`, and do not reach for
+  # `addressRanges` — that is honoured only under provider: host, which puts
+  # every daemon including the mons on host networking. A mon's address then
+  # becomes the node's DHCP lease, and when a lease moves the mon crash-loops
+  # on "Cannot assign requested address" with no node free to fail over to,
+  # because all three already hold the mon host ports.
   #
   # With `public` unset, client traffic stays on the pod network and VLAN 49
   # carries Ceph-internal traffic only — which is also why Rook's Multus
@@ -123,9 +104,6 @@ spec:
     provider: multus
     selectors:
       cluster: rook-ceph/ceph-san
-    addressRanges:
-      cluster:
-        - 10.99.0.0/29
   # Nodes listed explicitly, not useAllNodes: under useAllNodes the operator
   # re-provisions an OSD on any node whose device still carries a BlueStore
   # signature, so purging an OSD before pulling its drive is impossible — it
@@ -153,12 +131,10 @@ spec:
     useAllDevices: false
     # Two forms while the production drives are migrated:
     #
-    #   by-partlabel/r-osd-data  the target. Talos names the drive in the
-    #                            machine config (RawVolumeConfig osd-data), so
+    #   by-partlabel/r-ceph-osd  the target. Talos names the drive in the
+    #                            machine config (RawVolumeConfig ceph-osd), so
     #                            this file needs to know nothing about models,
-    #                            serials or hostnames. The label must not
-    #                            contain "ceph": ceph-volume rejects any such
-    #                            partition as a legacy ceph-disk member.
+    #                            serials or hostnames.
     #   by-id/nvme-SAMSUNG…      the three MS-01 drives as they are today:
     #                            bluestore written straight onto the raw disk,
     #                            with no partition table and therefore no
@@ -170,7 +146,7 @@ spec:
     #
     # The second half goes when those three are repartitioned, one node at a
     # time. Until then it is what keeps the running OSDs claimed.
-    devicePathFilter: "^/dev/disk/by-(partlabel/r-osd-data|id/nvme-SAMSUNG_MZQLB960HAJR)"
+    devicePathFilter: "^/dev/disk/by-(partlabel/r-ceph-osd|id/nvme-SAMSUNG_MZQLB960HAJR)"
     config:
       osdsPerDevice: "1"
   # Highest built-in class: kubelet evicts these last and the kernel gives

kubernetes/infrastructure/rook-ceph-cluster/san-network-attachment.yaml

@@ -3,26 +3,33 @@
 # without every Ceph daemon landing on host networking to get there. The
 # alternative and why it was rejected is in cephcluster.yaml.
 #
-# host-device: the SAN NIC is dedicated to Ceph and has exactly one consumer
-# per node — the OSD pod — so it is handed to that pod whole, not virtualised
-# under it. The device keeps its identity (MAC, MTU) and moves back to the
-# host when the pod goes away.
-#
-# `device` names the role, not hardware: every machine renames its own SAN
-# interface to ceph-san at boot (udev rule in its Talos patch). No kernel
-# interface name appears in this file.
-#
 # ISOLATION
 #
-# VLAN 49 carries Ceph-internal traffic and nothing else. No gateway and no
-# routes: there is no router on this segment and there must not be one.
-# whereabouts adds neither unless asked, so nothing here can leak onto or off
-# the SAN.
+# VLAN 49 carries Ceph-internal traffic and nothing else. Two consequences
+# are deliberate:
 #
-# The pool is the whole usable /29, .1 through .6 — headroom for a leaked
-# allocation to sit until the reconciler collects it. One address per node is
-# in use; a second OSD on a node would need a second SAN interface, which is a
-# hardware change, not a config one.
+#   No gateway and no routes. There is no router on this segment and there
+#   must not be one. whereabouts adds neither unless asked, so nothing here
+#   can leak onto or off the SAN. This also means no `sbr` chaining is
+#   needed — unlike the v30iot attachments, there is no DHCP-supplied default
+#   route to confine, because there is no DHCP and no default route.
+#
+#   The pool is the whole usable /29, .1 through .6. It was .4-.6 while the
+#   migration ran, because the nodes still held .1-.3 for as long as Ceph
+#   needed host networking to reach this link — three addresses for three
+#   OSDs, no spare. The nodes have since given those up (the link is only a
+#   macvlan parent now and needs no address of its own), so the reservation
+#   has nothing left to avoid.
+#
+#   Keeping the pool inside the /29 rather than moving it elsewhere on the
+#   segment is not cosmetic. There is no router here, so an address outside
+#   the /29 would be unreachable from anything still using one inside it —
+#   which is exactly what would break a future rolling change, where migrated
+#   and un-migrated OSDs have to keep talking to each other.
+#
+#   Six addresses for three OSDs is the headroom the migration could not
+#   afford: room for a leaked allocation to sit until the reconciler collects
+#   it, and room to add a second OSD per node without touching this file.
 #
 # IPAM
 #
@@ -30,18 +37,29 @@
 # a server that must not exist here, and host-local keeps its state per node
 # with no cluster-wide coordination, so two nodes would hand out the same
 # address on shared L2. See infrastructure/multus/whereabouts-daemonset.yaml.
+#
+# MTU 9000 matches the link. macvlan children inherit the master's MTU, but
+# it is stated so that a change to the parent does not silently halve
+# replication throughput without anything in the tree recording the
+# expectation.
 apiVersion: k8s.cni.cncf.io/v1
 kind: NetworkAttachmentDefinition
 metadata:
   name: ceph-san
   namespace: rook-ceph
 spec:
+  # `master` is an interface name. enp2s0f1np1 is the second X710 SFP+ port,
+  # a busPath-derived predictable name (net.ifnames=1 via the Image Factory
+  # schematic) that is identical on k8s-1/2/3 and cannot shuffle across
+  # reboots.
   config: |
     {
       "cniVersion": "0.3.1",
       "name": "ceph-san",
-      "type": "host-device",
-      "device": "ceph-san",
+      "type": "macvlan",
+      "master": "enp2s0f1np1",
+      "mode": "bridge",
+      "mtu": 9000,
       "ipam": {
         "type": "whereabouts",
         "range": "10.99.0.0/29",

kubernetes/infrastructure/rook-ceph-cluster/toolbox.yaml

@@ -38,7 +38,7 @@ spec:
       serviceAccountName: rook-ceph-default
       containers:
         - name: rook-ceph-tools
-          image: quay.io/ceph/ceph:v20.2.4 # {"$imagepolicy": "flux-system:ceph"}
+          image: quay.io/ceph/ceph:v19.2.6 # {"$imagepolicy": "flux-system:ceph"}
           command:
             - /bin/bash
             - -c

kubernetes/infrastructure/traefik/helmrelease.yaml

@@ -93,17 +93,13 @@ spec:
         # Real client-IP visibility. The netbird-ingress routing peer
         # re-originates every mesh + WAN-via-proxy connection from its own
         # pod IP (rootless userspace WireGuard), so Traefik's TCP peer for
-        # that traffic is always a pod IP — collapsing all external clients
-        # to one source. The NetBird reverse-proxy services are TLS
-        # passthrough and carry the WAN client address in a PROXY protocol
-        # v2 header (options.proxy_protocol, set by netbird-proxy-services);
-        # X-Forwarded-For stays trusted for HTTP-mode services. Both are
-        # accepted from the pod CIDRs only; connections without a header
-        # (direct LAN/mesh) keep their RemoteAddr.
-        proxyProtocol:
-          trustedIPs:
-            - "10.244.0.0/16"
-            - "fd10:244::/56"
+        # that traffic is always a pod IP (e.g. fd10:244::2de1) — collapsing
+        # all external clients to one source. The NetBird reverse-proxy
+        # (HTTP mode, on the VPS) sets X-Forwarded-For with the real WAN
+        # client IP; trust the pod CIDRs so Traefik reads it instead. Without
+        # this, a CrowdSec bouncer bans the shared pod IP and 403s everyone
+        # (the 2026-06-27 IPv6 outage). Harmless no-op when no XFF is present
+        # (direct LAN/mesh) — those keep their RemoteAddr.
         forwardedHeaders:
           trustedIPs:
             - "10.244.0.0/16"

kubernetes/infrastructure/traefik/kustomization.yaml

@@ -2,3 +2,6 @@ apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
   - helmrelease.yaml
+  - crowdsec-middleware.yaml
+  - security-headers-middleware.yaml
+  - rate-limit-middleware.yaml

kubernetes/infrastructure/velero/kustomization.yaml

@@ -1,9 +1,13 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
+  - externalsecret.yaml
+  - hetzner-creds-externalsecret.yaml
   - snapshot-readonly-storageclass.yaml
   - node-agent-config.yaml
   - helmrelease.yaml
+  - servicemonitor.yaml
+  - podmonitor.yaml
   - restore-test-rbac.yaml
   - restore-test-cronjob.yaml
 # Velero Schedule CRs live in kubernetes/infrastructure/post-crds/

kubernetes/talos/patches/hcloud-1.yaml

@@ -1,69 +0,0 @@
-# First machine of the throwaway cluster on Hetzner.
-#
-# enp1s0 is the public NIC; the private networks are enp7s0 and enp8s0, in the
-# order bootstrap.sh attaches them.
-machine:
-  install:
-    disk: /dev/sda
-  certSANs:
-    - hcloud-1
-  nodeLabels:
-    # Holds a Ceph OSD device; the CephCluster selects OSD nodes by this label.
-    storage.renner.uno/ceph-osd: "true"
-  udev:
-    # Renames the SAN NIC to the role name at boot. Matched by PCI bus path
-    # rather than MAC: the address follows from the order bootstrap.sh attaches
-    # the networks, while the MAC is assigned by Hetzner at attach time and is
-    # not known when this file is written.
-    rules:
-      - SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:08:00.0", NAME="ceph-san"
----
-# In the generated schema rather than machine.network.hostname: naming the host
-# in both is rejected at apply time. `auto: off` switches off the generated
-# `auto: stable`, which is mutually exclusive with a static name.
-apiVersion: v1alpha1
-kind: HostnameConfig
-auto: off
-hostname: hcloud-1
----
-# Every interface is stated, the public one included. Talos stops running its
-# default DHCP operators for the whole machine as soon as one new-style network
-# document exists, so an omitted interface gets no address.
-apiVersion: v1alpha1
-kind: LinkConfig
-name: enp1s0
-up: true
----
-apiVersion: v1alpha1
-kind: DHCPv4Config
-name: enp1s0
----
-apiVersion: v1alpha1
-kind: LinkConfig
-name: enp7s0
-up: true
----
-apiVersion: v1alpha1
-kind: DHCPv4Config
-name: enp7s0
----
-# Ceph cluster network, renamed to the role name by the udev rule in the
-# machine section. The node holds no address here; the link exists for the OSD
-# pod. No mtu: a Hetzner network carries 1450, not production's 9000.
-apiVersion: v1alpha1
-kind: LinkConfig
-name: ceph-san
-up: true
----
-apiVersion: v1alpha1
-kind: RawVolumeConfig
-name: osd-data
-provisioning:
-  diskSelector:
-    # Two disks: the system disk and the attached volume. Matching on transport
-    # as the qemu patches do selects nothing here — a Hetzner volume arrives
-    # over virtio-scsi, and a selector matching zero disks is skipped silently.
-    match: '!system_disk'
-  # bootstrap.sh reads this size and creates the volume to match.
-  minSize: 50GiB
-  grow: true

kubernetes/talos/patches/hcloud-2.yaml

@@ -1,50 +0,0 @@
-# Second machine. See hcloud-1.yaml.
-machine:
-  install:
-    disk: /dev/sda
-  certSANs:
-    - hcloud-2
-  nodeLabels:
-    storage.renner.uno/ceph-osd: "true"
-  udev:
-    # See hcloud-1.yaml.
-    rules:
-      - SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:08:00.0", NAME="ceph-san"
----
-apiVersion: v1alpha1
-kind: HostnameConfig
-auto: off
-hostname: hcloud-2
----
-apiVersion: v1alpha1
-kind: LinkConfig
-name: enp1s0
-up: true
----
-apiVersion: v1alpha1
-kind: DHCPv4Config
-name: enp1s0
----
-apiVersion: v1alpha1
-kind: LinkConfig
-name: enp7s0
-up: true
----
-apiVersion: v1alpha1
-kind: DHCPv4Config
-name: enp7s0
----
-# See hcloud-1.yaml.
-apiVersion: v1alpha1
-kind: LinkConfig
-name: ceph-san
-up: true
----
-apiVersion: v1alpha1
-kind: RawVolumeConfig
-name: osd-data
-provisioning:
-  diskSelector:
-    match: '!system_disk'
-  minSize: 50GiB
-  grow: true

kubernetes/talos/patches/hcloud-3.yaml

@@ -1,50 +0,0 @@
-# Third machine. See hcloud-1.yaml.
-machine:
-  install:
-    disk: /dev/sda
-  certSANs:
-    - hcloud-3
-  nodeLabels:
-    storage.renner.uno/ceph-osd: "true"
-  udev:
-    # See hcloud-1.yaml.
-    rules:
-      - SUBSYSTEM=="net", ACTION=="add", KERNELS=="0000:08:00.0", NAME="ceph-san"
----
-apiVersion: v1alpha1
-kind: HostnameConfig
-auto: off
-hostname: hcloud-3
----
-apiVersion: v1alpha1
-kind: LinkConfig
-name: enp1s0
-up: true
----
-apiVersion: v1alpha1
-kind: DHCPv4Config
-name: enp1s0
----
-apiVersion: v1alpha1
-kind: LinkConfig
-name: enp7s0
-up: true
----
-apiVersion: v1alpha1
-kind: DHCPv4Config
-name: enp7s0
----
-# See hcloud-1.yaml.
-apiVersion: v1alpha1
-kind: LinkConfig
-name: ceph-san
-up: true
----
-apiVersion: v1alpha1
-kind: RawVolumeConfig
-name: osd-data
-provisioning:
-  diskSelector:
-    match: '!system_disk'
-  minSize: 50GiB
-  grow: true

kubernetes/talos/patches/k8s-1.yaml

@@ -14,13 +14,6 @@ machine:
     # Disabling v6 on a VLAN parent does not affect the VLAN child.
     net.ipv6.conf.enp2s0f0np0.disable_ipv6: "1"
     net.ipv6.conf.enp87s0.disable_ipv6: "1"
-    # That network also carries the mgmt VLAN's ARP. Answer ARP only on the
-    # link that owns the address, or the UDM learns the node IP against
-    # enp87s0's MAC and delivers IPv4 to a NIC Cilium is not attached to (#793).
-    net.ipv4.conf.all.arp_ignore: "1"
-    net.ipv4.conf.default.arp_ignore: "1"
-    net.ipv4.conf.all.arp_announce: "2"
-    net.ipv4.conf.default.arp_announce: "2"
   install:
     # Select disks by serial only. A broad selector such as
     # `disk.transport == 'nvme'` matches every drive in the machine, and the
@@ -35,13 +28,6 @@ machine:
     # this label instead of listing hostnames, so which boxes carry storage is
     # stated once, here, by the machine that knows.
     storage.renner.uno/ceph-osd: "true"
-  udev:
-    # Renames the SAN port (X710 port 1, matched by its burned-in MAC) to the
-    # role name at boot, before the network stack opens links — a rule applied
-    # to a RUNNING node cannot rename (Resource busy) and takes effect at the
-    # next reboot.
-    rules:
-      - SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="58:47:ca:74:70:28", NAME="ceph-san"
 cluster:
   allowSchedulingOnControlPlanes: true
 ---
@@ -138,15 +124,14 @@ parent: enp2s0f0np0
 up: true
 ---
 # Dedicated Ceph cluster network on X710 port 1, VLAN 49, untagged on the
-# wire. OSD-to-OSD replication and nothing else. The udev rule in the machine
-# section above renames the port to ceph-san at boot, so shared manifests name
-# the role and only this file knows the hardware.
+# wire. OSD-to-OSD replication and nothing else.
 #
-# The node holds NO address here on purpose: the link is handed whole to the
-# OSD pod (host-device attachment). Giving the node an address would put it on
-# a segment reserved for Ceph-internal traffic.
+# The node holds NO address here on purpose. Rook attaches the link to the OSD
+# pods through a Multus macvlan, and a macvlan parent only needs to be up.
+# Giving the node an address would put it on a segment reserved for
+# Ceph-internal traffic.
 apiVersion: v1alpha1
 kind: LinkConfig
-name: ceph-san
+name: enp2s0f1np1
 up: true
 mtu: 9000

kubernetes/talos/patches/k8s-2.yaml

@@ -12,11 +12,6 @@ machine:
     # cluster IPv6 egress.
     net.ipv6.conf.enp2s0f0np0.disable_ipv6: "1"
     net.ipv6.conf.enp87s0.disable_ipv6: "1"
-    # Answer ARP only on the link that owns the address — see k8s-1.yaml (#793).
-    net.ipv4.conf.all.arp_ignore: "1"
-    net.ipv4.conf.default.arp_ignore: "1"
-    net.ipv4.conf.all.arp_announce: "2"
-    net.ipv4.conf.default.arp_announce: "2"
   install:
     # Select the installer target by serial so it cannot land on the OSD drive
     # or on EPHEMERAL, whatever order the kernel enumerates them in.
@@ -28,10 +23,6 @@ machine:
     topology.kubernetes.io/zone: saturn
     # Holds a Ceph OSD drive — see k8s-1.yaml.
     storage.renner.uno/ceph-osd: "true"
-  udev:
-    # Renames the SAN port to the role name at boot — see k8s-1.yaml.
-    rules:
-      - SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="58:47:ca:76:6f:74", NAME="ceph-san"
 cluster:
   allowSchedulingOnControlPlanes: true
 ---
@@ -104,12 +95,11 @@ up: true
 ---
 # Dedicated Ceph cluster network on X710 port 1, VLAN 49, untagged on the
 # wire. OSD-to-OSD replication only, not routed off its own segment.
-# Renamed to the role name by the udev rule above — see k8s-1.yaml.
 #
-# The node holds NO address here on purpose: the link is handed whole to the
-# OSD pod (host-device attachment).
+# The node holds NO address here on purpose. Rook attaches the link to the OSD
+# pods through a Multus macvlan, and a macvlan parent only needs to be up.
 apiVersion: v1alpha1
 kind: LinkConfig
-name: ceph-san
+name: enp2s0f1np1
 up: true
 mtu: 9000

kubernetes/talos/patches/k8s-3.yaml

@@ -8,11 +8,6 @@ machine:
     # k8s-1.yaml.
     net.ipv6.conf.enp2s0f0np0.disable_ipv6: "1"
     net.ipv6.conf.enp87s0.disable_ipv6: "1"
-    # Answer ARP only on the link that owns the address — see k8s-1.yaml (#793).
-    net.ipv4.conf.all.arp_ignore: "1"
-    net.ipv4.conf.default.arp_ignore: "1"
-    net.ipv4.conf.all.arp_announce: "2"
-    net.ipv4.conf.default.arp_announce: "2"
   install:
     # Select the installer target by serial so it cannot land on the OSD drive
     # or on EPHEMERAL, whatever order the kernel enumerates them in.
@@ -24,10 +19,6 @@ machine:
     topology.kubernetes.io/zone: neptune
     # Holds a Ceph OSD drive — see k8s-1.yaml.
     storage.renner.uno/ceph-osd: "true"
-  udev:
-    # Renames the SAN port to the role name at boot — see k8s-1.yaml.
-    rules:
-      - SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="58:47:ca:74:72:d4", NAME="ceph-san"
 cluster:
   allowSchedulingOnControlPlanes: true
 ---
@@ -87,9 +78,9 @@ parent: enp2s0f0np0
 up: true
 ---
 # Dedicated Ceph cluster network — see k8s-1.yaml. The node holds NO address
-# here on purpose; the link is handed whole to the OSD pod (host-device).
+# here on purpose; the link is only a macvlan parent for the OSD pods.
 apiVersion: v1alpha1
 kind: LinkConfig
-name: ceph-san
+name: enp2s0f1np1
 up: true
 mtu: 9000

kubernetes/talos/patches/site-hcloud.yaml

@@ -1,25 +0,0 @@
-# Site patch for the throwaway cluster on Hetzner Cloud.
-#
-# This site exists for one thing the other two cannot test: whether a node that
-# genuinely cannot reach gitea.renner.uno still comes up. At home the registry
-# answers, so containerd never reaches the second endpoint in
-# machine.registries.mirrors and the fallback stays unproven.
-#
-# Hetzner Cloud networks are layer 3. A macvlan child carries its own MAC and
-# frames from it do not cross between servers — measured: same-node ARP
-# resolves, cross-node does not. OSD replication therefore does not run here.
-#
-# What belongs in this file is what the cluster is. Interfaces, disks and names
-# are machine properties and live in the node patches.
-machine:
-  kubelet:
-    nodeIP:
-      # The segment bootstrap.sh provisions for node addresses. Unpinned, the
-      # kubelet can pick the public NIC.
-      validSubnets:
-        - 10.90.0.0/16
-cluster:
-  # Three control planes that also carry workloads, which is the production
-  # shape. The qemu site cannot mirror it: that provisioner gives a second disk
-  # to workers only, so a Ceph OSD forces 1 control plane and 3 workers there.
-  allowSchedulingOnControlPlanes: true

kubernetes/talos/patches/site-homelab.yaml

@@ -11,12 +11,6 @@ machine:
     # kubelet picks the first non-loopback IP, which could land on the v30iot
     # NIC (ens19, 10.2.30.0/24). Control-plane traffic and hostNetwork pod IPs
     # must stay on mgmt LAN.
-    # fd40::/64 is the mgmt LAN ULA, and it is listed so the Node object carries
-    # an IPv6 InternalIP at all. Without one, Cilium falls back to detecting the
-    # node's IPv6 from mgmt0 and picks the SLAAC address from the dynamic
-    # provider prefix, which it then probes until the agent restarts (#789).
-    # The GUA is not a control-plane address: BGP peers on fd40::/64 too.
     nodeIP:
       validSubnets:
         - 10.2.40.0/24
-        - fd40::/64

kubernetes/talos/patches/vm-1.yaml

@@ -43,15 +43,3 @@ apiVersion: v1alpha1
 kind: HostnameConfig
 auto: off
 hostname: vm-1
----
-# Ceph cluster network — see vm-2.yaml for why this is a VLAN, and why the
-# DHCPv4Config below is load-bearing.
-apiVersion: v1alpha1
-kind: VLANConfig
-name: ceph-san
-parent: enp0s2
-vlanID: 30
----
-apiVersion: v1alpha1
-kind: DHCPv4Config
-name: enp0s2

kubernetes/talos/patches/vm-2.yaml

@@ -21,7 +21,7 @@ auto: off
 hostname: vm-2
 ---
 # The OSD device, named by the machine that has it. Ceph then consumes
-# /dev/disk/by-partlabel/r-osd-data and needs to know nothing about models,
+# /dev/disk/by-partlabel/r-ceph-osd and needs to know nothing about models,
 # serials or hostnames — the same CephCluster works here and on the MS-01s.
 #
 # The selector is a CEL expression, so each machine says whatever identifies its
@@ -29,33 +29,9 @@ hostname: vm-2
 # real nodes it will name the Samsung.
 apiVersion: v1alpha1
 kind: RawVolumeConfig
-name: osd-data
+name: ceph-osd
 provisioning:
   diskSelector:
     match: disk.transport == "virtio" && !system_disk
   minSize: 5GiB
   grow: true
----
-# The Ceph cluster network link. Production gives it a second physical port;
-# this machine has one NIC, so the segment is a VLAN on it.
-#
-# A dummy link would be simpler and worthless — no traffic between nodes, so
-# Ceph would replicate over an interface going nowhere and the test would pass
-# having proven nothing. A VLAN rides the provisioner's bridge that every VM is
-# on, so the SAN carries real frames on its own tag.
-#
-# No MTU, unlike production's 9000: the qemu NIC is created with host_mtu=1500.
-apiVersion: v1alpha1
-kind: VLANConfig
-name: ceph-san
-parent: enp0s2
-vlanID: 30
----
-# Load-bearing. Declaring any network document makes Talos configure this link
-# explicitly, and the implicit DHCP that gives the node its address stops. A
-# first attempt with only the VLANConfig took vm-4 off the network — NotReady,
-# unreachable over the Talos API. Verified on a second throwaway node that this
-# document keeps the parent addressed.
-apiVersion: v1alpha1
-kind: DHCPv4Config
-name: enp0s2

kubernetes/talos/patches/vm-3.yaml

@@ -15,21 +15,9 @@ hostname: vm-3
 ---
 apiVersion: v1alpha1
 kind: RawVolumeConfig
-name: osd-data
+name: ceph-osd
 provisioning:
   diskSelector:
     match: disk.transport == "virtio" && !system_disk
   minSize: 5GiB
   grow: true
----
-# Ceph cluster network — see vm-2.yaml for why this is a VLAN, and why the
-# DHCPv4Config below is load-bearing.
-apiVersion: v1alpha1
-kind: VLANConfig
-name: ceph-san
-parent: enp0s2
-vlanID: 30
----
-apiVersion: v1alpha1
-kind: DHCPv4Config
-name: enp0s2

kubernetes/talos/patches/vm-4.yaml

@@ -19,21 +19,9 @@ hostname: vm-4
 ---
 apiVersion: v1alpha1
 kind: RawVolumeConfig
-name: osd-data
+name: ceph-osd
 provisioning:
   diskSelector:
     match: disk.transport == "virtio" && !system_disk
   minSize: 5GiB
   grow: true
----
-# Ceph cluster network — see vm-2.yaml for why this is a VLAN, and why the
-# DHCPv4Config below is load-bearing.
-apiVersion: v1alpha1
-kind: VLANConfig
-name: ceph-san
-parent: enp0s2
-vlanID: 30
----
-apiVersion: v1alpha1
-kind: DHCPv4Config
-name: enp0s2

kubernetes/vmtest-job.yaml

@@ -1,134 +0,0 @@
-# Runs `bootstrap.sh vmtest` inside the cluster instead of on a workstation.
-#
-# NOT reconciled by Flux — deliberately outside kubernetes/clusters. This is a
-# one-shot test, and a Kustomization would recreate it after every completion.
-# It is applied by the vmtest pipeline, or by hand for a single run.
-#
-# Why in-cluster at all: the test wants ~30G for four VMs (one control plane at
-# 6G, three OSD workers at 8G). The workstation has 31G with a desktop on it —
-# a single run left under 1G free and two overlapping runs OOM-killed the
-# session. The nodes have ~94G each with about 70G unused.
-#
-# The whole script runs here, flux and OpenBao included, so that what a rebuild
-# rehearses is what a real recovery would execute. The unseal key is not baked
-# in and not stored as a Secret: the operator streams all five secrets into an
-# in-memory emptyDir at the moment the run asks for them, and they die with the
-# Job without ever reaching etcd or a backup of it.
-apiVersion: v1
-kind: Namespace
-metadata:
-  name: vmtest
-  labels:
-    # The pod is privileged; say so rather than have the admission policy
-    # reject it for reasons that look unrelated.
-    pod-security.kubernetes.io/enforce: privileged
----
-apiVersion: batch/v1
-kind: Job
-metadata:
-  name: vmtest
-  namespace: vmtest
-spec:
-  backoffLimit: 0        # a failed rebuild test is a result, not something to retry
-  ttlSecondsAfterFinished: 86400
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/name: vmtest
-    spec:
-      restartPolicy: Never
-      # Nothing here should be able to reach the cluster it runs in. The phase
-      # points KUBECONFIG at its own throwaway cluster, and without a mounted
-      # token a stray kubectl cannot fall back to this one.
-      automountServiceAccountToken: false
-      containers:
-        - name: vmtest
-          # Pinned by digest, not :latest. Spegel mirrors mutable tags and keeps
-          # serving what it cached: a run on k8s-3 pulled a ci-vmtest:latest
-          # three builds old, which predated OVMF, and died with "the required
-          # flash image was not found" before the first VM booted. The tag is
-          # carried alongside for readability — containerd resolves the digest.
-          image: gitea.renner.uno/gitea_admin/ci-vmtest:2026-09-03@sha256:a8145a76896ef9aa6fdb58dc96b9bdc22b53f4e55f04d9e6de289f35b75dbd81
-          # Privileged for /dev/kvm and for the bridges and taps the provisioner
-          # builds. It needs no hostNetwork: the provisioner's network lives in
-          # this pod's own namespace, and the script that talks to 10.5.0.x runs
-          # in the same pod.
-          securityContext:
-            privileged: true
-          # A hard ceiling, because the failure mode is known: an unbounded run
-          # is what OOM-killed the workstation. 40G is the VMs (16G control
-          # plane + 3x8G workers), the rest is qemu overhead and the tooling.
-          # The workers carry an OSD, a mon and a mgr each; at 4G they ran out
-          # of room for the CNI DaemonSets themselves. The control plane went
-          # 6G → 8G → 16G: at both smaller sizes it ran at 99% of allocatable
-          # memory and the kubelet evicted kube-apiserver to admit a
-          # node-critical pod — bootstrap.sh carries the full account next to
-          # the sizing flags. The hosts have ~95GiB each, so this fits.
-          resources:
-            requests:
-              # Less than the 14 guest vCPUs the four VMs declare, on purpose:
-              # the guests are never all runnable at once (one cluster booting
-              # in sequence), and there is no CPU limit, so the pod may burst
-              # past the request when the node has headroom. The request only
-              # keeps the scheduler from stacking this next to another big
-              # burst consumer.
-              cpu: "6"
-              memory: 42Gi
-            limits:
-              memory: 48Gi
-          env:
-            # talosctl caches the image and the CNI bundle under $HOME; point it
-            # at the scratch volume rather than the container layer.
-            - name: HOME
-              value: /work
-            # The VM disks. Short on purpose — the provisioner puts a qemu
-            # monitor socket under it and sun_path caps the path at 108 bytes.
-            - name: VM_STATE
-              value: /work/vm
-            # Which branch to test. master unless a run is checking a change.
-            - name: BRANCH
-              value: master
-            # Secrets are streamed in here at the moment the run needs them,
-            # never baked into the image and never stored as a Secret. They live
-            # in this emptyDir for the length of the Job and go with it — so
-            # they do not reach etcd, or any backup of it.
-            - name: OPENBAO_SECRETS_DIR
-              value: /secrets
-          command: ["/bin/bash", "-ec"]
-          args:
-            - |
-              # From the GitHub mirror, not in-cluster Gitea. A rebuild test
-              # that clones from the cluster it is rehearsing the loss of
-              # cannot start when it matters — and a run already died here on
-              # resolving gitea.renner.uno.
-              git clone --depth 1 --branch "$BRANCH" \
-                https://github.com/konairius/Homelab.git /work/homelab
-              exec /work/homelab/kubernetes/bootstrap.sh vmtest
-          volumeMounts:
-            - name: work
-              mountPath: /work
-            - name: secrets
-              mountPath: /secrets
-            - name: dev-kvm
-              mountPath: /dev/kvm
-      volumes:
-        # The VM disks and the provisioner state. emptyDir on the node's disk
-        # rather than a PVC: it is scratch that dies with the Job, and putting it
-        # on Ceph would have the test's own writes compete with the storage it is
-        # meant to leave alone.
-        - name: work
-          emptyDir:
-            # The VM images are sparse, so this bounds what is actually written,
-            # not the 232GiB the geometry declares. 60Gi was already half spent
-            # by four 10GiB systems disks; the node carries this comfortably with
-            # ~810GB free on the filesystem behind it.
-            sizeLimit: 150Gi
-        # In memory, so the material never touches a node's disk.
-        - name: secrets
-          emptyDir:
-            medium: Memory
-            sizeLimit: 1Mi
-        - name: dev-kvm
-          hostPath:
-            path: /dev/kvm
-            type: CharDevice

scripts/embed-eval.sh

@@ -1,177 +0,0 @@
-#!/usr/bin/env bash
-# Retrieval evaluation for an embedding model, on German documents of the kind
-# this homelab actually files.
-#
-# Companion to scripts/ai-eval.sh, but it cannot work the same way. A chat
-# answer can be read and judged; an embedding is 1024 opaque numbers. What is
-# measurable is whether the right document comes back first, so this builds a
-# small labelled corpus, embeds it, and scores real questions against it.
-#
-# Metrics:
-#   Recall@1  the correct document ranked first — what Paperless's RAG actually
-#             feeds to the classifier
-#   Recall@3  it at least made the shortlist
-#   MRR       mean reciprocal rank, so near-misses are visible rather than
-#             collapsing to pass/fail
-#   margin    cosine gap between the correct document and the best wrong one.
-#             A model that is right by 0.002 is guessing; one that is right by
-#             0.15 has actually separated them.
-#
-# The corpus deliberately contains confusable pairs — two Allianz policies, four
-# documents that are all "Rechnungen", a car insurance and a vehicle
-# registration sharing a number plate. An easy corpus would rate everything 100%
-# and tell you nothing.
-#
-# Usage:
-#   scripts/embed-eval.sh                                    # cluster default
-#   scripts/embed-eval.sh -m some-model
-#   scripts/embed-eval.sh -e https://host/v1/embeddings -m model -k KEY
-#
-# Env equivalents: EMBED_EVAL_ENDPOINT, EMBED_EVAL_MODEL, EMBED_EVAL_KEY.
-#
-# To compare two models on our llama-embed, -m is not enough: swap --hf-repo in
-# kubernetes/apps/llama-server/embed.yaml, let it roll out, then re-run. Note
-# that changing the model also invalidates any vectors Paperless has stored, so
-# comparisons are best made before an index exists.
-#
-# Exit code is the number of queries whose correct document did not rank first.
-set -uo pipefail
-
-ENDPOINT="${EMBED_EVAL_ENDPOINT:-http://llama-embed.ai.svc.cluster.local:8080/v1/embeddings}"
-MODEL="${EMBED_EVAL_MODEL:-qwen3-embedding-0.6b}"
-APIKEY="${EMBED_EVAL_KEY:-}"
-# Run curl from inside this pod when the endpoint is a cluster-internal name.
-# llama-embed has no ingress by design, so there is nothing to reach from a
-# workstation.
-VIA_POD="${EMBED_EVAL_VIA_POD:-}"
-
-while [ $# -gt 0 ]; do
-    case "$1" in
-        -e|--endpoint) ENDPOINT="$2"; shift 2 ;;
-        -m|--model)    MODEL="$2";    shift 2 ;;
-        -k|--key)      APIKEY="$2";   shift 2 ;;
-        -p|--via-pod)  VIA_POD="$2";  shift 2 ;;
-        -h|--help)     sed -n '2,34p' "$0"; exit 0 ;;
-        *) echo "unknown argument: $1 (try --help)" >&2; exit 2 ;;
-    esac
-done
-
-HERE="$(cd "$(dirname "$0")" && pwd)"
-CASES="$HERE/embed-eval"
-WORK="$(mktemp -d)"
-trap 'rm -rf "$WORK"' EXIT
-
-for f in "$CASES/corpus.jsonl" "$CASES/queries.jsonl"; do
-    [ -r "$f" ] || { echo "missing fixture: $f" >&2; exit 2; }
-done
-
-# Default to reaching the Service from a pod that is allowed to, unless the
-# caller pointed us somewhere externally reachable.
-if [ -z "$VIA_POD" ] && [ "${ENDPOINT#*svc.cluster.local}" != "$ENDPOINT" ]; then
-    VIA_POD="deploy/llama-server -n ai"
-fi
-
-# embed <text> -> the vector as a JSON array on stdout
-embed() {
-    local payload out
-    payload=$(jq -nc --arg m "$MODEL" --arg t "$1" '{model:$m,input:$t}')
-    if [ -n "$VIA_POD" ]; then
-        # stderr is kept and surfaced on failure rather than discarded: a
-        # silently swallowed kubectl or curl error is indistinguishable from a
-        # model that returned nothing.
-        # shellcheck disable=SC2086
-        out=$(kubectl exec $VIA_POD -- curl -sS -m 120 \
-              -H 'Content-Type: application/json' \
-              ${APIKEY:+-H "Authorization: Bearer $APIKEY"} \
-              -d "$payload" "$ENDPOINT" 2>"$WORK/err.txt")
-    else
-        out=$(curl -sS -m 120 -H 'Content-Type: application/json' \
-              ${APIKEY:+-H "Authorization: Bearer $APIKEY"} \
-              -d "$payload" "$ENDPOINT")
-    fi
-    printf '%s' "$out" | jq -c '.data[0].embedding // empty'
-}
-
-echo "endpoint: $ENDPOINT"
-echo "model:    $MODEL"
-[ -n "$VIA_POD" ] && echo "via pod:  $VIA_POD"
-echo ""
-
-# ---- embed the corpus ------------------------------------------------------
-DOCS=$(wc -l < "$CASES/corpus.jsonl" | tr -d ' ')
-echo "embedding $DOCS documents..."
-: > "$WORK/vecs.jsonl"
-start=$(date +%s)
-while IFS= read -r line; do
-    id=$(printf '%s' "$line" | jq -r '.id')
-    text=$(printf '%s' "$line" | jq -r '.text')
-    vec=$(embed "$text")
-    if [ -z "$vec" ]; then
-        echo "FAILED to embed '$id' — endpoint returned no vector." >&2
-        [ -s "$WORK/err.txt" ] && { echo "stderr:" >&2; cat "$WORK/err.txt" >&2; }
-        exit 2
-    fi
-    jq -nc --arg id "$id" --argjson v "$vec" '{id:$id, vec:$v}' >> "$WORK/vecs.jsonl"
-done < "$CASES/corpus.jsonl"
-elapsed=$(( $(date +%s) - start ))
-DIM=$(head -1 "$WORK/vecs.jsonl" | jq '.vec | length')
-echo "  $DOCS documents, ${DIM} dimensions, ${elapsed}s"
-echo ""
-
-# ---- score the queries -----------------------------------------------------
-# Cosine similarity in jq. The vectors llama.cpp returns are normalised, but the
-# full form costs nothing and keeps this correct against servers that do not.
-rank() { # queryvec -> "id score" lines, best first
-    jq -s -r --argjson q "$1" '
-        def dot($a; $b): reduce range(0; $a|length) as $i (0; . + ($a[$i] * $b[$i]));
-        def norm($a): (reduce ($a[]) as $x (0; . + ($x * $x))) | sqrt;
-        map({id: .id, s: (dot(.vec; $q) / ((norm(.vec) * norm($q)) + 1e-12))})
-        | sort_by(-.s)[]
-        | "\(.id) \(.s)"
-    ' "$WORK/vecs.jsonl"
-}
-
-MISS=0; TOTAL=0; HIT1=0; HIT3=0; MRRSUM=0
-printf '%-46s %-22s %5s %8s\n' QUERY EXPECTED RANK MARGIN
-printf '%s\n' "------------------------------------------------------------------------------------"
-while IFS= read -r line; do
-    q=$(printf '%s' "$line" | jq -r '.q')
-    want=$(printf '%s' "$line" | jq -r '.want')
-    qvec=$(embed "$q")
-    if [ -z "$qvec" ]; then
-        echo "FAILED to embed query: $q" >&2
-        [ -s "$WORK/err.txt" ] && { echo "stderr:" >&2; cat "$WORK/err.txt" >&2; }
-        exit 2
-    fi
-    rank "$qvec" > "$WORK/ranked.txt"
-
-    pos=$(grep -n "^$want " "$WORK/ranked.txt" | cut -d: -f1)
-    pos=${pos:-999}
-    TOTAL=$((TOTAL + 1))
-    [ "$pos" -eq 1 ] && HIT1=$((HIT1 + 1)) || MISS=$((MISS + 1))
-    [ "$pos" -le 3 ] && HIT3=$((HIT3 + 1))
-    MRRSUM=$(awk -v s="$MRRSUM" -v p="$pos" 'BEGIN{printf "%.6f", s + 1/p}')
-
-    # Margin: the correct document's score minus the best incorrect one. On a
-    # miss this is negative, which says how badly it lost.
-    want_s=$(awk -v w="$want" '$1==w{print $2; exit}' "$WORK/ranked.txt")
-    best_other=$(awk -v w="$want" '$1!=w{print $2; exit}' "$WORK/ranked.txt")
-    margin=$(awk -v a="${want_s:-0}" -v b="${best_other:-0}" 'BEGIN{printf "%+.4f", a-b}')
-
-    if [ "$pos" -eq 1 ]; then mark=" "; else mark="!"; fi
-    printf '%s%-45.45s %-22s %5s %8s\n' "$mark" "$q" "$want" "$pos" "$margin"
-    if [ "$pos" -ne 1 ]; then
-        printf '   got instead: %s\n' "$(head -1 "$WORK/ranked.txt" | cut -d' ' -f1)"
-    fi
-done < "$CASES/queries.jsonl"
-
-R1=$(awk -v h="$HIT1" -v t="$TOTAL" 'BEGIN{printf "%.0f", 100*h/t}')
-R3=$(awk -v h="$HIT3" -v t="$TOTAL" 'BEGIN{printf "%.0f", 100*h/t}')
-MRR=$(awk -v s="$MRRSUM" -v t="$TOTAL" 'BEGIN{printf "%.3f", s/t}')
-
-echo ""
-echo "========================================"
-printf '  RESULT model=%s dim=%s recall@1=%s%% recall@3=%s%% mrr=%s\n' \
-    "$MODEL" "$DIM" "$R1" "$R3" "$MRR"
-echo "========================================"
-exit "$MISS"

scripts/embed-eval/corpus.jsonl

@@ -1,12 +0,0 @@
-{"id":"stadtwerke-strom","text":"Stadtwerke Muenchen GmbH. Jahresrechnung Strom. Rechnungsnummer 4711. Abrechnungszeitraum 01.01.2025 bis 31.12.2025. Zaehlernummer 1EMH0012345678. Verbrauch 2450 kWh zum Arbeitspreis von 31,2 Cent je kWh. Grundpreis 9,90 EUR monatlich. Zu zahlender Betrag 812,40 EUR faellig am 15.02.2026."}
-{"id":"telekom-mobilfunk","text":"Telekom Deutschland GmbH. Ihre Mobilfunkrechnung fuer Januar. Kundennummer 0891234567. Rufnummer 0170 1234567. Tarif MagentaMobil M mit 20 GB Datenvolumen. Grundpreis 39,95 EUR. Verbindungen ins Ausland 4,20 EUR. Rechnungsbetrag 44,15 EUR wird per Lastschrift eingezogen."}
-{"id":"finanzamt-steuer","text":"Finanzamt Muenchen Abteilung II. Bescheid ueber Einkommensteuer und Solidaritaetszuschlag fuer 2025. Steuernummer 143/256/78901. Zu versteuerndes Einkommen 54.320 EUR. Festgesetzte Einkommensteuer 8.412 EUR. Bereits geleistete Vorauszahlungen 9.100 EUR. Erstattungsbetrag 688 EUR. Rechtsbehelfsbelehrung siehe Seite 3."}
-{"id":"allianz-hausrat","text":"Allianz Versicherungs-AG. Versicherungsschein Hausratversicherung. Policennummer HR-88231. Versicherte Wohnflaeche 96 Quadratmeter. Versicherungssumme 65.000 EUR. Eingeschlossen sind Feuer, Leitungswasser, Sturm und Einbruchdiebstahl. Fahrraddiebstahl bis 1.500 EUR mitversichert. Jahresbeitrag 214 EUR."}
-{"id":"allianz-kfz","text":"Allianz Versicherungs-AG. Beitragsrechnung Kraftfahrtversicherung. Policennummer KFZ-55190. Amtliches Kennzeichen M-AB 1234. Fahrzeug VW Golf VIII. Vollkasko mit 500 EUR Selbstbeteiligung, Teilkasko 150 EUR. Schadenfreiheitsklasse SF 12. Jahresbeitrag 638 EUR."}
-{"id":"mietvertrag","text":"Mietvertrag ueber Wohnraum. Vermieter Immobilien Wagner GbR, Mieter Konstantin Renner. Mietobjekt Dreizimmerwohnung im zweiten Obergeschoss, 78 Quadratmeter, Lindenstrasse 14. Mietbeginn 01.04.2024. Kaltmiete 1.180 EUR monatlich, Betriebskostenvorauszahlung 240 EUR. Kaution drei Nettokaltmieten."}
-{"id":"arbeitsvertrag","text":"Arbeitsvertrag zwischen der Muster Technik GmbH und dem Arbeitnehmer. Taetigkeit als Softwareentwickler ab 01.09.2024. Bruttojahresgehalt 78.000 EUR zahlbar in zwoelf Monatsraten. Woechentliche Arbeitszeit 40 Stunden. Urlaubsanspruch 30 Arbeitstage im Kalenderjahr. Probezeit sechs Monate."}
-{"id":"aok-krankenkasse","text":"AOK Bayern. Beitragsbescheid zur gesetzlichen Kranken- und Pflegeversicherung. Versichertennummer A123456789. Beitragspflichtige Einnahmen 4.500 EUR monatlich. Allgemeiner Beitragssatz 14,6 Prozent zuzueglich Zusatzbeitrag 1,7 Prozent. Pflegeversicherung 3,4 Prozent. Monatlicher Gesamtbeitrag 887,50 EUR."}
-{"id":"sparkasse-kontoauszug","text":"Stadtsparkasse Muenchen. Kontoauszug Nummer 3 zum Girokonto IBAN DE89 7015 0000 0012 3456 78. Alter Kontostand 2.340,12 EUR. Gutschrift Gehalt 4.210,00 EUR. Lastschrift Miete 1.420,00 EUR. Dauerauftrag Sparplan 300,00 EUR. Neuer Kontostand 4.830,12 EUR."}
-{"id":"amazon-bestellung","text":"Amazon EU S.a.r.l. Bestellbestaetigung. Bestellnummer 302-4455667-8899001. Artikel Logitech MX Master 3S Wireless Maus, Menge 1, Preis 89,99 EUR. Versand durch Amazon, Lieferung voraussichtlich am 04.09. Zahlungsart Kreditkarte. Rechnungsbetrag inklusive Mehrwertsteuer 89,99 EUR."}
-{"id":"handwerker-rechnung","text":"Sanitaer Bergmeister GmbH. Rechnung fuer erbrachte Leistungen. Rechnungsnummer 2026-0417. Austausch eines defekten Heizungsthermostats sowie Entlueftung von drei Heizkoerpern. Arbeitszeit 2,5 Stunden zu 68 EUR. Material 42,30 EUR. Nettobetrag 212,30 EUR, Umsatzsteuer 19 Prozent, Gesamtbetrag 252,64 EUR."}
-{"id":"kfz-zulassung","text":"Landeshauptstadt Muenchen Kreisverwaltungsreferat. Zulassungsbescheinigung Teil I. Amtliches Kennzeichen M-AB 1234. Fahrzeugklasse Personenkraftwagen. Hersteller Volkswagen, Handelsbezeichnung Golf. Erstzulassung 14.03.2022. Hubraum 1498 Kubikzentimeter, Leistung 110 kW. Naechste Hauptuntersuchung 03.2028."}

scripts/embed-eval/queries.jsonl

@@ -1,12 +0,0 @@
-{"q":"Wie hoch war mein Stromverbrauch im letzten Jahr?","want":"stadtwerke-strom","note":"paraphrase: asks about consumption, the document says Verbrauch kWh"}
-{"q":"Was kostet mein Handyvertrag monatlich?","want":"telekom-mobilfunk","note":"paraphrase: Handyvertrag vs Mobilfunkrechnung/Tarif"}
-{"q":"Bescheid vom Finanzamt ueber die Einkommensteuer","want":"finanzamt-steuer","note":"near-literal, should be easy"}
-{"q":"Welchen Versicherungsschutz habe ich fuer mein Auto?","want":"allianz-kfz","note":"HARD: two Allianz policies. Must not pick the Hausrat one."}
-{"q":"Ist mein Fahrrad gegen Diebstahl versichert?","want":"allianz-hausrat","note":"HARD: same sender as kfz, and bicycles are a detail deep in the text"}
-{"q":"Wie hoch ist die Kaltmiete meiner Wohnung?","want":"mietvertrag","note":"must beat sparkasse, which also mentions Miete as a Lastschrift"}
-{"q":"Wie viele Urlaubstage stehen mir zu?","want":"arbeitsvertrag","note":"paraphrase: Urlaubsanspruch"}
-{"q":"Was zahle ich monatlich an die Krankenkasse?","want":"aok-krankenkasse","note":"Krankenkasse vs Kranken- und Pflegeversicherung"}
-{"q":"Wie viel Geld ist auf meinem Girokonto?","want":"sparkasse-kontoauszug","note":"Kontostand"}
-{"q":"Rechnung fuer die Reparatur der Heizung","want":"handwerker-rechnung","note":"HARD: several documents are Rechnungen; must pick the Sanitaer one"}
-{"q":"Wann muss mein Auto zum TUEV?","want":"kfz-zulassung","note":"HARD: TUEV never appears; the document says Hauptuntersuchung. Pure semantics, and must not be pulled to the Kfz-Versicherung by the shared plate number."}
-{"q":"Wann wurde meine Maus geliefert?","want":"amazon-bestellung","note":"colloquial vs Bestellbestaetigung/Lieferung"}

scripts/import-cluster-access.sh

@@ -13,18 +13,7 @@ set -euo pipefail
 ENDPOINT="https://fsn1.your-objectstorage.com"
 BUCKET="renner-uno-k8s-secrets"
 PREFIX="cluster-access"
-# Overridable, which the unconditional `export AWS_PROFILE="secrets"` was not.
-# The default is unchanged and deliberately so: "secrets" resolves to working
-# credentials and this script has always used it. Both "secrets" and "hetzner"
-# exist in ~/.aws/credentials on the workstation.
-#
-# The defect is only the assignment. Forcing the value defeated the env-based
-# path this file's own header offers, and it also overrode a caller who had
-# already selected a profile — which is how a 403 from HeadObject can appear
-# while the credentials the caller intended were perfectly good. That error
-# reads as a bucket permissions problem and sends you looking in the wrong
-# place.
-export AWS_PROFILE="${AWS_PROFILE:-secrets}"
+export AWS_PROFILE="secrets"
 
 
 command -v aws >/dev/null  || { echo "aws-cli not found"; exit 1; }

scripts/paperless-eval.sh

@@ -0,0 +1,207 @@
+#!/usr/bin/env bash
+# End-to-end evaluation of Paperless document suggestions against the real
+# archive, comparing the two engines Paperless ships:
+#
+#   /api/documents/<id>/suggestions/      the built-in scikit-learn classifier,
+#                                         trained on the existing corpus
+#   /api/documents/<id>/ai_suggestions/   the LLM path, optionally with RAG
+#
+# WHY THIS EXISTS
+#
+# scripts/embed-eval-real.sh measures whether a document's nearest neighbours
+# carry its labels. That is a proxy: Paperless does not act on the neighbours
+# directly, it hands them to an LLM as context, alongside the document text and
+# a candidate list of existing tags and correspondents. The LLM can override
+# poor context, or fail with good context. So a better embedding score does not
+# automatically mean better suggestions, and nothing so far has verified that it
+# does.
+#
+# This measures the thing that is actually wanted: does Paperless suggest the
+# labels a human already chose? Everything else — MTEB scores, retrieval
+# accuracy — is only interesting insofar as it predicts this number.
+#
+# It also answers a question that predates the LLM work: with ten years of
+# hand-curated labels, the built-in classifier has excellent training data. It
+# cannot invent a new tag or a title, which is the LLM's advantage — but for
+# assigning labels that already exist it may simply be better. Running both on
+# the same documents settles that rather than assuming it.
+#
+# Ground truth is the current filing. Where the two disagree with it, that is
+# not automatically an error: over ten years filing conventions change, and an
+# old document may carry an older convention. Treat the absolute numbers as
+# indicative and the comparison between engines as the result.
+#
+# Requires a Paperless API token in PAPERLESS_TOKEN. The API is only reachable
+# from inside the cluster (the namespace policy admits the Authentik gate
+# outpost alone), so requests are issued from within the paperless pod.
+#
+# Usage:
+#   PAPERLESS_TOKEN=... scripts/paperless-eval.sh [-n 100] [--engine both|classic|ai]
+set -uo pipefail
+
+SAMPLE="${PAPERLESS_EVAL_SAMPLE:-100}"
+ENGINE=both
+PG_POD="${PAPERLESS_EVAL_PG_POD:-pprl-pg-2}"
+APP_POD="${PAPERLESS_EVAL_APP_POD:-deploy/paperless}"
+
+while [ $# -gt 0 ]; do
+    case "$1" in
+        -n|--sample) SAMPLE="$2"; shift 2 ;;
+        --engine)    ENGINE="$2"; shift 2 ;;
+        -h|--help)   sed -n '2,40p' "$0"; exit 0 ;;
+        *) echo "unknown argument: $1 (try --help)" >&2; exit 2 ;;
+    esac
+done
+
+if [ -z "${PAPERLESS_TOKEN:-}" ]; then
+    echo "PAPERLESS_TOKEN is not set." >&2
+    echo "Create one for an existing user, then remove it when finished." >&2
+    exit 2
+fi
+
+WORK="$(mktemp -d)"
+trap 'rm -rf "$WORK"' EXIT
+
+# Deterministic sample, so repeated runs and different engines see the same
+# documents. Only documents that carry both labels are useful as ground truth.
+echo "reading $SAMPLE labelled documents from postgres ($PG_POD)..."
+# Names come along with the ids because the AI engine reports an unresolved
+# guess as a name string rather than an id; see the lenient scoring below.
+# Tab-separated rather than '|', which occurs in correspondent names.
+kubectl -n paperless exec "$PG_POD" -- psql -U postgres -d paperless -tA -F$'\t' -c \
+  "select d.id, d.correspondent_id, d.document_type_id, c.name, t.name
+   from documents_document d
+   join documents_correspondent c on c.id = d.correspondent_id
+   join documents_documenttype  t on t.id = d.document_type_id
+   where d.correspondent_id is not null
+     and d.document_type_id is not null
+     and length(d.content) > 400
+   order by md5(d.id::text)
+   limit $SAMPLE;" > "$WORK/docs.txt" 2>"$WORK/pg-err.txt"
+if ! [ -s "$WORK/docs.txt" ]; then
+    echo "no documents returned from postgres." >&2
+    [ -s "$WORK/pg-err.txt" ] && cat "$WORK/pg-err.txt" >&2
+    exit 2
+fi
+N=$(grep -c . "$WORK/docs.txt")
+
+# The floor: always guessing the most frequent label. Without it a number like
+# "70% correct" cannot be read, because a skewed archive gives that away for
+# free — one correspondent holds several hundred of ~3900 documents.
+kubectl -n paperless exec "$PG_POD" -- psql -U postgres -d paperless -tAc \
+  "select max(c) from (select count(*) c from documents_document
+   where correspondent_id is not null group by correspondent_id) x;" \
+  > "$WORK/corr_max.txt" 2>"$WORK/pg-err.txt"
+kubectl -n paperless exec "$PG_POD" -- psql -U postgres -d paperless -tAc \
+  "select count(*) from documents_document where correspondent_id is not null;" \
+  > "$WORK/corr_total.txt" 2>"$WORK/pg-err.txt"
+
+api() { # path -> body on stdout
+    kubectl -n paperless exec "$APP_POD" -c paperless -- \
+        curl -sS -m 300 -H "Authorization: Token $PAPERLESS_TOKEN" \
+        "http://127.0.0.1:8000$1" 2>"$WORK/err.txt"
+}
+
+# A hit under two definitions, because the AI engine can be right in a way the
+# API response does not record as a suggestion:
+#
+#   strict   the true label's id appears in `.correspondents` /
+#            `.document_types`. This is what the UI actually offers, so it is
+#            the number that describes the feature as experienced.
+#
+#   lenient  strict, OR the true label's name is contained wordwise in one of
+#            `.suggested_correspondents` / `.suggested_document_types`.
+#
+# The gap between them is one specific failure: the LLM names the right entity
+# but paperless_ai.matching cannot resolve the name to the existing object,
+# because difflib.get_close_matches runs at MATCH_THRESHOLD = 0.8 and a correct
+# but more verbose answer falls below it — observed with "Generali Deutschland
+# Krankenversicherung AG" against the existing "Generali AG", which yielded an
+# empty `correspondents` list.
+#
+# Reporting only the strict number would blame the model for a matcher problem;
+# reporting only the lenient one would claim a usefulness the UI does not
+# deliver. The classic engine emits no `suggested_*` fields at all, so both
+# numbers coincide for it and the comparison stays fair in either column.
+LENIENT_JQ='
+def norm: ascii_downcase | gsub("[^\\p{L}\\p{N}]+"; " ") | split(" ") | map(select(length > 0));
+def contains_all($want): . as $have | $want | all(. as $t | $have | index([$t]) != null);
+($want_name | norm) as $w
+| ((.[$id_field] // []) | index($want_id) != null)
+  or ([(.[$name_field] // [])[] | norm] | any(contains_all($w)))
+'
+
+# score <engine> <url-suffix>
+score() {
+    local engine="$1" suffix="$2"
+    local n=0 corr_hit=0 type_hit=0 corr_len=0 type_len=0 fails=0 start elapsed
+    echo ""
+    echo "=== $engine"
+    start=$(date +%s)
+    while IFS=$'\t' read -r id want_corr want_type want_corr_name want_type_name; do
+        [ -n "$id" ] || continue
+        n=$((n + 1))
+        body=$(api "/api/documents/$id/$suffix")
+        if ! printf '%s' "$body" | jq -e . >/dev/null 2>&1; then
+            fails=$((fails + 1))
+            [ "$fails" -le 3 ] && printf '    %s: keine gueltige Antwort: %s\n' \
+                "$id" "$(printf '%s' "$body" | tr '\n' ' ' | head -c 120)" >&2
+            continue
+        fi
+        if printf '%s' "$body" | jq -e --argjson w "$want_corr" \
+               '.correspondents // [] | index($w) != null' >/dev/null 2>&1; then
+            corr_hit=$((corr_hit + 1))
+        fi
+        if printf '%s' "$body" | jq -e --argjson w "$want_type" \
+               '.document_types // [] | index($w) != null' >/dev/null 2>&1; then
+            type_hit=$((type_hit + 1))
+        fi
+        if printf '%s' "$body" | jq -e \
+               --argjson want_id "$want_corr" --arg want_name "$want_corr_name" \
+               --arg id_field correspondents --arg name_field suggested_correspondents \
+               "$LENIENT_JQ" >/dev/null 2>&1; then
+            corr_len=$((corr_len + 1))
+        fi
+        if printf '%s' "$body" | jq -e \
+               --argjson want_id "$want_type" --arg want_name "$want_type_name" \
+               --arg id_field document_types --arg name_field suggested_document_types \
+               "$LENIENT_JQ" >/dev/null 2>&1; then
+            type_len=$((type_len + 1))
+        fi
+        [ $((n % 25)) -eq 0 ] && echo "    $n/$N"
+    done < "$WORK/docs.txt"
+    elapsed=$(( $(date +%s) - start ))
+    pct() { awk -v a="$1" -v b="$2" 'BEGIN{if (b>0) printf "%.1f", 100*a/b; else print "0.0"}'; }
+    printf '    correspondent, offered  : %5s%%   recognised: %5s%%\n' \
+        "$(pct "$corr_hit" "$n")" "$(pct "$corr_len" "$n")"
+    printf '    document type, offered  : %5s%%   recognised: %5s%%\n' \
+        "$(pct "$type_hit" "$n")" "$(pct "$type_len" "$n")"
+    printf '    no usable answer        : %s of %s\n' "$fails" "$n"
+    printf '    %ss total, %.1fs per document\n' "$elapsed" "$(awk -v e="$elapsed" -v n="$n" 'BEGIN{print e/n}')"
+    printf '%-9s %7s %7s %7s %7s %6s %8ss\n' "$engine" \
+        "$(pct "$corr_hit" "$n")" "$(pct "$corr_len" "$n")" \
+        "$(pct "$type_hit" "$n")" "$(pct "$type_len" "$n")" \
+        "$fails" "$elapsed" >> "$WORK/summary.txt"
+}
+
+: > "$WORK/summary.txt"
+FLOOR=$(awk -v m="$(cat "$WORK/corr_max.txt")" -v t="$(cat "$WORK/corr_total.txt")" \
+    'BEGIN{if (t>0) printf "%.1f", 100*m/t; else print "?"}')
+echo "documents: $N   most-frequent-correspondent floor: ${FLOOR}%"
+
+case "$ENGINE" in
+    both)    score classic suggestions/; score ai ai_suggestions/ ;;
+    classic) score classic suggestions/ ;;
+    ai)      score ai ai_suggestions/ ;;
+    *) echo "unknown engine: $ENGINE" >&2; exit 2 ;;
+esac
+
+echo ""
+echo "================================================================"
+printf '%-9s %7s %7s %7s %7s %6s %9s\n' \
+    ENGINE CORR CORR+ TYPE TYPE+ FAILS TIME
+cat "$WORK/summary.txt"
+printf '%-9s %7s\n' "floor" "$FLOOR"
+echo "================================================================"
+echo "CORR/TYPE  = true label offered as a resolved id (what the UI shows)"
+echo "CORR+/TYPE+ = also counting names the LLM got right but matching dropped"

scripts/paperless-eval/README.md

@@ -1,198 +0,0 @@
-# Comparing chat models on Paperless suggestions
-
-Which local LLM is worth running behind paperless-ngx's AI suggestions, measured
-against documents a human already filed, in the `paperless-eval` namespace
-rather than against the live archive.
-
-**Applied by hand, not by Flux.** A Job is immutable, so re-running a
-configuration means deleting and recreating it, and a Flux-managed Job would
-simply come back. The namespace itself *is* managed
-(`kubernetes/apps/paperless-eval/`); only the measurement is not.
-
-## Running one
-
-Screen first, always:
-
-```sh
-./run-candidate.sh qwen3-8b 5          # ~5 min to a rejection, ~10 to a pass
-kubectl -n paperless-eval logs -f job/eval-qwen3-8b -c eval
-
-kubectl -n paperless-eval delete job eval-qwen3-8b
-./run-candidate.sh qwen3-8b            # the real measurement, 50 documents
-```
-
-Three of the first four candidates could not do the task at all, and each was
-left running for an hour before anyone looked. eval.py now stops after three
-consecutive failures from the first document, so a screening run rejects a
-hopeless candidate in about five minutes.
-
-Screening uses genuine paperless requests. That matters — `probe-toolcall.sh`
-is cheaper still and worth a first pass, but it passed two candidates that then
-failed on document one, because a synthesised request cannot reproduce the RAG
-neighbours and taxonomy candidate lists that make a real prompt what it is.
-
-Never compare numbers across sample sizes. A screen is a gate; the comparison
-happens at 50.
-
-Candidates live in `candidates.tsv`; the runner substitutes the model-specific
-values into `job-template.yaml` and refreshes the `eval.py` ConfigMap. Adding a
-candidate is a line in that file, not a copy of the manifest — the sidecar
-configuration deliberately exists once.
-
-Two run at a time: the pod anti-affinity puts one per node, and only two nodes
-have a free iGPU. A third waits.
-
-Re-running after an edit needs the old Job gone first:
-`kubectl -n paperless-eval delete job eval-<label>`.
-
-Results land on the `paperless-eval-results` PVC as `<label>.jsonl` (one row per
-document) and `<label>.summary.json`, and in the pod log.
-
-## What is being compared
-
-The candidates against each other. The scikit-learn classifier runs in every
-job as a fixed reference line — it costs 15 seconds and keeps running in
-production whatever the LLM does, so `llm_rescued` against `llm_only_wrong`
-says what *this* model adds on top of it.
-
-Do not read the classifier's own score as a quality bar. `train()` fits on every
-non-inbox document with no holdout, so the evaluation documents are in its
-training set and its numbers are memorisation. That does not matter for ranking
-models — it is the same constant under all of them.
-
-## Reading the output
-
-`correspondent_offered` against `correspondent_recognised` is the
-`MATCH_THRESHOLD = 0.8` gap: the model names the right entity and difflib
-rejects it. Measured, not hypothetical — "GOTOOLS GmbH" against the stored
-"GOTOOLS" scores ~0.74 and is thrown away despite being an exact prefix.
-
-`type_specific` against `type_catchall` splits on whether the true type is
-"Sonstige", which holds 1133 of ~3900 documents and is the one label with no
-semantic content. A model reading the document has to infer "none of the above".
-
-`failed` counts documents where the model produced nothing usable. With
-`--n-predict 768` in place, that is mostly non-termination: paperless reports
-the truncated response as a flat `Invalid AI configuration` and discards the
-content, so what the model was doing is visible only in the llama-server log.
-
-## Results so far
-
-| model | corr. offered | corr. recognised | type | specific | catchall | failed | seconds |
-|---|---|---|---|---|---|---|---|
-| *classifier (reference)* | 72.0 | 72.0 | 100.0 | 100.0 | 100.0 | 0 | 18 |
-| Qwen3.6-35B-A3B Q4_K_M | 72.9 | 77.1 | 85.4 | 94.9 | 44.4 | 2 | 5494 |
-| Ministral-3 3B Q4_K_M | 71.4 | 76.2 | 76.2 | 83.3 | 33.3 | **29** | 5511 |
-| Ministral-3 8B Q4_K_M | — | — | — | — | — | 3/3 | aborted |
-| Granite 3.3 8B Q4_K_M | — | — | — | — | — | all | not measurable |
-
-Read the Ministral-3 3B row with its failure count, not without it. It scored 21
-documents and failed 29 — and on those 21 its correspondent numbers sit within
-the noise floor of Qwen3.6's. That is not a near-tie: the 21 are the documents
-it could answer at all, which is a set selected for being easy, and the
-classifier does better on them too (11 of 21 both right against 30 of 48).
-A model that declines three documents in five has no usable accuracy number.
-
-Qwen3.6 complement: 30 both right, 5 rescued, 4 broken, 9 both wrong — and two
-of those four "broken" are the difflib gap above, not model errors.
-
-### How big a difference means anything
-
-Qwen3.6 was measured twice over the same 50 documents, once with the deprecated
---chat-template-kwargs and once with --reasoning off:
-
-    70.8 / 75.0 / 83.3 / 94.9 / 33.3   two failures
-    72.9 / 77.1 / 85.4 / 94.9 / 44.4   two failures
-
-Every one of those gaps is a single document out of 48 — the catchall one is
-one document out of nine. So that is the noise floor: **a difference between
-two candidates smaller than about two documents says nothing.** Decoding is
-sampled, and the same model on the same input does not give the same answer
-twice.
-
-Worth knowing before reading a 2-point lead as a result.
-
-## The instrument is checked, not assumed
-
-Every AI run asks the model "Nenne die Hauptstadt von Frankreich" before the
-documents, every ten documents, and again if it aborts. If Paris does not come
-back, the run stops and says so.
-
-This exists because a whole day of results turned out to be measuring a broken
-instrument. On this Vulkan stack a prompt past some model-specific length stops
-llama.cpp producing language — it repeats one token, " de de de" or
-" phosphorus phosphorus" or whitespace — at HTTP 200 with finish_reason "stop"
-and nothing in the log. The slot then stays broken for every request after it
-until the pod restarts.
-
-So a run against a poisoned slot reports fifty failures that are one event, and
-reads exactly like a model that cannot classify. Six candidates were written up
-that way. gemma-4-26B-A4B answers "Die Hauptstadt von Frankreich ist Paris."
-perfectly on a fresh pod and produces whitespace at a 2334-token prompt; its
-recorded classification scores described a corrupted slot and nothing else.
-
-**A run whose canary breaks is void, not a result.** Failures adjacent to the
-break are one fault. Use `probe-length.sh` to find where the model's boundary
-actually sits before judging it — paperless needs ~5300 tokens.
-
-## Bounds of the harness, not of the model
-
-Two numbers in here are mine and both have produced misleading results:
-
-`--n-predict` caps generation server-side, default 768. A candidate that stops
-exactly there has hit MY bound. Six candidates were written up as incapable on
-that signal before anyone checked whether they would have finished at 900
-tokens. A working answer takes 241-291 tokens, so 768 looked generous — and
-"looked generous" is not evidence.
-
-`EVAL_TIMEOUT` bounds the client, default 300 s. Raising the token cap alone
-just moves the failure here: at 6 tok/s a 4096-token budget needs 620 s of
-generation, so the run fails with TimeoutError instead. Raise both together.
-
-    ./run-candidate.sh <label> <sample> <n_predict> <timeout>
-    ./run-candidate.sh gemma4-26b-a4b 3 4096 900
-
-The Q8 precision comparison had to be thrown away over this. Q8's uncapped
-answers run 460-520 tokens against Q4's 280-443 — about 60% longer for the same
-task — so 20 of 50 documents hit the 768 cap against Q4's one, and the run
-looked like "Q8 classifies worse" when it measured "Q8 is wordier than my
-bound". Before comparing two configurations, check that neither is sitting on
-the cap.
-
-## What to watch for
-
-**Prompt length.** Past roughly 14k tokens this stack stops producing language
-and starts producing garbage or nothing, without any error — and a single such
-request poisons the llama.cpp slot until the pod restarts. Paperless prompts sit
-near 5k, but the 252 documents over 14000 characters push toward the boundary
-with their RAG neighbours. A run that starts scoring zeros mid-way is more
-likely this than the model.
-
-**Ground truth.** It is the current filing, and over ten years conventions
-change. A disagreement with an old document is not automatically an error.
-
-## The models volume fills up
-
-Each candidate leaves its weights on the shared `llama-models` PVC — 2.5 GB for
-a small dense model, 17-25 GB for an MoE, 34 GB for a Q8. The volume is 120 GiB
-and nine screened candidates filled it completely.
-
-A full volume does not announce itself. llama-server cannot write its metadata
-cache, so it cannot resolve the repository listing, and says:
-
-    no GGUF files found in repository <repo>
-    Available GGUF files:
-      - <whatever it had cached before the disk filled>
-
-which reads as a wrong model reference. The pod then sits in
-`Init:CrashLoopBackOff`, which from outside is indistinguishable from a slow
-34 GB download. Two jobs spent three hours that way.
-
-**Check `df -h /models` before believing anything llama-server says about a
-repository.** Clear out rejected candidates when the volume gets tight — the
-weights are re-downloadable, so deleting them costs only bandwidth:
-
-```sh
-kubectl -n paperless-eval exec <a pod mounting llama-models> -- \
-  sh -c 'cd /models && du -sh * | sort -h'
-```

scripts/paperless-eval/candidates.tsv

@@ -1,238 +0,0 @@
-# label	hf-repo	hf-file	alias	mem-request	mem-limit
-#
-# Every candidate must do function calling: paperless calls the model through
-# chat_with_tools(..., tool_required=True) with DocumentClassifierSchema as a
-# tool, not via JSON mode. A model whose chat template has no tool support does
-# not score badly here, it fails outright.
-#
-# Instruct variants, never Reasoning ones. Qwen3.6 spent two of fifty documents
-# generating until something cut it off, and a model tuned to think first does
-# that by design. The one exception is Qwen3.6 itself, which is measured with
-# thinking switched off.
-#
-# Sizes. An earlier version of this note said "a smaller dense model gains at
-# both ends" and chose the Ministral pair on that basis. Measured, it is wrong:
-#
-#     Ministral-3 8B dense, ~4.7 GB weights   7.1-8.2 tok/s
-#     Qwen3.6 35B-A3B MoE,  ~20 GB weights    11 tok/s
-#
-# The largest model is the fastest one here. But the bandwidth explanation I
-# gave for it is wrong, and measuring Q8 disproved it:
-#
-#     Qwen3.6-35B-A3B Q4_K_M   ~21 GB   11.0 tok/s
-#     Qwen3.6-35B-A3B Q8_0     ~34 GB   10.3 tok/s
-#
-# Doubling the bytes per active parameter cost ten percent, not half. So decode
-# is NOT bandwidth-bound for this MoE — it is compute-bound on many small
-# per-expert matmuls, which also explains the earlier measurement of 21.4 GB/s
-# effective against a 53.9 GB/s ceiling: it was already leaving 60% of the bus
-# unused.
-#
-# Consequences: the bytes-per-token formula predicts dense models and not this
-# one, and higher precision is nearly free on an MoE here — about 10% slower
-# for double the weight precision.
-#
-# So the order to prefer is: MoE with few active parameters, then dense 3-9B,
-# then dense 10-14B only for an exceptional model (~4 tok/s, roughly three
-# hours for 50 documents), and nothing dense above 14B.
-qwen35	ggml-org/Qwen3.6-35B-A3B-GGUF	Qwen3.6-35B-A3B-Q4_K_M.gguf	qwen3.6-35b-a3b	24Gi	32Gi
-#
-# The same model at higher precision, to find out what Q4 costs. Everything in
-# this file is Q4_K_M, so the whole comparison could be sitting on quantisation
-# damage nobody has measured — and there is a concrete suspicion, not just a
-# worry: Ministral-3's leak of [THINK] and [TOOL_CALLS] as literal text is the
-# kind of fault a 4-bit quantisation can produce, and Q8 would separate "the
-# model cannot do this" from "Q4 broke it".
-#
-# ~34 GB resident against Q4's ~21, and roughly 6 tok/s against 11, so about
-# double the wall clock. Read the result against the noise floor in README.md:
-# below about two documents of difference, this says nothing.
-qwen35-q8	ggml-org/Qwen3.6-35B-A3B-GGUF	Qwen3.6-35B-A3B-Q8_0.gguf	qwen3.6-35b-a3b-q8	40Gi	48Gi
-#
-# --- MoE, few active parameters: the shape that wins on this hardware ---
-qwen3-30b-a3b	bartowski/Qwen_Qwen3-30B-A3B-Instruct-2507-GGUF	Qwen_Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf	qwen3-30b-a3b	22Gi	28Gi
-nemotron3-30b-a3b	bartowski/nvidia_Nemotron-3-Nano-30B-A3B-GGUF	nvidia_Nemotron-3-Nano-30B-A3B-Q4_K_M.gguf	nemotron3-30b-a3b	28Gi	34Gi
-glm47-flash	bartowski/zai-org_GLM-4.7-Flash-GGUF	zai-org_GLM-4.7-Flash-Q4_K_M.gguf	glm-4.7-flash	20Gi	26Gi
-gemma4-26b-a4b	unsloth/gemma-4-26B-A4B-it-GGUF	gemma-4-26B-A4B-it-UD-Q4_K_M.gguf	gemma-4-26b-a4b	20Gi	26Gi
-qwen35-35b-a3b	bartowski/Qwen_Qwen3.5-35B-A3B-GGUF	Qwen_Qwen3.5-35B-A3B-Q4_K_M.gguf	qwen3.5-35b-a3b	24Gi	30Gi
-nemotron35-30b-a3b	bartowski/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF	NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Q4_K_M.gguf	nemotron3.5-30b-a3b	28Gi	34Gi
-#
-# --- dense, small, with German named in the model's own card ---
-glm4-9b	bartowski/glm-4-9b-chat-GGUF	glm-4-9b-chat-Q4_K_M.gguf	glm-4-9b	10Gi	14Gi
-commandr7b	bartowski/c4ai-command-r7b-12-2024-GGUF	c4ai-command-r7b-12-2024-Q4_K_M.gguf	command-r7b	10Gi	14Gi
-qwen25-7b	bartowski/Qwen2.5-7B-Instruct-GGUF	Qwen2.5-7B-Instruct-Q4_K_M.gguf	qwen2.5-7b	10Gi	14Gi
-llama31-8b	bartowski/Meta-Llama-3.1-8B-Instruct-GGUF	Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf	llama-3.1-8b	10Gi	14Gi
-nemotron-nano-12b	bartowski/nvidia_NVIDIA-Nemotron-Nano-12B-v2-GGUF	nvidia_NVIDIA-Nemotron-Nano-12B-v2-Q4_K_M.gguf	nemotron-nano-12b	12Gi	16Gi
-phi4-mini	bartowski/microsoft_Phi-4-mini-instruct-GGUF	microsoft_Phi-4-mini-instruct-Q4_K_M.gguf	phi-4-mini	8Gi	12Gi
-#
-# --- dense, small, no German claim but no thinking risk either ---
-qwen3-4b-2507	bartowski/Qwen_Qwen3-4B-Instruct-2507-GGUF	Qwen_Qwen3-4B-Instruct-2507-Q4_K_M.gguf	qwen3-4b	8Gi	12Gi
-qwen35-9b	bartowski/Qwen_Qwen3.5-9B-GGUF	Qwen_Qwen3.5-9B-Q4_K_M.gguf	qwen3.5-9b	10Gi	14Gi
-#
-# WHAT THE RESEARCH RULED OUT, so nobody re-derives it
-#
-# Hard-excluded on tool calling, chat template inspected directly rather than
-# assumed — the string "tool" appears zero times in any of them:
-#   Gemma 3, every size (4B/12B/27B)   phi-4 14B (the mini DOES have it)
-#   Cohere Aya, all variants           InternLM 2.5 / 3
-#
-# Hard-excluded on German, stated by the vendor rather than merely missing:
-#   Falcon 3 — "supports EN, FR, ES, PT", German is not on the list
-#   EXAONE — the small ones are English/Korean; the ones with German are 33B+
-#
-# German fine-tunes are a dead end, and for one consistent reason: SauerkrautLM,
-# DiscoLM, LeoLM, Occiglot and Teuken all overwrite their base model's chat
-# template with a minimal ChatML loop that drops tool calling. VAGOsolutions
-# does this even on bases that have it (Llama-3.1, Qwen2.5). Assume any German
-# tune is tool-incapable until its own tokenizer_config.json says otherwise.
-#
-# On llama.cpp's tool parsing, which explains the Granite failure: current
-# builds have no per-vendor parser table any more. common/chat.cpp exposes
-# PEG_SIMPLE / PEG_NATIVE / PEG_GEMMA4 / PEG_MINIMAX_M3 and derives the grammar
-# from the model's own template. Hermes-style <tool_call> JSON is a first-class
-# tested fixture in tests/test-chat.cpp; Granite's format is not. So this is not
-# "Granite is unsupported" so much as "Granite's template shape is outside what
-# the auto-parser was validated against".
-#
-# SCREENING RESULTS, and a pattern that reframes the whole search
-#
-#   qwen35        MoE 35B total / 3B active   works, 2 of 50 failed
-#   qwen3-8b      dense 8B                    4 of 5 failed
-#   hermes4-14b   dense 14B                   3 of 3 failed, aborted
-#   ministral8b   dense 8B                    3 of 3 failed, aborted
-#   ministral3b   dense 3B                    29 of 50 failed
-#   granite8b     dense 8B                    every document failed
-#
-# Five dense models from 3B to 14B cannot produce a valid call against
-# paperless's 14-field schema; the one mixture-of-experts model can. Hermes is
-# the cleanest of these failures: its <tool_call> JSON is an explicitly tested
-# fixture in llama.cpp's own test suite, so the parser demonstrably handles the
-# format and the failure has to be the model.
-#
-# Working hypothesis: holding a long grammar-constrained structure needs model
-# capacity, and what supplies it is TOTAL parameters, not active ones. If that
-# is right, "go smaller for speed" is a dead end here — and the consolation is
-# that the fastest shape on this hardware, MoE with few active parameters, is
-# also the only one that works.
-#
-# qwen3-30b-a3b tests it directly: 30B total, 3B active.
-#
-# CORRECTION: Gemma is not excluded after all. Gemma *3* has no tool support —
-# "tool" appears zero times in its chat template at every size, checked
-# directly. Gemma *4* 26B-A4B does have it, documented in the template, plus
-# "140+ languages pretrained, 35+ out of the box". Excluding a family by its
-# previous generation was sloppy.
-#
-# Two of these carry a Vulkan risk worth checking in the llama-server log
-# before trusting a run: Nemotron-3.5 interleaves real Mamba2 layers and
-# Qwen3.5 uses DeltaNet linear attention, and llama.cpp's Vulkan SSM shaders
-# are less exercised on Iris Xe than on Arc. Issue #19957 reported exactly this
-# for Qwen3.5-35B-A3B and it turned out to be a stale build, fixed months
-# before b10731 — but confirm layers are actually offloaded rather than
-# silently falling back to CPU, which would read as a slow model.
-#
-# Ruled out after checking: ERNIE-4.5-21B-A3B (only the Thinking variant has
-# tool calling; the instruct one has none), Ling-3.0-flash (124B total),
-# Granite-4.0-H-Small (9B active), DeepSeek-V2-Lite and OLMoE (under 20B
-# total), Cohere North Mini (code-only, no ggml-org/bartowski conversion,
-# cohere2_moe support in llama.cpp unverified), gpt-oss-20b (a reasoning-effort
-# model in Harmony format rather than plain instruct).
-#
-# THE MoE HYPOTHESIS IS DEAD. qwen3-30b-a3b — MoE, 30B total, 3B active, the
-# shape that was supposed to explain everything — scored 1 of 5 and failed 4,
-# indistinguishable from the dense 8B.
-#
-# So neither architecture nor size predicts success:
-#
-#   Qwen3.6-35B-A3B   MoE 35B/3B   Apr 2026   2 of 50 failed
-#   Qwen3-30B-A3B     MoE 30B/3B   older      4 of 5 failed
-#   Qwen3-8B          dense 8B     older      4 of 5 failed
-#   Hermes-4-14B      dense 14B    —          3 of 3 failed
-#
-# The only thing correlating with success so far is how new and how strong the
-# model is. That is an unsatisfying hypothesis because it predicts nothing
-# cheaply — but it is what the data says, and two architectural theories have
-# already died here.
-#
-# qwen35-35b-a3b isolates it: same architecture, same total and active
-# parameters as the working baseline, one generation older. If it fails, the
-# generation is the variable and shape is noise.
-#
-# qwen35-35b-a3b PASSES the screen: 5 of 5, no failures, 584 s. So the
-# generation hypothesis is dead too — Qwen3.5 is a generation OLDER than the
-# working Qwen3.6 and handles the schema cleanly.
-#
-# What the two working models share is 35B total parameters; the one failing
-# MoE has 30B. But that is confounded — Qwen3-30B-A3B is both smaller and
-# older, and three data points cannot separate the two.
-#
-#   Qwen3.6-35B-A3B   35B total   newest   works
-#   Qwen3.5-35B-A3B   35B total   older    works
-#   Qwen3-30B-A3B     30B total   oldest   4 of 5 failed
-#
-# glm47-flash separates them: 30B total, newest generation. If it works,
-# generation is what matters and 35B is not a threshold. If it fails, total
-# size is, and the search should stop looking below ~35B.
-#
-# PREFILL SPEED IS A SELECTION CRITERION AND I WAS NOT TRACKING IT.
-# Measured on this hardware, prompt processing varies more than sixfold across
-# architectures, and it is roughly half of a run's wall clock:
-#
-#   dense 8B (Ministral)        240 tok/s
-#   Qwen3.6-35B-A3B             100 tok/s
-#   GLM-4.7-Flash                31 tok/s   decode 4.9 tok/s
-#
-# GLM-4.7-Flash is disqualified on that alone. At 31 tok/s a 5300-token prompt
-# takes 168 s, times ~2.6 requests per document, is over seven minutes per
-# document and about six hours for a run — before considering whether its
-# answers are any good.
-#
-# It also shows why the 300 s REQUEST_TIMEOUT needs care: 4453 prompt tokens at
-# 31.5 tok/s plus 768 generated at 4.9 tok/s is 298 s, so a slow model turns a
-# content failure into a TimeoutError and muddies the diagnosis. Read the
-# llama-server timings, not just the eval error.
-#
-# glm47-flash: aborted, 3 of 3 — but every one a TimeoutError, not an invalid
-# answer, so this does NOT settle the size-versus-generation question it was
-# started to settle. It failed on speed. Document 1 did reach the --n-predict
-# cap without producing a call, but 2 and 3 were cut off by the 300 s client
-# bound before anything could be concluded about them.
-#
-# Disqualified regardless: 31 tok/s prefill and 4.9 tok/s decode is over seven
-# minutes per document and about six hours for a full run.
-#
-# gemma4-26b-a4b is the better discriminator — 26B total, well under 35, and
-# newest generation. Works means generation and quality matter rather than a
-# 35B threshold; fails is consistent with total size being the line.
-#
-# gemma4-26b-a4b: aborted, 3 of 3, all HTTP 400 rather than timeouts, and the
-# server log shows it ran to the --n-predict cap on every one. So it is the
-# same capability failure as the small dense models, not a template mismatch.
-# (Prefill 45-50 tok/s, decode 6 tok/s — also not fast.)
-#
-# THE SIZE THRESHOLD SURVIVED ITS FIRST REAL TEST. gemma-4 was chosen
-# specifically because it could refute it — newest generation, 26B total — and
-# it did not:
-#
-#   Qwen3.6-35B-A3B   35B   newest    works
-#   Qwen3.5-35B-A3B   35B   older     works
-#   Qwen3-30B-A3B     30B   oldest    fails
-#   gemma-4-26B-A4B   26B   NEWEST    fails
-#   Hermes-4-14B      14B   —         fails
-#   dense 3-8B        —     —         fail
-#
-# Two generations work at 35B total; everything at 30B and below fails,
-# including the newest model in the field. Operating rule until something
-# contradicts it: do not spend a screening slot on anything under ~35B total.
-#
-# Not pinned further on purpose. Whether the line sits at 31B or 34B is
-# interesting and not decision-relevant — there are two working candidates and
-# the useful next number is a full 50-document run of the second one.
-#
-# Re-enabled for length probing, not for scoring. Their classification numbers
-# were taken against a slot that may already have been corrupted, so the only
-# honest thing to do with them is find out where each one's boundary sits.
-hermes4-14b	bartowski/NousResearch_Hermes-4-14B-GGUF	NousResearch_Hermes-4-14B-Q4_K_M.gguf	hemes-4-14b	14Gi	20Gi
-qwen3-8b	bartowski/Qwen_Qwen3-8B-GGUF	Qwen_Qwen3-8B-Q4_K_M.gguf	qwen3-8b	12Gi	16Gi
-ministral8b	bartowski/mistralai_Ministral-3-8B-Instruct-2512-GGUF	mistralai_Ministral-3-8B-Instruct-2512-Q4_K_M.gguf	ministral-3-8b	12Gi	16Gi
-granite8b	bartowski/ibm-granite_granite-3.3-8b-instruct-GGUF	ibm-granite_granite-3.3-8b-instruct-Q4_K_M.gguf	granite-3.3-8b	12Gi	16Gi

scripts/paperless-eval/eval.py

@@ -1,544 +0,0 @@
-#!/usr/bin/env python3
-"""Measure what Paperless suggests for documents whose labels a human already chose.
-
-Runs both engines against the same documents:
-
-    /api/documents/<id>/suggestions/      the built-in scikit-learn classifier
-    /api/documents/<id>/ai_suggestions/   the LLM path, with RAG
-
-WHAT THIS ANSWERS, AND WHAT AN EARLIER VERSION DID NOT
-
-The two engines are not alternatives — the classifier costs 0.6 s per document
-and keeps running whatever the LLM does. So the useful question is not "which
-scores higher" but "what does the LLM add where the classifier is wrong", and
-that cannot be recovered from two aggregate percentages: 74% and 72% could
-overlap almost entirely or barely at all, and the difference between those two
-worlds is the entire value of running an LLM.
-
-So every document is written out individually, with both engines' verdicts, and
-the summary is derived from that rather than counted along the way.
-
-SCORING
-
-Correspondent and document type are each scored twice:
-
-    offered      the true label's id came back in `correspondents` /
-                 `document_types` — what the UI actually presents
-    recognised   also counting the true label's name appearing wordwise in
-                 `suggested_correspondents` / `suggested_document_types`
-
-The gap between them is one specific failure: the model names the right entity
-but paperless_ai.matching cannot resolve it, because difflib runs at
-MATCH_THRESHOLD = 0.8 and a correct but more verbose answer falls below it —
-"Generali Deutschland Krankenversicherung AG" against the existing "Generali
-AG". Reporting only the first blames the model for a matcher problem; only the
-second claims a usefulness the UI does not deliver.
-
-Document type is additionally split on whether the true type is "Sonstige",
-which holds 1133 of ~3900 documents and is the one label with no semantic
-content — a restaurant reservation lands there because nothing else fits. The
-classifier can memorise that; a model reading the document has to infer "none
-of the above". Scored separately rather than dropped, because the model does
-sometimes get it right.
-
-GROUND TRUTH
-
-The current filing, which deserves care: over ten years conventions change, and
-a disagreement with an old document is not automatically an error. Treat the
-comparison between engines as the result and the absolute numbers as
-indicative.
-
-Env:
-    PAPERLESS_URL     default http://127.0.0.1:8000
-    PGHOST/PGDATABASE standard libpq, for ground truth and the API token
-    EVAL_SAMPLE       documents to score, default 50
-    EVAL_ENGINES      comma-separated subset of classic,ai — default both
-    EVAL_LABEL        free-text name for this configuration, recorded in the
-                      output so results from different jobs stay distinguishable
-    EVAL_OUT          directory for results, default /results
-"""
-
-from __future__ import annotations
-
-import hashlib
-import json
-import os
-import re
-import secrets
-import sys
-import time
-import urllib.error
-import urllib.request
-
-import psycopg
-
-PAPERLESS_URL = os.environ.get("PAPERLESS_URL", "http://127.0.0.1:8000")
-SAMPLE = int(os.environ.get("EVAL_SAMPLE", "50"))
-ENGINES = os.environ.get("EVAL_ENGINES", "classic,ai").split(",")
-LABEL = os.environ.get("EVAL_LABEL", "unlabelled")
-OUT_DIR = os.environ.get("EVAL_OUT", "/results")
-
-# A backstop, not the real bound. Generation is capped server-side by
-# --n-predict on the llama-server sidecar, which is what actually stops a model
-# that will not terminate; this catches everything else — a wedged sidecar, a
-# request that never reaches it.
-#
-# 300 s against the ~73 s a healthy document takes end to end, prompt included.
-# It does work: llama-server logs `stop: cancel task` on the disconnect and
-# starts the next request 0.4 s later.
-REQUEST_TIMEOUT = int(os.environ.get("EVAL_TIMEOUT", "300"))
-
-# Consecutive failures from the very first document that mean the engine is
-# unusable rather than merely weak. Three, because one could be a single hard
-# document and two could be coincidence.
-ABORT_AFTER = 3
-
-CATCHALL_TYPE = "Sonstige"
-
-
-def log(msg: str) -> None:
-    print(f"[eval] {msg}", flush=True)
-
-
-def api_token(conn) -> str:
-    """A token for this run, against a user belonging to this run alone.
-
-    The API needs one and this database is a disposable clone, so the run
-    issues its own rather than depending on a token that happened to come
-    across in the restore.
-
-    The per-label user is not tidiness. Django's authtoken holds one token per
-    user, so an earlier version took the first superuser and deleted its
-    existing token first — which meant the second of two candidates running
-    concurrently silently revoked the first one's credentials. That surfaced as
-    a mid-run `HTTP 401: Invalid token` sitting among genuine model failures,
-    where it reads like one more broken document rather than the harness
-    sabotaging itself. Both jobs share this database; only the user can differ.
-    """
-    token = secrets.token_hex(20)
-    username = f"eval-{LABEL}"
-    with conn.cursor() as cur:
-        cur.execute("select id from auth_user where username = %s", (username,))
-        row = cur.fetchone()
-        if row is None:
-            cur.execute(
-                """
-                insert into auth_user (password, is_superuser, username,
-                                       first_name, last_name, email, is_staff,
-                                       is_active, date_joined)
-                values ('!unusable', true, %s, '', '', '', true, true, now())
-                returning id
-                """,
-                (username,),
-            )
-            row = cur.fetchone()
-        user_id = row[0]
-        cur.execute("delete from authtoken_token where user_id = %s", (user_id,))
-        cur.execute(
-            "insert into authtoken_token (key, created, user_id) values (%s, now(), %s)",
-            (token, user_id),
-        )
-    conn.commit()
-    return token
-
-
-def sample(conn) -> list[dict]:
-    """The documents to score, ordered by md5 of the id.
-
-    Deterministic, so every engine and every configuration sees the same
-    documents and results stay comparable across jobs. Only documents carrying
-    both labels are usable as ground truth.
-    """
-    with conn.cursor() as cur:
-        cur.execute(
-            """
-            select d.id, d.correspondent_id, d.document_type_id, c.name, t.name
-              from documents_document d
-              join documents_correspondent c on c.id = d.correspondent_id
-              join documents_documenttype  t on t.id = d.document_type_id
-             where d.correspondent_id is not null
-               and d.document_type_id is not null
-               and length(d.content) > 400
-             order by md5(d.id::text)
-             limit %s
-            """,
-            (SAMPLE,),
-        )
-        return [
-            {
-                "id": r[0],
-                "correspondent_id": r[1],
-                "document_type_id": r[2],
-                "correspondent_name": r[3],
-                "document_type_name": r[4],
-            }
-            for r in cur.fetchall()
-        ]
-
-
-def floor(conn) -> float:
-    """Always guessing the most frequent correspondent.
-
-    Without it a number like "70% correct" cannot be read: one correspondent
-    holds several hundred of ~3900 documents.
-    """
-    with conn.cursor() as cur:
-        cur.execute(
-            """
-            select max(c), sum(c) from (
-                select count(*) c from documents_document
-                 where correspondent_id is not null group by correspondent_id
-            ) x
-            """,
-        )
-        top, total = cur.fetchone()
-    # float() on both: psycopg returns Postgres numerics as Decimal, and
-    # mixing that with a float literal raises TypeError rather than coercing.
-    return 100.0 * float(top) / float(total) if total else 0.0
-
-
-def fetch(path: str, token: str) -> tuple[dict | None, str | None, float]:
-    req = urllib.request.Request(
-        f"{PAPERLESS_URL}{path}",
-        headers={"Authorization": f"Token {token}"},
-    )
-    start = time.monotonic()
-    try:
-        with urllib.request.urlopen(req, timeout=REQUEST_TIMEOUT) as resp:
-            body = resp.read()
-        return json.loads(body), None, time.monotonic() - start
-    except urllib.error.HTTPError as err:
-        return None, f"HTTP {err.code}: {err.read()[:200]!r}", time.monotonic() - start
-    except Exception as err:  # noqa: BLE001 — a failed document is data, not a crash
-        return None, f"{type(err).__name__}: {err}", time.monotonic() - start
-
-
-def normalise(text: str) -> set[str]:
-    return {w for w in re.split(r"[^\w]+", text.lower()) if w}
-
-
-def recognised(payload: dict, field: str, want_name: str) -> bool:
-    """True when a suggested name contains every word of the true label's name.
-
-    Handles the observed failure directly: the model answers with a superset of
-    the stored name, which difflib rejects at 0.8 but a reader would accept.
-    """
-    want = normalise(want_name)
-    return any(want <= normalise(name) for name in payload.get(field) or [])
-
-
-def degenerate(payload: dict) -> str | None:
-    """Whether this looks like broken inference rather than a wrong guess.
-
-    Past roughly 14k prompt tokens this stack stops producing language — first
-    empty completions, then runs of digits and CJK characters — and one such
-    request poisons the llama.cpp slot until the pod restarts. Nothing in the
-    HTTP response says so: status 200, finish_reason "stop", well-formed JSON.
-
-    Scored as a wrong answer it would silently depress a candidate's numbers,
-    and because the slot stays poisoned it would take the rest of the run with
-    it. Flagged instead, so the failure is visible as itself.
-    """
-    title = payload.get("title") or ""
-    if not title.strip():
-        return "empty title"
-    # Digits, punctuation and whitespace are normal in a document title —
-    # "Rechnung 2025-5425541" is 8 letters in 21 characters and tripped an
-    # earlier version of this check. What broken inference produces is
-    # characters from other scripts, so look for those directly rather than
-    # inferring them from a shortage of Latin ones.
-    foreign = sum(c.isalpha() and not (c.isascii() or c in "äöüÄÖÜßáéèêíóúàçñ") for c in title)
-    if foreign > 0.2 * len(title):
-        return f"non-latin title: {title[:40]!r}"
-    return None
-
-
-def score_document(payload: dict, doc: dict) -> dict:
-    offered_corr = doc["correspondent_id"] in (payload.get("correspondents") or [])
-    offered_type = doc["document_type_id"] in (payload.get("document_types") or [])
-    return {
-        "correspondent_offered": offered_corr,
-        "correspondent_recognised": offered_corr
-        or recognised(payload, "suggested_correspondents", doc["correspondent_name"]),
-        "type_offered": offered_type,
-        "type_recognised": offered_type
-        or recognised(payload, "suggested_document_types", doc["document_type_name"]),
-        "suggested_correspondents": payload.get("suggested_correspondents") or [],
-        "suggested_document_types": payload.get("suggested_document_types") or [],
-        "title": payload.get("title"),
-    }
-
-
-LLM_ENDPOINT = os.environ.get("PAPERLESS_AI_LLM_ENDPOINT", "http://127.0.0.1:8080/v1")
-
-
-def canary() -> tuple[bool, str]:
-    """Ask the model something it cannot get wrong, and check it did not.
-
-    The instrument has to be shown working before its readings mean anything,
-    and this one can fail silently in a way that looks like a bad model. On
-    this Vulkan stack a prompt past some model-specific length stops llama.cpp
-    producing language: it repeats one token — " de de de", " phosphorus
-    phosphorus", whitespace — at HTTP 200 with finish_reason "stop". The slot
-    stays broken for every request after it until the pod restarts.
-
-    So a run that measured 50 documents against a poisoned slot reports 50
-    failures that are really one event, and reads as a model that cannot
-    classify. That is exactly what happened before this existed.
-
-    Run before the documents and again after: healthy then broken means the
-    failures in between are one fault, not evidence about the model.
-    """
-    body = json.dumps({
-        "model": os.environ.get("PAPERLESS_AI_LLM_MODEL", "probe"),
-        "max_tokens": 40,
-        "messages": [{"role": "user", "content": "Nenne die Hauptstadt von Frankreich."}],
-    }).encode()
-    req = urllib.request.Request(
-        f"{LLM_ENDPOINT}/chat/completions",
-        data=body,
-        headers={"Content-Type": "application/json"},
-    )
-    try:
-        with urllib.request.urlopen(req, timeout=120) as resp:
-            payload = json.loads(resp.read())
-        text = payload["choices"][0]["message"].get("content") or ""
-        text += payload["choices"][0]["message"].get("reasoning_content") or ""
-        return ("Paris" in text), text[:80]
-    except Exception as err:  # noqa: BLE001
-        return False, f"{type(err).__name__}: {err}"
-
-
-def run(engine: str, suffix: str, docs: list[dict], token: str) -> list[dict]:
-    log(f"=== {engine}: {len(docs)} documents")
-    rows = []
-    if engine == "ai":
-        ok, text = canary()
-        log(f"  canary before: {'OK' if ok else 'BROKEN'} — {text!r}")
-        if not ok:
-            log("  The model is not answering a trivial question correctly, so")
-            log("  nothing measured after this describes its classification.")
-            return [{"engine": engine, "canary_before": False, "canary_text": text}]
-    for n, doc in enumerate(docs, 1):
-        payload, error, elapsed = fetch(
-            f"/api/documents/{doc['id']}/{suffix}",
-            token,
-        )
-        row = {
-            "engine": engine,
-            "document_id": doc["id"],
-            "seconds": round(elapsed, 1),
-            "truth_correspondent": doc["correspondent_name"],
-            "truth_type": doc["document_type_name"],
-            "catchall_type": doc["document_type_name"] == CATCHALL_TYPE,
-        }
-        if payload is None:
-            # An authentication failure is never a fact about the model, and
-            # counted alongside real ones it reads like one more broken
-            # document. Stop loudly instead: something revoked this run's
-            # credentials mid-flight, and every number after it is worthless.
-            if error and error.startswith("HTTP 401"):
-                raise SystemExit(
-                    f"[eval] FATAL: {error}\n"
-                    f"[eval] This run's token stopped working part-way through. "
-                    f"Django's authtoken holds one token per user, so another "
-                    f"job sharing this database and this user revoked it.",
-                )
-            row["error"] = error
-            log(f"  {doc['id']}: {error}")
-        else:
-            row.update(score_document(payload, doc))
-            # Only the AI path returns a title; the classifier answers with ids
-            # alone, so an "empty title" there is normal rather than broken.
-            if engine == "ai" and (broken := degenerate(payload)) is not None:
-                row["degenerate"] = broken
-                log(f"  {doc['id']}: DEGENERATE — {broken}")
-        rows.append(row)
-
-        # Clear the slot between documents so the next one is measured on a
-        # clean server. The corruption this guards against is stateful: one bad
-        # request leaves the slot returning a single repeated token for
-        # everything after it, and without this a run counts that once per
-        # remaining document.
-        if engine == "ai":
-            try:
-                urllib.request.urlopen(
-                    urllib.request.Request(
-                        f"{LLM_ENDPOINT.rsplit('/v1', 1)[0]}/slots/0?action=erase",
-                        data=b"",
-                        method="POST",
-                    ),
-                    timeout=30,
-                )
-            except Exception as err:  # noqa: BLE001 — best effort, never fatal
-                if n == 1:
-                    log(f"  slot erase unavailable ({type(err).__name__}) — "
-                        f"failures after a corrupt document will not be independent")
-
-        if n % 10 == 0:
-            log(f"  {n}/{len(docs)}")
-            # Re-check the instrument, not just the progress. A slot corrupted
-            # part-way through turns every remaining document into a failure
-            # that says nothing about the model, and scattered failures — 29 of
-            # 50 in one earlier run — are exactly the shape that a
-            # start-of-run check alone would miss.
-            if engine == "ai":
-                ok, text = canary()
-                if not ok:
-                    log(f"  CANARY BROKEN at {n}/{len(docs)} — {text!r}")
-                    log("  The model can no longer answer a trivial question, so")
-                    log("  everything from here would measure a dead slot. Stopping.")
-                    log("  Failures recorded before this point are still usable;")
-                    log("  the ones adjacent to it are one fault, not several.")
-                    rows.append({"engine": engine, "canary_failed_at": n, "canary_text": text})
-                    break
-
-        # Stop early when a model cannot do the task at all, rather than
-        # spending an hour writing the same error fifty times. Three candidates
-        # in a row failed this way — Granite because llama.cpp cannot read its
-        # tool calls back, both Ministral sizes because they fill the tool
-        # call's first string with their own [THINK]/[TOOL_CALLS] control
-        # tokens until the token cap. In each case document one already said
-        # so, and the remaining forty-nine said nothing new.
-        #
-        # Deliberately not a ratio: a model that fails a third of the time is a
-        # result worth measuring in full. This only catches the case where
-        # nothing has ever worked.
-        if len(rows) == ABORT_AFTER and all("error" in r for r in rows):
-            ok, text = canary()
-            log(f"  canary after {ABORT_AFTER} failures: {'OK' if ok else 'BROKEN'} — {text!r}")
-            if not ok:
-                log("  The slot is corrupted, so these are ONE fault and not")
-                log(f"  {ABORT_AFTER} independent results. Probe the model's prompt-length")
-                log("  boundary with probe-length.sh before judging it.")
-            log(f"  ABORT — first {ABORT_AFTER} documents all failed:")
-            log(f"    {rows[0]['error']}")
-            log("  Nothing has succeeded, so the rest would measure nothing.")
-            log("  Check the model's tool calling before re-running:")
-            log("    curl -s $ENDPOINT/v1/chat/completions -d @probe.json")
-            log("  A usable candidate answers with finish_reason 'tool_calls'.")
-            break
-
-    # And once at the end, whatever happened. A short run whose first
-    # document succeeds trips neither the abort check nor the every-tenth
-    # one, so without this a five-document screen can finish having never
-    # re-verified the instrument it just used.
-    if engine == "ai":
-        ok, text = canary()
-        log(f"  canary after: {'OK' if ok else 'BROKEN'} — {text!r}")
-        if not ok:
-            log("  Healthy before, broken after: the failures in this run are")
-            log("  one fault and not independent results.")
-            rows.append({"engine": engine, "canary_after": False, "canary_text": text})
-    return rows
-
-
-def pct(hits: int, total: int) -> float:
-    return round(100.0 * hits / total, 1) if total else 0.0
-
-
-def summarise(rows: list[dict], engine: str) -> dict:
-    mine = [r for r in rows if r["engine"] == engine]
-    # Degenerate rows are excluded from the accuracy figures: they measure the
-    # backend giving up, not the model choosing wrongly, and leaving them in
-    # would quietly punish whichever candidate happened to trip the slot.
-    scored = [r for r in mine if "error" not in r and "degenerate" not in r]
-    failed = [r for r in mine if "error" in r]
-    broken = [r for r in mine if "degenerate" in r]
-    specific = [r for r in scored if not r["catchall_type"]]
-    catchall = [r for r in scored if r["catchall_type"]]
-    return {
-        "engine": engine,
-        "scored": len(scored),
-        "failed": len(failed),
-        # Non-zero means the run is suspect from that point on, not that the
-        # model is bad — the slot stays poisoned until the pod restarts.
-        "degenerate": len(broken),
-        "seconds_total": round(sum(r["seconds"] for r in mine)),
-        "correspondent_offered": pct(
-            sum(r["correspondent_offered"] for r in scored), len(scored)
-        ),
-        "correspondent_recognised": pct(
-            sum(r["correspondent_recognised"] for r in scored), len(scored)
-        ),
-        "type_offered": pct(sum(r["type_offered"] for r in scored), len(scored)),
-        "type_specific": pct(
-            sum(r["type_offered"] for r in specific), len(specific)
-        ),
-        "type_catchall": pct(
-            sum(r["type_offered"] for r in catchall), len(catchall)
-        ),
-    }
-
-
-def complement(rows: list[dict]) -> dict | None:
-    """What the LLM adds where the classifier is wrong, and what it costs.
-
-    The reason this script writes per-document rows at all. Only computed when
-    both engines ran, and only over documents both actually scored.
-    """
-    by_engine = {"classic": {}, "ai": {}}
-    for row in rows:
-        usable = "error" not in row and "degenerate" not in row
-        if usable and row["engine"] in by_engine:
-            by_engine[row["engine"]][row["document_id"]] = row
-    both = set(by_engine["classic"]) & set(by_engine["ai"])
-    if not both or not by_engine["ai"]:
-        return None
-
-    rescued = missed = broken = agreed = 0
-    for doc_id in both:
-        classic_hit = by_engine["classic"][doc_id]["correspondent_offered"]
-        ai_hit = by_engine["ai"][doc_id]["correspondent_offered"]
-        if classic_hit and ai_hit:
-            agreed += 1
-        elif not classic_hit and ai_hit:
-            rescued += 1
-        elif classic_hit and not ai_hit:
-            broken += 1
-        else:
-            missed += 1
-    return {
-        "documents": len(both),
-        "both_right": agreed,
-        "llm_rescued": rescued,
-        "llm_only_wrong": broken,
-        "both_wrong": missed,
-    }
-
-
-def main() -> int:
-    os.makedirs(OUT_DIR, exist_ok=True)
-    with psycopg.connect() as conn:
-        token = api_token(conn)
-        docs = sample(conn)
-        guess_floor = floor(conn)
-    log(f"label={LABEL} documents={len(docs)} floor={guess_floor:.1f}%")
-
-    rows: list[dict] = []
-    for engine, suffix in (("classic", "suggestions/"), ("ai", "ai_suggestions/")):
-        if engine in ENGINES:
-            rows.extend(run(engine, suffix, docs, token))
-
-    result = {
-        "label": LABEL,
-        "sample": len(docs),
-        "floor": round(guess_floor, 1),
-        "summaries": [summarise(rows, e) for e in ENGINES if e in ("classic", "ai")],
-        "complement": complement(rows),
-    }
-
-    with open(f"{OUT_DIR}/{LABEL}.jsonl", "w") as handle:
-        for row in rows:
-            handle.write(json.dumps(row, ensure_ascii=False) + "\n")
-    with open(f"{OUT_DIR}/{LABEL}.summary.json", "w") as handle:
-        json.dump(result, handle, ensure_ascii=False, indent=2)
-
-    # Also to stdout, so the numbers are in the pod log and reachable via Loki
-    # without mounting the volume.
-    log(json.dumps(result, ensure_ascii=False, indent=2))
-    return 0
-
-
-if __name__ == "__main__":
-    sys.exit(main())

scripts/paperless-eval/job-template.yaml

@@ -1,373 +0,0 @@
----
-# One configuration under test, complete in one pod.
-#
-# A paperless instance can only point at one chat endpoint, so measuring two
-# candidates at once means two paperless instances — which makes the pod, not
-# the namespace, the right unit. Everything a measurement needs is here except
-# the database and the embedding endpoint, both of which are read-only and
-# shared.
-#
-# The three services are native sidecars (initContainers with
-# restartPolicy: Always), not regular containers. That matters twice: they
-# start in order and each must pass its probes before the next begins, which
-# is what keeps paperless from racing the database; and when the eval
-# container exits they are torn down, so the Job actually completes instead of
-# running forever behind a llama-server that never returns.
-apiVersion: batch/v1
-kind: Job
-metadata:
-  name: eval-__LABEL__
-  namespace: paperless-eval
-  labels:
-    app.kubernetes.io/name: paperless-eval
-    app.kubernetes.io/component: eval
-spec:
-  # A failed run is re-read, not retried blind: a retry would spend another
-  # hour reproducing whatever went wrong.
-  backoffLimit: 0
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/name: paperless-eval
-        app.kubernetes.io/component: eval
-    spec:
-      restartPolicy: Never
-      enableServiceLinks: false
-      # One measurement per node. Two jobs sharing a node would share its iGPU,
-      # and every timing in the result would then describe contention rather
-      # than the model — which is exactly how a decode rate of 4.7 tokens/s got
-      # attributed to context length once already.
-      affinity:
-        podAntiAffinity:
-          requiredDuringSchedulingIgnoredDuringExecution:
-            - labelSelector:
-                matchLabels:
-                  app.kubernetes.io/component: eval
-              topologyKey: kubernetes.io/hostname
-      securityContext:
-        fsGroup: 1000
-        fsGroupChangePolicy: OnRootMismatch
-        seccompProfile:
-          type: RuntimeDefault
-      initContainers:
-        # A private copy of /data — the search index and, the expensive part,
-        # the LLM index the RAG path needs.
-        #
-        # It cannot be mounted live and shared: LLM_INDEX_DIR is hardcoded to
-        # DATA_DIR / "llm_index", the store behind it is SQLite, and SQLite
-        # locking across pods over CephFS is not dependable. Nor can it be left
-        # out — an empty llm_index makes retrieve_similar_nodes return nothing,
-        # and the job would quietly measure the no-RAG path while reporting a
-        # RAG result.
-        #
-        # ~160 MiB, so seconds. A regular initContainer: it must finish before
-        # the sidecars start.
-        - name: seed-data
-          image: ghcr.io/paperless-ngx/paperless-ngx:3.1.2
-          command:
-            - sh
-            - -c
-            - |
-              set -e
-              # -r, not -a: preserving ownership needs CHOWN, which this
-              # container does not have, and cp then exits non-zero after
-              # copying the files anyway. Running as 1000 makes the copy land
-              # with the ownership paperless needs — it writes to the LLM index
-              # while serving.
-              cp -r /golden/. /data/
-              du -sh /data/index /data/llm_index
-              # Refuse to start rather than measure the wrong thing. An empty
-              # llm_index makes retrieve_similar_nodes return nothing, and the
-              # run would report a RAG result having never retrieved anything —
-              # which is exactly what the first attempt at this job did, because
-              # the golden volume had not been populated yet. The index is
-              # ~125 MiB once built; anything under 1 MiB is not one.
-              bytes=$(du -sb /data/llm_index | cut -f1)
-              if [ "$bytes" -lt 1000000 ]; then
-                echo "FATAL: llm_index is ${bytes} bytes — build it on the" >&2
-                echo "smoke-test deployment first:" >&2
-                echo "  kubectl -n paperless-eval exec deploy/paperless -- \\" >&2
-                echo "    python manage.py document_llmindex rebuild" >&2
-                exit 1
-              fi
-          volumeMounts:
-            - name: golden
-              mountPath: /golden
-              readOnly: true
-            - name: data
-              mountPath: /data
-          securityContext:
-            runAsUser: 1000
-            runAsGroup: 1000
-            allowPrivilegeEscalation: false
-            capabilities:
-              drop: ["ALL"]
-        - name: llama-server
-          image: ghcr.io/ggml-org/llama.cpp:server-vulkan-b10731
-          restartPolicy: Always
-          args:
-            - --hf-repo
-            - __HF_REPO__
-            # The exact filename rather than the :QUANT shorthand. The
-            # shorthand matches by substring across the whole repo and repos do
-            # carry several files under one tag — ggml-org ships mmproj-, mtp-
-            # and dflash- variants at Q8_0 beside the model — so naming the file
-            # removes a real ambiguity and costs nothing.
-            #
-            # It is NOT what caused the three-hour crashloop that prompted this,
-            # and an earlier version of this comment claimed it was. The real
-            # cause was the llama-models volume being 100% full: llama-server
-            # could not write its metadata cache, so it could not resolve the
-            # repo listing, and reported
-            #     no GGUF files found in repository <repo>
-            # followed by a truncated list of what it had cached before the
-            # disk filled. That reads as a bad reference and is a full disk.
-            #
-            # If a model will not load, check `df -h /models` before believing
-            # anything llama-server says about the repository.
-            - --hf-file
-            - __HF_FILE__
-            - --alias
-            - __ALIAS__
-            - --n-gpu-layers
-            - "99"
-            - --threads
-            - "4"
-            # The production ceiling. Not because the flag value itself breaks
-            # anything — bisection showed the corruption boundary is prompt
-            # length, around 14000 tokens, whatever --ctx-size says — but
-            # because measuring against a different ceiling than production
-            # runs would not describe production.
-            - --ctx-size
-            - "16384"
-            - --parallel
-            - "1"
-            - --ubatch-size
-            - "1024"
-            # The default is -1, infinity: nothing on the server caps how long
-            # a request generates, and paperless sets no budget of its own and
-            # waits. A classification that needs ~300 tokens ran to 10269 at a
-            # healthy 11 tok/s, and three of the first eleven documents did it
-            # — 900 s each, most of an hour spent on four documents.
-            #
-            # 768 is generous against the ~300 a correct answer takes. Note
-            # that paperless reports a truncated response as a flat
-            # "Invalid AI configuration" and discards the content, so what the
-            # model was actually doing is visible only in this container's log.
-            - --n-predict
-            - "__N_PREDICT__"
-            # Without this, Qwen3.6 spends the response on reasoning_content
-            # and returns empty content — 983 tokens into a classification that
-            # needs a hundred, at a perfectly healthy 11 tok/s. It reads as a
-            # slow or broken model and is neither.
-            #
-            # NOT --chat-template-kwargs '{"enable_thinking": false}'. That is
-            # deprecated, llama-server says so at startup, and it is not the
-            # harmless no-op it looks like on a template that never reads the
-            # variable: injecting it into Mistral's render context produced
-            # token salad — "1UnComplete1Self1UnCompleteSDirect11S..." — on a
-            # 681-token prompt, while the identical request against a server
-            # without the flag returned a clean tool call. It reads exactly
-            # like a broken model and is a broken prompt.
-            - --reasoning
-            - "off"
-            # Enables POST /slots/0?action=erase, which the eval calls between
-            # documents. Without it the endpoint answers 501.
-            #
-            # This is what makes the measurements independent. A request that
-            # corrupts the slot otherwise takes every document after it, so a
-            # run reports N failures for one fault — "4 of 5 failed" was really
-            # "document 2 broke the slot". Erasing costs the prefix cache and
-            # some prefill speed, which is the right trade for numbers that
-            # mean what they say.
-            - --slot-save-path
-            - /tmp/slots
-            - --no-webui
-          env:
-            - name: LLAMA_CACHE
-              value: /models
-          ports:
-            - name: llm
-              containerPort: 8080
-          volumeMounts:
-            - name: models
-              mountPath: /models
-          # 60 minutes. Sized for a FIRST DOWNLOAD from HuggingFace, not for
-          # reading a warm PVC — an earlier version allowed 20 minutes on the
-          # latter reasoning and killed GLM-4.7-Flash mid-download at 17.8 GB.
-          # The container then restarts and begins again, so too small a budget
-          # is not "slow start", it is a loop that never finishes.
-          #
-          # The kill shows up as exitCode 137 with reason "Error" rather than
-          # OOMKilled, which is how to tell a probe kill from a memory limit.
-          startupProbe:
-            httpGet:
-              path: /health
-              port: llm
-            periodSeconds: 20
-            failureThreshold: 180
-          resources:
-            requests:
-              cpu: "1"
-              memory: __MEM_REQ__
-              gpu.intel.com/i915: "1"
-            limits:
-              memory: __MEM_LIM__
-              gpu.intel.com/i915: "1"
-          securityContext:
-            allowPrivilegeEscalation: false
-            capabilities:
-              drop: ["ALL"]
-        - name: redis
-          # Its own, not the namespace's: two paperless instances sharing a
-          # celery broker would take each other's tasks.
-          image: redis:8.10.1-alpine
-          restartPolicy: Always
-          args: ["redis-server", "--save", "", "--loglevel", "warning"]
-          startupProbe:
-            tcpSocket:
-              port: 6379
-            periodSeconds: 2
-            failureThreshold: 30
-          resources:
-            requests:
-              cpu: 20m
-              memory: 64Mi
-            limits:
-              memory: 256Mi
-          securityContext:
-            allowPrivilegeEscalation: false
-            readOnlyRootFilesystem: true
-            capabilities:
-              drop: ["ALL"]
-        - name: paperless
-          image: ghcr.io/paperless-ngx/paperless-ngx:3.1.2
-          restartPolicy: Always
-          envFrom:
-            - configMapRef:
-                name: paperless-eval-config
-          env:
-            # Overrides the namespace ConfigMap: this pod's own llama-server,
-            # and its own redis rather than the shared Service.
-            - name: PAPERLESS_AI_LLM_ENDPOINT
-              value: http://127.0.0.1:8080/v1
-            - name: PAPERLESS_AI_LLM_MODEL
-              value: __ALIAS__
-            - name: PAPERLESS_REDIS
-              value: redis://127.0.0.1:6379
-            - name: PAPERLESS_SECRET_KEY
-              valueFrom:
-                secretKeyRef:
-                  name: paperless-eval-secrets
-                  key: secret-key
-            - name: PAPERLESS_DBUSER
-              valueFrom:
-                secretKeyRef:
-                  name: pprleval-pg-app
-                  key: username
-            - name: PAPERLESS_DBPASS
-              valueFrom:
-                secretKeyRef:
-                  name: pprleval-pg-app
-                  key: password
-          ports:
-            - name: http
-              containerPort: 8000
-          volumeMounts:
-            - name: data
-              mountPath: /usr/src/paperless/data
-            - name: media
-              mountPath: /usr/src/paperless/media
-            - name: consume
-              mountPath: /usr/src/paperless/consume
-          startupProbe:
-            httpGet:
-              path: /accounts/login/
-              port: http
-            periodSeconds: 20
-            failureThreshold: 180
-          # Same as production, and required: the entrypoint starts as root,
-          # chowns the volumes and drops to uid 1000 via setgid/setuid.
-          securityContext:
-            runAsNonRoot: false
-            allowPrivilegeEscalation: true
-            capabilities:
-              drop: ["ALL"]
-              add: ["CHOWN", "DAC_OVERRIDE", "FOWNER", "SETGID", "SETUID"]
-          resources:
-            requests:
-              cpu: 200m
-              memory: 512Mi
-            limits:
-              memory: 2Gi
-      containers:
-        - name: eval
-          # The paperless image again, for psycopg and a python that matches
-          # the application — not because the script needs paperless itself.
-          image: ghcr.io/paperless-ngx/paperless-ngx:3.1.2
-          command: ["python3", "/eval/eval.py"]
-          env:
-            - name: EVAL_LABEL
-              value: __LABEL__
-            - name: EVAL_SAMPLE
-              value: "__SAMPLE__"
-            # Raised alongside --n-predict for a diagnostic run. The two bounds
-            # interact: lifting the token cap alone just moves the failure from
-            # "hit the cap" to "hit the clock", which is what happened the first
-            # time gemma-4 was retried at 4096.
-            - name: EVAL_TIMEOUT
-              value: "__TIMEOUT__"
-            - name: PGHOST
-              value: pprleval-pg-rw
-            - name: PGDATABASE
-              value: paperless
-            - name: PGUSER
-              valueFrom:
-                secretKeyRef:
-                  name: pprleval-pg-app
-                  key: username
-            - name: PGPASSWORD
-              valueFrom:
-                secretKeyRef:
-                  name: pprleval-pg-app
-                  key: password
-          volumeMounts:
-            - name: script
-              mountPath: /eval
-            - name: results
-              mountPath: /results
-          resources:
-            requests:
-              cpu: 100m
-              memory: 256Mi
-            limits:
-              memory: 512Mi
-          securityContext:
-            allowPrivilegeEscalation: false
-            capabilities:
-              drop: ["ALL"]
-      volumes:
-        - name: models
-          persistentVolumeClaim:
-            claimName: llama-models
-        - name: results
-          persistentVolumeClaim:
-            claimName: paperless-eval-results
-        - name: script
-          configMap:
-            name: paperless-eval-script
-        # The namespace's /data, mounted read-only and copied out by seed-data
-        # above. Never mounted as the live /data of a job.
-        - name: golden
-          persistentVolumeClaim:
-            claimName: paperless-eval-golden
-            readOnly: true
-        # paperless insists on all three paths existing. media and consume stay
-        # empty — nothing is consumed during a measurement — while data is
-        # seeded with the copy.
-        - name: data
-          emptyDir: {}
-        - name: media
-          emptyDir: {}
-        - name: consume
-          emptyDir: {}