fix(paperless-eval): score a right answer the matcher threw away

The AI engine reports an unresolved guess as a name in
`suggested_correspondents` rather than an id in `correspondents`, and
paperless_ai.matching only resolves it when difflib.get_close_matches
clears MATCH_THRESHOLD = 0.8. A correct but more verbose answer does not:
"Generali Deutschland Krankenversicherung AG" fails against the existing
"Generali AG", leaving `correspondents` empty.

Scored as it was, that counts as a miss, which would have blamed the
model for a matcher limit. Score both instead — the id as offered, and
the name matched wordwise against the true label — so the two causes stay
separable. The classic engine emits no `suggested_*` fields, so its two
columns coincide and the comparison stays fair either way.

Names now come out of postgres alongside the ids, tab-separated because
'|' occurs in correspondent names.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

.woodpecker/deploy-netbird-proxy.yml

@@ -10,18 +10,6 @@ when:
       # install-netbird-proxy.yml also applies the edge bouncer.
       - "ansible/roles/crowdsec-edge-bouncer/**"
       - "ansible/plays/install-netbird-proxy.yml"
-      # The variables that steer those roles, not just the roles themselves.
-      # Without these two, a change that only touches inventory never deploys
-      # and nothing says so: arming the bouncer lives in host_vars, and its
-      # LAPI key lives in group_vars, so both would sit in git looking applied
-      # while the hosts kept running the previous configuration.
-      #
-      # pve_hosts.yml is deliberately NOT listed. It is the single inventory
-      # file for the whole fleet, so listing it would redeploy the proxies for
-      # any unrelated host edit. The cost is that a change to [netbird_proxy]
-      # group membership needs a manual deployment run.
-      - "ansible/inventory/host_vars/proxy-*.yml"
-      - "ansible/inventory/group_vars/netbird_proxy/**"
       - ".woodpecker/deploy-netbird-proxy.yml"
   - event: deployment
     evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "deploy-netbird-proxy"'

ansible/inventory/host_vars/proxy-2.renner.uno.yml

@@ -1,19 +0,0 @@
----
-# Arms the CrowdSec firewall bouncer on THIS host only.
-#
-# The role defaults to dry-run for the whole netbird_proxy group, so the two
-# members can be armed one at a time. proxy-2 goes first deliberately: it is
-# the dual-stack member, while proxy-1 is IPv6-only and the one where every
-# network-shaped problem has surfaced first — the dhcpcd/resolv.conf breakage,
-# and the GitHub download that could not work at all.
-#
-# What flipping this does: the bouncer stops logging what it would block and
-# creates two nftables tables (ip crowdsec, ip6 crowdsec6) with the decisions
-# it holds. It touches no existing chain and sets no chain policy.
-#
-# Rollback is `systemctl stop crowdsec-firewall-bouncer` — a clean stop deletes
-# both tables itself. Setting this back to dry-run and re-running the play does
-# the same through the handler.
-#
-# Remove this file once proxy-1 follows and the role default moves to nftables.
-crowdsec_bouncer_mode: nftables

ansible/roles/crowdsec-edge-bouncer/tasks/main.yml

@@ -19,40 +19,13 @@
     state: directory
     mode: "0755"
 
-# Fetched on the CONTROLLER and pushed, not pulled by the host.
-#
-# proxy-1 is IPv6-only, and neither github.com nor objects.githubusercontent.com
-# publishes an AAAA record — a get_url on the host fails with "Network is
-# unreachable" and no amount of retrying fixes it. Mirroring into
-# gitea.renner.uno is no help either: its public records are the internal LB
-# VIP (10.2.99.86 / fd99::86), so it is unreachable from outside the LAN. And
-# upstream ships no container image for this bouncer, which is expected —
-# it manipulates host netfilter.
-#
-# The CI runner has v4, so it does the fetching. The checksum is verified
-# there, before anything is copied to a host.
-- name: Fetch the pinned bouncer release on the controller
+- name: Fetch the pinned bouncer release
   ansible.builtin.get_url:
     url: >-
       https://github.com/crowdsecurity/cs-firewall-bouncer/releases/download/{{
       crowdsec_bouncer_version }}/crowdsec-firewall-bouncer-linux-amd64.tgz
-    dest: /tmp/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}.tgz
-    checksum: sha256:{{ crowdsec_bouncer_sha256 }}
-    mode: "0644"
-  delegate_to: localhost
-  become: false
-  run_once: true
-
-# Only the download had to move to the controller — the extraction stays on
-# the host. The ci-ansible image ships a tar that ansible rejects ("detected as
-# tar type None. GNU tar required"), while both host distributions have GNU
-# tar. Shipping the archive rather than the unpacked binary also keeps the
-# artefact that the checksum was verified against intact all the way to the
-# host.
-- name: Copy the bouncer archive to the host
-  ansible.builtin.copy:
-    src: /tmp/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}.tgz
     dest: /opt/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}.tgz
+    checksum: sha256:{{ crowdsec_bouncer_sha256 }}
     mode: "0644"
 
 # Only the binary. The tarball also carries install.sh / upgrade.sh /

kubernetes/apps/llama-server/deployment.yaml

@@ -75,27 +75,11 @@ spec:
             # many small per-expert matmuls that sets the pace, not bandwidth.
             - --threads
             - "4"
-            # 32k because Home Assistant's Assist agent measured 18910 tokens
-            # for a single request: ~12k of exposed-entity overview plus ~7k of
-            # base instructions and tool schemas, whose parameters enumerate
-            # every exposed name, area and floor. At 16384 the request was
-            # refused outright.
-            #
-            # --ctx-size is per slot, not the total: /props reports
-            # n_ctx_slot equal to this value while running four slots, where a
-            # total would have divided down to 4096. Doubling it alone would
-            # therefore double the KV cache, so the slot count halves in step
-            # and the allocation stays where it is today — 24979 MiB of the
-            # 32Gi limit, most of it the 19.5 GiB of weights.
-            #
-            # Two slots is what is actually in use: one holds Home Assistant's
-            # prompt, one holds Paperless's. A third caller would queue rather
-            # than evict either, which is the better failure mode given a cold
-            # prefill of this prompt costs around two minutes.
+            # 16k rather than 8k because the measured prefill falloff is mild
+            # and this model's KV cache is unusually cheap — only 10 of its 40
+            # layers use full attention.
             - --ctx-size
-            - "32768"
-            - --parallel
-            - "2"
+            - "16384"
             # A micro-batch of 2048 or more makes this Vulkan/Mesa stack fail
             # with ErrorDeviceLost.
             - --ubatch-size

kubernetes/apps/llama-server/embed.yaml

@@ -41,7 +41,7 @@ spec:
           image: ghcr.io/ggml-org/llama.cpp:server-vulkan-b10731
           args:
             - --hf-repo
-            - gpustack/bge-m3-GGUF:Q8_0
+            - Qwen/Qwen3-Embedding-0.6B-GGUF:Q8_0
             # Serves POST /v1/embeddings.
             - --embeddings
             # A short, stable id for consumers. Without it the model is
@@ -50,41 +50,28 @@ spec:
             # One name only: llama-server keeps aliases in a sorted set and
             # reports the first as the id, so a second name can steal it.
             - --alias
-            - bge-m3
-            # bge-m3 is trained for CLS pooling — a different type from the
-            # Qwen family this replaces, which uses last-token. Left unspecified
-            # llama-server takes the GGUF default, which is not visible from
-            # outside and would silently produce vectors the model was not
-            # trained to emit. Read from the model's own 1_Pooling/config.json
-            # rather than assumed; of the candidates measured, three different
-            # pooling types were needed.
+            - qwen3-embedding-0.6b
+            # Qwen3-Embedding is trained for last-token pooling. Left
+            # unspecified llama-server takes the GGUF's default, which is not
+            # visible from outside and would silently produce vectors the model
+            # was not trained to emit. Set before anything is indexed: changing
+            # pooling later invalidates every stored embedding.
             - --pooling
-            - cls
+            - last
             - --n-gpu-layers
             - "99"
             - --threads
             - "2"
-            # Paperless chunks at 1024 tokens, but counts them with its own
-            # tokenizer while this server measures the batch in bge-m3's.
-            # Those disagree: XLM-RoBERTa splits German into more pieces than
-            # llama_index's counter does, so a chunk Paperless considers full
-            # at 1024 arrived here as 1069 and the request failed outright —
-            # "input (1069 tokens) is too large to process".
-            #
-            # The headroom therefore has to absorb a tokenizer ratio, not just
-            # a rounding error, and 1.5x covers far more than the 1.044 that
-            # broke it. The defaults are batch 2048 / ubatch 512, and with
-            # --embeddings llama-server clamps both to the smaller rather than
-            # run them mismatched, so both are set.
-            #
-            # 2048 stays avoided: that is where the chat model met
-            # ErrorDeviceLost on this Vulkan stack. That failure came under a
-            # 20 GB model's memory pressure and may not bind a 600 MB one, but
-            # this does not need to find out.
+            # Paperless chunks documents at 1024 tokens. The defaults are
+            # batch 2048 / ubatch 512, and with --embeddings llama-server
+            # clamps both down to 512 rather than run them mismatched — below
+            # the chunk size. Matching both to 1024 fits a whole chunk in one
+            # pass; 2048 is deliberately avoided, since that is where this
+            # Vulkan stack fails with ErrorDeviceLost.
             - --batch-size
-            - "1536"
+            - "1024"
             - --ubatch-size
-            - "1536"
+            - "1024"
             # Room for a full chunk plus overhead rather than exactly one.
             - --ctx-size
             - "4096"
@@ -104,14 +91,6 @@ spec:
             capabilities:
               drop:
                 - ALL
-          # Measured, not estimated: embedding 400 real documents peaked at
-          # 1.6 GiB (Prometheus max_over_time on the working set). The limit is
-          # well above that because the footprint is dominated by the four slots
-          # at 4096 context and the Vulkan working set rather than by the
-          # weights — the Qwen model this replaces peaked at 10.6 GiB on the
-          # same settings for the same 600 MiB of weights, and several
-          # candidates were OOMKilled at limits that looked generous against
-          # their model size.
           resources:
             requests:
               cpu: 250m
@@ -119,7 +98,7 @@ spec:
               gpu.intel.com/i915: "1"
             limits:
               cpu: "2"
-              memory: 8Gi
+              memory: 4Gi
               gpu.intel.com/i915: "1"
           # ~600 MiB rather than ~5 GiB, so a much shorter budget than the chat
           # model needs, but still sized for a cold download.

kubernetes/apps/monitoring-rules/crowdsec-servicemonitor.yaml

@@ -1,36 +0,0 @@
-# Scrapes CrowdSec. Until this landed, nothing did: `up{namespace="crowdsec"}`
-# returned zero series and Prometheus held no cs_* metric at all, so the
-# detection side of the ingress path was invisible — including whether the
-# agents were reading logs and whether any bouncer was pulling decisions.
-#
-# One monitor for both services. They carry the same shape (a `metrics` port on
-# 6060) and differ only in `app`, so a matchExpressions selector keeps this to
-# a single object rather than two near-identical ones:
-#
-#   crowdsec-service        the LAPI      — decisions, bouncer + machine requests
-#   crowdsec-agent-service  the agents    — acquisition, parsers, scenarios
-#
-# The crowdsec-ns CiliumNetworkPolicy already admits the monitoring namespace
-# without a port restriction, so no policy change is needed.
-#
-# `release: kube-prometheus-stack` is what the chart's Prometheus selects on.
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: crowdsec
-  namespace: crowdsec
-  labels:
-    app.kubernetes.io/name: crowdsec
-    release: kube-prometheus-stack
-spec:
-  selector:
-    matchExpressions:
-      - key: app
-        operator: In
-        values:
-          - crowdsec-service
-          - crowdsec-agent-service
-  endpoints:
-    - port: metrics
-      path: /metrics
-      interval: 30s

kubernetes/apps/monitoring-rules/kustomization.yaml

@@ -23,7 +23,6 @@ resources:
   - prometheusrule-cnpg-backup.yaml
   - alloy-servicemonitor.yaml
   - cnpg-podmonitors.yaml
-  - crowdsec-servicemonitor.yaml
   - home-assistant-servicemonitor.yaml
   - immich-servicemonitor.yaml
   - loki-servicemonitor.yaml

kubernetes/apps/monitoring/dashboards/crowdsec-overview.json

@@ -1,163 +0,0 @@
-{
-  "annotations": {"list": []},
-  "description": "CrowdSec: Erkennung im Cluster, Durchsetzung am NetBird-Edge",
-  "editable": true,
-  "fiscalYearStartMonth": 0,
-  "graphTooltip": 1,
-  "id": null,
-  "links": [],
-  "panels": [
-    {
-      "datasource": {"type": "prometheus", "uid": "prometheus"},
-      "description": "Aktive Ban-Entscheidungen in der LAPI, ganz überwiegend aus den CAPI-Community-Blocklisten. Das ist die Menge, die ein scharfgeschalteter Bouncer durchsetzt.",
-      "fieldConfig": {
-        "defaults": {"color": {"mode": "thresholds"}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "text", "value": null}]}, "unit": "short"},
-        "overrides": []
-      },
-      "gridPos": {"h": 4, "w": 6, "x": 0, "y": 0},
-      "id": 1,
-      "options": {"colorMode": "value", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
-      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(cs_active_decisions)", "refId": "A"}],
-      "title": "Aktive Decisions",
-      "type": "stat"
-    },
-    {
-      "datasource": {"type": "prometheus", "uid": "prometheus"},
-      "description": "Bouncer, die in den letzten 10 Minuten Decisions abgeholt haben. Fällt das unter die erwartete Zahl, holt ein Bouncer nichts mehr — und setzt damit einen veralteten Stand durch, ohne dass irgendetwas ausfällt.",
-      "fieldConfig": {
-        "defaults": {"color": {"mode": "thresholds"}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "red", "value": null}, {"color": "green", "value": 1}]}, "unit": "short"},
-        "overrides": []
-      },
-      "gridPos": {"h": 4, "w": 6, "x": 6, "y": 0},
-      "id": 2,
-      "options": {"colorMode": "value", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
-      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "count(sum by (bouncer) (rate(cs_lapi_bouncer_requests_total[10m])) > 0)", "refId": "A"}],
-      "title": "Bouncer, die abholen",
-      "type": "stat"
-    },
-    {
-      "datasource": {"type": "prometheus", "uid": "prometheus"},
-      "description": "Vom Agent gelesene Logzeilen pro Sekunde. Steht das auf 0, detektiert CrowdSec nichts mehr — der häufigste stille Ausfall, weil alles andere weiterläuft.",
-      "fieldConfig": {
-        "defaults": {"color": {"mode": "thresholds"}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "red", "value": null}, {"color": "green", "value": 0.01}]}, "unit": "reqps"},
-        "overrides": []
-      },
-      "gridPos": {"h": 4, "w": 6, "x": 12, "y": 0},
-      "id": 3,
-      "options": {"colorMode": "value", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
-      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(rate(cs_filesource_hits_total[5m]))", "refId": "A"}],
-      "title": "Gelesene Logzeilen/s",
-      "type": "stat"
-    },
-    {
-      "datasource": {"type": "prometheus", "uid": "prometheus"},
-      "description": "Szenario-Overflows der letzten 24 h — also lokal erkannte Angriffe, im Gegensatz zu den von CAPI übernommenen.",
-      "fieldConfig": {
-        "defaults": {"color": {"mode": "thresholds"}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "text", "value": null}]}, "unit": "short"},
-        "overrides": []
-      },
-      "gridPos": {"h": 4, "w": 6, "x": 18, "y": 0},
-      "id": 4,
-      "options": {"colorMode": "value", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
-      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(increase(cs_bucket_overflowed_total[24h]))", "refId": "A"}],
-      "title": "Lokale Overflows (24 h)",
-      "type": "stat"
-    },
-    {
-      "datasource": {"type": "prometheus", "uid": "prometheus"},
-      "description": "Abholrate je Bouncer. Der Firewall-Bouncer auf den Proxies fragt alle 10 s. Die LAPI legt bei jeder neuen beobachteten Client-Adresse einen Eintrag `<name>@<ip>` an, und beobachtet wird die Pod-IP des netbird-ingress-Peers — deshalb erscheint ein logischer Bouncer hier mehrfach und nach Pod-Rollen unter neuen Namen.",
-      "fieldConfig": {
-        "defaults": {"color": {"mode": "palette-classic"}, "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]}, "unit": "reqps"},
-        "overrides": []
-      },
-      "gridPos": {"h": 8, "w": 12, "x": 0, "y": 4},
-      "id": 5,
-      "options": {"legend": {"calcs": ["lastNotNull"], "displayMode": "table", "placement": "right", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
-      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum by (bouncer) (rate(cs_lapi_bouncer_requests_total[5m]))", "legendFormat": "{{bouncer}}", "refId": "A"}],
-      "title": "Bouncer-Abholrate",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {"type": "prometheus", "uid": "prometheus"},
-      "description": "Aktive Decisions nach Grund. `http:scan` dominiert und stammt aus den CAPI-Blocklisten.",
-      "fieldConfig": {
-        "defaults": {"color": {"mode": "palette-classic"}, "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 20, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "normal"}, "thresholdsStyle": {"mode": "off"}}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]}, "unit": "short"},
-        "overrides": []
-      },
-      "gridPos": {"h": 8, "w": 12, "x": 12, "y": 4},
-      "id": 6,
-      "options": {"legend": {"calcs": ["lastNotNull"], "displayMode": "table", "placement": "right", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
-      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum by (reason) (cs_active_decisions)", "legendFormat": "{{reason}}", "refId": "A"}],
-      "title": "Aktive Decisions nach Grund",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {"type": "prometheus", "uid": "prometheus"},
-      "description": "Welche Szenarien lokal auslösen. Das ist das, was CrowdSec selbst aus den Traefik-Logs erkennt.",
-      "fieldConfig": {
-        "defaults": {"color": {"mode": "palette-classic"}, "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "bars", "fillOpacity": 70, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "normal"}, "thresholdsStyle": {"mode": "off"}}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]}, "unit": "short"},
-        "overrides": []
-      },
-      "gridPos": {"h": 8, "w": 12, "x": 0, "y": 12},
-      "id": 7,
-      "options": {"legend": {"calcs": ["sum"], "displayMode": "table", "placement": "right", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
-      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum by (name) (increase(cs_bucket_overflowed_total[1h]))", "legendFormat": "{{name}}", "refId": "A"}],
-      "title": "Szenario-Overflows (je Stunde)",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {"type": "prometheus", "uid": "prometheus"},
-      "description": "Geparste gegen nicht geparste Zeilen. Ein Anstieg bei `ko` heißt meist, dass sich ein Logformat geändert hat und der Parser nicht mehr greift — CrowdSec liest dann weiter, erkennt aber nichts mehr.",
-      "fieldConfig": {
-        "defaults": {"color": {"mode": "palette-classic"}, "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]}, "unit": "reqps"},
-        "overrides": []
-      },
-      "gridPos": {"h": 8, "w": 12, "x": 12, "y": 12},
-      "id": 8,
-      "options": {"legend": {"calcs": ["lastNotNull"], "displayMode": "list", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
-      "targets": [
-        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(rate(cs_parser_hits_ok_total[5m]))", "legendFormat": "geparst", "refId": "A"},
-        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(rate(cs_parser_hits_ko_total[5m]))", "legendFormat": "nicht geparst", "refId": "B"}
-      ],
-      "title": "Parser",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {"type": "prometheus", "uid": "prometheus"},
-      "description": "Whitelist-Treffer nach Grund. Nützlich beim Scharfschalten: was hier zählt, wäre sonst ein Kandidat für einen Ban gewesen.",
-      "fieldConfig": {
-        "defaults": {"color": {"mode": "palette-classic"}, "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]}, "unit": "reqps"},
-        "overrides": []
-      },
-      "gridPos": {"h": 8, "w": 12, "x": 0, "y": 20},
-      "id": 9,
-      "options": {"legend": {"calcs": ["lastNotNull"], "displayMode": "table", "placement": "right", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
-      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum by (reason) (rate(cs_node_wl_hits_ok_total[5m]))", "legendFormat": "{{reason}}", "refId": "A"}],
-      "title": "Whitelist-Treffer",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {"type": "prometheus", "uid": "prometheus"},
-      "gridPos": {"h": 8, "w": 12, "x": 12, "y": 20},
-      "id": 10,
-      "options": {
-        "code": {"language": "plaintext", "showLineNumbers": false, "showMiniMap": false},
-        "content": "## Was dieses Dashboard NICHT zeigt\n\nAlle Panels stammen aus der LAPI und den Agents im Cluster. Die **Bouncer auf proxy-1 und proxy-2 exportieren eigene Metriken**, die hier fehlen:\n\n- `fw_bouncer_active_decisions` — was tatsächlich in den nftables-Sets steht\n- `fw_bouncer_dropped_packets` / `_bytes` — was wirklich verworfen wurde\n- `lapi_requests_total` / `_failures_total` — aus Sicht des Bouncers\n\nSie liegen auf `127.0.0.1:60601` der Proxy-Hosts. Die Hosts sind öffentlich und haben bisher keinerlei Metrik-Pfad; den Port ins Internet zu öffnen, nur um ein Scrape-Ziel zu bekommen, wäre der falsche Tausch.\n\n**Konsequenz:** „Bouncer-Abholrate\" oben zeigt, dass ein Bouncer *fragt* — nicht, dass er die Antwort auch angewandt hat. Solange der Modus `dry-run` ist, wendet er ohnehin nichts an.\n\nEin Scrape-Pfad für die Hosts ist eigene Arbeit und Voraussetzung dafür, dass „was wird wirklich geblockt\" beantwortbar wird."
-      },
-      "title": "Blinder Fleck",
-      "type": "text"
-    }
-  ],
-  "refresh": "30s",
-  "schemaVersion": 38,
-  "style": "dark",
-  "tags": ["homelab", "crowdsec", "security"],
-  "templating": {"list": []},
-  "time": {"from": "now-6h", "to": "now"},
-  "timepicker": {},
-  "timezone": "",
-  "title": "CrowdSec Overview",
-  "uid": "crowdsec-overview",
-  "version": 1,
-  "weekStart": ""
-}

kubernetes/apps/monitoring/dashboards/kustomization.yaml

@@ -38,9 +38,6 @@ configMapGenerator:
   - name: dashboard-blackbox-overview
     files:
       - blackbox-overview.json
-  - name: dashboard-crowdsec-overview
-    files:
-      - crowdsec-overview.json
   - name: dashboard-disk-health
     files:
       - disk-health.json

kubernetes/apps/paperless/paperless.yaml

@@ -82,10 +82,7 @@ data:
   # primarily English archives — the better fit for a German corpus.
   PAPERLESS_AI_LLM_EMBEDDING_BACKEND: "openai-like"
   PAPERLESS_AI_LLM_EMBEDDING_ENDPOINT: "http://llama-embed.ai.svc.cluster.local:8080/v1"
-  # Must match --alias in kubernetes/apps/llama-server/embed.yaml. Changing the
-  # embedding model invalidates every stored vector, so a swap means the LLM
-  # index has to be rebuilt.
-  PAPERLESS_AI_LLM_EMBEDDING_MODEL: "bge-m3"
+  PAPERLESS_AI_LLM_EMBEDDING_MODEL: "qwen3-embedding-0.6b"
   PAPERLESS_AI_LLM_EMBEDDING_CHUNK_SIZE: "1024"
 ---
 apiVersion: apps/v1

kubernetes/infrastructure/crowdsec/helmrelease.yaml

@@ -65,21 +65,6 @@ spec:
           - 45.157.177.26/32
           - 2a03:4000:4b:c1e:4474:43ff:fedf:8f85/128
           - 2a03:4000:28:f6e:a4c7:59ff:fe1e:9a29/128
-          # netbird.renner.uno — management, signal, relay and TURN.
-          #
-          # Not optional once a bouncer is armed. Its chain sits on the INPUT
-          # hook, the proxy hosts run no other firewall, and there is therefore
-          # no conntrack rule accepting established flows ahead of it. A ban on
-          # this address would drop the RETURN traffic of the proxies' own
-          # outbound control connection and take them off the mesh — with no
-          # inbound attack involved at all.
-          #
-          # Checked when this was added: none of these addresses appear in the
-          # current decision set. That is the state today, not a guarantee —
-          # these are rented addresses on a hoster whose ranges do end up on
-          # community blocklists.
-          - 162.55.50.97/32
-          - 2a01:4f8:c2c:4579::1/128
 
     # mTLS between agents and the LAPI. Do not switch back to password auth:
     # the LAPI runs without persistence, so its machine-registration DB is
@@ -123,24 +108,6 @@ spec:
 
     lapi:
       replicas: 1
-      service:
-        # Backend wiring for the public LAPI Ingress (lapi-ingress.yaml).
-        #
-        # These belong on the SERVICE, not on the Ingress: under the Kubernetes
-        # Ingress provider `traefik.ingress.kubernetes.io/service.*` are service
-        # annotations, and on an Ingress they are silently ignored. The symptom
-        # is Traefik speaking plaintext to a TLS-only backend, which the LAPI
-        # answers with "Client sent an HTTP request to an HTTPS server." and a
-        # 400 — an error that looks like a malformed request rather than a
-        # scheme mismatch.
-        #
-        # With tls.enabled the LAPI serves HTTPS on 8080 and offers no
-        # plaintext port, so https here is mandatory rather than a hardening
-        # choice. The ServersTransport carries the chart's own CA, since that
-        # certificate is self-signed.
-        annotations:
-          traefik.ingress.kubernetes.io/service.serversscheme: https
-          traefik.ingress.kubernetes.io/service.serverstransport: crowdsec-crowdsec-lapi@kubernetescrd
       # Far below the chart default, which reserves half a core per LAPI for a
       # workload that draws single-digit millicores — enough to leave the
       # cluster unschedulable with one node out. Only the reservation shrinks;

kubernetes/infrastructure/crowdsec/lapi-ingress.yaml

@@ -44,8 +44,8 @@ metadata:
   namespace: crowdsec
 spec:
   serverName: crowdsec-service.crowdsec.svc.cluster.local
-  rootCAs:
-    - secret: crowdsec-root-secret
+  rootCAsSecrets:
+    - crowdsec-root-secret
 ---
 apiVersion: networking.k8s.io/v1
 kind: Ingress
@@ -60,12 +60,8 @@ metadata:
     cert-manager.io/cluster-issuer: letsencrypt-cluster
     # NOT behind the Authentik gate: the client is a daemon with an API key,
     # not a browser that can complete an OIDC flow.
-    #
-    # The backend scheme and ServersTransport are NOT set here. Under the
-    # Kubernetes Ingress provider, `traefik.ingress.kubernetes.io/service.*`
-    # are SERVICE annotations — on an Ingress they are silently ignored, and
-    # Traefik then talks plaintext to a TLS-only backend. They live on the
-    # chart's Service instead, via lapi.service.annotations in helmrelease.yaml.
+    traefik.ingress.kubernetes.io/service.serversscheme: https
+    traefik.ingress.kubernetes.io/service.serverstransport: crowdsec-crowdsec-lapi@kubernetescrd
 spec:
   ingressClassName: traefik
   rules:

kubernetes/infrastructure/network-policies/ai-ns.yaml

@@ -34,17 +34,6 @@ spec:
         - ports:
             - port: "8080"
               protocol: TCP
-    # Home Assistant's llama.cpp conversation agent calls the chat model only —
-    # it has no use for the embedding endpoint, so unlike paperless above this
-    # grant does not extend to llama-embed.
-    - fromEndpoints:
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": home-assistant
-            app.kubernetes.io/name: home-assistant
-      toPorts:
-        - ports:
-            - port: "8080"
-              protocol: TCP
     - fromEndpoints:
         - matchLabels:
             "k8s:io.kubernetes.pod.namespace": monitoring

kubernetes/infrastructure/network-policies/home-assistant-ns.yaml

@@ -167,18 +167,6 @@ spec:
     - toEndpoints:
         - matchLabels:
             "k8s:io.kubernetes.pod.namespace": influxdb
-    # The cluster's own chat model, for the llama.cpp conversation agent that
-    # backs Assist. Scoped to the one pod and port rather than the ai namespace
-    # as a whole, so this grant cannot widen if something else is placed there —
-    # the embedding endpoint next to it is deliberately not included.
-    - toEndpoints:
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": ai
-            app.kubernetes.io/name: llama-server
-      toPorts:
-        - ports:
-            - port: "8080"
-              protocol: TCP
     # traefik for east-west OIDC roundtrips that go through the
     # public hostname via k8s_gateway.
     - toEndpoints:

scripts/paperless-eval.sh

@@ -0,0 +1,207 @@
+#!/usr/bin/env bash
+# End-to-end evaluation of Paperless document suggestions against the real
+# archive, comparing the two engines Paperless ships:
+#
+#   /api/documents/<id>/suggestions/      the built-in scikit-learn classifier,
+#                                         trained on the existing corpus
+#   /api/documents/<id>/ai_suggestions/   the LLM path, optionally with RAG
+#
+# WHY THIS EXISTS
+#
+# scripts/embed-eval-real.sh measures whether a document's nearest neighbours
+# carry its labels. That is a proxy: Paperless does not act on the neighbours
+# directly, it hands them to an LLM as context, alongside the document text and
+# a candidate list of existing tags and correspondents. The LLM can override
+# poor context, or fail with good context. So a better embedding score does not
+# automatically mean better suggestions, and nothing so far has verified that it
+# does.
+#
+# This measures the thing that is actually wanted: does Paperless suggest the
+# labels a human already chose? Everything else — MTEB scores, retrieval
+# accuracy — is only interesting insofar as it predicts this number.
+#
+# It also answers a question that predates the LLM work: with ten years of
+# hand-curated labels, the built-in classifier has excellent training data. It
+# cannot invent a new tag or a title, which is the LLM's advantage — but for
+# assigning labels that already exist it may simply be better. Running both on
+# the same documents settles that rather than assuming it.
+#
+# Ground truth is the current filing. Where the two disagree with it, that is
+# not automatically an error: over ten years filing conventions change, and an
+# old document may carry an older convention. Treat the absolute numbers as
+# indicative and the comparison between engines as the result.
+#
+# Requires a Paperless API token in PAPERLESS_TOKEN. The API is only reachable
+# from inside the cluster (the namespace policy admits the Authentik gate
+# outpost alone), so requests are issued from within the paperless pod.
+#
+# Usage:
+#   PAPERLESS_TOKEN=... scripts/paperless-eval.sh [-n 100] [--engine both|classic|ai]
+set -uo pipefail
+
+SAMPLE="${PAPERLESS_EVAL_SAMPLE:-100}"
+ENGINE=both
+PG_POD="${PAPERLESS_EVAL_PG_POD:-pprl-pg-2}"
+APP_POD="${PAPERLESS_EVAL_APP_POD:-deploy/paperless}"
+
+while [ $# -gt 0 ]; do
+    case "$1" in
+        -n|--sample) SAMPLE="$2"; shift 2 ;;
+        --engine)    ENGINE="$2"; shift 2 ;;
+        -h|--help)   sed -n '2,40p' "$0"; exit 0 ;;
+        *) echo "unknown argument: $1 (try --help)" >&2; exit 2 ;;
+    esac
+done
+
+if [ -z "${PAPERLESS_TOKEN:-}" ]; then
+    echo "PAPERLESS_TOKEN is not set." >&2
+    echo "Create one for an existing user, then remove it when finished." >&2
+    exit 2
+fi
+
+WORK="$(mktemp -d)"
+trap 'rm -rf "$WORK"' EXIT
+
+# Deterministic sample, so repeated runs and different engines see the same
+# documents. Only documents that carry both labels are useful as ground truth.
+echo "reading $SAMPLE labelled documents from postgres ($PG_POD)..."
+# Names come along with the ids because the AI engine reports an unresolved
+# guess as a name string rather than an id; see the lenient scoring below.
+# Tab-separated rather than '|', which occurs in correspondent names.
+kubectl -n paperless exec "$PG_POD" -- psql -U postgres -d paperless -tA -F$'\t' -c \
+  "select d.id, d.correspondent_id, d.document_type_id, c.name, t.name
+   from documents_document d
+   join documents_correspondent c on c.id = d.correspondent_id
+   join documents_documenttype  t on t.id = d.document_type_id
+   where d.correspondent_id is not null
+     and d.document_type_id is not null
+     and length(d.content) > 400
+   order by md5(d.id::text)
+   limit $SAMPLE;" > "$WORK/docs.txt" 2>"$WORK/pg-err.txt"
+if ! [ -s "$WORK/docs.txt" ]; then
+    echo "no documents returned from postgres." >&2
+    [ -s "$WORK/pg-err.txt" ] && cat "$WORK/pg-err.txt" >&2
+    exit 2
+fi
+N=$(grep -c . "$WORK/docs.txt")
+
+# The floor: always guessing the most frequent label. Without it a number like
+# "70% correct" cannot be read, because a skewed archive gives that away for
+# free — one correspondent holds several hundred of ~3900 documents.
+kubectl -n paperless exec "$PG_POD" -- psql -U postgres -d paperless -tAc \
+  "select max(c) from (select count(*) c from documents_document
+   where correspondent_id is not null group by correspondent_id) x;" \
+  > "$WORK/corr_max.txt" 2>"$WORK/pg-err.txt"
+kubectl -n paperless exec "$PG_POD" -- psql -U postgres -d paperless -tAc \
+  "select count(*) from documents_document where correspondent_id is not null;" \
+  > "$WORK/corr_total.txt" 2>"$WORK/pg-err.txt"
+
+api() { # path -> body on stdout
+    kubectl -n paperless exec "$APP_POD" -c paperless -- \
+        curl -sS -m 300 -H "Authorization: Token $PAPERLESS_TOKEN" \
+        "http://127.0.0.1:8000$1" 2>"$WORK/err.txt"
+}
+
+# A hit under two definitions, because the AI engine can be right in a way the
+# API response does not record as a suggestion:
+#
+#   strict   the true label's id appears in `.correspondents` /
+#            `.document_types`. This is what the UI actually offers, so it is
+#            the number that describes the feature as experienced.
+#
+#   lenient  strict, OR the true label's name is contained wordwise in one of
+#            `.suggested_correspondents` / `.suggested_document_types`.
+#
+# The gap between them is one specific failure: the LLM names the right entity
+# but paperless_ai.matching cannot resolve the name to the existing object,
+# because difflib.get_close_matches runs at MATCH_THRESHOLD = 0.8 and a correct
+# but more verbose answer falls below it — observed with "Generali Deutschland
+# Krankenversicherung AG" against the existing "Generali AG", which yielded an
+# empty `correspondents` list.
+#
+# Reporting only the strict number would blame the model for a matcher problem;
+# reporting only the lenient one would claim a usefulness the UI does not
+# deliver. The classic engine emits no `suggested_*` fields at all, so both
+# numbers coincide for it and the comparison stays fair in either column.
+LENIENT_JQ='
+def norm: ascii_downcase | gsub("[^\\p{L}\\p{N}]+"; " ") | split(" ") | map(select(length > 0));
+def contains_all($want): . as $have | $want | all(. as $t | $have | index([$t]) != null);
+($want_name | norm) as $w
+| ((.[$id_field] // []) | index($want_id) != null)
+  or ([(.[$name_field] // [])[] | norm] | any(contains_all($w)))
+'
+
+# score <engine> <url-suffix>
+score() {
+    local engine="$1" suffix="$2"
+    local n=0 corr_hit=0 type_hit=0 corr_len=0 type_len=0 fails=0 start elapsed
+    echo ""
+    echo "=== $engine"
+    start=$(date +%s)
+    while IFS=$'\t' read -r id want_corr want_type want_corr_name want_type_name; do
+        [ -n "$id" ] || continue
+        n=$((n + 1))
+        body=$(api "/api/documents/$id/$suffix")
+        if ! printf '%s' "$body" | jq -e . >/dev/null 2>&1; then
+            fails=$((fails + 1))
+            [ "$fails" -le 3 ] && printf '    %s: keine gueltige Antwort: %s\n' \
+                "$id" "$(printf '%s' "$body" | tr '\n' ' ' | head -c 120)" >&2
+            continue
+        fi
+        if printf '%s' "$body" | jq -e --argjson w "$want_corr" \
+               '.correspondents // [] | index($w) != null' >/dev/null 2>&1; then
+            corr_hit=$((corr_hit + 1))
+        fi
+        if printf '%s' "$body" | jq -e --argjson w "$want_type" \
+               '.document_types // [] | index($w) != null' >/dev/null 2>&1; then
+            type_hit=$((type_hit + 1))
+        fi
+        if printf '%s' "$body" | jq -e \
+               --argjson want_id "$want_corr" --arg want_name "$want_corr_name" \
+               --arg id_field correspondents --arg name_field suggested_correspondents \
+               "$LENIENT_JQ" >/dev/null 2>&1; then
+            corr_len=$((corr_len + 1))
+        fi
+        if printf '%s' "$body" | jq -e \
+               --argjson want_id "$want_type" --arg want_name "$want_type_name" \
+               --arg id_field document_types --arg name_field suggested_document_types \
+               "$LENIENT_JQ" >/dev/null 2>&1; then
+            type_len=$((type_len + 1))
+        fi
+        [ $((n % 25)) -eq 0 ] && echo "    $n/$N"
+    done < "$WORK/docs.txt"
+    elapsed=$(( $(date +%s) - start ))
+    pct() { awk -v a="$1" -v b="$2" 'BEGIN{if (b>0) printf "%.1f", 100*a/b; else print "0.0"}'; }
+    printf '    correspondent, offered  : %5s%%   recognised: %5s%%\n' \
+        "$(pct "$corr_hit" "$n")" "$(pct "$corr_len" "$n")"
+    printf '    document type, offered  : %5s%%   recognised: %5s%%\n' \
+        "$(pct "$type_hit" "$n")" "$(pct "$type_len" "$n")"
+    printf '    no usable answer        : %s of %s\n' "$fails" "$n"
+    printf '    %ss total, %.1fs per document\n' "$elapsed" "$(awk -v e="$elapsed" -v n="$n" 'BEGIN{print e/n}')"
+    printf '%-9s %7s %7s %7s %7s %6s %8ss\n' "$engine" \
+        "$(pct "$corr_hit" "$n")" "$(pct "$corr_len" "$n")" \
+        "$(pct "$type_hit" "$n")" "$(pct "$type_len" "$n")" \
+        "$fails" "$elapsed" >> "$WORK/summary.txt"
+}
+
+: > "$WORK/summary.txt"
+FLOOR=$(awk -v m="$(cat "$WORK/corr_max.txt")" -v t="$(cat "$WORK/corr_total.txt")" \
+    'BEGIN{if (t>0) printf "%.1f", 100*m/t; else print "?"}')
+echo "documents: $N   most-frequent-correspondent floor: ${FLOOR}%"
+
+case "$ENGINE" in
+    both)    score classic suggestions/; score ai ai_suggestions/ ;;
+    classic) score classic suggestions/ ;;
+    ai)      score ai ai_suggestions/ ;;
+    *) echo "unknown engine: $ENGINE" >&2; exit 2 ;;
+esac
+
+echo ""
+echo "================================================================"
+printf '%-9s %7s %7s %7s %7s %6s %9s\n' \
+    ENGINE CORR CORR+ TYPE TYPE+ FAILS TIME
+cat "$WORK/summary.txt"
+printf '%-9s %7s\n' "floor" "$FLOOR"
+echo "================================================================"
+echo "CORR/TYPE  = true label offered as a resolved id (what the UI shows)"
+echo "CORR+/TYPE+ = also counting names the LLM got right but matching dropped"