fix(crowdsec): move the LAPI backend annotations onto the Service

Both bouncers reach the LAPI and pull every 10s, and every pull gets a 400:

  2a03:4000:28:f6e:...:9a29 "GET /v1/decisions/stream?..." 400 48
  2a03:4000:4b:c1e:...:8f85 "GET /v1/decisions/stream?..." 400 48

The body says what it is: "Client sent an HTTP request to an HTTPS server."
Traefik was talking plaintext to a backend that serves TLS only.

serversscheme and serverstransport were on the Ingress. Under the Kubernetes
Ingress provider `traefik.ingress.kubernetes.io/service.*` are SERVICE
annotations; on an Ingress they are silently ignored. Nothing warns, and the
failure surfaces as a 400 that reads like a malformed request rather than a
scheme mismatch. They move to the chart's Service via lapi.service.annotations.

With tls.enabled the LAPI serves HTTPS on 8080 and offers no plaintext port,
so https is mandatory here rather than hardening.

Also switches the ServersTransport from rootCAsSecrets to rootCAs, which is
what Traefik has been warning about on every reconcile:

  RootCAsSecrets option is deprecated, please use the RootCA option instead.

Two of my own diagnostics were wrong along the way and both were measurement
artifacts. "Traefik sees no requests" came from a Loki query whose limit was
consumed by those deprecation warnings. And the source address that justified
dropping the IPAllowList was measured on the watchdog, which curls 127.0.0.1;
the bouncer resolves the public name and arrives on the proxy's public
address, as the log lines above show.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

ansible/roles/crowdsec-edge-bouncer/tasks/main.yml

@@ -19,14 +19,41 @@
     state: directory
     mode: "0755"
 
-- name: Fetch the pinned bouncer release
+# Fetched on the CONTROLLER and pushed, not pulled by the host.
+#
+# proxy-1 is IPv6-only, and neither github.com nor objects.githubusercontent.com
+# publishes an AAAA record — a get_url on the host fails with "Network is
+# unreachable" and no amount of retrying fixes it. Mirroring into
+# gitea.renner.uno is no help either: its public records are the internal LB
+# VIP (10.2.99.86 / fd99::86), so it is unreachable from outside the LAN. And
+# upstream ships no container image for this bouncer, which is expected —
+# it manipulates host netfilter.
+#
+# The CI runner has v4, so it does the fetching. The checksum is verified
+# there, before anything is copied to a host.
+- name: Fetch the pinned bouncer release on the controller
   ansible.builtin.get_url:
     url: >-
       https://github.com/crowdsecurity/cs-firewall-bouncer/releases/download/{{
       crowdsec_bouncer_version }}/crowdsec-firewall-bouncer-linux-amd64.tgz
-    dest: /opt/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}.tgz
+    dest: /tmp/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}.tgz
     checksum: sha256:{{ crowdsec_bouncer_sha256 }}
     mode: "0644"
+  delegate_to: localhost
+  become: false
+  run_once: true
+
+# Only the download had to move to the controller — the extraction stays on
+# the host. The ci-ansible image ships a tar that ansible rejects ("detected as
+# tar type None. GNU tar required"), while both host distributions have GNU
+# tar. Shipping the archive rather than the unpacked binary also keeps the
+# artefact that the checksum was verified against intact all the way to the
+# host.
+- name: Copy the bouncer archive to the host
+  ansible.builtin.copy:
+    src: /tmp/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}.tgz
+    dest: /opt/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}.tgz
+    mode: "0644"
 
 # Only the binary. The tarball also carries install.sh / upgrade.sh /
 # uninstall.sh, which manage systemd and config themselves — that is this

kubernetes/apps/llama-server/embed.yaml

@@ -41,7 +41,7 @@ spec:
           image: ghcr.io/ggml-org/llama.cpp:server-vulkan-b10731
           args:
             - --hf-repo
-            - Qwen/Qwen3-Embedding-0.6B-GGUF:Q8_0
+            - gpustack/bge-m3-GGUF:Q8_0
             # Serves POST /v1/embeddings.
             - --embeddings
             # A short, stable id for consumers. Without it the model is
@@ -50,14 +50,16 @@ spec:
             # One name only: llama-server keeps aliases in a sorted set and
             # reports the first as the id, so a second name can steal it.
             - --alias
-            - qwen3-embedding-0.6b
-            # Qwen3-Embedding is trained for last-token pooling. Left
-            # unspecified llama-server takes the GGUF's default, which is not
-            # visible from outside and would silently produce vectors the model
-            # was not trained to emit. Set before anything is indexed: changing
-            # pooling later invalidates every stored embedding.
+            - bge-m3
+            # bge-m3 is trained for CLS pooling — a different type from the
+            # Qwen family this replaces, which uses last-token. Left unspecified
+            # llama-server takes the GGUF default, which is not visible from
+            # outside and would silently produce vectors the model was not
+            # trained to emit. Read from the model's own 1_Pooling/config.json
+            # rather than assumed; of the candidates measured, three different
+            # pooling types were needed.
             - --pooling
-            - last
+            - cls
             - --n-gpu-layers
             - "99"
             - --threads
@@ -91,6 +93,14 @@ spec:
             capabilities:
               drop:
                 - ALL
+          # Measured, not estimated: embedding 400 real documents peaked at
+          # 1.6 GiB (Prometheus max_over_time on the working set). The limit is
+          # well above that because the footprint is dominated by the four slots
+          # at 4096 context and the Vulkan working set rather than by the
+          # weights — the Qwen model this replaces peaked at 10.6 GiB on the
+          # same settings for the same 600 MiB of weights, and several
+          # candidates were OOMKilled at limits that looked generous against
+          # their model size.
           resources:
             requests:
               cpu: 250m
@@ -98,7 +108,7 @@ spec:
               gpu.intel.com/i915: "1"
             limits:
               cpu: "2"
-              memory: 4Gi
+              memory: 8Gi
               gpu.intel.com/i915: "1"
           # ~600 MiB rather than ~5 GiB, so a much shorter budget than the chat
           # model needs, but still sized for a cold download.

kubernetes/apps/paperless/paperless.yaml

@@ -82,7 +82,10 @@ data:
   # primarily English archives — the better fit for a German corpus.
   PAPERLESS_AI_LLM_EMBEDDING_BACKEND: "openai-like"
   PAPERLESS_AI_LLM_EMBEDDING_ENDPOINT: "http://llama-embed.ai.svc.cluster.local:8080/v1"
-  PAPERLESS_AI_LLM_EMBEDDING_MODEL: "qwen3-embedding-0.6b"
+  # Must match --alias in kubernetes/apps/llama-server/embed.yaml. Changing the
+  # embedding model invalidates every stored vector, so a swap means the LLM
+  # index has to be rebuilt.
+  PAPERLESS_AI_LLM_EMBEDDING_MODEL: "bge-m3"
   PAPERLESS_AI_LLM_EMBEDDING_CHUNK_SIZE: "1024"
 ---
 apiVersion: apps/v1

kubernetes/infrastructure/crowdsec/helmrelease.yaml

@@ -108,6 +108,24 @@ spec:
 
     lapi:
       replicas: 1
+      service:
+        # Backend wiring for the public LAPI Ingress (lapi-ingress.yaml).
+        #
+        # These belong on the SERVICE, not on the Ingress: under the Kubernetes
+        # Ingress provider `traefik.ingress.kubernetes.io/service.*` are service
+        # annotations, and on an Ingress they are silently ignored. The symptom
+        # is Traefik speaking plaintext to a TLS-only backend, which the LAPI
+        # answers with "Client sent an HTTP request to an HTTPS server." and a
+        # 400 — an error that looks like a malformed request rather than a
+        # scheme mismatch.
+        #
+        # With tls.enabled the LAPI serves HTTPS on 8080 and offers no
+        # plaintext port, so https here is mandatory rather than a hardening
+        # choice. The ServersTransport carries the chart's own CA, since that
+        # certificate is self-signed.
+        annotations:
+          traefik.ingress.kubernetes.io/service.serversscheme: https
+          traefik.ingress.kubernetes.io/service.serverstransport: crowdsec-crowdsec-lapi@kubernetescrd
       # Far below the chart default, which reserves half a core per LAPI for a
       # workload that draws single-digit millicores — enough to leave the
       # cluster unschedulable with one node out. Only the reservation shrinks;

kubernetes/infrastructure/crowdsec/lapi-ingress.yaml

@@ -44,8 +44,8 @@ metadata:
   namespace: crowdsec
 spec:
   serverName: crowdsec-service.crowdsec.svc.cluster.local
-  rootCAsSecrets:
-    - crowdsec-root-secret
+  rootCAs:
+    - secret: crowdsec-root-secret
 ---
 apiVersion: networking.k8s.io/v1
 kind: Ingress
@@ -60,8 +60,12 @@ metadata:
     cert-manager.io/cluster-issuer: letsencrypt-cluster
     # NOT behind the Authentik gate: the client is a daemon with an API key,
     # not a browser that can complete an OIDC flow.
-    traefik.ingress.kubernetes.io/service.serversscheme: https
-    traefik.ingress.kubernetes.io/service.serverstransport: crowdsec-crowdsec-lapi@kubernetescrd
+    #
+    # The backend scheme and ServersTransport are NOT set here. Under the
+    # Kubernetes Ingress provider, `traefik.ingress.kubernetes.io/service.*`
+    # are SERVICE annotations — on an Ingress they are silently ignored, and
+    # Traefik then talks plaintext to a TLS-only backend. They live on the
+    # chart's Service instead, via lapi.service.annotations in helmrelease.yaml.
 spec:
   ingressClassName: traefik
   rules:

scripts/paperless-eval.sh

@@ -1,207 +0,0 @@
-#!/usr/bin/env bash
-# End-to-end evaluation of Paperless document suggestions against the real
-# archive, comparing the two engines Paperless ships:
-#
-#   /api/documents/<id>/suggestions/      the built-in scikit-learn classifier,
-#                                         trained on the existing corpus
-#   /api/documents/<id>/ai_suggestions/   the LLM path, optionally with RAG
-#
-# WHY THIS EXISTS
-#
-# scripts/embed-eval-real.sh measures whether a document's nearest neighbours
-# carry its labels. That is a proxy: Paperless does not act on the neighbours
-# directly, it hands them to an LLM as context, alongside the document text and
-# a candidate list of existing tags and correspondents. The LLM can override
-# poor context, or fail with good context. So a better embedding score does not
-# automatically mean better suggestions, and nothing so far has verified that it
-# does.
-#
-# This measures the thing that is actually wanted: does Paperless suggest the
-# labels a human already chose? Everything else — MTEB scores, retrieval
-# accuracy — is only interesting insofar as it predicts this number.
-#
-# It also answers a question that predates the LLM work: with ten years of
-# hand-curated labels, the built-in classifier has excellent training data. It
-# cannot invent a new tag or a title, which is the LLM's advantage — but for
-# assigning labels that already exist it may simply be better. Running both on
-# the same documents settles that rather than assuming it.
-#
-# Ground truth is the current filing. Where the two disagree with it, that is
-# not automatically an error: over ten years filing conventions change, and an
-# old document may carry an older convention. Treat the absolute numbers as
-# indicative and the comparison between engines as the result.
-#
-# Requires a Paperless API token in PAPERLESS_TOKEN. The API is only reachable
-# from inside the cluster (the namespace policy admits the Authentik gate
-# outpost alone), so requests are issued from within the paperless pod.
-#
-# Usage:
-#   PAPERLESS_TOKEN=... scripts/paperless-eval.sh [-n 100] [--engine both|classic|ai]
-set -uo pipefail
-
-SAMPLE="${PAPERLESS_EVAL_SAMPLE:-100}"
-ENGINE=both
-PG_POD="${PAPERLESS_EVAL_PG_POD:-pprl-pg-2}"
-APP_POD="${PAPERLESS_EVAL_APP_POD:-deploy/paperless}"
-
-while [ $# -gt 0 ]; do
-    case "$1" in
-        -n|--sample) SAMPLE="$2"; shift 2 ;;
-        --engine)    ENGINE="$2"; shift 2 ;;
-        -h|--help)   sed -n '2,40p' "$0"; exit 0 ;;
-        *) echo "unknown argument: $1 (try --help)" >&2; exit 2 ;;
-    esac
-done
-
-if [ -z "${PAPERLESS_TOKEN:-}" ]; then
-    echo "PAPERLESS_TOKEN is not set." >&2
-    echo "Create one for an existing user, then remove it when finished." >&2
-    exit 2
-fi
-
-WORK="$(mktemp -d)"
-trap 'rm -rf "$WORK"' EXIT
-
-# Deterministic sample, so repeated runs and different engines see the same
-# documents. Only documents that carry both labels are useful as ground truth.
-echo "reading $SAMPLE labelled documents from postgres ($PG_POD)..."
-# Names come along with the ids because the AI engine reports an unresolved
-# guess as a name string rather than an id; see the lenient scoring below.
-# Tab-separated rather than '|', which occurs in correspondent names.
-kubectl -n paperless exec "$PG_POD" -- psql -U postgres -d paperless -tA -F$'\t' -c \
-  "select d.id, d.correspondent_id, d.document_type_id, c.name, t.name
-   from documents_document d
-   join documents_correspondent c on c.id = d.correspondent_id
-   join documents_documenttype  t on t.id = d.document_type_id
-   where d.correspondent_id is not null
-     and d.document_type_id is not null
-     and length(d.content) > 400
-   order by md5(d.id::text)
-   limit $SAMPLE;" > "$WORK/docs.txt" 2>"$WORK/pg-err.txt"
-if ! [ -s "$WORK/docs.txt" ]; then
-    echo "no documents returned from postgres." >&2
-    [ -s "$WORK/pg-err.txt" ] && cat "$WORK/pg-err.txt" >&2
-    exit 2
-fi
-N=$(grep -c . "$WORK/docs.txt")
-
-# The floor: always guessing the most frequent label. Without it a number like
-# "70% correct" cannot be read, because a skewed archive gives that away for
-# free — one correspondent holds several hundred of ~3900 documents.
-kubectl -n paperless exec "$PG_POD" -- psql -U postgres -d paperless -tAc \
-  "select max(c) from (select count(*) c from documents_document
-   where correspondent_id is not null group by correspondent_id) x;" \
-  > "$WORK/corr_max.txt" 2>"$WORK/pg-err.txt"
-kubectl -n paperless exec "$PG_POD" -- psql -U postgres -d paperless -tAc \
-  "select count(*) from documents_document where correspondent_id is not null;" \
-  > "$WORK/corr_total.txt" 2>"$WORK/pg-err.txt"
-
-api() { # path -> body on stdout
-    kubectl -n paperless exec "$APP_POD" -c paperless -- \
-        curl -sS -m 300 -H "Authorization: Token $PAPERLESS_TOKEN" \
-        "http://127.0.0.1:8000$1" 2>"$WORK/err.txt"
-}
-
-# A hit under two definitions, because the AI engine can be right in a way the
-# API response does not record as a suggestion:
-#
-#   strict   the true label's id appears in `.correspondents` /
-#            `.document_types`. This is what the UI actually offers, so it is
-#            the number that describes the feature as experienced.
-#
-#   lenient  strict, OR the true label's name is contained wordwise in one of
-#            `.suggested_correspondents` / `.suggested_document_types`.
-#
-# The gap between them is one specific failure: the LLM names the right entity
-# but paperless_ai.matching cannot resolve the name to the existing object,
-# because difflib.get_close_matches runs at MATCH_THRESHOLD = 0.8 and a correct
-# but more verbose answer falls below it — observed with "Generali Deutschland
-# Krankenversicherung AG" against the existing "Generali AG", which yielded an
-# empty `correspondents` list.
-#
-# Reporting only the strict number would blame the model for a matcher problem;
-# reporting only the lenient one would claim a usefulness the UI does not
-# deliver. The classic engine emits no `suggested_*` fields at all, so both
-# numbers coincide for it and the comparison stays fair in either column.
-LENIENT_JQ='
-def norm: ascii_downcase | gsub("[^\\p{L}\\p{N}]+"; " ") | split(" ") | map(select(length > 0));
-def contains_all($want): . as $have | $want | all(. as $t | $have | index([$t]) != null);
-($want_name | norm) as $w
-| ((.[$id_field] // []) | index($want_id) != null)
-  or ([(.[$name_field] // [])[] | norm] | any(contains_all($w)))
-'
-
-# score <engine> <url-suffix>
-score() {
-    local engine="$1" suffix="$2"
-    local n=0 corr_hit=0 type_hit=0 corr_len=0 type_len=0 fails=0 start elapsed
-    echo ""
-    echo "=== $engine"
-    start=$(date +%s)
-    while IFS=$'\t' read -r id want_corr want_type want_corr_name want_type_name; do
-        [ -n "$id" ] || continue
-        n=$((n + 1))
-        body=$(api "/api/documents/$id/$suffix")
-        if ! printf '%s' "$body" | jq -e . >/dev/null 2>&1; then
-            fails=$((fails + 1))
-            [ "$fails" -le 3 ] && printf '    %s: keine gueltige Antwort: %s\n' \
-                "$id" "$(printf '%s' "$body" | tr '\n' ' ' | head -c 120)" >&2
-            continue
-        fi
-        if printf '%s' "$body" | jq -e --argjson w "$want_corr" \
-               '.correspondents // [] | index($w) != null' >/dev/null 2>&1; then
-            corr_hit=$((corr_hit + 1))
-        fi
-        if printf '%s' "$body" | jq -e --argjson w "$want_type" \
-               '.document_types // [] | index($w) != null' >/dev/null 2>&1; then
-            type_hit=$((type_hit + 1))
-        fi
-        if printf '%s' "$body" | jq -e \
-               --argjson want_id "$want_corr" --arg want_name "$want_corr_name" \
-               --arg id_field correspondents --arg name_field suggested_correspondents \
-               "$LENIENT_JQ" >/dev/null 2>&1; then
-            corr_len=$((corr_len + 1))
-        fi
-        if printf '%s' "$body" | jq -e \
-               --argjson want_id "$want_type" --arg want_name "$want_type_name" \
-               --arg id_field document_types --arg name_field suggested_document_types \
-               "$LENIENT_JQ" >/dev/null 2>&1; then
-            type_len=$((type_len + 1))
-        fi
-        [ $((n % 25)) -eq 0 ] && echo "    $n/$N"
-    done < "$WORK/docs.txt"
-    elapsed=$(( $(date +%s) - start ))
-    pct() { awk -v a="$1" -v b="$2" 'BEGIN{if (b>0) printf "%.1f", 100*a/b; else print "0.0"}'; }
-    printf '    correspondent, offered  : %5s%%   recognised: %5s%%\n' \
-        "$(pct "$corr_hit" "$n")" "$(pct "$corr_len" "$n")"
-    printf '    document type, offered  : %5s%%   recognised: %5s%%\n' \
-        "$(pct "$type_hit" "$n")" "$(pct "$type_len" "$n")"
-    printf '    no usable answer        : %s of %s\n' "$fails" "$n"
-    printf '    %ss total, %.1fs per document\n' "$elapsed" "$(awk -v e="$elapsed" -v n="$n" 'BEGIN{print e/n}')"
-    printf '%-9s %7s %7s %7s %7s %6s %8ss\n' "$engine" \
-        "$(pct "$corr_hit" "$n")" "$(pct "$corr_len" "$n")" \
-        "$(pct "$type_hit" "$n")" "$(pct "$type_len" "$n")" \
-        "$fails" "$elapsed" >> "$WORK/summary.txt"
-}
-
-: > "$WORK/summary.txt"
-FLOOR=$(awk -v m="$(cat "$WORK/corr_max.txt")" -v t="$(cat "$WORK/corr_total.txt")" \
-    'BEGIN{if (t>0) printf "%.1f", 100*m/t; else print "?"}')
-echo "documents: $N   most-frequent-correspondent floor: ${FLOOR}%"
-
-case "$ENGINE" in
-    both)    score classic suggestions/; score ai ai_suggestions/ ;;
-    classic) score classic suggestions/ ;;
-    ai)      score ai ai_suggestions/ ;;
-    *) echo "unknown engine: $ENGINE" >&2; exit 2 ;;
-esac
-
-echo ""
-echo "================================================================"
-printf '%-9s %7s %7s %7s %7s %6s %9s\n' \
-    ENGINE CORR CORR+ TYPE TYPE+ FAILS TIME
-cat "$WORK/summary.txt"
-printf '%-9s %7s\n' "floor" "$FLOOR"
-echo "================================================================"
-echo "CORR/TYPE  = true label offered as a resolved id (what the UI shows)"
-echo "CORR+/TYPE+ = also counting names the LLM got right but matching dropped"