fix(llama-embed): size the batch for bge-m3's tokenizer, not Paperless's

The LLM index rebuild ran for 79 minutes and then died:

    input (1069 tokens) is too large to process.
    increase the physical batch size (current batch size: 1024)

Setting the batch to Paperless's chunk size looked like an exact match
and was not. Paperless counts its 1024 tokens with llama_index's
tokenizer; this server measures the batch in bge-m3's, which is
XLM-RoBERTa and splits German into more pieces. A chunk Paperless
considered full arrived here 4.4% over the limit, and the request failed
rather than truncating.

1536 absorbs a tokenizer ratio rather than a rounding error. 2048 stays
out of reach: that is where the chat model hit ErrorDeviceLost on this
Vulkan stack, under a 20 GB model's memory pressure that a 600 MB one may
well not reproduce — but there is no reason to go looking.

The index is left partial by the failed run and has to be rebuilt with
`document_llmindex rebuild` once this is live.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

ansible/roles/crowdsec-edge-bouncer/tasks/main.yml

@@ -19,14 +19,41 @@
     state: directory
     mode: "0755"
 
-- name: Fetch the pinned bouncer release
+# Fetched on the CONTROLLER and pushed, not pulled by the host.
+#
+# proxy-1 is IPv6-only, and neither github.com nor objects.githubusercontent.com
+# publishes an AAAA record — a get_url on the host fails with "Network is
+# unreachable" and no amount of retrying fixes it. Mirroring into
+# gitea.renner.uno is no help either: its public records are the internal LB
+# VIP (10.2.99.86 / fd99::86), so it is unreachable from outside the LAN. And
+# upstream ships no container image for this bouncer, which is expected —
+# it manipulates host netfilter.
+#
+# The CI runner has v4, so it does the fetching. The checksum is verified
+# there, before anything is copied to a host.
+- name: Fetch the pinned bouncer release on the controller
   ansible.builtin.get_url:
     url: >-
       https://github.com/crowdsecurity/cs-firewall-bouncer/releases/download/{{
       crowdsec_bouncer_version }}/crowdsec-firewall-bouncer-linux-amd64.tgz
-    dest: /opt/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}.tgz
+    dest: /tmp/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}.tgz
     checksum: sha256:{{ crowdsec_bouncer_sha256 }}
     mode: "0644"
+  delegate_to: localhost
+  become: false
+  run_once: true
+
+# Only the download had to move to the controller — the extraction stays on
+# the host. The ci-ansible image ships a tar that ansible rejects ("detected as
+# tar type None. GNU tar required"), while both host distributions have GNU
+# tar. Shipping the archive rather than the unpacked binary also keeps the
+# artefact that the checksum was verified against intact all the way to the
+# host.
+- name: Copy the bouncer archive to the host
+  ansible.builtin.copy:
+    src: /tmp/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}.tgz
+    dest: /opt/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}.tgz
+    mode: "0644"
 
 # Only the binary. The tarball also carries install.sh / upgrade.sh /
 # uninstall.sh, which manage systemd and config themselves — that is this

kubernetes/apps/llama-server/embed.yaml

@@ -41,7 +41,7 @@ spec:
           image: ghcr.io/ggml-org/llama.cpp:server-vulkan-b10731
           args:
             - --hf-repo
-            - Qwen/Qwen3-Embedding-0.6B-GGUF:Q8_0
+            - gpustack/bge-m3-GGUF:Q8_0
             # Serves POST /v1/embeddings.
             - --embeddings
             # A short, stable id for consumers. Without it the model is
@@ -50,28 +50,41 @@ spec:
             # One name only: llama-server keeps aliases in a sorted set and
             # reports the first as the id, so a second name can steal it.
             - --alias
-            - qwen3-embedding-0.6b
-            # Qwen3-Embedding is trained for last-token pooling. Left
-            # unspecified llama-server takes the GGUF's default, which is not
-            # visible from outside and would silently produce vectors the model
-            # was not trained to emit. Set before anything is indexed: changing
-            # pooling later invalidates every stored embedding.
+            - bge-m3
+            # bge-m3 is trained for CLS pooling — a different type from the
+            # Qwen family this replaces, which uses last-token. Left unspecified
+            # llama-server takes the GGUF default, which is not visible from
+            # outside and would silently produce vectors the model was not
+            # trained to emit. Read from the model's own 1_Pooling/config.json
+            # rather than assumed; of the candidates measured, three different
+            # pooling types were needed.
             - --pooling
-            - last
+            - cls
             - --n-gpu-layers
             - "99"
             - --threads
             - "2"
-            # Paperless chunks documents at 1024 tokens. The defaults are
-            # batch 2048 / ubatch 512, and with --embeddings llama-server
-            # clamps both down to 512 rather than run them mismatched — below
-            # the chunk size. Matching both to 1024 fits a whole chunk in one
-            # pass; 2048 is deliberately avoided, since that is where this
-            # Vulkan stack fails with ErrorDeviceLost.
+            # Paperless chunks at 1024 tokens, but counts them with its own
+            # tokenizer while this server measures the batch in bge-m3's.
+            # Those disagree: XLM-RoBERTa splits German into more pieces than
+            # llama_index's counter does, so a chunk Paperless considers full
+            # at 1024 arrived here as 1069 and the request failed outright —
+            # "input (1069 tokens) is too large to process".
+            #
+            # The headroom therefore has to absorb a tokenizer ratio, not just
+            # a rounding error, and 1.5x covers far more than the 1.044 that
+            # broke it. The defaults are batch 2048 / ubatch 512, and with
+            # --embeddings llama-server clamps both to the smaller rather than
+            # run them mismatched, so both are set.
+            #
+            # 2048 stays avoided: that is where the chat model met
+            # ErrorDeviceLost on this Vulkan stack. That failure came under a
+            # 20 GB model's memory pressure and may not bind a 600 MB one, but
+            # this does not need to find out.
             - --batch-size
-            - "1024"
+            - "1536"
             - --ubatch-size
-            - "1024"
+            - "1536"
             # Room for a full chunk plus overhead rather than exactly one.
             - --ctx-size
             - "4096"
@@ -91,6 +104,14 @@ spec:
             capabilities:
               drop:
                 - ALL
+          # Measured, not estimated: embedding 400 real documents peaked at
+          # 1.6 GiB (Prometheus max_over_time on the working set). The limit is
+          # well above that because the footprint is dominated by the four slots
+          # at 4096 context and the Vulkan working set rather than by the
+          # weights — the Qwen model this replaces peaked at 10.6 GiB on the
+          # same settings for the same 600 MiB of weights, and several
+          # candidates were OOMKilled at limits that looked generous against
+          # their model size.
           resources:
             requests:
               cpu: 250m
@@ -98,7 +119,7 @@ spec:
               gpu.intel.com/i915: "1"
             limits:
               cpu: "2"
-              memory: 4Gi
+              memory: 8Gi
               gpu.intel.com/i915: "1"
           # ~600 MiB rather than ~5 GiB, so a much shorter budget than the chat
           # model needs, but still sized for a cold download.

kubernetes/apps/paperless/paperless.yaml

@@ -82,7 +82,10 @@ data:
   # primarily English archives — the better fit for a German corpus.
   PAPERLESS_AI_LLM_EMBEDDING_BACKEND: "openai-like"
   PAPERLESS_AI_LLM_EMBEDDING_ENDPOINT: "http://llama-embed.ai.svc.cluster.local:8080/v1"
-  PAPERLESS_AI_LLM_EMBEDDING_MODEL: "qwen3-embedding-0.6b"
+  # Must match --alias in kubernetes/apps/llama-server/embed.yaml. Changing the
+  # embedding model invalidates every stored vector, so a swap means the LLM
+  # index has to be rebuilt.
+  PAPERLESS_AI_LLM_EMBEDDING_MODEL: "bge-m3"
   PAPERLESS_AI_LLM_EMBEDDING_CHUNK_SIZE: "1024"
 ---
 apiVersion: apps/v1

scripts/paperless-eval.sh

@@ -1,207 +0,0 @@
-#!/usr/bin/env bash
-# End-to-end evaluation of Paperless document suggestions against the real
-# archive, comparing the two engines Paperless ships:
-#
-#   /api/documents/<id>/suggestions/      the built-in scikit-learn classifier,
-#                                         trained on the existing corpus
-#   /api/documents/<id>/ai_suggestions/   the LLM path, optionally with RAG
-#
-# WHY THIS EXISTS
-#
-# scripts/embed-eval-real.sh measures whether a document's nearest neighbours
-# carry its labels. That is a proxy: Paperless does not act on the neighbours
-# directly, it hands them to an LLM as context, alongside the document text and
-# a candidate list of existing tags and correspondents. The LLM can override
-# poor context, or fail with good context. So a better embedding score does not
-# automatically mean better suggestions, and nothing so far has verified that it
-# does.
-#
-# This measures the thing that is actually wanted: does Paperless suggest the
-# labels a human already chose? Everything else — MTEB scores, retrieval
-# accuracy — is only interesting insofar as it predicts this number.
-#
-# It also answers a question that predates the LLM work: with ten years of
-# hand-curated labels, the built-in classifier has excellent training data. It
-# cannot invent a new tag or a title, which is the LLM's advantage — but for
-# assigning labels that already exist it may simply be better. Running both on
-# the same documents settles that rather than assuming it.
-#
-# Ground truth is the current filing. Where the two disagree with it, that is
-# not automatically an error: over ten years filing conventions change, and an
-# old document may carry an older convention. Treat the absolute numbers as
-# indicative and the comparison between engines as the result.
-#
-# Requires a Paperless API token in PAPERLESS_TOKEN. The API is only reachable
-# from inside the cluster (the namespace policy admits the Authentik gate
-# outpost alone), so requests are issued from within the paperless pod.
-#
-# Usage:
-#   PAPERLESS_TOKEN=... scripts/paperless-eval.sh [-n 100] [--engine both|classic|ai]
-set -uo pipefail
-
-SAMPLE="${PAPERLESS_EVAL_SAMPLE:-100}"
-ENGINE=both
-PG_POD="${PAPERLESS_EVAL_PG_POD:-pprl-pg-2}"
-APP_POD="${PAPERLESS_EVAL_APP_POD:-deploy/paperless}"
-
-while [ $# -gt 0 ]; do
-    case "$1" in
-        -n|--sample) SAMPLE="$2"; shift 2 ;;
-        --engine)    ENGINE="$2"; shift 2 ;;
-        -h|--help)   sed -n '2,40p' "$0"; exit 0 ;;
-        *) echo "unknown argument: $1 (try --help)" >&2; exit 2 ;;
-    esac
-done
-
-if [ -z "${PAPERLESS_TOKEN:-}" ]; then
-    echo "PAPERLESS_TOKEN is not set." >&2
-    echo "Create one for an existing user, then remove it when finished." >&2
-    exit 2
-fi
-
-WORK="$(mktemp -d)"
-trap 'rm -rf "$WORK"' EXIT
-
-# Deterministic sample, so repeated runs and different engines see the same
-# documents. Only documents that carry both labels are useful as ground truth.
-echo "reading $SAMPLE labelled documents from postgres ($PG_POD)..."
-# Names come along with the ids because the AI engine reports an unresolved
-# guess as a name string rather than an id; see the lenient scoring below.
-# Tab-separated rather than '|', which occurs in correspondent names.
-kubectl -n paperless exec "$PG_POD" -- psql -U postgres -d paperless -tA -F$'\t' -c \
-  "select d.id, d.correspondent_id, d.document_type_id, c.name, t.name
-   from documents_document d
-   join documents_correspondent c on c.id = d.correspondent_id
-   join documents_documenttype  t on t.id = d.document_type_id
-   where d.correspondent_id is not null
-     and d.document_type_id is not null
-     and length(d.content) > 400
-   order by md5(d.id::text)
-   limit $SAMPLE;" > "$WORK/docs.txt" 2>"$WORK/pg-err.txt"
-if ! [ -s "$WORK/docs.txt" ]; then
-    echo "no documents returned from postgres." >&2
-    [ -s "$WORK/pg-err.txt" ] && cat "$WORK/pg-err.txt" >&2
-    exit 2
-fi
-N=$(grep -c . "$WORK/docs.txt")
-
-# The floor: always guessing the most frequent label. Without it a number like
-# "70% correct" cannot be read, because a skewed archive gives that away for
-# free — one correspondent holds several hundred of ~3900 documents.
-kubectl -n paperless exec "$PG_POD" -- psql -U postgres -d paperless -tAc \
-  "select max(c) from (select count(*) c from documents_document
-   where correspondent_id is not null group by correspondent_id) x;" \
-  > "$WORK/corr_max.txt" 2>"$WORK/pg-err.txt"
-kubectl -n paperless exec "$PG_POD" -- psql -U postgres -d paperless -tAc \
-  "select count(*) from documents_document where correspondent_id is not null;" \
-  > "$WORK/corr_total.txt" 2>"$WORK/pg-err.txt"
-
-api() { # path -> body on stdout
-    kubectl -n paperless exec "$APP_POD" -c paperless -- \
-        curl -sS -m 300 -H "Authorization: Token $PAPERLESS_TOKEN" \
-        "http://127.0.0.1:8000$1" 2>"$WORK/err.txt"
-}
-
-# A hit under two definitions, because the AI engine can be right in a way the
-# API response does not record as a suggestion:
-#
-#   strict   the true label's id appears in `.correspondents` /
-#            `.document_types`. This is what the UI actually offers, so it is
-#            the number that describes the feature as experienced.
-#
-#   lenient  strict, OR the true label's name is contained wordwise in one of
-#            `.suggested_correspondents` / `.suggested_document_types`.
-#
-# The gap between them is one specific failure: the LLM names the right entity
-# but paperless_ai.matching cannot resolve the name to the existing object,
-# because difflib.get_close_matches runs at MATCH_THRESHOLD = 0.8 and a correct
-# but more verbose answer falls below it — observed with "Generali Deutschland
-# Krankenversicherung AG" against the existing "Generali AG", which yielded an
-# empty `correspondents` list.
-#
-# Reporting only the strict number would blame the model for a matcher problem;
-# reporting only the lenient one would claim a usefulness the UI does not
-# deliver. The classic engine emits no `suggested_*` fields at all, so both
-# numbers coincide for it and the comparison stays fair in either column.
-LENIENT_JQ='
-def norm: ascii_downcase | gsub("[^\\p{L}\\p{N}]+"; " ") | split(" ") | map(select(length > 0));
-def contains_all($want): . as $have | $want | all(. as $t | $have | index([$t]) != null);
-($want_name | norm) as $w
-| ((.[$id_field] // []) | index($want_id) != null)
-  or ([(.[$name_field] // [])[] | norm] | any(contains_all($w)))
-'
-
-# score <engine> <url-suffix>
-score() {
-    local engine="$1" suffix="$2"
-    local n=0 corr_hit=0 type_hit=0 corr_len=0 type_len=0 fails=0 start elapsed
-    echo ""
-    echo "=== $engine"
-    start=$(date +%s)
-    while IFS=$'\t' read -r id want_corr want_type want_corr_name want_type_name; do
-        [ -n "$id" ] || continue
-        n=$((n + 1))
-        body=$(api "/api/documents/$id/$suffix")
-        if ! printf '%s' "$body" | jq -e . >/dev/null 2>&1; then
-            fails=$((fails + 1))
-            [ "$fails" -le 3 ] && printf '    %s: keine gueltige Antwort: %s\n' \
-                "$id" "$(printf '%s' "$body" | tr '\n' ' ' | head -c 120)" >&2
-            continue
-        fi
-        if printf '%s' "$body" | jq -e --argjson w "$want_corr" \
-               '.correspondents // [] | index($w) != null' >/dev/null 2>&1; then
-            corr_hit=$((corr_hit + 1))
-        fi
-        if printf '%s' "$body" | jq -e --argjson w "$want_type" \
-               '.document_types // [] | index($w) != null' >/dev/null 2>&1; then
-            type_hit=$((type_hit + 1))
-        fi
-        if printf '%s' "$body" | jq -e \
-               --argjson want_id "$want_corr" --arg want_name "$want_corr_name" \
-               --arg id_field correspondents --arg name_field suggested_correspondents \
-               "$LENIENT_JQ" >/dev/null 2>&1; then
-            corr_len=$((corr_len + 1))
-        fi
-        if printf '%s' "$body" | jq -e \
-               --argjson want_id "$want_type" --arg want_name "$want_type_name" \
-               --arg id_field document_types --arg name_field suggested_document_types \
-               "$LENIENT_JQ" >/dev/null 2>&1; then
-            type_len=$((type_len + 1))
-        fi
-        [ $((n % 25)) -eq 0 ] && echo "    $n/$N"
-    done < "$WORK/docs.txt"
-    elapsed=$(( $(date +%s) - start ))
-    pct() { awk -v a="$1" -v b="$2" 'BEGIN{if (b>0) printf "%.1f", 100*a/b; else print "0.0"}'; }
-    printf '    correspondent, offered  : %5s%%   recognised: %5s%%\n' \
-        "$(pct "$corr_hit" "$n")" "$(pct "$corr_len" "$n")"
-    printf '    document type, offered  : %5s%%   recognised: %5s%%\n' \
-        "$(pct "$type_hit" "$n")" "$(pct "$type_len" "$n")"
-    printf '    no usable answer        : %s of %s\n' "$fails" "$n"
-    printf '    %ss total, %.1fs per document\n' "$elapsed" "$(awk -v e="$elapsed" -v n="$n" 'BEGIN{print e/n}')"
-    printf '%-9s %7s %7s %7s %7s %6s %8ss\n' "$engine" \
-        "$(pct "$corr_hit" "$n")" "$(pct "$corr_len" "$n")" \
-        "$(pct "$type_hit" "$n")" "$(pct "$type_len" "$n")" \
-        "$fails" "$elapsed" >> "$WORK/summary.txt"
-}
-
-: > "$WORK/summary.txt"
-FLOOR=$(awk -v m="$(cat "$WORK/corr_max.txt")" -v t="$(cat "$WORK/corr_total.txt")" \
-    'BEGIN{if (t>0) printf "%.1f", 100*m/t; else print "?"}')
-echo "documents: $N   most-frequent-correspondent floor: ${FLOOR}%"
-
-case "$ENGINE" in
-    both)    score classic suggestions/; score ai ai_suggestions/ ;;
-    classic) score classic suggestions/ ;;
-    ai)      score ai ai_suggestions/ ;;
-    *) echo "unknown engine: $ENGINE" >&2; exit 2 ;;
-esac
-
-echo ""
-echo "================================================================"
-printf '%-9s %7s %7s %7s %7s %6s %9s\n' \
-    ENGINE CORR CORR+ TYPE TYPE+ FAILS TIME
-cat "$WORK/summary.txt"
-printf '%-9s %7s\n' "floor" "$FLOOR"
-echo "================================================================"
-echo "CORR/TYPE  = true label offered as a resolved id (what the UI shows)"
-echo "CORR+/TYPE+ = also counting names the LLM got right but matching dropped"