fix(llama-embed): size the batch for bge-m3's tokenizer, not Paperless's

The LLM index rebuild ran for 79 minutes and then died:

    input (1069 tokens) is too large to process.
    increase the physical batch size (current batch size: 1024)

Setting the batch to Paperless's chunk size looked like an exact match
and was not. Paperless counts its 1024 tokens with llama_index's
tokenizer; this server measures the batch in bge-m3's, which is
XLM-RoBERTa and splits German into more pieces. A chunk Paperless
considered full arrived here 4.4% over the limit, and the request failed
rather than truncating.

1536 absorbs a tokenizer ratio rather than a rounding error. 2048 stays
out of reach: that is where the chat model hit ErrorDeviceLost on this
Vulkan stack, under a 20 GB model's memory pressure that a 600 MB one may
well not reproduce — but there is no reason to go looking.

The index is left partial by the failed run and has to be rebuilt with
`document_llmindex rebuild` once this is live.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

ansible/roles/crowdsec-edge-bouncer/tasks/main.yml

@@ -19,14 +19,41 @@
     state: directory
     mode: "0755"
 
-- name: Fetch the pinned bouncer release
+# Fetched on the CONTROLLER and pushed, not pulled by the host.
+#
+# proxy-1 is IPv6-only, and neither github.com nor objects.githubusercontent.com
+# publishes an AAAA record — a get_url on the host fails with "Network is
+# unreachable" and no amount of retrying fixes it. Mirroring into
+# gitea.renner.uno is no help either: its public records are the internal LB
+# VIP (10.2.99.86 / fd99::86), so it is unreachable from outside the LAN. And
+# upstream ships no container image for this bouncer, which is expected —
+# it manipulates host netfilter.
+#
+# The CI runner has v4, so it does the fetching. The checksum is verified
+# there, before anything is copied to a host.
+- name: Fetch the pinned bouncer release on the controller
   ansible.builtin.get_url:
     url: >-
       https://github.com/crowdsecurity/cs-firewall-bouncer/releases/download/{{
       crowdsec_bouncer_version }}/crowdsec-firewall-bouncer-linux-amd64.tgz
-    dest: /opt/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}.tgz
+    dest: /tmp/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}.tgz
     checksum: sha256:{{ crowdsec_bouncer_sha256 }}
     mode: "0644"
+  delegate_to: localhost
+  become: false
+  run_once: true
+
+# Only the download had to move to the controller — the extraction stays on
+# the host. The ci-ansible image ships a tar that ansible rejects ("detected as
+# tar type None. GNU tar required"), while both host distributions have GNU
+# tar. Shipping the archive rather than the unpacked binary also keeps the
+# artefact that the checksum was verified against intact all the way to the
+# host.
+- name: Copy the bouncer archive to the host
+  ansible.builtin.copy:
+    src: /tmp/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}.tgz
+    dest: /opt/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}.tgz
+    mode: "0644"
 
 # Only the binary. The tarball also carries install.sh / upgrade.sh /
 # uninstall.sh, which manage systemd and config themselves — that is this

kubernetes/apps/llama-server/embed.yaml

@@ -41,7 +41,7 @@ spec:
           image: ghcr.io/ggml-org/llama.cpp:server-vulkan-b10731
           args:
             - --hf-repo
-            - Qwen/Qwen3-Embedding-0.6B-GGUF:Q8_0
+            - gpustack/bge-m3-GGUF:Q8_0
             # Serves POST /v1/embeddings.
             - --embeddings
             # A short, stable id for consumers. Without it the model is
@@ -50,28 +50,41 @@ spec:
             # One name only: llama-server keeps aliases in a sorted set and
             # reports the first as the id, so a second name can steal it.
             - --alias
-            - qwen3-embedding-0.6b
+            - bge-m3
-            # Qwen3-Embedding is trained for last-token pooling. Left
+            # bge-m3 is trained for CLS pooling — a different type from the
-            # unspecified llama-server takes the GGUF's default, which is not
+            # Qwen family this replaces, which uses last-token. Left unspecified
-            # visible from outside and would silently produce vectors the model
+            # llama-server takes the GGUF default, which is not visible from
-            # was not trained to emit. Set before anything is indexed: changing
+            # outside and would silently produce vectors the model was not
-            # pooling later invalidates every stored embedding.
+            # trained to emit. Read from the model's own 1_Pooling/config.json
+            # rather than assumed; of the candidates measured, three different
+            # pooling types were needed.
             - --pooling
-            - last
+            - cls
             - --n-gpu-layers
             - "99"
             - --threads
             - "2"
-            # Paperless chunks documents at 1024 tokens. The defaults are
+            # Paperless chunks at 1024 tokens, but counts them with its own
-            # batch 2048 / ubatch 512, and with --embeddings llama-server
+            # tokenizer while this server measures the batch in bge-m3's.
-            # clamps both down to 512 rather than run them mismatched — below
+            # Those disagree: XLM-RoBERTa splits German into more pieces than
-            # the chunk size. Matching both to 1024 fits a whole chunk in one
+            # llama_index's counter does, so a chunk Paperless considers full
-            # pass; 2048 is deliberately avoided, since that is where this
+            # at 1024 arrived here as 1069 and the request failed outright —
-            # Vulkan stack fails with ErrorDeviceLost.
+            # "input (1069 tokens) is too large to process".
+            #
+            # The headroom therefore has to absorb a tokenizer ratio, not just
+            # a rounding error, and 1.5x covers far more than the 1.044 that
+            # broke it. The defaults are batch 2048 / ubatch 512, and with
+            # --embeddings llama-server clamps both to the smaller rather than
+            # run them mismatched, so both are set.
+            #
+            # 2048 stays avoided: that is where the chat model met
+            # ErrorDeviceLost on this Vulkan stack. That failure came under a
+            # 20 GB model's memory pressure and may not bind a 600 MB one, but
+            # this does not need to find out.
             - --batch-size
-            - "1024"
+            - "1536"
             - --ubatch-size
-            - "1024"
+            - "1536"
             # Room for a full chunk plus overhead rather than exactly one.
             - --ctx-size
             - "4096"
@@ -91,6 +104,14 @@ spec:
             capabilities:
               drop:
                 - ALL
+          # Measured, not estimated: embedding 400 real documents peaked at
+          # 1.6 GiB (Prometheus max_over_time on the working set). The limit is
+          # well above that because the footprint is dominated by the four slots
+          # at 4096 context and the Vulkan working set rather than by the
+          # weights — the Qwen model this replaces peaked at 10.6 GiB on the
+          # same settings for the same 600 MiB of weights, and several
+          # candidates were OOMKilled at limits that looked generous against
+          # their model size.
           resources:
             requests:
               cpu: 250m
@@ -98,7 +119,7 @@ spec:
               gpu.intel.com/i915: "1"
             limits:
               cpu: "2"
-              memory: 4Gi
+              memory: 8Gi
               gpu.intel.com/i915: "1"
           # ~600 MiB rather than ~5 GiB, so a much shorter budget than the chat
           # model needs, but still sized for a cold download.

kubernetes/apps/llama-server/pvc.yaml

@@ -1,10 +1,20 @@
 ---
-# Holds the GGUF weights, downloaded once by the Deployment's initContainer.
+# Holds the GGUF weights, downloaded once by llama-server itself via --hf-repo.
-# Prune-disabled so retiring this Kustomization does not throw away several
+# Prune-disabled so retiring this Kustomization does not throw away tens of GiB
-# GiB of model files that would have to be pulled from Hugging Face again.
+# of model files that would have to be pulled from Hugging Face again.
 #
-# ReadWriteMany so a second replica or a one-off benchmark Job can mount the
+# ReadWriteMany so a second replica, a one-off benchmark Job, or a candidate
-# same weights; the models are read-only at inference time.
+# model being evaluated can mount the same weights; the models are read-only at
+# inference time.
+#
+# Sized well beyond the two deployed models. The cache is shared by every
+# candidate under evaluation (scripts/embed-eval-real.sh), and on a slow
+# connection the download is the long pole — a shared cache lets the large
+# candidates pull in the background while the small ones are already being
+# measured, and means a model is never fetched twice across runs.
+#
+# CephFS supports online expansion (allowVolumeExpansion on the storage class),
+# so growing this does not disturb the running pods.
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
@@ -18,4 +28,4 @@ spec:
   storageClassName: ceph-filesystem
   resources:
     requests:
-      storage: 40Gi
+      storage: 120Gi

kubernetes/apps/paperless/paperless.yaml

@@ -82,7 +82,10 @@ data:
   # primarily English archives — the better fit for a German corpus.
   PAPERLESS_AI_LLM_EMBEDDING_BACKEND: "openai-like"
   PAPERLESS_AI_LLM_EMBEDDING_ENDPOINT: "http://llama-embed.ai.svc.cluster.local:8080/v1"
-  PAPERLESS_AI_LLM_EMBEDDING_MODEL: "qwen3-embedding-0.6b"
+  # Must match --alias in kubernetes/apps/llama-server/embed.yaml. Changing the
+  # embedding model invalidates every stored vector, so a swap means the LLM
+  # index has to be rebuilt.
+  PAPERLESS_AI_LLM_EMBEDDING_MODEL: "bge-m3"
   PAPERLESS_AI_LLM_EMBEDDING_CHUNK_SIZE: "1024"
 ---
 apiVersion: apps/v1