fix(llama-embed): size the batch for bge-m3's tokenizer, not Paperless's

The LLM index rebuild ran for 79 minutes and then died: input (1069 tokens) is too large to process. increase the physical batch size (current batch size: 1024) Setting the batch to Paperless's chunk size looked like an exact match and was not. Paperless counts its 1024 tokens with llama_index's tokenizer; this server measures the batch in bge-m3's, which is XLM-RoBERTa and splits German into more pieces. A chunk Paperless considered full arrived here 4.4% over the limit, and the request failed rather than truncating. 1536 absorbs a tokenizer ratio rather than a rounding error. 2048 stays out of reach: that is where the chat model hit ErrorDeviceLost on this Vulkan stack, under a 20 GB model's memory pressure that a 600 MB one may well not reproduce — but there is no reason to go looking. The index is left partial by the failed run and has to be rebuilt with `document_llmindex rebuild` once this is live. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-02 15:13:54 +02:00
2 changed files with 21 additions and 21 deletions
@@ -120,14 +120,7 @@ spec:
    server:
      resources:
        requests:
-          # 500m, not 200m: the HPA below targets 70% of *this* request, so at
-          # 200m the scale-up threshold sat at 140m — below the pod's own
-          # ordinary peak. Measured over 7d, a warm server pod p99s at ~108m
-          # and peaks at ~190m, and a freshly-started one burns ~300m booting
-          # Django. That made the HPA oscillate 2<->3<->4 all day and pin at
-          # maxReplicas (KubeHpaMaxedOut, issue #761). At 500m the threshold
-          # is 350m/pod, above the boot burst, so only real load scales out.
-          cpu: 500m
+          cpu: 200m
          memory: 1Gi
        limits:
          memory: 2Gi
@@ -161,11 +154,7 @@ spec:
    worker:
      resources:
        requests:
-          # Same reasoning as server above, and the worker is burstier still:
-          # over 7d its per-pod p99 reaches ~336m and peaks hit ~630m while
-          # running blueprint applies and scheduled tasks. 200m understated
-          # that badly enough that routine task batches read as saturation.
-          cpu: 500m
+          cpu: 200m
          memory: 1Gi
        limits:
          memory: 2Gi
@@ -64,16 +64,27 @@ spec:
            - "99"
            - --threads
            - "2"
-            # Paperless chunks documents at 1024 tokens. The defaults are
-            # batch 2048 / ubatch 512, and with --embeddings llama-server
-            # clamps both down to 512 rather than run them mismatched — below
-            # the chunk size. Matching both to 1024 fits a whole chunk in one
-            # pass; 2048 is deliberately avoided, since that is where this
-            # Vulkan stack fails with ErrorDeviceLost.
+            # Paperless chunks at 1024 tokens, but counts them with its own
+            # tokenizer while this server measures the batch in bge-m3's.
+            # Those disagree: XLM-RoBERTa splits German into more pieces than
+            # llama_index's counter does, so a chunk Paperless considers full
+            # at 1024 arrived here as 1069 and the request failed outright —
+            # "input (1069 tokens) is too large to process".
+            #
+            # The headroom therefore has to absorb a tokenizer ratio, not just
+            # a rounding error, and 1.5x covers far more than the 1.044 that
+            # broke it. The defaults are batch 2048 / ubatch 512, and with
+            # --embeddings llama-server clamps both to the smaller rather than
+            # run them mismatched, so both are set.
+            #
+            # 2048 stays avoided: that is where the chat model met
+            # ErrorDeviceLost on this Vulkan stack. That failure came under a
+            # 20 GB model's memory pressure and may not bind a 600 MB one, but
+            # this does not need to find out.
            - --batch-size
-            - "1024"
+            - "1536"
            - --ubatch-size
-            - "1024"
+            - "1536"
            # Room for a full chunk plus overhead rather than exactly one.
            - --ctx-size
            - "4096"