From 0a101eb8ee60d3f42e569451a0e9c2271301ea64 Mon Sep 17 00:00:00 2001 From: Claude Bot Date: Wed, 2 Sep 2026 15:13:54 +0200 Subject: [PATCH] fix(llama-embed): size the batch for bge-m3's tokenizer, not Paperless's MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The LLM index rebuild ran for 79 minutes and then died: input (1069 tokens) is too large to process. increase the physical batch size (current batch size: 1024) Setting the batch to Paperless's chunk size looked like an exact match and was not. Paperless counts its 1024 tokens with llama_index's tokenizer; this server measures the batch in bge-m3's, which is XLM-RoBERTa and splits German into more pieces. A chunk Paperless considered full arrived here 4.4% over the limit, and the request failed rather than truncating. 1536 absorbs a tokenizer ratio rather than a rounding error. 2048 stays out of reach: that is where the chat model hit ErrorDeviceLost on this Vulkan stack, under a 20 GB model's memory pressure that a 600 MB one may well not reproduce — but there is no reason to go looking. The index is left partial by the failed run and has to be rebuilt with `document_llmindex rebuild` once this is live. Co-Authored-By: Claude Opus 5 (1M context) --- kubernetes/apps/llama-server/embed.yaml | 27 +++++++++++++++++-------- 1 file changed, 19 insertions(+), 8 deletions(-) diff --git a/kubernetes/apps/llama-server/embed.yaml b/kubernetes/apps/llama-server/embed.yaml index 515fb988..a3060e5b 100644 --- a/kubernetes/apps/llama-server/embed.yaml +++ b/kubernetes/apps/llama-server/embed.yaml @@ -64,16 +64,27 @@ spec: - "99" - --threads - "2" - # Paperless chunks documents at 1024 tokens. The defaults are - # batch 2048 / ubatch 512, and with --embeddings llama-server - # clamps both down to 512 rather than run them mismatched — below - # the chunk size. Matching both to 1024 fits a whole chunk in one - # pass; 2048 is deliberately avoided, since that is where this - # Vulkan stack fails with ErrorDeviceLost. + # Paperless chunks at 1024 tokens, but counts them with its own + # tokenizer while this server measures the batch in bge-m3's. + # Those disagree: XLM-RoBERTa splits German into more pieces than + # llama_index's counter does, so a chunk Paperless considers full + # at 1024 arrived here as 1069 and the request failed outright — + # "input (1069 tokens) is too large to process". + # + # The headroom therefore has to absorb a tokenizer ratio, not just + # a rounding error, and 1.5x covers far more than the 1.044 that + # broke it. The defaults are batch 2048 / ubatch 512, and with + # --embeddings llama-server clamps both to the smaller rather than + # run them mismatched, so both are set. + # + # 2048 stays avoided: that is where the chat model met + # ErrorDeviceLost on this Vulkan stack. That failure came under a + # 20 GB model's memory pressure and may not bind a 600 MB one, but + # this does not need to find out. - --batch-size - - "1024" + - "1536" - --ubatch-size - - "1024" + - "1536" # Room for a full chunk plus overhead rather than exactly one. - --ctx-size - "4096"