fix(llama-embed): size the batch for bge-m3 tokenizer, not Paperless #762

2026-09-02T13:14:20Z

claude-bot commented

2026-09-02 13:14:20 +00:00

The LLM index rebuild ran for 79 minutes and then died on the last stretch:

openai.InternalServerError: Error code: 500 -
  input (1069 tokens) is too large to process.
  increase the physical batch size (current batch size: 1024)

What was wrong

--batch-size 1024 was chosen to match PAPERLESS_AI_LLM_EMBEDDING_CHUNK_SIZE: 1024 exactly. That looked like a precise fit and was a mistake: the two numbers are counted in different tokenizers.

Paperless chunks with llama_index counter. This server measures its batch in the tokenizer of bge-m3, which is XLM-RoBERTa and splits German into more pieces. A chunk Paperless considered exactly full arrived here at 1069 tokens — 4.4% over — and llama-server rejected the request outright rather than truncating it.

The old comment in this file asserted the opposite, so it is replaced rather than amended.

Why 1536

The headroom has to absorb a tokenizer ratio, not a rounding error, and the ratio varies per document — a text heavy with numbers, compounds or umlauts inflates more than one that is not. 1.5x covers well beyond the 1.044 that broke this run.

2048 stays out of reach. That is where the chat model met ErrorDeviceLost on this Vulkan stack. That failure happened under a 20 GB model memory pressure and may not bind a 600 MB one at all — but nothing here needs to find out.

Both flags are set because --embeddings makes llama-server clamp batch and ubatch to the smaller of the two rather than run them mismatched.

Aftermath

The failed run leaves the index partial (90 MB, unknown completeness). After this is live it has to be rebuilt:

kubectl -n paperless exec deploy/paperless -c paperless -- \
  python manage.py document_llmindex rebuild

At the measured ~115 chunks/min that is roughly an hour.

What this delayed

scripts/paperless-eval.sh (#756) needs a complete index to compare the classic and AI engines on equal footing. The classic baseline is already measured — 73.0% correspondent, 100.0% document type on 100 documents — but the classifier was trained on those same documents, so that number is training-set accuracy and not yet comparable.

🤖 Generated with Claude Code

The LLM index rebuild ran for 79 minutes and then died on the last stretch: ``` openai.InternalServerError: Error code: 500 - input (1069 tokens) is too large to process. increase the physical batch size (current batch size: 1024) ``` ## What was wrong `--batch-size 1024` was chosen to match `PAPERLESS_AI_LLM_EMBEDDING_CHUNK_SIZE: 1024` exactly. That looked like a precise fit and was a mistake: **the two numbers are counted in different tokenizers.** Paperless chunks with llama_index counter. This server measures its batch in the tokenizer of bge-m3, which is XLM-RoBERTa and splits German into more pieces. A chunk Paperless considered exactly full arrived here at 1069 tokens — 4.4% over — and llama-server rejected the request outright rather than truncating it. The old comment in this file asserted the opposite, so it is replaced rather than amended. ## Why 1536 The headroom has to absorb a **tokenizer ratio**, not a rounding error, and the ratio varies per document — a text heavy with numbers, compounds or umlauts inflates more than one that is not. 1.5x covers well beyond the 1.044 that broke this run. 2048 stays out of reach. That is where the chat model met `ErrorDeviceLost` on this Vulkan stack. That failure happened under a 20 GB model memory pressure and may not bind a 600 MB one at all — but nothing here needs to find out. Both flags are set because `--embeddings` makes llama-server clamp batch and ubatch to the smaller of the two rather than run them mismatched. ## Aftermath The failed run leaves the index partial (90 MB, unknown completeness). After this is live it has to be rebuilt: ``` kubectl -n paperless exec deploy/paperless -c paperless -- \ python manage.py document_llmindex rebuild ``` At the measured ~115 chunks/min that is roughly an hour. ## What this delayed `scripts/paperless-eval.sh` (#756) needs a complete index to compare the classic and AI engines on equal footing. The classic baseline is already measured — 73.0% correspondent, 100.0% document type on 100 documents — but the classifier was trained on those same documents, so that number is training-set accuracy and not yet comparable. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit 2026-09-02 13:14:20 +00:00

fix(llama-embed): size the batch for bge-m3's tokenizer, not Paperless's

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

0a101eb8ee

The LLM index rebuild ran for 79 minutes and then died:

    input (1069 tokens) is too large to process.
    increase the physical batch size (current batch size: 1024)

Setting the batch to Paperless's chunk size looked like an exact match
and was not. Paperless counts its 1024 tokens with llama_index's
tokenizer; this server measures the batch in bge-m3's, which is
XLM-RoBERTa and splits German into more pieces. A chunk Paperless
considered full arrived here 4.4% over the limit, and the request failed
rather than truncating.

1536 absorbs a tokenizer ratio rather than a rounding error. 2048 stays
out of reach: that is where the chat model hit ErrorDeviceLost on this
Vulkan stack, under a 20 GB model's memory pressure that a 600 MB one may
well not reproduce — but there is no reason to go looking.

The index is left partial by the failed run and has to be rebuilt with
`document_llmindex rebuild` once this is live.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

review-bot reviewed

2026-09-02 13:15:55 +00:00

review-bot left a comment

🤖 Automated review — model claude-haiku-4-5-20251001 (complexity: trivial), commit 0a101eb8ee60d3f42e569451a0e9c2271301ea64.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

This PR fixes a batch-size configuration for the llama-embed embedding server that was causing index rebuild failures. The root cause is a tokenizer mismatch: Paperless counts tokens with its own counter, but llama-server measures batch size with bge-m3's XLM-RoBERTa tokenizer, which splits German text more aggressively. The fix increases batch-size and ubatch-size from 1024 to 1536 to provide headroom for this mismatch, and replaces an incorrect comment with one that accurately documents the problem and reasoning.

🟢 Nits

kubernetes/apps/llama-server/embed.yaml:76-77 — The phrase "run them mismatched" could be slightly clearer as "run them at different values" or "run asymmetrically", though the current wording is acceptable given the context from the preceding line.

Tests

This is a Kubernetes configuration change, not application code, so no unit tests are expected. The fix should be validated by running the index rebuild as mentioned in the PR body: the rebuild will either succeed with the new batch size or fail if the sizing is still insufficient. The PR author's plan to rebuild after merge is the appropriate validation approach.

Recommendation

Looks reasonable — no blocking issues; the fix is well-grounded in the actual failure mode and the reasoning for the specific value chosen (1536 for 1.5x headroom) is sound.

🤖 **Automated review** — model `claude-haiku-4-5-20251001` (complexity: trivial), commit `0a101eb8ee60d3f42e569451a0e9c2271301ea64`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary This PR fixes a batch-size configuration for the llama-embed embedding server that was causing index rebuild failures. The root cause is a tokenizer mismatch: Paperless counts tokens with its own counter, but llama-server measures batch size with bge-m3's XLM-RoBERTa tokenizer, which splits German text more aggressively. The fix increases batch-size and ubatch-size from 1024 to 1536 to provide headroom for this mismatch, and replaces an incorrect comment with one that accurately documents the problem and reasoning. ## 🟢 Nits **`kubernetes/apps/llama-server/embed.yaml:76-77`** — The phrase "run them mismatched" could be slightly clearer as "run them at different values" or "run asymmetrically", though the current wording is acceptable given the context from the preceding line. ## Tests This is a Kubernetes configuration change, not application code, so no unit tests are expected. The fix should be validated by running the index rebuild as mentioned in the PR body: the rebuild will either succeed with the new batch size or fail if the sizing is still insufficient. The PR author's plan to rebuild after merge is the appropriate validation approach. ## Recommendation Looks reasonable — no blocking issues; the fix is well-grounded in the actual failure mode and the reasoning for the specific value chosen (1536 for 1.5x headroom) is sound.

k.renner merged commit 75ecf020cc into master

2026-09-02 13:58:42 +00:00

k.renner referenced this issue from a commit

2026-09-02 13:58:43 +00:00

Merge pull request 'fix(llama-embed): size the batch for bge-m3 tokenizer, not Paperless' (#762) from embed-batch-fix into master

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#762