fix(llama-server): back to 16k — 32k per slot corrupts inference #770

2026-09-02T19:05:29Z

claude-bot commented

2026-09-02 19:05:29 +00:00

Paperless is broken right now, not slow. #767 raised the context to 32768 per slot and the model stopped producing language.

The evidence

A 650-token prompt — "classify this document, answer with JSON only, no explanation" — came back as:

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

1025 tokens of it, finish_reason: "stop", well-formed OpenAI envelope. That is a KV cache emitting numbers rather than text.

What it explains

Three symptoms that were each investigated as separate problems, all downstream of this one:

symptom	actual cause
Paperless ran past its 300 s timeout on every document	runaway generation of garbage
Home Assistant returned an empty completion	same
Home Assistant listed devices that do not exist	same

Each was blamed in turn on prompt size, on RAG context, and on the model being weak at long contexts. None of those were it, and none of it was visible without reading the generated text: the server starts, /props reports the requested size, /health passes.

Where the boundary is

Between the two changes rather than at either end:

#765 — 16384 per slot → coherent answers, correct tool calls
#767 — 32768 per slot → this

The same Vulkan/Mesa stack already fails hard at --ubatch-size 2048 with ErrorDeviceLost, so a ceiling is not surprising. That it degrades silently instead of erroring is the part worth remembering.

What this restores, and what it closes

--parallel goes too. Unset, the slot count returns to auto, which is what keeps --kv-unified enabled (llama-server --help: enabled "if number of slots is auto"), so the auto slots share one buffer and each may draw the full 16384.

Home Assistant measured 18 910 tokens for a single Assist request and cannot have them under this ceiling. That path is closed regardless of this PR. Paperless fits inside 16k, never needed more, and is what this brings back.

Verification after merge

Do not trust /props here — it reported the requested size the whole time the model was producing garbage. Read a generation:

kubectl -n paperless exec deploy/paperless -c paperless -- curl -sS -m 300 \
  -H "Authorization: Token $TOKEN" \
  http://127.0.0.1:8000/api/documents/1970/ai_suggestions/

Expect a populated suggestion object within ~90 s, which is what this endpoint did this morning before #765.

🤖 Generated with Claude Code

**Paperless is broken right now, not slow.** #767 raised the context to 32768 per slot and the model stopped producing language. ## The evidence A 650-token prompt — "classify this document, answer with JSON only, no explanation" — came back as: ``` !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ``` 1025 tokens of it, `finish_reason: "stop"`, well-formed OpenAI envelope. That is a KV cache emitting numbers rather than text. ## What it explains Three symptoms that were each investigated as separate problems, all downstream of this one: | symptom | actual cause | |---|---| | Paperless ran past its 300 s timeout on every document | runaway generation of garbage | | Home Assistant returned an empty completion | same | | Home Assistant listed devices that do not exist | same | Each was blamed in turn on prompt size, on RAG context, and on the model being weak at long contexts. None of those were it, and none of it was visible without reading the generated text: the server starts, `/props` reports the requested size, `/health` passes. ## Where the boundary is Between the two changes rather than at either end: - **#765** — 16384 per slot → coherent answers, correct tool calls - **#767** — 32768 per slot → this The same Vulkan/Mesa stack already fails hard at `--ubatch-size 2048` with `ErrorDeviceLost`, so a ceiling is not surprising. That it degrades **silently** instead of erroring is the part worth remembering. ## What this restores, and what it closes `--parallel` goes too. Unset, the slot count returns to auto, which is what keeps `--kv-unified` enabled (`llama-server --help`: enabled "if number of slots is auto"), so the auto slots share one buffer and each may draw the full 16384. Home Assistant measured **18 910 tokens** for a single Assist request and cannot have them under this ceiling. That path is closed regardless of this PR. Paperless fits inside 16k, never needed more, and is what this brings back. ## Verification after merge Do not trust `/props` here — it reported the requested size the whole time the model was producing garbage. Read a generation: ``` kubectl -n paperless exec deploy/paperless -c paperless -- curl -sS -m 300 \ -H "Authorization: Token $TOKEN" \ http://127.0.0.1:8000/api/documents/1970/ai_suggestions/ ``` Expect a populated suggestion object within ~90 s, which is what this endpoint did this morning before #765. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit 2026-09-02 19:05:30 +00:00

fix(llama-server): back to 16k — 32k per slot corrupts inference

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

5023a5971b

#767 broke the model. Asked to classify a document and answer with JSON
only, the server returned 1025 tokens of "!" against a 650-token prompt.
That is the KV cache emitting numbers rather than language, and it takes
everything downstream with it: Paperless ran past its 300 s timeout on
every document, Home Assistant got an empty completion once and a
hallucinated list of devices that do not exist another time.

None of that looked like a backend fault from the outside. The server
starts, /props reports the requested size, /health passes, and the
OpenAI response is well-formed with finish_reason "stop". Only the
generated text gives it away, which is why three separate symptoms got
blamed on prompt size, on tokenizer behaviour and on the RAG context
before anyone read the output.

The boundary is between the two changes rather than at either: #765 left
16384 per slot and produced coherent answers; #767 raised it to 32768 and
produced this. The same Vulkan/Mesa stack already fails at --ubatch-size
2048 with ErrorDeviceLost, so a ceiling here is not a surprise -- only
that it degrades quietly instead of erroring.

--parallel goes with it. Unset, slots return to auto, which is what keeps
--kv-unified enabled ("if number of slots is auto"), so the auto slots
share one buffer and each may draw the full 16384.

Home Assistant needed 18910 tokens for one Assist request and cannot have
them at this ceiling. That path is closed, and Paperless -- which fits in
16k and never needed more -- is what this restores.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

k.renner merged commit 403a8e0e90 into master

2026-09-02 19:06:25 +00:00

k.renner referenced this issue from a commit

2026-09-02 19:06:27 +00:00

Merge pull request 'fix(llama-server): back to 16k — 32k per slot corrupts inference' (#770) from llama-ctx-revert into master

review-bot reviewed

2026-09-02 19:07:31 +00:00

review-bot left a comment

🤖 Automated review — model claude-haiku-4-5-20251001 (complexity: trivial), commit 5023a5971bb407556f34be1140a06f007b3a5f1a.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

This PR reverts the llama-server context configuration from 32768 to 16384 tokens per slot, addressing a silent inference corruption issue where the model produced garbage output (repeating "!") instead of valid language. The change is well-documented with concrete evidence of the problem and is a sound production fix.

🔴 Blocking

None — no issues identified.

🟡 Suggestions

Verify the fix in production — The PR body includes a helpful kubectl command to verify the fix. Consider documenting a post-merge verification step or monitoring alert to catch if this regression happens again, since the corruption is silent (well-formed OpenAI envelope masking garbage output).

🟢 Nits

The comment at kubernetes/apps/llama-server/deployment.yaml:92-95 could mention explicitly that the tradeoff is between slot isolation (separate buffers per slot under --parallel) and buffer efficiency (shared buffer with auto slots). Currently clear but very dense.

Tests

This is infrastructure configuration, not code with tests. The verification method documented in the PR body (checking Paperless AI suggestions endpoint for populated results within ~90s) is appropriate. The configuration has been proven to work (references to successful operation at 16384 in #765) and now reverts away from proven-broken state (32768 in #767).

Recommendation

Looks reasonable — no blocking issues; suggestions above are optional improvements.

🤖 **Automated review** — model `claude-haiku-4-5-20251001` (complexity: trivial), commit `5023a5971bb407556f34be1140a06f007b3a5f1a`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary This PR reverts the llama-server context configuration from 32768 to 16384 tokens per slot, addressing a silent inference corruption issue where the model produced garbage output (repeating "!") instead of valid language. The change is well-documented with concrete evidence of the problem and is a sound production fix. ## 🔴 Blocking None — no issues identified. ## 🟡 Suggestions 1. **Verify the fix in production** — The PR body includes a helpful kubectl command to verify the fix. Consider documenting a post-merge verification step or monitoring alert to catch if this regression happens again, since the corruption is silent (well-formed OpenAI envelope masking garbage output). ## 🟢 Nits 1. The comment at `kubernetes/apps/llama-server/deployment.yaml:92-95` could mention explicitly that the tradeoff is between slot isolation (separate buffers per slot under `--parallel`) and buffer efficiency (shared buffer with auto slots). Currently clear but very dense. ## Tests This is infrastructure configuration, not code with tests. The verification method documented in the PR body (checking Paperless AI suggestions endpoint for populated results within ~90s) is appropriate. The configuration has been proven to work (references to successful operation at 16384 in #765) and now reverts away from proven-broken state (32768 in #767). ## Recommendation Looks reasonable — no blocking issues; suggestions above are optional improvements.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#770