From 5023a5971bb407556f34be1140a06f007b3a5f1a Mon Sep 17 00:00:00 2001 From: Claude Bot Date: Wed, 2 Sep 2026 21:05:03 +0200 Subject: [PATCH] =?UTF-8?q?fix(llama-server):=20back=20to=2016k=20?= =?UTF-8?q?=E2=80=94=2032k=20per=20slot=20corrupts=20inference?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #767 broke the model. Asked to classify a document and answer with JSON only, the server returned 1025 tokens of "!" against a 650-token prompt. That is the KV cache emitting numbers rather than language, and it takes everything downstream with it: Paperless ran past its 300 s timeout on every document, Home Assistant got an empty completion once and a hallucinated list of devices that do not exist another time. None of that looked like a backend fault from the outside. The server starts, /props reports the requested size, /health passes, and the OpenAI response is well-formed with finish_reason "stop". Only the generated text gives it away, which is why three separate symptoms got blamed on prompt size, on tokenizer behaviour and on the RAG context before anyone read the output. The boundary is between the two changes rather than at either: #765 left 16384 per slot and produced coherent answers; #767 raised it to 32768 and produced this. The same Vulkan/Mesa stack already fails at --ubatch-size 2048 with ErrorDeviceLost, so a ceiling here is not a surprise -- only that it degrades quietly instead of erroring. --parallel goes with it. Unset, slots return to auto, which is what keeps --kv-unified enabled ("if number of slots is auto"), so the auto slots share one buffer and each may draw the full 16384. Home Assistant needed 18910 tokens for one Assist request and cannot have them at this ceiling. That path is closed, and Paperless -- which fits in 16k and never needed more -- is what this restores. Co-Authored-By: Claude Opus 5 (1M context) --- kubernetes/apps/llama-server/deployment.yaml | 39 +++++++++----------- 1 file changed, 17 insertions(+), 22 deletions(-) diff --git a/kubernetes/apps/llama-server/deployment.yaml b/kubernetes/apps/llama-server/deployment.yaml index 1a947975..c15be8e5 100644 --- a/kubernetes/apps/llama-server/deployment.yaml +++ b/kubernetes/apps/llama-server/deployment.yaml @@ -75,31 +75,26 @@ spec: # many small per-expert matmuls that sets the pace, not bandwidth. - --threads - "4" - # 32k because Home Assistant's Assist agent measured 18910 tokens - # for a single request: ~12k of exposed-entity overview plus ~7k of - # base instructions and tool schemas, whose parameters enumerate - # every exposed name, area and floor. At 16384 the request was - # refused outright. + # 16k, and going above it corrupts inference rather than failing. + # At 32768 per slot this server answered a 650-token classification + # prompt with 1025 tokens of "!" — the KV cache producing numbers + # rather than language. It does not crash, it does not warn, and + # the OpenAI response looks well-formed, so the damage only shows + # in what callers get back: Paperless ran past its 300 s timeout, + # and Home Assistant got an empty completion and a hallucinated + # device list. # - # --ctx-size is the total and llama.cpp divides it across the - # slots, so 65536 over two slots is 32768 each. + # The same Vulkan/Mesa stack already fails hard at --ubatch-size + # 2048; this is the quiet version of that limit. Raise this only + # against a generation that is read, not against a server that + # merely starts. # - # It does not read that way without --parallel, and the reason is - # in `llama-server --help`: --kv-unified defaults to enabled only - # "if number of slots is auto". Leaving --parallel unset therefore - # bought a single KV buffer shared across sequences, where every - # slot could draw the full --ctx-size — which is why this server - # once reported n_ctx_slot 16384 across four slots. Naming a slot - # count turns that off and splits the context instead. - # - # Two slots is what is actually in use: one holds Home Assistant's - # prompt, one holds Paperless's. A third caller would queue rather - # than evict either, which is the better failure mode given a cold - # prefill of this prompt costs around two minutes. + # Slots stay unset, which is also what keeps --kv-unified on + # (`llama-server --help`: enabled "if number of slots is auto"), so + # the four auto slots share one buffer and each may draw the full + # 16384 rather than a quarter of it. - --ctx-size - - "65536" - - --parallel - - "2" + - "16384" # A micro-batch of 2048 or more makes this Vulkan/Mesa stack fail # with ErrorDeviceLost. - --ubatch-size -- 2.54.0