Compare commits

...
Author SHA1 Message Date
claude-botandClaude Opus 5 353b7882d7 fix(llama-server): one slot, so 16k belongs to one request
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Four Paperless requests overlapped and all four died:

    decode: failed to find free space in the KV cache
    srv decode: Context size has been exceeded.

They wanted 6209, 6472, 4303 and 1271 tokens from a shared pool of
16384. Auto slot count opens four of them and is also what enables
--kv-unified, so they draw on one buffer rather than a quarter each --
generous for a single caller, and impossible for four.

They overlapped because a client timeout does not stop the generation
behind it. The eval abandons a request after 300 s and moves to the next
document; the abandoned one keeps consuming the budget the next one
needs, and one slow document is enough to start the pile-up. With one
slot they queue instead, which is what should happen: nothing calls this
server in parallel on purpose, and Paperless's celery worker runs at
concurrency 1.

Deliberately no liveness probe alongside this. /health answered 200 in
0.4 ms throughout, which looks like the probe failing at its job -- but
the server drained the batch and served again by itself, so a probe that
restarted the pod would have turned a self-healing incident into an
outage. With a single slot, a probe asking for a real completion would
also queue behind live work and kill healthy pods mid-document.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-02 21:31:05 +02:00
k.renner 403a8e0e90 Merge pull request 'fix(llama-server): back to 16k — 32k per slot corrupts inference' (#770) from llama-ctx-revert into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#770
2026-09-02 19:06:25 +00:00
claude-botandClaude Opus 5 5023a5971b fix(llama-server): back to 16k — 32k per slot corrupts inference
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
#767 broke the model. Asked to classify a document and answer with JSON
only, the server returned 1025 tokens of "!" against a 650-token prompt.
That is the KV cache emitting numbers rather than language, and it takes
everything downstream with it: Paperless ran past its 300 s timeout on
every document, Home Assistant got an empty completion once and a
hallucinated list of devices that do not exist another time.

None of that looked like a backend fault from the outside. The server
starts, /props reports the requested size, /health passes, and the
OpenAI response is well-formed with finish_reason "stop". Only the
generated text gives it away, which is why three separate symptoms got
blamed on prompt size, on tokenizer behaviour and on the RAG context
before anyone read the output.

The boundary is between the two changes rather than at either: #765 left
16384 per slot and produced coherent answers; #767 raised it to 32768 and
produced this. The same Vulkan/Mesa stack already fails at --ubatch-size
2048 with ErrorDeviceLost, so a ceiling here is not a surprise -- only
that it degrades quietly instead of erroring.

--parallel goes with it. Unset, slots return to auto, which is what keeps
--kv-unified enabled ("if number of slots is auto"), so the auto slots
share one buffer and each may draw the full 16384.

Home Assistant needed 18910 tokens for one Assist request and cannot have
them at this ceiling. That path is closed, and Paperless -- which fits in
16k and never needed more -- is what this restores.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-02 21:05:03 +02:00
k.renner 1951497175 Merge pull request 'feat(crowdsec): arm the bouncer everywhere, drop the per-host override' (#769) from arm-crowdsec-proxy-1 into master
ci/woodpecker/push/deploy-netbird-proxy Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#769
2026-09-02 18:57:09 +00:00
security-bot ad32ab566b chore(image-automation): bump ghcr.io/home-assistant/home-assistant:2026.9.0
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Updates managed by image-automation-controller.
2026-09-02 16:23:17 +00:00
2 changed files with 35 additions and 22 deletions
@@ -95,7 +95,7 @@ spec:
memory: 32Mi
containers:
- name: home-assistant
image: ghcr.io/home-assistant/home-assistant:2026.8.3 # {"$imagepolicy": "flux-system:home-assistant"}
image: ghcr.io/home-assistant/home-assistant:2026.9.0 # {"$imagepolicy": "flux-system:home-assistant"}
env:
- name: TZ
value: Europe/Berlin
+34 -21
View File
@@ -75,31 +75,44 @@ spec:
# many small per-expert matmuls that sets the pace, not bandwidth.
- --threads
- "4"
# 32k because Home Assistant's Assist agent measured 18910 tokens
# for a single request: ~12k of exposed-entity overview plus ~7k of
# base instructions and tool schemas, whose parameters enumerate
# every exposed name, area and floor. At 16384 the request was
# refused outright.
# 16k, and going above it corrupts inference rather than failing.
# At 32768 per slot this server answered a 650-token classification
# prompt with 1025 tokens of "!" — the KV cache producing numbers
# rather than language. It does not crash, it does not warn, and
# the OpenAI response looks well-formed, so the damage only shows
# in what callers get back: Paperless ran past its 300 s timeout,
# and Home Assistant got an empty completion and a hallucinated
# device list.
#
# --ctx-size is the total and llama.cpp divides it across the
# slots, so 65536 over two slots is 32768 each.
# The same Vulkan/Mesa stack already fails hard at --ubatch-size
# 2048; this is the quiet version of that limit. Raise this only
# against a generation that is read, not against a server that
# merely starts.
#
# It does not read that way without --parallel, and the reason is
# in `llama-server --help`: --kv-unified defaults to enabled only
# "if number of slots is auto". Leaving --parallel unset therefore
# bought a single KV buffer shared across sequences, where every
# slot could draw the full --ctx-size — which is why this server
# once reported n_ctx_slot 16384 across four slots. Naming a slot
# count turns that off and splits the context instead.
#
# Two slots is what is actually in use: one holds Home Assistant's
# prompt, one holds Paperless's. A third caller would queue rather
# than evict either, which is the better failure mode given a cold
# prefill of this prompt costs around two minutes.
- --ctx-size
- "65536"
- "16384"
# One slot, so the 16384 belongs to one request at a time.
#
# Left on auto, llama.cpp opens four slots and — because auto is
# also what enables --kv-unified — has them share a single buffer
# of this size. Four Paperless requests at once wanted 6209, 6472,
# 4303 and 1271 tokens against those 16384, and all four died with
# "Context size has been exceeded".
#
# They were concurrent because a client timeout does not stop the
# generation behind it: each abandoned request keeps consuming the
# shared budget the next one needs, so one slow document is enough
# to start the pile-up. Queueing is the behaviour worth having
# here — nothing calls this server in parallel on purpose, and
# Paperless's celery worker runs at concurrency 1.
#
# No liveness probe change goes with this. The server recovered on
# its own once the batch drained, so a probe that restarted the pod
# would have made the incident worse; with a single slot, one that
# asked for a real completion would also queue behind live work and
# kill healthy pods during a long document.
- --parallel
- "2"
- "1"
# A micro-batch of 2048 or more makes this Vulkan/Mesa stack fail
# with ErrorDeviceLost.
- --ubatch-size