#767 broke the model. Asked to classify a document and answer with JSON
only, the server returned 1025 tokens of "!" against a 650-token prompt.
That is the KV cache emitting numbers rather than language, and it takes
everything downstream with it: Paperless ran past its 300 s timeout on
every document, Home Assistant got an empty completion once and a
hallucinated list of devices that do not exist another time.
None of that looked like a backend fault from the outside. The server
starts, /props reports the requested size, /health passes, and the
OpenAI response is well-formed with finish_reason "stop". Only the
generated text gives it away, which is why three separate symptoms got
blamed on prompt size, on tokenizer behaviour and on the RAG context
before anyone read the output.
The boundary is between the two changes rather than at either: #765 left
16384 per slot and produced coherent answers; #767 raised it to 32768 and
produced this. The same Vulkan/Mesa stack already fails at --ubatch-size
2048 with ErrorDeviceLost, so a ceiling here is not a surprise -- only
that it degrades quietly instead of erroring.
--parallel goes with it. Unset, slots return to auto, which is what keeps
--kv-unified enabled ("if number of slots is auto"), so the auto slots
share one buffer and each may draw the full 16384.
Home Assistant needed 18910 tokens for one Assist request and cannot have
them at this ceiling. That path is closed, and Paperless -- which fits in
16k and never needed more -- is what this restores.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
proxy-2 has been enforcing since 14:21:54 without incident, so the role
default moves to nftables and the host_vars override goes away rather than
being duplicated for proxy-1. proxy-2's rendered config is unchanged by this,
so it will not restart; only proxy-1 does.
The staged rollout did its job and can be read off the deploy log:
proxy-1: deployed in mode "dry-run"
proxy-2: deployed in mode "nftables"
and only proxy-2 restarted, which showed up as a 1,887,882-byte startup pull
from its address alone.
The delivery path was tested in enforcing mode with a decision on 192.0.2.1
(TEST-NET-1): both hosts had the ban 3s later as a 205-byte delta, and the
revocation as a 203-byte delta 1-2s after deletion.
What is still not proven is that a packet is actually dropped, and the
measurement for that argues for finishing the rollout rather than waiting.
Requests from banned addresses reaching Traefik stopped after proxy-2 armed —
1 at 14:02, 3 at 14:07, 22 at 14:08, 1 at 14:18, none after — but 25 of those
27 arrived in a single burst beforehand, so at this volume that is consistent
with enforcement rather than evidence of it. With both hosts armed the claim
becomes falsifiable instead: no request from an address in the decision set
should reach Traefik at all, and a single one would prove otherwise.
dry-run remains available per host through inventory/host_vars, which is what
a newly added proxy should get before it enforces.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Arming the bouncer on proxy-2 (#766) merged and deployed nothing. The switch
lives in ansible/inventory/host_vars/proxy-2.renner.uno.yml, and the trigger
list covered ansible/roles/** and the play but no inventory path at all. The
change sat in git looking applied while the host kept running dry-run, and
nothing reported a problem — there was simply no pipeline run to fail.
The bouncer's LAPI key has the same shape: it lives in
group_vars/netbird_proxy/vault.yml, so a rotation would never have reached the
hosts either.
Both paths are now listed. pve_hosts.yml deliberately is not: it is the single
inventory file for the whole fleet, so listing it would redeploy the proxies
for any unrelated host edit. The cost is that a change to [netbird_proxy] group
membership needs a manual deployment run.
Merging this triggers the run itself, since .woodpecker/deploy-netbird-proxy.yml
is in its own trigger list — so the arming from #766 applies with it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`llama-server --help` documents both halves of what #765 got wrong, and
it ships in the image:
-np, --parallel N number of server slots (default: -1, -1 = auto)
-kvu, --kv-unified use single unified KV buffer shared across all
sequences (default: enabled if number of slots
is auto)
So leaving --parallel unset did not merely pick a slot count, it enabled
a shared KV buffer that every slot could draw the full --ctx-size from.
That is why this server reported n_ctx_slot 16384 across four slots, and
the startup log said so outright: kv_unified = 'true'. Naming a slot
count switches that off and splits the context.
The value stays 65536. Only the reasoning in the comment changes, from an
inference drawn off two /props readings to the documented behaviour.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
#765 claimed --ctx-size was per slot and halved the slot count to pay for
doubling it. Both halves of that were wrong. With --parallel set,
llama.cpp divides --ctx-size across the slots, so 32768 over two slots
gave 16384 each -- exactly what was there before -- and the Assist
request fails with the same message it did then:
request (18910 tokens) exceeds the available context size (16384)
The flag does behave differently without --parallel, which is where the
wrong reading came from: the four-slot server reported n_ctx_slot equal
to --ctx-size rather than a quarter of it. Adding --parallel changed
that, and the memory reading confirms it -- the pod fell from 24979 MiB
to 21601 MiB, so total KV halved rather than holding steady.
65536 across two slots is 32768 each, and puts total KV back where it was
before #765 along with the memory that came with it.
Verify rather than assume this time:
kubectl -n ai exec deploy/llama-server -- curl -sS \
http://127.0.0.1:8080/props | jq .default_generation_settings.n_ctx
must report 32768.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-02 16:13:35 +02:00
5 changed files with 46 additions and 45 deletions
# many small per-expert matmuls that sets the pace, not bandwidth.
- --threads
- "4"
# 32k because Home Assistant's Assist agent measured 18910 tokens
# for a single request: ~12k of exposed-entity overview plus ~7k of
# base instructions and tool schemas, whose parameters enumerate
# every exposed name, area and floor. At 16384 the request was
# refused outright.
# 16k, and going above it corrupts inference rather than failing.
# At 32768 per slot this server answered a 650-token classification
# prompt with 1025 tokens of "!" — the KV cache producing numbers
# rather than language. It does not crash, it does not warn, and
# the OpenAI response looks well-formed, so the damage only shows
# in what callers get back: Paperless ran past its 300 s timeout,
# and Home Assistant got an empty completion and a hallucinated
# device list.
#
# --ctx-size is per slot, not the total: /props reports
# n_ctx_slot equal to this value while running four slots, where a
# total would have divided down to 4096. Doubling it alone would
# therefore double the KV cache, so the slot count halves in step
# and the allocation stays where it is today — 24979 MiB of the
# 32Gi limit, most of it the 19.5 GiB of weights.
# The same Vulkan/Mesa stack already fails hard at --ubatch-size
# 2048; this is the quiet version of that limit. Raise this only
# against a generation that is read, not against a server that
# merely starts.
#
# Two slots is what is actually in use: one holds Home Assistant's
# prompt, one holds Paperless's. A third caller would queue rather
# than evict either, which is the better failure mode given a cold
# prefill of this prompt costs around two minutes.
# Slots stay unset, which is also what keeps --kv-unified on
# (`llama-server --help`: enabled "if number of slots is auto"), so
# the four auto slots share one buffer and each may draw the full
# 16384 rather than a quarter of it.
- --ctx-size
- "32768"
- --parallel
- "2"
- "16384"
# A micro-batch of 2048 or more makes this Vulkan/Mesa stack fail
# with ErrorDeviceLost.
- --ubatch-size
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
claude-botandClaude Opus 5
k.renner
security-bot