fix(llama-server): --ctx-size is the total, so ask for 65536 #767

2026-09-02T14:13:55Z

claude-bot commented

2026-09-02 14:13:55 +00:00

#765 did not work. It claimed --ctx-size was per slot and halved the slot count to pay for doubling it. Both halves of that were wrong.

What actually happened

--ctx-size 32768, --parallel 2   ->   n_ctx_slot: 16384, total_slots: 2

With --parallel set, llama.cpp divides --ctx-size across the slots. Two slots of 16384 is exactly what was there before, so the Assist request fails with the identical message:

request (18910 tokens) exceeds the available context size (16384 tokens)

The memory reading confirms it independently: the pod fell from 24 979 MiB to 21 601 MiB. Total KV halved rather than holding steady, which is the opposite of what #765 predicted.

Where the wrong reading came from

Without --parallel, the same server reported n_ctx_slot: 16384 while running four slots. A total divided across four would have been 4096, so the flag genuinely does not divide in that configuration. Adding --parallel changed the behaviour. The observation was real; generalising it to the configuration being introduced in the same change was not, and nothing verified it afterwards.

The fix

65536 across 2 slots = 32768 each

That is the context the Assist request needs, and it puts total KV back where it was before #765 — along with the ~25 GiB memory that came with it, inside the 32Gi limit.

Verification after merge

kubectl -n ai exec deploy/llama-server -- curl -sS http://127.0.0.1:8080/props \
  | jq "{n_ctx_slot: .default_generation_settings.n_ctx, total_slots}"
kubectl -n ai top pod -l app.kubernetes.io/name=llama-server

Expect 32768 / 2 and memory near 25 GiB. Then the Assist request has to actually answer — that, not the flag value, is the check that matters.

🤖 Generated with Claude Code

#765 did not work. It claimed `--ctx-size` was per slot and halved the slot count to pay for doubling it. Both halves of that were wrong. ## What actually happened ``` --ctx-size 32768, --parallel 2 -> n_ctx_slot: 16384, total_slots: 2 ``` With `--parallel` set, llama.cpp **divides** `--ctx-size` across the slots. Two slots of 16384 is exactly what was there before, so the Assist request fails with the identical message: ``` request (18910 tokens) exceeds the available context size (16384 tokens) ``` The memory reading confirms it independently: the pod fell from **24 979 MiB to 21 601 MiB**. Total KV halved rather than holding steady, which is the opposite of what #765 predicted. ## Where the wrong reading came from Without `--parallel`, the same server reported `n_ctx_slot: 16384` while running four slots. A total divided across four would have been 4096, so the flag genuinely does not divide in that configuration. Adding `--parallel` changed the behaviour. The observation was real; generalising it to the configuration being introduced in the same change was not, and nothing verified it afterwards. ## The fix ``` 65536 across 2 slots = 32768 each ``` That is the context the Assist request needs, and it puts total KV back where it was before #765 — along with the ~25 GiB memory that came with it, inside the 32Gi limit. ## Verification after merge ``` kubectl -n ai exec deploy/llama-server -- curl -sS http://127.0.0.1:8080/props \ | jq "{n_ctx_slot: .default_generation_settings.n_ctx, total_slots}" kubectl -n ai top pod -l app.kubernetes.io/name=llama-server ``` Expect `32768` / `2` and memory near 25 GiB. Then the Assist request has to actually answer — that, not the flag value, is the check that matters. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit 2026-09-02 14:13:56 +00:00

fix(llama-server): --ctx-size is the total, so ask for 65536

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

7e74b93fb2

#765 claimed --ctx-size was per slot and halved the slot count to pay for
doubling it. Both halves of that were wrong. With --parallel set,
llama.cpp divides --ctx-size across the slots, so 32768 over two slots
gave 16384 each -- exactly what was there before -- and the Assist
request fails with the same message it did then:

    request (18910 tokens) exceeds the available context size (16384)

The flag does behave differently without --parallel, which is where the
wrong reading came from: the four-slot server reported n_ctx_slot equal
to --ctx-size rather than a quarter of it. Adding --parallel changed
that, and the memory reading confirms it -- the pod fell from 24979 MiB
to 21601 MiB, so total KV halved rather than holding steady.

65536 across two slots is 32768 each, and puts total KV back where it was
before #765 along with the memory that came with it.

Verify rather than assume this time:

    kubectl -n ai exec deploy/llama-server -- curl -sS \
      http://127.0.0.1:8080/props | jq .default_generation_settings.n_ctx

must report 32768.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

review-bot reviewed

2026-09-02 14:16:00 +00:00

review-bot left a comment

🤖 Automated review — model claude-haiku-4-5-20251001 (complexity: trivial), commit 7e74b93fb27aff593c72e3153fcd8fbb41b2998c.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

This PR fixes a broken llama-server configuration introduced in #765. The previous fix misunderstood how --ctx-size behaves with the --parallel flag: it is the total context size that llama.cpp divides across slots, not a per-slot value. Changing ctx-size from 32768 to 65536 with 2 parallel slots yields 32768 context per slot, which is what Home Assistant's Assist requests require. The fix is well-reasoned, includes accurate comments explaining the behavior, and restores memory usage to expected levels.

🟡 Suggestions

Post-merge validation: The PR description includes good verification commands (kubectl... curl... jq to check n_ctx_slot, and kubectl top pod to check memory). Consider documenting the expected post-merge state (n_ctx_slot: 32768, memory ~25 GiB) somewhere for the operator who merges this, since that validation step is important to confirm the fix actually worked.

Tests

This is a Kubernetes configuration change to a deployment manifest, not application code, so unit/integration tests are not applicable. The fix's correctness depends on:

The accurate understanding of llama.cpp's --ctx-size behavior with --parallel (which the PR author measured)
Post-deployment validation via the /props endpoint and memory monitoring (verification steps provided in the PR description)

Recommendation

Looks reasonable — no blocking issues; the fix correctly addresses a real production problem with a clear explanation of what went wrong before.

🤖 **Automated review** — model `claude-haiku-4-5-20251001` (complexity: trivial), commit `7e74b93fb27aff593c72e3153fcd8fbb41b2998c`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary This PR fixes a broken llama-server configuration introduced in #765. The previous fix misunderstood how `--ctx-size` behaves with the `--parallel` flag: it is the total context size that llama.cpp divides across slots, not a per-slot value. Changing ctx-size from 32768 to 65536 with 2 parallel slots yields 32768 context per slot, which is what Home Assistant's Assist requests require. The fix is well-reasoned, includes accurate comments explaining the behavior, and restores memory usage to expected levels. ## 🟡 Suggestions 1. **Post-merge validation**: The PR description includes good verification commands (`kubectl... curl... jq` to check `n_ctx_slot`, and `kubectl top pod` to check memory). Consider documenting the expected post-merge state (n_ctx_slot: 32768, memory ~25 GiB) somewhere for the operator who merges this, since that validation step is important to confirm the fix actually worked. ## Tests This is a Kubernetes configuration change to a deployment manifest, not application code, so unit/integration tests are not applicable. The fix's correctness depends on: 1. The accurate understanding of llama.cpp's `--ctx-size` behavior with `--parallel` (which the PR author measured) 2. Post-deployment validation via the `/props` endpoint and memory monitoring (verification steps provided in the PR description) ## Recommendation Looks reasonable — no blocking issues; the fix correctly addresses a real production problem with a clear explanation of what went wrong before.

claude-bot added 1 commit 2026-09-02 14:16:30 +00:00

docs(llama-server): cite the flag semantics instead of inferring them

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

021fd6a4ed

`llama-server --help` documents both halves of what #765 got wrong, and
it ships in the image:

    -np, --parallel N   number of server slots (default: -1, -1 = auto)
    -kvu, --kv-unified  use single unified KV buffer shared across all
                        sequences (default: enabled if number of slots
                        is auto)

So leaving --parallel unset did not merely pick a slot count, it enabled
a shared KV buffer that every slot could draw the full --ctx-size from.
That is why this server reported n_ctx_slot 16384 across four slots, and
the startup log said so outright: kv_unified = 'true'. Naming a slot
count switches that off and splits the context.

The value stays 65536. Only the reasoning in the comment changes, from an
inference drawn off two /props readings to the documented behaviour.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

k.renner merged commit 91849e31ff into master

2026-09-02 14:17:25 +00:00

k.renner referenced this issue from a commit

2026-09-02 14:17:26 +00:00

Merge pull request 'fix(llama-server): --ctx-size is the total, so ask for 65536' (#767) from llama-ctx-fix into master

claude-bot referenced this issue from a commit

2026-09-02 19:05:12 +00:00

fix(llama-server): back to 16k — 32k per slot corrupts inference

claude-bot referenced this pull request

2026-09-02 19:05:29 +00:00

fix(llama-server): back to 16k — 32k per slot corrupts inference #770

Sign in to join this conversation.