feat(llama-server): 32k context per slot for the Assist agent #765

2026-09-02T14:07:30Z

claude-bot commented

2026-09-02 14:07:30 +00:00

The Home Assistant conversation agent is now configured and running, and it refuses every request:

API error: request (18910 tokens) exceeds the available context size
(16384 tokens), try increasing it

Where 18910 tokens come from

	tokens
exposed-entity overview	~12 048
base instructions + tool schemas	~6 862
total	18 910

The second half is not reachable by curation: the Assist tool parameters enumerate every exposed name, area and floor, so they scale with the same set the overview does.

The first half already had a hard look. All 4156 live entities were reviewed individually and the exposed set went from 845 to 666 — firmware entities, per-device power and energy sensors, switch_as_x sources, battery maintenance, upstream supply switches. That moved the overview from 16 045 tokens to 12 048. Squeezing another 2 500 out of it would mean giving up rooms, not noise.

Why this does not cost memory

--ctx-size is per slot, not the total. /props reports n_ctx_slot equal to the flag while four slots are running — a total would have divided down to 4096. So doubling the flag on its own doubles the KV cache.

Halving the slot count in the same change keeps the allocation exactly where it is:

before   4 slots x 16384 = 65536 tokens of KV
after    2 slots x 32768 = 65536 tokens of KV

Measured now: 24 979 MiB against the 32Gi limit, of which ~19.5 GiB is weights.

Two slots is the right number

One holds the Home Assistant prompt, one holds the Paperless prompt, and llama.cpp assigns slots by prompt similarity so each stays warm. That matters more than it sounds: a cold prefill of this prompt costs about two minutes, against ~2 s once cached. A third caller queueing is a far better outcome than it evicting one of the two.

The prompt stays cache-stable across state changes because Home Assistant builds the overview with include_state=False — current values reach the model through the GetLiveContext tool instead, so a light switching on does not invalidate the prefix.

Verification after merge

kubectl -n ai exec deploy/llama-server -- curl -sS http://127.0.0.1:8080/props \
  | jq "{n_ctx_slot: .default_generation_settings.n_ctx, total_slots}"
kubectl -n ai top pod -l app.kubernetes.io/name=llama-server

Expect 32768 / 2, and memory near the current 24 979 MiB rather than above it. Then the same Assist request that returns the error above should answer instead.

🤖 Generated with Claude Code

The Home Assistant conversation agent is now configured and running, and it refuses every request: ``` API error: request (18910 tokens) exceeds the available context size (16384 tokens), try increasing it ``` ## Where 18910 tokens come from | | tokens | |---|---| | exposed-entity overview | ~12 048 | | base instructions + tool schemas | ~6 862 | | **total** | **18 910** | The second half is not reachable by curation: the Assist tool parameters enumerate every exposed name, area and floor, so they scale with the same set the overview does. The first half already had a hard look. All 4156 live entities were reviewed individually and the exposed set went from 845 to 666 — firmware entities, per-device power and energy sensors, `switch_as_x` sources, battery maintenance, upstream supply switches. That moved the overview from 16 045 tokens to 12 048. Squeezing another 2 500 out of it would mean giving up rooms, not noise. ## Why this does not cost memory `--ctx-size` is **per slot**, not the total. `/props` reports `n_ctx_slot` equal to the flag while four slots are running — a total would have divided down to 4096. So doubling the flag on its own doubles the KV cache. Halving the slot count in the same change keeps the allocation exactly where it is: ``` before 4 slots x 16384 = 65536 tokens of KV after 2 slots x 32768 = 65536 tokens of KV ``` Measured now: **24 979 MiB** against the 32Gi limit, of which ~19.5 GiB is weights. ## Two slots is the right number One holds the Home Assistant prompt, one holds the Paperless prompt, and llama.cpp assigns slots by prompt similarity so each stays warm. That matters more than it sounds: a **cold prefill of this prompt costs about two minutes**, against ~2 s once cached. A third caller queueing is a far better outcome than it evicting one of the two. The prompt stays cache-stable across state changes because Home Assistant builds the overview with `include_state=False` — current values reach the model through the `GetLiveContext` tool instead, so a light switching on does not invalidate the prefix. ## Verification after merge ``` kubectl -n ai exec deploy/llama-server -- curl -sS http://127.0.0.1:8080/props \ | jq "{n_ctx_slot: .default_generation_settings.n_ctx, total_slots}" kubectl -n ai top pod -l app.kubernetes.io/name=llama-server ``` Expect `32768` / `2`, and memory near the current 24 979 MiB rather than above it. Then the same Assist request that returns the error above should answer instead. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit 2026-09-02 14:07:30 +00:00

feat(llama-server): 32k context per slot for the Assist agent

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

60a3da9447

The Home Assistant conversation agent is configured and refuses every
request:

    API error: request (18910 tokens) exceeds the available
    context size (16384 tokens), try increasing it

18910 breaks down as ~12k for the exposed-entity overview and ~7k for
the base instructions and tool schemas, whose parameters enumerate every
exposed name, area and floor. Trimming the exposed set from 845 entities
to 666 already took the overview from 16045 tokens to 12048; the
remainder is not the sort of thing curation reaches.

--ctx-size is per slot rather than the total, which is what makes this
affordable: /props reports n_ctx_slot equal to the flag while running
four slots, where a total would have divided down to 4096. Doubling the
flag alone would double the KV cache, so the slot count halves in step
and the allocation lands where it already is -- 24979 MiB against the
32Gi limit, most of it the 19.5 GiB of weights.

Two slots matches the actual callers: one holds Home Assistant's prompt,
one holds Paperless's, and prompt caching keeps each warm. A cold prefill
of this prompt costs about two minutes, so a third caller queueing is a
better outcome than it evicting one of them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

review-bot reviewed

2026-09-02 14:09:00 +00:00

review-bot left a comment

🤖 Automated review — model claude-haiku-4-5-20251001 (complexity: trivial), commit 60a3da94470640abcb43041d3466b8e963534102.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

This PR increases the llama-server context size from 16384 to 32768 tokens and reduces parallel slots from 4 to 2, allowing the Home Assistant Assist agent to function. The change maintains constant KV cache memory by trading slots for context depth—a sound trade-off since only two workloads (Home Assistant and Paperless) are actively running.

🟡 Suggestions

kubernetes/apps/paperless/paperless.yaml:71 — The comment "Under llama-server's own 16384" is now outdated. Update it to reference the new 32768 limit to keep the reasoning clear for future maintainers. The value 12288 itself is still safe and does not need to change.

- # Under llama-server's own 16384, leaving room for the response.
+ # Under llama-server's own 32768, leaving room for the response.

🟢 Nits

None — the deployment comments are thorough and the reasoning is well documented.

Tests

No test files affected. The PR includes manual verification commands that should be run after deployment:

kubectl -n ai exec deploy/llama-server -- curl -sS http://127.0.0.1:8080/props \
  | jq "{n_ctx_slot: .default_generation_settings.n_ctx, total_slots}"
kubectl -n ai top pod -l app.kubernetes.io/name=llama-server

Verify that n_ctx_slot reports 32768 and total memory stays near 24979 MiB.

Recommendation

Looks reasonable — no blocking issues; update the comment in paperless.yaml for consistency.

🤖 **Automated review** — model `claude-haiku-4-5-20251001` (complexity: trivial), commit `60a3da94470640abcb43041d3466b8e963534102`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary This PR increases the llama-server context size from 16384 to 32768 tokens and reduces parallel slots from 4 to 2, allowing the Home Assistant Assist agent to function. The change maintains constant KV cache memory by trading slots for context depth—a sound trade-off since only two workloads (Home Assistant and Paperless) are actively running. ## 🟡 Suggestions **kubernetes/apps/paperless/paperless.yaml:71** — The comment "Under llama-server's own 16384" is now outdated. Update it to reference the new 32768 limit to keep the reasoning clear for future maintainers. The value 12288 itself is still safe and does not need to change. ```diff - # Under llama-server's own 16384, leaving room for the response. + # Under llama-server's own 32768, leaving room for the response. ``` ## 🟢 Nits None — the deployment comments are thorough and the reasoning is well documented. ## Tests No test files affected. The PR includes manual verification commands that should be run after deployment: ``` kubectl -n ai exec deploy/llama-server -- curl -sS http://127.0.0.1:8080/props \ | jq "{n_ctx_slot: .default_generation_settings.n_ctx, total_slots}" kubectl -n ai top pod -l app.kubernetes.io/name=llama-server ``` Verify that `n_ctx_slot` reports 32768 and total memory stays near 24979 MiB. ## Recommendation Looks reasonable — no blocking issues; update the comment in paperless.yaml for consistency.

k.renner merged commit d655d65ad9 into master

2026-09-02 14:09:07 +00:00

k.renner referenced this issue from a commit

2026-09-02 14:09:08 +00:00

Merge pull request 'feat(llama-server): 32k context per slot for the Assist agent' (#765) from llama-ctx-assist into master

claude-bot referenced this issue from a commit

2026-09-02 14:13:43 +00:00

fix(llama-server): --ctx-size is the total, so ask for 65536

claude-bot referenced this pull request

2026-09-02 14:13:55 +00:00

fix(llama-server): --ctx-size is the total, so ask for 65536 #767

claude-bot referenced this issue from a commit