Compare commits

..
Author SHA1 Message Date
claude-botandClaude Opus 5 021fd6a4ed docs(llama-server): cite the flag semantics instead of inferring them
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
`llama-server --help` documents both halves of what #765 got wrong, and
it ships in the image:

    -np, --parallel N   number of server slots (default: -1, -1 = auto)
    -kvu, --kv-unified  use single unified KV buffer shared across all
                        sequences (default: enabled if number of slots
                        is auto)

So leaving --parallel unset did not merely pick a slot count, it enabled
a shared KV buffer that every slot could draw the full --ctx-size from.
That is why this server reported n_ctx_slot 16384 across four slots, and
the startup log said so outright: kv_unified = 'true'. Naming a slot
count switches that off and splits the context.

The value stays 65536. Only the reasoning in the comment changes, from an
inference drawn off two /props readings to the documented behaviour.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-02 16:16:16 +02:00
claude-botandClaude Opus 5 7e74b93fb2 fix(llama-server): --ctx-size is the total, so ask for 65536
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
#765 claimed --ctx-size was per slot and halved the slot count to pay for
doubling it. Both halves of that were wrong. With --parallel set,
llama.cpp divides --ctx-size across the slots, so 32768 over two slots
gave 16384 each -- exactly what was there before -- and the Assist
request fails with the same message it did then:

    request (18910 tokens) exceeds the available context size (16384)

The flag does behave differently without --parallel, which is where the
wrong reading came from: the four-slot server reported n_ctx_slot equal
to --ctx-size rather than a quarter of it. Adding --parallel changed
that, and the memory reading confirms it -- the pod fell from 24979 MiB
to 21601 MiB, so total KV halved rather than holding steady.

65536 across two slots is 32768 each, and puts total KV back where it was
before #765 along with the memory that came with it.

Verify rather than assume this time:

    kubectl -n ai exec deploy/llama-server -- curl -sS \
      http://127.0.0.1:8080/props | jq .default_generation_settings.n_ctx

must report 32768.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-02 16:13:35 +02:00
4 changed files with 11 additions and 53 deletions
-12
View File
@@ -10,18 +10,6 @@ when:
# install-netbird-proxy.yml also applies the edge bouncer.
- "ansible/roles/crowdsec-edge-bouncer/**"
- "ansible/plays/install-netbird-proxy.yml"
# The variables that steer those roles, not just the roles themselves.
# Without these two, a change that only touches inventory never deploys
# and nothing says so: arming the bouncer lives in host_vars, and its
# LAPI key lives in group_vars, so both would sit in git looking applied
# while the hosts kept running the previous configuration.
#
# pve_hosts.yml is deliberately NOT listed. It is the single inventory
# file for the whole fleet, so listing it would redeploy the proxies for
# any unrelated host edit. The cost is that a change to [netbird_proxy]
# group membership needs a manual deployment run.
- "ansible/inventory/host_vars/proxy-*.yml"
- "ansible/inventory/group_vars/netbird_proxy/**"
- ".woodpecker/deploy-netbird-proxy.yml"
- event: deployment
evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "deploy-netbird-proxy"'
@@ -1,19 +0,0 @@
---
# Arms the CrowdSec firewall bouncer on THIS host only.
#
# The role defaults to dry-run for the whole netbird_proxy group, so the two
# members can be armed one at a time. proxy-2 goes first deliberately: it is
# the dual-stack member, while proxy-1 is IPv6-only and the one where every
# network-shaped problem has surfaced first — the dhcpcd/resolv.conf breakage,
# and the GitHub download that could not work at all.
#
# What flipping this does: the bouncer stops logging what it would block and
# creates two nftables tables (ip crowdsec, ip6 crowdsec6) with the decisions
# it holds. It touches no existing chain and sets no chain policy.
#
# Rollback is `systemctl stop crowdsec-firewall-bouncer` — a clean stop deletes
# both tables itself. Setting this back to dry-run and re-running the play does
# the same through the handler.
#
# Remove this file once proxy-1 follows and the role default moves to nftables.
crowdsec_bouncer_mode: nftables
+11 -7
View File
@@ -81,19 +81,23 @@ spec:
# every exposed name, area and floor. At 16384 the request was
# refused outright.
#
# --ctx-size is per slot, not the total: /props reports
# n_ctx_slot equal to this value while running four slots, where a
# total would have divided down to 4096. Doubling it alone would
# therefore double the KV cache, so the slot count halves in step
# and the allocation stays where it is today — 24979 MiB of the
# 32Gi limit, most of it the 19.5 GiB of weights.
# --ctx-size is the total and llama.cpp divides it across the
# slots, so 65536 over two slots is 32768 each.
#
# It does not read that way without --parallel, and the reason is
# in `llama-server --help`: --kv-unified defaults to enabled only
# "if number of slots is auto". Leaving --parallel unset therefore
# bought a single KV buffer shared across sequences, where every
# slot could draw the full --ctx-size — which is why this server
# once reported n_ctx_slot 16384 across four slots. Naming a slot
# count turns that off and splits the context instead.
#
# Two slots is what is actually in use: one holds Home Assistant's
# prompt, one holds Paperless's. A third caller would queue rather
# than evict either, which is the better failure mode given a cold
# prefill of this prompt costs around two minutes.
- --ctx-size
- "32768"
- "65536"
- --parallel
- "2"
# A micro-batch of 2048 or more makes this Vulkan/Mesa stack fail
@@ -65,21 +65,6 @@ spec:
- 45.157.177.26/32
- 2a03:4000:4b:c1e:4474:43ff:fedf:8f85/128
- 2a03:4000:28:f6e:a4c7:59ff:fe1e:9a29/128
# netbird.renner.uno — management, signal, relay and TURN.
#
# Not optional once a bouncer is armed. Its chain sits on the INPUT
# hook, the proxy hosts run no other firewall, and there is therefore
# no conntrack rule accepting established flows ahead of it. A ban on
# this address would drop the RETURN traffic of the proxies' own
# outbound control connection and take them off the mesh — with no
# inbound attack involved at all.
#
# Checked when this was added: none of these addresses appear in the
# current decision set. That is the state today, not a guarantee —
# these are rented addresses on a hoster whose ranges do end up on
# community blocklists.
- 162.55.50.97/32
- 2a01:4f8:c2c:4579::1/128
# mTLS between agents and the LAPI. Do not switch back to password auth:
# the LAPI runs without persistence, so its machine-registration DB is