llama-server, llama-embed, wiki-embed (gpu.intel.com/i915) and the wiki
behind them are bound to the Intel iGPU. On a site without one (Hetzner,
restore rehearsal runs 27/28) their pods stayed Pending, and the GPU plugin
and exporter DaemonSets sat on every node with nothing to serve.
The node patches k8s-1..3 label their machines
hardware.renner.uno/intel-gpu=true. The GPU plugin, the exporter, the
three llama Deployments and the wiki now select on it. Elsewhere the
DaemonSets have no pods and the Deployments stay unscheduled.
Production rollout order: the label must be on the nodes (talosctl patch,
no reboot) before these manifests reconcile, or the GPU plugin and the
llama/wiki pods are removed or left Pending.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The LLM index rebuild ran for 79 minutes and then died:
input (1069 tokens) is too large to process.
increase the physical batch size (current batch size: 1024)
Setting the batch to Paperless's chunk size looked like an exact match
and was not. Paperless counts its 1024 tokens with llama_index's
tokenizer; this server measures the batch in bge-m3's, which is
XLM-RoBERTa and splits German into more pieces. A chunk Paperless
considered full arrived here 4.4% over the limit, and the request failed
rather than truncating.
1536 absorbs a tokenizer ratio rather than a rounding error. 2048 stays
out of reach: that is where the chat model hit ErrorDeviceLost on this
Vulkan stack, under a 20 GB model's memory pressure that a 600 MB one may
well not reproduce — but there is no reason to go looking.
The index is left partial by the failed run and has to be rebuilt with
`document_llmindex rebuild` once this is live.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
First candidate to be carried through to an end-to-end measurement. The
embedding comparison on 400 real archive documents puts bge-m3 level with the
deployed Qwen3-Embedding-0.6B where it matters and ahead everywhere else:
correspondent 74.0% vs 75.0% (ceiling 82.5%, so 90% vs 91% of achievable)
document type 70.0% vs 68.2%
time 147s vs 380s for 400 documents
peak memory 1.6 GiB vs 10.6 GiB
Same 1024 dimensions, so nothing downstream changes shape.
Note the pooling type differs — bge-m3 wants CLS where the Qwen family wants
last-token. Read from the model's own 1_Pooling/config.json rather than assumed;
across the candidates measured, three different pooling types were required, and
a wrong one produces plausible but untrained vectors with no error anywhere.
The memory limit is now measured rather than estimated. It also supersedes #745,
which raised the limit for the model being replaced: the footprint is dominated
by four slots at 4096 context plus the Vulkan working set, not by the weights —
which is why several candidates were OOMKilled at limits that looked generous
against their model size.
This is not a final answer. Every candidate has to go through the end-to-end
test (scripts/paperless-eval.sh) before the embedding choice means anything,
because nothing has yet verified that the retrieval metric predicts what
Paperless actually suggests. bge-m3 is simply the one to start with.
Changing paperless-config is also what lets that test run at all: the AI
settings from #739 have never reached the running pod, which predates the
Reloader annotation from #741. This edit gives Reloader something real to react
to, rather than forcing a restart by hand or inventing a change to trigger one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Paperless 3.x has LLM suggestions built in, so this is configuration rather
than a sidecar: no extra image, no second database, no Paperless API token to
mint. Both endpoints are ClusterIP Services, so document text never leaves the
cluster and there is no key to rotate.
The make-or-break requirement was verified against the live endpoint before any
of this was written. Paperless's openai-like client calls chat_with_tools with
tool_required=True and error_on_no_tool_call=True — there is no free-text JSON
fallback, so the model must emit real tool_calls. It does: finish_reason
"tool_calls", a populated tool_calls array, empty content. A German utility
bill came back tagged Rechnung / Strom / Versorger.
That same probe showed the tags in German but document_types in English, which
is the documented weak spot and exactly what OUTPUT_LANGUAGE's second
localisation pass addresses. The value must be "de-de": it is matched against
Django's settings.LANGUAGES, where a bare "de" does not resolve.
llama-embed needed three fixes to serve as the RAG embedder:
- --alias, so the model is not advertised as its whole --hf-repo string.
That string would otherwise sit verbatim in this ConfigMap and break on the
next model swap.
- --pooling last. Qwen3-Embedding is trained for it; unspecified,
llama-server takes the GGUF default, which is invisible from outside. Set
now because nothing is indexed yet — changing pooling later invalidates
every stored vector.
- --batch-size/--ubatch-size 1024. The logs showed llama-server clamping both
to 512 under --embeddings, below Paperless's 1024-token chunk size. 2048 is
avoided deliberately: that is where this Vulkan stack fails with
ErrorDeviceLost.
Both network policies are scoped to the two pods and the one port on each side,
so neither grant widens if something else is placed in either namespace.
REQUEST_TIMEOUT goes to 300s. At ~11 tok/s a classification is 2-4k prompt
tokens plus a second round trip for localisation, and the 120s default is
marginal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two defects surfaced once the endpoint was live, and both trace back to
router mode.
The embedding model was selectable in the chat UI. Every model on the port
appears in the picker, and choosing the embedding one returns fluent nonsense
("capital capital capital ...") rather than an error, because --embeddings does
not actually refuse chat completions. llama.cpp offers no way to hide a model
from the list: --tags is informational, --ui-config only sets UI preferences.
So the embedding model moves to its own Deployment on a ClusterIP-only Service
with --no-webui, which is what keeps it out of the picker.
Readiness was also wrong. The router answers /health 200 as soon as it listens,
while models still download behind it, so the pod went Ready seconds after start
and traffic arrived at a server that could not answer — that is how the
embedding model got reached at all. A single-model server returns 503 until the
weights are resident, which is what the probes needed all along.
Splitting fixes both, at the cost of router mode's single endpoint. Consumers
now use llama-server for chat and llama-embed for embeddings; both share one
PVC as LLAMA_CACHE, so nothing is downloaded twice.
The chat model changes to a mixture-of-experts. Llama-3.1-8B was picked only
because it was the model benchmarked, which is continuity rather than a quality
argument, and it showed: fluent but wrong. Since decode is set by bytes read per
token, an MoE reading ~1.85 GB/token beats a dense model reading ~16.5 GB.
Measured on k8s-1 with this image:
tg128 11.59 tok/s (dense 8B: 10.78)
pp512 120.93 t/s
pp2048 126.47 t/s
pp8192 92.79 t/s
pp16384 69.84 t/s
Four times the model, slightly faster. The prefill falloff is sublinear, so the
Gated Delta Net collapse of llama.cpp#27638 does not occur here and long
documents stay viable; context therefore goes to 16k, which is cheap because
only 10 of 40 layers use full attention.
GLM-5.2/5.3 were evaluated and are impossible on this hardware at 754B
parameters (~400 GB), as is GLM-5.3-Flash at 321B with an architecture the
pinned build does not implement. Dense Qwen3.8-27B reads 16.5 GB/token and
manages 3.3 tok/s.
Memory goes to 24Gi/32Gi: Vulkan allocates the iGPU working set from system RAM
and it counts against the pod cgroup, measured at 7.54 GiB RSS for 5.2 GiB of
weights. --ubatch-size 1024 because 2048 and above fail with ErrorDeviceLost on
this Vulkan/Mesa stack.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
claude-botandClaude Opus 5.5
renovate-bot
k.renner
issue-fix-bot