claude-botandClaude Opus 5
1ec6825b66
The measurements were coupled. One request that corrupts the slot leaves it
returning a single repeated token for everything after it, so a run counted
that fault once per remaining document. "Qwen3-8B failed 4 of 5" was really
"document 1970 succeeded, document 363 broke the slot, and the remaining three
measured the wreckage".
Proved mid-run rather than inferred: with two documents failed, the model
answered "Nenne die Hauptstadt von Frankreich" with
" \" \" \" \" \" \" \" \" \" \" ...
llama-server clears a slot on POST /slots/0?action=erase, but answers 501
without --slot-save-path, so the flag goes on the sidecar and eval.py calls it
after every AI document. It costs the prefix cache and some prefill speed,
which is the right trade for numbers that mean what they say.
Also worth recording: a synthetic reproduction of the poisoning failed. The
real paperless request corrupts the slot; my own 5000-token probe with the same
14-field schema does not, and the canary stayed healthy through it. Whatever
triggers this lives in the parts of a real request I was not reproducing — the
system prompt, the RAG context, the full taxonomy lists. Third time today that
a synthesised test passed where the real one fails, so the harness should stop
trying to stand in for the real request and instead detect when it has gone
wrong, which is what the canary now does.
Re-screening qwen3-8b to find out whether it scores more than one of five once
the documents are actually independent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
374 lines
15 KiB
YAML
374 lines
15 KiB
YAML
---
|
|
# One configuration under test, complete in one pod.
|
|
#
|
|
# A paperless instance can only point at one chat endpoint, so measuring two
|
|
# candidates at once means two paperless instances — which makes the pod, not
|
|
# the namespace, the right unit. Everything a measurement needs is here except
|
|
# the database and the embedding endpoint, both of which are read-only and
|
|
# shared.
|
|
#
|
|
# The three services are native sidecars (initContainers with
|
|
# restartPolicy: Always), not regular containers. That matters twice: they
|
|
# start in order and each must pass its probes before the next begins, which
|
|
# is what keeps paperless from racing the database; and when the eval
|
|
# container exits they are torn down, so the Job actually completes instead of
|
|
# running forever behind a llama-server that never returns.
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata:
|
|
name: eval-__LABEL__
|
|
namespace: paperless-eval
|
|
labels:
|
|
app.kubernetes.io/name: paperless-eval
|
|
app.kubernetes.io/component: eval
|
|
spec:
|
|
# A failed run is re-read, not retried blind: a retry would spend another
|
|
# hour reproducing whatever went wrong.
|
|
backoffLimit: 0
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app.kubernetes.io/name: paperless-eval
|
|
app.kubernetes.io/component: eval
|
|
spec:
|
|
restartPolicy: Never
|
|
enableServiceLinks: false
|
|
# One measurement per node. Two jobs sharing a node would share its iGPU,
|
|
# and every timing in the result would then describe contention rather
|
|
# than the model — which is exactly how a decode rate of 4.7 tokens/s got
|
|
# attributed to context length once already.
|
|
affinity:
|
|
podAntiAffinity:
|
|
requiredDuringSchedulingIgnoredDuringExecution:
|
|
- labelSelector:
|
|
matchLabels:
|
|
app.kubernetes.io/component: eval
|
|
topologyKey: kubernetes.io/hostname
|
|
securityContext:
|
|
fsGroup: 1000
|
|
fsGroupChangePolicy: OnRootMismatch
|
|
seccompProfile:
|
|
type: RuntimeDefault
|
|
initContainers:
|
|
# A private copy of /data — the search index and, the expensive part,
|
|
# the LLM index the RAG path needs.
|
|
#
|
|
# It cannot be mounted live and shared: LLM_INDEX_DIR is hardcoded to
|
|
# DATA_DIR / "llm_index", the store behind it is SQLite, and SQLite
|
|
# locking across pods over CephFS is not dependable. Nor can it be left
|
|
# out — an empty llm_index makes retrieve_similar_nodes return nothing,
|
|
# and the job would quietly measure the no-RAG path while reporting a
|
|
# RAG result.
|
|
#
|
|
# ~160 MiB, so seconds. A regular initContainer: it must finish before
|
|
# the sidecars start.
|
|
- name: seed-data
|
|
image: ghcr.io/paperless-ngx/paperless-ngx:3.1.2
|
|
command:
|
|
- sh
|
|
- -c
|
|
- |
|
|
set -e
|
|
# -r, not -a: preserving ownership needs CHOWN, which this
|
|
# container does not have, and cp then exits non-zero after
|
|
# copying the files anyway. Running as 1000 makes the copy land
|
|
# with the ownership paperless needs — it writes to the LLM index
|
|
# while serving.
|
|
cp -r /golden/. /data/
|
|
du -sh /data/index /data/llm_index
|
|
# Refuse to start rather than measure the wrong thing. An empty
|
|
# llm_index makes retrieve_similar_nodes return nothing, and the
|
|
# run would report a RAG result having never retrieved anything —
|
|
# which is exactly what the first attempt at this job did, because
|
|
# the golden volume had not been populated yet. The index is
|
|
# ~125 MiB once built; anything under 1 MiB is not one.
|
|
bytes=$(du -sb /data/llm_index | cut -f1)
|
|
if [ "$bytes" -lt 1000000 ]; then
|
|
echo "FATAL: llm_index is ${bytes} bytes — build it on the" >&2
|
|
echo "smoke-test deployment first:" >&2
|
|
echo " kubectl -n paperless-eval exec deploy/paperless -- \\" >&2
|
|
echo " python manage.py document_llmindex rebuild" >&2
|
|
exit 1
|
|
fi
|
|
volumeMounts:
|
|
- name: golden
|
|
mountPath: /golden
|
|
readOnly: true
|
|
- name: data
|
|
mountPath: /data
|
|
securityContext:
|
|
runAsUser: 1000
|
|
runAsGroup: 1000
|
|
allowPrivilegeEscalation: false
|
|
capabilities:
|
|
drop: ["ALL"]
|
|
- name: llama-server
|
|
image: ghcr.io/ggml-org/llama.cpp:server-vulkan-b10731
|
|
restartPolicy: Always
|
|
args:
|
|
- --hf-repo
|
|
- __HF_REPO__
|
|
# The exact filename rather than the :QUANT shorthand. The
|
|
# shorthand matches by substring across the whole repo and repos do
|
|
# carry several files under one tag — ggml-org ships mmproj-, mtp-
|
|
# and dflash- variants at Q8_0 beside the model — so naming the file
|
|
# removes a real ambiguity and costs nothing.
|
|
#
|
|
# It is NOT what caused the three-hour crashloop that prompted this,
|
|
# and an earlier version of this comment claimed it was. The real
|
|
# cause was the llama-models volume being 100% full: llama-server
|
|
# could not write its metadata cache, so it could not resolve the
|
|
# repo listing, and reported
|
|
# no GGUF files found in repository <repo>
|
|
# followed by a truncated list of what it had cached before the
|
|
# disk filled. That reads as a bad reference and is a full disk.
|
|
#
|
|
# If a model will not load, check `df -h /models` before believing
|
|
# anything llama-server says about the repository.
|
|
- --hf-file
|
|
- __HF_FILE__
|
|
- --alias
|
|
- __ALIAS__
|
|
- --n-gpu-layers
|
|
- "99"
|
|
- --threads
|
|
- "4"
|
|
# The production ceiling. Not because the flag value itself breaks
|
|
# anything — bisection showed the corruption boundary is prompt
|
|
# length, around 14000 tokens, whatever --ctx-size says — but
|
|
# because measuring against a different ceiling than production
|
|
# runs would not describe production.
|
|
- --ctx-size
|
|
- "16384"
|
|
- --parallel
|
|
- "1"
|
|
- --ubatch-size
|
|
- "1024"
|
|
# The default is -1, infinity: nothing on the server caps how long
|
|
# a request generates, and paperless sets no budget of its own and
|
|
# waits. A classification that needs ~300 tokens ran to 10269 at a
|
|
# healthy 11 tok/s, and three of the first eleven documents did it
|
|
# — 900 s each, most of an hour spent on four documents.
|
|
#
|
|
# 768 is generous against the ~300 a correct answer takes. Note
|
|
# that paperless reports a truncated response as a flat
|
|
# "Invalid AI configuration" and discards the content, so what the
|
|
# model was actually doing is visible only in this container's log.
|
|
- --n-predict
|
|
- "__N_PREDICT__"
|
|
# Without this, Qwen3.6 spends the response on reasoning_content
|
|
# and returns empty content — 983 tokens into a classification that
|
|
# needs a hundred, at a perfectly healthy 11 tok/s. It reads as a
|
|
# slow or broken model and is neither.
|
|
#
|
|
# NOT --chat-template-kwargs '{"enable_thinking": false}'. That is
|
|
# deprecated, llama-server says so at startup, and it is not the
|
|
# harmless no-op it looks like on a template that never reads the
|
|
# variable: injecting it into Mistral's render context produced
|
|
# token salad — "1UnComplete1Self1UnCompleteSDirect11S..." — on a
|
|
# 681-token prompt, while the identical request against a server
|
|
# without the flag returned a clean tool call. It reads exactly
|
|
# like a broken model and is a broken prompt.
|
|
- --reasoning
|
|
- "off"
|
|
# Enables POST /slots/0?action=erase, which the eval calls between
|
|
# documents. Without it the endpoint answers 501.
|
|
#
|
|
# This is what makes the measurements independent. A request that
|
|
# corrupts the slot otherwise takes every document after it, so a
|
|
# run reports N failures for one fault — "4 of 5 failed" was really
|
|
# "document 2 broke the slot". Erasing costs the prefix cache and
|
|
# some prefill speed, which is the right trade for numbers that
|
|
# mean what they say.
|
|
- --slot-save-path
|
|
- /tmp/slots
|
|
- --no-webui
|
|
env:
|
|
- name: LLAMA_CACHE
|
|
value: /models
|
|
ports:
|
|
- name: llm
|
|
containerPort: 8080
|
|
volumeMounts:
|
|
- name: models
|
|
mountPath: /models
|
|
# 60 minutes. Sized for a FIRST DOWNLOAD from HuggingFace, not for
|
|
# reading a warm PVC — an earlier version allowed 20 minutes on the
|
|
# latter reasoning and killed GLM-4.7-Flash mid-download at 17.8 GB.
|
|
# The container then restarts and begins again, so too small a budget
|
|
# is not "slow start", it is a loop that never finishes.
|
|
#
|
|
# The kill shows up as exitCode 137 with reason "Error" rather than
|
|
# OOMKilled, which is how to tell a probe kill from a memory limit.
|
|
startupProbe:
|
|
httpGet:
|
|
path: /health
|
|
port: llm
|
|
periodSeconds: 20
|
|
failureThreshold: 180
|
|
resources:
|
|
requests:
|
|
cpu: "1"
|
|
memory: __MEM_REQ__
|
|
gpu.intel.com/i915: "1"
|
|
limits:
|
|
memory: __MEM_LIM__
|
|
gpu.intel.com/i915: "1"
|
|
securityContext:
|
|
allowPrivilegeEscalation: false
|
|
capabilities:
|
|
drop: ["ALL"]
|
|
- name: redis
|
|
# Its own, not the namespace's: two paperless instances sharing a
|
|
# celery broker would take each other's tasks.
|
|
image: redis:8.10.1-alpine
|
|
restartPolicy: Always
|
|
args: ["redis-server", "--save", "", "--loglevel", "warning"]
|
|
startupProbe:
|
|
tcpSocket:
|
|
port: 6379
|
|
periodSeconds: 2
|
|
failureThreshold: 30
|
|
resources:
|
|
requests:
|
|
cpu: 20m
|
|
memory: 64Mi
|
|
limits:
|
|
memory: 256Mi
|
|
securityContext:
|
|
allowPrivilegeEscalation: false
|
|
readOnlyRootFilesystem: true
|
|
capabilities:
|
|
drop: ["ALL"]
|
|
- name: paperless
|
|
image: ghcr.io/paperless-ngx/paperless-ngx:3.1.2
|
|
restartPolicy: Always
|
|
envFrom:
|
|
- configMapRef:
|
|
name: paperless-eval-config
|
|
env:
|
|
# Overrides the namespace ConfigMap: this pod's own llama-server,
|
|
# and its own redis rather than the shared Service.
|
|
- name: PAPERLESS_AI_LLM_ENDPOINT
|
|
value: http://127.0.0.1:8080/v1
|
|
- name: PAPERLESS_AI_LLM_MODEL
|
|
value: __ALIAS__
|
|
- name: PAPERLESS_REDIS
|
|
value: redis://127.0.0.1:6379
|
|
- name: PAPERLESS_SECRET_KEY
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: paperless-eval-secrets
|
|
key: secret-key
|
|
- name: PAPERLESS_DBUSER
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: pprleval-pg-app
|
|
key: username
|
|
- name: PAPERLESS_DBPASS
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: pprleval-pg-app
|
|
key: password
|
|
ports:
|
|
- name: http
|
|
containerPort: 8000
|
|
volumeMounts:
|
|
- name: data
|
|
mountPath: /usr/src/paperless/data
|
|
- name: media
|
|
mountPath: /usr/src/paperless/media
|
|
- name: consume
|
|
mountPath: /usr/src/paperless/consume
|
|
startupProbe:
|
|
httpGet:
|
|
path: /accounts/login/
|
|
port: http
|
|
periodSeconds: 20
|
|
failureThreshold: 180
|
|
# Same as production, and required: the entrypoint starts as root,
|
|
# chowns the volumes and drops to uid 1000 via setgid/setuid.
|
|
securityContext:
|
|
runAsNonRoot: false
|
|
allowPrivilegeEscalation: true
|
|
capabilities:
|
|
drop: ["ALL"]
|
|
add: ["CHOWN", "DAC_OVERRIDE", "FOWNER", "SETGID", "SETUID"]
|
|
resources:
|
|
requests:
|
|
cpu: 200m
|
|
memory: 512Mi
|
|
limits:
|
|
memory: 2Gi
|
|
containers:
|
|
- name: eval
|
|
# The paperless image again, for psycopg and a python that matches
|
|
# the application — not because the script needs paperless itself.
|
|
image: ghcr.io/paperless-ngx/paperless-ngx:3.1.2
|
|
command: ["python3", "/eval/eval.py"]
|
|
env:
|
|
- name: EVAL_LABEL
|
|
value: __LABEL__
|
|
- name: EVAL_SAMPLE
|
|
value: "__SAMPLE__"
|
|
# Raised alongside --n-predict for a diagnostic run. The two bounds
|
|
# interact: lifting the token cap alone just moves the failure from
|
|
# "hit the cap" to "hit the clock", which is what happened the first
|
|
# time gemma-4 was retried at 4096.
|
|
- name: EVAL_TIMEOUT
|
|
value: "__TIMEOUT__"
|
|
- name: PGHOST
|
|
value: pprleval-pg-rw
|
|
- name: PGDATABASE
|
|
value: paperless
|
|
- name: PGUSER
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: pprleval-pg-app
|
|
key: username
|
|
- name: PGPASSWORD
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: pprleval-pg-app
|
|
key: password
|
|
volumeMounts:
|
|
- name: script
|
|
mountPath: /eval
|
|
- name: results
|
|
mountPath: /results
|
|
resources:
|
|
requests:
|
|
cpu: 100m
|
|
memory: 256Mi
|
|
limits:
|
|
memory: 512Mi
|
|
securityContext:
|
|
allowPrivilegeEscalation: false
|
|
capabilities:
|
|
drop: ["ALL"]
|
|
volumes:
|
|
- name: models
|
|
persistentVolumeClaim:
|
|
claimName: llama-models
|
|
- name: results
|
|
persistentVolumeClaim:
|
|
claimName: paperless-eval-results
|
|
- name: script
|
|
configMap:
|
|
name: paperless-eval-script
|
|
# The namespace's /data, mounted read-only and copied out by seed-data
|
|
# above. Never mounted as the live /data of a job.
|
|
- name: golden
|
|
persistentVolumeClaim:
|
|
claimName: paperless-eval-golden
|
|
readOnly: true
|
|
# paperless insists on all three paths existing. media and consume stay
|
|
# empty — nothing is consumed during a measurement — while data is
|
|
# seeded with the copy.
|
|
- name: data
|
|
emptyDir: {}
|
|
- name: media
|
|
emptyDir: {}
|
|
- name: consume
|
|
emptyDir: {}
|