Companion to ai-eval.sh, for the half of the stack that was never measured. The
chat model was chosen from a four-way benchmark; the embedder was taken from a
first recommendation and deployed unexamined. It now backs Paperless's RAG over
a German archive, where retrieval quality decides which past documents the
classifier gets to see.
It cannot work like ai-eval.sh. A chat answer can be read and judged; an
embedding is 1024 opaque numbers. What is measurable is whether the right
document comes back first, so this embeds a labelled German corpus and scores
real questions against it: recall@1, recall@3, MRR, and the cosine margin
between the correct document and the best wrong one — a model that is right by
0.002 is guessing, one that is right by 0.15 has actually separated them.
The corpus is built to be failable. Two Allianz policies, four documents that
are all Rechnungen, a car insurance and a vehicle registration sharing a number
plate. Queries are paraphrases rather than keyword matches — "Wann muss mein
Auto zum TUEV" against a document that only ever says Hauptuntersuchung. An easy
corpus would score every model 100% and rank nothing.
Baseline for the deployed model:
RESULT model=qwen3-embedding-0.6b dim=1024 recall@1=92% recall@3=100% mrr=0.958
The single miss is the intended hard case — the car-insurance question lands on
the household policy, rank 2, margin -0.0132. The margin column also exposes
what pass/fail would hide: arbeitsvertrag is won by +0.0059, a coin flip
recorded as a hit.
Reaches llama-embed from inside the cluster by default, since that Service has
no ingress by design.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
