claude-botandClaude Opus 5
18d3f3c850
Companion to ai-eval.sh, for the half of the stack that was never measured. The chat model was chosen from a four-way benchmark; the embedder was taken from a first recommendation and deployed unexamined. It now backs Paperless's RAG over a German archive, where retrieval quality decides which past documents the classifier gets to see. It cannot work like ai-eval.sh. A chat answer can be read and judged; an embedding is 1024 opaque numbers. What is measurable is whether the right document comes back first, so this embeds a labelled German corpus and scores real questions against it: recall@1, recall@3, MRR, and the cosine margin between the correct document and the best wrong one — a model that is right by 0.002 is guessing, one that is right by 0.15 has actually separated them. The corpus is built to be failable. Two Allianz policies, four documents that are all Rechnungen, a car insurance and a vehicle registration sharing a number plate. Queries are paraphrases rather than keyword matches — "Wann muss mein Auto zum TUEV" against a document that only ever says Hauptuntersuchung. An easy corpus would score every model 100% and rank nothing. Baseline for the deployed model: RESULT model=qwen3-embedding-0.6b dim=1024 recall@1=92% recall@3=100% mrr=0.958 The single miss is the intended hard case — the car-insurance question lands on the household policy, rank 2, margin -0.0132. The margin column also exposes what pass/fail would hide: arbeitsvertrag is won by +0.0059, a coin flip recorded as a hit. Reaches llama-embed from inside the cluster by default, since that Service has no ingress by design. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>