4 Commits

Include renames

Author	SHA1	Message	Date
claude-botandClaude Sonnet 5.5	cb6f6e803a	feat(embed-eval): paired comparison of two kNN benchmark runs ... Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>	2026-10-10 20:35:54 +02:00
claude-botandClaude Sonnet 5.5	775419c2a4	feat(embed-eval): shorten texts the model reports as over its token limit ... Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>	2026-10-10 20:30:13 +02:00
claude-botandClaude Sonnet 5.5	daf55565ee	feat(embed-eval): leave-one-out kNN benchmark for embedding models ... Embeds a labelled Paperless corpus and scores nearest-neighbour correspondent and document-type accuracy against ceiling and floor. Standard library only. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>	2026-10-10 20:23:48 +02:00
claude-botandClaude Opus 5	18d3f3c850	feat(scripts): retrieval eval for embedding models on German documents ... Companion to ai-eval.sh, for the half of the stack that was never measured. The chat model was chosen from a four-way benchmark; the embedder was taken from a first recommendation and deployed unexamined. It now backs Paperless's RAG over a German archive, where retrieval quality decides which past documents the classifier gets to see. It cannot work like ai-eval.sh. A chat answer can be read and judged; an embedding is 1024 opaque numbers. What is measurable is whether the right document comes back first, so this embeds a labelled German corpus and scores real questions against it: recall@1, recall@3, MRR, and the cosine margin between the correct document and the best wrong one — a model that is right by 0.002 is guessing, one that is right by 0.15 has actually separated them. The corpus is built to be failable. Two Allianz policies, four documents that are all Rechnungen, a car insurance and a vehicle registration sharing a number plate. Queries are paraphrases rather than keyword matches — "Wann muss mein Auto zum TUEV" against a document that only ever says Hauptuntersuchung. An easy corpus would score every model 100% and rank nothing. Baseline for the deployed model: RESULT model=qwen3-embedding-0.6b dim=1024 recall@1=92% recall@3=100% mrr=0.958 The single miss is the intended hard case — the car-insurance question lands on the household policy, rank 2, margin -0.0132. The margin column also exposes what pass/fail would hide: arbeitsvertrag is won by +0.0059, a coin flip recorded as a hit. Reaches llama-embed from inside the cluster by default, since that Service has no ingress by design. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 07:28:31 +02:00