claude-botandClaude Opus 5
18d3f3c850
Companion to ai-eval.sh, for the half of the stack that was never measured. The chat model was chosen from a four-way benchmark; the embedder was taken from a first recommendation and deployed unexamined. It now backs Paperless's RAG over a German archive, where retrieval quality decides which past documents the classifier gets to see. It cannot work like ai-eval.sh. A chat answer can be read and judged; an embedding is 1024 opaque numbers. What is measurable is whether the right document comes back first, so this embeds a labelled German corpus and scores real questions against it: recall@1, recall@3, MRR, and the cosine margin between the correct document and the best wrong one — a model that is right by 0.002 is guessing, one that is right by 0.15 has actually separated them. The corpus is built to be failable. Two Allianz policies, four documents that are all Rechnungen, a car insurance and a vehicle registration sharing a number plate. Queries are paraphrases rather than keyword matches — "Wann muss mein Auto zum TUEV" against a document that only ever says Hauptuntersuchung. An easy corpus would score every model 100% and rank nothing. Baseline for the deployed model: RESULT model=qwen3-embedding-0.6b dim=1024 recall@1=92% recall@3=100% mrr=0.958 The single miss is the intended hard case — the car-insurance question lands on the household policy, rank 2, margin -0.0132. The margin column also exposes what pass/fail would hide: arbeitsvertrag is won by +0.0059, a coin flip recorded as a hit. Reaches llama-embed from inside the cluster by default, since that Service has no ingress by design. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
13 lines
1.7 KiB
JSON
13 lines
1.7 KiB
JSON
{"q":"Wie hoch war mein Stromverbrauch im letzten Jahr?","want":"stadtwerke-strom","note":"paraphrase: asks about consumption, the document says Verbrauch kWh"}
|
|
{"q":"Was kostet mein Handyvertrag monatlich?","want":"telekom-mobilfunk","note":"paraphrase: Handyvertrag vs Mobilfunkrechnung/Tarif"}
|
|
{"q":"Bescheid vom Finanzamt ueber die Einkommensteuer","want":"finanzamt-steuer","note":"near-literal, should be easy"}
|
|
{"q":"Welchen Versicherungsschutz habe ich fuer mein Auto?","want":"allianz-kfz","note":"HARD: two Allianz policies. Must not pick the Hausrat one."}
|
|
{"q":"Ist mein Fahrrad gegen Diebstahl versichert?","want":"allianz-hausrat","note":"HARD: same sender as kfz, and bicycles are a detail deep in the text"}
|
|
{"q":"Wie hoch ist die Kaltmiete meiner Wohnung?","want":"mietvertrag","note":"must beat sparkasse, which also mentions Miete as a Lastschrift"}
|
|
{"q":"Wie viele Urlaubstage stehen mir zu?","want":"arbeitsvertrag","note":"paraphrase: Urlaubsanspruch"}
|
|
{"q":"Was zahle ich monatlich an die Krankenkasse?","want":"aok-krankenkasse","note":"Krankenkasse vs Kranken- und Pflegeversicherung"}
|
|
{"q":"Wie viel Geld ist auf meinem Girokonto?","want":"sparkasse-kontoauszug","note":"Kontostand"}
|
|
{"q":"Rechnung fuer die Reparatur der Heizung","want":"handwerker-rechnung","note":"HARD: several documents are Rechnungen; must pick the Sanitaer one"}
|
|
{"q":"Wann muss mein Auto zum TUEV?","want":"kfz-zulassung","note":"HARD: TUEV never appears; the document says Hauptuntersuchung. Pure semantics, and must not be pulled to the Kfz-Versicherung by the shared plate number."}
|
|
{"q":"Wann wurde meine Maus geliefert?","want":"amazon-bestellung","note":"colloquial vs Bestellbestaetigung/Lieferung"}
|