feat(scripts): retrieval eval for embedding models on German documents

Companion to ai-eval.sh, for the half of the stack that was never measured. The chat model was chosen from a four-way benchmark; the embedder was taken from a first recommendation and deployed unexamined. It now backs Paperless's RAG over a German archive, where retrieval quality decides which past documents the classifier gets to see. It cannot work like ai-eval.sh. A chat answer can be read and judged; an embedding is 1024 opaque numbers. What is measurable is whether the right document comes back first, so this embeds a labelled German corpus and scores real questions against it: recall@1, recall@3, MRR, and the cosine margin between the correct document and the best wrong one — a model that is right by 0.002 is guessing, one that is right by 0.15 has actually separated them. The corpus is built to be failable. Two Allianz policies, four documents that are all Rechnungen, a car insurance and a vehicle registration sharing a number plate. Queries are paraphrases rather than keyword matches — "Wann muss mein Auto zum TUEV" against a document that only ever says Hauptuntersuchung. An easy corpus would score every model 100% and rank nothing. Baseline for the deployed model: RESULT model=qwen3-embedding-0.6b dim=1024 recall@1=92% recall@3=100% mrr=0.958 The single miss is the intended hard case — the car-insurance question lands on the household policy, rank 2, margin -0.0132. The margin column also exposes what pass/fail would hide: arbeitsvertrag is won by +0.0059, a coin flip recorded as a hit. Reaches llama-embed from inside the cluster by default, since that Service has no ingress by design. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-02 07:28:31 +02:00
10 changed files with 233 additions and 61 deletions
@@ -2,12 +2,7 @@

 Converts one MS-01 from "Proxmox host running a Talos VM" to "Talos on bare
 metal", and swaps its Ceph OSD onto a Samsung PM983 at the same time. Written
-after doing jupiter on 2026-08-25.
-
-All three nodes are now on bare metal — `talosctl get systeminformation`
-reports Minisforum MS-01 hardware with distinct chassis serials for k8s-1,
-k8s-2 and k8s-3, with no hypervisor in between. Kept as the procedure for a
-rebuild, not as outstanding work.
+after doing jupiter on 2026-08-25; saturn and neptune remain.

 Everything here that reads like an over-specific warning is something that
 actually happened. The conversion itself is straightforward — the failures were
@@ -111,23 +111,9 @@ statistics             max_id=13323904    max_start_ts=1787162400.0
 statistics_short_term  max_id=159205762   max_start_ts=1787166000.0
 ```

-## Phase 2c — cutover: DONE
+## Phase 2c — cutover (REMAINING)

-Verified 2026-09-02 against the live database:
-
-| table | rows | span |
-|---|---|---|
-| `statistics` | 11,092,586 | 2024-03-13 20:00 → current |
-| `statistics_short_term` | 1,883,709 | rolling ~7 days |
-| `statistics_meta` | 1,480 | |
-
-The 2024-03-13 origin is the proof that matters. It is unchanged from the
-pre-migration count in "What is at risk" above, so the history was copied
-across engines rather than started from zero — which is the failure this
-document exists to prevent. `states` carries current rows, so `db_url` points
-at Postgres.
-
-The procedure below is kept as the record of what was done.
+Do not start until phase 1 is verified.

 **Migrate statistics only.** Copying all 18M `states` rows is slow, needs a long
 HA outage, and buys a table that is discarded within 7 days anyway.
@@ -56,9 +56,10 @@ spec:
        # Re-rendered on every pod start, so a CNPG password rotation takes
        # effect rather than silently leaving a stale URI behind.
        #
-        # Load-bearing: `db_url: !secret recorder_db_url` in the homeassistant
-        # repo's configuration.yaml reads this key, so HA does not start
-        # without it. Removing this init container takes the recorder down.
+        # INERT until the recorder is repointed: nothing reads this key until
+        # `db_url: !secret recorder_db_url` lands in configuration.yaml in the
+        # homeassistant repo. That ordering is required — HA refuses to start
+        # when a referenced secret is missing.
        - name: render-db-url
          image: busybox:1.36
          command:
@@ -18,12 +18,12 @@
 # That is the point of the move — it attacks the multiplier rather than trimming
 # what gets recorded, which would only hide the problem.
 #
-# This is the live recorder database. HA writes here; the SQLite file is no
-# longer the system of record. The 2.4 years of long-term statistics were
-# copied across engines rather than regenerated — HA does not migrate data
-# between database engines, so an empty Postgres would have silently started
-# from zero and taken the energy dashboard's history with it.
-# MIGRATION-recorder-postgres.md holds the sequence that was followed.
+# STAGING
+# Phase 1 (this manifest): the cluster comes up EMPTY and HA keeps using SQLite.
+# Nothing depends on it, so a failure here is inert. Verify Ready, WAL archiving
+# to Hetzner, and a successful base backup before going further.
+# Phase 2: migrate the statistics tables and repoint recorder.db_url.
+# See MIGRATION-recorder-postgres.md for the sequence and the rollback.
 apiVersion: postgresql.cnpg.io/v1
 kind: Cluster
 metadata:
@@ -60,8 +60,9 @@ resources:
  - zigbee2mqtt-gate-service.yaml
  - zigbee2mqtt-ingress.yaml
  # Postgres backend for the recorder (MIGRATION-recorder-postgres.md).
-  # Live: recorder.db_url in the homeassistant repo points here, and the
-  # long-term statistics carry their full history from 2024.
+  # PHASE 1: this cluster comes up EMPTY and nothing consumes it — HA is still
+  # on SQLite. Deploying it is therefore inert; phase 2 migrates the statistics
+  # tables and repoints recorder.db_url in the homeassistant repo.
  - ha-pg-hetzner-creds-externalsecret.yaml
  - ha-pg-objectstore.yaml
  - ha-pg-cluster.yaml
@@ -10,15 +10,15 @@ metadata:
    # Kept permanently as a standing guard.
    kustomize.toolkit.fluxcd.io/prune: disabled
 spec:
-  # A replica is viable because nothing here writes pg_vectors any more. Both
-  # vector indexes run on vchordrq and the vectors extension is gone, so the
-  # long pg_vectors/indexes/<uuid> paths are absent — pg_basebackup's tar
-  # format cannot represent them, and that is what blocks a replica join.
-  instances: 2
-  # See ak-pg cluster.yaml for rationale. The barman-plugin shutdown wedge
-  # (cloudnative-pg#9770) holds an instance up until SIGKILL, so bounding
-  # terminationGracePeriod (stopDelay) is what keeps a failover from waiting
-  # on it. smartShutdownTimeout keeps pod restarts quick.
+  # Single instance during pgvecto.rs → VectorChord migration window;
+  # pg_basebackup tar format chokes on the long pg_vectors/indexes/<uuid>
+  # paths, blocking replica join. Bump back to 2 once Immich's in-app
+  # vchord migration drops the vectors extension.
+  instances: 1
+  # See ak-pg cluster.yaml for rationale. Single-instance, so no failover, but
+  # the barman-plugin shutdown wedge (cloudnative-pg#9770) takes the whole DB
+  # down until SIGKILL — bounding terminationGracePeriod (stopDelay) matters
+  # even without a standby. smartShutdownTimeout keeps pod restarts quick.
  smartShutdownTimeout: 10
  stopDelay: 60
  imageName: ghcr.io/corentingiraud/cnpg-pgvector-vectorchord:16-migration
@@ -36,10 +36,8 @@ spec:
      database: immich
      owner: immich
  postgresql:
-    # vchord only. The image also ships pgvecto.rs (vectors.so), but its
-    # extension is not installed and preloading it would recreate pg_vectors
-    # in PGDATA for nothing. pgvector needs no preloading.
    shared_preload_libraries:
+      - vectors.so
      - vchord.so
    parameters:
      shared_buffers: "512MB"
@@ -1,20 +1,10 @@
 ---
-# Holds the GGUF weights, downloaded once by llama-server itself via --hf-repo.
-# Prune-disabled so retiring this Kustomization does not throw away tens of GiB
-# of model files that would have to be pulled from Hugging Face again.
+# Holds the GGUF weights, downloaded once by the Deployment's initContainer.
+# Prune-disabled so retiring this Kustomization does not throw away several
+# GiB of model files that would have to be pulled from Hugging Face again.
 #
-# ReadWriteMany so a second replica, a one-off benchmark Job, or a candidate
-# model being evaluated can mount the same weights; the models are read-only at
-# inference time.
-#
-# Sized well beyond the two deployed models. The cache is shared by every
-# candidate under evaluation (scripts/embed-eval-real.sh), and on a slow
-# connection the download is the long pole — a shared cache lets the large
-# candidates pull in the background while the small ones are already being
-# measured, and means a model is never fetched twice across runs.
-#
-# CephFS supports online expansion (allowVolumeExpansion on the storage class),
-# so growing this does not disturb the running pods.
+# ReadWriteMany so a second replica or a one-off benchmark Job can mount the
+# same weights; the models are read-only at inference time.
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
@@ -28,4 +18,4 @@ spec:
  storageClassName: ceph-filesystem
  resources:
    requests:
-      storage: 120Gi
+      storage: 40Gi
@@ -0,0 +1,177 @@
+#!/usr/bin/env bash
+# Retrieval evaluation for an embedding model, on German documents of the kind
+# this homelab actually files.
+#
+# Companion to scripts/ai-eval.sh, but it cannot work the same way. A chat
+# answer can be read and judged; an embedding is 1024 opaque numbers. What is
+# measurable is whether the right document comes back first, so this builds a
+# small labelled corpus, embeds it, and scores real questions against it.
+#
+# Metrics:
+#   Recall@1  the correct document ranked first — what Paperless's RAG actually
+#             feeds to the classifier
+#   Recall@3  it at least made the shortlist
+#   MRR       mean reciprocal rank, so near-misses are visible rather than
+#             collapsing to pass/fail
+#   margin    cosine gap between the correct document and the best wrong one.
+#             A model that is right by 0.002 is guessing; one that is right by
+#             0.15 has actually separated them.
+#
+# The corpus deliberately contains confusable pairs — two Allianz policies, four
+# documents that are all "Rechnungen", a car insurance and a vehicle
+# registration sharing a number plate. An easy corpus would rate everything 100%
+# and tell you nothing.
+#
+# Usage:
+#   scripts/embed-eval.sh                                    # cluster default
+#   scripts/embed-eval.sh -m some-model
+#   scripts/embed-eval.sh -e https://host/v1/embeddings -m model -k KEY
+#
+# Env equivalents: EMBED_EVAL_ENDPOINT, EMBED_EVAL_MODEL, EMBED_EVAL_KEY.
+#
+# To compare two models on our llama-embed, -m is not enough: swap --hf-repo in
+# kubernetes/apps/llama-server/embed.yaml, let it roll out, then re-run. Note
+# that changing the model also invalidates any vectors Paperless has stored, so
+# comparisons are best made before an index exists.
+#
+# Exit code is the number of queries whose correct document did not rank first.
+set -uo pipefail
+
+ENDPOINT="${EMBED_EVAL_ENDPOINT:-http://llama-embed.ai.svc.cluster.local:8080/v1/embeddings}"
+MODEL="${EMBED_EVAL_MODEL:-qwen3-embedding-0.6b}"
+APIKEY="${EMBED_EVAL_KEY:-}"
+# Run curl from inside this pod when the endpoint is a cluster-internal name.
+# llama-embed has no ingress by design, so there is nothing to reach from a
+# workstation.
+VIA_POD="${EMBED_EVAL_VIA_POD:-}"
+
+while [ $# -gt 0 ]; do
+    case "$1" in
+        -e|--endpoint) ENDPOINT="$2"; shift 2 ;;
+        -m|--model)    MODEL="$2";    shift 2 ;;
+        -k|--key)      APIKEY="$2";   shift 2 ;;
+        -p|--via-pod)  VIA_POD="$2";  shift 2 ;;
+        -h|--help)     sed -n '2,34p' "$0"; exit 0 ;;
+        *) echo "unknown argument: $1 (try --help)" >&2; exit 2 ;;
+    esac
+done
+
+HERE="$(cd "$(dirname "$0")" && pwd)"
+CASES="$HERE/embed-eval"
+WORK="$(mktemp -d)"
+trap 'rm -rf "$WORK"' EXIT
+
+for f in "$CASES/corpus.jsonl" "$CASES/queries.jsonl"; do
+    [ -r "$f" ] || { echo "missing fixture: $f" >&2; exit 2; }
+done
+
+# Default to reaching the Service from a pod that is allowed to, unless the
+# caller pointed us somewhere externally reachable.
+if [ -z "$VIA_POD" ] && [ "${ENDPOINT#*svc.cluster.local}" != "$ENDPOINT" ]; then
+    VIA_POD="deploy/llama-server -n ai"
+fi
+
+# embed <text> -> the vector as a JSON array on stdout
+embed() {
+    local payload out
+    payload=$(jq -nc --arg m "$MODEL" --arg t "$1" '{model:$m,input:$t}')
+    if [ -n "$VIA_POD" ]; then
+        # stderr is kept and surfaced on failure rather than discarded: a
+        # silently swallowed kubectl or curl error is indistinguishable from a
+        # model that returned nothing.
+        # shellcheck disable=SC2086
+        out=$(kubectl exec $VIA_POD -- curl -sS -m 120 \
+              -H 'Content-Type: application/json' \
+              ${APIKEY:+-H "Authorization: Bearer $APIKEY"} \
+              -d "$payload" "$ENDPOINT" 2>"$WORK/err.txt")
+    else
+        out=$(curl -sS -m 120 -H 'Content-Type: application/json' \
+              ${APIKEY:+-H "Authorization: Bearer $APIKEY"} \
+              -d "$payload" "$ENDPOINT")
+    fi
+    printf '%s' "$out" | jq -c '.data[0].embedding // empty'
+}
+
+echo "endpoint: $ENDPOINT"
+echo "model:    $MODEL"
+[ -n "$VIA_POD" ] && echo "via pod:  $VIA_POD"
+echo ""
+
+# ---- embed the corpus ------------------------------------------------------
+DOCS=$(wc -l < "$CASES/corpus.jsonl" | tr -d ' ')
+echo "embedding $DOCS documents..."
+: > "$WORK/vecs.jsonl"
+start=$(date +%s)
+while IFS= read -r line; do
+    id=$(printf '%s' "$line" | jq -r '.id')
+    text=$(printf '%s' "$line" | jq -r '.text')
+    vec=$(embed "$text")
+    if [ -z "$vec" ]; then
+        echo "FAILED to embed '$id' — endpoint returned no vector." >&2
+        [ -s "$WORK/err.txt" ] && { echo "stderr:" >&2; cat "$WORK/err.txt" >&2; }
+        exit 2
+    fi
+    jq -nc --arg id "$id" --argjson v "$vec" '{id:$id, vec:$v}' >> "$WORK/vecs.jsonl"
+done < "$CASES/corpus.jsonl"
+elapsed=$(( $(date +%s) - start ))
+DIM=$(head -1 "$WORK/vecs.jsonl" | jq '.vec | length')
+echo "  $DOCS documents, ${DIM} dimensions, ${elapsed}s"
+echo ""
+
+# ---- score the queries -----------------------------------------------------
+# Cosine similarity in jq. The vectors llama.cpp returns are normalised, but the
+# full form costs nothing and keeps this correct against servers that do not.
+rank() { # queryvec -> "id score" lines, best first
+    jq -s -r --argjson q "$1" '
+        def dot($a; $b): reduce range(0; $a|length) as $i (0; . + ($a[$i] * $b[$i]));
+        def norm($a): (reduce ($a[]) as $x (0; . + ($x * $x))) | sqrt;
+        map({id: .id, s: (dot(.vec; $q) / ((norm(.vec) * norm($q)) + 1e-12))})
+        | sort_by(-.s)[]
+        | "\(.id) \(.s)"
+    ' "$WORK/vecs.jsonl"
+}
+
+MISS=0; TOTAL=0; HIT1=0; HIT3=0; MRRSUM=0
+printf '%-46s %-22s %5s %8s\n' QUERY EXPECTED RANK MARGIN
+printf '%s\n' "------------------------------------------------------------------------------------"
+while IFS= read -r line; do
+    q=$(printf '%s' "$line" | jq -r '.q')
+    want=$(printf '%s' "$line" | jq -r '.want')
+    qvec=$(embed "$q")
+    if [ -z "$qvec" ]; then
+        echo "FAILED to embed query: $q" >&2
+        [ -s "$WORK/err.txt" ] && { echo "stderr:" >&2; cat "$WORK/err.txt" >&2; }
+        exit 2
+    fi
+    rank "$qvec" > "$WORK/ranked.txt"
+
+    pos=$(grep -n "^$want " "$WORK/ranked.txt" | cut -d: -f1)
+    pos=${pos:-999}
+    TOTAL=$((TOTAL + 1))
+    [ "$pos" -eq 1 ] && HIT1=$((HIT1 + 1)) || MISS=$((MISS + 1))
+    [ "$pos" -le 3 ] && HIT3=$((HIT3 + 1))
+    MRRSUM=$(awk -v s="$MRRSUM" -v p="$pos" 'BEGIN{printf "%.6f", s + 1/p}')
+
+    # Margin: the correct document's score minus the best incorrect one. On a
+    # miss this is negative, which says how badly it lost.
+    want_s=$(awk -v w="$want" '$1==w{print $2; exit}' "$WORK/ranked.txt")
+    best_other=$(awk -v w="$want" '$1!=w{print $2; exit}' "$WORK/ranked.txt")
+    margin=$(awk -v a="${want_s:-0}" -v b="${best_other:-0}" 'BEGIN{printf "%+.4f", a-b}')
+
+    if [ "$pos" -eq 1 ]; then mark=" "; else mark="!"; fi
+    printf '%s%-45.45s %-22s %5s %8s\n' "$mark" "$q" "$want" "$pos" "$margin"
+    if [ "$pos" -ne 1 ]; then
+        printf '   got instead: %s\n' "$(head -1 "$WORK/ranked.txt" | cut -d' ' -f1)"
+    fi
+done < "$CASES/queries.jsonl"
+
+R1=$(awk -v h="$HIT1" -v t="$TOTAL" 'BEGIN{printf "%.0f", 100*h/t}')
+R3=$(awk -v h="$HIT3" -v t="$TOTAL" 'BEGIN{printf "%.0f", 100*h/t}')
+MRR=$(awk -v s="$MRRSUM" -v t="$TOTAL" 'BEGIN{printf "%.3f", s/t}')
+
+echo ""
+echo "========================================"
+printf '  RESULT model=%s dim=%s recall@1=%s%% recall@3=%s%% mrr=%s\n' \
+    "$MODEL" "$DIM" "$R1" "$R3" "$MRR"
+echo "========================================"
+exit "$MISS"
@@ -0,0 +1,12 @@
+{"id":"stadtwerke-strom","text":"Stadtwerke Muenchen GmbH. Jahresrechnung Strom. Rechnungsnummer 4711. Abrechnungszeitraum 01.01.2025 bis 31.12.2025. Zaehlernummer 1EMH0012345678. Verbrauch 2450 kWh zum Arbeitspreis von 31,2 Cent je kWh. Grundpreis 9,90 EUR monatlich. Zu zahlender Betrag 812,40 EUR faellig am 15.02.2026."}
+{"id":"telekom-mobilfunk","text":"Telekom Deutschland GmbH. Ihre Mobilfunkrechnung fuer Januar. Kundennummer 0891234567. Rufnummer 0170 1234567. Tarif MagentaMobil M mit 20 GB Datenvolumen. Grundpreis 39,95 EUR. Verbindungen ins Ausland 4,20 EUR. Rechnungsbetrag 44,15 EUR wird per Lastschrift eingezogen."}
+{"id":"finanzamt-steuer","text":"Finanzamt Muenchen Abteilung II. Bescheid ueber Einkommensteuer und Solidaritaetszuschlag fuer 2025. Steuernummer 143/256/78901. Zu versteuerndes Einkommen 54.320 EUR. Festgesetzte Einkommensteuer 8.412 EUR. Bereits geleistete Vorauszahlungen 9.100 EUR. Erstattungsbetrag 688 EUR. Rechtsbehelfsbelehrung siehe Seite 3."}
+{"id":"allianz-hausrat","text":"Allianz Versicherungs-AG. Versicherungsschein Hausratversicherung. Policennummer HR-88231. Versicherte Wohnflaeche 96 Quadratmeter. Versicherungssumme 65.000 EUR. Eingeschlossen sind Feuer, Leitungswasser, Sturm und Einbruchdiebstahl. Fahrraddiebstahl bis 1.500 EUR mitversichert. Jahresbeitrag 214 EUR."}
+{"id":"allianz-kfz","text":"Allianz Versicherungs-AG. Beitragsrechnung Kraftfahrtversicherung. Policennummer KFZ-55190. Amtliches Kennzeichen M-AB 1234. Fahrzeug VW Golf VIII. Vollkasko mit 500 EUR Selbstbeteiligung, Teilkasko 150 EUR. Schadenfreiheitsklasse SF 12. Jahresbeitrag 638 EUR."}
+{"id":"mietvertrag","text":"Mietvertrag ueber Wohnraum. Vermieter Immobilien Wagner GbR, Mieter Konstantin Renner. Mietobjekt Dreizimmerwohnung im zweiten Obergeschoss, 78 Quadratmeter, Lindenstrasse 14. Mietbeginn 01.04.2024. Kaltmiete 1.180 EUR monatlich, Betriebskostenvorauszahlung 240 EUR. Kaution drei Nettokaltmieten."}
+{"id":"arbeitsvertrag","text":"Arbeitsvertrag zwischen der Muster Technik GmbH und dem Arbeitnehmer. Taetigkeit als Softwareentwickler ab 01.09.2024. Bruttojahresgehalt 78.000 EUR zahlbar in zwoelf Monatsraten. Woechentliche Arbeitszeit 40 Stunden. Urlaubsanspruch 30 Arbeitstage im Kalenderjahr. Probezeit sechs Monate."}
+{"id":"aok-krankenkasse","text":"AOK Bayern. Beitragsbescheid zur gesetzlichen Kranken- und Pflegeversicherung. Versichertennummer A123456789. Beitragspflichtige Einnahmen 4.500 EUR monatlich. Allgemeiner Beitragssatz 14,6 Prozent zuzueglich Zusatzbeitrag 1,7 Prozent. Pflegeversicherung 3,4 Prozent. Monatlicher Gesamtbeitrag 887,50 EUR."}
+{"id":"sparkasse-kontoauszug","text":"Stadtsparkasse Muenchen. Kontoauszug Nummer 3 zum Girokonto IBAN DE89 7015 0000 0012 3456 78. Alter Kontostand 2.340,12 EUR. Gutschrift Gehalt 4.210,00 EUR. Lastschrift Miete 1.420,00 EUR. Dauerauftrag Sparplan 300,00 EUR. Neuer Kontostand 4.830,12 EUR."}
+{"id":"amazon-bestellung","text":"Amazon EU S.a.r.l. Bestellbestaetigung. Bestellnummer 302-4455667-8899001. Artikel Logitech MX Master 3S Wireless Maus, Menge 1, Preis 89,99 EUR. Versand durch Amazon, Lieferung voraussichtlich am 04.09. Zahlungsart Kreditkarte. Rechnungsbetrag inklusive Mehrwertsteuer 89,99 EUR."}
+{"id":"handwerker-rechnung","text":"Sanitaer Bergmeister GmbH. Rechnung fuer erbrachte Leistungen. Rechnungsnummer 2026-0417. Austausch eines defekten Heizungsthermostats sowie Entlueftung von drei Heizkoerpern. Arbeitszeit 2,5 Stunden zu 68 EUR. Material 42,30 EUR. Nettobetrag 212,30 EUR, Umsatzsteuer 19 Prozent, Gesamtbetrag 252,64 EUR."}
+{"id":"kfz-zulassung","text":"Landeshauptstadt Muenchen Kreisverwaltungsreferat. Zulassungsbescheinigung Teil I. Amtliches Kennzeichen M-AB 1234. Fahrzeugklasse Personenkraftwagen. Hersteller Volkswagen, Handelsbezeichnung Golf. Erstzulassung 14.03.2022. Hubraum 1498 Kubikzentimeter, Leistung 110 kW. Naechste Hauptuntersuchung 03.2028."}
@@ -0,0 +1,12 @@
+{"q":"Wie hoch war mein Stromverbrauch im letzten Jahr?","want":"stadtwerke-strom","note":"paraphrase: asks about consumption, the document says Verbrauch kWh"}
+{"q":"Was kostet mein Handyvertrag monatlich?","want":"telekom-mobilfunk","note":"paraphrase: Handyvertrag vs Mobilfunkrechnung/Tarif"}
+{"q":"Bescheid vom Finanzamt ueber die Einkommensteuer","want":"finanzamt-steuer","note":"near-literal, should be easy"}
+{"q":"Welchen Versicherungsschutz habe ich fuer mein Auto?","want":"allianz-kfz","note":"HARD: two Allianz policies. Must not pick the Hausrat one."}
+{"q":"Ist mein Fahrrad gegen Diebstahl versichert?","want":"allianz-hausrat","note":"HARD: same sender as kfz, and bicycles are a detail deep in the text"}
+{"q":"Wie hoch ist die Kaltmiete meiner Wohnung?","want":"mietvertrag","note":"must beat sparkasse, which also mentions Miete as a Lastschrift"}
+{"q":"Wie viele Urlaubstage stehen mir zu?","want":"arbeitsvertrag","note":"paraphrase: Urlaubsanspruch"}
+{"q":"Was zahle ich monatlich an die Krankenkasse?","want":"aok-krankenkasse","note":"Krankenkasse vs Kranken- und Pflegeversicherung"}
+{"q":"Wie viel Geld ist auf meinem Girokonto?","want":"sparkasse-kontoauszug","note":"Kontostand"}
+{"q":"Rechnung fuer die Reparatur der Heizung","want":"handwerker-rechnung","note":"HARD: several documents are Rechnungen; must pick the Sanitaer one"}
+{"q":"Wann muss mein Auto zum TUEV?","want":"kfz-zulassung","note":"HARD: TUEV never appears; the document says Hauptuntersuchung. Pure semantics, and must not be pulled to the Kfz-Versicherung by the shared plate number."}
+{"q":"Wann wurde meine Maus geliefert?","want":"amazon-bestellung","note":"colloquial vs Bestellbestaetigung/Lieferung"}