feat(embed-eval): leave-one-out kNN benchmark for embedding models #1237

2026-10-10T18:43:28Z

claude-bot commented

2026-10-10 18:43:28 +00:00

Two stdlib-only scripts for comparing embedding models on a labelled Paperless corpus.

knn-bench.py embeds a JSONL corpus (id, correspondent, doctype, text) and scores leave-one-out nearest-neighbour accuracy per label field, with ceiling and floor. Works against a direct OpenAI-style endpoint or through a pod (--via-pod) for ClusterIP-only servers. Texts the API reports as over its token limit are shortened to fit.
knn-compare.py compares two cached runs document by document: outcome overlap, exact McNemar test, share of identical nearest neighbours.

The corpus is exported read-only from the paperless database and is not part of the repo; the export query is in the knn-bench.py docstring.

Used on 400 real documents: bge-m3 and mistral-embed were statistically indistinguishable (McNemar p=0.85 correspondent, p=0.27 document type), so embeddings stay local.

🤖 Generated with Claude Code

Two stdlib-only scripts for comparing embedding models on a labelled Paperless corpus. - knn-bench.py embeds a JSONL corpus (id, correspondent, doctype, text) and scores leave-one-out nearest-neighbour accuracy per label field, with ceiling and floor. Works against a direct OpenAI-style endpoint or through a pod (--via-pod) for ClusterIP-only servers. Texts the API reports as over its token limit are shortened to fit. - knn-compare.py compares two cached runs document by document: outcome overlap, exact McNemar test, share of identical nearest neighbours. The corpus is exported read-only from the paperless database and is not part of the repo; the export query is in the knn-bench.py docstring. Used on 400 real documents: bge-m3 and mistral-embed were statistically indistinguishable (McNemar p=0.85 correspondent, p=0.27 document type), so embeddings stay local. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 3 commits 2026-10-10 18:43:29 +00:00

feat(embed-eval): leave-one-out kNN benchmark for embedding models daf55565ee

Embeds a labelled Paperless corpus and scores nearest-neighbour correspondent and document-type accuracy against ceiling and floor. Standard library only.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

feat(embed-eval): shorten texts the model reports as over its token limit 775419c2a4

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

feat(embed-eval): paired comparison of two kNN benchmark runs

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

cb6f6e803a

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

k.renner merged commit 358ae278d4 into master

2026-10-10 20:00:49 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#1237