feat(llama-embed): switch to bge-m3, activating the Paperless AI config #757

2026-09-02T11:40:13Z

claude-bot commented

2026-09-02 11:40:13 +00:00

First candidate carried through to an end-to-end measurement — and the change that makes that measurement possible at all.

The comparison, on 400 real archive documents

	bge-m3	Qwen3-Embedding-0.6B (deployed)
correspondent, nearest neighbour	74.0% (90% of achievable)	75.0% (91%)
document type, nearest neighbour	70.0%	68.2%
time for 400 documents	147 s	380 s
peak memory under load	1.6 GiB	10.6 GiB
dimensions	1024	1024

Level where it matters, ahead everywhere else, and same vector width so nothing downstream changes shape.

Pooling differs — and that is the trap

bge-m3 wants CLS pooling where the Qwen family wants last-token. Read from the model's own 1_Pooling/config.json, not assumed: across the candidates measured, three different pooling types were required. A wrong one produces plausible but untrained vectors, with no error message anywhere.

The memory limit is measured now

1.6 GiB peak (Prometheus max_over_time on the working set), limit set to 8Gi.

This supersedes #745, which raised the limit for the model being replaced. The lesson from that OOM holds regardless of model: the footprint is dominated by four slots at 4096 context plus the Vulkan working set, not by the weights. The 600 MiB Qwen model peaked at 10.6 GiB on identical settings, and several candidates were OOMKilled at limits that looked generous against their parameter count.

This is not the final answer

Every candidate still has to go through scripts/paperless-eval.sh (#756) before the embedding choice means anything. Nothing has yet verified that the retrieval metric predicts what Paperless actually suggests — and the last two times an upstream proxy was trusted, MTEB turned out to be anti-correlated with the real measurement. bge-m3 is simply the one to start with.

Why this unblocks the test

The AI settings from #739 have never reached the running pod: it predates the Reloader annotation added in #741, and Reloader does not act retroactively. Editing paperless-config gives it something real to react to — rather than an imperative rollout restart, or a change invented purely to force one.

Verification after merge

kubectl -n ai get pods -l app.kubernetes.io/name=llama-embed        # restarted, RESTARTS 0
kubectl -n ai exec deploy/llama-server -- curl -s \
  http://llama-embed.ai.svc.cluster.local:8080/v1/models | jq '.data[0].id'
kubectl -n paperless exec deploy/paperless -c paperless -- env | grep -c PAPERLESS_AI

Expect bge-m3, a restarted paperless pod, and 11 AI variables present — the last of which has been the missing precondition all along.

Then scripts/paperless-eval.sh --engine classic gives the baseline the LLM has to beat, and the full run follows once the LLM index is built.

🤖 Generated with Claude Code

First candidate carried through to an end-to-end measurement — and the change that makes that measurement possible at all. ## The comparison, on 400 real archive documents | | bge-m3 | Qwen3-Embedding-0.6B (deployed) | |---|---|---| | correspondent, nearest neighbour | 74.0% (90% of achievable) | 75.0% (91%) | | document type, nearest neighbour | **70.0%** | 68.2% | | time for 400 documents | **147 s** | 380 s | | peak memory under load | **1.6 GiB** | 10.6 GiB | | dimensions | 1024 | 1024 | Level where it matters, ahead everywhere else, and same vector width so nothing downstream changes shape. ## Pooling differs — and that is the trap bge-m3 wants **CLS** pooling where the Qwen family wants **last-token**. Read from the model's own `1_Pooling/config.json`, not assumed: across the candidates measured, **three different pooling types** were required. A wrong one produces plausible but untrained vectors, with no error message anywhere. ## The memory limit is measured now `1.6 GiB` peak (Prometheus `max_over_time` on the working set), limit set to 8Gi. This supersedes **#745**, which raised the limit for the model being replaced. The lesson from that OOM holds regardless of model: the footprint is dominated by four slots at 4096 context plus the Vulkan working set, **not** by the weights. The 600 MiB Qwen model peaked at 10.6 GiB on identical settings, and several candidates were OOMKilled at limits that looked generous against their parameter count. ## This is not the final answer Every candidate still has to go through `scripts/paperless-eval.sh` (#756) before the embedding choice means anything. Nothing has yet verified that the retrieval metric predicts what Paperless actually *suggests* — and the last two times an upstream proxy was trusted, MTEB turned out to be anti-correlated with the real measurement. bge-m3 is simply the one to start with. ## Why this unblocks the test The AI settings from #739 have never reached the running pod: it predates the Reloader annotation added in #741, and Reloader does not act retroactively. Editing `paperless-config` gives it something real to react to — rather than an imperative `rollout restart`, or a change invented purely to force one. ## Verification after merge ``` kubectl -n ai get pods -l app.kubernetes.io/name=llama-embed # restarted, RESTARTS 0 kubectl -n ai exec deploy/llama-server -- curl -s \ http://llama-embed.ai.svc.cluster.local:8080/v1/models | jq '.data[0].id' kubectl -n paperless exec deploy/paperless -c paperless -- env | grep -c PAPERLESS_AI ``` Expect `bge-m3`, a restarted paperless pod, and 11 AI variables present — the last of which has been the missing precondition all along. Then `scripts/paperless-eval.sh --engine classic` gives the baseline the LLM has to beat, and the full run follows once the LLM index is built. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit 2026-09-02 11:40:14 +00:00

feat(llama-embed): switch to bge-m3, activating the Paperless AI config

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

3bde2a4316

First candidate to be carried through to an end-to-end measurement. The
embedding comparison on 400 real archive documents puts bge-m3 level with the
deployed Qwen3-Embedding-0.6B where it matters and ahead everywhere else:

  correspondent  74.0% vs 75.0%   (ceiling 82.5%, so 90% vs 91% of achievable)
  document type  70.0% vs 68.2%
  time           147s vs 380s for 400 documents
  peak memory    1.6 GiB vs 10.6 GiB

Same 1024 dimensions, so nothing downstream changes shape.

Note the pooling type differs — bge-m3 wants CLS where the Qwen family wants
last-token. Read from the model's own 1_Pooling/config.json rather than assumed;
across the candidates measured, three different pooling types were required, and
a wrong one produces plausible but untrained vectors with no error anywhere.

The memory limit is now measured rather than estimated. It also supersedes #745,
which raised the limit for the model being replaced: the footprint is dominated
by four slots at 4096 context plus the Vulkan working set, not by the weights —
which is why several candidates were OOMKilled at limits that looked generous
against their model size.

This is not a final answer. Every candidate has to go through the end-to-end
test (scripts/paperless-eval.sh) before the embedding choice means anything,
because nothing has yet verified that the retrieval metric predicts what
Paperless actually suggests. bge-m3 is simply the one to start with.

Changing paperless-config is also what lets that test run at all: the AI
settings from #739 have never reached the running pod, which predates the
Reloader annotation from #741. This edit gives Reloader something real to react
to, rather than forcing a restart by hand or inventing a change to trigger one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

k.renner merged commit 4742b26047 into master

2026-09-02 11:41:20 +00:00

k.renner referenced this issue from a commit

2026-09-02 11:41:21 +00:00

Merge pull request 'feat(llama-embed): switch to bge-m3, activating the Paperless AI config' (#757) from embed-bgem3 into master

review-bot reviewed

2026-09-02 11:41:47 +00:00

review-bot left a comment

🤖 Automated review — model claude-haiku-4-5-20251001 (complexity: trivial), commit 3bde2a4316b005b89e08a039adc317e833305c42.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

This PR switches the embedding model from Qwen3-Embedding-0.6B to bge-m3, with corresponding configuration updates and memory limit increases. The change is well-justified with concrete measurement data and proper cross-file consistency. No blocking issues identified.

🔴 Blocking

None.

🟡 Suggestions

None. The PR is thorough as-is.

🟢 Nits

None.

Tests

This is a Kubernetes configuration change with no code tests required. The PR includes verification steps in its body (kubectl commands and script execution) to confirm the model is correctly loaded and the Paperless config is properly activated. The change requires rebuilding the embeddings index (documented in the comments), which is a known consequence of model swaps.

Recommendation

Looks reasonable — no blocking issues; the change is comprehensive and well-documented.

🤖 **Automated review** — model `claude-haiku-4-5-20251001` (complexity: trivial), commit `3bde2a4316b005b89e08a039adc317e833305c42`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary This PR switches the embedding model from Qwen3-Embedding-0.6B to bge-m3, with corresponding configuration updates and memory limit increases. The change is well-justified with concrete measurement data and proper cross-file consistency. No blocking issues identified. ## 🔴 Blocking None. ## 🟡 Suggestions None. The PR is thorough as-is. ## 🟢 Nits None. ## Tests This is a Kubernetes configuration change with no code tests required. The PR includes verification steps in its body (`kubectl` commands and script execution) to confirm the model is correctly loaded and the Paperless config is properly activated. The change requires rebuilding the embeddings index (documented in the comments), which is a known consequence of model swaps. ## Recommendation Looks reasonable — no blocking issues; the change is comprehensive and well-documented.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#757