feat(scripts): end-to-end evaluation of Paperless suggestions #756

2026-09-02T11:33:03Z

claude-bot commented

2026-09-02 11:33:03 +00:00

The harness for measuring what actually matters: does Paperless suggest the labels a human already chose?

Why this is needed

scripts/embed-eval-real.sh scores whether a document's nearest neighbours carry its labels. That is a proxy. Paperless does not act on the neighbours directly — it hands them to an LLM together with the document text and a candidate list of existing tags and correspondents, and the LLM decides. It can override poor context, or fail with good context.

So a better embedding score does not automatically mean better suggestions, and that link has never been verified. Optimising a proxy without checking it predicts the outcome is exactly the error made with MTEB, one level up — and MTEB turned out to be anti-correlated with the embedding metric.

It also settles an older question

Paperless ships two engines, and this runs both on the same documents:

endpoint	engine
`/api/documents/<id>/suggestions/`	built-in scikit-learn classifier
`/api/documents/<id>/ai_suggestions/`	the LLM path

With ten years of hand-curated labels the built-in classifier has excellent training data. It can never invent a new tag or a title — that is the LLM's advantage — but for assigning labels that already exist it may simply be better. That was flagged as a possibility back in #739 and never tested. This answers it with a number instead of an assumption.

Reading the output

The most-frequent-correspondent floor is reported alongside the scores, for the same reason embed-eval-real.sh reports it: the archive is skewed enough (one correspondent holds 672 of 3939 documents) that "70% correct" is unreadable on its own.

Ground truth is the current filing, and that deserves care: over ten years conventions change, so a disagreement with an old document is not automatically an error. The comparison between the two engines is the durable result; the absolute numbers are indicative.

Usage

PAPERLESS_TOKEN=... scripts/paperless-eval.sh -n 100
PAPERLESS_TOKEN=... scripts/paperless-eval.sh -n 100 --engine classic

Requests are issued from inside the paperless pod, because the namespace policy admits only the Authentik gate outpost. Sampling is deterministic (md5 of the id), so both engines and repeated runs see identical documents.

Not yet runnable — and deliberately so

ai_suggestions returns 400 unless ai_enabled is set. The settings landed in the ConfigMap in #739, but the running pod predates the Reloader annotation from #741, so it still holds the old environment.

That restart should come from a real configuration change — the embedding model, once the comparison finishes — not from an imperative rollout restart and not from a change invented to force one. Merging this adds the harness; running it waits for that.

The classic engine needs none of this and can be measured immediately, which gives the baseline the LLM has to beat.

🤖 Generated with Claude Code

The harness for measuring what actually matters: **does Paperless suggest the labels a human already chose?** ## Why this is needed `scripts/embed-eval-real.sh` scores whether a document's nearest neighbours carry its labels. That is a **proxy**. Paperless does not act on the neighbours directly — it hands them to an LLM together with the document text *and* a candidate list of existing tags and correspondents, and the LLM decides. It can override poor context, or fail with good context. So a better embedding score does not automatically mean better suggestions, and that link has never been verified. Optimising a proxy without checking it predicts the outcome is exactly the error made with MTEB, one level up — and MTEB turned out to be *anti-correlated* with the embedding metric. ## It also settles an older question Paperless ships two engines, and this runs both on the same documents: | endpoint | engine | |---|---| | `/api/documents/<id>/suggestions/` | built-in scikit-learn classifier | | `/api/documents/<id>/ai_suggestions/` | the LLM path | With **ten years of hand-curated labels** the built-in classifier has excellent training data. It can never invent a new tag or a title — that is the LLM's advantage — but for assigning labels that already exist it may simply be better. That was flagged as a possibility back in #739 and never tested. This answers it with a number instead of an assumption. ## Reading the output The **most-frequent-correspondent floor** is reported alongside the scores, for the same reason `embed-eval-real.sh` reports it: the archive is skewed enough (one correspondent holds 672 of 3939 documents) that "70% correct" is unreadable on its own. Ground truth is the current filing, and that deserves care: over ten years conventions change, so a disagreement with an old document is not automatically an error. **The comparison between the two engines is the durable result; the absolute numbers are indicative.** ## Usage ```sh PAPERLESS_TOKEN=... scripts/paperless-eval.sh -n 100 PAPERLESS_TOKEN=... scripts/paperless-eval.sh -n 100 --engine classic ``` Requests are issued from inside the paperless pod, because the namespace policy admits only the Authentik gate outpost. Sampling is deterministic (md5 of the id), so both engines and repeated runs see identical documents. ## Not yet runnable — and deliberately so `ai_suggestions` returns `400` unless `ai_enabled` is set. The settings landed in the ConfigMap in #739, but the running pod predates the Reloader annotation from #741, so it still holds the old environment. That restart should come from a **real** configuration change — the embedding model, once the comparison finishes — not from an imperative `rollout restart` and not from a change invented to force one. Merging this adds the harness; running it waits for that. The `classic` engine needs none of this and can be measured immediately, which gives the baseline the LLM has to beat. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit 2026-09-02 11:33:03 +00:00

feat(scripts): end-to-end evaluation of Paperless suggestions

ci/woodpecker/pr/pr-review Pipeline was successful

Details

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ba1125b1e3

Measures the thing that is actually wanted, which nothing so far has:
does Paperless suggest the labels a human already chose?

scripts/embed-eval-real.sh scores whether a document's nearest neighbours carry
its labels. That is a proxy. Paperless does not act on the neighbours directly —
it hands them to an LLM together with the document text and a candidate list of
existing tags and correspondents, and the LLM decides. A better embedding score
therefore does not automatically mean better suggestions, and that link has
never been verified. Optimising the proxy without checking it predicts the
outcome is the same error as trusting MTEB, one level up.

It runs both engines Paperless ships, on the same documents:

  /api/documents/<id>/suggestions/      the built-in scikit-learn classifier
  /api/documents/<id>/ai_suggestions/   the LLM path

which also settles a question that predates the LLM work. With ten years of
hand-curated labels the built-in classifier has excellent training data. It can
never invent a new tag or a title — that is the LLM's advantage — but for
assigning labels that already exist it may simply be better. Running both
answers that instead of assuming it.

Reports the most-frequent-correspondent floor alongside the scores, for the same
reason embed-eval-real.sh does: the archive is skewed enough that a number like
"70% correct" is unreadable without it.

Ground truth is the current filing, which is worth reading with care — over ten
years conventions change, so a disagreement is not automatically an error. The
comparison between engines is the durable result; the absolute numbers are
indicative.

Requests go from inside the paperless pod, since the namespace policy admits
only the Authentik gate outpost.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

review-bot reviewed

2026-09-02 11:37:38 +00:00

review-bot left a comment

🤖 Automated review — model claude-sonnet-5 (complexity: normal), commit ba1125b1e345e3345ee43263167b2e407e783c60.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

Adds scripts/paperless-eval.sh, a new manual ops script that samples labelled documents from the Paperless Postgres DB and scores both the classic (/suggestions/) and LLM (/ai_suggestions/) engines against the existing correspondent/document-type filing, via kubectl exec. The script closely follows the conventions already established by scripts/ai-eval.sh (arg parsing, set -uo pipefail, mktemp workdir with trap cleanup, -h/--help via sed over the header comment) and is not wired into anything else, so the blast radius is limited to this one new, opt-in script. No blocking issues; one real (cosmetic) bug in --help and a documentation inconsistency worth confirming before merge.

🟡 Suggestions

scripts/paperless-eval.sh:38 / kubernetes/apps/llama-server/pvc.yaml:11 — the script's own rationale (and the PR description) leans entirely on scripts/embed-eval-real.sh as an existing, working script ("scripts/embed-eval-real.sh measures whether a document's nearest neighbours carry its labels"). That file does not exist anywhere in this repository — not in the current tree, not at the PR's merge-base (60cfcf9e), not on any other local branch (git branch -a shows only master). It's referenced the same way, and is equally absent, in kubernetes/apps/llama-server/pvc.yaml:11, so this predates this PR rather than being introduced by it — but this PR's whole framing ("that is a proxy... nothing so far has verified that it does") depends on that script actually existing and producing the embedding scores being compared against. Worth confirming it's real (maybe added in an unmerged branch/PR) rather than a stale/miscopied reference, since as written it will mislead anyone reading the header comment into looking for a script that isn't there.
scripts/paperless-eval.sh:100-104 — PAPERLESS_TOKEN is passed as a literal curl -H "Authorization: Token $PAPERLESS_TOKEN" argument inside kubectl exec. That puts the bearer token in the exec'd process's argv, visible to anything with ps//proc access inside the pod and potentially captured in API-server audit logs for the exec request itself. For a short-lived, manually-created token per the script's own instructions ("create one... then remove it when finished") the exposure window is bounded, but passing it via a header file (curl -K - / --header @file) or an env var set on the exec'd process would avoid the argv leak entirely.
scripts/paperless-eval.sh:92-98 — the two corr_max/corr_total psql calls both redirect stderr to the same $WORK/pg-err.txt, so the second call's redirect (line 96-98) silently overwrites any error from the first (line 92-95), and neither is ever surfaced to the user (unlike the main docs.txt query, whose pg-err.txt is cat'd on failure at line 84). If either of these two queries fails, the script just prints floor: ?% with no indication why.

🟢 Nits

scripts/paperless-eval.sh:57 — sed -n '2,40p' "$0" for --help includes line 40, which is set -uo pipefail itself, not a comment. Confirmed by running sed -n '2,40p' scripts/paperless-eval.sh: the last line of --help output is the literal set -uo pipefail. Should be 2,39p (the sibling script ai-eval.sh:44 uses 2,30p, correctly stopping one line before its own set -uo pipefail on line 33).
scripts/paperless-eval.sh:119 — the debug line on a bad response is German ("keine gueltige Antwort") while every other user-facing string in the script (including in ai-eval.sh) is English; looks like a stray leftover rather than intentional, since the archive content being classified is German but this message isn't part of that content.

Tests

No tests, and none expected — this mirrors scripts/ai-eval.sh, which is also an untested manual ops/diagnostic script in this repo, and both require a live cluster (kubectl exec into Postgres and the paperless pod) that isn't available in CI. The PR body is explicit that the AI-engine path is not yet runnable pending a real pod restart, which is a reasonable reason to not gate this on CI.

Recommendation

Looks reasonable — no blocking issues; comments above are optional (though the embed-eval-real.sh reference and the --help off-by-one are cheap to fix before merge).

🤖 **Automated review** — model `claude-sonnet-5` (complexity: normal), commit `ba1125b1e345e3345ee43263167b2e407e783c60`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary Adds `scripts/paperless-eval.sh`, a new manual ops script that samples labelled documents from the Paperless Postgres DB and scores both the classic (`/suggestions/`) and LLM (`/ai_suggestions/`) engines against the existing correspondent/document-type filing, via `kubectl exec`. The script closely follows the conventions already established by `scripts/ai-eval.sh` (arg parsing, `set -uo pipefail`, `mktemp` workdir with trap cleanup, `-h`/`--help` via `sed` over the header comment) and is not wired into anything else, so the blast radius is limited to this one new, opt-in script. No blocking issues; one real (cosmetic) bug in `--help` and a documentation inconsistency worth confirming before merge. ## 🟡 Suggestions - `scripts/paperless-eval.sh:38` / `kubernetes/apps/llama-server/pvc.yaml:11` — the script's own rationale (and the PR description) leans entirely on `scripts/embed-eval-real.sh` as an existing, working script ("`scripts/embed-eval-real.sh` measures whether a document's nearest neighbours carry its labels"). That file does not exist anywhere in this repository — not in the current tree, not at the PR's merge-base (`60cfcf9e`), not on any other local branch (`git branch -a` shows only `master`). It's referenced the same way, and is equally absent, in `kubernetes/apps/llama-server/pvc.yaml:11`, so this predates this PR rather than being introduced by it — but this PR's whole framing ("that is a proxy... nothing so far has verified that it does") depends on that script actually existing and producing the embedding scores being compared against. Worth confirming it's real (maybe added in an unmerged branch/PR) rather than a stale/miscopied reference, since as written it will mislead anyone reading the header comment into looking for a script that isn't there. - `scripts/paperless-eval.sh:100-104` — `PAPERLESS_TOKEN` is passed as a literal `curl -H "Authorization: Token $PAPERLESS_TOKEN"` argument inside `kubectl exec`. That puts the bearer token in the exec'd process's argv, visible to anything with `ps`/`/proc` access inside the pod and potentially captured in API-server audit logs for the `exec` request itself. For a short-lived, manually-created token per the script's own instructions ("create one... then remove it when finished") the exposure window is bounded, but passing it via a header file (`curl -K -` / `--header @file`) or an env var set on the exec'd process would avoid the argv leak entirely. - `scripts/paperless-eval.sh:92-98` — the two `corr_max`/`corr_total` `psql` calls both redirect stderr to the same `$WORK/pg-err.txt`, so the second call's redirect (line 96-98) silently overwrites any error from the first (line 92-95), and neither is ever surfaced to the user (unlike the main `docs.txt` query, whose `pg-err.txt` is `cat`'d on failure at line 84). If either of these two queries fails, the script just prints `floor: ?%` with no indication why. ## 🟢 Nits - `scripts/paperless-eval.sh:57` — `sed -n '2,40p' "$0"` for `--help` includes line 40, which is `set -uo pipefail` itself, not a comment. Confirmed by running `sed -n '2,40p' scripts/paperless-eval.sh`: the last line of `--help` output is the literal `set -uo pipefail`. Should be `2,39p` (the sibling script `ai-eval.sh:44` uses `2,30p`, correctly stopping one line before its own `set -uo pipefail` on line 33). - `scripts/paperless-eval.sh:119` — the debug line on a bad response is German ("keine gueltige Antwort") while every other user-facing string in the script (including in `ai-eval.sh`) is English; looks like a stray leftover rather than intentional, since the archive content being classified is German but this message isn't part of that content. ## Tests No tests, and none expected — this mirrors `scripts/ai-eval.sh`, which is also an untested manual ops/diagnostic script in this repo, and both require a live cluster (kubectl exec into Postgres and the paperless pod) that isn't available in CI. The PR body is explicit that the AI-engine path is not yet runnable pending a real pod restart, which is a reasonable reason to not gate this on CI. ## Recommendation Looks reasonable — no blocking issues; comments above are optional (though the `embed-eval-real.sh` reference and the `--help` off-by-one are cheap to fix before merge).

claude-bot referenced this pull request

2026-09-02 11:40:13 +00:00

feat(llama-embed): switch to bge-m3, activating the Paperless AI config #757

claude-bot added 1 commit 2026-09-02 12:05:06 +00:00

fix(paperless-eval): score a right answer the matcher threw away

ci/woodpecker/pr/pr-review Pipeline was successful

Details

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

be27e33735

The AI engine reports an unresolved guess as a name in
`suggested_correspondents` rather than an id in `correspondents`, and
paperless_ai.matching only resolves it when difflib.get_close_matches
clears MATCH_THRESHOLD = 0.8. A correct but more verbose answer does not:
"Generali Deutschland Krankenversicherung AG" fails against the existing
"Generali AG", leaving `correspondents` empty.

Scored as it was, that counts as a miss, which would have blamed the
model for a matcher limit. Score both instead — the id as offered, and
the name matched wordwise against the true label — so the two causes stay
separable. The classic engine emits no `suggested_*` fields, so its two
columns coincide and the comparison stays fair either way.

Names now come out of postgres alongside the ids, tab-separated because
'|' occurs in correspondent names.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

review-bot reviewed

2026-09-02 12:13:27 +00:00

review-bot left a comment

🤖 Automated review — model claude-sonnet-5 (complexity: normal), commit be27e33735a7bbe76aa4a4ca8a69b97ba68b11b2.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

Adds scripts/paperless-eval.sh, a new operator-run harness that samples labelled Paperless documents from Postgres and scores both suggestion engines (suggestions/ and ai_suggestions/) against the existing filing, with a strict/lenient hit definition to separate genuine LLM misses from a matcher (difflib) that drops correct-but-verbose answers. It's a single new executable script, no other files touched. The logic is careful and its odd pieces (deterministic md5(id) sampling, the lenient-match jq, the most-frequent-correspondent floor) are all explained and check out against the checkout (pod names, container name, PAPERLESS_AI_ENABLED, and the Reloader annotations referenced in the PR body all match kubernetes/apps/paperless/*.yaml). No blocking issues found — the items below are all robustness/polish suggestions.

🟡 Suggestions

scripts/paperless-eval.sh:79 — limit $SAMPLE interpolates the -n/--sample value (or $PAPERLESS_EVAL_SAMPLE) directly into the SQL text run via psql -c. A non-numeric value breaks the query with a raw Postgres error surfaced only via pg-err.txt; anything else is arbitrary SQL run as the postgres superuser (enableSuperuserAccess: true in kubernetes/apps/paperless/cluster.yaml). Given this is an operator-invoked local tool, not attacker input, this is low real risk, but a one-line case $SAMPLE in ''|*[!0-9]*) echo "..." >&2; exit 2;; esac would close it cheaply and turn a cryptic psql error into a clear one.
scripts/paperless-eval.sh:11 — the "why" section states as fact that scripts/embed-eval-real.sh measures nearest-neighbour label overlap, but that file doesn't exist anywhere in this repository (checked working tree and git grep for embed-eval; the only other hit is a comment in kubernetes/apps/llama-server/pvc.yaml:11 referencing the same nonexistent script). Either it lives in an unmerged branch/PR and the reference is premature, or it was never committed — worth confirming before merge so the comment doesn't point future readers at a script that was never there.
scripts/paperless-eval.sh:100-102 — the Paperless API token is passed as a curl -H argument inside kubectl exec, so the expanded token value travels through the exec request (and is visible to anything reading that pod's process args, e.g. ps from another process in the same pod). The script already tells the operator to use a scoped, throwaway token ("Create one for an existing user, then remove it when finished"), which mitigates the impact, but piping it via a header file (curl -K -) or an in-pod env var set through kubectl exec -- env ... would avoid the argv exposure entirely.
scripts/paperless-eval.sh:192-197 — $ENGINE is only validated in the case at the end, after both Postgres round-trips (docs.txt, corr_max.txt, corr_total.txt) have already run. A typo like --engine ali fails late with wasted work. Validating right after arg parsing (next to the PAPERLESS_TOKEN check) would fail fast.

🟢 Nits

scripts/paperless-eval.sh:51 — sed -n '2,40p' "$0" for -h/--help includes line 40 (set -uo pipefail) in the printed help text, since the header comment block actually ends at line 39. Compare scripts/ai-eval.sh:44, sed -n '2,30p', where 30 is correctly the last comment line before set -uo pipefail on line 33 — this one script is one line eager. Running --help will print the stray set -uo pipefail line after the Usage block.
scripts/paperless-eval.sh:130 / :128 — index([$t]) (subarray search for a one-element array) is equivalent to index($t) here since $have is always an array of plain strings; the extra wrapping array works but reads oddly against the destination and doesn't decode as originally intended for anyone unfamiliar with jq's array-vs-scalar index semantics.
scripts/paperless-eval.sh:102 — $WORK/err.txt is reused (overwritten) on every api() call inside the per-document loop, so only the last request's stderr survives if several fail; not currently surfaced anywhere in score() anyway, so low value as-is.

Tests

No automated test is added, but this matches the existing convention for this class of script: scripts/ai-eval.sh (the closest sibling — a manually-invoked comparison/eval tool) also ships without a test, and .woodpecker/ci-scripts-test.yml only exercises scripts/ci/*. Also note .woodpecker/ci-manifests-test.yml's shellcheck step only globs kubernetes/**/*.sh, so this script (like its siblings under scripts/) isn't linted by CI either — again consistent with existing scripts, not a gap introduced here. Nothing in this PR needs a regression test given the script is explicitly "not yet runnable" for the ai engine and the classic path is a straightforward read-and-report tool.

Recommendation

Looks reasonable — no blocking issues; comments above are optional.

🤖 **Automated review** — model `claude-sonnet-5` (complexity: normal), commit `be27e33735a7bbe76aa4a4ca8a69b97ba68b11b2`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary Adds `scripts/paperless-eval.sh`, a new operator-run harness that samples labelled Paperless documents from Postgres and scores both suggestion engines (`suggestions/` and `ai_suggestions/`) against the existing filing, with a strict/lenient hit definition to separate genuine LLM misses from a matcher (`difflib`) that drops correct-but-verbose answers. It's a single new executable script, no other files touched. The logic is careful and its odd pieces (deterministic `md5(id)` sampling, the lenient-match jq, the most-frequent-correspondent floor) are all explained and check out against the checkout (pod names, container name, `PAPERLESS_AI_ENABLED`, and the Reloader annotations referenced in the PR body all match `kubernetes/apps/paperless/*.yaml`). No blocking issues found — the items below are all robustness/polish suggestions. ## 🟡 Suggestions - `scripts/paperless-eval.sh:79` — `limit $SAMPLE` interpolates the `-n`/`--sample` value (or `$PAPERLESS_EVAL_SAMPLE`) directly into the SQL text run via `psql -c`. A non-numeric value breaks the query with a raw Postgres error surfaced only via `pg-err.txt`; anything else is arbitrary SQL run as the `postgres` superuser (`enableSuperuserAccess: true` in `kubernetes/apps/paperless/cluster.yaml`). Given this is an operator-invoked local tool, not attacker input, this is low real risk, but a one-line `case $SAMPLE in ''|*[!0-9]*) echo "..." >&2; exit 2;; esac` would close it cheaply and turn a cryptic psql error into a clear one. - `scripts/paperless-eval.sh:11` — the "why" section states as fact that `scripts/embed-eval-real.sh` measures nearest-neighbour label overlap, but that file doesn't exist anywhere in this repository (checked working tree and `git grep` for `embed-eval`; the only other hit is a comment in `kubernetes/apps/llama-server/pvc.yaml:11` referencing the same nonexistent script). Either it lives in an unmerged branch/PR and the reference is premature, or it was never committed — worth confirming before merge so the comment doesn't point future readers at a script that was never there. - `scripts/paperless-eval.sh:100-102` — the Paperless API token is passed as a `curl -H` argument inside `kubectl exec`, so the expanded token value travels through the exec request (and is visible to anything reading that pod's process args, e.g. `ps` from another process in the same pod). The script already tells the operator to use a scoped, throwaway token ("Create one for an existing user, then remove it when finished"), which mitigates the impact, but piping it via a header file (`curl -K -`) or an in-pod env var set through `kubectl exec -- env ...` would avoid the argv exposure entirely. - `scripts/paperless-eval.sh:192-197` — `$ENGINE` is only validated in the `case` at the end, after both Postgres round-trips (`docs.txt`, `corr_max.txt`, `corr_total.txt`) have already run. A typo like `--engine ali` fails late with wasted work. Validating right after arg parsing (next to the `PAPERLESS_TOKEN` check) would fail fast. ## 🟢 Nits - `scripts/paperless-eval.sh:51` — `sed -n '2,40p' "$0"` for `-h/--help` includes line 40 (`set -uo pipefail`) in the printed help text, since the header comment block actually ends at line 39. Compare `scripts/ai-eval.sh:44`, `sed -n '2,30p'`, where 30 is correctly the last comment line before `set -uo pipefail` on line 33 — this one script is one line eager. Running `--help` will print the stray `set -uo pipefail` line after the Usage block. - `scripts/paperless-eval.sh:130` / `:128` — `index([$t])` (subarray search for a one-element array) is equivalent to `index($t)` here since `$have` is always an array of plain strings; the extra wrapping array works but reads oddly against the destination and doesn't decode as originally intended for anyone unfamiliar with jq's array-vs-scalar `index` semantics. - `scripts/paperless-eval.sh:102` — `$WORK/err.txt` is reused (overwritten) on every `api()` call inside the per-document loop, so only the last request's stderr survives if several fail; not currently surfaced anywhere in `score()` anyway, so low value as-is. ## Tests No automated test is added, but this matches the existing convention for this class of script: `scripts/ai-eval.sh` (the closest sibling — a manually-invoked comparison/eval tool) also ships without a test, and `.woodpecker/ci-scripts-test.yml` only exercises `scripts/ci/*`. Also note `.woodpecker/ci-manifests-test.yml`'s `shellcheck` step only globs `kubernetes/**/*.sh`, so this script (like its siblings under `scripts/`) isn't linted by CI either — again consistent with existing scripts, not a gap introduced here. Nothing in this PR needs a regression test given the script is explicitly "not yet runnable" for the `ai` engine and the `classic` path is a straightforward read-and-report tool. ## Recommendation Looks reasonable — no blocking issues; comments above are optional.

claude-bot referenced this pull request

2026-09-02 13:14:20 +00:00

fix(llama-embed): size the batch for bge-m3 tokenizer, not Paperless #762

claude-bot commented

2026-09-05 07:31:58 +00:00

Geschlossen als überholt durch #778.

Dieser Branch ist der Vorgänger: er würde vollständig löschen und durch das alte Einzelskript ersetzen. Der Diff gegen master ist -2992/+253 Zeilen.

Was hier begonnen wurde, steht in master — inklusive der Erkenntnis, dass die erste Ergebnisrunde ungültig war: llama-server-Slots werden von echten Paperless-Anfragen dauerhaft zerstört, und ein Harness mit einem Slot für fünfzig Dokumente misst danach nur noch sich selbst. Der Harness in master erkennt das jetzt (Kanarienvogel) und verhindert es (Slot-Löschung zwischen Dokumenten).

Geschlossen als überholt durch #778. Dieser Branch ist der Vorgänger: er würde vollständig **löschen** und durch das alte Einzelskript ersetzen. Der Diff gegen master ist -2992/+253 Zeilen. Was hier begonnen wurde, steht in master — inklusive der Erkenntnis, dass die erste Ergebnisrunde ungültig war: llama-server-Slots werden von echten Paperless-Anfragen dauerhaft zerstört, und ein Harness mit einem Slot für fünfzig Dokumente misst danach nur noch sich selbst. Der Harness in master erkennt das jetzt (Kanarienvogel) und verhindert es (Slot-Löschung zwischen Dokumenten).

claude-bot closed this pull request

2026-09-05 07:31:59 +00:00

claude-bot commented

2026-09-05 07:32:15 +00:00

Geschlossen als überholt durch #778.

Dieser Branch ist der Vorgänger: er würde scripts/paperless-eval/ vollständig LÖSCHEN und durch das alte Einzelskript scripts/paperless-eval.sh ersetzen. Der Diff gegen master ist -2992/+253 Zeilen.

Was hier begonnen wurde, steht in master — inklusive der Erkenntnis, dass die erste Ergebnisrunde ungültig war: llama-server-Slots werden von echten Paperless-Anfragen dauerhaft zerstört, und ein Harness mit einem Slot für fünfzig Dokumente misst danach nur noch sich selbst. Der Harness in master erkennt das jetzt über eine Kontrollfrage mit bekannter Antwort und verhindert es durch Slot-Löschung zwischen den Dokumenten.

Geschlossen als überholt durch #778. Dieser Branch ist der Vorgänger: er würde scripts/paperless-eval/ vollständig LÖSCHEN und durch das alte Einzelskript scripts/paperless-eval.sh ersetzen. Der Diff gegen master ist -2992/+253 Zeilen. Was hier begonnen wurde, steht in master — inklusive der Erkenntnis, dass die erste Ergebnisrunde ungültig war: llama-server-Slots werden von echten Paperless-Anfragen dauerhaft zerstört, und ein Harness mit einem Slot für fünfzig Dokumente misst danach nur noch sich selbst. Der Harness in master erkennt das jetzt über eine Kontrollfrage mit bekannter Antwort und verhindert es durch Slot-Löschung zwischen den Dokumenten.

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#756