claude-botandClaude Opus 5
1ec6825b66
The measurements were coupled. One request that corrupts the slot leaves it
returning a single repeated token for everything after it, so a run counted
that fault once per remaining document. "Qwen3-8B failed 4 of 5" was really
"document 1970 succeeded, document 363 broke the slot, and the remaining three
measured the wreckage".
Proved mid-run rather than inferred: with two documents failed, the model
answered "Nenne die Hauptstadt von Frankreich" with
" \" \" \" \" \" \" \" \" \" \" ...
llama-server clears a slot on POST /slots/0?action=erase, but answers 501
without --slot-save-path, so the flag goes on the sidecar and eval.py calls it
after every AI document. It costs the prefix cache and some prefill speed,
which is the right trade for numbers that mean what they say.
Also worth recording: a synthetic reproduction of the poisoning failed. The
real paperless request corrupts the slot; my own 5000-token probe with the same
14-field schema does not, and the canary stayed healthy through it. Whatever
triggers this lives in the parts of a real request I was not reproducing — the
system prompt, the RAG context, the full taxonomy lists. Third time today that
a synthesised test passed where the real one fails, so the harness should stop
trying to stand in for the real request and instead detect when it has gone
wrong, which is what the canary now does.
Re-screening qwen3-8b to find out whether it scores more than one of five once
the documents are actually independent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
545 lines
23 KiB
Python
545 lines
23 KiB
Python
#!/usr/bin/env python3
|
|
"""Measure what Paperless suggests for documents whose labels a human already chose.
|
|
|
|
Runs both engines against the same documents:
|
|
|
|
/api/documents/<id>/suggestions/ the built-in scikit-learn classifier
|
|
/api/documents/<id>/ai_suggestions/ the LLM path, with RAG
|
|
|
|
WHAT THIS ANSWERS, AND WHAT AN EARLIER VERSION DID NOT
|
|
|
|
The two engines are not alternatives — the classifier costs 0.6 s per document
|
|
and keeps running whatever the LLM does. So the useful question is not "which
|
|
scores higher" but "what does the LLM add where the classifier is wrong", and
|
|
that cannot be recovered from two aggregate percentages: 74% and 72% could
|
|
overlap almost entirely or barely at all, and the difference between those two
|
|
worlds is the entire value of running an LLM.
|
|
|
|
So every document is written out individually, with both engines' verdicts, and
|
|
the summary is derived from that rather than counted along the way.
|
|
|
|
SCORING
|
|
|
|
Correspondent and document type are each scored twice:
|
|
|
|
offered the true label's id came back in `correspondents` /
|
|
`document_types` — what the UI actually presents
|
|
recognised also counting the true label's name appearing wordwise in
|
|
`suggested_correspondents` / `suggested_document_types`
|
|
|
|
The gap between them is one specific failure: the model names the right entity
|
|
but paperless_ai.matching cannot resolve it, because difflib runs at
|
|
MATCH_THRESHOLD = 0.8 and a correct but more verbose answer falls below it —
|
|
"Generali Deutschland Krankenversicherung AG" against the existing "Generali
|
|
AG". Reporting only the first blames the model for a matcher problem; only the
|
|
second claims a usefulness the UI does not deliver.
|
|
|
|
Document type is additionally split on whether the true type is "Sonstige",
|
|
which holds 1133 of ~3900 documents and is the one label with no semantic
|
|
content — a restaurant reservation lands there because nothing else fits. The
|
|
classifier can memorise that; a model reading the document has to infer "none
|
|
of the above". Scored separately rather than dropped, because the model does
|
|
sometimes get it right.
|
|
|
|
GROUND TRUTH
|
|
|
|
The current filing, which deserves care: over ten years conventions change, and
|
|
a disagreement with an old document is not automatically an error. Treat the
|
|
comparison between engines as the result and the absolute numbers as
|
|
indicative.
|
|
|
|
Env:
|
|
PAPERLESS_URL default http://127.0.0.1:8000
|
|
PGHOST/PGDATABASE standard libpq, for ground truth and the API token
|
|
EVAL_SAMPLE documents to score, default 50
|
|
EVAL_ENGINES comma-separated subset of classic,ai — default both
|
|
EVAL_LABEL free-text name for this configuration, recorded in the
|
|
output so results from different jobs stay distinguishable
|
|
EVAL_OUT directory for results, default /results
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import hashlib
|
|
import json
|
|
import os
|
|
import re
|
|
import secrets
|
|
import sys
|
|
import time
|
|
import urllib.error
|
|
import urllib.request
|
|
|
|
import psycopg
|
|
|
|
PAPERLESS_URL = os.environ.get("PAPERLESS_URL", "http://127.0.0.1:8000")
|
|
SAMPLE = int(os.environ.get("EVAL_SAMPLE", "50"))
|
|
ENGINES = os.environ.get("EVAL_ENGINES", "classic,ai").split(",")
|
|
LABEL = os.environ.get("EVAL_LABEL", "unlabelled")
|
|
OUT_DIR = os.environ.get("EVAL_OUT", "/results")
|
|
|
|
# A backstop, not the real bound. Generation is capped server-side by
|
|
# --n-predict on the llama-server sidecar, which is what actually stops a model
|
|
# that will not terminate; this catches everything else — a wedged sidecar, a
|
|
# request that never reaches it.
|
|
#
|
|
# 300 s against the ~73 s a healthy document takes end to end, prompt included.
|
|
# It does work: llama-server logs `stop: cancel task` on the disconnect and
|
|
# starts the next request 0.4 s later.
|
|
REQUEST_TIMEOUT = int(os.environ.get("EVAL_TIMEOUT", "300"))
|
|
|
|
# Consecutive failures from the very first document that mean the engine is
|
|
# unusable rather than merely weak. Three, because one could be a single hard
|
|
# document and two could be coincidence.
|
|
ABORT_AFTER = 3
|
|
|
|
CATCHALL_TYPE = "Sonstige"
|
|
|
|
|
|
def log(msg: str) -> None:
|
|
print(f"[eval] {msg}", flush=True)
|
|
|
|
|
|
def api_token(conn) -> str:
|
|
"""A token for this run, against a user belonging to this run alone.
|
|
|
|
The API needs one and this database is a disposable clone, so the run
|
|
issues its own rather than depending on a token that happened to come
|
|
across in the restore.
|
|
|
|
The per-label user is not tidiness. Django's authtoken holds one token per
|
|
user, so an earlier version took the first superuser and deleted its
|
|
existing token first — which meant the second of two candidates running
|
|
concurrently silently revoked the first one's credentials. That surfaced as
|
|
a mid-run `HTTP 401: Invalid token` sitting among genuine model failures,
|
|
where it reads like one more broken document rather than the harness
|
|
sabotaging itself. Both jobs share this database; only the user can differ.
|
|
"""
|
|
token = secrets.token_hex(20)
|
|
username = f"eval-{LABEL}"
|
|
with conn.cursor() as cur:
|
|
cur.execute("select id from auth_user where username = %s", (username,))
|
|
row = cur.fetchone()
|
|
if row is None:
|
|
cur.execute(
|
|
"""
|
|
insert into auth_user (password, is_superuser, username,
|
|
first_name, last_name, email, is_staff,
|
|
is_active, date_joined)
|
|
values ('!unusable', true, %s, '', '', '', true, true, now())
|
|
returning id
|
|
""",
|
|
(username,),
|
|
)
|
|
row = cur.fetchone()
|
|
user_id = row[0]
|
|
cur.execute("delete from authtoken_token where user_id = %s", (user_id,))
|
|
cur.execute(
|
|
"insert into authtoken_token (key, created, user_id) values (%s, now(), %s)",
|
|
(token, user_id),
|
|
)
|
|
conn.commit()
|
|
return token
|
|
|
|
|
|
def sample(conn) -> list[dict]:
|
|
"""The documents to score, ordered by md5 of the id.
|
|
|
|
Deterministic, so every engine and every configuration sees the same
|
|
documents and results stay comparable across jobs. Only documents carrying
|
|
both labels are usable as ground truth.
|
|
"""
|
|
with conn.cursor() as cur:
|
|
cur.execute(
|
|
"""
|
|
select d.id, d.correspondent_id, d.document_type_id, c.name, t.name
|
|
from documents_document d
|
|
join documents_correspondent c on c.id = d.correspondent_id
|
|
join documents_documenttype t on t.id = d.document_type_id
|
|
where d.correspondent_id is not null
|
|
and d.document_type_id is not null
|
|
and length(d.content) > 400
|
|
order by md5(d.id::text)
|
|
limit %s
|
|
""",
|
|
(SAMPLE,),
|
|
)
|
|
return [
|
|
{
|
|
"id": r[0],
|
|
"correspondent_id": r[1],
|
|
"document_type_id": r[2],
|
|
"correspondent_name": r[3],
|
|
"document_type_name": r[4],
|
|
}
|
|
for r in cur.fetchall()
|
|
]
|
|
|
|
|
|
def floor(conn) -> float:
|
|
"""Always guessing the most frequent correspondent.
|
|
|
|
Without it a number like "70% correct" cannot be read: one correspondent
|
|
holds several hundred of ~3900 documents.
|
|
"""
|
|
with conn.cursor() as cur:
|
|
cur.execute(
|
|
"""
|
|
select max(c), sum(c) from (
|
|
select count(*) c from documents_document
|
|
where correspondent_id is not null group by correspondent_id
|
|
) x
|
|
""",
|
|
)
|
|
top, total = cur.fetchone()
|
|
# float() on both: psycopg returns Postgres numerics as Decimal, and
|
|
# mixing that with a float literal raises TypeError rather than coercing.
|
|
return 100.0 * float(top) / float(total) if total else 0.0
|
|
|
|
|
|
def fetch(path: str, token: str) -> tuple[dict | None, str | None, float]:
|
|
req = urllib.request.Request(
|
|
f"{PAPERLESS_URL}{path}",
|
|
headers={"Authorization": f"Token {token}"},
|
|
)
|
|
start = time.monotonic()
|
|
try:
|
|
with urllib.request.urlopen(req, timeout=REQUEST_TIMEOUT) as resp:
|
|
body = resp.read()
|
|
return json.loads(body), None, time.monotonic() - start
|
|
except urllib.error.HTTPError as err:
|
|
return None, f"HTTP {err.code}: {err.read()[:200]!r}", time.monotonic() - start
|
|
except Exception as err: # noqa: BLE001 — a failed document is data, not a crash
|
|
return None, f"{type(err).__name__}: {err}", time.monotonic() - start
|
|
|
|
|
|
def normalise(text: str) -> set[str]:
|
|
return {w for w in re.split(r"[^\w]+", text.lower()) if w}
|
|
|
|
|
|
def recognised(payload: dict, field: str, want_name: str) -> bool:
|
|
"""True when a suggested name contains every word of the true label's name.
|
|
|
|
Handles the observed failure directly: the model answers with a superset of
|
|
the stored name, which difflib rejects at 0.8 but a reader would accept.
|
|
"""
|
|
want = normalise(want_name)
|
|
return any(want <= normalise(name) for name in payload.get(field) or [])
|
|
|
|
|
|
def degenerate(payload: dict) -> str | None:
|
|
"""Whether this looks like broken inference rather than a wrong guess.
|
|
|
|
Past roughly 14k prompt tokens this stack stops producing language — first
|
|
empty completions, then runs of digits and CJK characters — and one such
|
|
request poisons the llama.cpp slot until the pod restarts. Nothing in the
|
|
HTTP response says so: status 200, finish_reason "stop", well-formed JSON.
|
|
|
|
Scored as a wrong answer it would silently depress a candidate's numbers,
|
|
and because the slot stays poisoned it would take the rest of the run with
|
|
it. Flagged instead, so the failure is visible as itself.
|
|
"""
|
|
title = payload.get("title") or ""
|
|
if not title.strip():
|
|
return "empty title"
|
|
# Digits, punctuation and whitespace are normal in a document title —
|
|
# "Rechnung 2025-5425541" is 8 letters in 21 characters and tripped an
|
|
# earlier version of this check. What broken inference produces is
|
|
# characters from other scripts, so look for those directly rather than
|
|
# inferring them from a shortage of Latin ones.
|
|
foreign = sum(c.isalpha() and not (c.isascii() or c in "äöüÄÖÜßáéèêíóúàçñ") for c in title)
|
|
if foreign > 0.2 * len(title):
|
|
return f"non-latin title: {title[:40]!r}"
|
|
return None
|
|
|
|
|
|
def score_document(payload: dict, doc: dict) -> dict:
|
|
offered_corr = doc["correspondent_id"] in (payload.get("correspondents") or [])
|
|
offered_type = doc["document_type_id"] in (payload.get("document_types") or [])
|
|
return {
|
|
"correspondent_offered": offered_corr,
|
|
"correspondent_recognised": offered_corr
|
|
or recognised(payload, "suggested_correspondents", doc["correspondent_name"]),
|
|
"type_offered": offered_type,
|
|
"type_recognised": offered_type
|
|
or recognised(payload, "suggested_document_types", doc["document_type_name"]),
|
|
"suggested_correspondents": payload.get("suggested_correspondents") or [],
|
|
"suggested_document_types": payload.get("suggested_document_types") or [],
|
|
"title": payload.get("title"),
|
|
}
|
|
|
|
|
|
LLM_ENDPOINT = os.environ.get("PAPERLESS_AI_LLM_ENDPOINT", "http://127.0.0.1:8080/v1")
|
|
|
|
|
|
def canary() -> tuple[bool, str]:
|
|
"""Ask the model something it cannot get wrong, and check it did not.
|
|
|
|
The instrument has to be shown working before its readings mean anything,
|
|
and this one can fail silently in a way that looks like a bad model. On
|
|
this Vulkan stack a prompt past some model-specific length stops llama.cpp
|
|
producing language: it repeats one token — " de de de", " phosphorus
|
|
phosphorus", whitespace — at HTTP 200 with finish_reason "stop". The slot
|
|
stays broken for every request after it until the pod restarts.
|
|
|
|
So a run that measured 50 documents against a poisoned slot reports 50
|
|
failures that are really one event, and reads as a model that cannot
|
|
classify. That is exactly what happened before this existed.
|
|
|
|
Run before the documents and again after: healthy then broken means the
|
|
failures in between are one fault, not evidence about the model.
|
|
"""
|
|
body = json.dumps({
|
|
"model": os.environ.get("PAPERLESS_AI_LLM_MODEL", "probe"),
|
|
"max_tokens": 40,
|
|
"messages": [{"role": "user", "content": "Nenne die Hauptstadt von Frankreich."}],
|
|
}).encode()
|
|
req = urllib.request.Request(
|
|
f"{LLM_ENDPOINT}/chat/completions",
|
|
data=body,
|
|
headers={"Content-Type": "application/json"},
|
|
)
|
|
try:
|
|
with urllib.request.urlopen(req, timeout=120) as resp:
|
|
payload = json.loads(resp.read())
|
|
text = payload["choices"][0]["message"].get("content") or ""
|
|
text += payload["choices"][0]["message"].get("reasoning_content") or ""
|
|
return ("Paris" in text), text[:80]
|
|
except Exception as err: # noqa: BLE001
|
|
return False, f"{type(err).__name__}: {err}"
|
|
|
|
|
|
def run(engine: str, suffix: str, docs: list[dict], token: str) -> list[dict]:
|
|
log(f"=== {engine}: {len(docs)} documents")
|
|
rows = []
|
|
if engine == "ai":
|
|
ok, text = canary()
|
|
log(f" canary before: {'OK' if ok else 'BROKEN'} — {text!r}")
|
|
if not ok:
|
|
log(" The model is not answering a trivial question correctly, so")
|
|
log(" nothing measured after this describes its classification.")
|
|
return [{"engine": engine, "canary_before": False, "canary_text": text}]
|
|
for n, doc in enumerate(docs, 1):
|
|
payload, error, elapsed = fetch(
|
|
f"/api/documents/{doc['id']}/{suffix}",
|
|
token,
|
|
)
|
|
row = {
|
|
"engine": engine,
|
|
"document_id": doc["id"],
|
|
"seconds": round(elapsed, 1),
|
|
"truth_correspondent": doc["correspondent_name"],
|
|
"truth_type": doc["document_type_name"],
|
|
"catchall_type": doc["document_type_name"] == CATCHALL_TYPE,
|
|
}
|
|
if payload is None:
|
|
# An authentication failure is never a fact about the model, and
|
|
# counted alongside real ones it reads like one more broken
|
|
# document. Stop loudly instead: something revoked this run's
|
|
# credentials mid-flight, and every number after it is worthless.
|
|
if error and error.startswith("HTTP 401"):
|
|
raise SystemExit(
|
|
f"[eval] FATAL: {error}\n"
|
|
f"[eval] This run's token stopped working part-way through. "
|
|
f"Django's authtoken holds one token per user, so another "
|
|
f"job sharing this database and this user revoked it.",
|
|
)
|
|
row["error"] = error
|
|
log(f" {doc['id']}: {error}")
|
|
else:
|
|
row.update(score_document(payload, doc))
|
|
# Only the AI path returns a title; the classifier answers with ids
|
|
# alone, so an "empty title" there is normal rather than broken.
|
|
if engine == "ai" and (broken := degenerate(payload)) is not None:
|
|
row["degenerate"] = broken
|
|
log(f" {doc['id']}: DEGENERATE — {broken}")
|
|
rows.append(row)
|
|
|
|
# Clear the slot between documents so the next one is measured on a
|
|
# clean server. The corruption this guards against is stateful: one bad
|
|
# request leaves the slot returning a single repeated token for
|
|
# everything after it, and without this a run counts that once per
|
|
# remaining document.
|
|
if engine == "ai":
|
|
try:
|
|
urllib.request.urlopen(
|
|
urllib.request.Request(
|
|
f"{LLM_ENDPOINT.rsplit('/v1', 1)[0]}/slots/0?action=erase",
|
|
data=b"",
|
|
method="POST",
|
|
),
|
|
timeout=30,
|
|
)
|
|
except Exception as err: # noqa: BLE001 — best effort, never fatal
|
|
if n == 1:
|
|
log(f" slot erase unavailable ({type(err).__name__}) — "
|
|
f"failures after a corrupt document will not be independent")
|
|
|
|
if n % 10 == 0:
|
|
log(f" {n}/{len(docs)}")
|
|
# Re-check the instrument, not just the progress. A slot corrupted
|
|
# part-way through turns every remaining document into a failure
|
|
# that says nothing about the model, and scattered failures — 29 of
|
|
# 50 in one earlier run — are exactly the shape that a
|
|
# start-of-run check alone would miss.
|
|
if engine == "ai":
|
|
ok, text = canary()
|
|
if not ok:
|
|
log(f" CANARY BROKEN at {n}/{len(docs)} — {text!r}")
|
|
log(" The model can no longer answer a trivial question, so")
|
|
log(" everything from here would measure a dead slot. Stopping.")
|
|
log(" Failures recorded before this point are still usable;")
|
|
log(" the ones adjacent to it are one fault, not several.")
|
|
rows.append({"engine": engine, "canary_failed_at": n, "canary_text": text})
|
|
break
|
|
|
|
# Stop early when a model cannot do the task at all, rather than
|
|
# spending an hour writing the same error fifty times. Three candidates
|
|
# in a row failed this way — Granite because llama.cpp cannot read its
|
|
# tool calls back, both Ministral sizes because they fill the tool
|
|
# call's first string with their own [THINK]/[TOOL_CALLS] control
|
|
# tokens until the token cap. In each case document one already said
|
|
# so, and the remaining forty-nine said nothing new.
|
|
#
|
|
# Deliberately not a ratio: a model that fails a third of the time is a
|
|
# result worth measuring in full. This only catches the case where
|
|
# nothing has ever worked.
|
|
if len(rows) == ABORT_AFTER and all("error" in r for r in rows):
|
|
ok, text = canary()
|
|
log(f" canary after {ABORT_AFTER} failures: {'OK' if ok else 'BROKEN'} — {text!r}")
|
|
if not ok:
|
|
log(" The slot is corrupted, so these are ONE fault and not")
|
|
log(f" {ABORT_AFTER} independent results. Probe the model's prompt-length")
|
|
log(" boundary with probe-length.sh before judging it.")
|
|
log(f" ABORT — first {ABORT_AFTER} documents all failed:")
|
|
log(f" {rows[0]['error']}")
|
|
log(" Nothing has succeeded, so the rest would measure nothing.")
|
|
log(" Check the model's tool calling before re-running:")
|
|
log(" curl -s $ENDPOINT/v1/chat/completions -d @probe.json")
|
|
log(" A usable candidate answers with finish_reason 'tool_calls'.")
|
|
break
|
|
|
|
# And once at the end, whatever happened. A short run whose first
|
|
# document succeeds trips neither the abort check nor the every-tenth
|
|
# one, so without this a five-document screen can finish having never
|
|
# re-verified the instrument it just used.
|
|
if engine == "ai":
|
|
ok, text = canary()
|
|
log(f" canary after: {'OK' if ok else 'BROKEN'} — {text!r}")
|
|
if not ok:
|
|
log(" Healthy before, broken after: the failures in this run are")
|
|
log(" one fault and not independent results.")
|
|
rows.append({"engine": engine, "canary_after": False, "canary_text": text})
|
|
return rows
|
|
|
|
|
|
def pct(hits: int, total: int) -> float:
|
|
return round(100.0 * hits / total, 1) if total else 0.0
|
|
|
|
|
|
def summarise(rows: list[dict], engine: str) -> dict:
|
|
mine = [r for r in rows if r["engine"] == engine]
|
|
# Degenerate rows are excluded from the accuracy figures: they measure the
|
|
# backend giving up, not the model choosing wrongly, and leaving them in
|
|
# would quietly punish whichever candidate happened to trip the slot.
|
|
scored = [r for r in mine if "error" not in r and "degenerate" not in r]
|
|
failed = [r for r in mine if "error" in r]
|
|
broken = [r for r in mine if "degenerate" in r]
|
|
specific = [r for r in scored if not r["catchall_type"]]
|
|
catchall = [r for r in scored if r["catchall_type"]]
|
|
return {
|
|
"engine": engine,
|
|
"scored": len(scored),
|
|
"failed": len(failed),
|
|
# Non-zero means the run is suspect from that point on, not that the
|
|
# model is bad — the slot stays poisoned until the pod restarts.
|
|
"degenerate": len(broken),
|
|
"seconds_total": round(sum(r["seconds"] for r in mine)),
|
|
"correspondent_offered": pct(
|
|
sum(r["correspondent_offered"] for r in scored), len(scored)
|
|
),
|
|
"correspondent_recognised": pct(
|
|
sum(r["correspondent_recognised"] for r in scored), len(scored)
|
|
),
|
|
"type_offered": pct(sum(r["type_offered"] for r in scored), len(scored)),
|
|
"type_specific": pct(
|
|
sum(r["type_offered"] for r in specific), len(specific)
|
|
),
|
|
"type_catchall": pct(
|
|
sum(r["type_offered"] for r in catchall), len(catchall)
|
|
),
|
|
}
|
|
|
|
|
|
def complement(rows: list[dict]) -> dict | None:
|
|
"""What the LLM adds where the classifier is wrong, and what it costs.
|
|
|
|
The reason this script writes per-document rows at all. Only computed when
|
|
both engines ran, and only over documents both actually scored.
|
|
"""
|
|
by_engine = {"classic": {}, "ai": {}}
|
|
for row in rows:
|
|
usable = "error" not in row and "degenerate" not in row
|
|
if usable and row["engine"] in by_engine:
|
|
by_engine[row["engine"]][row["document_id"]] = row
|
|
both = set(by_engine["classic"]) & set(by_engine["ai"])
|
|
if not both or not by_engine["ai"]:
|
|
return None
|
|
|
|
rescued = missed = broken = agreed = 0
|
|
for doc_id in both:
|
|
classic_hit = by_engine["classic"][doc_id]["correspondent_offered"]
|
|
ai_hit = by_engine["ai"][doc_id]["correspondent_offered"]
|
|
if classic_hit and ai_hit:
|
|
agreed += 1
|
|
elif not classic_hit and ai_hit:
|
|
rescued += 1
|
|
elif classic_hit and not ai_hit:
|
|
broken += 1
|
|
else:
|
|
missed += 1
|
|
return {
|
|
"documents": len(both),
|
|
"both_right": agreed,
|
|
"llm_rescued": rescued,
|
|
"llm_only_wrong": broken,
|
|
"both_wrong": missed,
|
|
}
|
|
|
|
|
|
def main() -> int:
|
|
os.makedirs(OUT_DIR, exist_ok=True)
|
|
with psycopg.connect() as conn:
|
|
token = api_token(conn)
|
|
docs = sample(conn)
|
|
guess_floor = floor(conn)
|
|
log(f"label={LABEL} documents={len(docs)} floor={guess_floor:.1f}%")
|
|
|
|
rows: list[dict] = []
|
|
for engine, suffix in (("classic", "suggestions/"), ("ai", "ai_suggestions/")):
|
|
if engine in ENGINES:
|
|
rows.extend(run(engine, suffix, docs, token))
|
|
|
|
result = {
|
|
"label": LABEL,
|
|
"sample": len(docs),
|
|
"floor": round(guess_floor, 1),
|
|
"summaries": [summarise(rows, e) for e in ENGINES if e in ("classic", "ai")],
|
|
"complement": complement(rows),
|
|
}
|
|
|
|
with open(f"{OUT_DIR}/{LABEL}.jsonl", "w") as handle:
|
|
for row in rows:
|
|
handle.write(json.dumps(row, ensure_ascii=False) + "\n")
|
|
with open(f"{OUT_DIR}/{LABEL}.summary.json", "w") as handle:
|
|
json.dump(result, handle, ensure_ascii=False, indent=2)
|
|
|
|
# Also to stdout, so the numbers are in the pod log and reachable via Loki
|
|
# without mounting the volume.
|
|
log(json.dumps(result, ensure_ascii=False, indent=2))
|
|
return 0
|
|
|
|
|
|
if __name__ == "__main__":
|
|
sys.exit(main())
|