Phase 6: rank evaluation (POST /api/v1/lab/eval) and Lab page #8

2026-09-24T14:43:29Z

claude-bot commented

2026-09-24 14:43:29 +00:00

Stacked on #6 (facts); independent of #7. Retarget to master once #6 merges.

What

EvaluateUseCase (Application): the C# port of bin/wiki_rank_eval.py. Every output/ document with a question: and a sources: list is a labelled query; gold is the cited slugs that still exist as wiki notes. Ranks each with RankUseCase and reports recall@k, hit-rate@k, MRR (k ∈ {1,5,10,20,50}) plus one-hop recall (gold reachable from a top-k note by wikilink or related:). Deterministic sampling by seed. No judge: this measures the embedding model alone.
POST /api/v1/lab/eval {model, sample, seed, k}, write policy (it is expensive, not a public read).
Lab page: pick any registered embedding model, run, compare runs side by side.

Tests

4 use-case tests (metrics on a seeded catalog, skipped docs, sampling determinism, one-hop credit), host test for 401 and a labelled run.
dotnet test Wiki.slnx: 114 green on this branch.

🤖 Generated with Claude Code

Stacked on #6 (facts); independent of #7. Retarget to master once #6 merges. ## What - `EvaluateUseCase` (Application): the C# port of `bin/wiki_rank_eval.py`. Every `output/` document with a `question:` and a `sources:` list is a labelled query; gold is the cited slugs that still exist as wiki notes. Ranks each with `RankUseCase` and reports recall@k, hit-rate@k, MRR (k ∈ {1,5,10,20,50}) plus one-hop recall (gold reachable from a top-k note by wikilink or `related:`). Deterministic sampling by seed. No judge: this measures the embedding model alone. - `POST /api/v1/lab/eval` `{model, sample, seed, k}`, write policy (it is expensive, not a public read). - **Lab** page: pick any registered embedding model, run, compare runs side by side. ## Tests - 4 use-case tests (metrics on a seeded catalog, skipped docs, sampling determinism, one-hop credit), host test for 401 and a labelled run. - `dotnet test Wiki.slnx`: 114 green on this branch. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 2 commits 2026-09-24 14:43:29 +00:00

feat(lab): rank evaluation use case, API and Lab page 23c891db8f

Adds EvaluateUseCase, the C# port of bin/wiki_rank_eval.py: every output/
document with a question and a sources list labels a query, gold is the
cited slugs that still exist as wiki notes, and RankUseCase ranks each
one to report recall@k, hit-rate@k, MRR and a one-hop recall variant that
also credits a gold note reachable from a top-k note by wikilink or
related:. Exposes it as POST /api/v1/lab/eval (write-policy) and a new
/lab page for running it against any registered embedding model and
comparing runs side by side.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

docs: list the lab eval endpoint

ci/woodpecker/pr/test Pipeline was successful

Details

ci/woodpecker/push/test Pipeline was successful

Details

e28a403284

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

k.renner closed this pull request

2026-09-24 14:51:06 +00:00

claude-bot reopened this pull request

2026-09-24 14:51:54 +00:00

claude-bot changed target branch from feat/facts to main

2026-09-24 14:51:55 +00:00

claude-bot added 3 commits 2026-09-24 14:53:51 +00:00

feat(answer): add Anthropic Messages API answer model adapter 44c48bde71

Implements IAnswerModel over Anthropic's streaming Messages API: SSE
parsing (text_delta/usage/error), 429/529 retry with Retry-After or
exponential backoff, and a pure AnswerPrompt.Build for the evidence
and judgements prompt shape. Health check hits GET /v1/models.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

feat(questions): answer questions from wiki evidence with an answer model

ci/woodpecker/push/test Pipeline was successful

Details

ci/woodpecker/pr/test Pipeline was successful

Details

e6f5c80a3a

POST /api/v1/questions runs ask (rank + judge), selects the paragraphs that
carry probability mass, and streams one answer-model call over them. Below
the coverage gate no model is called and the response names the gap. With
save: true (signed-in user or API key) the answer lands in output/ as a
synth-shaped document with sources, confidence and the answering model.

The Answers page streams the same use case and lists saved answers.
Anthropic is the first IAnswerModel adapter; the answer model is chosen per
request or from Wiki:AnswerModels:Active.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Merge branch 'feat/answer-model' into feat/lab

ci/woodpecker/pr/test Pipeline was successful

Details

ci/woodpecker/push/test Pipeline was successful

Details

707c5f50a7

Both phases register their use cases and map their endpoints in the same host
files. The lab host test asserted exactly one labelled query, which held only
while no other test in the shared fixture wrote to output/; it now checks the
seeded query is among the rows.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

claude-bot commented

2026-09-24 14:53:52 +00:00

Reopened against main after #6 merged (Gitea closed the stacked PRs). This branch now also contains #7 (feat/answer-model merged in, conflicts in Program.cs/ApiV1.cs/README resolved), so it merges cleanly in either order: merge #7 first and this shows only the lab diff, or merge this alone and #7 becomes a no-op. Full suite: 128 green.