konairius/Wiki
10
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

feat(encyclopedia): bounded planning — no model call sees the whole wiki #19

Merged
k.renner merged 16 commits from feat/bounded-planning into main 2026-10-26 09:28:27 +00:00
Conversation 0 Commits 16 Files Changed 36
+3615 -688
claude-bot commented 2026-10-26 09:23:07 +00:00
Member
Copy Link
Copy Source

Replaces the one-call article planner (which hit the 32k output cap on 1977 notes) with a stored plan that each run updates through bounded calls. Spec: docs/superpowers/specs/2026-10-26-encyclopedia-bounded-planning-design.md; plan: docs/superpowers/plans/2026-10-26-encyclopedia-bounded-planning.md.

What changes

  • articles/_plan.json is state: articles (with a one-line scope), note→article assignments, redirects, plus retired slugs and publicRedirects (added in review). Old format reads as "no plan" → first run bootstraps; old _state.json reads fine.
  • A run: diff notes → assign new notes in batches of ≤20 similar notes against ≤20 nearby articles (found via the existing per-note vector index) → split articles over 300k chars / 120 sources (≤120 sources in) → fold thin articles into a neighbour (≤10 candidates) → write stale articles with ≤10 nearby link targets. Every prompt is capped by a setting; nothing grows with the wiki.
  • Per-step caps per night (120 assign batches, 5 splits, 5 folds, 10 writes); a failed batch is remembered and not paid for again until its notes change; one transient model error skips only that call.
  • Thin articles awaiting a fold decision are not written yet (no money on pages that would vanish).
  • Addresses never break: split keeps the slug as overview; fold leaves a redirect (301). Anonymous readers follow a redirect only if the folded article was public at fold time and the final target is visible — otherwise the identical 404 of a missing slug. Retired slugs are never reused.
  • Endpoint answers 502 when the plan is empty after the run (so the CronJob goes red).

Tests: dotnet test Wiki.slnx green — Core 174, Application 219, Contracts 83, Host 160. Each task reviewed (spec + quality), whole-branch review + one fix wave re-reviewed.

Known residual (not fixed here): an Anthropic HttpClient timeout while waiting for response headers (MessagesTransport, 120 s) surfaces as TaskCanceledException, not HttpRequestException, so it bypasses the new per-call catch and fails that night's planning (staged work is committed). Follow-up: convert it in the transport like the stream-idle case.

Deploy notes: homelab sets Wiki__Encyclopedia__MaxArticles=0, so the first run only plans (no articles written) — then pick pilot slugs from _plan.json. Consider Wiki__Encyclopedia__MaxFoldsPerRun well above 5 (fold calls are ~1k tokens) so thin articles don't wait weeks for their fold decision.

🤖 Generated with Claude Code

Replaces the one-call article planner (which hit the 32k output cap on 1977 notes) with a stored plan that each run updates through bounded calls. Spec: `docs/superpowers/specs/2026-10-26-encyclopedia-bounded-planning-design.md`; plan: `docs/superpowers/plans/2026-10-26-encyclopedia-bounded-planning.md`. **What changes** - `articles/_plan.json` is state: articles (with a one-line scope), note→article assignments, redirects, plus `retired` slugs and `publicRedirects` (added in review). Old format reads as "no plan" → first run bootstraps; old `_state.json` reads fine. - A run: diff notes → **assign** new notes in batches of ≤20 similar notes against ≤20 nearby articles (found via the existing per-note vector index) → **split** articles over 300k chars / 120 sources (≤120 sources in) → **fold** thin articles into a neighbour (≤10 candidates) → **write** stale articles with ≤10 nearby link targets. Every prompt is capped by a setting; nothing grows with the wiki. - Per-step caps per night (120 assign batches, 5 splits, 5 folds, 10 writes); a failed batch is remembered and not paid for again until its notes change; one transient model error skips only that call. - Thin articles awaiting a fold decision are not written yet (no money on pages that would vanish). - Addresses never break: split keeps the slug as overview; fold leaves a redirect (301). Anonymous readers follow a redirect only if the folded article was public at fold time and the final target is visible — otherwise the identical 404 of a missing slug. Retired slugs are never reused. - Endpoint answers 502 when the plan is empty after the run (so the CronJob goes red). **Tests:** `dotnet test Wiki.slnx` green — Core 174, Application 219, Contracts 83, Host 160. Each task reviewed (spec + quality), whole-branch review + one fix wave re-reviewed. **Known residual (not fixed here):** an Anthropic HttpClient timeout while waiting for response headers (MessagesTransport, 120 s) surfaces as TaskCanceledException, not HttpRequestException, so it bypasses the new per-call catch and fails that night's planning (staged work is committed). Follow-up: convert it in the transport like the stream-idle case. **Deploy notes:** homelab sets `Wiki__Encyclopedia__MaxArticles=0`, so the first run only plans (no articles written) — then pick pilot slugs from `_plan.json`. Consider `Wiki__Encyclopedia__MaxFoldsPerRun` well above 5 (fold calls are ~1k tokens) so thin articles don't wait weeks for their fold decision. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
claude-bot added 16 commits 2026-10-26 09:23:08 +00:00
docs: bounded encyclopedia planning design
ci/woodpecker/push/test Pipeline was successful
Details
8d51f1958d 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docs: bounded encyclopedia planning implementation plan
ci/woodpecker/push/test Pipeline was successful
Details
fa45a96a81 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat(encyclopedia): plan v2 model with assignments and redirects 996ff182c0 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat(encyclopedia): state v2 with per-batch and keep memory 13d5495899 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat(encyclopedia): nearby articles through the note index 45c76ac4ba 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test(encyclopedia): cover self-hit exclusion and ranking before cap 348f8e4f75 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat(encyclopedia): assign notes in bounded batches e4384a2c54 
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
fix(encyclopedia): tighten assign tests and first-paragraph handling a5322e71d2 
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
feat(encyclopedia): split oversized articles ad45fe4ac2 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat(encyclopedia): fold thin articles into a neighbour 2d94f9a8d8 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat(encyclopedia): writer links only to bounded targets 7a6cfc5124 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat(encyclopedia): reconcile through bounded assign, split and fold b41bee9743 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat(house): follow article redirects 41a54a5844 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix(house): redirect lookup survives store failures 61af595614 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix(encyclopedia): planning survives model errors, thin articles wait for fold 4706e49fd7 
- A text-model HttpRequestException skips its assign batch (not remembered),
  split or fold article; only embedding/index failures end planning.
- Neighbour timeouts (TaskCanceledException, run not cancelled) count as
  service down.
- Thin articles with a nearby article and no keep at their inputs are not
  written until their fold decision; split children of the run are not
  folded.
- NearestAsync widens its query up to 8x the batch size, then fills from
  the open notes in slug order.
- Deleted articles' slugs (and redirects to them) stay reserved in
  _plan.json "retired"; a note left without articles is unassigned, not
  skipped.
- Fold records whether the folded article was public in both languages
  ("publicRedirects"); repointed redirects keep their flag.
- Assigned counts only notes assigned to an article.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fix(house): anonymous readers follow only redirects public at fold time
ci/woodpecker/push/test Pipeline was successful
Details
ci/woodpecker/pr/test Pipeline was successful
Details
bf274fb5a3 
ArticleQueries serves anonymous readers only the redirects listed in
publicRedirects, so a retired slug whose article was not public (or a chain
through one) answers the usual 404, identical to a missing slug. Signed-in
readers follow every redirect. Body links of the anonymous view resolve
through the same filtered set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
k.renner merged commit 8df8992f43 into main 2026-10-26 09:28:27 +00:00
k.renner deleted branch feat/bounded-planning 2026-10-26 09:28:27 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/Wiki#19