Compare commits
19
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
e93da3990c | ||
|
|
206c4629aa | ||
|
|
a42c978a99 | ||
|
|
1d5df07ad0 | ||
|
|
e240bea5b1 | ||
|
|
e560fd13a7 | ||
|
|
fbd5a2529a | ||
|
|
50fb31b950 | ||
|
|
311f59d82a | ||
|
|
4820bc9f6d | ||
|
|
0ecf1aaac5 | ||
|
|
53c03a8e53 | ||
|
|
c94afb004c | ||
|
|
125911a4e6 | ||
|
|
8b8e56d1b2 | ||
|
|
e0a0cf3085 | ||
|
|
09ec2cf1d9 | ||
|
|
4cd8ef1dd4 | ||
|
|
347ac9d7a6 |
@@ -15,7 +15,14 @@ test/Wiki.Contracts one abstract suite per port; every adapter passes its sui
|
||||
|
||||
Adapters are selected in `appsettings.json` under `Wiki:`; secrets come from environment
|
||||
variables (`Wiki__NoteStore__Gitea__Token`, `Wiki__Webhook__Secret`, `Auth__ApiKeys__<name>`,
|
||||
`Wiki__Judges__Models__<name>__ApiKey`, `Wiki__AnswerModels__Models__<name>__ApiKey`).
|
||||
`Wiki__Judges__Models__<name>__ApiKey`, `Wiki__AnswerModels__Models__<name>__ApiKey`). Each
|
||||
answer-model entry also provides the fact drafter of the same name (`/facts`).
|
||||
|
||||
Humans sign in with OpenID Connect (public client, PKCE): `Auth__Oidc__Authority` and
|
||||
`Auth__Oidc__ClientId` are required; the redirect URI is `https://<request host>/signin-oidc`. Machines send `X-Api-Key`, or BasicAuth
|
||||
checked against the htpasswd lines in `Auth__Basic__UsersPath`, a file or a mounted Secret directory
|
||||
(bcrypt, apr1 or SHA1; read API only). Only `/health`, `/ready` and the HMAC-signed `/webhook`
|
||||
answer without either.
|
||||
|
||||
## Run
|
||||
|
||||
|
||||
@@ -0,0 +1,792 @@
|
||||
# House Encyclopedia Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** The Wiki service generates bilingual (de/en) Wikipedia-style articles from its notes. It commits them to the notes repo and serves them at `wiki.renner.uno/house`: public parts are open to anonymous readers, and private parts are visible only to signed-in wiki-group members.
|
||||
|
||||
**Architecture:**
|
||||
- A new `Article` document kind lives under `articles/<lang>/<slug>.md` in the notes repo and is indexed by the existing sync.
|
||||
- Once a night (06:00 Europe/Berlin, after the 04:30 distill), an encyclopedia reconciler syncs, then regenerates only stale articles and commits all changes in one multi-file Gitea commit. An article is stale when a source hash changed, it is missing, or its subject was removed. Sync and the webhook do not trigger it.
|
||||
- The reconciler runs only when the catalog is at the store's head. It first plans the article set, stored in `articles/_plan.json`.
|
||||
- For each stale article it then runs: write German, apply the deterministic leak filter, run the LLM leak check, translate to English block by block, and filter and leak-check the English again.
|
||||
- Attempt memory in `articles/_state.json` bounds cost: a rejected subject is retried only when its inputs change, and a failed plan is not retried for the same note set.
|
||||
- The reconciler never stops the host: every failure except cancellation is recorded for `/status` and the run returns.
|
||||
- Article pages are static-SSR Razor components with their own Wikipedia-like layout and `[AllowAnonymous]`. Private blocks are removed server-side for anonymous readers.
|
||||
- Group gating happens in Authentik: a policy binding on the `wiki` OIDC application.
|
||||
|
||||
**Tech Stack:** .NET 10, ASP.NET Core minimal APIs, Blazor (static SSR for article pages), Markdig 1.4.0, MudBlazor 9.10.0 (existing pages only), raw `HttpClient` against the Anthropic Messages API, the Gitea contents API, xunit 2.9.3 with hand-written fakes, Mermaid (vendored JS).
|
||||
|
||||
**Spec:** `docs/superpowers/specs/2026-10-24-house-encyclopedia-design.md`. Read it before starting any task.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- Repo `konairius/Wiki`: `TreatWarningsAsErrors=true`, `Nullable` enabled, primary-constructor sealed classes, `CancellationToken ct` last, no mocking library, hand-written fakes only.
|
||||
- Verification for every task: `dotnet build Wiki.slnx -c Release` then `dotnet test Wiki.slnx -c Release --logger "console;verbosity=minimal"`. Both must be green.
|
||||
- Notes repo is `konairius/notes`, branch `main`. Articles live at `articles/<lang>/<slug>.md`, where `<lang>` is `de` or `en`. The plan file is `articles/_plan.json` and the attempt memory is `articles/_state.json`; `Classify` ignores both because of the `_` prefix.
|
||||
- Article slugs are English kebab-case and are the same in both languages. Titles are localized.
|
||||
- Private content is never present in HTML served to an anonymous reader. It is removed server-side, never hidden by CSS.
|
||||
- `wiki.renner.uno` stays off the internet: its Ingress carries neither `dns.renner.uno/public` nor `netbird.renner.uno/public`.
|
||||
- Generic examples that name nothing concrete in this house stay public.
|
||||
- When in doubt, content is private. This covers an unterminated private block, a failed leak check, unparseable model output, and a failed filter.
|
||||
- Public article pages send `<meta name="robots" content="noindex">` and the response headers `X-Robots-Tag: noindex` and `Cache-Control: no-store`.
|
||||
- An anonymous reader never sees a private article's title, a link to it, or a listing entry for it. An article is private when either language's file says so; the writer always gives both languages the same visibility.
|
||||
- No CDN at runtime: Mermaid is vendored under `wwwroot/lib/mermaid/`.
|
||||
- Article generation reuses the Anthropic key at `Wiki:AnswerModels:Models:claude-sonnet:ApiKey`. No new secret is added.
|
||||
- Spend limits are enforced on the API side. On a quota or overload error, stop the run, keep everything generated so far, and retry on the next nightly run. Any other Anthropic HTTP 400 is a rejection of that subject, remembered like any other rejection, not a transient.
|
||||
- Commits go to branch `feat/house-encyclopedia` with author `Claude Bot <claude-bot@renner.uno>`, via a PR. Never push to `main` directly.
|
||||
|
||||
## Review Focus
|
||||
|
||||
1. **Unterminated `::: private` block.** An article body opens a private block and never closes it. Everything after the opener must be treated as private (Task 2: `RedactPrivate_UnterminatedBlock_DropsRest`).
|
||||
2. **The translator moves a private boundary.** A translated unit that contains a `:::` or fence line would close a private block early in English. The model never sees or controls block boundaries; such a reply is rejected (Task 7: `Translate_UnitWithStructuralLine_Throws`, `Translate_PrivateBlockStaysPrivate`).
|
||||
3. **A private subject leaks by name.** For an anonymous reader, links, infobox entries and listings that point at a private article are dropped with their text (Task 10: `Render_Anonymous_LinkToInvisibleSlug_DroppedWithText`; Task 11: `PrivateArticle_Anonymous_ShowsNoTitle`, `PublicArticle_Anonymous_NoPrivateArticleTitle`).
|
||||
4. **Model returns non-JSON or truncated output** from planning, the leak check or the translator. The article is not published and the previous version stays (Task 7: `Write_LeakCheckUnparseable_Throws`, `Translate_WrongBlockCount_Throws`). Planning: the old plan stays (Task 6: `Plan_ModelGarbage_KeepsExistingPlan`).
|
||||
5. **A reconciler exception stops the host.** An unhandled exception in a `BackgroundService` stops the whole wiki. A missing Anthropic key, a JSON error or a Gitea error is recorded on `/status` and the service keeps running (Task 9: `Service_ReconcilerThrows_KeepsRunning`).
|
||||
6. **A missing or shrunken plan deletes articles.** Without a usable plan nothing is written; deletes happen only for subjects the previous plan had and the new plan dropped; the pilot never deletes (Task 8: `Reconcile_NoPlan_WritesNothing`, `Reconcile_PilotCapShrinks_DeletesNothing`, `Reconcile_SubjectDropped_DeletesBothLanguages`).
|
||||
7. **Paying again for the same failure.** A rejected subject or a failed plan would otherwise be retried every night on unchanged input, and a non-idempotent run would rewrite everything. A rerun on unchanged input makes no model call and no commit (Task 8: `Reconcile_RejectedSameInputs_NotRetried`, `Reconcile_PlanFailedSameNotes_DoesNotReplan`, `Reconcile_AfterOwnCommit_IsNoop`; Task 9: `Service_RunsOncePerDayAtRunAt`).
|
||||
|
||||
---
|
||||
|
||||
### Task 1: `Article` document kind
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/Wiki.Core/Model/Document.cs` (enum `DocumentKind`)
|
||||
- Modify: `src/Wiki.Core/Parsing/DocumentParser.cs` (`Classify`, `PathFor`)
|
||||
- Modify: `src/Wiki.Application/Sync/SyncUseCase.cs` (`AllKinds`)
|
||||
- Modify: `src/Wiki.Adapters.Gitea/GiteaRepository.cs` (`ListAsync` kind filter, if it filters by path)
|
||||
- Test: `test/Wiki.Core.Tests/DocumentParserTests.cs` (extend; create it if absent)
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `DocumentKind.Article`.
|
||||
- `DocumentParser.Classify("articles/de/wall-switches.md") == (DocumentKind.Article, "de/wall-switches")`.
|
||||
- `DocumentParser.PathFor(DocumentKind.Article, "en/heating") == "articles/en/heating.md"`.
|
||||
- Article slugs in the catalog are `<lang>/<slug>`.
|
||||
|
||||
- [ ] **Step 1: Write failing tests**
|
||||
|
||||
```csharp
|
||||
[Theory]
|
||||
[InlineData("articles/de/wall-switches.md", "de/wall-switches")]
|
||||
[InlineData("articles/en/heating.md", "en/heating")]
|
||||
public void Classify_Article(string path, string slug)
|
||||
=> Assert.Equal((DocumentKind.Article, slug), DocumentParser.Classify(path));
|
||||
|
||||
[Theory]
|
||||
[InlineData("articles/_plan.json")]
|
||||
[InlineData("articles/_state.json")]
|
||||
[InlineData("articles/fr/x.md")] // only de|en
|
||||
[InlineData("articles/de/_draft.md")]
|
||||
[InlineData("articles/de/sub/x.md")] // no nesting
|
||||
[InlineData("articles/x.md")]
|
||||
public void Classify_ArticleRejects(string path) => Assert.Null(DocumentParser.Classify(path));
|
||||
|
||||
[Fact]
|
||||
public void PathFor_Article() => Assert.Equal("articles/en/heating.md", DocumentParser.PathFor(DocumentKind.Article, "en/heating"));
|
||||
```
|
||||
|
||||
Add a `SyncUseCaseTests` case named `Sync_IndexesArticles_ButDoesNotEmbedThem`. The `FakeStore` holds one `wiki/` note and one `articles/de/x.md`. After `RunAsync`, the catalog contains both, and `FakeEmbeddings` was called only for the wiki note.
|
||||
|
||||
- [ ] **Step 2: Run the tests.** `dotnet test Wiki.slnx --filter "FullyQualifiedName~DocumentParserTests|FullyQualifiedName~SyncUseCaseTests"`. Expected: FAIL (`Article` is not defined).
|
||||
- [ ] **Step 3: Implement.** Add the enum member, the `Classify` branch (exactly two path segments after `articles/`, language `de|en`, the existing `_`-prefix rule), the `PathFor` case, and `Article` in `AllKinds`. Keep embedding restricted to `DocumentKind.Wiki`. If `GiteaRepository.ListAsync` maps paths through `Classify`, no change is needed there; verify it.
|
||||
- [ ] **Step 4: Run the full suite.** Expected: PASS.
|
||||
- [ ] **Step 5: Commit.** Message: `feat(core): Article document kind under articles/<lang>/`.
|
||||
|
||||
---
|
||||
|
||||
### Task 2: Article format: frontmatter, private blocks, redaction
|
||||
|
||||
**Files:**
|
||||
- Create: `src/Wiki.Core/Model/Article.cs`
|
||||
- Create: `src/Wiki.Core/Parsing/ArticleBody.cs`
|
||||
- Create: `src/Wiki.Core/Parsing/ArticleFile.cs`
|
||||
- Test: `test/Wiki.Core.Tests/ArticleBodyTests.cs`, `test/Wiki.Core.Tests/ArticleFileTests.cs`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces:
|
||||
```csharp
|
||||
public enum ArticleVisibility { Public, Private }
|
||||
public enum ArticleCategory { Room, System, Device, Topic }
|
||||
public sealed record ArticleSource(string Slug, string ContentHash);
|
||||
public sealed record Article(string Lang, string Slug, string Title, ArticleCategory Category,
|
||||
ArticleVisibility Visibility, IReadOnlyList<ArticleSource> Sources, DateOnly Generated,
|
||||
string Model, string Body);
|
||||
public sealed record BodyBlock(int Index, string Text);
|
||||
|
||||
public static class ArticleBody
|
||||
{
|
||||
public const string PrivateOpen = "::: private";
|
||||
public const string Close = ":::";
|
||||
public static string RedactPrivate(string body); // anonymous view
|
||||
public static IReadOnlyList<BodyBlock> PublicBlocks(string body); // outside private blocks & code fences
|
||||
public static string Privatize(string body, IReadOnlySet<int> blockIndexes); // wrap those public blocks in ::: private
|
||||
// translation (Task 7)
|
||||
public static IReadOnlyList<TranslationUnit> TranslationUnits(string body);
|
||||
public static string Reassemble(string body, IReadOnlyList<string> translated); // throws FormatException
|
||||
public static IReadOnlyList<string> Skeleton(string body); // structural lines in order
|
||||
}
|
||||
public sealed record TranslationUnit(int Index, bool Private, string Text);
|
||||
public static class ArticleFile
|
||||
{
|
||||
public static string Build(Article article); // frontmatter + body
|
||||
public static Article Parse(Document document); // from catalog Document of kind Article
|
||||
}
|
||||
```
|
||||
- Body conventions, which the Task 7 prompts must teach the writer:
|
||||
- `::: infobox` … `:::` holds a two-column pipe table.
|
||||
- `::: private` … `:::` marks a private block.
|
||||
- Mermaid diagrams go in ```` ```mermaid ```` fences.
|
||||
- Links to other articles use `[[slug]]`.
|
||||
- Nesting is not allowed.
|
||||
- Translation units, which keep private boundaries out of the model's hands:
|
||||
- Structural lines are `::: private`, `::: infobox`, `:::` and code-fence lines. They are never sent to the translator.
|
||||
- A unit is a maximal run of non-blank, non-structural lines. The whole body of a ```` ```mermaid ```` fence is one unit, because its labels need translating. The body of any other code fence is not a unit and is copied verbatim.
|
||||
- `Private` is true for units inside a `::: private` block, and for everything after an unterminated opener, the same rule as `RedactPrivate`.
|
||||
- `Reassemble` replaces unit `i` with `translated[i]` and copies every structural line and blank line from `body`. It throws `FormatException` when the count differs from the unit count, when a translated text contains a structural line, when a translated non-mermaid unit contains a blank line (it would split a block and shift the numbering), or when a translated text's multiset of `[[…]]` links differs from its unit's.
|
||||
- `Skeleton` returns the structural lines in order. Two bodies with equal skeletons have the same private blocks and the same `PublicBlocks` numbering.
|
||||
- Frontmatter keys:
|
||||
- `title`, `lang`, `subject` (the slug without language), `category` (lowercase), `visibility` (lowercase)
|
||||
- `sources`: a YAML list of `slug@contenthash`
|
||||
- `generated` (`yyyy-MM-dd`), `model`
|
||||
|
||||
- [ ] **Step 1: Write failing tests.**
|
||||
- `RedactPrivate_RemovesBlock_KeepsLockedMarker`: given `"A\n\n::: private\nsecret\n:::\n\nB"`, the result contains `A`, `B` and `::: private-locked`, and does not contain `secret`.
|
||||
- `RedactPrivate_UnterminatedBlock_DropsRest`: given `"A\n\n::: private\nsecret\n\nmore"`, the result does not contain `secret` or `more`.
|
||||
- `RedactPrivate_IgnoresFenceInsideCode`: a `::: private` line inside a ```` ``` ```` code fence is not treated as an opener.
|
||||
- `PublicBlocks_SkipsPrivateAndNumbersSequentially`: on `"A\n\n::: private\nx\n:::\n\nB"` the indexes are `[0:"A", 1:"B"]`.
|
||||
- `PublicBlocks_InfoboxIsOneBlock`: an infobox container counts as one block.
|
||||
- `Privatize_WrapsChosenBlocks`: `Privatize(body, {1})` wraps only `B`. Applying `PublicBlocks` again returns only `A`.
|
||||
- `TranslationUnits_MarksPrivateUnits`: on `"A\n\n::: private\nx\n:::\n\nB"` the units are `[0:A public, 1:x private, 2:B public]`.
|
||||
- `TranslationUnits_MermaidFenceIsOneUnit_CodeFenceIsNot`.
|
||||
- `TranslationUnits_UnterminatedPrivate_RestIsPrivate`.
|
||||
- `Reassemble_KeepsSkeleton`: `Skeleton(Reassemble(body, texts)) == Skeleton(body)`.
|
||||
- `Reassemble_WrongCount_Throws`.
|
||||
- `Reassemble_UnitWithStructuralLine_Throws`: a translated private unit `"x\n:::\nleak"` throws `FormatException`.
|
||||
- `Reassemble_LinkSetChanged_Throws`: a unit with `[[heating]]` translated without it throws.
|
||||
- `ArticleFile_RoundTrip`: `Build`, then `DocumentParser.Parse`, then `ArticleFile.Parse`, gives an equal `Article`, including two sources `schalter-konvention-blueprint-system@abc123` and `heizung@def456`.
|
||||
- [ ] **Step 2: Run them.** Expected: FAIL (the types are not defined).
|
||||
- [ ] **Step 3: Implement.** `RedactPrivate` replaces each private block with the two lines `::: private-locked` and `:::`. A line-based scanner tracks code-fence state. Build the frontmatter by hand with a `StringBuilder`, following the `DepositFactUseCase.BuildNote` style, and reuse its YAML scalar quoting rules.
|
||||
- [ ] **Step 4: Run the suite.** Expected: PASS.
|
||||
- [ ] **Step 5: Commit.** Message: `feat(core): article body format and private-block redaction`.
|
||||
|
||||
---
|
||||
|
||||
### Task 3: Deterministic leak filter
|
||||
|
||||
**Files:**
|
||||
- Create: `src/Wiki.Core/Parsing/LeakScanner.cs`
|
||||
- Test: `test/Wiki.Core.Tests/LeakScannerTests.cs`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `ArticleBody.PublicBlocks` and `ArticleBody.Privatize` (Task 2).
|
||||
- Produces:
|
||||
```csharp
|
||||
public sealed record LeakMatch(int BlockIndex, string Rule, string Value);
|
||||
public static class LeakScanner
|
||||
{
|
||||
public static IReadOnlyList<LeakMatch> Scan(string body, IReadOnlyList<string> sources); // public blocks only
|
||||
public static string PrivatizeLeaks(string body, IReadOnlyList<string> sources); // Privatize(body, blocks with any match)
|
||||
}
|
||||
```
|
||||
- Only this house's identifiers count, so generic examples stay public (spec: "Generic examples stay public"):
|
||||
- a candidate counts as a match when its value occurs, case-insensitively, in any of `sources` (the full texts of the article's source notes);
|
||||
- a name ending in `renner.uno` always counts, sources or not;
|
||||
- reserved example values never count, even when they occur in sources: `192.0.2.0/24`, `198.51.100.0/24`, `203.0.113.0/24`, `2001:db8::/32`, and `example.com`, `example.org`, `example.net` or any name under `.example`.
|
||||
- Rules (rule names are used in the tests):
|
||||
- `ipv4`
|
||||
- `ipv6`, matching at least two `:` groups of hex digits, e.g. `fd00::1`
|
||||
- `mac`
|
||||
- `fqdn`: any `label.label.tld` with at least 2 dots, or any name ending in `.renner.uno`, `.local`, `.lan`, `.internal`, or `.svc.cluster.local`
|
||||
- `entity-id`: `(light|switch|sensor|binary_sensor|cover|climate|script|automation|scene|input_\w+|button|number|select|fan|media_player|person|device_tracker|lock|alarm_control_panel|camera|vacuum|water_heater|update)\.[a-z0-9_]+`
|
||||
- `path`: tokens starting with `/`, `~/` or `./`, containing a further `/`
|
||||
- `secret-path`: `apps/<x>/<y>`, `secret/…`, `kv/…`, `vault_[a-z_]+`
|
||||
|
||||
- [ ] **Step 1: Write failing tests.** Use a `[Theory]` with one `InlineData` per rule:
|
||||
- `ipv4`: `"Der Server 10.2.40.1 antwortet"`
|
||||
- `ipv6`: `"fd00::1"`
|
||||
- `mac`: `"aa:bb:cc:dd:ee:ff"`
|
||||
- `fqdn`: `"wiki.renner.uno"`, `"ak-outpost.authentik.svc.cluster.local"`
|
||||
- `entity-id`: `"light.eg_kuche_ambiente"`
|
||||
- `path`: `"/var/run/secrets/x"`
|
||||
- `secret-path`: `"apps/ai/wiki-anthropic"`
|
||||
|
||||
Each case passes its value in `sources` and yields that rule name. Then:
|
||||
- `Scan_IgnoresPrivateBlocks`: the same string inside `::: private` yields no match.
|
||||
- `Scan_GenericExampleNotInSources_NoMatch`: `"Eine Entität heißt zum Beispiel light.wohnzimmer"` with sources not containing `light.wohnzimmer` yields no match.
|
||||
- `Scan_MatchIsCaseInsensitive`: body `LIGHT.EG_KUCHE_AMBIENTE`, source `light.eg_kuche_ambiente`, yields `entity-id`.
|
||||
- `Scan_ReservedExample_NoMatch`: `"192.0.2.10"`, `"2001:db8::1"` and `"nas.example.com"` yield no match, even when present in sources.
|
||||
- `Scan_RennerUno_AlwaysMatches`: `"gitea.renner.uno"` with empty sources yields `fqdn`.
|
||||
- `Scan_NoFalsePositive_OnPlainProse`: `"Die Wärmepumpe heizt den Estrich. Version 2.5 ist aktuell."` yields no match, so a version number is not an IPv4 address.
|
||||
- `PrivatizeLeaks_MovesOnlyLeakingBlock`.
|
||||
- [ ] **Step 2: Run them.** Expected: FAIL.
|
||||
- [ ] **Step 3: Implement** the rules as `[GeneratedRegex]` partial methods. Inline code spans are scanned too, because a hostname in backticks is still a leak.
|
||||
- [ ] **Step 4: Run them.** Expected: PASS.
|
||||
- [ ] **Step 5: Commit.** Message: `feat(core): deterministic leak scanner for public article text`.
|
||||
|
||||
---
|
||||
|
||||
### Task 4: Note store: multi-file commit, text and binary read by path
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/Wiki.Core/Ports/Ports.cs` (`INoteStore`)
|
||||
- Modify: `src/Wiki.Core/Model/Retrieval.cs` (new records)
|
||||
- Modify: `src/Wiki.Adapters.Gitea/GiteaRepository.cs`
|
||||
- Modify: `test/Wiki.Application.Tests/Fakes.cs`, `test/Wiki.Host.Tests/Fakes.cs` (`FakeStore`)
|
||||
- Modify: `test/Wiki.Contracts/Gitea/FakeGiteaHandler.cs`, `test/Wiki.Contracts/NoteStoreContract.cs`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces:
|
||||
```csharp
|
||||
public enum FileChangeOp { Create, Update, Delete }
|
||||
public sealed record FileChange(FileChangeOp Op, string Path, string? Text, string? BlobSha);
|
||||
public sealed record StoredText(string Text, string BlobSha);
|
||||
public sealed class StoreConflictException(string message) : Exception(message);
|
||||
// INoteStore additions:
|
||||
Task<string> CommitAsync(IReadOnlyList<FileChange> changes, string message, CancellationToken ct); // the new commit sha
|
||||
Task<StoredText?> ReadTextAsync(string path, string commit, CancellationToken ct); // null on 404
|
||||
Task<byte[]> ReadBytesAsync(string path, string commit, CancellationToken ct);
|
||||
```
|
||||
- `BlobSha` is required for `Update` and `Delete`. For articles it is the `ContentHash` from `ListAsync`, which is the Gitea blob SHA. For files outside the catalog (`articles/_plan.json`, `articles/_state.json`) it is `StoredText.BlobSha`.
|
||||
- `ReadTextAsync` exists because `ReadAsync` takes a `DocumentRef`, which `_`-prefixed files never have. It replaces a separate blob-SHA lookup: one call returns both.
|
||||
- `CommitAsync` must produce exactly one commit.
|
||||
- Gitea mapping:
|
||||
- `POST api/v1/repos/{repo}/contents` with body `{ branch, message, files: [{ operation: "create"|"update"|"delete", path, content (base64), sha }] }`. The returned sha is the response's `commit.sha`.
|
||||
- HTTP 409 or 422 throws `StoreConflictException`.
|
||||
- `ReadTextAsync` uses `GET contents/{path}?ref=commit` and returns the base64-decoded `.content` with `.sha`. A 404 returns null.
|
||||
- `ReadBytesAsync` uses `raw/{path}?ref=commit`.
|
||||
- `FakeStore.CommitAsync` applies the changes, advances `Head` to a new value (`c2`, `c3`, …) and returns it, so a sync after a fake commit sees a new head. The fake's blob SHA is the same SHA-1 of the text that `ListAsync` reports.
|
||||
|
||||
- [ ] **Step 1: Write failing contract tests** in `NoteStoreContract`:
|
||||
- `Commit_CreatesUpdatesDeletesInOneCommit`: after the call, `GetHeadAsync` advanced once and equals the returned sha, and `ListAsync` shows the create, the update with new content, and the delete gone.
|
||||
- `Commit_StaleSha_ThrowsConflict`.
|
||||
- `ReadText_ReturnsTextAndUsableSha`: read `articles/_plan.json`, then an `Update` with that `BlobSha` succeeds.
|
||||
- `ReadText_MissingFile_IsNull`.
|
||||
- `ReadBytes_ReturnsExactBytes`, using a 3-byte PNG header.
|
||||
- [ ] **Step 2: Run them.** Expected: FAIL (compile errors in the fakes).
|
||||
- [ ] **Step 3: Implement** the port members, the Gitea adapter, `FakeGiteaHandler` support for the batch contents endpoint and the contents GET, and both `FakeStore`s. The `FakeStore`s record a `Commits` list of `(message, changes)`.
|
||||
- [ ] **Step 4: Run the suite.** Expected: PASS.
|
||||
- [ ] **Step 5: Commit.** Message: `feat(store): multi-file commit, text and binary read by path`.
|
||||
|
||||
---
|
||||
|
||||
### Task 5: Text model port and Anthropic adapter
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/Wiki.Core/Ports/Ports.cs`
|
||||
- Create: `src/Wiki.Core/Model/TextModel.cs`
|
||||
- Create: `src/Wiki.Adapters.Anthropic/AnthropicTextModel.cs`
|
||||
- Modify: `src/Wiki.Adapters.Anthropic/MessagesTransport.cs`, plus the registration extensions file in the same project
|
||||
- Test: `test/Wiki.Contracts/Anthropic/AnthropicTextModelTests.cs`, reusing the existing fake handler
|
||||
|
||||
**Interfaces:**
|
||||
- Produces:
|
||||
```csharp
|
||||
public sealed record TextReply(string Text, string ModelId, long InputTokens, long OutputTokens, bool Truncated);
|
||||
public sealed class ModelUnavailableException(string message, Exception? inner = null) : Exception(message, inner);
|
||||
public interface ITextModel : IAdapterHealth
|
||||
{
|
||||
string ModelId { get; }
|
||||
Task<TextReply> CompleteAsync(string system, string user, int maxTokens, CancellationToken ct);
|
||||
}
|
||||
// registration
|
||||
public static IServiceCollection AddAnthropicTextModel(this IServiceCollection services, Action<AnthropicOptions> configure);
|
||||
```
|
||||
- `Truncated` is true when the stream reports `stop_reason == "max_tokens"`.
|
||||
- The transport throws `ModelUnavailableException` in three cases:
|
||||
- 429 or 529 after `MaxRetries`;
|
||||
- a stream `error` event of type `rate_limit_error` or `overloaded_error`;
|
||||
- HTTP 400 whose error message mentions credit, billing or a usage limit. This covers API-side spend limits.
|
||||
|
||||
All other failures keep throwing `HttpRequestException`, now constructed with its `StatusCode` set, so Task 7 can tell a 400 (a rejection) from a 5xx or network error (a transient).
|
||||
- A missing API key keeps throwing `InvalidOperationException` (`MessagesTransport.PostWithRetryAsync`). It is not caught by the use cases; Task 9's coordinator records it.
|
||||
- Existing answer and fact adapters inherit the new exception type. `ModelUnavailableException` must map to 502 in the host, the same as `HttpRequestException`.
|
||||
|
||||
- [ ] **Step 1: Write failing tests.**
|
||||
- `Complete_AccumulatesStreamedText`.
|
||||
- `Complete_MaxTokensStop_SetsTruncated`.
|
||||
- `Complete_429AfterRetries_ThrowsModelUnavailable`, using an injected no-op `delay`.
|
||||
- `Complete_CreditError400_ThrowsModelUnavailable`.
|
||||
- `Complete_Other400_ThrowsHttpRequestWithStatus`: `StatusCode == HttpStatusCode.BadRequest`.
|
||||
- `Complete_PassesMaxTokensAndSystem`: assert the request JSON has `max_tokens` 16000 and the `system` string.
|
||||
- [ ] **Step 2: Run them.** Expected: FAIL.
|
||||
- [ ] **Step 3: Implement.** Add an overload `MessagesTransport.StreamAsync(..., int maxTokens, ...)` and keep the existing callers unchanged. `AnthropicTextModel.Name` is `anthropic-text:{ModelId}`.
|
||||
- [ ] **Step 4: Run the suite.** Expected: PASS.
|
||||
- [ ] **Step 5: Commit.** Message: `feat(anthropic): long-form text model port with typed unavailability`.
|
||||
|
||||
---
|
||||
|
||||
### Task 6: Article-set planning
|
||||
|
||||
**Files:**
|
||||
- Create: `src/Wiki.Application/Encyclopedia/ArticlePlan.cs`
|
||||
- Create: `src/Wiki.Application/Encyclopedia/PlanArticlesUseCase.cs`
|
||||
- Create: `src/Wiki.Application/Encyclopedia/EncyclopediaState.cs`
|
||||
- Create: `src/Wiki.Application/Encyclopedia/Prompts.cs` (all prompt texts, as `const string`, plus `const int Version`)
|
||||
- Test: `test/Wiki.Application.Tests/PlanArticlesUseCaseTests.cs`, `test/Wiki.Application.Tests/EncyclopediaStateTests.cs`, with a `FakeTextModel` added to `Fakes.cs`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `ITextModel` (Task 5), `IDocumentCatalog.ListDocumentsAsync(DocumentKind.Wiki, ct)`.
|
||||
- Produces:
|
||||
```csharp
|
||||
public sealed record PlannedArticle(string Slug, string TitleDe, string TitleEn, ArticleCategory Category, IReadOnlyList<string> Sources, IReadOnlyList<string> Related);
|
||||
public sealed record ArticlePlan(IReadOnlyList<PlannedArticle> Articles, IReadOnlyList<string> CoveredNotes); // CoveredNotes = all wiki slugs seen when planned
|
||||
public static class ArticlePlanJson { public static string Write(ArticlePlan p); public static ArticlePlan? Read(string json); } // Read: null on malformed JSON, never throws
|
||||
public sealed class PlanArticlesUseCase(ITextModel model, EncyclopediaSettings settings, ILogger<PlanArticlesUseCase> log)
|
||||
{
|
||||
public static bool NeedsReplan(ArticlePlan? current, PlanAttempt? failedPlan, IReadOnlyList<Document> notes);
|
||||
public async Task<ArticlePlan?> PlanAsync(ArticlePlan? current, IReadOnlyList<Document> notes, CancellationToken ct); // null = keep current
|
||||
}
|
||||
public sealed record EncyclopediaSettings(int PlanMaxTokens, int ArticleMaxTokens, int MaxArticlesPerRun,
|
||||
int MaxSourceCharsPerArticle, IReadOnlyList<string> PilotSlugs, int? MaxArticles);
|
||||
|
||||
// attempt memory, persisted as articles/_state.json
|
||||
public enum AttemptOutcome { Written, Rejected }
|
||||
public sealed record SubjectAttempt(string Fingerprint, AttemptOutcome Outcome, DateTimeOffset At);
|
||||
public sealed record PlanAttempt(string NoteSetFingerprint, DateTimeOffset At);
|
||||
public sealed record EncyclopediaState(IReadOnlyDictionary<string, SubjectAttempt> Subjects, PlanAttempt? FailedPlan)
|
||||
{
|
||||
public static EncyclopediaState Empty { get; }
|
||||
public static string NoteSetFingerprint(IReadOnlyList<Document> notes); // SHA-256 over Prompts.Version + sorted wiki slugs
|
||||
public static string SubjectFingerprint(PlannedArticle planned, IReadOnlyList<Document> notes); // SHA-256 over Prompts.Version, titles, category, sorted slug@ContentHash of its sources
|
||||
}
|
||||
public static class EncyclopediaStateJson { public static string Write(EncyclopediaState s); public static EncyclopediaState Read(string json); } // Read: Empty on malformed JSON
|
||||
```
|
||||
- `EncyclopediaSettings` lives in Application so the use cases don't depend on Host options. Task 9 maps it.
|
||||
- `NeedsReplan` is true when `current` is null or its `CoveredNotes` differ from the note slugs, **and** `failedPlan` is not for the same `NoteSetFingerprint`. A failed plan is therefore retried only when notes are added or removed, or `Prompts.Version` changes.
|
||||
- `Prompts.Version` is bumped by hand whenever a prompt changes, so a prompt fix retries earlier rejections.
|
||||
- Prompt contract (`Prompts.Plan`):
|
||||
- Input: the current plan's JSON, if any. Then one line per note: `slug | title | tags | body length in chars`.
|
||||
- Instruction: the Wikipedia notability rule from the spec. A subject gets its own article when there is enough material, e.g. many automations or a complicated history. Otherwise its notes go into the parent system's article as a section.
|
||||
- Instruction: every note is used by at least one article or deliberately skipped (meta and wiki-tooling notes).
|
||||
- Instruction: keep the existing slugs for existing subjects.
|
||||
- Instruction: list each article's sources most important first. When an article's sources add up to more than `MaxSourceCharsPerArticle` characters of body, split the subject into narrower articles.
|
||||
- Reply: JSON only, in the shape `{"articles":[{"slug","titleDe","titleEn","category":"room|system|device|topic","sources":[...],"related":[...]}]}`.
|
||||
- Validation after parsing:
|
||||
- drop sources that are not known note slugs;
|
||||
- drop articles with zero sources;
|
||||
- de-duplicate slugs;
|
||||
- slugs must match `^[a-z0-9]+(-[a-z0-9]+)*$`;
|
||||
- source cap: keep sources in the listed order while their summed body length stays within `MaxSourceCharsPerArticle` (default 300 000, about 100k tokens). The first source is always kept. Drop the rest, the lowest-ranked, and log a warning with the slug and the dropped count.
|
||||
- The plan is never truncated by the pilot settings. `PilotSlugs` and `MaxArticles` limit what Task 8 writes, not what is planned.
|
||||
- Failure results: garbage, truncated, zero valid articles, or an `HttpRequestException` with status 400 return null (logged). `ModelUnavailableException` and any other exception propagate to the reconciler.
|
||||
|
||||
- [ ] **Step 1: Write failing tests.**
|
||||
- `NeedsReplan_NullPlan_True`.
|
||||
- `NeedsReplan_SameSlugs_False`.
|
||||
- `NeedsReplan_NoteAdded_True`.
|
||||
- `NeedsReplan_FailedForSameNoteSet_False`: `current` null, `failedPlan` has the notes' fingerprint.
|
||||
- `NeedsReplan_FailedForOtherNoteSet_True`.
|
||||
- `Plan_ParsesAndValidates`: the fake returns JSON containing one unknown source and one bad slug. Both are dropped.
|
||||
- `Plan_ModelGarbage_KeepsExistingPlan`: the fake returns `"Sure! Here is"`, and `PlanAsync` returns null.
|
||||
- `Plan_Truncated_KeepsExistingPlan`: the reply has `Truncated=true`, and `PlanAsync` returns null.
|
||||
- `Plan_Http400_ReturnsNull`.
|
||||
- `Plan_ModelUnavailable_Propagates`.
|
||||
- `Plan_SourcesOverCharCap_DropsLowestRanked`: three sources of 200k chars each with the cap at 300k keep only the first.
|
||||
- `Plan_PromptContainsExistingSlugs`: assert the user prompt sent to the fake contains the current plan's slugs.
|
||||
- `ArticlePlanJson_Malformed_ReadsNull`.
|
||||
- `EncyclopediaStateJson_RoundTrip` and `EncyclopediaStateJson_Malformed_ReadsEmpty`.
|
||||
- `SubjectFingerprint_ChangesWithSourceHashOrTitle`: equal for identical inputs, different when one source's `ContentHash` or `TitleDe` changes.
|
||||
- [ ] **Step 2: Run them.** Expected: FAIL.
|
||||
- [ ] **Step 3: Implement.** Use `System.Text.Json`. Strip a ```` ```json ```` fence from the reply if one is present.
|
||||
- [ ] **Step 4: Run them.** Expected: PASS.
|
||||
- [ ] **Step 5: Commit.** Message: `feat(encyclopedia): article-set planning and attempt memory`.
|
||||
|
||||
---
|
||||
|
||||
### Task 7: Article writing pipeline
|
||||
|
||||
**Files:**
|
||||
- Create: `src/Wiki.Application/Encyclopedia/WriteArticleUseCase.cs`
|
||||
- Modify: `src/Wiki.Application/Encyclopedia/Prompts.cs` (`Write`, `LeakCheck`, `Translate`)
|
||||
- Test: `test/Wiki.Application.Tests/WriteArticleUseCaseTests.cs`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes:
|
||||
- `ITextModel`;
|
||||
- `LeakScanner.Scan`, `ArticleBody.PublicBlocks`, `ArticleBody.Privatize`;
|
||||
- `ArticleBody.TranslationUnits`, `Reassemble`, `Skeleton`;
|
||||
- `ArticleFile.Build`;
|
||||
- `PlannedArticle`;
|
||||
- source `Document`s;
|
||||
- `TimeProvider`.
|
||||
- Produces:
|
||||
```csharp
|
||||
public sealed record WrittenArticle(Article De, Article En);
|
||||
public sealed class WriteArticleUseCase(ITextModel model, EncyclopediaSettings settings, TimeProvider clock)
|
||||
{
|
||||
public async Task<WrittenArticle> WriteAsync(PlannedArticle planned, IReadOnlyList<Document> sources,
|
||||
IReadOnlyList<PlannedArticle> allArticles, CancellationToken ct);
|
||||
}
|
||||
public sealed class ArticleRejectedException(string slug, string reason) : Exception($"{slug}: {reason}");
|
||||
```
|
||||
- Pipeline:
|
||||
1. **Write.** `Prompts.Write` produces German markdown. The first line of the reply is `VISIBILITY: public|private`, followed by the body.
|
||||
2. **Filter.** `LeakScanner.Scan` the German body, passing the source notes' full texts as `sources`, and `Privatize` the matching blocks.
|
||||
3. **Leak check.** `Prompts.LeakCheck` receives only the numbered `PublicBlocks` and replies `{"privatize":[ints]}`. Apply `Privatize`. The prompt says that a generic example explaining a principle, naming nothing concrete in this house, is not a leak.
|
||||
4. **Translate block by block.** Take `TranslationUnits` of the German body and send them as a numbered JSON array `[{"i":0,"text":"…"},…]`. `Prompts.Translate` requires a JSON array of exactly as many strings back, in order, keeping `[[slug]]` links and mermaid syntax unchanged. Build the English body with `Reassemble`. Private and public units go in the same request; the model sees no container lines and so cannot move a private boundary.
|
||||
5. **Filter and leak-check the English.** `Scan` the English body with the same `sources`, then run `Prompts.LeakCheck` on the English `PublicBlocks`. The English check is not skipped: the translation is new model text and may reintroduce a name the German check removed. Every block index privatized in this step is privatized in the German body too. Both bodies share one skeleton, so the indexes align and the two languages keep the same private blocks.
|
||||
6. **Check structure.** If `Skeleton(de) != Skeleton(en)`, throw `ArticleRejectedException`. This is a defensive invariant; steps 4 and 5 should make it unreachable.
|
||||
- Visibility: one value from the `VISIBILITY` line, applied to both languages. `En.Visibility == De.Visibility` always.
|
||||
- Links: the writer may link any slug in `allArticles` anywhere in the body. It does not know which planned articles will turn out private, so hiding is the renderer's job (Task 10 drops links to invisible slugs with their text). `Prompts.Write` tells it to refer to other articles only through `[[slug]]` and never to repeat their title next to the link; the link renders the title.
|
||||
- `Prompts.Write` must state:
|
||||
- the audience (non-technical first, then deeper);
|
||||
- the order: lead → infobox → principle → examples → a short History at the end;
|
||||
- the body conventions from Task 2;
|
||||
- the link targets: only slugs in `allArticles`;
|
||||
- that every claim comes from the sources;
|
||||
- that the newest confirmed fact is used and disputed claims are omitted;
|
||||
- the private rule: hostnames, IPs, device IDs, network and security layout, restart and server procedures, and attacker-useful detail are private, and private is the choice when unsure;
|
||||
- that generic examples explaining a principle stay public, and that formats are illustrated with reserved example values (`192.0.2.x`, `2001:db8::`, `example.com`), never with this house's real ones;
|
||||
- that no PR numbers, incident logs or entity IDs appear in public text;
|
||||
- that interior photos of equipment are private unless clearly harmless;
|
||||
- which images are available: the `attachments` frontmatter of the sources.
|
||||
- Fail-closed rules:
|
||||
- **Any model reply `Truncated`:** throw `ArticleRejectedException`.
|
||||
- **Unparseable leak-check JSON:** throw.
|
||||
- **Translator reply not a JSON array of strings, or of the wrong length, or `Reassemble` throws `FormatException`:** throw `ArticleRejectedException`.
|
||||
- **`HttpRequestException` with status 400:** throw `ArticleRejectedException` (a credit or limit 400 is already `ModelUnavailableException`, Task 5). Other `HttpRequestException`s propagate as transients.
|
||||
- **Missing `VISIBILITY` line:** treat as private.
|
||||
- **Leak-check indexes out of range:** ignore those indexes.
|
||||
|
||||
- [ ] **Step 1: Write failing tests**, using a `FakeTextModel` that returns scripted replies in order.
|
||||
- `Write_RunsWriteLeakCheckTranslateLeakCheckInOrder`: assert four calls, with system prompts starting with `[write]`, `[leak-check]`, `[translate]`, `[leak-check]`.
|
||||
- `Write_LeakCheckPrivatizesChosenBlocks`.
|
||||
- `Write_DeterministicFilterRunsBeforeLeakCheck`: a source note contains `10.2.40.1`, and so does block 1 of the German body. The leak-check prompt does not contain `10.2.40.1`.
|
||||
- `Write_GenericExampleStaysPublic`: the German body explains the address format with `192.0.2.10` in a public block, and the leak check returns `{"privatize":[]}`. The block is still public in both languages.
|
||||
- `Write_PromptsAllowGenericExamples`: the `[write]` and `[leak-check]` system prompts both contain the phrase about generic examples.
|
||||
- `Write_EnglishLeakFilteredToo`: the translation introduces `wiki.renner.uno` in unit 2, which ends up in a private block in English, and the matching German block is private too.
|
||||
- `Write_EnglishLeakCheckPrivatizesBothLanguages`: the second leak check returns `{"privatize":[0]}`; block 0 is private in both bodies and `Skeleton(De.Body) == Skeleton(En.Body)`.
|
||||
- `Write_LeakCheckUnparseable_Throws`.
|
||||
- `Write_Truncated_Throws`.
|
||||
- `Write_MissingVisibility_IsPrivate`.
|
||||
- `Write_VisibilitySameInBothLanguages`.
|
||||
- `Write_SourcesCarryHashes`: `De.Sources` equals the source docs' `(Slug, ContentHash)`.
|
||||
- `Write_Http400_IsRejected`.
|
||||
- `Translate_SendsNoStructuralLines`: the `[translate]` user prompt contains no line starting with `:::`.
|
||||
- `Translate_WrongBlockCount_Throws`: the translator returns one string fewer than units sent.
|
||||
- `Translate_NotJsonArray_Throws`.
|
||||
- `Translate_UnitWithStructuralLine_Throws`: the translation of a private unit contains a `:::` line.
|
||||
- `Translate_PrivateBlockStaysPrivate`: a German private unit `GEHEIM` translated as `SECRET` is inside `::: private` in English, and `ArticleBody.RedactPrivate(En.Body)` does not contain `SECRET`.
|
||||
- [ ] **Step 2: Run them.** Expected: FAIL.
|
||||
- [ ] **Step 3: Implement.** Each prompt's system string begins with a role marker (`[write]`, `[leak-check]`, `[translate]`) so the tests can tell the calls apart. Source notes are passed in full, each wrapped as `<note slug="…" updated="…">…</note>`.
|
||||
- [ ] **Step 4: Run them.** Expected: PASS.
|
||||
- [ ] **Step 5: Commit.** Message: `feat(encyclopedia): write, leak-check and translate pipeline`.
|
||||
|
||||
---
|
||||
|
||||
### Task 8: Encyclopedia reconciler
|
||||
|
||||
**Files:**
|
||||
- Create: `src/Wiki.Application/Encyclopedia/EncyclopediaReconciler.cs`
|
||||
- Test: `test/Wiki.Application.Tests/EncyclopediaReconcilerTests.cs`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes:
|
||||
- `INoteStore.GetHeadAsync`, `CommitAsync`, `ReadTextAsync` (Task 4);
|
||||
- `IDocumentCatalog.GetStateAsync`, `ListDocumentsAsync(DocumentKind.Wiki, ct)` and `ListDocumentsAsync(DocumentKind.Article, ct)`, called separately;
|
||||
- `PlanArticlesUseCase` (Task 6) and `WriteArticleUseCase` (Task 7);
|
||||
- `ArticleFile.Parse`, `ArticleFile.Build`;
|
||||
- `ArticlePlanJson`, `EncyclopediaState`, `EncyclopediaStateJson` (Task 6);
|
||||
- `TimeProvider`.
|
||||
- Produces:
|
||||
```csharp
|
||||
public sealed record ReconcileSummary(int Planned, int Written, int Deleted, int Rejected, int Remembered,
|
||||
bool StoppedByModel, string? Skipped, string? Commit); // Remembered = rejected earlier, inputs unchanged, not retried
|
||||
public sealed class EncyclopediaReconciler(INoteStore store, IDocumentCatalog catalog, PlanArticlesUseCase planner,
|
||||
WriteArticleUseCase writer, EncyclopediaSettings settings, TimeProvider clock, ILogger<EncyclopediaReconciler> log)
|
||||
: IEncyclopediaRun
|
||||
{
|
||||
public async Task<ReconcileSummary> RunAsync(CancellationToken ct);
|
||||
public static IReadOnlyList<PlannedArticle> Stale(ArticlePlan plan, IReadOnlyList<Document> notes, IReadOnlyList<Article> existing);
|
||||
public static IReadOnlyList<PlannedArticle> Due(IReadOnlyList<PlannedArticle> stale, ArticlePlan plan,
|
||||
IReadOnlyList<Document> notes, EncyclopediaState state, EncyclopediaSettings settings);
|
||||
}
|
||||
public interface IEncyclopediaRun { Task<ReconcileSummary> RunAsync(CancellationToken ct); }
|
||||
```
|
||||
- Algorithm:
|
||||
1. **Freshness gate.** If `catalog.GetStateAsync().Commit != store.GetHeadAsync().Commit`, return `Skipped = "catalog-behind"` with no model call and no write. Task 9 syncs right before each run, so this is a safety check; the next nightly run retries.
|
||||
2. Read `articles/_plan.json` and `articles/_state.json` with `ReadTextAsync` at the head. Null means none yet. A malformed plan reads as null and a malformed state as `Empty`, both logged.
|
||||
3. If `NeedsReplan(current, state.FailedPlan, notes)`, call `PlanAsync`.
|
||||
- Non-null result: it becomes the plan, staged as `_plan.json`, and `FailedPlan` is cleared.
|
||||
- Null result: record `FailedPlan = (NoteSetFingerprint(notes), now)` and keep `current`.
|
||||
- `ModelUnavailableException`: stop the run. Keep the current plan, stage nothing, return `StoppedByModel = true`. No memory is recorded; it is a transient.
|
||||
4. **No usable plan** (`current` null and planning failed): return `Skipped = "no-plan"`. The only file committed is `_state.json` with the failed-plan memory; no article file is created, updated or deleted.
|
||||
5. Compute `Stale`. An article is stale when any of these is true:
|
||||
- it is missing in either language;
|
||||
- the set of its sources differs from the plan;
|
||||
- any source hash differs from the catalog;
|
||||
- the plan's title changed.
|
||||
6. Compute `Due` from the stale list, in plan order:
|
||||
- If `PilotSlugs` is non-empty, keep only those slugs. Otherwise, if `MaxArticles` is set, keep only articles among the first `MaxArticles` of the plan.
|
||||
- Drop a subject whose `state.Subjects[slug]` is `Rejected` with the same `SubjectFingerprint`; count it as `Remembered`.
|
||||
- Take at most `MaxArticlesPerRun`.
|
||||
7. Write each due article:
|
||||
- success: stage both files and record `(fingerprint, Written, now)`;
|
||||
- `ArticleRejectedException` (includes a non-credit HTTP 400, Task 7): log, count as rejected, record `(fingerprint, Rejected, now)`, continue. The old version stays;
|
||||
- other `HttpRequestException`: log, count as rejected, record nothing, continue. The next night retries;
|
||||
- `ModelUnavailableException`: stop the loop and set `StoppedByModel=true`. Keep what is staged.
|
||||
8. **Deletes.** Only when step 3 produced a new plan: for each slug in the previous plan and not in the new one, stage a `Delete` in both languages for the files that exist, and drop its state entry. Articles outside both plans, and articles excluded by `PilotSlugs` or `MaxArticles`, are never deleted.
|
||||
9. Stage `_state.json` only when its serialized content changed. If anything is staged, call `CommitAsync` once with the message `articles: N written, M deleted, R rejected`.
|
||||
- On `StoreConflictException`, log and return. The next run retries.
|
||||
- Idempotence: after its own commit and a sync, nothing is stale or due and the state is unchanged, so a second run makes no model call and no commit. Add no extra guard.
|
||||
|
||||
- [ ] **Step 1: Write failing tests**, using `FakeStore`, `FakeCatalog`, `FakeTextModel` and `FakeTimeProvider`. Seed the catalog state with the store's `Head` unless the test says otherwise.
|
||||
- `Reconcile_FirstRun_PlansWritesCommitsOnce`: exactly one commit, containing `_plan.json`, `_state.json`, plus `de/` and `en/` files.
|
||||
- `Reconcile_CatalogBehindHead_MakesNoModelCalls`: catalog state `c1`, store head `c2`; zero model calls, zero commits, `Skipped == "catalog-behind"`.
|
||||
- `Reconcile_AfterOwnCommit_IsNoop`: run (the fake commit advances `Head`, Task 4), sync the catalog with `SyncUseCase.RunAsync(force: false)`, which now sees a new head, then run again. The second run passes the freshness gate and makes zero commits and zero model calls.
|
||||
- `Reconcile_SourceChanged_RewritesOnlyThat`.
|
||||
- `Reconcile_RespectsMaxArticlesPerRun`.
|
||||
- `Reconcile_ModelUnavailable_CommitsWhatIsDone`: the second article throws. The commit contains the first article, and `StoppedByModel` is true.
|
||||
- `Reconcile_PlanModelUnavailable_StopsKeepsPlan`: `PlanAsync` throws; zero commits, `_plan.json` unchanged, `StoppedByModel` true.
|
||||
- `Reconcile_Rejected_KeepsOldVersion`: no change for that article's paths; `_state.json` records `Rejected`.
|
||||
- `Reconcile_RejectedSameInputs_NotRetried`: a second run with unchanged sources makes no model call for that subject and counts it `Remembered`.
|
||||
- `Reconcile_RejectedInputsChanged_Retried`: after one source's hash changes, the subject is written again.
|
||||
- `Reconcile_Http400_RememberedAsRejected`.
|
||||
- `Reconcile_PlanFailedSameNotes_DoesNotReplan`: planning returns garbage; a second run on the same notes makes zero model calls.
|
||||
- `Reconcile_NoPlan_WritesNothing`: no plan exists and planning fails; no `articles/de/` or `articles/en/` path is touched, the only committed path is `articles/_state.json`, and existing article files remain.
|
||||
- `Reconcile_SubjectDropped_DeletesBothLanguages`: the new plan lacks a slug the previous plan had.
|
||||
- `Reconcile_ArticleNotInAnyPlan_NotDeleted`.
|
||||
- `Reconcile_PilotCapShrinks_DeletesNothing`: three articles exist; `PilotSlugs` is changed to one of them; no `Delete` is staged and no commit is made.
|
||||
- `Reconcile_PilotSlugs_WritesOnlyListed`.
|
||||
- `Reconcile_StoreConflict_ReturnsWithoutThrow`.
|
||||
- [ ] **Step 2: Run them.** Expected: FAIL.
|
||||
- [ ] **Step 3: Implement.**
|
||||
- [ ] **Step 4: Run them.** Expected: PASS.
|
||||
- [ ] **Step 5: Commit.** Message: `feat(encyclopedia): incremental reconciler with single-commit writes`.
|
||||
|
||||
---
|
||||
|
||||
### Task 9: Composition and post-sync trigger
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/Wiki.Host/Composition/WikiOptions.cs` (new `EncyclopediaOptions Encyclopedia`)
|
||||
- Create: `src/Wiki.Application/Encyclopedia/EncyclopediaCoordinator.cs`
|
||||
- Create: `src/Wiki.Host/Composition/EncyclopediaBackgroundService.cs`
|
||||
- Modify: `src/Wiki.Application/Status/StatusQuery.cs`, `src/Wiki.Host/Components/Pages/Status.razor`
|
||||
- Modify: `src/Wiki.Host/Program.cs`, `src/Wiki.Host/appsettings.json`
|
||||
- Test: `test/Wiki.Application.Tests/EncyclopediaCoordinatorTests.cs`, `test/Wiki.Host.Tests/EncyclopediaBackgroundServiceTests.cs`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `IEncyclopediaRun` and `ReconcileSummary` (Task 8), `SyncCoordinator.RunAsync(bool force, CancellationToken ct)`, `TimeProvider`.
|
||||
- Produces:
|
||||
```csharp
|
||||
public sealed class EncyclopediaOptions
|
||||
{
|
||||
public bool Enabled { get; set; } = false;
|
||||
public string AnswerModel { get; set; } = "claude-sonnet"; // key read from Wiki:AnswerModels:Models:{AnswerModel}:ApiKey
|
||||
public string Model { get; set; } = "claude-sonnet-5";
|
||||
public TimeOnly RunAt { get; set; } = new(6, 0); // local time in TimeZone; after the 04:30 distill
|
||||
public string TimeZone { get; set; } = "Europe/Berlin";
|
||||
public int ArticleMaxTokens { get; set; } = 16000;
|
||||
public int PlanMaxTokens { get; set; } = 32000;
|
||||
public int MaxArticlesPerRun { get; set; } = 10;
|
||||
public int MaxSourceCharsPerArticle { get; set; } = 300_000;
|
||||
public List<string> PilotSlugs { get; set; } = []; // when non-empty, only these subjects are written
|
||||
public int? MaxArticles { get; set; } = null; // fallback write cap: first N planned subjects
|
||||
}
|
||||
// Application: one run at a time, failures recorded, never thrown (the SyncCoordinator pattern)
|
||||
public sealed class EncyclopediaCoordinator(IEncyclopediaRun run, TimeProvider clock, ILogger<EncyclopediaCoordinator> log)
|
||||
{
|
||||
public bool Running { get; }
|
||||
public ReconcileSummary? LastRun { get; }
|
||||
public string? LastError { get; }
|
||||
public DateTimeOffset? LastAttemptAt { get; }
|
||||
public Task<ReconcileSummary?> RunAsync(CancellationToken ct);
|
||||
}
|
||||
// Host
|
||||
public sealed class EncyclopediaBackgroundService(SyncCoordinator sync, EncyclopediaCoordinator encyclopedia,
|
||||
EncyclopediaOptions options, TimeProvider clock, ILogger<EncyclopediaBackgroundService> log) : BackgroundService;
|
||||
```
|
||||
- `appsettings.json` gets `"Encyclopedia": { "Enabled": false }`.
|
||||
- When `Enabled` is false, register nothing encyclopedia-related besides the options.
|
||||
- **Schedule.** The service runs once a night at `RunAt` in `TimeZone`. It does not run on startup. It computes the next occurrence from `clock.GetUtcNow()` via `TimeZoneInfo` (so DST is handled) and waits with `Task.Delay(delay, clock, ct)`, so tests drive it with `FakeTimeProvider`. A run takes as long as it takes; the next occurrence is computed after it ends, so runs never overlap.
|
||||
- **Before each run** it awaits `sync.RunAsync(force: false, ct)` so the catalog is current. The Task 8 freshness gate stays as a safety check. Sync and the push webhook no longer trigger the encyclopedia; the reconciler's own commit causes an ordinary sync that does not reconcile.
|
||||
- **Never stops the host.** `EncyclopediaCoordinator.RunAsync` catches every exception except `OperationCanceledException` when `ct` is cancelled, sets `LastError` (message truncated to 300 chars), logs and returns null, exactly like `SyncCoordinator.RunAsync`. This covers a missing Anthropic key (`InvalidOperationException` from `MessagesTransport`), a Gitea error, and anything else. The service loop wraps each iteration in the same catch, so even a sync or scheduling bug only logs.
|
||||
- `EncyclopediaSettings` is mapped from the options in `Program.cs`.
|
||||
- Register the `ITextModel` via `AddAnthropicTextModel`, using `Model` and the API key described above, and add it to the `IReadOnlyList<IAdapterHealth>` list in `Program.cs` (`.. sp.GetServices<ITextModel>()`, empty when disabled) so `/status` shows it.
|
||||
- `StatusQuery` takes an optional `EncyclopediaCoordinator? encyclopedia = null` and adds `ReconcileSummary? LastEncyclopediaRun`, `string? LastEncyclopediaError` and `DateTimeOffset? LastEncyclopediaAttemptAt` to its output. `/status` shows them.
|
||||
|
||||
- [ ] **Step 1: Write failing tests.**
|
||||
- `Coordinator_RunThrows_RecordsTruncatedErrorReturnsNull`: the fake run throws `InvalidOperationException` with a 400-char message; `RunAsync` returns null and `LastError` has 300 chars.
|
||||
- `Coordinator_Cancelled_Propagates`.
|
||||
- `Service_RunsOncePerDayAtRunAt`: start the `FakeTimeProvider` at 05:00 Berlin. No run at start; one run after advancing to 06:00; still one at 05:59 the next day; two at 06:00.
|
||||
- `Service_DoesNotRunOnStartup`: start at 07:00; no run until 06:00 the next day.
|
||||
- `Service_SyncsBeforeRun`: the store head moves to `c2` before 06:00; when the fake run is invoked, the catalog state commit is `c2`.
|
||||
- `Service_ReconcilerThrows_KeepsRunning`: the fake run throws `JsonException` on the first night; the service is still running, `LastError` is set, and the second night runs again.
|
||||
- `Host_EncyclopediaDisabled_NoTextModelRegistered`: build the `WikiHostFactory` and assert `ITextModel` is not resolvable.
|
||||
- `Host_EncyclopediaEnabled_TextModelInAdapterHealth`: the resolved `IReadOnlyList<IAdapterHealth>` contains an adapter named `anthropic-text:…`.
|
||||
- [ ] **Step 2: Run them.** Expected: FAIL.
|
||||
- [ ] **Step 3: Implement.**
|
||||
- [ ] **Step 4: Run the suite.** Expected: PASS.
|
||||
- [ ] **Step 5: Commit.** Message: `feat(host): run the encyclopedia nightly after a sync`.
|
||||
|
||||
---
|
||||
|
||||
### Task 10: Article rendering
|
||||
|
||||
**Files:**
|
||||
- Create: `src/Wiki.Host/Rendering/ArticleMarkdown.cs`
|
||||
- Create: `src/Wiki.Host/wwwroot/lib/mermaid/mermaid.min.js` (vendored `mermaid@11.4.1` `dist/mermaid.min.js`, with its LICENSE beside it)
|
||||
- Create: `src/Wiki.Host/wwwroot/house.css`
|
||||
- Modify: `src/Wiki.Core/Model/Attachment.cs` (expose the extension list)
|
||||
- Test: `test/Wiki.Host.Tests/ArticleMarkdownTests.cs`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `ArticleBody.RedactPrivate` (Task 2), `Attachment` (`src/Wiki.Core/Model/Attachment.cs`).
|
||||
- Produces:
|
||||
```csharp
|
||||
public sealed record TocEntry(int Level, string Id, string Text);
|
||||
public sealed record RenderedArticle(string Html, string? InfoboxHtml, IReadOnlyList<TocEntry> Toc);
|
||||
public static class ArticleMarkdown
|
||||
{
|
||||
// signedIn=false ⇒ body passed through RedactPrivate first.
|
||||
// visibleTitles: slug → title in `lang` of every article this reader may see.
|
||||
public static RenderedArticle Render(string body, string lang, string slug, bool signedIn, IReadOnlyDictionary<string, string> visibleTitles);
|
||||
// the valid raw/ image paths of a body, in order; Render and Task 11 both index into this list
|
||||
public static IReadOnlyList<string> ImagePaths(string visibleBody);
|
||||
}
|
||||
// Attachment.cs addition
|
||||
public static IReadOnlyCollection<string> FileExtensions { get; } // the values of the existing Extensions map: jpg, png, webp, gif
|
||||
```
|
||||
- Rendering rules:
|
||||
- **Pipeline:** `UseAdvancedExtensions().DisableHtml()`, which gives custom containers, auto-identifiers and diagrams. Raw HTML stays disabled, so the model can never inject markup.
|
||||
- **Infobox:** the `::: infobox` container is extracted into `InfoboxHtml` and removed from `Html`.
|
||||
- **Private blocks:** `::: private` renders as `<div class="private">` with a lock label. `::: private-locked` renders as `<div class="private-locked">`, containing the localized "Privater Abschnitt — anmelden zum Lesen" or "Private section — log in to read" and a link to `/house/login?returnUrl=…`.
|
||||
- **Links:** `[[slug]]` with `slug` in `visibleTitles` renders as a link to `/house/{lang}/{slug}` whose text is the title. A slug not in `visibleTitles` is dropped entirely, text included, for every reader: anonymous readers must not learn a private subject's name, and signed-in readers see no dead links. Dropping a link also drops an adjacent list separator (`, ` or ` · `), and a list item or infobox row left with no text is removed. This covers "related" lists and infobox entries, which are written as `[[slug]]` too.
|
||||
- **Mermaid:** a ```` ```mermaid ```` fence becomes `<pre class="mermaid">` or Markdig's `<div class="mermaid">`. Pin whichever Markdig 1.4.0 emits in the test.
|
||||
- **Images:** `ImagePaths` returns the image references whose URL matches `^raw/[A-Za-z0-9._-]+\.(jpg|png|webp|gif)$`, the extension group built from `Attachment.FileExtensions`. Those are rewritten to `/house/media/{lang}/{slug}/{n}`, where `n` is the index into `ImagePaths` of the redacted body. Any other image reference (other prefix, `..`, a subdirectory, a non-image extension, an external URL) is dropped. Task 11 serves them.
|
||||
- **TOC:** built from the h2 and h3 headings, with the ids Markdig assigns. Each TOC entry links to `#id`.
|
||||
- **Overlap fix:** in `house.css`, the infobox floats right. `.mermaid`, `figure`, `table` and `h2` get `clear: right`, so a diagram never sits under the infobox. This is the mockup bug the owner flagged.
|
||||
|
||||
- [ ] **Step 1: Write failing tests.**
|
||||
- `Render_Anonymous_ContainsNoPrivateText`: the private block contains `GEHEIM`, and `Html` does not.
|
||||
- `Render_SignedIn_ShowsPrivateBlock`.
|
||||
- `Render_InfoboxExtracted`.
|
||||
- `Render_TocMatchesHeadingIds`: each `TocEntry.Id` appears as `id="…"` in `Html`.
|
||||
- `Render_LinkUsesTitle`: `[[heating]]` with `visibleTitles["heating"] = "Heizung"` renders `<a href="/house/de/heating">Heizung</a>`.
|
||||
- `Render_Anonymous_LinkToInvisibleSlug_DroppedWithText`: neither `network-internals` nor its title appears in `Html`.
|
||||
- `Render_InfoboxRowWithOnlyInvisibleLink_Removed`: an infobox row `| Verwandt | [[network-internals]] |` is gone; a row `[[heating]], [[network-internals]]` keeps only the heating link and no trailing comma.
|
||||
- `Render_RawHtmlEscaped`: the input `<script>` does not appear unescaped.
|
||||
- `Render_MermaidFenceBecomesDiagramElement`.
|
||||
- `Render_RawImageRewrittenToMediaEndpoint`: for an anonymous reader, an image inside a private block does not shift the index of a later public image.
|
||||
- `ImagePaths_RejectsTraversalAndNonImages`: `raw/../wiki/x.png`, `raw/sub/x.png`, `raw/x.md`, `raw/x.jpeg` and `https://example.com/x.png` are not returned; `raw/zaehler-1.jpg` is.
|
||||
- [ ] **Step 2: Run them.** Expected: FAIL.
|
||||
- [ ] **Step 3: Implement.** Download Mermaid with `curl -fsSLo src/Wiki.Host/wwwroot/lib/mermaid/mermaid.min.js https://cdn.jsdelivr.net/npm/mermaid@11.4.1/dist/mermaid.min.js` and do the same for `LICENSE`.
|
||||
- [ ] **Step 4: Run them.** Expected: PASS.
|
||||
- [ ] **Step 5: Commit.** Message: `feat(host): article markdown rendering with redaction, infobox, TOC, mermaid`.
|
||||
|
||||
---
|
||||
|
||||
### Task 11: Encyclopedia pages, anonymous access, media
|
||||
|
||||
**Files:**
|
||||
- Create: `src/Wiki.Host/Components/Layout/HouseLayout.razor` (no MudBlazor; Wikipedia-like layout)
|
||||
- Create: `src/Wiki.Host/Components/Pages/House.razor` (`/house`, `/house/{Lang}`)
|
||||
- Create: `src/Wiki.Host/Components/Pages/HouseArticle.razor` (`/house/{Lang}/{Slug}`)
|
||||
- Create: `src/Wiki.Host/Api/HouseEndpoints.cs` (`/house/login`, `/house/media/{Lang}/{Slug}/{N:int}`)
|
||||
- Create: `src/Wiki.Application/Reading/ArticleQueries.cs`
|
||||
- Create: `src/Wiki.Host/Api/HouseMediaCache.cs`
|
||||
- Modify: `src/Wiki.Host/Program.cs` (map endpoints; `MapStaticAssets().AllowAnonymous()`; exclude `/house` from the status-code-pages branch)
|
||||
- Modify: `src/Wiki.Host/Components/Pages/Error.razor` (`@attribute [AllowAnonymous]`)
|
||||
- Modify: `src/Wiki.Host/Components/App.razor` (load `house.css`; load `mermaid.min.js` with `mermaid.initialize({startOnLoad: true, securityLevel: 'strict'})` only on house pages)
|
||||
- Modify: `src/Wiki.Host/Components/Layout/MainLayout.razor` (nav link "Haus")
|
||||
- Test: `test/Wiki.Host.Tests/HouseTests.cs`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes:
|
||||
- `ArticleMarkdown.Render` (Task 10);
|
||||
- `ArticleFile.Parse` (Task 2);
|
||||
- `IDocumentCatalog`;
|
||||
- `INoteStore.ReadBytesAsync` (Task 4);
|
||||
- `ArticleMarkdown.ImagePaths` (Task 10).
|
||||
- Produces:
|
||||
```csharp
|
||||
public sealed class ArticleQueries(IDocumentCatalog catalog)
|
||||
{
|
||||
public async Task<Article?> GetAsync(string lang, string slug, CancellationToken ct);
|
||||
public async Task<IReadOnlyList<Article>> ListAsync(string lang, CancellationToken ct);
|
||||
// slug → title in `lang`; anonymous: only slugs whose de AND en files are both Public
|
||||
public async Task<IReadOnlyDictionary<string, string>> VisibleTitlesAsync(string lang, bool signedIn, CancellationToken ct);
|
||||
public async Task<bool> IsPrivateAsync(string slug, CancellationToken ct); // either language Private ⇒ true
|
||||
}
|
||||
public sealed class HouseMediaCache // singleton; key (commit, path); evicts oldest beyond 64 MiB total
|
||||
{
|
||||
public Task<byte[]> GetAsync(string commit, string path, Func<CancellationToken, Task<byte[]>> load, CancellationToken ct);
|
||||
}
|
||||
```
|
||||
- Behaviour:
|
||||
- **Render mode and auth.** The pages are static SSR (no `@rendermode`), with `@attribute [AllowAnonymous]` and `@layout HouseLayout`. "Signed in" means `User.Identity.IsAuthenticated` and claim `wiki:kind == "human"`.
|
||||
- **`/house`.** Redirects to `/house/de`.
|
||||
- **`/house/{Lang}`.** The front page lists the articles grouped by category (Räume/Rooms, Systeme/Systems, Geräte/Devices, Themen/Topics). For anonymous readers it lists only slugs in `VisibleTitlesAsync(lang, false)`, so an article private in either language never appears, not even by title.
|
||||
- **Article page.**
|
||||
- Shows the title, a "From the house encyclopedia" line with the generated date, the infobox floated right, and the linked TOC. Then the body.
|
||||
- Shows a language switch to the same slug in the other language, and a "log in to see more" link when anonymous.
|
||||
- Passes `VisibleTitlesAsync(lang, signedIn)` to `Render`. Signed-in readers get all existing articles.
|
||||
- **Private article, anonymous reader.** `IsPrivateAsync` decides. Serve a generic page: HTTP 200 with "Dieser Artikel ist privat — anmelden" and a login link. It must not contain the title, the slug's title or the body.
|
||||
- **Missing article or unknown language.** The house page itself sets status 404 and renders a generic localized "Artikel nicht gefunden" / "Article not found" inside `HouseLayout`, with a link to the front page. It does not use `NotFound.razor`: that page uses `MainLayout` and falls under the fallback policy, and `UseStatusCodePagesWithReExecute("/not-found")` runs before `UseAuthentication`/`UseAuthorization` (`Program.cs`), so an anonymous 404 would become a login redirect. Exclude the `/house` prefix from the `UseWhen` status-code-pages branch so house 404s, including media 404s, keep their own response. `Error.razor` gets `[AllowAnonymous]` (it shows only a request id) so a server error on a house page is not a login redirect either.
|
||||
- **Response headers.** Every `/house` response carries `<meta name="robots" content="noindex">` on pages, and the headers `X-Robots-Tag: noindex` and `Cache-Control: no-store`. Add both headers with a small middleware scoped to the `/house` path prefix, registered before the endpoints.
|
||||
- **`/house/login?returnUrl=`.** Requires authentication. The challenge goes to OIDC, then a local redirect to `returnUrl`. Accept only local URLs starting with `/house/`.
|
||||
- **`/house/media/{Lang}/{Slug}/{N}`.** Allowed anonymously.
|
||||
- 404 when the article is missing, or private and the reader anonymous.
|
||||
- Otherwise apply `RedactPrivate` when anonymous and take `ArticleMarkdown.ImagePaths(visibleBody)[N]`. Only paths passing Task 10's `raw/` pattern can be returned, so a model-written `raw/../…` or `raw/x.md` is never read. Out of range: 404.
|
||||
- Read the bytes at the catalog's commit through `HouseMediaCache` (`catalog.GetStateAsync().Commit`, the same commit the article body came from) and send them with the content type mapped from the extension (`jpg` → `image/jpeg`, …).
|
||||
- Images referenced only inside private blocks are therefore unreachable anonymously. The indexing matches Task 10's URL rewrite.
|
||||
- **Everything else stays behind the fallback policy.**
|
||||
|
||||
- [ ] **Step 1: Write failing host tests.** Use `WikiHostFactory`, a `FakeStore` or `FakeCatalog` seeded with one public article containing a private block (`GEHEIM`) and one private article titled `Netzwerk-Interna`. Use the existing `FakeIdp` sign-in helper for signed-in requests.
|
||||
- `PublicArticle_Anonymous_200_NoPrivateText`.
|
||||
- `PublicArticle_SignedIn_ShowsPrivateText`.
|
||||
- `PrivateArticle_Anonymous_ShowsNoTitle`: the body does not contain `Netzwerk-Interna`.
|
||||
- `PublicArticle_Anonymous_NoPrivateArticleTitle`: the public article links `[[network-internals]]` in its body and infobox; the anonymous response contains neither `network-internals` nor `Netzwerk-Interna`.
|
||||
- `FrontPage_Anonymous_ListsOnlyPublic`.
|
||||
- `FrontPage_Anonymous_HidesArticlePrivateInOneLanguage`: an article whose `en` file is private and `de` file public is not listed on `/house/de`.
|
||||
- `MissingArticle_Anonymous_Is404NotRedirect`: `GET /house/de/does-not-exist` returns 404, no `Location` header, the localized not-found text, and the `HouseLayout` markup.
|
||||
- `House_SendsNoindexAndNoStore`: header `X-Robots-Tag: noindex`, header `Cache-Control: no-store`, and the meta tag.
|
||||
- `House_MermaidInitStrict`: the article page contains `securityLevel: 'strict'`.
|
||||
- `Notes_Anonymous_StillRejected`: `GET /notes/x` without auth is not 200. `/api/v1/notes` returns 401.
|
||||
- `Login_RejectsNonHouseReturnUrl`: `returnUrl=https://evil` redirects to `/house/de`.
|
||||
- `Media_ImageOnlyInPrivateBlock_Anonymous404`.
|
||||
- `Media_PublicImage_Anonymous200`: correct bytes and `Content-Type: image/png`.
|
||||
- `Media_PathTraversal_404`: the article's only image is `raw/../articles/_state.json`; `/house/media/de/x/0` is 404 and the store's `ReadBytesAsync` was not called.
|
||||
- `Media_NonImage_404`: image `raw/notes.md` gives 404.
|
||||
- `Media_SecondRequest_ServedFromCache`: two requests read the store once.
|
||||
- `StaticAssets_Anonymous200`: `/house.css` returns 200.
|
||||
- [ ] **Step 2: Run them.** Expected: FAIL.
|
||||
- [ ] **Step 3: Implement.**
|
||||
- [ ] **Step 4: Run the full suite.** Expected: PASS.
|
||||
- [ ] **Step 5: Commit.** Message: `feat(host): house encyclopedia pages with anonymous public view`.
|
||||
- [ ] **Step 6: Push and open a PR** to `main` as claude-bot. Wait for the Woodpecker `test` pipeline to go green. The owner merges; never self-merge. A release image is built on merge, and Flux image automation rolls it out.
|
||||
|
||||
---
|
||||
|
||||
### Task 12 (homelab repo): group gate and enabling the pilot
|
||||
|
||||
The work is in a separate repo, `/home/krenner/source/homelab`. Create a worktree with `EnterWorktree`, use a branch, and open a PR as claude-bot.
|
||||
|
||||
**Files:**
|
||||
- Modify: `kubernetes/apps/authentik/blueprints-configmap.yaml`. Add the group, and add a policy binding of that group to the `wiki` application. Update the comment "No policy binding: every Authentik user may sign in."
|
||||
- Modify: `kubernetes/apps/wiki/deployment.yaml`. Add the env vars `Wiki__Encyclopedia__Enabled=true` and `Wiki__Encyclopedia__MaxArticles=0`. The first nightly run then only plans: slugs come from the planner and are not known before it runs, so `PilotSlugs` cannot be filled yet. `MaxArticles` stays as the fallback cap; a non-empty `PilotSlugs` takes precedence over it.
|
||||
|
||||
- [ ] **Step 1: Check the blueprint format.** Read how other OAuth applications in the blueprint bind a group, if any do, and confirm the `authentik_policies.policybinding` model fields against the Authentik docs for the deployed version before writing it. Add:
|
||||
```yaml
|
||||
# wiki.renner.uno signs in via OIDC application `wiki`; only this group may sign in.
|
||||
- model: authentik_core.group
|
||||
identifiers:
|
||||
name: Wiki Readers
|
||||
state: present
|
||||
attrs:
|
||||
is_superuser: false
|
||||
```
|
||||
Bind it with `group: !Find [authentik_core.group, [name, Wiki Readers]]` and `target: !Find [authentik_core.application, [slug, wiki]]`, `order: 0`.
|
||||
- [ ] **Step 2: Check before opening the PR.** Run `kustomize build kubernetes/apps/authentik` and `kustomize build kubernetes/apps/wiki`. Both must render without error. The rendered `wiki` Ingress must carry neither the `dns.renner.uno/public` nor the `netbird.renner.uno/public` label; either one publishes the host through `netbird-service-sync`.
|
||||
- [ ] **Step 3: Open the PR.** State in it that the owner must add their own account, and anyone else who should see private content, to "Wiki Readers" before merging. Otherwise the owner gets locked out of the wiki UI. Membership lives in the Authentik DB, not the blueprint.
|
||||
- [ ] **Step 4: Verify after the owner merges and Flux applies.**
|
||||
- An anonymous `curl -sI https://wiki.renner.uno/house/de` returns 200 with `X-Robots-Tag: noindex` and `Cache-Control: no-store`.
|
||||
- An anonymous `curl -sI https://wiki.renner.uno/notes/x` does not return 200.
|
||||
- `/status` lists the `anthropic-text:…` adapter as healthy.
|
||||
- After the next 06:00 run, `/status` shows `LastEncyclopediaRun` with `Planned > 0`, `Written == 0`, and `articles/_plan.json` exists in `konairius/notes`.
|
||||
- [ ] **Step 5: Pick the pilot slugs.** Read `articles/_plan.json` and find the slugs the planner gave the three spec pilot subjects ("Wall switches", "Heating", "EG Schlafzimmer"). Ask the owner to confirm them, then open a follow-up PR adding `Wiki__Encyclopedia__PilotSlugs__0..2` with those slugs. Keep `MaxArticles=0`.
|
||||
- [ ] **Step 6: Verify the pilot** after the next 06:00 run:
|
||||
- `/status` shows `LastEncyclopediaRun` with `Written == 3` (or the rejections, with `LastEncyclopediaError` empty).
|
||||
- `articles/de/` and `articles/en/` in `konairius/notes` hold exactly the 3 pilot articles each.
|
||||
- In a browser, signed in as a group member, a private block is visible.
|
||||
- [ ] **Step 7: Hand the 3 pilot articles to the owner for review.** Removing `PilotSlugs` and `MaxArticles` is a follow-up PR after their approval.
|
||||
@@ -0,0 +1,123 @@
|
||||
# House encyclopedia — design
|
||||
|
||||
Date: 2026-10-24. Target codebase: the .NET Wiki service (`konairius/Wiki`), deployed in the homelab cluster (`kubernetes/apps/wiki`).
|
||||
|
||||
## Intent
|
||||
|
||||
The wiki service already holds most of what is known about the house (≈2 000 distilled notes: switches per room, the Schalter-Konvention, the Vitocal heat pump, Zigbee, network…). Those notes are fact- and incident-shaped and written for an agent: correct, but not readable by a person who just wants to understand the house.
|
||||
|
||||
The goal is a **Wikipedia-style encyclopedia of the house**, generated from the existing wiki, that anyone who needs to know something about the house can read:
|
||||
|
||||
- the owner,
|
||||
- the household,
|
||||
- a friend fixing a non-working light switch while the owner is on a business trip,
|
||||
- a guest who wants to know how the lighting automation works.
|
||||
|
||||
Readers range from non-technical to expert, so every article opens plainly and gets more technical further down.
|
||||
|
||||
## Goals
|
||||
|
||||
1. **Articles in Wikipedia form.** Each article has:
|
||||
- a lead paragraph a non-technical reader understands;
|
||||
- a **properties panel** (infobox) on the right with the key facts;
|
||||
- a **linked** table of contents;
|
||||
- a body ordered **from general principles to concrete examples** (e.g. "How it works" → "The rules" → "Examples by room");
|
||||
- **diagrams** that explain mechanisms, and **photos** where the wiki has them;
|
||||
- optionally a **History** section, which stays short and comes last.
|
||||
2. **Articles for rooms and for systems, linked to each other**, like Wikipedia: "EG Küche" links to "Lighting automation", "Heating" links to the rooms it serves. A front page is just an entry point into this web.
|
||||
3. **Granularity follows the material, like notability on Wikipedia.** A subject gets its own article when the wiki holds enough on it: e.g. a heating valve controller with 17 automations and a complicated history. A subject that was set once and forgotten is a section of its parent system's article.
|
||||
4. **Bilingual: German and English.** Every article exists in both languages, with a language switch on the page.
|
||||
5. **Public and private.**
|
||||
- Articles are public by default and readable without login.
|
||||
- **Private sections** inside a public article are visible only to members of the wiki group after an Authentik login. In the approved mockup this is a locked block within the article; logged-out readers see that something is there but not the content.
|
||||
- Whole articles can be private.
|
||||
- **The AI writing the article decides what is private. When unsure, it marks the content private.**
|
||||
6. **Generated, never hand-written.** Articles are produced from the wiki by the AI and regenerated nightly, after the distill job. New facts become wiki notes through the nightly distill job, and the articles built on those notes are then refreshed. Nightly freshness is acceptable to the owner.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Human editing of articles. The wiki is the source; corrections go in as wiki facts.
|
||||
- Replacing the existing wiki UI or the notes. The encyclopedia is an extra layer on top.
|
||||
- A separate application. It lives inside the current Wiki service and reuses its Anthropic key and config.
|
||||
|
||||
## Anti-goals: a failure even if it technically works
|
||||
|
||||
- **Private content reachable by anyone outside the wiki group**: hostnames, IP addresses, device IDs, credentials, secret paths, network and security layout, server and restart procedures, and anything else that helps an attacker. This includes the raw notes, the search, and the existing APIs being reachable without login because encyclopedia routes were opened.
|
||||
- Articles that read like the source notes: incident logs, PR numbers, entity IDs in the main text, history dominating.
|
||||
- Invented facts. Every claim must come from wiki notes.
|
||||
|
||||
## Constraints
|
||||
|
||||
- Stack: inside `konairius/Wiki`: .NET (`net10.0`), Blazor Server with MudBlazor, ports-and-adapters layout. Notes are stored in the Gitea repo `konairius/notes` via the existing `INoteStore`; markdown is rendered with Markdig. The LLM is the existing Anthropic adapter. Its key comes from the ExternalSecret `wiki-anthropic`.
|
||||
- Auth: `wiki.renner.uno` is plain Traefik, LAN-only. The service signs people in itself with OIDC against the Authentik application `wiki`. Every page requires an authenticated user via the fallback policy. Today any Authentik user may sign in, because there is no policy binding. Machine clients use API keys or BasicAuth.
|
||||
- Homelab rules:
|
||||
- every change goes through git and a PR;
|
||||
- the CiliumNetworkPolicy, gateway and egress house patterns apply;
|
||||
- nothing exists only on a PVC (generated articles must be in git).
|
||||
- Reconciliation: the service re-syncs from Gitea every 30 min and on the push webhook (`SyncBackgroundService` / `SyncUseCase`). Distillation from `raw/` into `wiki/` runs nightly at 04:30 on the workstation.
|
||||
- Images: the wiki accepts photo attachments on facts (`raw/<stem>-N.<ext>`) since 2026-09-25. Today there are almost none, and the UI does not yet serve them.
|
||||
- The current Anthropic adapter is configured for short answers (`MaxTokens` 1200). Articles are much longer.
|
||||
|
||||
## Decisions made by the owner
|
||||
|
||||
- It is an encyclopedia layer generated from the current wiki, not written by hand.
|
||||
- Its audience is everyone who needs to know things about the house, from the owner to guests.
|
||||
- The article shape is the approved mockup: lead, infobox, linked TOC, principle → examples, short history last, and private sections as locked blocks.
|
||||
- The property panel must never overlap diagrams or figures. This was a layout bug in the mockup.
|
||||
- Articles cover rooms and systems, linked; there is no separate navigation scheme.
|
||||
- The split is public and private, with login via Authentik. The AI decides, and private is the default when it is unsure.
|
||||
- Articles are in German and English.
|
||||
- It lives inside the current Wiki software and uses the existing AI key.
|
||||
- Articles are regenerated nightly, after the distill job, not after every sync.
|
||||
- A subject gets its own article when the material warrants it; otherwise it is a section of its parent article.
|
||||
- **Scope covers the whole wiki**: the house and the homelab IT behind it (Kubernetes, servers, network). Infrastructure articles exist too. Their sensitive parts are private under the same rule.
|
||||
- **Private content is visible only to Authentik accounts in the matching wiki group.** A plain login is not enough. No such group exists yet. It is created following the house pattern: the group is declared in the Authentik blueprint without `users:`, and membership is managed in Authentik. The group is bound to the `wiki` OIDC application, so only members can sign in to the wiki at all. Friends and helpers get access by being added to that group.
|
||||
- **The site is not on the internet.** Both the public and the logged-in hostname are reachable only from the LAN or via NetBird. There is no external DNS record and no route through the public gateway. "Public" means "no login needed on the home network or VPN", not internet-public.
|
||||
- **Spending limits are enforced on the Anthropic API side.** The service needs no budget logic of its own beyond incremental regeneration. It must handle a limit being reached: articles stay as last generated and the job retries later.
|
||||
|
||||
## Calls made by the designer (owner to check)
|
||||
|
||||
- **Articles are stored in the notes repo** under their own prefix, e.g. `articles/<lang>/<slug>.md`. Frontmatter records:
|
||||
- the source note slugs and their content hashes;
|
||||
- the article's visibility;
|
||||
- the language.
|
||||
|
||||
Private sections are marked in the body. This keeps generated content in git, like everything else, and lets regeneration skip articles whose sources are unchanged.
|
||||
- **Regeneration is incremental.** Each night, after the distill job, an article is regenerated only when one of its source notes changed, or when the set of articles itself changes. Planning the article set is a separate, rarer step: which subjects deserve an article, and which notes feed each one. That step runs when notes are added or removed.
|
||||
- **Failures are remembered.** An article the pipeline rejected is retried only when its sources change, and a failed planning attempt only when the set of notes changes. Retrying the same input every night would only spend money.
|
||||
- **There is one hostname: `wiki.renner.uno`.** The service already does OIDC itself, so it knows whether a reader is signed in. Only wiki-group members can sign in, because of the policy binding.
|
||||
- The encyclopedia routes allow anonymous readers. Every other route keeps requiring a login.
|
||||
- For an anonymous reader, private sections and private articles are left out of the HTML entirely, not merely hidden. Links and listings pointing at a private article are left out too, so its title never appears. A "log in to see more" link starts the OIDC login.
|
||||
- The host stays LAN- and NetBird-only, as it is today: the `wiki` Ingress carries neither the `dns.renner.uno/public` nor the `netbird.renner.uno/public` label. Either one makes `netbird-service-sync` publish the host.
|
||||
- **Articles go live without human approval**, as the owner decided. The service commits generated articles directly to the notes repo, the same way it already commits raw deposits. Leak protection is therefore entirely automatic and has three layers:
|
||||
- the writer marks content private;
|
||||
- a deterministic filter on the public text catches IP addresses (v4 and v6), hostnames and FQDNs, HA entity IDs, MACs, file paths, and OpenBao or secret paths. It only catches this house's own identifiers: a match counts when the value appears in the article's source notes or is a name under `renner.uno`. A match moves the paragraph into a private section;
|
||||
- the independent LLM check described in the next item.
|
||||
- **Generic examples stay public.** An example that explains a principle without naming anything concrete in this house, e.g. what an entity ID or an IP address looks like, belongs in the public text. The writer illustrates formats with reserved example values (`192.0.2.x`, `2001:db8::`, `example.com`), and those never count as leaks. "Private when unsure" applies to facts about this house, not to generic examples.
|
||||
- **When notes disagree or are out of date**, the article states the newest confirmed fact and leaves out anything that stays in dispute. It never asserts a disputed claim confidently to a non-technical reader.
|
||||
- **Group membership covers the whole wiki.** Binding the group to the `wiki` application restricts the whole existing wiki UI, including raw notes and search, to that group. That matches "accounts in the matching wiki group". It also means a helper added to the group can browse the raw notes, not just the private sections.
|
||||
- **Public pages send `noindex`**, as a precaution even on the LAN.
|
||||
- **Two passes guard against leaks.** After writing an article, a second, independent check reads only its public parts and flags anything that looks private. Generic examples are not flagged. Flagged content moves into a private section. When something is in doubt, the private side wins.
|
||||
- **German is written first and English is translated from it**, so both versions say the same thing. The order of the two languages is minor and left to the builder. Translation goes paragraph by paragraph, and the private markers are copied over by the service, never by the AI, so a private section cannot become public in translation.
|
||||
- **Diagrams are produced as text, e.g. Mermaid or a constrained SVG subset, and rendered in the page.** They are not image files generated outside the service. The builder picks the format.
|
||||
- **A photo follows the same visibility rule as text.** Interior photos showing devices or equipment are private unless clearly harmless.
|
||||
|
||||
## Left to the builder
|
||||
|
||||
- Model choice and token budget per article, and how the article-set planning step is prompted.
|
||||
- Exact URL scheme, front page layout, and styling. The style should be close to Wikipedia's look; the approved mockup is a reference.
|
||||
- How links between articles are resolved, including links to articles that don't exist yet.
|
||||
- Serving of attachment images.
|
||||
- Rollout order. A sensible pilot is a handful of articles, e.g. "Wall switches", "Heating", "EG Schlafzimmer", reviewed before generating the whole set.
|
||||
|
||||
## Reference mockup
|
||||
|
||||
The approved mockup is an article on "Wall switches", built from the note `schalter-konvention-blueprint-system`. It has:
|
||||
|
||||
- an infobox listing type, brands, controller, rule set, "works without network?", rooms, and related articles;
|
||||
- the lead sentence: "Wall switches in the house are push buttons that don't switch power directly…";
|
||||
- a flow diagram: switch → Home Assistant → lamp/blinds;
|
||||
- a gesture table;
|
||||
- room examples for EG Schlafzimmer and EG Bad;
|
||||
- a troubleshooting section with a private block (device names, restart procedure, which server);
|
||||
- a two-line History section.
|
||||
@@ -1,8 +1,3 @@
|
||||
using System.Diagnostics;
|
||||
using System.Runtime.CompilerServices;
|
||||
using System.Text;
|
||||
using System.Text.Json;
|
||||
using System.Text.Json.Nodes;
|
||||
using Wiki.Core.Model;
|
||||
using Wiki.Core.Ports;
|
||||
|
||||
@@ -10,216 +5,28 @@ namespace Wiki.Adapters.Anthropic;
|
||||
|
||||
/// <summary>
|
||||
/// Answers questions via Anthropic's Messages API, streamed over server-sent events. One HTTP
|
||||
/// request per <see cref="AnswerAsync"/> call, retried on 429/529 with the server's Retry-After
|
||||
/// (or exponential backoff), never retried for anything else.
|
||||
/// request per <see cref="AnswerAsync"/> call; see <see cref="MessagesTransport"/> for retries.
|
||||
/// </summary>
|
||||
public sealed class AnthropicAnswerModel : IAnswerModel
|
||||
{
|
||||
private const string AnthropicVersion = "2023-06-01";
|
||||
|
||||
private static readonly JsonSerializerOptions JsonOptions = new(JsonSerializerDefaults.Web);
|
||||
|
||||
private readonly HttpClient _http;
|
||||
private readonly AnthropicOptions _options;
|
||||
private readonly Func<TimeSpan, CancellationToken, Task> _delay;
|
||||
private readonly MessagesTransport _transport;
|
||||
|
||||
public AnthropicAnswerModel(HttpClient http, AnthropicOptions options, Func<TimeSpan, CancellationToken, Task>? delay = null)
|
||||
{
|
||||
_http = http;
|
||||
_options = options;
|
||||
_delay = delay ?? Task.Delay;
|
||||
_transport = new MessagesTransport(http, options, delay);
|
||||
ModelId = options.ModelId;
|
||||
}
|
||||
|
||||
public string ModelId => _options.ModelId;
|
||||
public string ModelId { get; }
|
||||
|
||||
public string Name => $"anthropic:{ModelId}";
|
||||
|
||||
public async IAsyncEnumerable<AnswerChunk> AnswerAsync(AnswerRequest request, [EnumeratorCancellation] CancellationToken ct)
|
||||
public IAsyncEnumerable<AnswerChunk> AnswerAsync(AnswerRequest request, CancellationToken ct)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(request);
|
||||
ct.ThrowIfCancellationRequested();
|
||||
|
||||
var (system, user) = AnswerPrompt.Build(request);
|
||||
var body = BuildRequestBody(system, user);
|
||||
|
||||
var stopwatch = Stopwatch.StartNew();
|
||||
using var response = await PostWithRetryAsync(body, ct).ConfigureAwait(false);
|
||||
var stream = await response.Content.ReadAsStreamAsync(ct).ConfigureAwait(false);
|
||||
await using (stream.ConfigureAwait(false))
|
||||
using (var reader = new StreamReader(stream, Encoding.UTF8))
|
||||
{
|
||||
long inputTokens = 0;
|
||||
long outputTokens = 0;
|
||||
string? eventName = null;
|
||||
|
||||
string? line;
|
||||
while ((line = await reader.ReadLineAsync(ct).ConfigureAwait(false)) is not null)
|
||||
{
|
||||
if (line.Length == 0)
|
||||
{
|
||||
eventName = null;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (line.StartsWith("event: ", StringComparison.Ordinal))
|
||||
{
|
||||
eventName = line["event: ".Length..];
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!line.StartsWith("data: ", StringComparison.Ordinal))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var data = line["data: ".Length..];
|
||||
using var payload = JsonDocument.Parse(data);
|
||||
var root = payload.RootElement;
|
||||
|
||||
switch (eventName)
|
||||
{
|
||||
case "message_start":
|
||||
inputTokens = root.GetProperty("message").GetProperty("usage").GetProperty("input_tokens").GetInt64();
|
||||
break;
|
||||
|
||||
case "content_block_delta":
|
||||
var delta = root.GetProperty("delta");
|
||||
if (delta.GetProperty("type").GetString() == "text_delta")
|
||||
{
|
||||
yield return new AnswerChunk.Text(delta.GetProperty("text").GetString() ?? "");
|
||||
}
|
||||
|
||||
break;
|
||||
|
||||
case "message_delta":
|
||||
outputTokens = root.GetProperty("usage").GetProperty("output_tokens").GetInt64();
|
||||
break;
|
||||
|
||||
case "error":
|
||||
throw new HttpRequestException($"anthropic: {root.GetProperty("error").GetProperty("message").GetString()}");
|
||||
|
||||
// message_stop, ping, content_block_start, content_block_stop: nothing to record.
|
||||
default:
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
stopwatch.Stop();
|
||||
yield return new AnswerChunk.Done(ModelId, inputTokens, outputTokens, stopwatch.Elapsed);
|
||||
}
|
||||
return _transport.StreamAsync(system, user, ModelId, ct);
|
||||
}
|
||||
|
||||
public async Task<AdapterHealth> CheckAsync(CancellationToken ct)
|
||||
{
|
||||
var stopwatch = Stopwatch.StartNew();
|
||||
try
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(_options.ApiKey))
|
||||
{
|
||||
throw new InvalidOperationException("Anthropic API key is not configured.");
|
||||
}
|
||||
|
||||
using var requestMessage = new HttpRequestMessage(HttpMethod.Get, $"{_options.BaseUrl}/v1/models");
|
||||
requestMessage.Headers.Add("x-api-key", _options.ApiKey);
|
||||
requestMessage.Headers.Add("anthropic-version", AnthropicVersion);
|
||||
|
||||
using var response = await _http.SendAsync(requestMessage, ct).ConfigureAwait(false);
|
||||
var responseBody = await response.Content.ReadAsStringAsync(ct).ConfigureAwait(false);
|
||||
|
||||
if (!response.IsSuccessStatusCode)
|
||||
{
|
||||
var truncated = responseBody.Length > 500 ? responseBody[..500] : responseBody;
|
||||
return new AdapterHealth(Name, false, $"HTTP {(int)response.StatusCode}: {truncated}", stopwatch.Elapsed, DateTimeOffset.UtcNow);
|
||||
}
|
||||
|
||||
using var document = JsonDocument.Parse(responseBody);
|
||||
var count = document.RootElement.GetProperty("data").GetArrayLength();
|
||||
return new AdapterHealth(Name, true, $"{count} models", stopwatch.Elapsed, DateTimeOffset.UtcNow);
|
||||
}
|
||||
catch (Exception ex) when (ex is HttpRequestException or TaskCanceledException or JsonException or InvalidOperationException)
|
||||
{
|
||||
return new AdapterHealth(Name, false, ex.Message, stopwatch.Elapsed, DateTimeOffset.UtcNow);
|
||||
}
|
||||
}
|
||||
|
||||
private JsonObject BuildRequestBody(string system, string user) => new()
|
||||
{
|
||||
["model"] = _options.Model,
|
||||
["max_tokens"] = _options.MaxTokens,
|
||||
["system"] = system,
|
||||
["messages"] = new JsonArray
|
||||
{
|
||||
new JsonObject
|
||||
{
|
||||
["role"] = "user",
|
||||
["content"] = user,
|
||||
},
|
||||
},
|
||||
["stream"] = true,
|
||||
};
|
||||
|
||||
private async Task<HttpResponseMessage> PostWithRetryAsync(JsonObject body, CancellationToken ct)
|
||||
{
|
||||
// A missing key is a runtime condition the status page shows, not a reason the host cannot start.
|
||||
if (string.IsNullOrWhiteSpace(_options.ApiKey))
|
||||
{
|
||||
throw new InvalidOperationException("Anthropic API key is not configured.");
|
||||
}
|
||||
|
||||
var json = body.ToJsonString(JsonOptions);
|
||||
var delay = TimeSpan.FromSeconds(1);
|
||||
|
||||
for (var attempt = 0; ; attempt++)
|
||||
{
|
||||
using var requestMessage = new HttpRequestMessage(HttpMethod.Post, $"{_options.BaseUrl}/v1/messages")
|
||||
{
|
||||
Content = new StringContent(json, Encoding.UTF8, "application/json"),
|
||||
};
|
||||
requestMessage.Headers.Add("x-api-key", _options.ApiKey);
|
||||
requestMessage.Headers.Add("anthropic-version", AnthropicVersion);
|
||||
|
||||
var response = await _http.SendAsync(requestMessage, HttpCompletionOption.ResponseHeadersRead, ct).ConfigureAwait(false);
|
||||
if (response.IsSuccessStatusCode)
|
||||
{
|
||||
return response;
|
||||
}
|
||||
|
||||
var statusCode = (int)response.StatusCode;
|
||||
if ((statusCode == 429 || statusCode == 529) && attempt < _options.MaxRetries)
|
||||
{
|
||||
var wait = RetryAfter(response) ?? delay;
|
||||
response.Dispose();
|
||||
await _delay(wait, ct).ConfigureAwait(false);
|
||||
delay = TimeSpan.FromSeconds(Math.Min(delay.TotalSeconds * 2, 30));
|
||||
continue;
|
||||
}
|
||||
|
||||
var errorBody = await response.Content.ReadAsStringAsync(ct).ConfigureAwait(false);
|
||||
response.Dispose();
|
||||
var truncated = errorBody.Length > 500 ? errorBody[..500] : errorBody;
|
||||
throw new HttpRequestException($"anthropic: HTTP {statusCode}: {truncated}");
|
||||
}
|
||||
}
|
||||
|
||||
private static TimeSpan? RetryAfter(HttpResponseMessage response)
|
||||
{
|
||||
var header = response.Headers.RetryAfter;
|
||||
if (header is null)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
if (header.Delta is { } delta)
|
||||
{
|
||||
return delta;
|
||||
}
|
||||
|
||||
if (header.Date is { } date)
|
||||
{
|
||||
var diff = date - DateTimeOffset.UtcNow;
|
||||
return diff > TimeSpan.Zero ? diff : TimeSpan.Zero;
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
public Task<AdapterHealth> CheckAsync(CancellationToken ct) => _transport.CheckAsync(Name, ct);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,55 @@
|
||||
using System.Text;
|
||||
using System.Text.Json.Nodes;
|
||||
using Wiki.Core.Model;
|
||||
using Wiki.Core.Ports;
|
||||
|
||||
namespace Wiki.Adapters.Anthropic;
|
||||
|
||||
/// <summary>Turns a statement into raw-source fields with one Messages API call. Same transport and options as the answer model.</summary>
|
||||
public sealed class AnthropicFactDrafter : IFactDrafter
|
||||
{
|
||||
private readonly MessagesTransport _transport;
|
||||
|
||||
public AnthropicFactDrafter(HttpClient http, AnthropicOptions options, Func<TimeSpan, CancellationToken, Task>? delay = null)
|
||||
{
|
||||
_transport = new MessagesTransport(http, options, delay);
|
||||
ModelId = options.ModelId;
|
||||
}
|
||||
|
||||
public string ModelId { get; }
|
||||
|
||||
public string Name => $"anthropic-facts:{ModelId}";
|
||||
|
||||
public async Task<FactDraft> DraftAsync(FactStatement statement, CancellationToken ct)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(statement);
|
||||
if (string.IsNullOrWhiteSpace(statement.Text))
|
||||
throw new ArgumentException("the statement is empty", nameof(statement));
|
||||
|
||||
var (system, user) = FactDraftPrompt.Build(statement);
|
||||
var content = new JsonArray();
|
||||
foreach (var a in statement.Attachments)
|
||||
{
|
||||
content.Add(new JsonObject
|
||||
{
|
||||
["type"] = "image",
|
||||
["source"] = new JsonObject
|
||||
{
|
||||
["type"] = "base64",
|
||||
["media_type"] = a.MediaType,
|
||||
["data"] = Convert.ToBase64String(a.Bytes),
|
||||
},
|
||||
});
|
||||
}
|
||||
content.Add(new JsonObject { ["type"] = "text", ["text"] = user });
|
||||
|
||||
var text = new StringBuilder();
|
||||
await foreach (var chunk in _transport.StreamAsync(system, content, ModelId, ct).ConfigureAwait(false))
|
||||
{
|
||||
if (chunk is AnswerChunk.Text t) text.Append(t.Delta);
|
||||
}
|
||||
return FactDraftPrompt.Parse(text.ToString());
|
||||
}
|
||||
|
||||
public Task<AdapterHealth> CheckAsync(CancellationToken ct) => _transport.CheckAsync(Name, ct);
|
||||
}
|
||||
@@ -0,0 +1,71 @@
|
||||
using System.Globalization;
|
||||
using System.Text;
|
||||
using System.Text.Json;
|
||||
using Wiki.Core.Model;
|
||||
|
||||
namespace Wiki.Adapters.Anthropic;
|
||||
|
||||
/// <summary>Builds the prompt that turns a statement into raw-source fields, and parses the model's JSON. Pure.</summary>
|
||||
public static class FactDraftPrompt
|
||||
{
|
||||
private const string SystemPrompt =
|
||||
"You turn one statement a person makes about their private homelab into the fields of a raw fact record for a wiki. " +
|
||||
"Reply with a single JSON object and nothing else: {\"title\": string, \"content\": string, \"tags\": [string]}. " +
|
||||
"title: a short noun phrase in the language of the statement, no trailing period. " +
|
||||
"content: the statement restated faithfully as complete sentences in the language of the statement; keep every detail and every exception, add nothing that was not said, at least 40 characters. " +
|
||||
"tags: one to five lowercase ASCII slugs (a-z, 0-9, hyphen) naming the topics, English. " +
|
||||
"Images attached to the statement are part of it: transcribe every identifier printed on them exactly as printed (manufacturer, model, type, serial number, ratings, dates), mark characters you cannot read as [unreadable], never guess. " +
|
||||
"No markdown, no code fences, no commentary.";
|
||||
|
||||
public static (string System, string User) Build(FactStatement statement)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(statement);
|
||||
var user = new StringBuilder();
|
||||
user.Append("Stated by ").Append(statement.By.Name)
|
||||
.Append(" on ").Append(statement.StatedAt.ToString("yyyy-MM-dd", CultureInfo.InvariantCulture)).Append(":\n\n");
|
||||
user.Append(statement.Text);
|
||||
if (statement.Attachments.Count > 0)
|
||||
user.Append("\n\n(").Append(statement.Attachments.Count.ToString(CultureInfo.InvariantCulture)).Append(" image(s) attached.)");
|
||||
return (SystemPrompt, user.ToString());
|
||||
}
|
||||
|
||||
/// <summary>The draft in the model's reply. Tolerates text around the JSON object; rejects anything without title and content.</summary>
|
||||
public static FactDraft Parse(string text)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(text);
|
||||
var start = text.IndexOf('{');
|
||||
var end = text.LastIndexOf('}');
|
||||
if (start < 0 || end <= start)
|
||||
throw new InvalidOperationException("the drafter returned no JSON object");
|
||||
|
||||
try
|
||||
{
|
||||
using var document = JsonDocument.Parse(text[start..(end + 1)]);
|
||||
var root = document.RootElement;
|
||||
var title = root.TryGetProperty("title", out var t) && t.ValueKind == JsonValueKind.String ? t.GetString()?.Trim() : null;
|
||||
var content = root.TryGetProperty("content", out var c) && c.ValueKind == JsonValueKind.String ? c.GetString()?.Trim() : null;
|
||||
var tags = root.TryGetProperty("tags", out var g) && g.ValueKind == JsonValueKind.Array
|
||||
? g.EnumerateArray().Where(x => x.ValueKind == JsonValueKind.String).Select(x => Slug(x.GetString() ?? "")).Where(s => s.Length > 0).Distinct().Take(5).ToList()
|
||||
: [];
|
||||
if (string.IsNullOrWhiteSpace(title) || string.IsNullOrWhiteSpace(content))
|
||||
throw new InvalidOperationException("the drafter returned no title or content");
|
||||
return new FactDraft(title, content, tags);
|
||||
}
|
||||
catch (JsonException e)
|
||||
{
|
||||
throw new InvalidOperationException($"the drafter returned malformed JSON: {e.Message}", e);
|
||||
}
|
||||
}
|
||||
|
||||
private static string Slug(string tag)
|
||||
{
|
||||
var sb = new StringBuilder(tag.Length);
|
||||
var dash = false;
|
||||
foreach (var ch in tag.ToLowerInvariant())
|
||||
{
|
||||
if (ch is (>= 'a' and <= 'z') or (>= '0' and <= '9')) { sb.Append(ch); dash = false; }
|
||||
else if (!dash) { sb.Append('-'); dash = true; }
|
||||
}
|
||||
return sb.ToString().Trim('-');
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,221 @@
|
||||
using System.Diagnostics;
|
||||
using System.Runtime.CompilerServices;
|
||||
using System.Text;
|
||||
using System.Text.Json;
|
||||
using System.Text.Json.Nodes;
|
||||
using Wiki.Core.Model;
|
||||
|
||||
namespace Wiki.Adapters.Anthropic;
|
||||
|
||||
/// <summary>
|
||||
/// One streamed Messages API call, shared by every port the adapter serves. Retried on 429/529
|
||||
/// with the server's Retry-After (or exponential backoff), never retried for anything else.
|
||||
/// </summary>
|
||||
internal sealed class MessagesTransport
|
||||
{
|
||||
private const string AnthropicVersion = "2023-06-01";
|
||||
|
||||
private static readonly JsonSerializerOptions JsonOptions = new(JsonSerializerDefaults.Web);
|
||||
|
||||
private readonly HttpClient _http;
|
||||
private readonly AnthropicOptions _options;
|
||||
private readonly Func<TimeSpan, CancellationToken, Task> _delay;
|
||||
|
||||
public MessagesTransport(HttpClient http, AnthropicOptions options, Func<TimeSpan, CancellationToken, Task>? delay)
|
||||
{
|
||||
_http = http;
|
||||
_options = options;
|
||||
_delay = delay ?? Task.Delay;
|
||||
}
|
||||
|
||||
/// <summary>One user turn of text.</summary>
|
||||
public IAsyncEnumerable<AnswerChunk> StreamAsync(string system, string user, string modelId, CancellationToken ct) =>
|
||||
StreamAsync(system, JsonValue.Create(user), modelId, ct);
|
||||
|
||||
/// <summary>One user turn of content blocks (text, images).</summary>
|
||||
public async IAsyncEnumerable<AnswerChunk> StreamAsync(string system, JsonNode userContent, string modelId, [EnumeratorCancellation] CancellationToken ct)
|
||||
{
|
||||
ct.ThrowIfCancellationRequested();
|
||||
|
||||
var body = BuildRequestBody(system, userContent);
|
||||
var stopwatch = Stopwatch.StartNew();
|
||||
using var response = await PostWithRetryAsync(body, ct).ConfigureAwait(false);
|
||||
var stream = await response.Content.ReadAsStreamAsync(ct).ConfigureAwait(false);
|
||||
await using (stream.ConfigureAwait(false))
|
||||
using (var reader = new StreamReader(stream, Encoding.UTF8))
|
||||
{
|
||||
long inputTokens = 0;
|
||||
long outputTokens = 0;
|
||||
string? eventName = null;
|
||||
|
||||
string? line;
|
||||
while ((line = await reader.ReadLineAsync(ct).ConfigureAwait(false)) is not null)
|
||||
{
|
||||
if (line.Length == 0)
|
||||
{
|
||||
eventName = null;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (line.StartsWith("event: ", StringComparison.Ordinal))
|
||||
{
|
||||
eventName = line["event: ".Length..];
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!line.StartsWith("data: ", StringComparison.Ordinal))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
var data = line["data: ".Length..];
|
||||
using var payload = JsonDocument.Parse(data);
|
||||
var root = payload.RootElement;
|
||||
|
||||
switch (eventName)
|
||||
{
|
||||
case "message_start":
|
||||
inputTokens = root.GetProperty("message").GetProperty("usage").GetProperty("input_tokens").GetInt64();
|
||||
break;
|
||||
|
||||
case "content_block_delta":
|
||||
var delta = root.GetProperty("delta");
|
||||
if (delta.GetProperty("type").GetString() == "text_delta")
|
||||
{
|
||||
yield return new AnswerChunk.Text(delta.GetProperty("text").GetString() ?? "");
|
||||
}
|
||||
|
||||
break;
|
||||
|
||||
case "message_delta":
|
||||
outputTokens = root.GetProperty("usage").GetProperty("output_tokens").GetInt64();
|
||||
break;
|
||||
|
||||
case "error":
|
||||
throw new HttpRequestException($"anthropic: {root.GetProperty("error").GetProperty("message").GetString()}");
|
||||
|
||||
// message_stop, ping, content_block_start, content_block_stop: nothing to record.
|
||||
default:
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
stopwatch.Stop();
|
||||
yield return new AnswerChunk.Done(modelId, inputTokens, outputTokens, stopwatch.Elapsed);
|
||||
}
|
||||
}
|
||||
|
||||
public async Task<AdapterHealth> CheckAsync(string name, CancellationToken ct)
|
||||
{
|
||||
var stopwatch = Stopwatch.StartNew();
|
||||
try
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(_options.ApiKey))
|
||||
{
|
||||
throw new InvalidOperationException("Anthropic API key is not configured.");
|
||||
}
|
||||
|
||||
using var requestMessage = new HttpRequestMessage(HttpMethod.Get, $"{_options.BaseUrl}/v1/models");
|
||||
requestMessage.Headers.Add("x-api-key", _options.ApiKey);
|
||||
requestMessage.Headers.Add("anthropic-version", AnthropicVersion);
|
||||
|
||||
using var response = await _http.SendAsync(requestMessage, ct).ConfigureAwait(false);
|
||||
var responseBody = await response.Content.ReadAsStringAsync(ct).ConfigureAwait(false);
|
||||
|
||||
if (!response.IsSuccessStatusCode)
|
||||
{
|
||||
var truncated = responseBody.Length > 500 ? responseBody[..500] : responseBody;
|
||||
return new AdapterHealth(name, false, $"HTTP {(int)response.StatusCode}: {truncated}", stopwatch.Elapsed, DateTimeOffset.UtcNow);
|
||||
}
|
||||
|
||||
using var document = JsonDocument.Parse(responseBody);
|
||||
var count = document.RootElement.GetProperty("data").GetArrayLength();
|
||||
return new AdapterHealth(name, true, $"{count} models", stopwatch.Elapsed, DateTimeOffset.UtcNow);
|
||||
}
|
||||
catch (Exception ex) when (ex is HttpRequestException or TaskCanceledException or JsonException or InvalidOperationException)
|
||||
{
|
||||
return new AdapterHealth(name, false, ex.Message, stopwatch.Elapsed, DateTimeOffset.UtcNow);
|
||||
}
|
||||
}
|
||||
|
||||
private JsonObject BuildRequestBody(string system, JsonNode userContent) => new()
|
||||
{
|
||||
["model"] = _options.Model,
|
||||
["max_tokens"] = _options.MaxTokens,
|
||||
["system"] = system,
|
||||
["messages"] = new JsonArray
|
||||
{
|
||||
new JsonObject
|
||||
{
|
||||
["role"] = "user",
|
||||
["content"] = userContent,
|
||||
},
|
||||
},
|
||||
["stream"] = true,
|
||||
};
|
||||
|
||||
private async Task<HttpResponseMessage> PostWithRetryAsync(JsonObject body, CancellationToken ct)
|
||||
{
|
||||
// A missing key is a runtime condition the status page shows, not a reason the host cannot start.
|
||||
if (string.IsNullOrWhiteSpace(_options.ApiKey))
|
||||
{
|
||||
throw new InvalidOperationException("Anthropic API key is not configured.");
|
||||
}
|
||||
|
||||
var json = body.ToJsonString(JsonOptions);
|
||||
var delay = TimeSpan.FromSeconds(1);
|
||||
|
||||
for (var attempt = 0; ; attempt++)
|
||||
{
|
||||
using var requestMessage = new HttpRequestMessage(HttpMethod.Post, $"{_options.BaseUrl}/v1/messages")
|
||||
{
|
||||
Content = new StringContent(json, Encoding.UTF8, "application/json"),
|
||||
};
|
||||
requestMessage.Headers.Add("x-api-key", _options.ApiKey);
|
||||
requestMessage.Headers.Add("anthropic-version", AnthropicVersion);
|
||||
|
||||
var response = await _http.SendAsync(requestMessage, HttpCompletionOption.ResponseHeadersRead, ct).ConfigureAwait(false);
|
||||
if (response.IsSuccessStatusCode)
|
||||
{
|
||||
return response;
|
||||
}
|
||||
|
||||
var statusCode = (int)response.StatusCode;
|
||||
if ((statusCode == 429 || statusCode == 529) && attempt < _options.MaxRetries)
|
||||
{
|
||||
var wait = RetryAfter(response) ?? delay;
|
||||
response.Dispose();
|
||||
await _delay(wait, ct).ConfigureAwait(false);
|
||||
delay = TimeSpan.FromSeconds(Math.Min(delay.TotalSeconds * 2, 30));
|
||||
continue;
|
||||
}
|
||||
|
||||
var errorBody = await response.Content.ReadAsStringAsync(ct).ConfigureAwait(false);
|
||||
response.Dispose();
|
||||
var truncated = errorBody.Length > 500 ? errorBody[..500] : errorBody;
|
||||
throw new HttpRequestException($"anthropic: HTTP {statusCode}: {truncated}");
|
||||
}
|
||||
}
|
||||
|
||||
private static TimeSpan? RetryAfter(HttpResponseMessage response)
|
||||
{
|
||||
var header = response.Headers.RetryAfter;
|
||||
if (header is null)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
if (header.Delta is { } delta)
|
||||
{
|
||||
return delta;
|
||||
}
|
||||
|
||||
if (header.Date is { } date)
|
||||
{
|
||||
var diff = date - DateTimeOffset.UtcNow;
|
||||
return diff > TimeSpan.Zero ? diff : TimeSpan.Zero;
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
}
|
||||
@@ -39,4 +39,38 @@ public static class ServiceCollectionExtensions
|
||||
|
||||
return services;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Registers an <see cref="IFactDrafter"/> backed by Anthropic's Messages API, keyed by
|
||||
/// <paramref name="name"/>. Usually registered alongside the answer model of the same name.
|
||||
/// </summary>
|
||||
public static IServiceCollection AddAnthropicFactDrafter(
|
||||
this IServiceCollection services,
|
||||
string name,
|
||||
Action<AnthropicOptions> configure)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(services);
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(name);
|
||||
ArgumentNullException.ThrowIfNull(configure);
|
||||
|
||||
var options = new AnthropicOptions();
|
||||
configure(options);
|
||||
options.Validate();
|
||||
|
||||
var httpClientName = $"anthropic-facts:{name}";
|
||||
|
||||
services.AddHttpClient(httpClientName, client =>
|
||||
{
|
||||
client.Timeout = options.Timeout;
|
||||
});
|
||||
|
||||
services.AddKeyedSingleton<IFactDrafter>(name, (sp, _) =>
|
||||
{
|
||||
var httpClientFactory = sp.GetRequiredService<IHttpClientFactory>();
|
||||
var client = httpClientFactory.CreateClient(httpClientName);
|
||||
return new AnthropicFactDrafter(client, options);
|
||||
});
|
||||
|
||||
return services;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -79,13 +79,19 @@ public sealed class GiteaRepository : INoteStore
|
||||
}
|
||||
|
||||
/// <summary>POST contents: Gitea answers 422 when the path already exists on the branch.</summary>
|
||||
public async Task<StoreWrite> CreateAsync(string path, string text, string message, CancellationToken ct)
|
||||
public Task<StoreWrite> CreateAsync(string path, string text, string message, CancellationToken ct) =>
|
||||
CreateAsync(path, Encoding.UTF8.GetBytes(text), message, ct);
|
||||
|
||||
public Uri? BrowseUrl(string path) =>
|
||||
new($"{_options.BaseUrl.TrimEnd('/')}/{_options.Repository}/src/branch/{_options.Branch}/{path}");
|
||||
|
||||
public async Task<StoreWrite> CreateAsync(string path, byte[] content, string message, CancellationToken ct)
|
||||
{
|
||||
var repo = GiteaUrls.EncodeSegments(_options.Repository);
|
||||
var encodedPath = GiteaUrls.EncodeSegments(path);
|
||||
var request = new
|
||||
{
|
||||
content = Convert.ToBase64String(Encoding.UTF8.GetBytes(text)),
|
||||
content = Convert.ToBase64String(content),
|
||||
message,
|
||||
branch = _options.Branch,
|
||||
};
|
||||
|
||||
@@ -20,7 +20,7 @@ public enum ContextScope
|
||||
Documents,
|
||||
}
|
||||
|
||||
public sealed record AskOptions(int K = 6, int DocChars = 12_000, int StateChars = 60_000, int MaxParallel = 8);
|
||||
public sealed record AskOptions(int K = 6, int DocChars = 12_000, int StateChars = 60_000);
|
||||
|
||||
public sealed record AskRequest(IReadOnlyList<string> Queries, IReadOnlyDictionary<string, AskQuestion>? Questions, AskOptions Options);
|
||||
|
||||
@@ -52,7 +52,7 @@ public sealed record AskResult(
|
||||
/// <summary>
|
||||
/// wiki_ask.py as a use case: rank every query, unite the top notes into one context (per-document
|
||||
/// and total budget), expand context choices over the paragraph or document ids plus NONE, add the
|
||||
/// covered gate, judge (every context choice in its own request, the rest batched), and attach the
|
||||
/// covered gate, judge every question in one request, and attach the
|
||||
/// answering paragraphs' text to the results. Judgements and quotations, never prose.
|
||||
/// </summary>
|
||||
public sealed class AskUseCase(RankUseCase rank, IDocumentCatalog catalog, IChunker chunker)
|
||||
@@ -69,7 +69,7 @@ public sealed class AskUseCase(RankUseCase rank, IDocumentCatalog catalog, IChun
|
||||
var questions = request.Questions ?? DefaultQuestions(request.Queries);
|
||||
var (expanded, scopes) = Expand(questions, context, request.Queries, judge.MaxChoiceOptions);
|
||||
|
||||
var (answers, usage) = await JudgeInParallelAsync(judge, context, expanded, scopes, request.Options.MaxParallel, ct);
|
||||
var (answers, usage) = await JudgeAsync(judge, context, expanded, ct);
|
||||
var resolved = Resolve(answers, scopes, context);
|
||||
double? covered = answers.TryGetValue(CoveredId, out var c) && c is Judgement.Noul n ? Math.Round(n.Probability, 3) : null;
|
||||
return new AskResult(request.Queries, context, meta, resolved, covered, usage, judge.ModelId, Stopwatch.GetElapsedTime(started));
|
||||
@@ -183,9 +183,12 @@ public sealed class AskUseCase(RankUseCase rank, IDocumentCatalog catalog, IChun
|
||||
return (expanded, scopes);
|
||||
}
|
||||
|
||||
private static async Task<(IReadOnlyDictionary<string, Judgement> Answers, JudgeUsage Usage)> JudgeInParallelAsync(
|
||||
IJudge judge, IReadOnlyList<ContextDocument> context, IReadOnlyDictionary<string, TypedQuestion> questions,
|
||||
IReadOnlyDictionary<string, ContextScope> scopes, int maxParallel, CancellationToken ct)
|
||||
/// <summary>
|
||||
/// One request for every question: the state dominates the request, and System One scores each
|
||||
/// question in isolation against it, so splitting only re-sends the state and adds round trips.
|
||||
/// </summary>
|
||||
private static async Task<(IReadOnlyDictionary<string, Judgement> Answers, JudgeUsage Usage)> JudgeAsync(
|
||||
IJudge judge, IReadOnlyList<ContextDocument> context, IReadOnlyDictionary<string, TypedQuestion> questions, CancellationToken ct)
|
||||
{
|
||||
var state = new
|
||||
{
|
||||
@@ -196,29 +199,8 @@ public sealed class AskUseCase(RankUseCase rank, IDocumentCatalog catalog, IChun
|
||||
paragraphs = d.Paragraphs.Select(p => new { id = p.Id, heading = p.Heading, text = p.Text }),
|
||||
}),
|
||||
};
|
||||
var groups = new List<Dictionary<string, TypedQuestion>>();
|
||||
foreach (var (id, q) in questions.Where(kv => scopes.ContainsKey(kv.Key)))
|
||||
groups.Add(new Dictionary<string, TypedQuestion> { [id] = q });
|
||||
var rest = questions.Where(kv => !scopes.ContainsKey(kv.Key)).ToDictionary(kv => kv.Key, kv => kv.Value);
|
||||
if (rest.Count > 0) groups.Add(rest);
|
||||
|
||||
using var gate = new SemaphoreSlim(Math.Max(1, maxParallel));
|
||||
var tasks = groups.Select(async g =>
|
||||
{
|
||||
await gate.WaitAsync(ct);
|
||||
try { return await judge.JudgeAsync(new JudgeRequest(state, g), ct); }
|
||||
finally { gate.Release(); }
|
||||
}).ToList();
|
||||
var results = await Task.WhenAll(tasks);
|
||||
|
||||
var answers = new Dictionary<string, Judgement>();
|
||||
var usage = JudgeUsage.None;
|
||||
foreach (var r in results)
|
||||
{
|
||||
foreach (var (id, a) in r.Answers) answers[id] = a;
|
||||
usage = usage.Add(r.Usage);
|
||||
}
|
||||
return (answers, usage);
|
||||
var result = await judge.JudgeAsync(new JudgeRequest(state, questions), ct);
|
||||
return (result.Answers, result.Usage);
|
||||
}
|
||||
|
||||
private static IReadOnlyDictionary<string, AskAnswer> Resolve(
|
||||
|
||||
@@ -24,28 +24,42 @@ public sealed class DepositFactUseCase(INoteStore store, TimeProvider clock)
|
||||
var now = clock.GetUtcNow();
|
||||
var datePrefix = now.UtcDateTime.ToString("yyyy-MM-dd", CultureInfo.InvariantCulture);
|
||||
var slug = Slugify(fact.Title);
|
||||
var text = BuildNote(fact, by, datePrefix);
|
||||
var message = $"raw: {fact.Title}";
|
||||
|
||||
var path = $"raw/{datePrefix}-{slug}.md";
|
||||
for (var attempt = 2; ; attempt++)
|
||||
{
|
||||
// Attachments take the note's stem, so they sort next to it and survive a suffix.
|
||||
var stem = path[(path.LastIndexOf('/') + 1)..^3];
|
||||
var names = fact.Attachments.Select((a, i) => $"{stem}-{i + 1}.{a.Extension}").ToList();
|
||||
var text = BuildNote(fact, by, datePrefix, names);
|
||||
StoreWrite written;
|
||||
try
|
||||
{
|
||||
var written = await store.CreateAsync(path, text, message, ct);
|
||||
return new FactReceipt(written.Path, written.Commit, now);
|
||||
written = await store.CreateAsync(path, text, message, ct);
|
||||
}
|
||||
catch (DocumentExistsException) when (attempt <= MaxSuffixAttempts)
|
||||
{
|
||||
path = $"raw/{datePrefix}-{slug}-{attempt}.md";
|
||||
continue;
|
||||
}
|
||||
for (var i = 0; i < names.Count; i++)
|
||||
written = await store.CreateAsync($"raw/{names[i]}", fact.Attachments[i].Bytes, $"{message} (attachment {i + 1})", ct);
|
||||
return new FactReceipt(path, written.Commit, now);
|
||||
}
|
||||
}
|
||||
|
||||
public static string BuildNote(RawSource fact, Identity by, string capturedDate)
|
||||
public static string BuildNote(RawSource fact, Identity by, string capturedDate, IReadOnlyList<string>? attachments = null)
|
||||
{
|
||||
attachments ??= [];
|
||||
var body = fact.Content;
|
||||
if (!body.EndsWith('\n')) body += "\n";
|
||||
if (attachments.Count > 0)
|
||||
{
|
||||
body += "\n";
|
||||
for (var i = 0; i < attachments.Count; i++)
|
||||
body += $"![{fact.Attachments[i].Caption ?? $"attachment {i + 1}"}]({attachments[i]})\n";
|
||||
}
|
||||
|
||||
var sb = new StringBuilder();
|
||||
sb.Append("---\n");
|
||||
@@ -56,6 +70,7 @@ public sealed class DepositFactUseCase(INoteStore store, TimeProvider clock)
|
||||
sb.Append("captured_by: ").Append(by.Name).Append('\n');
|
||||
sb.Append("observed: ").Append(fact.ObservedAt.ToString("yyyy-MM-ddTHH:mm:ssK", CultureInfo.InvariantCulture)).Append('\n');
|
||||
sb.Append("tags: [").Append(string.Join(", ", fact.Tags)).Append("]\n");
|
||||
if (attachments.Count > 0) sb.Append("attachments: [").Append(string.Join(", ", attachments)).Append("]\n");
|
||||
sb.Append("---\n");
|
||||
sb.Append('\n');
|
||||
sb.Append(body);
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
using System.Globalization;
|
||||
using Wiki.Core.Model;
|
||||
using Wiki.Core.Ports;
|
||||
|
||||
namespace Wiki.Application.Facts;
|
||||
|
||||
public sealed record RecordedStatement(FactDraft Draft, RawSource Fact, FactReceipt Receipt);
|
||||
|
||||
/// <summary>
|
||||
/// A person states something; a drafter normalises it into title, content and tags; the raw
|
||||
/// source that lands carries the drafter's restatement and the statement verbatim, with the
|
||||
/// person as source. Provenance is the person's word, and the note says so.
|
||||
/// </summary>
|
||||
public sealed class RecordStatementUseCase(DepositFactUseCase deposit, TimeProvider clock)
|
||||
{
|
||||
public async Task<RecordedStatement> RecordAsync(string statement, IReadOnlyList<Attachment> attachments, Identity by, IFactDrafter drafter, CancellationToken ct)
|
||||
{
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(statement);
|
||||
ArgumentNullException.ThrowIfNull(attachments);
|
||||
ArgumentNullException.ThrowIfNull(by);
|
||||
ArgumentNullException.ThrowIfNull(drafter);
|
||||
|
||||
var now = clock.GetUtcNow();
|
||||
var text = statement.Trim();
|
||||
var draft = await drafter.DraftAsync(new FactStatement(text, by, now) { Attachments = attachments }, ct);
|
||||
var fact = new RawSource(
|
||||
draft.Title,
|
||||
$"stated by {by.Name} on {now.ToString("yyyy-MM-dd", CultureInfo.InvariantCulture)} via the wiki",
|
||||
Body(draft.Content, text),
|
||||
now,
|
||||
draft.Tags) { Attachments = attachments };
|
||||
var errors = fact.Validate();
|
||||
if (errors.Count > 0)
|
||||
throw new InvalidOperationException($"the draft is not a valid raw source: {string.Join("; ", errors)}");
|
||||
|
||||
var receipt = await deposit.DepositAsync(fact, by, ct);
|
||||
return new RecordedStatement(draft, fact, receipt);
|
||||
}
|
||||
|
||||
public static string Body(string content, string statement) =>
|
||||
content.TrimEnd() + "\n\nStatement, verbatim:\n\n> " + statement.Trim().Replace("\n", "\n> ", StringComparison.Ordinal) + "\n";
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
namespace Wiki.Core.Model;
|
||||
|
||||
/// <summary>A binary that belongs to a raw source and is kept next to it as evidence, a photo of a type label for instance.</summary>
|
||||
public sealed record Attachment(string MediaType, byte[] Bytes, string? Caption = null)
|
||||
{
|
||||
public const int MaxBytes = 8 * 1024 * 1024;
|
||||
|
||||
private static readonly Dictionary<string, string> Extensions = new(StringComparer.OrdinalIgnoreCase)
|
||||
{
|
||||
["image/jpeg"] = "jpg",
|
||||
["image/png"] = "png",
|
||||
["image/webp"] = "webp",
|
||||
["image/gif"] = "gif",
|
||||
};
|
||||
|
||||
/// <summary>The file extension for a supported media type, or null.</summary>
|
||||
public static string? ExtensionFor(string mediaType) => Extensions.TryGetValue(mediaType, out var e) ? e : null;
|
||||
|
||||
public string? Extension => ExtensionFor(MediaType);
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
namespace Wiki.Core.Model;
|
||||
|
||||
/// <summary>What a person said, by whom and when. The drafter sees nothing else.</summary>
|
||||
public sealed record FactStatement(string Text, Identity By, DateTimeOffset StatedAt)
|
||||
{
|
||||
/// <summary>Pictures that are part of the statement, a type label for instance.</summary>
|
||||
public IReadOnlyList<Attachment> Attachments { get; init; } = [];
|
||||
}
|
||||
|
||||
/// <summary>A statement normalised into the fields of a raw source. Source and provenance are the application's to add.</summary>
|
||||
public sealed record FactDraft(string Title, string Content, IReadOnlyList<string> Tags);
|
||||
@@ -13,6 +13,9 @@ public sealed record RawSource(
|
||||
{
|
||||
public const int MinimumContentLength = 40;
|
||||
|
||||
/// <summary>Binaries stored next to the note as evidence. Empty for most facts.</summary>
|
||||
public IReadOnlyList<Attachment> Attachments { get; init; } = [];
|
||||
|
||||
/// <summary>Validation errors, empty when the fact may be stored.</summary>
|
||||
public IReadOnlyList<string> Validate()
|
||||
{
|
||||
@@ -21,6 +24,13 @@ public sealed record RawSource(
|
||||
if (string.IsNullOrWhiteSpace(Source)) errors.Add("source is required: where the fact was observed or quoted from");
|
||||
if (Content is null || Content.Trim().Length < MinimumContentLength)
|
||||
errors.Add($"content must be at least {MinimumContentLength} characters of observation");
|
||||
for (var i = 0; i < Attachments.Count; i++)
|
||||
{
|
||||
var a = Attachments[i];
|
||||
if (a.Extension is null) errors.Add($"attachment {i + 1}: unsupported media type {a.MediaType}");
|
||||
if (a.Bytes.Length == 0) errors.Add($"attachment {i + 1}: empty");
|
||||
if (a.Bytes.Length > Attachment.MaxBytes) errors.Add($"attachment {i + 1}: larger than {Attachment.MaxBytes / (1024 * 1024)} MiB");
|
||||
}
|
||||
return errors;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -26,6 +26,12 @@ public interface INoteStore : IAdapterHealth
|
||||
|
||||
/// <summary>Creates a new document at the path on the current head. Throws <see cref="DocumentExistsException"/> when the path is taken.</summary>
|
||||
Task<StoreWrite> CreateAsync(string path, string text, string message, CancellationToken ct);
|
||||
|
||||
/// <summary>Creates a new binary (an attachment) at the path. Same rules as the text overload.</summary>
|
||||
Task<StoreWrite> CreateAsync(string path, byte[] content, string message, CancellationToken ct);
|
||||
|
||||
/// <summary>Where a person can look at the path in the store's own UI, or null when the store has none.</summary>
|
||||
Uri? BrowseUrl(string path);
|
||||
}
|
||||
|
||||
/// <summary>Text to vector. One instance per configured model.</summary>
|
||||
@@ -88,3 +94,10 @@ public interface IAnswerModel : IAdapterHealth
|
||||
string ModelId { get; }
|
||||
IAsyncEnumerable<AnswerChunk> AnswerAsync(AnswerRequest request, CancellationToken ct);
|
||||
}
|
||||
|
||||
/// <summary>A person's statement into the fields of a raw source. Sees only the statement.</summary>
|
||||
public interface IFactDrafter : IAdapterHealth
|
||||
{
|
||||
string ModelId { get; }
|
||||
Task<FactDraft> DraftAsync(FactStatement statement, CancellationToken ct);
|
||||
}
|
||||
|
||||
@@ -16,7 +16,7 @@ public static class ApiV1
|
||||
v1.MapQuestionEndpoints();
|
||||
|
||||
v1.MapGet("/status", async (StatusQuery status, CancellationToken ct) =>
|
||||
Results.Ok(await status.GetAsync(includeAdapterChecks: true, ct))).AllowAnonymous();
|
||||
Results.Ok(await status.GetAsync(includeAdapterChecks: true, ct))).RequireAuthorization(Auth.Policies.Read);
|
||||
|
||||
v1.MapPost("/sync", async (SyncCoordinator sync, bool? force, CancellationToken ct) =>
|
||||
{
|
||||
|
||||
@@ -79,7 +79,7 @@ public static class AskEndpoints
|
||||
{
|
||||
return Results.Json(new { error = e.Message }, statusCode: StatusCodes.Status502BadGateway);
|
||||
}
|
||||
}).AllowAnonymous();
|
||||
}).RequireAuthorization(Auth.Policies.Read);
|
||||
}
|
||||
|
||||
public static AskQuestion Parse(string id, AskQuestionDto dto)
|
||||
|
||||
@@ -20,8 +20,8 @@ public static class IndexEndpoints
|
||||
{
|
||||
public static void MapIndexEndpoints(this WebApplication app)
|
||||
{
|
||||
app.MapPost("/rank", RankAsync).AllowAnonymous();
|
||||
app.MapPost("/api/v1/rank", RankAsync).AllowAnonymous();
|
||||
app.MapPost("/rank", RankAsync).RequireAuthorization(Auth.Policies.Read);
|
||||
app.MapPost("/api/v1/rank", RankAsync).RequireAuthorization(Auth.Policies.Read);
|
||||
|
||||
app.MapPost("/webhook", async (HttpRequest request, IConfiguration configuration, WikiOptions options, SyncBackgroundService sync, CancellationToken ct) =>
|
||||
{
|
||||
|
||||
@@ -1,9 +1,7 @@
|
||||
using System.Text;
|
||||
using System.Text.Json.Serialization;
|
||||
using Microsoft.AspNetCore.Authentication;
|
||||
using Wiki.Application.Questions;
|
||||
using Wiki.Core.Ports;
|
||||
using Wiki.Host.Auth;
|
||||
|
||||
namespace Wiki.Host.Api;
|
||||
|
||||
@@ -25,7 +23,7 @@ public static class QuestionEndpoints
|
||||
{
|
||||
v1.MapPost("/questions", async (
|
||||
QuestionRequestDto body, QuestionUseCase questions, SaveAnswerUseCase save, IServiceProvider services,
|
||||
IEmbeddingModel activeModel, IJudge activeJudge, IAnswerModel activeAnswer, HttpContext http, CancellationToken ct) =>
|
||||
IEmbeddingModel activeModel, IJudge activeJudge, IAnswerModel activeAnswer, CancellationToken ct) =>
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(body.Question))
|
||||
return Results.BadRequest(new { error = "question required" });
|
||||
@@ -35,8 +33,6 @@ public static class QuestionEndpoints
|
||||
if (err is not null) return Results.BadRequest(new { error = err });
|
||||
var answerModel = Pick(services, activeAnswer, body.AnswerModel, out err) ?? activeAnswer;
|
||||
if (err is not null) return Results.BadRequest(new { error = err });
|
||||
if (body.Save == true && !await IsAuthenticatedAsync(http))
|
||||
return Results.Json(new { error = "saving needs a signed-in user or an API key" }, statusCode: StatusCodes.Status401Unauthorized);
|
||||
|
||||
var text = new StringBuilder();
|
||||
QuestionEvent.Evidence? evidence = null;
|
||||
@@ -90,18 +86,9 @@ public static class QuestionEndpoints
|
||||
askUsage = ask?.Usage,
|
||||
saved,
|
||||
});
|
||||
}).AllowAnonymous();
|
||||
}).RequireAuthorization(Auth.Policies.Read);
|
||||
}
|
||||
|
||||
/// <summary>Humans arrive authenticated by the default (proxy header) scheme; machines present an API key.</summary>
|
||||
private static async Task<bool> IsAuthenticatedAsync(HttpContext http)
|
||||
{
|
||||
if (http.User.Identity?.IsAuthenticated == true) return true;
|
||||
var result = await http.AuthenticateAsync(ApiKeyAuthenticationHandler.SchemeName);
|
||||
if (!result.Succeeded) return false;
|
||||
http.User = result.Principal;
|
||||
return true;
|
||||
}
|
||||
|
||||
private static T? Pick<T>(IServiceProvider services, T active, string? name, out string? error) where T : class
|
||||
{
|
||||
|
||||
@@ -23,7 +23,7 @@ public static class ReadEndpoints
|
||||
return Results.Ok(await queries.SearchAsync(m, q, take, ct));
|
||||
}
|
||||
return Results.Ok(await queries.ListAsync(tag, take, ct));
|
||||
}).AllowAnonymous();
|
||||
}).RequireAuthorization(Auth.Policies.Read);
|
||||
|
||||
v1.MapGet("/notes/{slug}", async (NoteQueries queries, string slug, CancellationToken ct) =>
|
||||
{
|
||||
@@ -40,28 +40,28 @@ public static class ReadEndpoints
|
||||
sources = view.Sources.Select(s => new { s.Text, kind = s.Kind?.ToString(), s.Slug }),
|
||||
unresolved = view.UnresolvedLinks,
|
||||
});
|
||||
}).AllowAnonymous();
|
||||
}).RequireAuthorization(Auth.Policies.Read);
|
||||
|
||||
v1.MapGet("/tags", async (NoteQueries queries, CancellationToken ct) =>
|
||||
Results.Ok((await queries.TagsAsync(ct)).OrderByDescending(kv => kv.Value).ThenBy(kv => kv.Key).Select(kv => new { tag = kv.Key, count = kv.Value })))
|
||||
.AllowAnonymous();
|
||||
.RequireAuthorization(Auth.Policies.Read);
|
||||
|
||||
v1.MapGet("/raw", async (NoteQueries queries, CancellationToken ct) =>
|
||||
Results.Ok(await queries.RawQueueAsync(ct))).AllowAnonymous();
|
||||
Results.Ok(await queries.RawQueueAsync(ct))).RequireAuthorization(Auth.Policies.Read);
|
||||
|
||||
v1.MapGet("/raw/{slug}", async (NoteQueries queries, string slug, CancellationToken ct) =>
|
||||
{
|
||||
var doc = await queries.GetDocumentAsync(DocumentKind.Raw, slug, ct) ?? await queries.GetDocumentAsync(DocumentKind.RawProcessed, slug, ct);
|
||||
return doc is null ? Results.NotFound(new { error = "no such raw source" }) : Results.Ok(new { doc.Slug, doc.Title, doc.Kind, doc.Path, doc.Body });
|
||||
}).AllowAnonymous();
|
||||
}).RequireAuthorization(Auth.Policies.Read);
|
||||
|
||||
v1.MapGet("/reports", async (NoteQueries queries, string? type, int? limit, CancellationToken ct) =>
|
||||
Results.Ok(await queries.ReportsAsync(type, limit ?? 50, ct))).AllowAnonymous();
|
||||
Results.Ok(await queries.ReportsAsync(type, limit ?? 50, ct))).RequireAuthorization(Auth.Policies.Read);
|
||||
|
||||
v1.MapGet("/reports/{slug}", async (NoteQueries queries, string slug, CancellationToken ct) =>
|
||||
{
|
||||
var doc = await queries.GetDocumentAsync(DocumentKind.Output, slug, ct);
|
||||
return doc is null ? Results.NotFound(new { error = "no such report" }) : Results.Ok(new { doc.Slug, doc.Title, doc.Type, doc.Path, doc.Sources, doc.Body });
|
||||
}).AllowAnonymous();
|
||||
}).RequireAuthorization(Auth.Policies.Read);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -50,5 +50,13 @@ public sealed class ApiKeyAuthenticationHandler(
|
||||
|
||||
public static class Policies
|
||||
{
|
||||
/// <summary>A signed-in human (OIDC cookie), any API key or valid BasicAuth credentials.</summary>
|
||||
public const string Read = "read";
|
||||
/// <summary>An API key only, so a write carries a named machine identity.</summary>
|
||||
public const string Write = "write";
|
||||
}
|
||||
|
||||
public static class Claims
|
||||
{
|
||||
public const string Kind = "wiki:kind";
|
||||
}
|
||||
|
||||
@@ -0,0 +1,144 @@
|
||||
using System.Collections.Concurrent;
|
||||
using System.Security.Claims;
|
||||
using System.Security.Cryptography;
|
||||
using System.Text;
|
||||
using System.Text.Encodings.Web;
|
||||
using Microsoft.AspNetCore.Authentication;
|
||||
using Microsoft.Extensions.Options;
|
||||
|
||||
namespace Wiki.Host.Auth;
|
||||
|
||||
/// <summary>
|
||||
/// Machine callers with the wiki-index BasicAuth credentials: the service checks
|
||||
/// `Authorization: Basic` itself against the htpasswd lines in Auth:Basic:UsersPath, a file or a
|
||||
/// mounted Secret directory (every key is read, as Traefik's middleware reads the same Secret), so
|
||||
/// nothing depends on the request having passed a proxy.
|
||||
/// Hashes: bcrypt ($2a$, $2b$, $2y$), Apache MD5 ($apr1$), SHA1 ({SHA}), the formats Traefik
|
||||
/// accepts. Fail closed: no users, no matching user or an unknown hash format authenticates nobody.
|
||||
/// </summary>
|
||||
public sealed class BasicAuthenticationHandler(
|
||||
IOptionsMonitor<AuthenticationSchemeOptions> options,
|
||||
ILoggerFactory logger,
|
||||
UrlEncoder encoder,
|
||||
IConfiguration configuration)
|
||||
: AuthenticationHandler<AuthenticationSchemeOptions>(options, logger, encoder)
|
||||
{
|
||||
public const string SchemeName = "Basic";
|
||||
|
||||
// bcrypt costs ~100 ms per check; a verified (user, password, hash) triple is remembered.
|
||||
private static readonly ConcurrentDictionary<string, bool> Verified = new();
|
||||
|
||||
protected override async Task<AuthenticateResult> HandleAuthenticateAsync()
|
||||
{
|
||||
var authorization = Request.Headers.Authorization.FirstOrDefault();
|
||||
if (authorization?.StartsWith("Basic ", StringComparison.OrdinalIgnoreCase) != true)
|
||||
return AuthenticateResult.NoResult();
|
||||
|
||||
string user, password;
|
||||
try
|
||||
{
|
||||
var decoded = Encoding.UTF8.GetString(Convert.FromBase64String(authorization["Basic ".Length..].Trim()));
|
||||
var colon = decoded.IndexOf(':');
|
||||
if (colon <= 0) return AuthenticateResult.Fail("malformed Basic credentials");
|
||||
(user, password) = (decoded[..colon], decoded[(colon + 1)..]);
|
||||
}
|
||||
catch (FormatException)
|
||||
{
|
||||
return AuthenticateResult.Fail("malformed Basic credentials");
|
||||
}
|
||||
|
||||
var lines = await ReadUsersAsync(configuration["Auth:Basic:UsersPath"], Context.RequestAborted);
|
||||
if (lines.Count == 0)
|
||||
return AuthenticateResult.Fail("no BasicAuth users configured");
|
||||
|
||||
foreach (var line in lines)
|
||||
{
|
||||
var colon = line.IndexOf(':');
|
||||
if (colon <= 0 || line[..colon] != user) continue;
|
||||
var hash = line[(colon + 1)..].Trim();
|
||||
var key = Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes($"{user}\n{password}\n{hash}")));
|
||||
if (Verified.GetOrAdd(key, _ => Htpasswd.Verify(password, hash)))
|
||||
{
|
||||
var identity = new ClaimsIdentity([new Claim(ClaimTypes.Name, user), new Claim(Claims.Kind, "machine")], Scheme.Name);
|
||||
return AuthenticateResult.Success(new AuthenticationTicket(new ClaimsPrincipal(identity), Scheme.Name));
|
||||
}
|
||||
break;
|
||||
}
|
||||
return AuthenticateResult.Fail("invalid Basic credentials");
|
||||
}
|
||||
|
||||
/// <summary>All lines of the file, or of every visible file in the directory (a Secret mount's
|
||||
/// ..data and ..timestamp entries are skipped).</summary>
|
||||
private static async Task<List<string>> ReadUsersAsync(string? path, CancellationToken ct)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(path)) return [];
|
||||
IEnumerable<string> files = File.Exists(path) ? [path]
|
||||
: Directory.Exists(path) ? Directory.EnumerateFiles(path).Where(f => !Path.GetFileName(f).StartsWith('.')).Order()
|
||||
: [];
|
||||
var lines = new List<string>();
|
||||
foreach (var file in files) lines.AddRange(await File.ReadAllLinesAsync(file, ct));
|
||||
return lines;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Verifies a password against one htpasswd hash.</summary>
|
||||
public static class Htpasswd
|
||||
{
|
||||
public static bool Verify(string password, string hash)
|
||||
{
|
||||
if (hash.StartsWith("$2a$") || hash.StartsWith("$2b$") || hash.StartsWith("$2y$"))
|
||||
{
|
||||
try { return BCrypt.Net.BCrypt.Verify(password, hash); }
|
||||
catch (BCrypt.Net.SaltParseException) { return false; }
|
||||
}
|
||||
if (hash.StartsWith("$apr1$"))
|
||||
{
|
||||
var salt = hash["$apr1$".Length..].Split('$')[0];
|
||||
return FixedEquals(Apr1(password, salt), hash);
|
||||
}
|
||||
if (hash.StartsWith("{SHA}"))
|
||||
return FixedEquals("{SHA}" + Convert.ToBase64String(SHA1.HashData(Encoding.UTF8.GetBytes(password))), hash);
|
||||
return false;
|
||||
}
|
||||
|
||||
private static bool FixedEquals(string a, string b) =>
|
||||
CryptographicOperations.FixedTimeEquals(Encoding.ASCII.GetBytes(a), Encoding.ASCII.GetBytes(b));
|
||||
|
||||
/// <summary>Apache's MD5-crypt variant, as written by `htpasswd -m`.</summary>
|
||||
public static string Apr1(string password, string salt)
|
||||
{
|
||||
const string magic = "$apr1$";
|
||||
if (salt.Length > 8) salt = salt[..8];
|
||||
var pw = Encoding.UTF8.GetBytes(password);
|
||||
var sl = Encoding.UTF8.GetBytes(salt);
|
||||
var mg = Encoding.ASCII.GetBytes(magic);
|
||||
|
||||
var alt = MD5.HashData([.. pw, .. sl, .. pw]);
|
||||
var ctx = new List<byte>([.. pw, .. mg, .. sl]);
|
||||
for (var left = pw.Length; left > 0; left -= 16) ctx.AddRange(alt.Take(Math.Min(16, left)));
|
||||
for (var i = pw.Length; i > 0; i >>= 1) ctx.Add((i & 1) != 0 ? (byte)0 : pw.Length > 0 ? pw[0] : (byte)0);
|
||||
var final = MD5.HashData(ctx.ToArray());
|
||||
|
||||
for (var i = 0; i < 1000; i++)
|
||||
{
|
||||
var round = new List<byte>();
|
||||
round.AddRange((i & 1) != 0 ? pw : final);
|
||||
if (i % 3 != 0) round.AddRange(sl);
|
||||
if (i % 7 != 0) round.AddRange(pw);
|
||||
round.AddRange((i & 1) != 0 ? final : pw);
|
||||
final = MD5.HashData(round.ToArray());
|
||||
}
|
||||
|
||||
var sb = new StringBuilder(magic).Append(salt).Append('$');
|
||||
void To64(int v, int n) { for (; n > 0; n--, v >>= 6) sb.Append(Itoa64[v & 0x3f]); }
|
||||
To64((final[0] << 16) | (final[6] << 8) | final[12], 4);
|
||||
To64((final[1] << 16) | (final[7] << 8) | final[13], 4);
|
||||
To64((final[2] << 16) | (final[8] << 8) | final[14], 4);
|
||||
To64((final[3] << 16) | (final[9] << 8) | final[15], 4);
|
||||
To64((final[4] << 16) | (final[10] << 8) | final[5], 4);
|
||||
To64(final[11], 2);
|
||||
return sb.ToString();
|
||||
}
|
||||
|
||||
private const string Itoa64 = "./0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz";
|
||||
}
|
||||
@@ -1,71 +0,0 @@
|
||||
using System.Net;
|
||||
using System.Security.Claims;
|
||||
using System.Text.Encodings.Web;
|
||||
using Microsoft.AspNetCore.Authentication;
|
||||
using Microsoft.Extensions.Options;
|
||||
|
||||
namespace Wiki.Host.Auth;
|
||||
|
||||
/// <summary>
|
||||
/// Humans arrive through the Authentik proxy outpost, which sets X-Authentik-Username after
|
||||
/// validating the session. The header is trusted only from the peers listed in
|
||||
/// Auth:ReverseProxy:TrustedProxies; with none configured the scheme authenticates nobody.
|
||||
/// </summary>
|
||||
public sealed class ProxyHeaderAuthenticationHandler(
|
||||
IOptionsMonitor<AuthenticationSchemeOptions> options,
|
||||
ILoggerFactory logger,
|
||||
UrlEncoder encoder,
|
||||
IConfiguration configuration)
|
||||
: AuthenticationHandler<AuthenticationSchemeOptions>(options, logger, encoder)
|
||||
{
|
||||
public const string SchemeName = "ProxyHeader";
|
||||
public const string DefaultHeaderName = "X-Authentik-Username";
|
||||
|
||||
protected override Task<AuthenticateResult> HandleAuthenticateAsync()
|
||||
{
|
||||
var headerName = configuration["Auth:ReverseProxy:HeaderName"];
|
||||
if (string.IsNullOrWhiteSpace(headerName)) headerName = DefaultHeaderName;
|
||||
|
||||
var user = Request.Headers[headerName].FirstOrDefault();
|
||||
if (string.IsNullOrWhiteSpace(user))
|
||||
return Task.FromResult(AuthenticateResult.NoResult());
|
||||
|
||||
var peer = Context.Connection.RemoteIpAddress;
|
||||
if (peer is null || !IsTrustedProxy(peer))
|
||||
{
|
||||
Logger.LogWarning("ignoring {Header} from untrusted peer {Peer}", headerName, peer);
|
||||
return Task.FromResult(AuthenticateResult.NoResult());
|
||||
}
|
||||
|
||||
var identity = new ClaimsIdentity([new Claim(ClaimTypes.Name, user), new Claim(Claims.Kind, "human")], Scheme.Name);
|
||||
return Task.FromResult(AuthenticateResult.Success(new AuthenticationTicket(new ClaimsPrincipal(identity), Scheme.Name)));
|
||||
}
|
||||
|
||||
protected override Task HandleChallengeAsync(AuthenticationProperties properties)
|
||||
{
|
||||
Response.StatusCode = StatusCodes.Status401Unauthorized;
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
|
||||
private bool IsTrustedProxy(IPAddress peer)
|
||||
{
|
||||
if (peer.IsIPv4MappedToIPv6) peer = peer.MapToIPv4();
|
||||
var trusted = configuration.GetSection("Auth:ReverseProxy:TrustedProxies").Get<string[]>() ?? [];
|
||||
foreach (var entry in trusted)
|
||||
{
|
||||
if (IPAddress.TryParse(entry, out var single))
|
||||
{
|
||||
if (single.Equals(peer)) return true;
|
||||
continue;
|
||||
}
|
||||
if (IPNetwork.TryParse(entry, out var network) && network.Contains(peer))
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
public static class Claims
|
||||
{
|
||||
public const string Kind = "wiki:kind";
|
||||
}
|
||||
@@ -0,0 +1,97 @@
|
||||
using System.Security.Claims;
|
||||
using Microsoft.AspNetCore.Authentication;
|
||||
using Microsoft.AspNetCore.Authentication.Cookies;
|
||||
using Microsoft.AspNetCore.Authentication.OpenIdConnect;
|
||||
using Microsoft.AspNetCore.Authorization;
|
||||
using Microsoft.IdentityModel.Protocols.OpenIdConnect;
|
||||
|
||||
namespace Wiki.Host.Auth;
|
||||
|
||||
/// <summary>
|
||||
/// Humans sign in with OpenID Connect against Authentik (public client, PKCE): the username comes
|
||||
/// from the ID token, whose signature the handler checks against the provider's JWKS, so no
|
||||
/// network path or request header can assert an identity. Machines present an API key or the
|
||||
/// BasicAuth credentials of the wiki-index machine door, which the service checks itself.
|
||||
/// Every endpoint needs one of these unless it opts out (/health, /ready, the HMAC webhook).
|
||||
/// </summary>
|
||||
public static class WikiAuthentication
|
||||
{
|
||||
public const string UsernameClaim = "preferred_username";
|
||||
|
||||
public static IServiceCollection AddWikiAuthentication(this IServiceCollection services, IConfiguration configuration)
|
||||
{
|
||||
var authority = configuration["Auth:Oidc:Authority"];
|
||||
var clientId = configuration["Auth:Oidc:ClientId"];
|
||||
if (string.IsNullOrWhiteSpace(authority) || string.IsNullOrWhiteSpace(clientId))
|
||||
throw new InvalidOperationException("Auth:Oidc:Authority and Auth:Oidc:ClientId are required; humans have no other way to sign in.");
|
||||
|
||||
services.AddAuthentication(options =>
|
||||
{
|
||||
options.DefaultScheme = CookieAuthenticationDefaults.AuthenticationScheme;
|
||||
options.DefaultChallengeScheme = OpenIdConnectDefaults.AuthenticationScheme;
|
||||
})
|
||||
.AddCookie(options =>
|
||||
{
|
||||
options.Cookie.Name = "wiki.auth";
|
||||
options.Cookie.SecurePolicy = CookieSecurePolicy.Always;
|
||||
options.Cookie.SameSite = SameSiteMode.Lax;
|
||||
options.SlidingExpiration = true;
|
||||
options.ExpireTimeSpan = TimeSpan.FromHours(12);
|
||||
// Only the OIDC scheme sends anyone to a login; the cookie scheme answers API callers with a status.
|
||||
options.Events.OnRedirectToLogin = ctx => { ctx.Response.StatusCode = StatusCodes.Status401Unauthorized; return Task.CompletedTask; };
|
||||
options.Events.OnRedirectToAccessDenied = ctx => { ctx.Response.StatusCode = StatusCodes.Status403Forbidden; return Task.CompletedTask; };
|
||||
})
|
||||
.AddOpenIdConnect(options =>
|
||||
{
|
||||
options.Authority = authority;
|
||||
options.ClientId = clientId;
|
||||
options.ResponseType = OpenIdConnectResponseType.Code;
|
||||
options.UsePkce = true;
|
||||
options.Scope.Clear();
|
||||
options.Scope.Add("openid");
|
||||
options.Scope.Add("profile");
|
||||
options.MapInboundClaims = false;
|
||||
options.GetClaimsFromUserInfoEndpoint = true;
|
||||
options.ClaimActions.MapUniqueJsonKey(UsernameClaim, UsernameClaim);
|
||||
options.TokenValidationParameters.NameClaimType = UsernameClaim;
|
||||
options.SaveTokens = false;
|
||||
options.Events.OnRedirectToIdentityProvider = ctx =>
|
||||
{
|
||||
// TLS ends at Traefik, so the request scheme is http; browsers only ever reach the wiki over https.
|
||||
ctx.ProtocolMessage.RedirectUri = $"https://{ctx.Request.Host}{ctx.Options.CallbackPath}";
|
||||
return Task.CompletedTask;
|
||||
};
|
||||
options.Events.OnTokenValidated = ctx =>
|
||||
{
|
||||
if (ctx.Principal?.Identity is ClaimsIdentity identity)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(identity.Name))
|
||||
{
|
||||
ctx.Fail($"ID token carries no {UsernameClaim}");
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
identity.AddClaim(new Claim(Claims.Kind, "human"));
|
||||
}
|
||||
return Task.CompletedTask;
|
||||
};
|
||||
})
|
||||
.AddScheme<AuthenticationSchemeOptions, ApiKeyAuthenticationHandler>(ApiKeyAuthenticationHandler.SchemeName, null)
|
||||
.AddScheme<AuthenticationSchemeOptions, BasicAuthenticationHandler>(BasicAuthenticationHandler.SchemeName, null);
|
||||
|
||||
services.AddAuthorization(options =>
|
||||
{
|
||||
// UI pages, static assets and the Blazor hub: a signed-in human. No explicit scheme, so the
|
||||
// challenge goes to the default (OIDC) and an anonymous browser is sent to Authentik.
|
||||
options.FallbackPolicy = new AuthorizationPolicyBuilder()
|
||||
.RequireAuthenticatedUser()
|
||||
.Build();
|
||||
options.AddPolicy(Policies.Read, p => p
|
||||
.AddAuthenticationSchemes(CookieAuthenticationDefaults.AuthenticationScheme, ApiKeyAuthenticationHandler.SchemeName, BasicAuthenticationHandler.SchemeName)
|
||||
.RequireAuthenticatedUser());
|
||||
options.AddPolicy(Policies.Write, p => p
|
||||
.AddAuthenticationSchemes(ApiKeyAuthenticationHandler.SchemeName)
|
||||
.RequireAuthenticatedUser());
|
||||
});
|
||||
return services;
|
||||
}
|
||||
}
|
||||
@@ -18,6 +18,7 @@
|
||||
<ReconnectModal />
|
||||
<script src="@Assets["_framework/blazor.web.js"]"></script>
|
||||
<script src="@Assets["_content/MudBlazor/MudBlazor.min.js"]"></script>
|
||||
<script src="@Assets["facts.js"]"></script>
|
||||
</body>
|
||||
|
||||
</html>
|
||||
|
||||
@@ -14,6 +14,7 @@
|
||||
<MudNavMenu Color="Color.Primary" Bordered="false">
|
||||
<MudNavLink Href="" Match="NavLinkMatch.All" Icon="@Icons.Material.Outlined.Search">Search</MudNavLink>
|
||||
<MudNavLink Href="answers" Icon="@Icons.Material.Outlined.QuestionAnswer">Answers</MudNavLink>
|
||||
<MudNavLink Href="facts" Icon="@Icons.Material.Outlined.NoteAdd">Facts</MudNavLink>
|
||||
<MudNavLink Href="browse" Icon="@Icons.Material.Outlined.Label">Browse</MudNavLink>
|
||||
<MudNavLink Href="maintenance" Icon="@Icons.Material.Outlined.Build">Maintenance</MudNavLink>
|
||||
<MudNavLink Href="lab" Icon="@Icons.Material.Outlined.Science">Lab</MudNavLink>
|
||||
|
||||
@@ -0,0 +1,192 @@
|
||||
@page "/facts"
|
||||
@rendermode InteractiveServer
|
||||
@using Microsoft.AspNetCore.Components.Authorization
|
||||
@using Wiki.Application.Facts
|
||||
@inject RecordStatementUseCase Record
|
||||
@inject IFactDrafter Drafter
|
||||
@inject NoteQueries Notes
|
||||
@inject ISnackbar Snackbar
|
||||
@inject IJSRuntime JS
|
||||
|
||||
<PageTitle>Facts</PageTitle>
|
||||
|
||||
<MudText Typo="Typo.h4" GutterBottom="true">Facts</MudText>
|
||||
<MudText Typo="Typo.body2" Class="mb-4" Color="Color.Secondary">
|
||||
Say what you observed, in one sentence or a few, and add or paste a picture when the thing itself says it best, a type label for instance. @Drafter.ModelId turns it into a raw source under your name, with your words kept verbatim and the pictures next to the note, and the nightly distill turns that into wiki notes. Observations only, never hypotheses.
|
||||
</MudText>
|
||||
|
||||
@if (_user is null)
|
||||
{
|
||||
<MudAlert Severity="Severity.Warning" Class="mb-4">Not signed in. Facts are attributed to the person stating them, so this page needs the Authentik sign-in.</MudAlert>
|
||||
}
|
||||
else
|
||||
{
|
||||
<form @onsubmit="RecordAsync" @onsubmit:preventDefault="true">
|
||||
<MudTextField T="string" @bind-Value="_statement" Immediate="true" Lines="3" Label="What did you observe?" Variant="Variant.Outlined" Disabled="@_running" Class="mb-2" />
|
||||
<InputFile id="fact-files" OnChange="OnFilesAsync" multiple accept="image/*" style="display:none" />
|
||||
@if (_images.Count > 0)
|
||||
{
|
||||
<MudStack Row Wrap="Wrap.Wrap" Spacing="2" Class="mb-2">
|
||||
@foreach (var img in _images)
|
||||
{
|
||||
<MudPaper Outlined="true" Class="pa-1" Style="position: relative">
|
||||
<img src="@img.DataUrl" alt="attachment" style="max-height: 140px; display: block" />
|
||||
<MudIconButton Icon="@Icons.Material.Filled.Close" Size="Size.Small" Style="position: absolute; top: 0; right: 0" OnClick="@(() => _images.Remove(img))" Disabled="@_running" />
|
||||
</MudPaper>
|
||||
}
|
||||
</MudStack>
|
||||
}
|
||||
<MudStack Row Justify="Justify.SpaceBetween" AlignItems="AlignItems.Center" Class="mb-4">
|
||||
<MudStack Row AlignItems="AlignItems.Center" Spacing="2">
|
||||
<MudButton Variant="Variant.Outlined" StartIcon="@Icons.Material.Outlined.AddPhotoAlternate" OnClick="PickAsync" Disabled="@_running">Add picture</MudButton>
|
||||
<MudText Typo="Typo.caption" Color="Color.Secondary">or paste one · captured_by: @_user</MudText>
|
||||
</MudStack>
|
||||
<MudButton ButtonType="ButtonType.Submit" Variant="Variant.Filled" Color="Color.Primary" Disabled="@(_running || string.IsNullOrWhiteSpace(_statement))">Record</MudButton>
|
||||
</MudStack>
|
||||
</form>
|
||||
}
|
||||
|
||||
@if (_running)
|
||||
{
|
||||
<MudProgressLinear Indeterminate="true" Class="mb-4" />
|
||||
}
|
||||
@if (_error is not null)
|
||||
{
|
||||
<MudAlert Severity="Severity.Error" Class="mb-4">@_error</MudAlert>
|
||||
}
|
||||
@if (_recorded is not null)
|
||||
{
|
||||
<MudPaper Class="pa-4 mb-6" Outlined="true">
|
||||
<MudStack Row Justify="Justify.SpaceBetween" AlignItems="AlignItems.Center" Class="mb-2">
|
||||
<MudText Typo="Typo.h6">@_recorded.Draft.Title</MudText>
|
||||
<MudLink Href="@($"raw/{Slug(_recorded.Receipt.Path)}")">@_recorded.Receipt.Path</MudLink>
|
||||
</MudStack>
|
||||
<MudText Typo="Typo.body1" Style="white-space: pre-wrap" Class="mb-2">@_recorded.Draft.Content</MudText>
|
||||
<MudStack Row Spacing="1">
|
||||
@foreach (var t in _recorded.Draft.Tags)
|
||||
{
|
||||
<MudChip T="string" Size="Size.Small" Variant="Variant.Outlined">@t</MudChip>
|
||||
}
|
||||
</MudStack>
|
||||
</MudPaper>
|
||||
}
|
||||
|
||||
<MudText Typo="Typo.subtitle1" Class="mb-2">Raw sources awaiting distillation</MudText>
|
||||
@if (_raw is null)
|
||||
{
|
||||
<MudProgressLinear Indeterminate="true" />
|
||||
}
|
||||
else if (_raw.Count == 0)
|
||||
{
|
||||
<MudText Typo="Typo.body2" Color="Color.Secondary">Nothing queued.</MudText>
|
||||
}
|
||||
else
|
||||
{
|
||||
<MudSimpleTable Dense="true" Hover="true">
|
||||
<thead>
|
||||
<tr><th>Captured</th><th>Title</th><th>Source</th></tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
@foreach (var r in _raw.OrderByDescending(r => r.Captured).ThenByDescending(r => r.Slug).Take(30))
|
||||
{
|
||||
<tr>
|
||||
<td>@(r.Captured?.ToString("yyyy-MM-dd") ?? "")</td>
|
||||
<td><MudLink Href="@($"raw/{r.Slug}")">@r.Title</MudLink></td>
|
||||
<td><MudText Typo="Typo.caption" Color="Color.Secondary">@r.Source</MudText></td>
|
||||
</tr>
|
||||
}
|
||||
</tbody>
|
||||
</MudSimpleTable>
|
||||
}
|
||||
|
||||
@code {
|
||||
[CascadingParameter] private Task<AuthenticationState>? AuthState { get; set; }
|
||||
|
||||
private bool _running;
|
||||
private string? _user;
|
||||
private string? _statement;
|
||||
private string? _error;
|
||||
private RecordedStatement? _recorded;
|
||||
private IReadOnlyList<RawQueueEntry>? _raw;
|
||||
private readonly List<PendingImage> _images = [];
|
||||
|
||||
private sealed record PendingImage(string MediaType, byte[] Bytes, string DataUrl);
|
||||
|
||||
protected override async Task OnInitializedAsync()
|
||||
{
|
||||
if (AuthState is not null)
|
||||
{
|
||||
var state = await AuthState;
|
||||
_user = state.User.Identity?.IsAuthenticated == true ? state.User.Identity.Name : null;
|
||||
}
|
||||
await LoadRawAsync();
|
||||
}
|
||||
|
||||
protected override async Task OnAfterRenderAsync(bool firstRender)
|
||||
{
|
||||
if (firstRender && _user is not null)
|
||||
await JS.InvokeVoidAsync("wikiFacts.listenPaste", "fact-files");
|
||||
}
|
||||
|
||||
private async Task LoadRawAsync() => _raw = await Notes.RawQueueAsync(CancellationToken.None);
|
||||
|
||||
private async Task PickAsync() => await JS.InvokeVoidAsync("wikiFacts.pick", "fact-files");
|
||||
|
||||
private async Task OnFilesAsync(InputFileChangeEventArgs e)
|
||||
{
|
||||
foreach (var file in e.GetMultipleFiles(10))
|
||||
{
|
||||
if (!file.ContentType.StartsWith("image/", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
Snackbar.Add($"{file.Name}: not an image", Severity.Warning);
|
||||
continue;
|
||||
}
|
||||
try
|
||||
{
|
||||
// Resized in the browser: labels stay legible at 1600px and the upload stays small.
|
||||
var resized = await file.RequestImageFileAsync("image/jpeg", 1600, 1600);
|
||||
using var ms = new MemoryStream();
|
||||
await using (var stream = resized.OpenReadStream(Attachment.MaxBytes))
|
||||
await stream.CopyToAsync(ms);
|
||||
var bytes = ms.ToArray();
|
||||
_images.Add(new PendingImage("image/jpeg", bytes, "data:image/jpeg;base64," + Convert.ToBase64String(bytes)));
|
||||
}
|
||||
catch (IOException ex)
|
||||
{
|
||||
Snackbar.Add($"{file.Name}: {ex.Message}", Severity.Error);
|
||||
}
|
||||
}
|
||||
StateHasChanged();
|
||||
}
|
||||
|
||||
private async Task RecordAsync()
|
||||
{
|
||||
if (_running || _user is null || string.IsNullOrWhiteSpace(_statement)) return;
|
||||
_running = true;
|
||||
_error = null;
|
||||
_recorded = null;
|
||||
try
|
||||
{
|
||||
var attachments = _images.Select(i => new Attachment(i.MediaType, i.Bytes)).ToList();
|
||||
_recorded = await Record.RecordAsync(_statement, attachments, new Identity(_user, IdentityKind.Human), Drafter, CancellationToken.None);
|
||||
Snackbar.Add($"recorded {_recorded.Receipt.Path}", Severity.Success);
|
||||
_statement = null;
|
||||
_images.Clear();
|
||||
await LoadRawAsync();
|
||||
}
|
||||
catch (Exception e) when (e is HttpRequestException or InvalidOperationException)
|
||||
{
|
||||
_error = e.Message;
|
||||
}
|
||||
finally
|
||||
{
|
||||
_running = false;
|
||||
}
|
||||
}
|
||||
|
||||
private static string Slug(string path)
|
||||
{
|
||||
var file = path[(path.LastIndexOf('/') + 1)..];
|
||||
return file.EndsWith(".md", StringComparison.Ordinal) ? file[..^3] : file;
|
||||
}
|
||||
}
|
||||
@@ -1,6 +1,7 @@
|
||||
@page "/raw/{slug}"
|
||||
@rendermode InteractiveServer
|
||||
@inject NoteQueries Notes
|
||||
@inject INoteStore Store
|
||||
|
||||
<PageTitle>@(_doc?.Title ?? Slug)</PageTitle>
|
||||
|
||||
@@ -33,6 +34,19 @@ else
|
||||
<div class="wiki-body">
|
||||
@((MarkupString)_html!)
|
||||
</div>
|
||||
@if (Attachments(_doc) is { Count: > 0 } attachments)
|
||||
{
|
||||
<MudText Typo="Typo.subtitle2" Class="mt-4 mb-1">Attachments</MudText>
|
||||
<MudList T="string" Dense="true">
|
||||
@foreach (var a in attachments)
|
||||
{
|
||||
var href = Store.BrowseUrl($"raw/{a}")?.ToString();
|
||||
<MudListItem T="string" Icon="@Icons.Material.Outlined.Image">
|
||||
@if (href is null) { @a } else { <MudLink Href="@href" Target="_blank">@a</MudLink> }
|
||||
</MudListItem>
|
||||
}
|
||||
</MudList>
|
||||
}
|
||||
}
|
||||
|
||||
@code {
|
||||
@@ -52,4 +66,11 @@ else
|
||||
}
|
||||
|
||||
private static string? Scalar(Document d, string key) => d.Frontmatter.TryGetValue(key, out var v) ? v.Scalar : null;
|
||||
|
||||
private static IReadOnlyList<string> Attachments(Document d)
|
||||
{
|
||||
if (!d.Frontmatter.TryGetValue("attachments", out var v)) return [];
|
||||
if (v.List is { } list) return list;
|
||||
return (v.Scalar ?? "").Trim('[', ']').Split(',', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries);
|
||||
}
|
||||
}
|
||||
|
||||
+13
-11
@@ -1,4 +1,3 @@
|
||||
using Microsoft.AspNetCore.Authentication;
|
||||
using MudBlazor.Services;
|
||||
using Wiki.Adapters.FileIndex;
|
||||
using Wiki.Adapters.Gitea;
|
||||
@@ -74,14 +73,16 @@ foreach (var (name, answer) in wiki.AnswerModels.Models)
|
||||
switch (answer.Adapter)
|
||||
{
|
||||
case "Anthropic":
|
||||
builder.Services.AddAnthropicAnswerModel(name, o =>
|
||||
Action<Wiki.Adapters.Anthropic.AnthropicOptions> anthropic = o =>
|
||||
{
|
||||
o.BaseUrl = answer.Url;
|
||||
o.Model = answer.Model;
|
||||
o.ModelId = name;
|
||||
o.MaxTokens = answer.MaxTokens;
|
||||
o.ApiKey = builder.Configuration[$"Wiki:AnswerModels:Models:{name}:ApiKey"] ?? string.Empty;
|
||||
});
|
||||
};
|
||||
builder.Services.AddAnthropicAnswerModel(name, anthropic);
|
||||
builder.Services.AddAnthropicFactDrafter(name, anthropic);
|
||||
break;
|
||||
default:
|
||||
throw new InvalidOperationException($"unknown answer-model adapter '{answer.Adapter}' for '{name}'");
|
||||
@@ -90,6 +91,10 @@ foreach (var (name, answer) in wiki.AnswerModels.Models)
|
||||
builder.Services.AddSingleton<IReadOnlyList<IAnswerModel>>(sp =>
|
||||
wiki.AnswerModels.Models.Keys.Select(sp.GetRequiredKeyedService<IAnswerModel>).ToList());
|
||||
builder.Services.AddSingleton<IAnswerModel>(sp => sp.GetRequiredKeyedService<IAnswerModel>(wiki.AnswerModels.Active));
|
||||
// Fact drafters ride with the answer models: same entries, same keys.
|
||||
builder.Services.AddSingleton<IReadOnlyList<IFactDrafter>>(sp =>
|
||||
wiki.AnswerModels.Models.Keys.Select(sp.GetRequiredKeyedService<IFactDrafter>).ToList());
|
||||
builder.Services.AddSingleton<IFactDrafter>(sp => sp.GetRequiredKeyedService<IFactDrafter>(wiki.AnswerModels.Active));
|
||||
// One adapter may serve several ports; list it once.
|
||||
builder.Services.AddSingleton<IReadOnlyList<IAdapterHealth>>(sp =>
|
||||
{
|
||||
@@ -101,6 +106,7 @@ builder.Services.AddSingleton<IReadOnlyList<IAdapterHealth>>(sp =>
|
||||
.. sp.GetRequiredService<IReadOnlyList<IEmbeddingModel>>(),
|
||||
.. sp.GetRequiredService<IReadOnlyList<IJudge>>(),
|
||||
.. sp.GetRequiredService<IReadOnlyList<IAnswerModel>>(),
|
||||
.. sp.GetRequiredService<IReadOnlyList<IFactDrafter>>(),
|
||||
];
|
||||
return all.Distinct().ToList();
|
||||
});
|
||||
@@ -114,6 +120,7 @@ builder.Services.AddSingleton<Wiki.Application.Reading.NoteQueries>();
|
||||
builder.Services.AddSingleton<IChunker, Wiki.Core.Parsing.ParagraphChunker>();
|
||||
builder.Services.AddSingleton<Wiki.Application.Ask.AskUseCase>();
|
||||
builder.Services.AddSingleton<Wiki.Application.Facts.DepositFactUseCase>();
|
||||
builder.Services.AddSingleton<Wiki.Application.Facts.RecordStatementUseCase>();
|
||||
builder.Services.AddSingleton<Wiki.Application.Evaluation.EvaluateUseCase>();
|
||||
builder.Services.AddSingleton<Wiki.Application.Questions.QuestionUseCase>();
|
||||
builder.Services.AddSingleton<Wiki.Application.Questions.SaveAnswerUseCase>();
|
||||
@@ -121,17 +128,12 @@ builder.Services.AddSingleton<StatusQuery>();
|
||||
builder.Services.AddSingleton<SyncBackgroundService>();
|
||||
builder.Services.AddHostedService(sp => sp.GetRequiredService<SyncBackgroundService>());
|
||||
|
||||
// ---- auth: humans arrive through the Authentik outpost, machines with an API key ---------
|
||||
builder.Services.AddAuthentication(ProxyHeaderAuthenticationHandler.SchemeName)
|
||||
.AddScheme<AuthenticationSchemeOptions, ProxyHeaderAuthenticationHandler>(ProxyHeaderAuthenticationHandler.SchemeName, null)
|
||||
.AddScheme<AuthenticationSchemeOptions, ApiKeyAuthenticationHandler>(ApiKeyAuthenticationHandler.SchemeName, null);
|
||||
builder.Services.AddAuthorization(options =>
|
||||
{
|
||||
options.AddPolicy(Policies.Write, p => p.AddAuthenticationSchemes(ApiKeyAuthenticationHandler.SchemeName).RequireAuthenticatedUser());
|
||||
});
|
||||
// ---- auth: humans sign in with OIDC against Authentik, machines with an API key ----------
|
||||
builder.Services.AddWikiAuthentication(builder.Configuration);
|
||||
|
||||
// ---- UI ----------------------------------------------------------------------------------
|
||||
builder.Services.AddRazorComponents().AddInteractiveServerComponents();
|
||||
builder.Services.AddCascadingAuthenticationState();
|
||||
builder.Services.AddMudServices();
|
||||
|
||||
var app = builder.Build();
|
||||
|
||||
@@ -10,7 +10,9 @@
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<PackageReference Include="BCrypt.Net-Next" Version="4.2.1" />
|
||||
<PackageReference Include="Markdig" Version="1.4.0" />
|
||||
<PackageReference Include="Microsoft.AspNetCore.Authentication.OpenIdConnect" Version="10.0.13" />
|
||||
<PackageReference Include="MudBlazor" Version="9.10.0" />
|
||||
</ItemGroup>
|
||||
|
||||
|
||||
@@ -34,7 +34,8 @@
|
||||
"Sync": { "IntervalSeconds": 1800, "RetrySeconds": 60 }
|
||||
},
|
||||
"Auth": {
|
||||
"ReverseProxy": { "HeaderName": "X-Authentik-Username", "TrustedProxies": [] },
|
||||
"Oidc": { "Authority": "", "ClientId": "" },
|
||||
"Basic": { "UsersPath": "" },
|
||||
"ApiKeys": {}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
// Pasted images become files on the page's hidden <input type="file">, so Blazor's InputFile
|
||||
// streams them like a picked file. One document listener; it looks the input up per paste.
|
||||
window.wikiFacts = (() => {
|
||||
let armed = false;
|
||||
return {
|
||||
listenPaste(inputId) {
|
||||
if (armed) return;
|
||||
armed = true;
|
||||
document.addEventListener('paste', (ev) => {
|
||||
const input = document.getElementById(inputId);
|
||||
if (!input || !ev.clipboardData) return;
|
||||
const files = [];
|
||||
for (const item of ev.clipboardData.items) {
|
||||
if (item.kind === 'file' && item.type.startsWith('image/')) {
|
||||
const f = item.getAsFile();
|
||||
if (f) files.push(f);
|
||||
}
|
||||
}
|
||||
if (files.length === 0) return;
|
||||
ev.preventDefault();
|
||||
const dt = new DataTransfer();
|
||||
for (const f of files) dt.items.add(f);
|
||||
input.files = dt.files;
|
||||
input.dispatchEvent(new Event('change', { bubbles: true }));
|
||||
});
|
||||
},
|
||||
pick(inputId) {
|
||||
document.getElementById(inputId)?.click();
|
||||
},
|
||||
};
|
||||
})();
|
||||
@@ -81,10 +81,10 @@ public class AskUseCaseTests
|
||||
Assert.Equal("udm-dns#p1", where.Top[0].Id);
|
||||
Assert.Contains("answers every query", where.Top[0].Text);
|
||||
Assert.Equal("UDM intercepts DNS", where.Top[0].Title);
|
||||
// the context choice went alone, the noul questions together
|
||||
Assert.Equal(2, _judge.Requests.Count);
|
||||
Assert.Contains(_judge.Requests, r => r.Questions.Count == 1 && r.Questions.ContainsKey("where_1"));
|
||||
Assert.Equal(2, result.Usage.Requests);
|
||||
// every question rides in the one request that carries the state
|
||||
var sentToJudge = Assert.Single(_judge.Requests);
|
||||
Assert.Equal(3, sentToJudge.Questions.Count);
|
||||
Assert.Equal(1, result.Usage.Requests);
|
||||
Assert.Equal("fake-judge", result.JudgeModel);
|
||||
}
|
||||
|
||||
|
||||
@@ -44,6 +44,18 @@ public sealed class FakeStore : INoteStore
|
||||
return Task.FromResult(new StoreWrite(path, Convert.ToHexStringLower(SHA1.HashData(Encoding.UTF8.GetBytes(text)))));
|
||||
}
|
||||
|
||||
public Dictionary<string, byte[]> Binaries { get; } = new();
|
||||
|
||||
public Task<StoreWrite> CreateAsync(string path, byte[] content, string message, CancellationToken ct)
|
||||
{
|
||||
if (Files.ContainsKey(path) || Binaries.ContainsKey(path)) throw new DocumentExistsException(path);
|
||||
Binaries[path] = content;
|
||||
Created.Add(($"{message}", path));
|
||||
return Task.FromResult(new StoreWrite(path, Convert.ToHexStringLower(SHA1.HashData(content))));
|
||||
}
|
||||
|
||||
public Uri? BrowseUrl(string path) => null;
|
||||
|
||||
public Task<AdapterHealth> CheckAsync(CancellationToken ct) => Task.FromResult(new AdapterHealth(Name, true, null, TimeSpan.Zero, DateTimeOffset.UnixEpoch));
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,88 @@
|
||||
using Microsoft.Extensions.Time.Testing;
|
||||
using Wiki.Application.Facts;
|
||||
using Wiki.Core.Model;
|
||||
using Wiki.Core.Ports;
|
||||
|
||||
namespace Wiki.Application.Tests;
|
||||
|
||||
public sealed class FakeFactDrafter : IFactDrafter
|
||||
{
|
||||
public List<FactStatement> Statements { get; } = [];
|
||||
public FactDraft Draft { get; set; } = new("Fußbodenheizung im Erdgeschoss", "Alle Räume im Erdgeschoss haben Fußbodenheizung; die Garderobe hat keine.", ["heating", "ground-floor"]);
|
||||
public string ModelId => "fake-drafter";
|
||||
public string Name => "fake-drafter";
|
||||
|
||||
public Task<FactDraft> DraftAsync(FactStatement statement, CancellationToken ct)
|
||||
{
|
||||
Statements.Add(statement);
|
||||
return Task.FromResult(Draft);
|
||||
}
|
||||
|
||||
public Task<AdapterHealth> CheckAsync(CancellationToken ct) => Task.FromResult(new AdapterHealth(Name, true, null, TimeSpan.Zero, DateTimeOffset.UnixEpoch));
|
||||
}
|
||||
|
||||
public class RecordStatementUseCaseTests
|
||||
{
|
||||
private static readonly DateTimeOffset Now = new(2026, 9, 25, 8, 0, 0, TimeSpan.Zero);
|
||||
private static readonly Identity By = new("k.renner", IdentityKind.Human);
|
||||
private const string Statement = "Alle Räume im Erdgeschoss haben Fußbodenheizung außer die Garderobe";
|
||||
|
||||
[Fact]
|
||||
public async Task StatementBecomesARawSourceUnderThePersonWithTheVerbatimText()
|
||||
{
|
||||
var store = new FakeStore();
|
||||
var drafter = new FakeFactDrafter();
|
||||
var useCase = new RecordStatementUseCase(new DepositFactUseCase(store, new FakeTimeProvider(Now)), new FakeTimeProvider(Now));
|
||||
|
||||
var recorded = await useCase.RecordAsync(" " + Statement + "\n", [], By, drafter, CancellationToken.None);
|
||||
|
||||
var sent = Assert.Single(drafter.Statements);
|
||||
Assert.Equal(Statement, sent.Text);
|
||||
Assert.Equal(By, sent.By);
|
||||
Assert.Equal("raw/2026-09-25-fu-bodenheizung-im-erdgeschoss.md", recorded.Receipt.Path);
|
||||
var text = store.Files[recorded.Receipt.Path];
|
||||
Assert.Contains("title: Fußbodenheizung im Erdgeschoss\n", text);
|
||||
Assert.Contains("source: stated by k.renner on 2026-09-25 via the wiki\n", text);
|
||||
Assert.Contains("captured_by: k.renner\n", text);
|
||||
Assert.Contains("tags: [heating, ground-floor]\n", text);
|
||||
Assert.Contains("die Garderobe hat keine.\n\nStatement, verbatim:\n\n> " + Statement + "\n", text);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ADraftThatIsNoRawSourceIsNotStored()
|
||||
{
|
||||
var store = new FakeStore();
|
||||
var drafter = new FakeFactDrafter { Draft = new FactDraft(" ", "the verbatim statement always makes the body long enough; only a missing title fails", []) };
|
||||
var useCase = new RecordStatementUseCase(new DepositFactUseCase(store, new FakeTimeProvider(Now)), new FakeTimeProvider(Now));
|
||||
|
||||
await Assert.ThrowsAsync<InvalidOperationException>(() => useCase.RecordAsync(Statement, [], By, drafter, CancellationToken.None));
|
||||
Assert.Empty(store.Files);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task EmptyStatementsNeverReachTheDrafter()
|
||||
{
|
||||
var drafter = new FakeFactDrafter();
|
||||
var useCase = new RecordStatementUseCase(new DepositFactUseCase(new FakeStore(), new FakeTimeProvider(Now)), new FakeTimeProvider(Now));
|
||||
await Assert.ThrowsAsync<ArgumentException>(() => useCase.RecordAsync(" \n", [], By, drafter, CancellationToken.None));
|
||||
Assert.Empty(drafter.Statements);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task PicturesReachTheDrafterAndLandNextToTheNote()
|
||||
{
|
||||
var store = new FakeStore();
|
||||
var drafter = new FakeFactDrafter { Draft = new FactDraft("Geschirrspüler Typenschild", "Der Geschirrspüler ist ein Bosch SMV4HCX48E, Seriennummer [unreadable].", ["dishwasher"]) };
|
||||
var useCase = new RecordStatementUseCase(new DepositFactUseCase(store, new FakeTimeProvider(Now)), new FakeTimeProvider(Now));
|
||||
var photo = new Attachment("image/jpeg", [0xFF, 0xD8, 0xFF, 1, 2, 3], "type label");
|
||||
|
||||
var recorded = await useCase.RecordAsync("Das ist das Typenschild unseres Geschirrspülers.", [photo], By, drafter, CancellationToken.None);
|
||||
|
||||
Assert.Same(photo, Assert.Single(Assert.Single(drafter.Statements).Attachments));
|
||||
Assert.Equal("raw/2026-09-25-geschirrsp-ler-typenschild.md", recorded.Receipt.Path);
|
||||
Assert.Equal(photo.Bytes, store.Binaries["raw/2026-09-25-geschirrsp-ler-typenschild-1.jpg"]);
|
||||
var text = store.Files[recorded.Receipt.Path];
|
||||
Assert.Contains("attachments: [2026-09-25-geschirrsp-ler-typenschild-1.jpg]\n", text);
|
||||
Assert.EndsWith("\n", text);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,63 @@
|
||||
using Wiki.Adapters.Anthropic;
|
||||
using Wiki.Core.Ports;
|
||||
|
||||
namespace Wiki.Contracts.Anthropic;
|
||||
|
||||
/// <summary>Runs the shared <see cref="FactDrafterContract"/> against the Anthropic adapter over a fake server that replies with a JSON draft.</summary>
|
||||
public sealed class AnthropicFactDrafterContractTests : FactDrafterContract
|
||||
{
|
||||
internal const string DraftJson =
|
||||
"{\"title\": \"Fußbodenheizung im Erdgeschoss\", " +
|
||||
"\"content\": \"Alle Räume im Erdgeschoss haben Fußbodenheizung. Die Garderobe ist die einzige Ausnahme.\", " +
|
||||
"\"tags\": [\"Heating\", \"floor heating\", \"ground-floor\", \"heating\"]}";
|
||||
|
||||
protected override Task<IFactDrafter> CreateDrafterAsync()
|
||||
{
|
||||
var handler = new FakeAnthropicHandler { ResponseText = DraftJson };
|
||||
var client = new HttpClient(handler);
|
||||
var options = new AnthropicOptions
|
||||
{
|
||||
BaseUrl = "https://anthropic.test",
|
||||
Model = "claude-sonnet-5",
|
||||
ApiKey = "test-key",
|
||||
};
|
||||
|
||||
IFactDrafter drafter = new AnthropicFactDrafter(client, options, delay: (_, _) => Task.CompletedTask);
|
||||
return Task.FromResult(drafter);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task PicturesTravelAsBase64ImageBlocksBeforeTheText()
|
||||
{
|
||||
var handler = new FakeAnthropicHandler { ResponseText = DraftJson };
|
||||
var drafter = new AnthropicFactDrafter(new HttpClient(handler), new AnthropicOptions { BaseUrl = "https://anthropic.test", ApiKey = "k" }, delay: (_, _) => Task.CompletedTask);
|
||||
var statement = Statement() with { Attachments = [new Wiki.Core.Model.Attachment("image/jpeg", [1, 2, 3])] };
|
||||
|
||||
await drafter.DraftAsync(statement, CancellationToken.None);
|
||||
|
||||
using var request = System.Text.Json.JsonDocument.Parse(handler.LastRequestJson!);
|
||||
var content = request.RootElement.GetProperty("messages")[0].GetProperty("content");
|
||||
Assert.Equal(2, content.GetArrayLength());
|
||||
Assert.Equal("image", content[0].GetProperty("type").GetString());
|
||||
Assert.Equal("image/jpeg", content[0].GetProperty("source").GetProperty("media_type").GetString());
|
||||
Assert.Equal("AQID", content[0].GetProperty("source").GetProperty("data").GetString());
|
||||
Assert.Equal("text", content[1].GetProperty("type").GetString());
|
||||
Assert.Contains("1 image(s) attached", content[1].GetProperty("text").GetString());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ParseTakesTheObjectOutOfSurroundingTextAndNormalisesTags()
|
||||
{
|
||||
var draft = FactDraftPrompt.Parse("Here you go:\n```json\n" + DraftJson + "\n```");
|
||||
Assert.Equal("Fußbodenheizung im Erdgeschoss", draft.Title);
|
||||
Assert.Equal(["heating", "floor-heating", "ground-floor"], draft.Tags);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ParseRejectsRepliesWithoutTitleOrContent()
|
||||
{
|
||||
Assert.Throws<InvalidOperationException>(() => FactDraftPrompt.Parse("no json here"));
|
||||
Assert.Throws<InvalidOperationException>(() => FactDraftPrompt.Parse("{\"title\": \"x\"}"));
|
||||
Assert.Throws<InvalidOperationException>(() => FactDraftPrompt.Parse("{\"title\": \"x\", \"content\": }"));
|
||||
}
|
||||
}
|
||||
@@ -18,6 +18,9 @@ internal sealed class FakeAnthropicHandler : HttpMessageHandler
|
||||
|
||||
public int TooManyRequestsBeforeSuccess { get; set; }
|
||||
|
||||
/// <summary>The text the fake streams back, in three deltas.</summary>
|
||||
public string ResponseText { get; set; } = EchoText;
|
||||
|
||||
public bool RespondWithErrorEvent { get; set; }
|
||||
|
||||
public string? LastRequestJson { get; private set; }
|
||||
@@ -72,7 +75,7 @@ internal sealed class FakeAnthropicHandler : HttpMessageHandler
|
||||
throw new InvalidOperationException($"Malformed Anthropic request: {body}");
|
||||
}
|
||||
|
||||
var sse = RespondWithErrorEvent ? BuildErrorSse() : BuildSuccessSse(body.Length);
|
||||
var sse = RespondWithErrorEvent ? BuildErrorSse() : BuildSuccessSse(body.Length, ResponseText);
|
||||
|
||||
return new HttpResponseMessage(HttpStatusCode.OK)
|
||||
{
|
||||
@@ -80,10 +83,10 @@ internal sealed class FakeAnthropicHandler : HttpMessageHandler
|
||||
};
|
||||
}
|
||||
|
||||
private static string BuildSuccessSse(int requestBodyLength)
|
||||
private static string BuildSuccessSse(int requestBodyLength, string text)
|
||||
{
|
||||
var inputTokens = requestBodyLength / 4;
|
||||
var words = EchoText.Split(' ');
|
||||
var words = text.Split(' ');
|
||||
var third = words.Length / 3;
|
||||
var deltas = new[]
|
||||
{
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
using Wiki.Core.Model;
|
||||
using Wiki.Core.Ports;
|
||||
|
||||
namespace Wiki.Contracts;
|
||||
|
||||
/// <summary>
|
||||
/// Every <see cref="IFactDrafter"/> adapter derives from this. The adapter may talk to a fake
|
||||
/// server; the contract checks the draft's shape and that empty input is refused before any call.
|
||||
/// </summary>
|
||||
public abstract class FactDrafterContract
|
||||
{
|
||||
protected abstract Task<IFactDrafter> CreateDrafterAsync();
|
||||
|
||||
protected static FactStatement Statement() => new(
|
||||
"Alle Räume im Erdgeschoss haben Fußbodenheizung außer die Garderobe.",
|
||||
new Identity("k.renner", IdentityKind.Human),
|
||||
new DateTimeOffset(2026, 9, 25, 8, 0, 0, TimeSpan.Zero));
|
||||
|
||||
[Fact]
|
||||
public async Task DraftsTitleContentAndSlugTags()
|
||||
{
|
||||
var drafter = await CreateDrafterAsync();
|
||||
var draft = await drafter.DraftAsync(Statement(), CancellationToken.None);
|
||||
|
||||
Assert.False(string.IsNullOrWhiteSpace(draft.Title));
|
||||
Assert.True(draft.Content.Trim().Length >= RawSource.MinimumContentLength);
|
||||
Assert.NotEmpty(draft.Tags);
|
||||
Assert.All(draft.Tags, t => Assert.Matches("^[a-z0-9]+(-[a-z0-9]+)*$", t));
|
||||
Assert.Equal(draft.Tags.Count, draft.Tags.Distinct().Count());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task RefusesAnEmptyStatement()
|
||||
{
|
||||
var drafter = await CreateDrafterAsync();
|
||||
var empty = Statement() with { Text = " " };
|
||||
await Assert.ThrowsAsync<ArgumentException>(() => drafter.DraftAsync(empty, CancellationToken.None));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ReportsHealth()
|
||||
{
|
||||
var drafter = await CreateDrafterAsync();
|
||||
var health = await drafter.CheckAsync(CancellationToken.None);
|
||||
Assert.Equal(drafter.Name, health.Adapter);
|
||||
}
|
||||
}
|
||||
@@ -76,6 +76,18 @@ public abstract class NoteStoreContract
|
||||
store.CreateAsync("wiki/udm-dns.md", "other", "x", CancellationToken.None));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task CreatesABinaryAndRefusesAnExistingPath()
|
||||
{
|
||||
var store = await CreateStoreAsync();
|
||||
var bytes = new byte[] { 0xFF, 0xD8, 0xFF, 0xE0, 1, 2, 3 };
|
||||
var written = await store.CreateAsync("raw/2026-09-24-new-1.jpg", bytes, "raw: New (attachment 1)", CancellationToken.None);
|
||||
Assert.Equal("raw/2026-09-24-new-1.jpg", written.Path);
|
||||
Assert.False(string.IsNullOrWhiteSpace(written.Commit));
|
||||
await Assert.ThrowsAsync<DocumentExistsException>(() =>
|
||||
store.CreateAsync("raw/2026-09-24-new-1.jpg", bytes, "x", CancellationToken.None));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task HealthReportsOk()
|
||||
{
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
using System.Net;
|
||||
using System.Net.Http.Headers;
|
||||
using System.Net.Http.Json;
|
||||
using System.Text;
|
||||
using Microsoft.AspNetCore.Mvc.Testing;
|
||||
using Wiki.Host.Auth;
|
||||
|
||||
namespace Wiki.Host.Tests;
|
||||
|
||||
/// <summary>A Secret-like directory: one user per hash format Traefik accepts, split over two keys, plus a
|
||||
/// hidden entry that must be skipped. Every password is the user name.</summary>
|
||||
public static class BasicUsers
|
||||
{
|
||||
// From the Traefik BasicAuth docs (htpasswd -m, password "test"); openssl passwd -apr1 agrees.
|
||||
public const string Apr1Line = "test:$apr1$H6uskkkW$IgXLP6ewTrSuBkTrqE8wj/";
|
||||
|
||||
public static readonly string Directory = Write();
|
||||
|
||||
private static string Write()
|
||||
{
|
||||
var dir = Path.Combine(Path.GetTempPath(), "wiki-host-tests", $"htpasswd-{Guid.NewGuid():N}");
|
||||
System.IO.Directory.CreateDirectory(dir);
|
||||
var bcrypt = BCrypt.Net.BCrypt.HashPassword("agent", 4).Replace("$2a$", "$2y$");
|
||||
var sha = "{SHA}" + Convert.ToBase64String(System.Security.Cryptography.SHA1.HashData(Encoding.UTF8.GetBytes("legacy")));
|
||||
File.WriteAllLines(Path.Combine(dir, "users"), [Apr1Line, $"agent:{bcrypt}"]);
|
||||
File.WriteAllLines(Path.Combine(dir, "more"), [$"legacy:{sha}"]);
|
||||
File.WriteAllLines(Path.Combine(dir, "..hidden"), ["ghost:{SHA}" + Convert.ToBase64String(System.Security.Cryptography.SHA1.HashData(Encoding.UTF8.GetBytes("ghost")))]);
|
||||
return dir;
|
||||
}
|
||||
}
|
||||
|
||||
public class BasicAuthTests(WikiHostFactory factory) : IClassFixture<WikiHostFactory>
|
||||
{
|
||||
private HttpClient As(string user, string password)
|
||||
{
|
||||
var client = factory.CreateClient(new WebApplicationFactoryClientOptions { AllowAutoRedirect = false });
|
||||
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Basic",
|
||||
Convert.ToBase64String(Encoding.UTF8.GetBytes($"{user}:{password}")));
|
||||
return client;
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Apr1MatchesApacheHtpasswd() =>
|
||||
Assert.Equal("$apr1$H6uskkkW$IgXLP6ewTrSuBkTrqE8wj/", Htpasswd.Apr1("test", "H6uskkkW"));
|
||||
|
||||
[Theory]
|
||||
[InlineData("test", "test")] // apr1
|
||||
[InlineData("agent", "agent")] // bcrypt $2y$
|
||||
[InlineData("legacy", "legacy")] // {SHA}
|
||||
public async Task TheMachineDoorCredentialsReadTheApi(string user, string password)
|
||||
{
|
||||
var client = As(user, password);
|
||||
Assert.Equal(HttpStatusCode.OK, (await client.GetAsync("/api/v1/tags")).StatusCode);
|
||||
Assert.Equal(HttpStatusCode.BadRequest, (await client.PostAsJsonAsync("/rank", new { query = " " })).StatusCode);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData("test", "wrong")]
|
||||
[InlineData("nobody", "test")]
|
||||
[InlineData("agent", "")]
|
||||
[InlineData("ghost", "ghost")] // only in a hidden entry
|
||||
public async Task WrongCredentialsAre401(string user, string password) =>
|
||||
Assert.Equal(HttpStatusCode.Unauthorized, (await As(user, password).GetAsync("/api/v1/tags")).StatusCode);
|
||||
|
||||
[Fact]
|
||||
public async Task BasicCredentialsCannotWrite()
|
||||
{
|
||||
var r = await As("test", "test").PostAsJsonAsync("/api/v1/facts", new { title = "t", source = "s", kind = "fact", content = new string('x', 50) });
|
||||
Assert.Equal(HttpStatusCode.Unauthorized, r.StatusCode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task BasicCredentialsDoNotOpenTheUi() =>
|
||||
Assert.Equal(HttpStatusCode.Redirect, (await As("test", "test").GetAsync("/facts")).StatusCode);
|
||||
}
|
||||
@@ -20,16 +20,20 @@ public sealed class WikiHostFactory : WebApplicationFactory<Program>
|
||||
public const string WebhookSecret = "hook-secret";
|
||||
public const string ApiKey = "machine-key";
|
||||
public FakeStore Store { get; } = new();
|
||||
public FakeIdp Idp { get; } = new();
|
||||
|
||||
protected override void ConfigureWebHost(IWebHostBuilder builder)
|
||||
{
|
||||
builder.UseEnvironment("Testing");
|
||||
builder.UseSetting("Wiki:Webhook:Secret", WebhookSecret);
|
||||
builder.UseSetting("Auth:ApiKeys:tester", ApiKey);
|
||||
builder.UseSetting("Auth:ReverseProxy:TrustedProxies:0", "127.0.0.1");
|
||||
builder.UseSetting("Auth:Oidc:Authority", FakeIdp.Issuer);
|
||||
builder.UseSetting("Auth:Oidc:ClientId", FakeIdp.ClientId);
|
||||
builder.UseSetting("Auth:Basic:UsersPath", BasicUsers.Directory);
|
||||
builder.UseSetting("Wiki:Index:File:Directory", Path.Combine(Path.GetTempPath(), "wiki-host-tests", Guid.NewGuid().ToString("N")));
|
||||
builder.ConfigureServices(services =>
|
||||
{
|
||||
Idp.Register(services);
|
||||
services.RemoveAll<INoteStore>();
|
||||
services.AddSingleton<INoteStore>(Store);
|
||||
// The embedding model registered under the active name is replaced by a deterministic fake.
|
||||
@@ -48,6 +52,11 @@ public sealed class WikiHostFactory : WebApplicationFactory<Program>
|
||||
services.RemoveAll<IAnswerModel>();
|
||||
services.AddSingleton<IAnswerModel>(answer);
|
||||
services.AddSingleton<IReadOnlyList<IAnswerModel>>([answer]);
|
||||
var drafter = new FakeFactDrafter();
|
||||
services.AddKeyedSingleton<IFactDrafter>("claude-sonnet", drafter);
|
||||
services.RemoveAll<IFactDrafter>();
|
||||
services.AddSingleton<IFactDrafter>(drafter);
|
||||
services.AddSingleton<IReadOnlyList<IFactDrafter>>([drafter]);
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -56,6 +65,17 @@ public class EndpointTests(WikiHostFactory factory) : IClassFixture<WikiHostFact
|
||||
{
|
||||
private HttpClient Client => factory.CreateClient();
|
||||
|
||||
/// <summary>A machine caller: the read API needs a credential since humans sign in with OIDC.</summary>
|
||||
private HttpClient Keyed
|
||||
{
|
||||
get
|
||||
{
|
||||
var client = factory.CreateClient();
|
||||
client.DefaultRequestHeaders.Add("X-Api-Key", WikiHostFactory.ApiKey);
|
||||
return client;
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task HealthIsAlwaysOk()
|
||||
{
|
||||
@@ -91,9 +111,10 @@ public class EndpointTests(WikiHostFactory factory) : IClassFixture<WikiHostFact
|
||||
[Fact]
|
||||
public async Task RankRejectsEmptyQueryAndUnknownModel()
|
||||
{
|
||||
var empty = await Client.PostAsJsonAsync("/rank", new { query = " " });
|
||||
var client = Keyed;
|
||||
var empty = await client.PostAsJsonAsync("/rank", new { query = " " });
|
||||
Assert.Equal(HttpStatusCode.BadRequest, empty.StatusCode);
|
||||
var unknown = await Client.PostAsJsonAsync("/rank", new { query = "x", model = "nope" });
|
||||
var unknown = await client.PostAsJsonAsync("/rank", new { query = "x", model = "nope" });
|
||||
Assert.Equal(HttpStatusCode.BadRequest, unknown.StatusCode);
|
||||
}
|
||||
|
||||
@@ -103,9 +124,9 @@ public class EndpointTests(WikiHostFactory factory) : IClassFixture<WikiHostFact
|
||||
factory.Store.Files["wiki/udm-dns.md"] = "---\ntitle: UDM intercepts DNS\ntype: wiki\ntags: [dns]\n---\n\n# UDM intercepts DNS\n\nThe UDM answers.\n";
|
||||
var req = new HttpRequestMessage(HttpMethod.Post, "/api/v1/sync?force=true");
|
||||
req.Headers.Add("X-Api-Key", WikiHostFactory.ApiKey);
|
||||
await Client.SendAsync(req);
|
||||
await Keyed.SendAsync(req);
|
||||
|
||||
var r = await Client.PostAsJsonAsync("/rank", new { query = "udm dns", k = 3 });
|
||||
var r = await Keyed.PostAsJsonAsync("/rank", new { query = "udm dns", k = 3 });
|
||||
Assert.Equal(HttpStatusCode.OK, r.StatusCode);
|
||||
var body = await r.Content.ReadFromJsonAsync<RankResponse>();
|
||||
Assert.NotNull(body);
|
||||
@@ -144,29 +165,29 @@ public class EndpointTests(WikiHostFactory factory) : IClassFixture<WikiHostFact
|
||||
factory.Store.Files["output/2026-09-21-wiki-audit.md"] = "---\ntitle: Wiki audit\ntype: audit\ngenerated: 2026-09-21\n---\n\nreport\n";
|
||||
var req = new HttpRequestMessage(HttpMethod.Post, "/api/v1/sync?force=true");
|
||||
req.Headers.Add("X-Api-Key", WikiHostFactory.ApiKey);
|
||||
Assert.Equal(HttpStatusCode.OK, (await Client.SendAsync(req)).StatusCode);
|
||||
Assert.Equal(HttpStatusCode.OK, (await Keyed.SendAsync(req)).StatusCode);
|
||||
|
||||
var notes = await Client.GetFromJsonAsync<List<NoteRow>>("/api/v1/notes?tag=ceph");
|
||||
var notes = await Keyed.GetFromJsonAsync<List<NoteRow>>("/api/v1/notes?tag=ceph");
|
||||
Assert.Equal("ceph-osd", Assert.Single(notes!).Slug);
|
||||
|
||||
var note = await Client.GetFromJsonAsync<NoteBody>("/api/v1/notes/ceph-osd");
|
||||
var note = await Keyed.GetFromJsonAsync<NoteBody>("/api/v1/notes/ceph-osd");
|
||||
Assert.NotNull(note);
|
||||
Assert.Contains("See [[udm-dns]]", note.Body);
|
||||
Assert.Equal(2, note.Links.Count);
|
||||
Assert.Equal(HttpStatusCode.NotFound, (await Client.GetAsync("/api/v1/notes/nope")).StatusCode);
|
||||
Assert.Equal(HttpStatusCode.NotFound, (await Keyed.GetAsync("/api/v1/notes/nope")).StatusCode);
|
||||
|
||||
var back = await Client.GetFromJsonAsync<NoteBody>("/api/v1/notes/udm-dns");
|
||||
var back = await Keyed.GetFromJsonAsync<NoteBody>("/api/v1/notes/udm-dns");
|
||||
Assert.Equal(2, back!.Backlinks.Count);
|
||||
|
||||
var tags = await Client.GetFromJsonAsync<List<TagRow>>("/api/v1/tags");
|
||||
var tags = await Keyed.GetFromJsonAsync<List<TagRow>>("/api/v1/tags");
|
||||
Assert.Contains(tags!, t => t.Tag == "ceph" && t.Count == 1);
|
||||
|
||||
var raw = await Client.GetFromJsonAsync<List<RawRow>>("/api/v1/raw");
|
||||
var raw = await Keyed.GetFromJsonAsync<List<RawRow>>("/api/v1/raw");
|
||||
Assert.Contains(raw!, r => r.Slug == "2026-09-23-pending");
|
||||
|
||||
var reports = await Client.GetFromJsonAsync<List<ReportRow>>("/api/v1/reports?type=audit");
|
||||
var reports = await Keyed.GetFromJsonAsync<List<ReportRow>>("/api/v1/reports?type=audit");
|
||||
Assert.Equal("Wiki audit", Assert.Single(reports!).Title);
|
||||
Assert.Equal(HttpStatusCode.OK, (await Client.GetAsync("/api/v1/reports/2026-09-21-wiki-audit")).StatusCode);
|
||||
Assert.Equal(HttpStatusCode.OK, (await Keyed.GetAsync("/api/v1/reports/2026-09-21-wiki-audit")).StatusCode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -175,12 +196,12 @@ public class EndpointTests(WikiHostFactory factory) : IClassFixture<WikiHostFact
|
||||
factory.Store.Files["wiki/udm-dns.md"] = "---\ntitle: UDM intercepts DNS\ntype: wiki\ntags: [dns]\n---\n\n# UDM intercepts DNS\n\nThe UDM answers every query.\n";
|
||||
var req = new HttpRequestMessage(HttpMethod.Post, "/api/v1/sync?force=true");
|
||||
req.Headers.Add("X-Api-Key", WikiHostFactory.ApiKey);
|
||||
await Client.SendAsync(req);
|
||||
await Keyed.SendAsync(req);
|
||||
|
||||
var bad = await Client.PostAsJsonAsync("/api/v1/ask", new { queries = Array.Empty<string>() });
|
||||
var bad = await Keyed.PostAsJsonAsync("/api/v1/ask", new { queries = Array.Empty<string>() });
|
||||
Assert.Equal(HttpStatusCode.BadRequest, bad.StatusCode);
|
||||
|
||||
var r = await Client.PostAsJsonAsync("/api/v1/ask", new
|
||||
var r = await Keyed.PostAsJsonAsync("/api/v1/ask", new
|
||||
{
|
||||
queries = new[] { "udm dns" },
|
||||
k = 3,
|
||||
@@ -247,13 +268,13 @@ public class EndpointTests(WikiHostFactory factory) : IClassFixture<WikiHostFact
|
||||
factory.Store.Files["wiki/udm-dns.md"] = "---\ntitle: UDM intercepts DNS\ntype: wiki\ntags: [dns]\n---\n\n# UDM intercepts DNS\n\nThe UDM answers every query.\n";
|
||||
var req = new HttpRequestMessage(HttpMethod.Post, "/api/v1/sync?force=true");
|
||||
req.Headers.Add("X-Api-Key", WikiHostFactory.ApiKey);
|
||||
await Client.SendAsync(req);
|
||||
await Keyed.SendAsync(req);
|
||||
|
||||
Assert.Equal(HttpStatusCode.BadRequest, (await Client.PostAsJsonAsync("/api/v1/questions", new { question = " " })).StatusCode);
|
||||
Assert.Equal(HttpStatusCode.BadRequest, (await Keyed.PostAsJsonAsync("/api/v1/questions", new { question = " " })).StatusCode);
|
||||
var unauthorizedSave = await Client.PostAsJsonAsync("/api/v1/questions", new { question = "udm dns", save = true });
|
||||
Assert.Equal(HttpStatusCode.Unauthorized, unauthorizedSave.StatusCode);
|
||||
|
||||
var r = await Client.PostAsJsonAsync("/api/v1/questions", new { question = "udm dns", k = 3 });
|
||||
var r = await Keyed.PostAsJsonAsync("/api/v1/questions", new { question = "udm dns", k = 3 });
|
||||
Assert.Equal(HttpStatusCode.OK, r.StatusCode);
|
||||
var body = await r.Content.ReadFromJsonAsync<QuestionResponse>();
|
||||
Assert.NotNull(body);
|
||||
@@ -264,7 +285,7 @@ public class EndpointTests(WikiHostFactory factory) : IClassFixture<WikiHostFact
|
||||
|
||||
var save = new HttpRequestMessage(HttpMethod.Post, "/api/v1/questions") { Content = JsonContent.Create(new { question = "udm dns", k = 3, save = true }) };
|
||||
save.Headers.Add("X-Api-Key", WikiHostFactory.ApiKey);
|
||||
var saved = await (await Client.SendAsync(save)).Content.ReadFromJsonAsync<QuestionResponse>();
|
||||
var saved = await (await Keyed.SendAsync(save)).Content.ReadFromJsonAsync<QuestionResponse>();
|
||||
Assert.NotNull(saved?.Saved);
|
||||
Assert.StartsWith("output/", saved.Saved.Path);
|
||||
Assert.Contains("type: output", factory.Store.Files[saved.Saved.Path]);
|
||||
@@ -313,7 +334,8 @@ public class EndpointTests(WikiHostFactory factory) : IClassFixture<WikiHostFact
|
||||
[Fact]
|
||||
public async Task StatusListsAdaptersAndIndexes()
|
||||
{
|
||||
var status = await Client.GetFromJsonAsync<StatusResponse>("/api/v1/status");
|
||||
var client = Keyed;
|
||||
var status = await client.GetFromJsonAsync<StatusResponse>("/api/v1/status");
|
||||
Assert.NotNull(status);
|
||||
Assert.Contains(status.Indexes, i => i.ModelId == "bge-m3" && i.Active);
|
||||
Assert.NotEmpty(status.Adapters);
|
||||
|
||||
@@ -0,0 +1,95 @@
|
||||
using System.Net;
|
||||
using System.Security.Claims;
|
||||
using System.Security.Cryptography;
|
||||
using System.Text;
|
||||
using System.Text.Json;
|
||||
using Microsoft.AspNetCore.Authentication.OpenIdConnect;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.IdentityModel.JsonWebTokens;
|
||||
using Microsoft.IdentityModel.Protocols;
|
||||
using Microsoft.IdentityModel.Protocols.OpenIdConnect;
|
||||
using Microsoft.IdentityModel.Tokens;
|
||||
|
||||
namespace Wiki.Host.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// An in-process OpenID provider: static discovery, a token endpoint that issues RS256 ID tokens and a
|
||||
/// userinfo endpoint, reached through the OIDC handler's backchannel. Records what the wiki sent.
|
||||
/// </summary>
|
||||
public sealed class FakeIdp : HttpMessageHandler
|
||||
{
|
||||
public const string Issuer = "https://idp.test/application/o/wiki/";
|
||||
public const string ClientId = "wiki";
|
||||
private const string AccessToken = "access-token";
|
||||
|
||||
private readonly RsaSecurityKey _key = new(RSA.Create(2048)) { KeyId = "idp-key" };
|
||||
|
||||
/// <summary>The user the next token response is issued for.</summary>
|
||||
public string Username { get; set; } = "alice";
|
||||
/// <summary>Signs the ID token with a key the wiki does not trust when set.</summary>
|
||||
public bool SignWithForeignKey { get; set; }
|
||||
/// <summary>The nonce the wiki put into the authorize request; the ID token must echo it.</summary>
|
||||
public string? Nonce { get; set; }
|
||||
public List<Dictionary<string, string>> TokenRequests { get; } = [];
|
||||
|
||||
public OpenIdConnectConfiguration Configuration { get; }
|
||||
|
||||
public FakeIdp()
|
||||
{
|
||||
Configuration = new OpenIdConnectConfiguration
|
||||
{
|
||||
Issuer = Issuer,
|
||||
AuthorizationEndpoint = "https://idp.test/application/o/authorize/",
|
||||
TokenEndpoint = "https://idp.test/application/o/token/",
|
||||
UserInfoEndpoint = "https://idp.test/application/o/userinfo/",
|
||||
};
|
||||
Configuration.SigningKeys.Add(_key);
|
||||
Configuration.CodeChallengeMethodsSupported.Add("S256");
|
||||
}
|
||||
|
||||
public void Register(IServiceCollection services) =>
|
||||
services.PostConfigure<OpenIdConnectOptions>(OpenIdConnectDefaults.AuthenticationScheme, o =>
|
||||
{
|
||||
o.Configuration = Configuration;
|
||||
o.ConfigurationManager = new StaticConfigurationManager<OpenIdConnectConfiguration>(Configuration);
|
||||
o.Backchannel = new HttpClient(this);
|
||||
});
|
||||
|
||||
protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken ct)
|
||||
{
|
||||
var url = request.RequestUri!.ToString();
|
||||
if (url == Configuration.TokenEndpoint)
|
||||
{
|
||||
var form = await request.Content!.ReadAsStringAsync(ct);
|
||||
TokenRequests.Add(form.Split('&').Select(kv => kv.Split('=', 2))
|
||||
.ToDictionary(kv => Uri.UnescapeDataString(kv[0]), kv => Uri.UnescapeDataString(kv[1].Replace('+', ' '))));
|
||||
return Json(new { access_token = AccessToken, token_type = "Bearer", expires_in = 300, id_token = IdToken() });
|
||||
}
|
||||
if (url == Configuration.UserInfoEndpoint)
|
||||
return Json(new { sub = Sub, preferred_username = Username });
|
||||
return new HttpResponseMessage(HttpStatusCode.NotFound);
|
||||
}
|
||||
|
||||
private string Sub => Convert.ToHexStringLower(SHA256.HashData(Encoding.UTF8.GetBytes(Username)));
|
||||
|
||||
private string IdToken()
|
||||
{
|
||||
var key = SignWithForeignKey ? new RsaSecurityKey(RSA.Create(2048)) { KeyId = "idp-key" } : _key;
|
||||
var atHash = Base64UrlEncoder.Encode(SHA256.HashData(Encoding.ASCII.GetBytes(AccessToken))[..16]);
|
||||
var now = DateTime.UtcNow;
|
||||
return new JsonWebTokenHandler().CreateToken(new SecurityTokenDescriptor
|
||||
{
|
||||
Issuer = Issuer,
|
||||
Audience = ClientId,
|
||||
IssuedAt = now,
|
||||
NotBefore = now,
|
||||
Expires = now.AddMinutes(5),
|
||||
Subject = new ClaimsIdentity([new Claim("sub", Sub), new Claim("preferred_username", Username)]),
|
||||
Claims = new Dictionary<string, object> { ["nonce"] = Nonce ?? string.Empty, ["at_hash"] = atHash },
|
||||
SigningCredentials = new SigningCredentials(key, SecurityAlgorithms.RsaSha256),
|
||||
});
|
||||
}
|
||||
|
||||
private static HttpResponseMessage Json(object body) =>
|
||||
new(HttpStatusCode.OK) { Content = new StringContent(JsonSerializer.Serialize(body), Encoding.UTF8, "application/json") };
|
||||
}
|
||||
@@ -41,6 +41,17 @@ public sealed class FakeStore : INoteStore
|
||||
return Task.FromResult(new StoreWrite(path, "c"));
|
||||
}
|
||||
|
||||
public Dictionary<string, byte[]> Binaries { get; } = new();
|
||||
|
||||
public Task<StoreWrite> CreateAsync(string path, byte[] content, string message, CancellationToken ct)
|
||||
{
|
||||
if (Files.ContainsKey(path) || Binaries.ContainsKey(path)) throw new DocumentExistsException(path);
|
||||
Binaries[path] = content;
|
||||
return Task.FromResult(new StoreWrite(path, Convert.ToHexStringLower(SHA1.HashData(content))));
|
||||
}
|
||||
|
||||
public Uri? BrowseUrl(string path) => null;
|
||||
|
||||
public Task<AdapterHealth> CheckAsync(CancellationToken ct) => Task.FromResult(new AdapterHealth(Name, true, null, TimeSpan.Zero, DateTimeOffset.UnixEpoch));
|
||||
}
|
||||
|
||||
@@ -105,3 +116,18 @@ public sealed class FakeEmbeddings(string modelId) : IEmbeddingModel
|
||||
|
||||
public Task<AdapterHealth> CheckAsync(CancellationToken ct) => Task.FromResult(new AdapterHealth(Name, true, null, TimeSpan.Zero, DateTimeOffset.UnixEpoch));
|
||||
}
|
||||
|
||||
public sealed class FakeFactDrafter : IFactDrafter
|
||||
{
|
||||
public string ModelId => "fake-drafter";
|
||||
public string Name => "fake-drafter";
|
||||
|
||||
public Task<FactDraft> DraftAsync(FactStatement statement, CancellationToken ct)
|
||||
{
|
||||
var words = statement.Text.Split(' ', StringSplitOptions.RemoveEmptyEntries);
|
||||
var title = string.Join(' ', words.Take(5));
|
||||
return Task.FromResult(new FactDraft(title, statement.Text, ["fake"]));
|
||||
}
|
||||
|
||||
public Task<AdapterHealth> CheckAsync(CancellationToken ct) => Task.FromResult(new AdapterHealth(Name, true, null, TimeSpan.Zero, DateTimeOffset.UnixEpoch));
|
||||
}
|
||||
|
||||
@@ -0,0 +1,126 @@
|
||||
using System.Net;
|
||||
using System.Net.Http.Json;
|
||||
using Microsoft.AspNetCore.Mvc.Testing;
|
||||
using Microsoft.AspNetCore.WebUtilities;
|
||||
|
||||
namespace Wiki.Host.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Human sign-in: OIDC against the provider, nothing else. A browser without a session is sent to the
|
||||
/// provider, API callers get a status, and no request header names a user.
|
||||
/// </summary>
|
||||
public class OidcTests(WikiHostFactory factory) : IClassFixture<WikiHostFactory>
|
||||
{
|
||||
private HttpClient Browser() => factory.CreateClient(new WebApplicationFactoryClientOptions
|
||||
{
|
||||
AllowAutoRedirect = false,
|
||||
HandleCookies = true,
|
||||
BaseAddress = new Uri("https://localhost"),
|
||||
});
|
||||
|
||||
[Fact]
|
||||
public async Task AnonymousBrowserIsSentToTheProviderWithPkce()
|
||||
{
|
||||
var page = await Browser().GetAsync("/facts");
|
||||
Assert.Equal(HttpStatusCode.Redirect, page.StatusCode);
|
||||
var location = page.Headers.Location!;
|
||||
Assert.StartsWith(factory.Idp.Configuration.AuthorizationEndpoint, location.ToString());
|
||||
var query = QueryHelpers.ParseQuery(location.Query);
|
||||
Assert.Equal(FakeIdp.ClientId, query["client_id"]);
|
||||
Assert.Equal("code", query["response_type"]);
|
||||
Assert.Equal("S256", query["code_challenge_method"]);
|
||||
Assert.False(string.IsNullOrEmpty(query["code_challenge"]));
|
||||
Assert.Equal("https://localhost/signin-oidc", query["redirect_uri"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task TheRedirectUriIsHttpsOnTheRequestHostEvenBehindTlsTermination()
|
||||
{
|
||||
// Traefik ends TLS and forwards plain http with the original Host.
|
||||
var behindProxy = factory.CreateClient(new WebApplicationFactoryClientOptions
|
||||
{
|
||||
AllowAutoRedirect = false,
|
||||
BaseAddress = new Uri("http://wiki.test"),
|
||||
});
|
||||
var page = await behindProxy.GetAsync("/facts");
|
||||
var query = QueryHelpers.ParseQuery(page.Headers.Location!.Query);
|
||||
Assert.Equal("https://wiki.test/signin-oidc", query["redirect_uri"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task TheAuthentikHeaderSignsNobodyIn()
|
||||
{
|
||||
var browser = Browser();
|
||||
var page = new HttpRequestMessage(HttpMethod.Get, "/facts");
|
||||
page.Headers.Add("X-Authentik-Username", "k.renner");
|
||||
Assert.Equal(HttpStatusCode.Redirect, (await browser.SendAsync(page)).StatusCode);
|
||||
|
||||
var api = new HttpRequestMessage(HttpMethod.Get, "/api/v1/notes");
|
||||
api.Headers.Add("X-Authentik-Username", "k.renner");
|
||||
Assert.Equal(HttpStatusCode.Unauthorized, (await browser.SendAsync(api)).StatusCode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ApiWithoutCredentialsAnswers401NotARedirect()
|
||||
{
|
||||
var browser = Browser();
|
||||
Assert.Equal(HttpStatusCode.Unauthorized, (await browser.GetAsync("/api/v1/notes")).StatusCode);
|
||||
Assert.Equal(HttpStatusCode.Unauthorized, (await browser.GetAsync("/api/v1/status")).StatusCode);
|
||||
Assert.Equal(HttpStatusCode.Unauthorized, (await browser.PostAsJsonAsync("/rank", new { query = "x" })).StatusCode);
|
||||
Assert.Equal(HttpStatusCode.Unauthorized, (await browser.PostAsJsonAsync("/api/v1/questions", new { question = "x" })).StatusCode);
|
||||
Assert.Equal(HttpStatusCode.OK, (await browser.GetAsync("/health")).StatusCode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SignInTakesTheUsernameFromTheVerifiedIdToken()
|
||||
{
|
||||
var browser = Browser();
|
||||
var (code, redirect) = await SignInAsync(browser, "alice");
|
||||
Assert.Equal(HttpStatusCode.Redirect, code);
|
||||
Assert.Equal("/facts", redirect);
|
||||
|
||||
var token = factory.Idp.TokenRequests[^1];
|
||||
Assert.Equal("https://localhost/signin-oidc", token["redirect_uri"]);
|
||||
Assert.True(token.ContainsKey("code_verifier"));
|
||||
Assert.False(token.ContainsKey("client_secret"));
|
||||
|
||||
var page = await browser.GetAsync("/facts");
|
||||
Assert.Equal(HttpStatusCode.OK, page.StatusCode);
|
||||
var html = await page.Content.ReadAsStringAsync();
|
||||
Assert.Contains("captured_by: alice", html);
|
||||
|
||||
// A signed-in human may read the API with the session cookie.
|
||||
Assert.Equal(HttpStatusCode.OK, (await browser.GetAsync("/api/v1/tags")).StatusCode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task AnIdTokenSignedByAnotherKeyIsRejected()
|
||||
{
|
||||
factory.Idp.SignWithForeignKey = true;
|
||||
try
|
||||
{
|
||||
var browser = Browser();
|
||||
await SignInAsync(browser, "mallory");
|
||||
Assert.Equal(HttpStatusCode.Redirect, (await browser.GetAsync("/facts")).StatusCode);
|
||||
}
|
||||
finally
|
||||
{
|
||||
factory.Idp.SignWithForeignKey = false;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Follows the code flow by hand: challenge, then the provider's form_post back to the callback.</summary>
|
||||
private async Task<(HttpStatusCode, string?)> SignInAsync(HttpClient browser, string username)
|
||||
{
|
||||
var challenge = await browser.GetAsync("/facts");
|
||||
var query = QueryHelpers.ParseQuery(challenge.Headers.Location!.Query);
|
||||
factory.Idp.Username = username;
|
||||
factory.Idp.Nonce = query["nonce"];
|
||||
var callback = await browser.PostAsync("/signin-oidc", new FormUrlEncodedContent(new Dictionary<string, string>
|
||||
{
|
||||
["code"] = "the-code",
|
||||
["state"] = query["state"].ToString(),
|
||||
}));
|
||||
return (callback.StatusCode, callback.Headers.Location?.OriginalString);
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user