homelab/kubernetes/PROPOSAL-wiki-web-api.md

Wiki: local web interface and agent API, as a modular .NET solution

Goal

A web interface for the Notes wiki that runs entirely in the cluster, and an HTTP API through which agents ask the wiki typed questions and deposit facts. Built as one C# solution with a MudBlazor front end, and built so that every external dependency is replaceable: where the notes live, which model embeds them, what judges a typed question, what runs an agent, how the vector index is stored. The first adapters are what the homelab uses today. The point of the design is that the next one is a project and a config line, not a rewrite.

Where we are

• wiki-index (Python, ns ai) syncs wiki/ from k.renner/notes by blob SHA, embeds through wiki-embed (bge-m3 on llama-server), answers POST /rank. It keeps slug, title and vector, not the text. LAN-only behind Traefik BasicAuth; the Gitea webhook hits the ClusterIP.
• ~/Notes/bin/wiki_rank.py and wiki_ask.py rank through the index, build a document context from a local vault checkout, send it with typed questions to TypeSafe, gate on covered. The wiki-synth agent composes answers from those judgements.
• New knowledge enters as a file in raw/ (facts with provenance, never hypotheses), distilled nightly in CI.
• The house C# project, shelly-manager, sets the conventions: net10.0, an .slnx with src/ and test/, xUnit, MudBlazor 9, Blazor Web App with interactive server rendering, minimal APIs, an Authentik proxy-header authentication handler for humans and an API-key handler for machines, a multi-stage Dockerfile, dotnet test on every push and a kaniko release pipeline to the Gitea registry. Human-facing apps sit behind an Authentik outpost (<app>-gate) with a CNP that admits only that outpost.

Architecture: ports in the core, adapters around it

                 ┌──────────────────────────────────────────┐
                 │  Wiki.Host  (ASP.NET Core + Blazor/Mud)  │  composition root,
                 │  /api/v1, /rank, /webhook, UI pages      │  config, auth
                 └───────────────┬──────────────────────────┘
                                 │ uses
                 ┌───────────────▼──────────────────────────┐
                 │  Wiki.Application  (use cases)           │  Sync, Rank, Ask,
                 │                                          │  DepositFact, Queue-
                 └───────────────┬──────────────────────────┘  Question, Evaluate
                                 │ depends only on
                 ┌───────────────▼──────────────────────────┐
                 │  Wiki.Core  (domain + ports)             │  Note, RawSource,
                 │  no package references                   │  Report, LinkGraph,
                 └──────────────────────────────────────────┘  the interfaces
   implemented by ▲            ▲            ▲            ▲            ▲
   Adapters.Gitea  Adapters.LlamaCpp  Adapters.TypeSafe  Adapters.FileIndex  Adapters.Anthropic
   (later: Database, Ollama/Onnx, local Jev, pgvector/Qdrant, OpenAI-compatible answer models)

Rules that keep it modular, enforced by project references and a test:

• Wiki.Core references nothing. It holds the domain types and the port interfaces. Adapters reference Core only; they never reference each other or the host. Wiki.Host is the only project that knows which adapters exist.
• Every port has a contract test suite in test/Wiki.Contracts that runs against every adapter of that port. A new adapter is done when the suite is green. That is what makes "swap the storage" a safe operation.
• Adapters are selected by configuration and registered by name (keyed services), so several can be active at once. The UI's lab page and the evaluator use that to compare, the API uses the one marked active.
• Nothing in Core or Application knows about HTTP, Gitea, TypeSafe, files or Kubernetes.

Ports

port	what it abstracts	first adapter	examples of the next ones
INoteStore	read notes, raw sources and reports at a head; create a new document (a fact becomes a raw note in the application layer, the store only moves text)	Gitea repository API (k.renner/notes, blob SHA)	git checkout on a volume; a database (CNPG Postgres) with the vault imported; an Obsidian sync folder
IEmbeddingModel	text → vector, with a model id and dimension	llama-server /v1/embeddings (bge-m3 via wiki-embed, the model that won the German-document comparison on accuracy per cost)	the qwen3 embedding variants from that comparison; ONNX in-process; any OpenAI-compatible endpoint
IVectorIndex	store vectors per (model id, document), query top-k	in-memory cosine with a file snapshot per model, as today	pgvector in the database; Qdrant
IChunker	split a document into addressable units for typed questions	paragraph splitter (headings, fences), as in wiki_ask.py	sentence windows; semantic chunks
IJudge	typed questions over a state: Choice, Noul, Score, with probabilities	TypeSafe System One (jev-latest, cloud; the wiki records no locally hosted Jev today)	a locally hosted Jev with the same request shape, should TypeSafe offer one; a constrained-output local LLM adapter, flagged as uncalibrated
IAnswerModel	prose from a question plus evidence: the selected paragraphs with their notes and the judgements; must cite by slug; streaming	Anthropic Messages API (Claude, model from config, Sonnet by default)	any OpenAI-compatible endpoint, including the local llama-server; a different Claude model per question
IEvaluator	measure a retrieval or judge configuration against the labelled set (questions and cited sources in output/)	recall@k, hit rate, MRR, as wiki_rank_eval.py	judge accuracy against hand-labelled paragraphs
IIdentity	who is calling, for attribution of facts and saved answers	Authentik proxy headers (humans), API key (machines), as in shelly-manager	OIDC bearer tokens

Two ports deserve a sentence of scope each. The judge returns judgements with probabilities, never prose, and the UI never presents its output as an answer, only as ranked, quoted evidence. The answer model turns that evidence into words and is allowed nothing else: it sees the selected paragraphs and the judgements, not the whole context, and its output is shown next to the evidence it was given. Both are single model calls inside the request. The service runs no agent: no tool loop, no vault edits. The wiki-synth agent in a Claude session remains the path for artifacts and for multi-question synthesis, and it can call /questions itself.

Solution layout

Wiki.slnx
src/
  Wiki.Core/               domain types, ports, frontmatter and link parsing
  Wiki.Application/        use cases; depends on Core only
  Wiki.Adapters.Gitea/     INoteStore (read at a head, create)
  Wiki.Adapters.LlamaCpp/  IEmbeddingModel (OpenAI-compatible embeddings)
  Wiki.Adapters.TypeSafe/  IJudge (System One client)
  Wiki.Adapters.FileIndex/ IVectorIndex (memory + snapshot per model)
  Wiki.Adapters.Anthropic/ IAnswerModel (Messages API, streaming)
  Wiki.Host/               ASP.NET Core: API, webhook, MudBlazor UI, auth,
                           composition root, appsettings
test/
  Wiki.Core.Tests/         parsing, link graph, chunker
  Wiki.Contracts/          one suite per port, parameterised over adapters
  Wiki.Application.Tests/  use cases with fake ports
  Wiki.Host.Tests/         API surface, auth handlers

Own repository k.renner/wiki, same shape as shelly-manager: Dockerfile, .woodpecker/test.yml on pull requests, .woodpecker/release.yml building the image with kaniko into the Gitea registry on push to main, tagged build.<unix-ts> plus the commit SHA so Flux image automation can follow it. The homelab repo holds only the manifests. claude-bot needs write access to the new repository.

Two things the shelly-manager history settles in advance: the TypeSafe client is written from the documented field contract (endpoint, bearer key, jev-latest, Noul/Choice/Score, 255-option cap, 32k-token state budget), there is no prior C# client to copy; and the UI needs no login page, because identity comes from the outpost's headers, which avoids the Blazor-circuit sign-in constraint shelly-manager ran into.

Configuration

Adapters are declared in appsettings and selected by name; secrets come through environment variables, filled by ExternalSecret, never from the file.

"Wiki": {
  "NoteStore": { "Active": "gitea", "gitea": { "Repository": "k.renner/notes", "Branch": "main" } },
  "Embeddings": {
    "Active": "bge-m3",
    "Models": {
      "bge-m3":   { "Adapter": "LlamaCpp", "Url": "http://wiki-embed.ai.svc:8080", "Model": "bge-m3" },
      "e5-large": { "Adapter": "Ollama",   "Url": "http://ollama.ai.svc:11434",    "Model": "multilingual-e5-large" }
    }
  },
  "Judges": {
    "Active": "typesafe",
    "Models": {
      "typesafe":  { "Adapter": "TypeSafe", "Url": "https://api.typesafe.ai/v1/systemone", "Model": "jev-latest" },
      "jev-local": { "Adapter": "TypeSafe", "Url": "http://jev.ai.svc:8080/v1/systemone",  "Model": "jev" }   // illustration: same adapter, other endpoint
    }
  },
  "Index": { "Active": "file", "file": { "Path": "/data/index" } },
  "AnswerModels": {
    "Active": "claude-sonnet",
    "Models": {
      "claude-sonnet": { "Adapter": "Anthropic", "Model": "claude-sonnet-5" },
      "qwen-local":    { "Adapter": "OpenAI",    "Url": "http://llama-server.ai.svc:8080/v1", "Model": "qwen3.6-35b-a3b" }
    }
  }
}

Every embedding model gets its own index namespace. Switching the active model is a config change after the new index has been built in the background; until then both exist and can be compared. Nothing is re-embedded in place.

The web interface (MudBlazor)

Blazor Web App, interactive server rendering, hosted in Wiki.Host. Static assets vendored into the image, no CDN. Pages:

1. Search: a query box, results ranked by the active index with score and tags; for a phrased question, optionally the paragraph-level hits from the judge with their probabilities, shown as quoted evidence.
2. Note: frontmatter, rendered body with [[wikilinks]] resolved, backlinks, sources linking to their raw file.
3. Browse: by tag, by updated, recently changed, orphans.
4. Answers: a question box. The answer streams in within seconds: rank, judge, compose, all in the request. Below it the evidence: the paragraphs the judge selected with their notes and probabilities, and the covered gate; a low gate is shown as "the wiki does not cover this" instead of an answer. A save button writes the answer to output/ with the wiki-synth frontmatter (question, sources, confidence), so it joins the labelled set. Underneath, the saved answers, newest first, including the ones the wiki-synth agent wrote from a session.
5. Maintenance: the raw queue with ages, the last consolidation and audit reports, index and sync status.
6. Lab: pick any registered embedding model and judge, run the same query through them side by side, and run the evaluator on the labelled set to get recall@k per configuration. This is the page for "play around with different models".
7. Settings: read-only view of the active adapters and their health.

No editing of notes in the browser. The vault stays a git repository edited by the consolidate agent and by hand.

API v1

Reads are authenticated by the hostname's gate (humans) or API key (machines); writes always need an API key so a fact carries its author.

method and path	purpose
GET /api/v1/status	index commit and age, counts, active adapters
GET /api/v1/notes?q=&tag=&limit=	list; q ranks through the active index
GET /api/v1/notes/{slug}	frontmatter, body, links, backlinks, sources
GET /api/v1/tags	tags with counts
POST /api/v1/rank	{question, k, model?} → ranked slugs; also served at /rank for the existing scripts
POST /api/v1/ask	{queries, questions, k, judge?, model?} → context, typed judgements with the answering paragraphs verbatim, covered. For agents and code.
POST /api/v1/questions	{question, k?, judge?, model?, answerModel?, save?} → the answer text with [[slug]] citations, the evidence (paragraphs, probabilities), covered, model and token usage; streamed as server-sent events when requested; with save, the output/ path and commit
POST /api/v1/facts	deposit a raw source: title, source, kind: fact, observed_at, content, tags → path and commit
GET /api/v1/raw	the distill queue with ages
GET /api/v1/reports	output/ documents by type, newest first
POST /api/v1/lab/eval	run the evaluator for a (model, judge) pair
POST /webhook	Gitea push, HMAC as today

POST /api/v1/facts is rejected without source, with a kind other than fact, or with content under a minimum length. The raw rule is enforced by shape: there is no field for a hypothesis. The file gets the vault's provenance frontmatter with captured_by set from the API key's identity.

POST /api/v1/ask is wiki_ask.py moved server-side, so the scripts become thin clients and agents no longer need a vault checkout. Its result is judgements and quotations, never prose.

POST /api/v1/questions is /ask plus one answer-model call, in process: rank the question, build the context, run the default typed questions (answered, where, covered), hand the selected paragraphs and judgements to the answer model, stream the prose back. Below the covered threshold it returns the gap instead of an answer. Time is the sum of the parts, about 0.2 s rank, 1 to 4 s judge, a few seconds of generation. Cost is one model call over a few thousand tokens, a few cents on Sonnet, against a quarter dollar for a full agent run.

Auth and network

• Humans: wiki.renner.uno, LAN-only (no dns.renner.uno/public label), Ingress backend ak-outpost-wiki-gate:9000, a wiki-gate-egress CNP in authentik, the proxy-header handler trusts X-Authentik-* only from that outpost.
• Machines: wiki-index.renner.uno with the API-key handler; the workstation key stays where it is. CI reaches the ClusterIP.
• Secrets (Gitea token, webhook secret, API keys, TypeSafe key, Anthropic key) reach the pod as environment variables from Kubernetes Secrets filled by ExternalSecret from the openbao ClusterSecretStore. The pod itself has no OpenBao access. The Python service fetched its secrets from OpenBao directly through its own Kubernetes-auth role; that role, its policy and the ai → openbao CNP entry are retired with it.
• Egress from the pod: Gitea, wiki-embed, api.typesafe.ai:443, api.anthropic.com:443. Every further adapter that talks to a new endpoint gets its own CNP line, which is the one place adding an adapter touches the cluster.

Health endpoints

Three endpoints with three different meanings, wired to the .NET health checks with tags so that the probes and the humans see the right thing:

endpoint	meaning	probe
GET /health	the process serves HTTP. Always 200 once Kestrel listens; no dependency checks.	liveness
GET /ready	the active note store has synced at least once and the active index is loaded; 503 while the first sync or index build runs. A failed later sync keeps the last good state and stays ready.	readiness, plus a startup probe with enough budget for the first sync (the Python service was killed mid-build when its /health returned 503 during the initial sync)
GET /api/v1/status	the full picture for humans and the Settings page: index commit and age, counts, queue lengths, and one entry per registered adapter with its last check result and latency.	none

Adapter checks (Gitea reachable, embeddings endpoint answering, judge endpoint answering, answer model answering) are reported in /status and on the Settings page and are never part of /ready: a judge outage must not take the reader down. Their failures show up as red entries, not as restarts.

Phases

One PR each, verified before the next. The Python wiki-index keeps running until phase 3 proves parity.

0. Skeleton. Repository, solution, Wiki.Core with the ports and domain types, Wiki.Contracts with the suites, Wiki.Host serving /health, /ready and /api/v1/status. CI: test and release pipelines. Verify: image in the registry, pipeline green, the Deployment's three probes wired and the pod Ready.
1. Parity with the Python service. Gitea, LlamaCpp and FileIndex adapters, the sync use case, /rank and /webhook. Verify: the 40-query recall eval gives the same numbers as the Python service on the same commit; wiki_rank.py --probe-index against the new service passes.
2. Read API and UI. Notes, tags, raw, reports; Search, Note, Browse, Maintenance pages; wiki-gate. Verify: a note renders with resolved links; an unauthenticated request is redirected to Authentik.
3. Ask. Chunker, TypeSafe adapter, /api/v1/ask, scripts as clients. Verify: test_wiki_ask.py passes against the API; a wiki-synth run with the definition unchanged. Then the Python service is retired and the wiki-index hostname points at the new one.
4. Facts. API keys as an ExternalSecret, the deposit-fact use case, /api/v1/facts. Verify: a fact posted from the workstation shows as a claude-bot commit and in the next distill.
5. Questions. Anthropic adapter, /api/v1/questions with streaming, the Answers page with save. Verify: a question answered in under ten seconds with citations that resolve to the shown evidence; a question the wiki does not cover returns the gap; a saved answer appears in output/ with the wiki-synth frontmatter and is picked up by the evaluator.
6. Lab. Evaluator, a second embedding model and a second answer model registered, the Lab page. Verify: two models compared on the labelled set from the UI.

Not in scope

• Editing notes in the browser.
• Replacing Obsidian; the vault stays a git repository and notes-push.path keeps pushing.
• Public exposure; LAN and VPN only.
• Running or triggering agents. The service makes single model calls inside a request; it never runs a tool loop or edits the vault. Distill, consolidate and multi-question synthesis stay Claude processes in CI or in a session; the service shows their inputs and outputs and serves their API calls.