Ingress routes only the /status prefix; /hook and /metrics stay
cluster-internal. No public label, so it is not on the NetBird proxy.
The new CNP lets Traefik reach the pod on 8080.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every phase of a pull request's review is now visible in two places:
- a commit status "review-bot" on the reviewed head: pending while
queued, running or waiting to retry; success once reviewed (with the
verdict) or skipped; error with the reason when the review failed;
warning when new commits superseded the reviewed head. The statuses
are posted one at a time in order, so a later phase never lands before
an earlier one, and never block the queue.
- GET /status and GET /status/{owner}/{repo}/{n}: the in-memory state
of every PR seen since start, with the queue position while queued.
Finished entries are dropped after a day.
Until now a failed review posted nothing, so a PR whose review failed
looked the same as one still waiting.
Also gofmt report_test.go, which the CI gofmt check flags on master.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Chart 4.0.0 (installed 2026-10-08 via the open semver range) replaced
domain/watchedResources/fallthrough/extraZonePlugins with a servers:
list. The old keys were ignored and the chart default ran instead:
k8s_gateway example.com watching Service/HTTPRoute, forward to
/etc/resolv.conf, Service type LoadBalancer. No renner.uno Ingress name
has been answered in-cluster since.
The new values render the same Corefile as the last 3.x release
(helm rev 7): .:1053, k8s_gateway renner.uno on Ingress with
fallthrough renner.uno., cache with serve_stale, forward to the UDM,
Service ClusterIP 10.96.0.11 port 53, runAsUser 1000. Verified with
helm template against chart 4.0.1.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stage 3 of unifying the auth flows. After #1285/#1286 no provider, brand
or flow references the passkey-only objects any more (checked live:
0 providers on either flow; passkey-webauthn-validation and
passkey-passwordless-identification are bound only inside the two
retired flows). They are now declared absent: both flows (their stage
bindings and the policy bindings on those cascade), the
passkey-passwordless-identification stage, the passkey-webauthn-validation
stage and the passkey-user-has-webauthn policy. The shared default
stages they bound (password, MFA validation, login) stay.
The passkey-stepup.yaml key keeps its name, so the blueprint instance
stays the same; it now only carries the OAuth2 providers and the
removals.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A drifted proxy provider was PATCHed with only the drifted fields.
ProxyProviderSerializer.validate (providers/proxy/api.py, 2026.8.3)
checks internal_host against mode from the request attrs alone,
defaulting mode to proxy, so a flow-only PATCH fails with 400
"Internal host cannot be empty when forward auth is disabled." Seen
live after #1285: 8 gates could not move to the new flows and stayed on
their old ones.
The fake Authentik in the tests now applies the same validation, and a
test covers a flow change on an existing gate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stage 2 of unifying the auth flows, after the llama and paperless trial
of stage 1 was confirmed:
- Brand authentik.srv.renner.uno: flow_authentication =
sso-authentication (was NULL, i.e. the slug-first
default-authentication-flow with password + MFA). Every provider
without its own authentication_flow logs in through it, PVE and PBS
included.
- The 7 blueprint OIDC providers (Home Assistant, NetBird,
Audiobookshelf, Immich, Gitea, Grafana, Matrix MAS): authentication
flow unset (brand), authorization implicit consent.
- authentik-gate-sync: default authorization flow implicit consent, via
DEFAULT_AUTHORIZATION_FLOW on the Deployment and in the code; the
per-gate login/authorization annotations are dropped. Paperless keeps
paperless-authorization.
Every flow referenced here already exists, so the gate controller does
not hit the missing-flow removal seen in stage 1. Stage 3 removes the
passkey-only flows once nothing references them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stage 1 of unifying the auth flows into two security levels on one SSO
login:
- sso-authentication: one identification stage that takes the password
inline or a passkey via autofill (webauthn_stage); either factor alone
logs in.
- paperless-authorization: password and passkey on every authorize.
Authorization flows are planned without a pending user (oauth2
authorize.py plan context), so a bare PasswordStage there fails with
"No pending user"; the password is therefore checked inline by an
identification stage, a deny stage rejects an identified user that
differs from the session user, and a passkey validation with
last_auth_threshold 0 follows (no authentik_mfa cookie, so it always
asks).
Trial targets: llama-server (normal level: sso-authentication +
implicit consent) and paperless (high level). The brand, the OIDC
providers and the remaining gates follow once both are confirmed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The remaining backed-up clusters still declared bootstrap.initdb, so a
cold rebuild came up with empty Home Assistant, Matrix, Paperclip and
TRMNL databases. Each now declares bootstrap.recovery from its
ObjectStore (serverName = cluster name). mx-pg drops its initdb
encoding/locale settings: the physical restore keeps the C collation.
RECOVERY.md lists all seven clusters; gf-pg has no ObjectStore and stays
on initdb.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Same change as ak-pg (#1281): im-pg and pprl-pg declared
bootstrap.initdb, so a cold rebuild came up with empty Immich and
Paperless databases despite 30 days of backups. Both now declare
bootstrap.recovery from their ObjectStore (serverName = cluster name).
bootstrap is honoured only when PGDATA is absent, so the live clusters
are unaffected.
RECOVERY.md drops the manual initdb->recovery swap and revert steps:
all three clusters it covers now restore on their own.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The bot has no cluster access, so any PR citing live-cluster evidence
gets a ⚪ Unverified section and could never be approved (e.g. #1281).
Approval now requires only no 🔴/🟡 section, matching the review banner
("approves only when it found nothing red or yellow"); the prompt's
verdict rule is aligned.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ak-pg declared bootstrap.initdb, a leftover of the retired seed-job design,
so a cold rebuild came up with an empty Authentik database: no users,
passkeys, groups or gate membership. It now declares bootstrap.recovery
from ak-pg-store (serverName ak-pg), the stanza RECOVERY.md already
documented and the pattern wp-pg uses.
bootstrap is honoured only when PGDATA is absent, so the live cluster is
unaffected. An empty object store makes bootstrap hang instead of coming
up empty. RECOVERY.md now limits the manual initdb->recovery swap to im-pg
and pprl-pg.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
All wiki interaction now goes through the wiki service API; CI agents that
distill and consolidate by writing the notes repo directly are retired.
Removes .woodpecker/wiki-maintenance.yml and ci/agents/wiki-maintenance/.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drops the per-step resources from the previous commit and removes every
depends_on: [], so Woodpecker runs the kaniko steps serially in file order.
A full rebuild then holds one step pod at a time and cannot exceed the
woodpecker-ci-quota.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A change to build-ci-images.yml starts all 14 kaniko steps at once. With
the LimitRange default of 2 CPU each they asked for ~28 CPU in limits
against the 24-CPU woodpecker-ci-quota, so whichever pods were created last
were rejected ("exceeded quota: woodpecker-ci-quota"): alpine-jq and
cni-plugins in pipeline 13723, vmtest, tofu and secret-rotator in 13726.
Every step now sets its resources from 30 days of Prometheus data (peak
working set / CPU per step):
- ansible, k8s, vmtest, tofu (peaks <=983Mi, <=0.54 cores): 1 CPU, 2Gi
- secret-rotator, review-bot (Go builds, ~2 cores): 2 CPU, 1Gi
- the remaining eight (<=271Mi, <=0.45 cores): 1 CPU, 1Gi
Requests stay at the LimitRange defaults (100m / 256Mi).
Total 16 CPU / 18Gi in limits, leaving room in the quota for a concurrent
PR check. The quota itself is unchanged; its comment no longer claims the
workflow runs 4 pods.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Like any ingress controller, authentik-gate-sync now owns the load-balancer
status of the Ingresses of its class: each pass it copies the address Traefik
reports for the generated route onto the source Ingress, and clears it while
the gate has no route. Previously the migrated Ingresses kept the address
Traefik wrote before their class changed, which Traefik never clears.
Status publishing is non-fatal (logged), so it never holds or removes a gate.
RBAC gains ingresses/status; the change is additive, so the order in which
Flux applies RBAC and the new image does not matter.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
paperless, shelly-manager, zone-configurator, esphome, zigbee2mqtt,
alertmanager, prometheus, wiki, paperclip, navidrome, lidarr and wordpress
now gate through authentik-gate-sync: each source Ingress names its real
backend and carries the old provider's settings as annotations (group,
login-flow, authorization-flow where it was implicit consent, paperless'
skip-path).
Removed per gate: the ExternalName Service, the egress CNP in authentik,
the namespace CNP rule admitting the old outpost, and the blueprint
provider/application/binding/outpost (state: absent, outpost ->
application -> provider). Groups stay declared without users lists;
membership is managed in Authentik.
Per-gate specifics:
- paperless: the docs.renner.uno alias backends the generated
paperless-gate Service (its middleware redirects before the backend); the
auth.renner.uno/outpost.goauthentik.io route is removed, proxy outposts
take their callback on the app host.
- alertmanager / prometheus: source Ingresses renamed to alertmanager /
prometheus; the controller generates the *-gate route Ingresses.
- wordpress-admin: gains a tls block for www.renner.uno (wordpress-tls,
certificate still owned by the public Ingress), so the gate's external
host stays https.
- wiki, paperclip: only the gate peer/rule leaves their shared CNPs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
First group gate on authentik-gate-sync. The Ingress switches to
ingressClassName authentik-gate with its real backend
(home-assistant-vscode:80) and annotations for the group (Home Assistant
Config Editors) and the code-server skip path; the default
passkey-stepup-authorization flow matches the old provider.
Removed: the ExternalName Service, the vscode-gate egress CNP, the
home-assistant-ns ingress rule for the old outpost, and the blueprint
provider/application/binding/outpost (state: absent, outpost ->
application -> provider). The group stays declared in the blueprint but
without a users list: membership is now managed in Authentik, and current
members are kept.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A named targetPort was resolved to a number through the Service's
EndpointSlices, so with no ready endpoints (app restarting, crashlooping)
the gate counted as invalid: its route was removed and its Authentik
objects torn down, to be rebuilt once the pods came back. Cilium accepts
named ports in toPorts and resolves them against the selected pods'
container ports, so the name now goes into both CNPs as-is and the gate no
longer depends on the backend being up. The EndpointSlice read (and its
RBAC) is gone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
First gate on authentik-gate-sync. The Ingress keeps its host, TLS, public
label and cert-manager annotation, switches to ingressClassName
authentik-gate and names its real backend (llama-server:8080); the
passwordless login flow and the manifest/service-worker skip path move to
auth.renner.uno/* annotations. The controller now owns the outpost, the
Traefik route and both CNPs.
Removed: the hand-written ExternalName Service, the llama-gate egress CNP,
the ai-ns ingress rule for the old outpost, and the llama-gate blueprint
objects (state: absent, outpost -> application -> provider).
Expect a short gap on llm.renner.uno at switch-over: Traefik drops the route
when the class changes and serves the new one once the controller's next pass
has started the new outpost.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
build-authentik-gate-sync depended on a separate test step, so it only
started after the other steps had run, and Woodpecker skips not-yet-started
steps once the workflow has a failure: on the #1272 merge build-alpine-jq and
build-cni-plugins failed and the image was never built.
The unit tests now run as a stage of the image build (the final stage copies
the controller out of it), and the step starts with depends_on: [] like the
others.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Proxy providers are filtered by name__iexact (plain name= is ignored and
returned the first provider alphabetically).
- Applications are retrieved by slug, never listed: the list endpoint is
policy-filtered for the token's user and cached per user regardless of
filters. Cleanup finds owned apps through the proxy provider list and
confirms meta_publisher + provider one by one.
- Group gates bind an expression policy (ak_is_group_member by name, the name
as a repr() literal) instead of the group: deleting a group cascades its
bindings away and would open the app, the policy keeps denying. New bindings
are created before stale ones are removed.
- Per-gate errors no longer abort the pass; a gate whose Ingress is invalid
loses its route immediately. Network errors become ApiError.
- Gate names carry a hash of namespace/name: unambiguous and bounded to fit
label values and ak-outpost-<name>.
- Teardown removes only an outpost serving exactly the gate's provider; a
provider orphaned by an interrupted teardown is adopted on recreation.
- Role gains the expression-policy permissions; tests cover the Authentik
reconcile against an in-memory API.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Branch pushes do not build in-house images, so no dated tag exists before merge; the master build pushes :latest alongside it and the setter marker replaces it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds authentik-gate-sync, an in-house controller (ci/images/authentik-gate-sync,
stdlib Python like netbird-service-sync) that implements an `authentik-gate`
IngressClass. An Ingress of that class names its real backend; the controller
turns it into an Authentik proxy gate and wires the upstream ingress provider:
- Authentik: proxy provider, application (meta_publisher marks ownership),
outpost on the local Kubernetes connection, optional group binding.
- App namespace: ExternalName Service <ingress>-gate -> outpost, Traefik
Ingress <ingress>-gate (source hosts/paths/TLS/labels/annotations, backend
swapped, cert-manager annotations dropped so only the source issues), and a
CNP admitting only this outpost to the backend pods.
- authentik namespace: CNP letting the outpost reach only those pods.
Annotations: auth.renner.uno/group (members managed in Authentik; none = any
user), /login-flow, /authorization-flow (default passkey-stepup-authorization),
/skip-path. Fail closed: the Traefik route is applied last and removed when the
gate's config is invalid (missing group/flow/backend); transient API errors
leave the gate as the last good pass left it.
The API token is zero-touch: an ESO Password generator (CreatedOnce) feeds
both the controller and AUTHENTIK_GATE_SYNC_TOKEN on server/worker, which the
new gate-sync blueprint sets as the key of a non-expiring token for a
service account whose role covers only gate objects.
No existing gate moves yet; llama is migrated in a follow-up once this runs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A second build on the same day reused the tag, so the Flux image policy saw nothing new and the
Deployment kept the old digest (#1270 built but did not roll out). Tags are now YYYY-MM-DD-HHMM and
the policy accepts both forms, newest wins.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The agent can read its earlier review but nothing told it to. A re-review now opens with the
state of every earlier point, judges the new commits in full, and leaves unchanged code as it
was judged: no new suggestions, unverified points or nits about it, and no requests for more
tests or hardening of code that did not change.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The workflow-level path filter lacked review-bot/**, so a push touching only that directory
created no pipeline and the step-level filter was never evaluated. #1258 and #1268 merged
without an image build.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A 502 after ten minutes ended a review of homeassistant#382 with nothing posted. 502, 503 and
504 are retried like the 429, with the same waits and the same retry budget. The metrics are
now retries_total and retry_giveups_total.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The guessed wait (RATE_LIMIT_WAIT x retry) was uncapped while Retry-After was capped, so a large
RATE_LIMIT_WAIT could outlast the review budget. Both now share one constant.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A Blocking section that said 'none' but held a sentence turned the review into
REQUEST_CHANGES. The agent now ends its report with 'Verdict: request-changes|comment|approve';
no verdict means a plain comment, and an approval is downgraded when the report still holds
red, yellow or unverified sections.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Drops the text matching on tool results. The observed limit is HTTP 429 on starting a
conversation, which is handled by status; guessing at tool output text cannot be made
correct and dropped reviews when it was wrong.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A review whose head moved during the conversation is not posted; the PR is reviewed
again at the new head. A request that arrives while a PR is being reviewed is no
longer lost: the PR is reviewed once more when the running review ends. Tool errors
count as rate limits only with a limit phrase (429 as a status, too many requests,
rate limit reached/exceeded), so file names like rate_limit.go do not match. The whole
review is capped at one conversation timeout plus the longest waits.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A tool result that merely contains the words 'rate limit' (a file about rate limits)
no longer marks the answer incomplete. Only short error messages naming a limit
(rate limit/rate_limit/ratelimit, too many requests, 429) count. Each conversation
gets the full review timeout, the waits between retries do not eat into it, and a
give-up is counted in rate_limit_giveups_total. Tests pin the first retry delay,
the Retry-After cap and the false positive.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Replaces the hand-rolled retry loop with a failsafe-go retry policy: handles 429 and
rate-limited tool results only, delay from Retry-After (capped at 5m) else linear
backoff, context-aware.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Mistral documents requests per second (concurrent requests), tokens per minute and tokens per month, and X-RateLimit-Remaining, but no Retry-After and nothing about connector tool calls. When a 429 carries Retry-After it is used instead of the guessed wait (capped at five minutes); the Retry-After and rate limit headers of every 429 are logged.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A conversation that got HTTP 429, or whose tool calls failed on the rate limit, is asked again after a growing wait (RATE_LIMIT_RETRIES, RATE_LIMIT_WAIT) instead of failing or posting a review built on half the files. The default is one worker, and the review timeout covers the waits.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The review-bot service reviews pull requests now. The pipeline starts only when triggered by hand and stays as a reference.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Namespace, Deployment (one replica, scratch image, health and readiness probes), Service, ExternalSecret (webhook secret from apps/review-bot, Mistral key from apps/wiki, the rotated review-bot token), CiliumNetworkPolicies (Gitea in, Traefik and api.mistral.ai out), the namespace in the ClusterSecretStore allowlist, Gitea's egress to the service and its webhook host list, and Flux image automation for the review-bot image.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Gitea's pull_request webhook (HMAC-checked) queues the review at once; a periodic sweep over the configured repositories catches lost deliveries. The service keeps the listed Mistral agent in sync with the embedded prompt, asks it for a review, refuses answers that called non-read-only tools, cleans the report, derives the review event and posts as the review bot. State is the pr-review:sha marker in the bot's own review. /healthz, /readyz and /metrics. Standard library only, tests against fake Gitea and Mistral servers, built as a scratch image by build-ci-images.yml.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
run.sh asks a Mistral agent (mistral-large-4-0) through the Conversations API. The agent reads the repo and the PR through the read-only Gitea connector, so the review step no longer holds a Claude token or a local checkout. The Claude CLI invocation, the model picker and the context clones stay in run.sh as comments. The answer is cut at the first heading and empty or None-only sections are dropped before the review event is derived. A tool call that is not read-only fails the run and nothing is posted. A failed request now fails the pipeline.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
gitea-mcp advertises its -H URL as the authorization server in the OAuth protected-resource metadata, so clients doing discovery were sent to http://gitea.gitea.svc.cluster.local:3000. Switch -H to https://gitea.renner.uno; egress now goes to Traefik (as woodpecker-server and wiki already do) and the gitea:3000 grant on both sides is removed. Tool results' clone_url/html_url become public URLs too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gitea-mcp always writes a lumberjack log (100MB rotation, 10 backups) and falls back to /tmp on a read-only root. A 16Mi in-memory emptyDir would fill and get the pod evicted; the log also counted against the memory limit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Runs gitea-mcp in -oauth mode with no server token: tokenless /mcp requests get 401 instead of reaching Gitea anonymously, and a plain PAT in the Authorization header is passed through to Gitea. Published on the NetBird proxy via netbird.renner.uno/public; LAN DNS stays on the Traefik VIP.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Master build of #1238 (pipeline 13250). Activates the proxy-only label, so
the gitea Ingress is published on the NetBird proxy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dns.renner.uno/public publishes a host on the NetBird proxy AND drops its
UDM record (external-dns-unifi labelFilter). For gitea that sends LAN SSH
to the proxy host's own sshd and node image pulls over the WAN, since the
hairpin covers only 80/443 and not the node VLAN.
netbird-service-sync now also selects netbird.renner.uno/public, which
external-dns-unifi does not filter on, so the host is published on the
proxy while the LAN record on the Traefik VIP stays. The gitea Ingress
switches to the new label; it takes effect once the master build of this
image is pinned in the deployment.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Label the gitea Ingress dns.renner.uno/public so netbird-service-sync
publishes git.renner.uno and gitea.renner.uno on the NetBird proxy. The
gitea-ssh IngressRouteTCP stays unlabeled and LAN-only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Embeds a labelled Paperless corpus and scores nearest-neighbour correspondent and document-type accuracy against ceiling and floor. Standard library only.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Point PAPERLESS_AI_LLM_* at api.mistral.ai (mistral-small-latest), key from paperless-secrets minstral-token. Embeddings stay on llama-embed; an explicit embedding key stops the Mistral token falling back to it.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
- vm-verify: `kopia snapshot verify` read the loop's stdin, so anything
reading stdin would eat the remaining plan entries; it now reads
/dev/null. KOPIA_CHECK_FOR_UPDATES=false.
- cloud-init failing on the VM no longer waits out SSH_WAIT_S: ssh exit
codes other than 255 mean cloud-init itself failed, so the orchestrator
logs `cloud-init status --long` and gives up at once.
- A failure staging files on the VM is no longer fatal: the run still
prints the per-entry summary (no result from VM) and exits 1.
- A server create that times out after the server exists no longer
blocks the fallback type with a duplicate name: labelled servers are
deleted between attempts.
- cloud-config is rendered as JSON (a YAML subset) with jq, so the
multi-line host key is escaped by jq and the test can parse it.
- Tests: secrets parsing (with/without spaces around =), user-data
structure, every server type unavailable, retry after a leftover
server, push failure, cloud-init failure.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- AUTH_INSECURE_* is no longer excluded. #527 is done: every CephX key
is aes256k since 2026-09-23 (#931, #934, #935) and all six
AUTH_INSECURE_* checks report 0, so a new one is a real regression.
- The description now only says what to do: BLUESTORE_SLOW_OP_ALERT
outlives the stall by 24h; find the OSD in `ceph health detail` and
check that drive's SMART data. No history in the annotation.
- Comment: drop the AUTH_INSECURE paragraph and the restore-test note;
the "80-87% endurance used" claim was wrong for the OSDs (nvme2 at
2-3% per smartctl_device_percentage_used, the 83-91% drives are the
nvme1 EPHEMERAL disks), so it goes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The alert told the reader that BLUESTORE_SLOW_OP_ALERT stalls between
08:00-14:00Z on the 1st are expected because of the monthly Velero
restore-test sweep. restore-test was removed in #1233 (replaced by
backup-verify, which writes nothing to the OSDs), so a stall in that
window is no longer explained by a scheduled job and must not be
waved through. Expression, for and labels are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
backup-verify passed its live runs on 2026-10-10:
- apps only: PASS=2, 61 s, VM torn down
- negative (forged snapshot IDs): FAIL=2 "snapshot not in repository",
Job Failed, VM torn down
- abort (Job deleted after ssh ready): TERM handled in 3 s, server,
firewall and key gone within 16 s; next run swept nothing
- full: PASS=39 FAIL=0 SKIP=2, 3688 s, cx23, ~410 GB verified, nothing
left behind
The CronJob is unsuspended. restore-test (CronJob, RBAC, script,
ConfigMap) and its Prometheus :9090 ingress rule in monitoring-ns go;
Flux prune removes them and the failed restore-test-29847360 Job with
its CronJob, which clears the open KubeJobFailed (velero) alert.
RECOVERY.md describes backup-verify instead of the OSD capacity gate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The k8s backend defaults to IfNotPresent, so a node kept serving a stale
:latest (Claude Code 2.1.218) after the refresh pipeline pushed a new one.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
- TERM while waiting for cloud-init over SSH was deferred until the
probe returned; the probe now runs in the background and is waited
on, so a Job deleted during VM bring-up still tears down in time.
- /run/verify is its own 1500m tmpfs (cloud-init runcmd); the Kopia
cache no longer competes with the ~10% /run of a 4 GB VM.
- Summary rows carry backup name and snapshot ID; a final line reports
elapsed seconds and the server type used (spec section 5).
- kopia verify runs with --file-parallelism=8 (default NumCPU=2 is
latency-bound against S3).
- /work emptyDir 32Mi -> 128Mi (kubectl cache + Velero object dumps),
memory limit 256Mi -> 384Mi to cover it; overflow would evict the pod
without the teardown trap.
Each fix has a test that failed first (term during ssh wait, tmpfs
mount in user-data, summary row/elapsed line, verify parallelism).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Runs kubernetes/infrastructure/velero/backup-verify/tests/run.sh in the
ci-backup-verify image (bash, jq, ssh-keygen), offline.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- CronJob velero/backup-verify, monthly 0 8 1 * * UTC, suspended until
the live runs pass; 4 h deadline, 120 s grace for teardown; runs as
65532 with a read-only root fs, tmpfs /work.
- Role limited to get/list schedules, backups, datauploads in velero.
- ExternalSecret hcloud-backup-verify from OpenBao
infrastructure/hcloud-backup-verify.
- CiliumNetworkPolicy: apiserver, DNS, api.hetzner.cloud:443, tcp/22.
- Scripts packaged as ConfigMap backup-verify-scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
backup-verify.sh: sweep leftovers by label, build the plan from Velero,
create ssh-key/firewall/server (cx23, fallback cax11) with a pre-made
host key pinned in known_hosts, push check.jq + vm-verify.sh, stream
secrets on stdin, summarize, and delete everything labelled
managed-by=backup-verify on EXIT. The VM call runs in the background
and is waited on so a TERM (Job deadline/deletion) tears down at once.
FORGE_SNAPSHOT_ID replaces planned snapshot IDs for the live negative
test.
Tests stub hcloud/kubectl/ssh: happy path, type fallback, VM FAIL,
forged IDs, empty plan, and TERM mid-run (fails if the VM call runs in
the foreground).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
vm-verify.sh installs kopia 0.16.0 (sha256-pinned, x64/arm64), connects
read-only per namespace (kopia/<ns>/), runs check.jq and then
`snapshot verify --verify-files-percent=100` per snapshot root. Secrets
come on stdin only. Exactly one RESULT line per entry, then DONE;
connect or list failures turn every entry of that namespace into FAIL.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
check.jq matches planned DataUploads against kopia snapshot list output:
missing snapshot, incomplete snapshot, entries failed at backup time and
size mismatch against DataUpload totalBytes are FAIL before any verify.
summary.jq turns a VERIFY entry without a VM result into FAIL and never
reports success when nothing was verified.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
plan.jq turns Schedules, Backups and DataUploads into VERIFY/FAIL/SKIP
entries: newest finished backup per schedule (an in-flight backup does
not count), PartiallyFailed/Failed -> FAIL, no DataUploads -> SKIP.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
OpenProject Community hides every OIDC provider without an Enterprise
token (auth_plugin.rb, EnterpriseToken.allows_to?(:sso_auth_providers)),
so it cannot sign in through Authentik.
Removes the app, its Flux Kustomization, namespace, network policy,
op-pg PodMonitor, blackbox probe, Velero schedule and ClusterSecretStore
entry. Authentik drops the OIDC secret env and marks the OpenProject
application, provider and group absent. Gitea no longer allows
openproject.renner.uno as a webhook host.
Refs: REN-1363
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Alpine with bash, jq, kubectl, openssh and the hcloud CLI (v1.69.0,
copied from the official image). A verify user (uid 65532) exists
because the OpenSSH client refuses to run as a UID without a passwd
entry. Built by build-ci-images.yml with a dated tag; Flux image
automation tracks the newest date.
First step of docs/superpowers/plans/2026-10-10-backup-verify-cloud-vm.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Implementation plan for docs/superpowers/specs/2026-10-10-backup-verify-cloud-vm-design.md:
PR A image, PR B jq filters + scripts + manifests (CronJob suspended)
with offline fixture/stub tests, live runs, PR C cut-over.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Design for replacing the monthly Velero restore-test: an in-cluster
CronJob creates a throwaway VM in fsn1, which runs kopia snapshot verify
(100% of files, read-only) against every latest DataUpload snapshot and
checks sizes against DataUpload totalBytes. No restored data is written
to the Ceph OSDs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The event is derived by the pipeline from the report's severity sections:
Blocking -> REQUEST_CHANGES; none of Blocking/Suggestions/Unverified on a
non-empty diff -> APPROVE; otherwise COMMENT.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Drop the complexity triage (it downgraded most PRs to a small model) and pick
the model via scripts/ci/pick-model.sh from the live /v1/models catalog,
steered to the strongest and newest. Falls back to the CLI default.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Move the review logic out of .woodpecker/pr-review.yml into
ci/agents/pr-review/run.sh; the per-repo file becomes a logic-free stub that
clones homelab@master and runs it, so the four copies cannot drift.
Prompt: unverified suspicions must be checked (checkout + docs) and are never
blocking. Review agent gets WebFetch/WebSearch.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Gitea 1.27 deprecates [webhook] ALLOWED_HOST_LIST (fallback removed in
v28.0.0). The on-PVC app.ini had it set to `*`. Track the replacement
in git and narrow it to the hosts the configured webhooks call:
- ci.renner.uno (Woodpecker, 10 repo hooks)
- paperclip-api.renner.uno (6 repo hooks)
- wiki.ai.svc.cluster.local (1 repo hook)
- openproject.renner.uno (OpenProject GitHub-style webhook, REN-1363)
Hostname entries match before any IP check (modules/hostmatcher), so
the private Traefik VIP behind these names does not need `private`.
The deprecated [webhook] ALLOWED_HOST_LIST and the admin-panel-managed
[picture] DISABLE_GRAVATAR / ENABLE_FEDERATED_AVATAR keys were removed
from the on-PVC app.ini by hand, and the same [security] value written
there, since environment-to-ini can only add keys.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Authentik gets an OpenProject confidential OAuth2 provider, application and
OpenProject Users group in the blueprint; the client secret is read from
OpenBao apps/openproject through an ExternalSecret and !Env. OpenProject
seeds the matching provider from env, and its egress policy allows
Authentik via Traefik.
Refs: REN-1363
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The udm_cef regex expected UNIFIhost immediately after the CEF header and
UNIFIdeviceName/-Model/-Ip/-Mac/-Version before msg. The UDM emits
UNIFIcategory first and the extension keys differ per event type, so no line
matched and no labels were promoted.
Anchor on the header, UNIFIcategory, UNIFIhost and the trailing msg, and
promote vendor, product, signature_id, event_name and category.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Upstream wait-for-db boots Rails via `rails runner`, and the
06-pending_migrations_check initializer aborts boot while migrations are
pending. Ahead of the seeder it can never pass on a fresh database: the
pod crashlooped in Init:Error with 154 pending migrations.
The seeder runs docker/prod/migrate (structure load on an empty DB, else
db:migrate) under rake, which skips that check; the kubelet retries it
until op-pg accepts connections.
Refs: REN-1363
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
OpenProject reads Gitea webhooks only from 1.22 on. 1.27.3 is the last
1.x release; 28.x changes the egress host lists and follows separately.
Adds a startupProbe so the liveness probe cannot kill Gitea while the
first start runs the database migrations.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Point-in-time copy of data-gitea-0 before 1.21.4 -> 1.27.3. The new
version migrates the SQLite database; a restore from this snapshot is the
way back. Restore steps are in the file.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New openproject namespace and Flux KS: web + worker pod (17.9.1-slim,
digest-pinned), op-pg CNPG 17 with Barman to Hetzner, attachments PVC
with a Velero schedule, ESO-generated SECRET_KEY_BASE and admin password,
public Ingress on openproject.renner.uno via the NetBird edge, default-deny
CNP, op-pg PodMonitor and a blackbox health probe.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sandbox pods got 403 Route not allowed on /api/agents/me/secrets*.
Adds the five /api/agents/me secret routes to the callback bridge
allowlist; company-wide secret routes stay denied.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The SML power-meter reader was retired by owner decision on 2026-09-19:
grid data comes from the E380 on the CAN bus (wiki
sml-to-e380-grid-meter-migration). Its reader has been unplugged since then.
- register: edge-1 loses write sml2mqtt/# (live, no broker restart)
- bridge templates: no sml2mqtt topic, no legacy sml2mqtt bridge
- remove role power-meter-sml, play install-power-meter.yml, inventory
group power_meter and the vault indirection
- edge-mqtt-bridge play: pre_tasks stop and remove the leftover compose
project on edge-1 on the next pipeline run (idempotent)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The register account `edge-1` replaces the legacy password users open3e
and sml2mqtt for edge-1's local mosquitto bridge. Controller main-cb96b1a
(REN-1308) delivers the password to edge-mqtt-cred over HTTPS with its
client certificate and pins edge-1's server certificate on first contact.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
edge-1's bridge password moves out of ansible-vault. A small receiver,
edge-mqtt-cred, speaks the controller's esp32 API over HTTPS with mTLS
(only CN mqtt-account-controller from the MQTT CA) and writes one bridge
`edge-1` for open3e and sml2mqtt. Until the first push the two
vault-password bridges stay in bridge.d/legacy.conf, so nothing changes
on the cluster side yet.
A Woodpecker pipeline runs the role-only play on merge, so no manual play
run is needed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mqtt-account-controller (REN-1284) now pushes the CA and the client
certificate of the register account home-assistant into HA's MQTT config
entry via HA's reconfigure flow, only when tls.crt changed and only inside
02:00-05:00 Europe/Berlin. Mounts mqtt-home-assistant-tls and the HA token
(OpenBao apps/home-assistant/mqtt-cert-sync-token, service user MQTT Cert
Sync), sets the HA env and adds Secret mqtt-ha-cert-sync to the Role.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Step 2 of adopt-then-prune. #1196 put the Secret in the home-assistant
Flux inventory (Ready at 67bc773a). Removing the manifest lets prune
delete it. Nothing reads it since #1191.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Step 1 of adopt-then-prune. The removed shellies bootstrap Job created
this Secret outside Git; nothing reads it. An empty manifest with the
same name and type puts it in the Flux inventory; the follow-up PR
removes the file so prune deletes the Secret.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
homelab#1194 added ExternalSecret gitea/gitea-package-retention, but the
openbao ClusterSecretStore only admits listed namespaces. ESO denies it
("denied by spec.condition") and the gitea Kustomization health check
waits on it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nothing pruned old CI builds in Gitea's registry, so data-gitea-0 was
down to 24% free four days after the grow to 60Gi (~2.2 GB/day).
Gitea 1.21 has no API for cleanup rules and the package owners are
human accounts, so the rule runs as a CronJob in the gitea namespace,
with the write:package-only token Hermes seeded in REN-1204.
Keeps the 5 newest images per CI build package, main/latest, and any
tag a pod runs now. Skips gitea_admin/*, date-tagged packages and
agent-runtime images.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The stock agent-runtime-claude base has no /usr/share/zoneinfo:
TZ=Europe/Berlin date prints UTC and Python ZoneInfo raises
ZoneInfoNotFoundError (REN-1195). k8s-base and k8s-go run these images.
Install tzdata and fail the build if Europe/Berlin does not resolve.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
All Shellies log in with their own Dynamic Security account; the ESP32
open3e-int is decommissioned (owner, REN-1109). Drops both users from the
passwd/ACL, their env and Secret refs, the shellies bootstrap Job, the
open3e-int ExternalSecret, rotator and policy file.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reverts homelab#1183 (merge 2f20f214). Controller main-982231a adds the
wake role to its own dynsec client; the broker kicks the client on that
("disconnected by administrative action" 45 ms after every connect), so
the controller never holds a connection and /healthz stays 503.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
main-1d03144 adds the per-account register field `rotate: false`
(mqtt-account-controller PR #6, reviewed in REN-1073). Set it on shelly-2
only. Both changes go in one commit: main-a307693 rejects a register with
the unknown field.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bump mqtt-account-controller to main-4adc9a6 (shelly-manager#93 client),
give it the shelly-manager API key and URL, and register shelly-5 (Plus i4)
as the first auth: password account. The legacy user shellies stays.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent runs mostly execute in the paperclip pod, not in a sandbox (REN-1001:
TCP timeout to :8884 from paperclip-b65cc7768). Adds the egress rule in
paperclip-egress and the paperclip pod as a source in mosquitto-agent-ingress.
:8884 only; it requires an mqtt-ca client certificate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Register entry agents-read (read open3e/#, zigbee2mqtt/#, homeassistant/#),
delivered to the paperclip namespace; agent kubectl identities may get that
one Secret; dedicated CNPs open agent sandbox pods -> mosquitto :8884.
Nothing restarts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HA keeps MQTT client certificates only as PEMs inside its config entry, so a
renewal could not reach it. A ghostunnel client sidecar listens on
127.0.0.1:11883 and forwards to mosquitto :8884 with the home-assistant
certificate, reloading it on renewal. HA's config entry is pointed there in a
separate HA-side step.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Register entry for Home Assistant (ACL readwrite # as the legacy user
homeassistant, different name so it cannot shadow that login) and its Secret
on the controller's revocation list. Nothing restarts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
evcc gets broker URL and the PEMs of the register account evcc-client by
env (EVCC_MQTT_*), which override the mqtt block of evcc.yaml in OpenBao.
Reloader restarts evcc when cert-manager renews the certificate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Register entry evcc-client (deliver evcc, ACL as the legacy user evcc), a
controller Role/RoleBinding in the evcc namespace for its Certificate and
Secret, and :8884 next to :1883 in both evcc<->mosquitto network policies.
Nothing restarts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Z2M, discovery-guard and interview-watch use the certificate of the
register account zigbee2mqtt-bridge on the mTLS listener :8884 instead of
the legacy password. Z2M gets server/ca/cert/key by env because its
configuration.yaml on the PVC is never re-seeded. The init container no
longer renders mqtt_password. The legacy broker user stays until a later
broker roll.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds the register entry for Zigbee2MQTT (same ACL as the legacy user
zigbee2mqtt, different name so it cannot shadow the running password login)
and its Secret to the controller's revocation list. Nothing restarts; the
controller picks the register up live and issues mqtt-zigbee2mqtt-bridge-tls
before the Z2M switch lands.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds the register account mosquitto-probe (auth: cert, readwrite health/#),
mounts its certificate into the broker and switches the liveness/readiness
probes to the mTLS listener :8884. Removes the legacy password user health
from passwd, acl, env and the Reloader annotation. The ExternalSecret and
rotator stay until the REN-802 cleanup step.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review 1473: shelly-manager signs in whoever X-authentik-username names when
the peer is in the pod CIDR, so opening :8080 to agent pods let any run sign
in with a forged header and no API key.
Agents now get an nginx sidecar on ${POD_IP}:8081 (Service
shelly-manager-api) that drops X-authentik-username and X-Forwarded-User
before proxying to :8080. Both agent CNPs move from 8080 to 8081, so :8080
stays outpost-only and every agent call needs the API key.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Egress from paperclip-renner agent pods and a dedicated ingress CNP on
shelly-manager, container port 8080 (socket-LB). All agents: sandbox pods
carry no per-agent label; the API key gates calls.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The REN-902 build denies publishClientReceive by default and grants receive
per read ACL. On its first loop it rewrites the Secret mqtt-dynsec-seed with
that default. The seed must be rewritten before the production broker loads
it (homelab #1159). Still points at the test broker.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
REN-802 step 2. The production mosquitto loads the Dynamic Security plugin
and gets a mutual-TLS listener :8884 (server certificate from mqtt-ca).
mqtt-account-controller now manages this broker instead of the test broker.
The nine legacy users keep password_file and acl_file unchanged.
The Dynamic Security default denies publishClientReceive: with acl_file in
the chain, mosquitto 2.0 does not gate SUBSCRIBE, so a receive default of
allow would let every Dynamic Security client read every topic.
The image policy stays on 2.0.x, because 2.1 changes the plugin chain.
The obsolete delete grant on mqtt-test-revoke-tls is dropped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The controller must delete Certificate mqtt-test-revoke, Secret
mqtt-test-revoke-tls and the dynsec client and role acct-test-revoke on its
next loop. Only the register ConfigMap changes; nothing restarts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
REN-802 step 1. Adds:
- infrastructure/post-crds/mqtt-ca.yaml: private MQTT CA (self-signed root
in cert-manager ns) and ClusterIssuer mqtt-ca.
- apps/home-assistant/mosquitto-dynsec-test.yaml: throwaway mosquitto 2.0.22
with the dynsec plugin, mTLS listener (require_certificate,
use_identity_as_username), initContainer that loads Secret
mqtt-dynsec-seed or a static base file. ClusterIP only; not labelled
mosquitto, so prod Services and the world-ingress policy do not select it.
- apps/home-assistant/mqtt-account-controller.yaml: ServiceAccount, RBAC
(cluster-wide list certificates; certificates in home-assistant; secrets
by resourceName only), controller client cert CN mqtt-account-controller,
Deployment main-47213ef pointed at the test broker.
- apps/home-assistant/mqtt-accounts.yaml: register with two cert test
accounts, delivered as ConfigMap mqtt-accounts.
The production broker is not changed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The kubernetes sandbox plugin's release and destroy hooks returned void,
so no lease got a termination receipt and every cancelled pod run left
its card waiting on remote_cleanup. Return state destroyed once the
Sandbox CR / Job (and pod) reads 404, within 20 s; otherwise no receipt.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The kubernetes sandbox plugin returns no termination receipt, so the
leases of cancelled runs e08976cd (REN-837) and 144ae3bf (REN-825) were
released without one and the queued owner comments wait on
remote_cleanup forever. The controller log shows both Sandbox CRs
deleted. Write the missing receipt for exactly those leases.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The quarterly rotation on 2026-12-01 would change the broker password while
the ESP32 gateway keeps the old one, taking it offline.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One-shot Job that sends three read commands on open3e/cmnd to edge-1's
running open3e (read-json + read-raw on 0x6c3, one read-json on 0x6e0)
and prints the replies for Loki. No write mode, no config change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The capture answered REN-805 after 24 min (no setpoint in bytes 42-56);
the owner stopped the remaining 24 h window. Flux prune deletes the Job
and its pod.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
open3e names the topics ..._ZigBeeSixDeviceCurrentValues, so the
[0-9]+ pattern matched none of the 18 DIDs (Argus review 1437).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Job subscribes to open3e/+ as the read-only evcc broker user and prints the
18 One Base FBH ZigBee<N>DeviceCurrentValues messages with a UTC timestamp
to stdout (Loki). Ends after 24 h via mosquitto_sub -W.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Server and LDAP outpost images built from k.renner/authentik-samba-bridge
872f7546 (Authentik 2026.8.3), in lockstep with the 2026.8.3 chart.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026.8 honours forwarded headers only from AUTHENTIK_LISTEN__TRUSTED_PROXY_CIDRS.
Set it to the pod CIDRs Traefik connects from. Bridge image tags follow
once authentik-samba-bridge#4 is merged and built (lockstep pin).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The authentik-samba-bridge hook token was burned in a run log (REN-729).
Woodpecker 3.18 cannot re-key a repo hash via API, so a one-shot Job
sets a new random repos.hash generated inside Postgres. Remove after run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
No pod has attached net1 since cni-plugins-installer moved to
containernetworking/plugins v1.9.1 (#1142). Hestia's verify item: restart
mdns-relay and confirm a 10.2.30.x DHCP lease. Atlas has no cluster write,
so the restart goes through a pod-template annotation.
Refs: REN-722
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#1132 staged OpenBao 2.7.1 in the StatefulSet template, but OnDelete
leaves the pods on 2.6.1 until someone deletes them, and no agent holds
pod delete in openbao. The static seal makes OnDelete unnecessary: the
readiness probe (bao status) passes only once a pod is unsealed, so the
controller replaces one pod at a time and waits for each.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Point the installer at cni-plugins:2026-10-05, built from #1140
(plugins v1.5.1 -> v1.9.1). Version comments updated in the installer
and cni-dhcp-daemon.
Refs: REN-722
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A VolumeSnapshot of influxdb-influxdb2 on ceph-block-snapclass, to be
merged and ready before homelab #1130 (2.7.4 -> 2.9.1) so the upgrade has
a restore point beyond `influxd downgrade`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Chart 0.30 moves the server 2.6.1 -> 2.7.1. Record the rollout rule
next to the pin: raft snapshot first, OnDelete one pod at a time, undo
by snapshot restore once a pod has run the new server.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
InfluxDB 2.9 hashes tokens on startup by default. The HA and Grafana
token Jobs read tokens back via `influx auth list`, which returns no
token once hashed, and a downgrade below 2.8 would erase all tokens.
Set INFLUXD_USE_HASHED_TOKENS=false so the 2.9.1 bump stays reversible
with `influxd downgrade 2.4`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refresh the plugins pin alongside multus v4.3.1 (#1131), as the
cni-plugins-installer.yaml comment asks. SHA256 cross-checked against
upstream's cni-plugins-linux-amd64-v1.9.1.tgz.sha256 asset. The tarball
still ships macvlan, static, host-local and dhcp.
Image only: CI builds cni-plugins:<date>. The DaemonSet tag bump follows
in a separate PR once the image exists.
Refs: REN-722
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Upstream deployments/multus-daemonset-thick.yml v4.2.4 -> v4.3.1 adds:
- ClusterRole rule: resource.k8s.io resourceclaims, resourceclaims/status,
resourceslices; get, list (new DRA client)
- terminationGracePeriodSeconds 10 -> 30
- CRD shortName nad
Header comment updated to v4.3.1 (v4.3.1 also has an unsuffixed tag; we
keep -thick to match the manifest).
Refs: REN-722
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Job pod templates are immutable. Without kustomize.toolkit.fluxcd.io/force,
an image bump (Renovate #1129, #1053, #1054) fails the server-side dry-run
and blocks the home-assistant and otbr Kustomizations. Same pattern as the
influxdb token Jobs.
The mosquitto Jobs exit 0 when their Secret exists. The otbr Job now also
exits 0 when the otbr Deployment is scaled to 0 (disabled), instead of
waiting for a pod that never comes and failing; Role gains get on
apps/deployments for that check.
Refs: REN-722
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tini (PID 1, root) cannot signal node (uid 1000) without CAP_KILL, exits
with EPERM on every rollout, and the server is killed without its
graceful run drain. In-flight runs are orphaned instead of being
interrupted and retried.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One Matrix room per card labelled 'matrix' (REN-668). Sets
PAPERCLIP_PUBLIC_URL for the card link in the room topic.
Rollback: build.1791177680 and drop the env var.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Route agent comments on author account only, from konairius/matrix-bridge#5
(9a83e70, REN-668). No env, secret, port or network change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The three Matrix Ingresses had no dns.renner.uno/public label, so
netbird-service-sync never created a NetBird proxy service for them and the
public proxy rejects their SNI (tlsv1 alert internal error). Label them like
audiobooks/immich and add Gatus checks for the three hosts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
as_token and hs_token come from an ESO Password generator (CreatedOnce)
instead of OpenBao, so no one has to seed them by hand. Synapse mounts
only the generated Secret and no longer depends on the OpenBao path;
OpenBao homelab/apps/matrix-bridge now holds only paperclip_api_key.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds the matrix-bridge Deployment, Service and ExternalSecret in the
matrix namespace, loads its appservice registration in Synapse, and opens
the bridge -> Paperclip API path (:3100) in both namespaces' policies.
Tokens and the board key come from OpenBao homelab/apps/matrix-bridge.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Authentik 2026.5 checks each authorize against the provider's
grant_types, and a provider created by a blueprint starts with an empty
list. Every Matrix login through Authentik failed with invalid_request
(log: "Invalid grant_type for provider", grant_type=authorization_code).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner wants QR sign-in from the desktop app as well. Linking a phone from
a signed-in Element (Link new device) only needs the server side from the
previous commit (MSC4108 + MAS device-code grant). This flag also lets a
new browser sign in by QR.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Element X signs in only with a password or through MAS. Put MAS in front
of Synapse with Authentik (passkey) as the upstream provider, and enable
the QR rendezvous (MSC4108/MSC4388).
- apps/matrix/mas.yaml: MAS 1.26.0, `mas` database in mx-pg (CNPG
Database), account.matrix.renner.uno.
- synapse.yaml: matrix_authentication_service, password_config off; a
syn2mas init container moves the existing accounts and sessions once
(marker on synapse-data).
- Drop the owner-account Job; MAS owns accounts now.
- Authentik: public PKCE client `matrix-mas`, passkey flows, only the
Matrix Users group (k.renner).
- matrix-ns: MAS egress to Traefik :8443 (auth.renner.uno) only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Scoped access for the zwave-js restart (REN-628): forced-command user
atlas-zwave on edge-1. One /32, TCP 22 only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Loki runs SingleBinary with auth disabled and answers GET on
/ingester/shutdown and /flush, so an RBAC `get` on services/proxy for
loki:3100 would not be read-only. Put an nginx allow-list in front of it
(query, query_range, labels, label/<name>/values, series; GET only;
upstream URI rebuilt per location) and grant the paperclip read SA
`get` on that Service only. Owner decision: REN-622 (option A2).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The primary pclp-pg-2 was OOMKilled 6 times on 3-4 Oct at the 1Gi
limit. Each kill drops Paperclip's DB and fails in-flight agent runs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#1116 left spec.volumeName out of pvc.yaml. kustomize-controller has owned
that field since the namespace move pre-created the claim (eb23d3ac), so
server-side apply tried to clear it and the API server rejected the claim
as an immutable-spec change. The gitea Kustomization is Ready=False and the
claim stays at 40Gi. Pin volumeName to the live value.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The volume filled on REN-580. volumeClaimTemplates is immutable, so own the
live claim data-gitea-0 in apps/gitea/pvc.yaml (prune disabled) and let Flux
server-side apply grow it; ceph-block expands online. Add GiteaDataVolumeLow
(<25% free for 15m) with a promtool suite and two mutation controls.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stable session fingerprint for the per-run managed AI home paths, plus a
persistent CLAUDE_CONFIG_DIR/projects so the resumed session finds its
transcript. Local runs only. Fix-forward of #1111 (reverted in #1113).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New `matrix` namespace and Flux KS: Synapse v1.162.0 on a CNPG Postgres
(mx-pg, Barman to Hetzner), media store + signing key on synapse-data
(Velero matrix-daily), Element Web locked to matrix.renner.uno, and an
idempotent Job that creates the one owner account from OpenBao
apps/matrix. Federation off, registration and guests closed, LAN-only
ingress, egress limited to the in-cluster DB and the push gateway.
Appservice slot left empty for REN-565 Part 2.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GITEA_VERIFY_REPO still named k.renner/shelly-manager, which now
answers 307 to the new path. Point it at the new path directly.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The host/remote-node rule had no toPorts, so any node or hostNetwork pod
could reach the relay's unauthenticated gRPC on :4245. Restrict it to the
kubelet gRPC probe port (chart deployment.yaml probes use port 4222).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
yaml-citations failed on external-secrets-podmonitor.yaml:35, which cited the
upstream chart's values.yaml by line. That file is not in the repo, so no
path#anchor form exists; name the keys instead. Comment-only change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hubble is already on in every agent; the relay gives one gRPC endpoint so
`hubble observe -n paperclip-renner --verdict DROPPED` runs without exec
into kube-system. The chart renders no change to cilium-config or the agent
DaemonSet, so the reloader does not roll the CNI.
The relay has no client auth: a new CNP admits only the paperclip pod (and
nodes for probes), and paperclip-egress gains the matching :4245 rule.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Argus's review of PR #1032: the name exists in neither pkg/ nor the metrics
docs for 1.20.2, and the agent skips unknown --metrics names silently. Zeus
dropped it from scope. metricsService, the Service delete and the new
selector stay.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- prometheus.metrics: +cilium_policy_verdict_total (opt-in; per-direction
policy verdict attribution on top of cilium_drop_count_total).
- prometheus.metricsService: true, so the chart renders the headless
`cilium-agent` metrics Service.
- Drop the hand-declared `cilium-agent-metrics` Service from
cilium-servicemonitor.yaml and point the ServiceMonitor at the chart's
Service, so the endpoints are scraped once, not twice.
This is a values change: helm-controller upgrades the CNI release, the
cilium-config `metrics` key changes, and the reloader annotation rolls the
agents node by node. Merge only inside the window Zeus set on REN-166.
Revert: git revert this commit; Flux restores the old Service and selector
and rolls the agents back the same way.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The local-login connection stores only the ~8h OAuth access token and
Paperclip never refreshes Anthropic credentials (subscription values
are injected read-only as CLAUDE_CODE_OAUTH_TOKEN). It expired 8h after
the login and every run failed with 'terminal access failure'.
Replacing the secret value with a claude setup-token token fixed it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-25 21:59:31 +02:00
372 changed files with 14662 additions and 5204 deletions
MODEL=$$(sh scripts/ci/pick-model.sh "distill raw source notes into atomic wiki notes: read sources, check the wiki for existing coverage, write or merge notes; careful writing, moderate reasoning")
if [ -n "$$MODEL" ]; then echo "distill: selected model $$MODEL"; else echo "distill: using the CLI default model"; fi
cd notes
cat ../ci/agents/wiki-maintenance/distill.md | claude $${MODEL:+--model $$MODEL} \
# ---- Clone, pick model, review, post ---------------------------------
set -eu
[ -s /tmp/pr-review/worklist.jsonl ]||{echo"Nothing to review.";exit 0;}
[ -n "${MISTRAL_API_KEY:-}"]||{echo"ERROR: MISTRAL_API_KEY is not set — grant the mistral_api_key secret in this repo's .woodpecker/pr-review.yml stub.";exit 1;}
REVIEW_FAILED=0
API="https://gitea.renner.uno/api/v1"
AUTH="Authorization: token $REVIEW_TOKEN"
WS="$PWD"# workspace = this repo, checked out at the PR head by Woodpecker
MAX_DIFF_LINES=20000
# This script, the prompts and scripts/ci/ all come from the homelab@master
# clone the per-repo stub made (see .woodpecker/pr-review.yml) — never from the
# PR checkout, so a PR cannot change the rubric or the helper scripts it is
# reviewed with.
PROMPTS="$(cd"$(dirname "$0")"&&pwd)"
HOMELAB="$(cd"$PROMPTS/../../.."&&pwd)"# repo root holding ci/ AND scripts/ci/
# || echo "No public context clone of $REPO (skipped)."
# done
#
# # Model: newest from the live Anthropic catalog (needs CLAUDE_CODE_OAUTH_TOKEN).
# MODEL=$(env -u REVIEW_TOKEN sh "$HOMELAB/scripts/ci/pick-model.sh" \
# "careful, high-stakes code review of a pull request: verify claims against the code and external documentation, find real defects, avoid false positives; always pick the most capable and newest model available, regardless of PR size" || true)
# if [ -n "$MODEL" ]; then echo "Review model: $MODEL"; else echo "Review model: CLI default (catalog/picker unavailable)"; fi
#
# # Review with the chosen model.
# ( cd "$WS" && cat "$PROMPTS/prompt.md" | env -u REVIEW_TOKEN claude ${MODEL:+--model $MODEL} \
printf'🤖 **Automated review** — model `%s`, commit `%s`.\n\n'\
"${MODEL:-CLI default}""${HEAD}"
printf'> Automated review by the `review-bot` pipeline: it approves only when it found nothing red or yellow, and a human still merges. Push new commits to get a fresh review.\n\n---\n\n'
cat "$D/report.md"
printf'\n\n<!-- pr-review:sha=%s -->\n'"$HEAD"
} > "$D/body.md"
jq -n --rawfile b "$D/body.md" --arg e "$EVENT" --arg c "$HEAD"\
You are running as the scheduled wiki-maintenance job in CI. The Notes vault is checked out at ~/Notes (the repository k.renner/notes); the homelab and homeassistant repositories are checked out under ~/source/. `kubectl` reads the live cluster with a read-only identity.
Run the wiki-consolidate agent now for today's scheduled consolidation pass. Follow its instructions exactly: snapshot the vault to git first, abort if that fails, select today's rotating cohort of 20 from the state file, verify claims against both their sources and the live systems, act on the verdicts, write the dated report to ~/Notes/output/, update the state file, and snapshot again at the end. Do not push; the pipeline pushes the run's commits.
Report the counts and the coverage progress when you are done.
You are running as the scheduled wiki-maintenance job in CI. The Notes vault is checked out at ~/Notes (the repository k.renner/notes). Do exactly this, nothing else:
Glob ~/Notes/raw/*.md (top level only, never ~/Notes/raw/_processed/). Distill every file you find into ~/Notes/wiki/ following your process, including the ranker check for every concept. Move each fully processed source to ~/Notes/raw/_processed/. If raw/ has no top-level files besides _index.md, say so and stop.
Do not commit or push; the pipeline does that after you finish. Do not touch anything outside ~/Notes/wiki/ and ~/Notes/raw/.
End with one line per file: which wiki notes it produced or merged into, or "nothing distillable".
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Deploy the official Gitea MCP server in-cluster at `https://gitea-mcp.renner.uno/mcp`, published on the NetBird proxy, authenticating every request with the caller's own Gitea PAT and rejecting tokenless requests.
**Architecture:** One stateless Deployment in a new `gitea-mcp` namespace, Flux-managed through its own Kustomization. Traefik Ingress with the proxy-only public label; netbird-service-sync publishes it. gitea-mcp runs in `-oauth` mode purely to get the 401-on-missing-token gate; it talks to Gitea via the in-cluster Service.
- Produces: namespace `gitea-mcp`; ingress to pods labelled `app.kubernetes.io/name: gitea-mcp` on 8080 from Traefik; egress from `gitea-mcp` to `gitea` pod :3000; Gitea-side ingress allowance from namespace `gitea-mcp` on :3000.
- [ ]**Step 1: Add the Namespace** — insert after the `name: gitea` Namespace document in `namespaces.yaml`:
```yaml
---
# Gitea MCP server. Declared here so it exists before infrastructure-crds applies its CNPs.
apiVersion:v1
kind:Namespace
metadata:
name:gitea-mcp
```
- [ ]**Step 2: Create `gitea-mcp-ns.yaml`**
```yaml
---
# Baseline policy for the `gitea-mcp` namespace.
apiVersion:cilium.io/v2
kind:CiliumNetworkPolicy
metadata:
name:default-deny
namespace:gitea-mcp
spec:
endpointSelector:{}
ingress:
- fromEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": gitea-mcp
- fromEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": monitoring
- fromEntities:
- host
- remote-node
- fromEndpoints:
- matchExpressions:
- key:io.kubernetes.pod.namespace
operator:Exists
icmps:
- fields:
- {type:0, family:IPv4 }
- {type:3, family:IPv4 }
- {type:8, family:IPv4 }
- {type:11, family:IPv4 }
- {type:12, family:IPv4 }
- {type:1, family:IPv6 }
- {type:2, family:IPv6 }
- {type:3, family:IPv6 }
- {type:4, family:IPv6 }
- {type:128, family:IPv6 }
- {type:129, family:IPv6 }
- {type:133, family:IPv6 }
- {type:134, family:IPv6 }
- {type:135, family:IPv6 }
- {type:136, family:IPv6 }
- {type:143, family:IPv6 }
egress:
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": gitea-mcp
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": kube-system
k8s-app:kube-dns
toPorts:
- ports:
- port:"53"
protocol:UDP
- port:"53"
protocol:TCP
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": gitea
app.kubernetes.io/name:gitea
toPorts:
- ports:
- port:"3000"
protocol:TCP
---
apiVersion:cilium.io/v2
kind:CiliumNetworkPolicy
metadata:
name:gitea-mcp-allow-traefik
namespace:gitea-mcp
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name:gitea-mcp
ingress:
- fromEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": ingress
app.kubernetes.io/name:traefik
toPorts:
- ports:
- port:"8080"
protocol:TCP
```
- [ ]**Step 3: Register it** — in `network-policies/kustomization.yaml` add ` - gitea-mcp-ns.yaml` directly after ` - gitea-ns.yaml`.
- [ ]**Step 4: Gitea-side allowance** — append to `kubernetes/apps/gitea/networkpolicy.yaml`:
```yaml
---
# gitea-mcp reaches the in-cluster Service; egress half is in network-policies/gitea-mcp-ns.yaml.
Confirm `kubernetes/clusters/homelab/` is picked up without a listing file: `ls kubernetes/clusters/homelab/flux-system` and read its `kustomization.yaml`; if it lists files explicitly, add `gitea-mcp.yaml` where `gitea.yaml` is listed.
`git -c user.name='Claude Bot' -c user.email='claude-bot@renner.uno' commit -m "feat(gitea-mcp): deploy the Gitea MCP server at gitea-mcp.renner.uno" -m "Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"`
Expected: `HTTP/2 401` with `WWW-Authenticate: Bearer resource_metadata=…`.
- [ ]**Step 5: Invalid token is rejected** — same request plus `-H 'Authorization: Bearer invalid'` → 401 with `error="invalid_token"`.
- [ ]**Step 6: Authenticated calls work** — same `initialize` request with `--oauth2-bearer "$(cat /home/krenner/.config/claude-bot/gitea-token)"` → 200 with `serverInfo`. Then `tools/call``{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"list_my_repos","arguments":{}}}` with the same token → non-empty repo list; `list_my_repos` resolves the caller from the token, so a result proves the PAT was used. If the server demands `Mcp-Protocol-Version`/`Mcp-Method` headers, take the required values from the error and resend.
- [ ]**Step 7: Image policy** — `flux get image policy gitea-mcp` → latest image `docker.gitea.com/gitea-mcp-server:1.8.1` or newer.
- [ ]**Step 8: Wiki** — route through `wiki-ingest` → `wiki-distill`: netbird-service-sync label mechanism (`dns.renner.uno/public` vs `netbird.renner.uno/public`), and gitea-mcp deployment facts (oauth-mode-as-auth-gate, PAT passthrough, `/healthz` outside the Host check).
@@ -16,7 +16,7 @@ credentials are out of scope (see "Deferred").
|---|---|
| Execution model | **B — server-local runs** (upstream default): agent CLIs are child processes of the server, in the server container. Trial only. |
| Upgrade path | If Paperclip is kept: **C** — `@paperclipai/plugin-kubernetes` sandbox provider + kubernetes-sigs agent-sandbox controller (`agents.x-k8s.io`), per-run Pods and Secrets. Separate spec. |
| Claude auth | **One shared Claude subscription AI connection**, created once via Paperclip's local sign-in (`claude auth login` into `ai-local-logins/<id>`); Paperclip stores and refreshes it. Server-env `CLAUDE_CODE_OAUTH_TOKEN` is not used: `stripAiAuthBindings` blanks it for managed runs, and upstream accepts no pasted setup-token (the device-code flow needs a Daytona sandbox). |
| Claude auth | **One shared Claude subscription AI connection**, created once via Paperclip's local sign-in (`claude auth login` into `ai-local-logins/<id>`), then its value replaced with a long-lived `claude setup-token` token. Paperclip stores the value as company secret "My Claude subscription" and injects it per run as `CLAUDE_CODE_OAUTH_TOKEN`; it never refreshes Anthropic credentials, so the login's ~8h access token expires (runs then fail with "ACP agent reported a terminal access failure", 0 tokens). Renew by rotating that secret with a new setup-token (~yearly). Server-env `CLAUDE_CODE_OAUTH_TOKEN` is not used: `stripAiAuthBindings` blanks it for managed runs, and upstream accepts no pasted setup-token (the device-code flow needs a Daytona sandbox). |
| Bot credentials | Out of scope. Future "AIR-Manager" Paperclip role handles permission requests. |
| Exposure | LAN-only, behind an Authentik forward-auth gate, then Paperclip's own login. |
@@ -189,7 +189,8 @@ available since v0.35.0). First use of digest reflection in this repo.
**as the node user** (the entrypoint makes exec sessions root, and root-owned 0600
# Backup verification on an ephemeral Hetzner Cloud VM — design
Date: 2026-10-10
Status: draft, awaiting review
## Goal
Replace the monthly Velero `restore-test` with a check that proves every PVC
backup is readable and complete, **without writing restored data to the
homelab Ceph OSDs**. The work runs on a throwaway Hetzner Cloud VM next to the
backup bucket; the cluster only orchestrates.
Full disaster-recovery rehearsals (Velero restores of manifests into a fresh
cluster) are a later, separate project. This spec only covers data
verifiability.
## Why
-`restore-test` restores every schedule into the production Ceph pools. On
2026-10-01 the immich restore took the fullest OSD from 74% to 95% and Ceph
went `OSD_FULL` (REN-451). The test is a large share of the OSD write budget.
- Its file-count parity compares the restored PVC against the *live* volume.
That needs a source-side snapshot probe per StorageClass (it broke for
`ceph-filesystem-bulk`, see PR #1224) and volatile-path exclusions.
- The restore only covers `persistentvolumeclaims`/`persistentvolumes`, so a
data-level check loses no coverage that the current test has.
## Decisions
| Topic | Decision |
|---|---|
| What is verified | Data only (option A): `kopia snapshot verify --verify-files-percent=100` on the latest Kopia snapshot of every DataUpload of the latest `Completed` backup per schedule, plus a size check against Velero's own `DataUpload.status.progress.totalBytes`. No restore, no Kubernetes on the VM. |
| Where it runs | Ephemeral Hetzner Cloud VM in `fsn1` (same location as bucket `renner-uno-k8s-velero`). Bucket traffic stays inside Hetzner's eu-central zone. |
| Orchestrator | In-cluster CronJob `backup-verify` (namespace `velero`), monthly `0 8 1 * *` UTC. It reads Kubernetes, drives the VM via the Hetzner API and SSH, and writes nothing but logs. |
| Alerting | Exit 1 on any `FAIL` → existing `KubeJobFailed`. Results in the Job log (Loki). |
| Coverage | All DataUploads, including CNPG volumes (read-only, no cost). Schedules discovered by label `app.kubernetes.io/part-of=homelab-backup`. |
| SSH source restriction | None. Key-only auth with a per-run key, pinned host key, password auth off, VM lives ~1 h. |
| Kopia version | CLI pinned to **0.16.0**, the version Velero 1.18.2 vendors (`project-velero/kopia` fork of 0.16.0). sha256 pinned per arch. |
Run the official Gitea MCP server (`gitea.com/gitea/gitea-mcp`) in the cluster
and publish it on the NetBird proxy, so MCP clients anywhere (Claude Code on
the workstation, Paperclip agents, other machines) can use Gitea through MCP
with their own Gitea personal access token.
## Decisions
| Topic | Decision |
|---|---|
| Hosting | In-cluster HTTP service, own namespace `gitea-mcp`, one stateless replica. |
| Exposure | Public via NetBird proxy, host `gitea-mcp.renner.uno`, label `netbird.renner.uno/public` (proxy-only; the LAN record on the Traefik VIP stays, so LAN clients don't hairpin). |
| Client auth | Bearer PAT per request (`Authorization: Bearer <pat>` or `token <pat>`). The server holds no token. Every call runs with exactly the caller's Gitea permissions. |
| Anonymous access | Rejected with 401. Achieved by running with `-oauth`: in that mode a missing token is refused, a present token is passed through unchanged and checked with one `GET /api/v1/user`. No OAuth app or client secret is needed server-side. |
| Gitea backend | `https://gitea.renner.uno` via Traefik. The `-H` URL is also what the OAuth protected-resource metadata advertises as authorization server, so it must be the public one. Pods reach it the same way woodpecker-server and wiki already do. |
| Image | `docker.gitea.com/gitea-mcp-server`, starting at `1.8.1`, tracked by Flux image automation (semver). |
| Read-only mode | Off; write tools exposed. Write access is bounded by each caller's token. |
## Why the `-oauth` flag with PATs
Gitea answers anonymous API calls: `GET /api/v1/repos/search` without
credentials returns `k.renner/homelab`, `konairius/Wiki` and
`gitea_admin/_cargo-index`. In plain HTTP mode (no `-oauth`, no server token)
gitea-mcp forwards tokenless requests as an anonymous Gitea client, so the MCP
endpoint would serve those repos to anyone. Upstream has no "require token"
flag outside OAuth mode.
With `-oauth -oauth-public-url https://gitea-mcp.renner.uno` (upstream
`pkg/oauth/oauth.go`):
-`/mcp` without a token → 401 with an RFC 9728 `WWW-Authenticate` challenge.
-`/mcp` with a token → `TokenRejected` calls Gitea `GET /user` with it; only a
401 there rejects. The token is then used as-is for the API calls, so a plain
PAT works. There is no JWT/introspection/audience check.
- The server only publishes `/.well-known/oauth-protected-resource` metadata;
config is the public URL and the Gitea URL, nothing else.
-`/mcp` additionally checks `Host` against the public URL's host (or
loopback) and returns 421 otherwise. `/healthz` is outside both checks.
Costs: one extra `/user` call per MCP request. The metadata document advertises
the in-cluster Gitea URL as authorization server; this only matters for
OAuth-flow clients (claude.ai connectors), which are out of scope.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
