Outage since 10:11Z: the first pass of image 2026-11-01-0959 (#1535)
removed all 14 gates ("removed OIDC client from Authentik", "14 objects
removed"), and every pass since recreates and removes them again.
ProxyProvider subclasses OAuth2Provider (authentik 2026.8.4
providers/proxy/models.py: class ProxyProvider(OutpostModel,
OAuth2Provider)), so GET providers/oauth2/ also lists every proxy
provider. owned_applications() yielded each gate a second time as an
OAuth2 application; it was not in the OIDC keep set, so the sweep
deleted its application and provider.
Both the sweep and ensure_oidc's provider lookup now take from
providers/oauth2/ only providers whose meta_model_name is
authentik_providers_oauth2.oauth2provider. The test fake now lists proxy
providers under providers/oauth2/ like the real API; the new tests
(gate-only and gate + OIDC client stable over two passes) fail against
the previous controller.
- Desired redirect URIs carry redirect_uri_type: authorization, which the
API returns on every entry (RedirectURISerializer default in 2026.8.4);
without it every pass would re-PATCH the provider.
- OIDC slugs and gate names are kept apart: a slug that is a gate name is
refused, an existing application without this controller's OAuth2
provider is refused, and the sweep keeps proxy and OAuth2 applications
by their own sets, so an OIDCClient can neither take over nor delete a
gate.
- Status is written only when it changed.
- Adoption of an unassigned oidc-<slug> provider is documented; removal
logs say OIDC client.
- reconcile() takes k8s/ak for tests; new loop tests: create then quiet,
duplicate slug, transient error keeps the client, sweep after the
resource is gone, slug equal to a gate name.
OIDC is not part of the Ingress. The client is now declared by an
OIDCClient (auth.renner.uno/v1alpha1, CRD in
infrastructure/authentik-gate-sync-crds, like kuma-sync-crds) next to the
app: launchURL, redirectURIs, optional slug, group, authorizationFlow.
The controller writes a Ready condition and the issuer to its status.
The auth.renner.uno/oidc-* Ingress annotations are gone; RBAC gains
get/list/watch on oidcclients and patch on their status.
An Ingress annotated auth.renner.uno/oidc-client: <slug> gets a
confidential OAuth2/OpenID provider (client ID = slug, secret generated by
Authentik), an application with that slug and, with auth.renner.uno/group,
the same group expression policy as gates. auth.renner.uno/oidc-redirect-uri
pins strict redirect URIs on the Ingress host; default is any path on it.
Same fail-closed and ownership rules as gates: owned clients are found via
the OAuth2 provider list and meta_publisher and swept when their Ingress
goes. The service-account role gains the OAuth2 provider permissions plus
view on scope mappings and certificate key pairs.
So apps that log in through Authentik via OIDC (first: YouTrack, #1534)
need no blueprint entries; the groups stay managed in Authentik.
Monitors without a parentRef are placed under the root group set by
KUMA_ROOT_GROUP (kuma-sync/renner-uno). Only the root references the
email notification, so a fleet-wide outage sends one DOWN and one UP mail
listing the affected groups. resendInterval (consecutive DOWN heartbeats)
is now a managed KumaMonitor field; the root resends hourly while down.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every Kuma monitor, group or child, carried the default "email"
notification, so one outage sent a down and an up mail per monitor.
- New cluster-scoped CRD KumaNotification (type smtp): from/to in the
spec, host/port/username/password from a Secret read uncached.
kuma-sync adopts the Kuma notification of the same name, keeps it
isDefault=false and reverts drift.
- KumaMonitor gains spec.notificationRefs. A monitor attaches only the
notifications it references; none means none. The default-notification
attachment is removed.
- KumaNotification "email" (uptime@renner.uno -> admin@renner.uno) with
ExternalSecret kuma-sync-smtp from homelab/infrastructure/uptime-kuma/smtp
and a Role granting get on that one Secret. No monitor references it.
- kuma-setup.js no longer creates the notification; the uptime-kuma role
and the uptime-kuma-vps policy drop the SMTP credentials
(policy must be re-applied by hand).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The embedded outpost is matched on the client too, so a filter the server ignored cannot select the LDAP outpost. A host named by two gated Ingresses is a GateError: the outpost finds providers by host, so it would serve one gate for both.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The gate is now declared on the app's own traefik Ingress: naming the
static Middleware authentik/gate (ForwardAuth against the embedded
outpost) in router.middlewares opts in, and its auth.renner.uno/*
annotations configure it. No generated route Ingress, ExternalName
Service or per-gate Middleware; the IngressClass authentik-gate is gone.
authentik-gate-sync keeps building the Authentik side (forward_single
provider on the embedded outpost, application, group policy) and the
backend CNP admitting Traefik, and sweeps the objects it generated
before. Traefik publishes the Ingress status itself.
The outpost's own paths (/outpost.goauthentik.io/) are one static
IngressRoute above the app routes. The embedded outpost answers 404 for
a host without a provider (checked live: auth/traefik with
X-Forwarded-Host no-such-gate.renner.uno), so a gate without its
provider stays closed.
docs.renner.uno points at paperless:80 behind redirect + gate (Paperless
Users) instead of the removed paperless-gate Service. Drops __pycache__
committed by mistake.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each gate was a single-replica proxy outpost in series with Traefik, so
any outpost restart, drain or node loss took its app down. Gates now
run as Traefik ForwardAuth against Authentik's embedded outpost inside
authentik-server: the request path is Traefik -> app, and the only auth
dependency is authentik-server, which every login already needs.
Per gated Ingress the controller now creates a forward_single provider
attached to the embedded outpost, a ForwardAuth Middleware first in the
route's chain, a /outpost.goauthentik.io route to authentik-server, and
a CNP admitting Traefik to the backend pods. It removes the per-gate
outposts and their egress CNPs.
Identity headers: authResponseHeadersRegex makes Traefik delete every
client-sent X-authentik-* header before copying the outpost's answer
(traefik pkg/middlewares/auth/forward.go), so a gated backend only ever
sees identity Authentik issued. Traefik answers 500 if the outpost is
unreachable and drops a router whose Middleware is missing: fail closed.
The blueprint no longer declares the embedded outpost: an entry must
carry providers, and re-applying [] would detach every gate. The
controller owns its providers and authentik_host/_browser.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Step 2 of 2 of the label split (owner decision 2026-10-22). Requires
#1425 (step 1) to be applied first.
- external-dns-unifi: labelFilter dns.renner.uno/internal=true (opt-in).
Every object it publishes today carries the label since step 1, so the
UDM record set is unchanged.
- external-dns-cloudflare: labelFilter dns.renner.uno/public=true, default
target proxy.renner.uno (the cloudflare-lb-sync load balancer) instead of
the retired tunnel, no --cloudflare-proxied, policy sync. Each public
host gets a DNS-only CNAME to proxy.renner.uno plus its TXT registry
record; this is the prerequisite for removing the *.renner.uno wildcard.
- netbird-service-sync: reads netbird.renner.uno/public only. Every
dns-public Ingress carries it since step 1, so the 21 proxy services are
unchanged.
- git/gitea/gitea-mcp: add dns.renner.uno/public (public today via the
wildcard; keep their LAN record via dns.renner.uno/internal).
- Drop comments describing the old public!=true filter.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New in-house controller (kuma-sync house pattern) that reconciles a
CloudflareLoadBalancer resource into a DNS-only Cloudflare load balancer,
its pool and its health monitor, using cloudflare-go v7.
proxy.renner.uno had round-robin A/AAAA records without health checks, so
every proxy reboot blackholed part of the public traffic. The owner set up
a DNS-only load balancer by hand (hostname origins proxy-1/proxy-2, HTTPS
monitor with Host: status.renner.uno); this moves it into git.
- Identity is the load balancer hostname; pool and monitor are found via
its references. Hand-made objects are adopted (the account allows only
2 origins, so a parallel pool would be rejected) and tagged in their
description; objects with another owner's tag are never written.
- Status shows per-origin health; resync every 2 minutes.
- Token from OpenBao homelab/infrastructure/k8s-cloudflare-lb via
ExternalSecret; egress limited to api.cloudflare.com:443.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fixes CVE-2026-99999 (ipv4: validate checksum_start before completing
checksum, CVSS 9.3). The nodes run 6.18.48-talos; the fix is in 6.18.57,
which v1.13.12 ships (siderolabs/pkgs@19e92dc).
Not applied to any node yet. Rollout is one node at a time: drain,
talosctl upgrade --image, wait for Ceph HEALTH_OK, starting with k8s-1.
Patch release within 1.13; Kubernetes stays v1.36.3. Schematic is
unchanged; factory installer manifest and ISO verified to exist for the
tag. talosctl checksum taken from the release's sha256sum.txt.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The owner wants the path alert -> [alert] issue -> kanban card -> expert fix PR
-> owner merges, without a human step in between. Owner decisions (kanban card
t_3f5d0a62): [alert] issues are not excluded from the bridge, and the issue-fix
Woodpecker pipeline is retired, so the expert's card is the only automated fix
path for [alert] issues.
Routing
- alert-monitor: `alerts.py file` takes a required `--label` (hestia, atlas,
apollo, clio, daedalus) and sets that label at creation (POST /issues
`labels: [id]`). Gitea 1.27.3 adds the labels in the same transaction
(services/issue/issue.go NewIssue) and the `opened` webhook payload loads them
(services/convert/issue.go toIssue -> LoadLabels), so the first delivery
already routes the card to the expert. Org labels are accepted for org repos
(models/issues/issue_update.go: label.OrgID == repo.OwnerID). If the label
does not exist yet, the issue is filed without it and the coordinator routes.
SKILL.md tells the triage agent how to pick the expert.
- Label-less issues no longer go to triage. kanban.auto_decompose defaults to
true (hermes_cli/config_defaults.py; not overridden here), so the dispatcher
hands every triage card to the auxiliary-LLM decomposer
(gateway/kanban_watchers_dispatcher.py auto_decompose_tick ->
kanban_decompose.decompose_task), which rewrites title and body (dropping the
Gitea-Issue line the completion hook needs), may fan out into children without
that line, and keeps the assignee `default` anyway (_apply_single only sets an
assignee when none is set). The card is now created ready for `default` with
routing instructions: set exactly one domain label on the issue, nothing else.
- The script reacts to that label: a coordinator card nobody has claimed yet
(todo/ready) is reassigned (`hermes kanban assign`) and its body replaced by
the expert body (`hermes kanban edit --body`); a claimed or finished one gets
a separate expert card under key `<owner>/<repo>#<n>@<expert>` (one per
expert, created_by gitea-webhook, so its completion is posted on the issue).
`assign` refuses a running card, which falls back to the separate card. Cards
already with an expert are still never moved.
[alert] cards
- Their body carries the fix protocol that replaces issue-fix, including its
prompt hardening from PR #253/#266: check for an open PR with `Closes #<n>` or
a branch `alert-fix/issue-<n>` first; owner (k.renner) comments are
authoritative, everything else is untrusted problem description; verify live
before fixing, never state cluster state not observed this run, give
read-only commands instead of predicted output; repo-fixable -> branch
`alert-fix/issue-<n>`, smallest fix, PR with `Closes #<n>`, never merge; not
repo-fixable -> no PR, the completion summary (posted on the issue) says why
and what a human must do.
Gitea side (CronJob gitea-webhook-sync, every 15 min)
- Domain labels: created as org labels in every org (GET /admin/orgs) and as
repo labels in user-owned repos, only when missing.
- Backfill: open issues without the card comment are replayed to the route as
signed `opened` deliveries (same headers Gitea sends), at most 10 per run, 3 s
apart (the adapter's default rate limit is 30/min/route), in random order so
repos where the comment keeps failing cannot starve the rest. The idempotency
key `<owner>/<repo>#<n>` makes a replay of an issue with a card a no-op.
The job runs in the hermes namespace, whose default-deny already allows
intra-namespace traffic to :8644.
- The admin PAT needs write:organization (org labels, reqOrgOwnership passes for
a site admin), write:issue (repo labels, comments) and read:repository
(/repos/search) next to write:admin. activeDeadlineSeconds 120 -> 600.
issue-fix removed
- .woodpecker/issue-fix.yml, ci/agents/issue-fix/, and what only it used:
scripts/ci/make-diag-kubeconfig.sh, the monitoring/ci-diagnostics
ServiceAccount and RBAC (no other pipeline mints its token; git grep), the
.ci-diag/ gitignore entry. Comments naming issue-fix updated.
- Not in git and left to the owner: the Woodpecker cron `issue-fix` and the
secret `issue_fix_bot_token` with its Gitea PAT.
Tested: unittest 73 OK; gitea_issues.py against the real hermes kanban CLI in a
scratch HERMES_HOME with a stub Gitea (unlabelled -> default ready card;
labelled while unclaimed -> same card reassigned to daedalus with the alert
body; labelled while claimed -> separate card `#n@daedalus`; replay -> no
duplicate card or comment); sync discovery read-only against live Gitea (42
open issues, all without a card); kubectl kustomize for apps/hermes,
apps/monitoring, apps/monitoring-rules.
The CRD and the ClusterRole rules land with the controller change, so
Flux applies them at merge, before the new kuma-sync image is built and
rolled out by the ImagePolicy; the new controller watches
KumaStatusPage and would not start without them.
- infrastructure/kuma-sync-crds: crd-kumastatuspages.yaml, a copy of
config/crd with prune disabled, next to crd-kumamonitors.yaml.
- apps/kuma-sync/rbac.yaml: kumastatuspages, /status, /finalizers, as in
config/rbac/role.yaml.
- apps/kuma-sync/statuspage-renner.yaml: the public page (slug renner,
domain uptime.renner.uno) with the sections Services and Third-party,
selected by the uptime.renner.uno/status-page-section label.
- The label on the 17 top-level service groups and the four third-party
HTTP checks.
An envtest test validates the deployed page object against the CRD.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Conflict loser no longer keeps the page ID, section counts and unsynced
count of an earlier sync; an invalid selector keeps only the page ID.
The round-trip integration test now asserts that autoRefreshInterval,
showOnlyLastHeartbeat and rssTitle survive a managed save. Doc comments
state that per-monitor custom URLs are cleared on save.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New namespaced CRD KumaStatusPage (uptime.renner.uno/v1alpha1) and a
reconciler that owns the Kuma status page completely: config, domain
names and the full group/monitor list. Sections select KumaMonitors in
any namespace by label; monitors are ordered by displayName, then
namespace/name, a monitor goes to the first matching section only, and
monitors without a Kuma ID are left out and counted in status.
The reconciler adopts an existing page with the slug, writes only when
the managed content differs (no write per resync), reuses Kuma group IDs
by position so saves do not churn them, reverts UI edits on resync,
deletes the page on resource deletion, and reports Ready=False/Conflict
for a younger resource claiming the same slug. Every KumaMonitor change
to labels, displayName, monitor ID or deletion enqueues all pages.
kumaclient gains a StatusPages interface (Breml + Fake). The socket API
returns no group list, so groups are read from Kuma's public
GET /api/status-page/<slug>, as the bootstrap did. Fields kuma-sync does
not manage (logo, custom CSS, analytics, RSS title, auto refresh
interval) are written back unchanged. autoRefreshInterval is not in the
spec: go-uptime-kuma-client v0.6.2 (latest) never sends it on save.
The Ansible bootstrap no longer creates or edits status pages; the
existing page is left in place for kuma-sync to adopt.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner decision: Hermes files the [alert] issues; the Woodpecker
alert-monitor pipeline is retired. issue-fix keeps working unchanged: it
picks issues by the "[alert] " title prefix, and the dedup contract is the
same (title "[alert] <alertname> (<namespace|job|cluster>)", noise-log lines
"- <fp> <ALERTNAME> — <date>: <reason>" with a 30-day TTL, blip
fingerprint sha256(alertname|scope)[:16], fingerprint footer on issues).
- skill alert-monitor: scripts/alerts.py (stdlib) is the cron pre-run gate:
Alertmanager active alerts minus heartbeats and null-routed ones, plus
ALERTS blips of the last 75 min, minus open [alert] issues (paginated, the
pipeline read only the first 100) and noise fingerprints. Nothing new ->
{"wakeAgent": false}, no LLM run, no message. `file` and `noise` write the
verdicts in the fixed formats. SKILL.md ports the triage prompt: wiki
first, verify live (kubectl, Prometheus, Loki, talosctl), problem/noise,
root cause vs symptoms; Matrix report per run that found something.
- Cron job alert-monitor, hourly at :20, deliver matrix. The init container
(renamed cron-jobs) now ensures both jobs by name and copies the gate
script to /opt/data/scripts (Hermes only runs scripts from there).
- HermesDown: blackbox probe of https://hermes.renner.uno/api/status plus an
alert, since a Hermes outage now stops alert -> issue (Alertmanager email
is unaffected).
- Removed .woodpecker/alert-monitor.yml and ci/agents/alert-monitor/;
comments that named them updated.
Tested: the gate run inside the hermes pod against live Alertmanager,
Prometheus and Gitea: 2 active, 36 open issues, 50 noise suppressions, 0 new
-> wakeAgent false. With a 3-day lookback and no noise TTL it produced 4
blips with the expected titles. file/noise against a fake Gitea (61 issues,
pagination, duplicate title skipped, footer, noise log created then
appended, parsed back by the gate; bad title/body/fingerprint refused).
Init script run twice on a scratch HERMES_HOME holding an existing
ha-updates job: ha-updates edited in place, alert-monitor created once,
script copied. Blackbox probe of the new target: 200. kustomize, citations
and server dry-run pass.
After merge:
- Delete the Woodpecker cron alert-monitor (id 4); without its pipeline file
it has nothing to run.
- The alert_monitor_bot_token secret and the alert-monitor-bot account are
unused afterwards.
Trade-off: the pipeline posted with alert-monitor-bot, a PAT limited to
write:issue. Hermes posts as hermes-bot, which also has repo write; the
agent reads untrusted alert text with a broader credential than before.
Every host the NetBird proxies serve (exactly the Ingresses labelled
dns.renner.uno/public or netbird.renner.uno/public; verified by SNI probe of
proxy-1 and proxy-2) gets a group with three kinds of check:
- DNS: cluster services have no own record, so the check asserts the
*.renner.uno wildcard answers (CNAME == proxy.renner.uno, Cloudflare
resolver, IPv6 first). External names (apex, proxies, third-party) assert
their own AAAA/A record instead.
- HTTP reachability: a URL routed to the backend, redirects not followed,
any status below 500 counts.
- Response validation: health/API JSON or a page keyword; services fully
behind the Authentik gate assert the login page.
The proxies get their own group: proxy.renner.uno, and per instance its
DNS plus reachability and a Gatus health check pinned to that proxy (URL
host is the proxy, Host header status.renner.uno; Node derives SNI from the
Host header, so the proxy routes it like a client request).
Layout: monitors live in the kustomization of the app they monitor;
external ones (proxies, apex, third-party) in apps/monitoring. The CRD moves
to infrastructure/ so every app Kustomization can consume it; it carries
prune: disabled, so leaving kuma-sync's inventory does not delete it. The
old kuma-sync/monitors objects go; their Kuma monitors are recreated under
the new owner keys (history since this morning is lost).
App Kustomizations wait on their resources, and KumaMonitor Ready reflects
whether Uptime Kuma on the VPS is reachable. Each consuming Kustomization
therefore treats KumaMonitor as always current, so a Kuma outage cannot
stall Flux.
kuma-sync: dns monitors gain spec.dns.conditions (Kuma's "record"
variable with its string operators, ANDed), mapped both ways so drift is
detected and an emptied list clears Kuma's conditions. Until the new image
rolls out the controller drops the field and the DNS checks only test
resolvability.
Pipeline branch filters and PR bases, Flux GitRepository ref, image automation
checkout and push, restore-test, vmtest and bootstrap.sh. Merge right before
the master -> main rename.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Flux source URL, flux-git rotator verify repo, review-bot REPOS, pipeline API,
clone and push URLs, the heating-can source label and the review-bot and
kuma-sync Go module paths. Merge right after the transfer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drops the daily cron and the upstream-polling decide step. Every version
in ci/images/hermes/Containerfile is now an image reference (FROM for
nousresearch/hermes-agent, COPY --from for talosctl, kubectl, flux, bao
and tea), which Renovate's enabled dockerfile manager updates and
digest-pins. A merged Renovate PR pushes to ci/images/hermes/**, the
pipeline builds build.<unix-ts>, and Flux image automation rolls it out.
No checksums to maintain: the binaries come from digest-pinned images.
renovate.json: the hermes Containerfile gets a PR of its own instead of
the "CI non-major" group (that group is for changes that never reach the
cluster), and nousresearch/hermes-agent is held to v0.x because the
retired date-versioned v2026.x tags sort above it.
Verified: each COPY --from path exists in the pinned image and the
binary runs on the hermes-agent v0.22.2 base. renovate-config-validator
37.440.7 reports only the two pre-existing errors on master
(flux/kubernetes.managerFilePatterns).
nousresearch/hermes-agent ships no talosctl, kubectl, flux, bao, tea or
jq. The kubectl init container dropped its binary in /opt/data/.local/bin,
which the agent's terminal never has on PATH: Hermes snapshots the env
from a bash login shell, and Debian's /etc/profile resets PATH for
non-root users. /usr/local/bin survives that reset.
Upstream's documented route for always-present tools is a derived image
(docs/user-guide/docker, "Adding tools"). ci/images/hermes builds FROM
the upstream release and adds the binaries, versions matched to the
cluster (Talos v1.13.10, Kubernetes v1.36.3, Flux v2.9.5, OpenBao 2.7.1)
and checksummed against upstream release checksums.
.woodpecker/hermes-image.yml rebuilds daily when upstream publishes a
new release (compared against the uno.renner.hermes.version label on
:latest), and on any push to the Containerfile. Tags are
build.<unix-ts>, the scheme Flux image automation already tracks for
wiki and shelly-manager.
Owner step after merge: register the cron
woodpecker repo cron add --repo k.renner/homelab \
--name hermes-image --branch master --schedule '@daily'
The Deployment switch follows in a separate PR once the first build
exists, so the Recreate rollout never points at a missing tag.
Kuma 2.5.6 applies "default enabled" notifications only in the web UI
(src/pages/EditMonitor.vue:4128-4132); the server's add handler stores
exactly the notificationIDList the client sends (server/server.js:748,791).
Monitors created by kuma-sync therefore had no notifications.
The notification set is now a managed field: exactly the IDs of all
notifications with isDefault set, sorted, applied on create and update
(group monitors included). Monitor.NotificationIDs is part of
Normalized(), so a changed default set or a hand-made change triggers an
update on the next resync and an unchanged one does not.
Client.DefaultNotifications reads the library's cached notificationList
(go-uptime-kuma-client v0.6.2 notification.go:17). Kuma has no command to
request the list; it pushes it at login and to every session of the user
after each change (server/client.js:31, server/server.js:1573,1594,1836).
A failed read is KumaUnavailable with no create or update.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After a Kuma database loss or restore, IDs restart and a resource's
status.monitorID can point at a monitor that now carries another
resource's ownership tag. That proves our monitor is gone, so the
controller creates a new one instead of parking at NotOwned forever.
Only an untagged monitor at the previous ID still blocks the create
(tag removed by hand), as before.
Residual Important finding from the kuma-sync final review.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
renovate-bot
k.renner
hermes-bot
claude-botandClaude Opus 5.5