The archive was fetched into the controller's /tmp, so every run after a
/tmp cleanup downloaded it again and reported a change. It now lives in
~/.cache/ansible/crowdsec, like the netbird-client download.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- ssh-ca reloads sshd only when the CA key, the Include or the drop-in
changed; principals and revoked keys are read per login.
- On DSM the sudo rules are checked with visudo -c -f - on the
controller instead of a temporary file, which reported three changes
per run.
- Facts are read through ansible_facts (INJECT_FACTS_AS_VARS
deprecation) in common, qemu-guest-agent and keepalived-ntp.
- common masks systemd-timesyncd on guests only; a KVM host such as a
NAS running VMM has no such unit.
- netbird-client uses deb822_repository (apt_repository deprecation) and
removes the old one-line source.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
root on nas-1 accepts no owner key or certificate yet, so the first
run has to go through ladmin with its become password.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With the SSH CA in place, root accepts the owner certificate on DSM
(PermitRootLogin without-password), so runs need no become password,
as on every other host.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nas-1 is powered on again. It joins [synology_nas] and [ssh_ca] like
nas-2; the roles and plays dropped with it in 9c9e45d1 stay dropped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On DSM claude-bot is in administrators, whose sudo rule asks for a
password nobody knows. ssh_ca_claude_sudo_all (default false) adds a
passwordless rule; synology_nas sets it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nas-2 (Synology DSM 7.3) joins [ssh_ca] under a new [synology_nas] group,
so claude-bot reaches it with its 1h certificate like every other host.
DSM differs from the fleet in ways the ssh-ca role now handles:
- Python 3.8 only; plays/configure-synology-python.yml installs Synology's
Python3.9 package via raw, and group_vars point the interpreter at it.
- No getent: admin logins come from /etc/passwd on every host.
- No useradd: the account is made with synouser and added to
administrators (the only group DSM lets in over SSH). synogroup --member
replaces the list, so the current members are read and passed along,
and the run stops if the owner login is not among them.
- No visudo: the sudo rules are rendered and checked on the controller.
- sshd_config has no Include; the role puts it first.
- No sftp/scp for ladmin: transfers are piped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
App endpoints are <function>.<app>.renner.uno (ftp.paperless.renner.uno,
syncthing.paperless.renner.uno).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
[::]:8082 also listened on net1. POD_IPS (status.podIPs) lets the app bind
8080/8082/9090 to the pod IPv4 and IPv6 only; ASPNETCORE_URLS falls back
to the pod IPv4 for images that do not read POD_IPS.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Captured on k8s-1 (cilium-dbg monitor, endpoint 1708): an IPv6 SYN from
edge-1 (fd02:30::/64) to fd99::8a reached the pod on 8082, but no reply
left via eth0; the pod has on-link and default IPv6 routes on net1 from
RAs. sbr only confines IPv4. Turn RA processing off on net1 with the
tuning plugin (as matter-server already uses it) and roll the pod.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The auto-assigned IPv4 address 10.2.99.0 timed out from the LAN while
the pod answered on 8082 directly; pin 10.2.99.90/fd99::8a like the
other bgp-pool Services. Kestrel listened on 8082 only on the pod IPv4,
so the IPv6 endpoint had no listener; bind 8082 on [::].
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
shelly-manager now checks and rolls out Shelly firmware itself
(konairius/shelly-manager#121). Devices download from it on :8082 via a
BGP LoadBalancer named shelly-firmware.renner.uno; it reads Shelly's
index from *.shelly.cloud over HTTPS; /metrics on :9090 is scraped and
alerts on halted rollouts, a stale index and invalid rollout config.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pocket Casts now serves an Up Next recommendation row whose single item is
the Up Next playlist, titled with the episode at the head of the queue
(podcast as subtitle, podcast artwork). The frontend builds Top Picks for
You from the first items of the visible recommendation rows, so the head of
Up Next shows there; clicking or playing it loads the whole playlist. An
empty Up Next yields an empty row, which the frontend hides together with
the pick.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The k8s-outposts blueprint pinned the patched LDAP outpost image by hand.
Flux setters cannot patch text inside the blueprint ConfigMap string, and
Gitea's cleanup rule deletes a pinned tag once a newer build ages it out.
The image now lives in the authentik Flux Kustomization's
postBuild.substitute, with the authentik-samba-bridge-ldap-outpost
ImagePolicy marker, and the blueprint reads ${SAMBA_BRIDGE_LDAP_OUTPOST_IMAGE}.
kustomize-controller renders the literal before Authentik reads the file.
`flux build kustomization authentik` differs from the unsubstituted render
only in that line; kubernetes/apps/authentik has no other ${...} literals.
The SAMBA_BRIDGE_LDAP_OUTPOST_IMAGE env var in global.env, used by the
earlier !Env attempt, is removed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#1510 set the k8s-outposts blueprint's container_image through
`!Env SAMBA_BRIDGE_LDAP_OUTPOST_IMAGE`. It did not resolve to the patched
image: at 07:07:39Z the outpost controller recreated
ak-outpost-ldap-outpost with Authentik's default
ghcr.io/goauthentik/ldap:2026.8.4, although every worker carries the env
var. The upstream outpost lacks the sambaSamAccount/sambaDomain object
classes DSM's smbd queries for.
container_image is a literal again, on the samba-bridge outpost build that
exists now (2026-10-28-0655).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Gitea's konairius package cleanup rule keeps the most recent version and
counts every tag as a version. Builds that push several tags (build.<ts>,
sha, main, latest) keep only the last-pushed one, so the tag a workload
runs gets deleted: vsftpd kept only `main` and ImagePolicy vsftpd fails
with "version list argument cannot be empty" (#1509); ical-proxy's pinned
main-926ead0 is gone too.
Every konairius image is now pushed with exactly one tag, YYYY-MM-DD-HHMM
(UTC), the scheme review-bot and the gitea_admin images use, and every
consumer follows it through an ImagePolicy instead of a hand-written pin:
- hermes-image.yml: date tag only, no :latest.
- build-ci-images.yml: alpine-jq, cni-plugins, alpine-openssl tag
YYYY-MM-DD-HHMM; the konairius/ and ghcr copies drop :latest.
gitea_admin/ copies are unchanged.
- ImagePolicies for hermes, wiki, shelly-manager, vsftpd switch to the
date filter (alphabetical, accepts YYYY-MM-DD for the older utility
builds). New policies and markers: ical-proxy, mqtt-account-controller,
authentik-samba-bridge-server (HelmRelease tag), the samba-bridge LDAP
outpost, alpine-jq, alpine-openssl, cni-plugins.
- The outpost image moves out of the blueprint string, which Flux setters
cannot patch, into global.env SAMBA_BRIDGE_LDAP_OUTPOST_IMAGE, read by
the blueprint via !Env (same pattern as LDAP_OUTPOST_TLS_* and
AUTHENTIK_GATE_SYNC_TOKEN).
- renovate.json: the new Flux-tracked images join the disabled list.
- Hermes homelab-gitea-pr guide: date tag, one tag, no hand pins.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
shelly-manager's /api/v1/mqtt/push is being removed in favour of
group settings (konairius/shelly-manager group-settings M3).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner decision on #1455 (2026-10-27): package retention moves to Gitea's
built-in package cleanup rules (docs.gitea.com/usage/packages/storage),
created in the UI; Gitea 1.27.3 has no API for them.
Removed: package-retention.yaml (CronJob, ExternalSecret, ServiceAccount,
ClusterRole/Binding), package-retention.py and its configMapGenerator
entry, the gitea-package-retention-kube-apiserver CiliumNetworkPolicy, and
`gitea` from the openbao ClusterSecretStore allowlist. The retention
ExternalSecret was the only one in namespace gitea using that store;
gitea-ssh-ca uses its own namespaced SecretStore openbao-ssh-ca.
The job had failed nightly since 2026-10-18 with a dead token (hermes-bot
recreated, no longer site admin). Gitea's cleanup_packages cron in
statefulset.yaml stays: it runs the cleanup rules and frees blobs.
ImageRepository shelly-manager scans gitea.renner.uno/konairius/shelly-manager
and renovate's Flux-managed list names the new path. The Deployment line is
left to image automation: its marker is the full-reference form, so the first
konairius build.<ts> rewrites name and tag together. No manual tag.
konairius/shelly-manager already holds build.1791038240 (2026-10-03, older
than the running k.renner build.1792997278), so this must merge only after the
konairius/shelly-manager release pipeline has pushed a newer build there;
otherwise the policy would roll the Deployment back to the October image.
Owner decision on #1455: no packages under k.renner.
The bot posted COMMENT whenever the agent said comment, gave no
verdict, or approved with a red or yellow section. A COMMENT is no
review verdict, so the PR showed no approval or rejection by the bot.
Every review is now APPROVED or REQUEST_CHANGES. Only blocking findings
request changes: the verdict request-changes, a Blocking section, or an
empty change. Suggestions, nits and unverified points stay in the body
of an approval. An answer without a verdict line is not posted; the
review fails instead of guessing. The prompt offers only request-changes
and approve.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The sweep leaves queued and running reviews alone. Enqueueing one
marked it for another run when it ended, so a failing review was
chained back-to-back and FAILURE_COOLDOWN never applied.
- Mistral's read-only check accepts _compare_, so a re-review that
calls compare_commits is not rejected as a violation.
- SELECTOR_MIN_CONFIDENCE=NaN is rejected instead of disabling the
fallback.
- The prompt always lists recent base commits on the changed files, so
a change already on the base is still caught when mergeable is true.
- The requested-review search pages until an empty page.
- Doc comments describe the request-only, any-repository trigger.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
REPOS is gone. The webhook takes a review request from any repository,
and the sweep asks Gitea for every open pull request that requests a
review from the bot (issues/search review_requested=true), so a repo
needs no webhook and no config entry: requesting review-bot is enough.
The sweep runs every 5 minutes, since it is now one search plus two
reads per requested PR. A head whose review failed is left to the next
request or FAILURE_COOLDOWN (1h), so a failing review is not rerun on
every sweep.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reviews start only when review-bot is a requested reviewer: the
pull_request webhook with action review_requested for the bot, or the
sweep finding the bot in requested_reviewers. Gitea drops the request
when the bot posts its review, so each request yields one review. Pushes
and opened PRs alone start nothing.
The prompt reads the diff first, opens whole files only where the hunks
are not enough, reads the base side only when mergeable is false or the
base moved, and aims at 10-20 tool calls. Re-reviews read only the
commits since the last marker via compare_commits, now an enabled tool.
Every review logs and exports input, output and cache tokens
(review_bot_tokens_total, review_bot_tool_calls_total).
The selector falls back to the default model when it picks another one
with less than SELECTOR_MIN_CONFIDENCE (0.6), and asks for the cheapest
capable model. Sonnet is the default, Opus the exception, Haiku is gone
(41-50 tool calls on trivial PRs), Mistral stays as a separate budget.
Anthropic effort defaults to medium.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Shelly devices are not Ansible-managed. Drops
plays/configure-shelly-ntp.yml, its only helper files/discover_shelly.py,
and the mention in site.yml.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
install-ntp-ha.yml:
- 'chronyc clients | head -10' could not fail: without pipefail the rc
is head's, and the result was never used. 'chronyc -n tracking'
fails when chronyd does not answer.
- The udp/123 check had failed_when: false and only printed
LISTENING/NOT LISTENING. 'ss -Hlunp sport = :123' now fails unless
chronyd owns the socket.
- The v4/v6 VIP presence checks had failed_when: false, so an ip
error (e.g. a wrong interface name) read as 'BACKUP'. They now fail
on a non-zero rc; an empty result still means BACKUP.
update-dist-upgrade.yml: failed_when already passes 'No new release
found'; ignore_errors: true on top of it turned a failed
do-release-upgrade into 'ignored' and the play went on to reboot.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
chronyc -h <vip> talks to chronyd's command port, which chrony.conf binds
to loopback only, so it always returned '506 Cannot talk to daemon'. The
'|| echo' turned that into rc 0, so the task reported ok on every run.
chronyd -Q with a server directive on the command line does a one-shot
NTP query: no config file means no pidfile, and client-only mode zeroes
the NTP and command ports, so it runs alongside the live daemon. -t 10
makes it fail rather than hang when the VIP does not answer.
The VIP presence check used 'ip addr show | grep 10.2.40.4', which also
matches 10.2.40.40-49; it now uses 'ip -4 addr show to <vip>' like the
v6 check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The dashboard builds its Content-Security-Policy from AUTH_AUTHORITY, which
for Authentik is a path (https://auth.renner.uno/application/o/netbird/).
A CSP source with a path only matches URLs below it, so the browser blocked
the token endpoint (/application/o/token/): no code exchange, no refresh,
and the dashboard showed "Error: Unauthenticated" until site data was
cleared. Verified in Chromium against the live header; NETBIRD_CSP adds
https://auth.renner.uno, derived from netbird_auth_issuer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- crowdsec-edge-bouncer, netbird-client: the controller downloads also run
in check mode; they only write on the controller and the next step needs
the archive
- uptime-kuma: reads and writes OpenBao through its HTTP API (uri) instead
of community.hashi_vault, so the controller needs no hvac library. The
token comes from VAULT_TOKEN or ~/.vault-token, as with the bao CLI; the
owner OIDC role gets policy uptime-kuma-vps.
- deploy-uptime-kuma pipeline removed: its deploy step was gone, only the
renewal cron for the CI token remained.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Check mode skips command and uri tasks, so every assert or condition on a
read-only result failed or saw nothing (ssh-ca: "dict has no attribute
content" on the CA key). The tasks that only read state run in check mode
too (check_mode: false): the CA key fetch, sshd unit lookup, key
fingerprints, sshd listeners, netbird status, grep/pgrep/ip/chronyc/
timedatectl checks. The sshd -T assert is skipped in check mode, as sshd
reads the files check mode has not written.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
It was never installed on any managed host (/opt/node-exporter absent on
all four, site.yml --check) and Prometheus scrapes no :9100 target outside
the cluster. The role also downloaded from github.com on the host, which
proxy-1 (IPv6-only) cannot reach. Play and role removed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ansible-playbook site.yml, from ansible/, applies every state play in
dependency order; the inventory and vault password come from ansible.cfg,
now as paths relative to it. One-off actions (update-os,
update-dist-upgrade, rotate-netbird-secrets, configure-shelly-ntp) stay
separate.
- gathering=smart: facts once per host and run, not per play
- ControlPersist=30m so a run costs one YubiKey PIN per host;
plays/close-connections.yml ends site.yml and the action playbooks and
closes the connections
- qemu-guest-agent only on VMs (edge-1 is bare metal)
- removed: edge-mqtt-bridge.yml (install-edge-host.yml carries the role,
sml2mqtt is gone from edge-1) and the decommissioned claude-ws (inventory,
group_vars, role, play)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
service_facts took 3.5-11.7 s per host (profile over the four [ssh_ca]
hosts: 32 s of a 62 s run). The role only needs the running sshd unit,
which systemctl list-units answers in milliseconds; on Ubuntu the
sshd.service alias resolves to ssh.service.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Without it every task is a temp dir, an sftp upload, a chmod, the run and
a cleanup, each a separate SSH round trip. Measured over the mesh against
netbird.renner.uno with a warm ControlMaster: 1.85 s per command task
without pipelining, 0.85 s with it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
edge-1 and netbird accepted password authentication (OpenSSH default; no
config line sets it), the proxies did not. The ssh-ca drop-in turns off
password and keyboard-interactive authentication and the role asserts it
in sshd -T. The edge-host ISO renders the same drop-in.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replaces the docker-logs flag with one fixed set for every [ssh_ca] host:
- systemd-journal group instead of sudo journalctl/systemctl status, which
also allowed journalctl --vacuum-*/--rotate
- sudo: ss -tulpn, sshd -T, nft list ruleset (fixed args: nft joins argv
into one command line), netbird status [*], docker ps/logs/stats and
docker restart (no docker start: -a/-i attach to stdin)
- per-host restartable units: proxies netbird, netbird-proxy-flood-watchdog,
crowdsec-firewall-bouncer; netbird netbird; edge-1 mosquitto,
edge-mqtt-cred, edge-alloy, keepalived, chronyd
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The NetBird services on netbird.renner.uno log only to docker, which
claude-bot cannot reach. New role flag ssh_ca_claude_docker_logs adds
sudo for docker ps and docker logs; enabled for netbird.renner.uno.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
proxy-1 and proxy-2 were verified: only *%wt0:22 listens and the public
addresses refuse port 22.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From a workstation every new SSH login costs a YubiKey PIN and touch.
ssh-mesh-only reconnected on every run; the reconnect now sits in an
include that runs only when sshd is switched, since meta reset_connection
ignores when:. configure-ssh-ca is one play: with serial: 1, the second
play reached each host after its 60 s ControlPersist had expired, so it
logged in again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
proxy-2 was verified: only *%wt0:22 listens, the public IPv4 and IPv6
addresses refuse port 22, and mesh SSH works after a full reboot.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New role ssh-mesh-only and play configure-ssh-mesh-only.yml. sshd moves to a
systemd socket with BindToDevice=wt0: on Arch a new sshd.socket (Accept=yes,
packaged sshd@.service), on Ubuntu a drop-in for ssh.socket that masks the
sshd-socket-generator output, plus BindsTo= so ssh.service stops with it.
NetBird recreates wt0 on every engine restart; BindToDevice stops the socket
with the old interface and a udev SYSTEMD_WANTS rule starts it with the new
one. The socket is not in sockets.target, which would hold boot until wt0.
The switchover runs in a transient unit and arms a revert timer first; the
play cancels it only after logging in again through the new listener and
seeing no listener outside wt0. [ssh_mesh_only] starts with proxy-2.
ssh-ca reloads sshd only when one sshd unit is running, as there is none
with per-connection or stopped socket-activated sshd.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The build fetches the CA public key from the ssh-ca role URL and renders the
role drop-in, revocation list and root principal into the live system;
edge-install copies them to the installed system and checks sshd -T uses
the CA. Owner certificates log in as root from the first boot, with the
break-glass key as fallback.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The edge-host ISO carried krenner@fortress, ansible-runner@ci-new and
krenner@krenner-XPS-13-9310; it now carries only the break-glass key.
The netbird-server revoke task for netbird-backup@homelab-cluster goes:
the exclusive managed authorized_keys already excludes it. OpenTofu drops
the unused ssh_public_key/ssh_private_key variables and the Proxmox
provider ssh block.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The woodpecker-ci@homelab key is dropped from the managed authorized_keys
and from the edge-host ISO. The Ansible pipelines that used it are deleted;
plays run from a workstation. deploy-uptime-kuma keeps only the OpenBao
token renewal cron, and deploy-infrastructure no longer installs the key
(no root-module resource uses the Proxmox provider SSH).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
proxy-1, proxy-2 and netbird are addressed by their mesh names, so Ansible
for them runs from a mesh-connected workstation. CI cannot reach the mesh;
the pipelines targeting these hosts lose their push triggers and run only
on manual deployment.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first workstation run failed on proxy-1 with 'Network is
unreachable' fetching the release: proxy-1 is IPv6-only and github.com
has no IPv6 address. The tarball is now downloaded and checksum-verified
on the controller (~/.cache/ansible/netbird) and unarchive copies it to
the host, so the hosts need no route to GitHub.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the workstation each new SSH session asks for the YubiKey PIN and
a touch; with the default forks both proxies prompted at once. serial: 1
like the other plays that touch the proxies.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The play gathered facts on every inventory host before skipping those
without netbird_setup_key, so a workstation run opened an SSH session
(YubiKey PIN + touch) to edge-1 and netbird too, and timed out on the
decommissioned claude-ws. Facts are gathered after the skip now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ansible.cfg forced ~/.ssh/id_ed25519, which does not exist on the
owner's workstation: runs from there failed with 'no such identity'
and bypassed the ssh config that picks the plugged-in YubiKey's key
and certificate. CI never read this file ('world writable directory,
ignoring it as an ansible.cfg source'); its pipelines write the CI key
to ~/.ssh/id_ed25519, ssh's default identity, so they are unaffected.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Step 1 of SSH-only-over-NetBird: proxy-1/2 (Arch) need a host NetBird
client with wt0 before sshd can stop listening on their public
addresses. The role was apt-only.
- Arch: release tarball from GitHub, verified against the release's
checksums file (get_url checksum URL), binary to /usr/local/bin,
'netbird service install'. Version netbird_client_version, 0.75.0
like netbird-ingress; Debian/Ubuntu keep following the apt repo.
- netbird_disable_dns adds --disable-dns; set for [netbird_proxy],
whose resolv.conf the netbird-proxy role pins.
- The service is started before 'netbird up' (the Arch path has no
package that starts it).
- The connected check matched 'Connected' inside 'Disconnected'; it
now matches the 'Management: Connected' line.
- 'netbird up' runs with no_log: its argv holds the setup key.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review of #1481 (merged with it open):
- The Job printed the operator's streaming command with the pod's
HOME expanded (/work/.config/...), unusable on the workstation. The
defaults stay a literal ~ now; eval expands it on the workstation.
- 'Collecting secrets' captured stderr, where age-plugin-yubikey asks
for the PIN and touch: a silent hang. stderr stays on the terminal.
- Runbook/migration doc still named pass as the DR store in places;
tooling list gains age + age-plugin-yubikey; the file list gains
hetzner-cloud-api-secret.age.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The cold-rebuild secrets (seal key, Hetzner restore keys, GHCR pull
token) came from the pass store: one gpg key file on the owner's laptop,
readable by any process of that user while gpg-agent caches the
passphrase, and with no second copy.
bootstrap.sh now decrypts ~/.config/homelab-secrets/<name>.age, files
encrypted to both YubiKeys (touch) and an offline paper key; a copy is
on nas-2. HOMELAB_AGE_IDENTITY switches to the paper key when both
sticks are gone. The in-cluster Job path is unchanged: the printed
streaming command is the age one. Runbook and the two OpenBao alert
texts point at the new files.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
proxy-1 is verified after #1477/#1479: authorized_keys holds exactly
breakglass-neo-6532348 and the CI key, and CI still reaches it.
Removed with this run, per the [ssh_ca] report and the sshd journals
(no root login with any of them except id_ecdsa, the owner's old key):
- proxy-2: id_ecdsa, ansible-runner@ci-new
- netbird: id_ecdsa, ansible-runner@ci-new, krenner@fortress,
netbird-tunnel (SHA256:cNy8...), a second woodpecker-ci key
(SHA256:hQxz...); none used for a root login since 2026-08-29
- edge-1: id_ecdsa, ansible-runner@ci-new, krenner@fortress
The CI key keeps from= the pod network on edge-1 (LAN, CI arrives from
10.244.2.x) and has none on proxy-2 and netbird (CI arrives NATed).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replaces the per-service Music Assistant/Immich parser whitelists with
one postoverflow whitelist: http-crawl-non_statics overflows are dropped
unless at least half of the requests in the alert are 4xx.
Every self-ban in the alert history (MA 350/368/477, Immich 149/150) was
a burst of 200s from a signed-in UI. Every scanner ban on this scenario
was 404s (/.env, /.git/...). The per-service rules only covered apps
that had already failed and muted all scenarios on their paths; this
covers any image-heavy UI and touches only this scenario.
Verified in a crowdsecurity/crowdsec:v1.8.1 replay: a 60x200 burst is
whitelisted, a 60x404 burst with distinct paths still overflows and is
not whitelisted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#1477 locked CI out of proxy-1: the CI key got from= the pod network,
but CI reaches the internet hosts NATed from the home WAN address
(proxy-2 journal: root logins from 84.171.37.13). The from= came from
plays/configure-ssh-keys.yml, which no pipeline ever ran, so it was
never in effect on those hosts. LAN hosts do see the pod address
(edge-1 journal: 10.244.2.x).
ssh_ci_key_from is now per host: the pod CIDRs by default, empty for
[netbird_proxy] and netbird.renner.uno. The owner already removed the
from= on proxy-1 by hand; this keeps the next run from restoring it.
Restricting SSH on internet hosts to the NetBird mesh is the follow-up.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Scrolling the Immich timeline fetches a /api/timeline/bucket and then
every /api/assets/<uuid> in it; 46-54 requests in 3-9s fired
crowdsecurity/http-crawl-non_statics against a residential Telekom IPv6
(alerts 149/150, 2026-10-23).
Same parser-stage pattern as the Music Assistant whitelist, scoped to
the Immich Traefik router and those two path prefixes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The MA UI loads every cover through /imageproxy/<hash>?size=512 in one
burst; a library view (47 requests in 3s) fired
crowdsecurity/http-crawl-non_statics and banned a residential Telekom
IPv6 (alert 477, 2026-10-27).
Parser-stage whitelist scoped to the MA Traefik router and the
/imageproxy/ path. The Traefik parser sets no target_fqdn, so the
router name is used as host scope.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
First host in [ssh_keys_managed]. root on proxy-1 currently has
id_ecdsa (krenner@krenner-XPS-13-9310), woodpecker-ci@homelab
(SHA256:DQWPS...) and ansible-runner@ci-new (dead since #184). After
this run: breakglass-neo-6532348 and the same CI key with its from=
pod-network restriction; people log in with CA certificates. proxy-2
is the twin, so the public edge survives a mistake here.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 4 of the SSH CA, step 1. Nothing is removed from any host yet.
- ssh_admin_authorized_keys (group_vars/all) is the single list of
plain keys the admin logins may accept: the break-glass PIV key on
the offline YubiKey NEO and the CI deploy key (from= pod network).
- ssh-ca reports, on every [ssh_ca] host, the fingerprints of the
admin logins' current authorized_keys.
- configure-ssh-ca.yml gets a second play that sets the admin logins'
authorized_keys exclusively to that list on [ssh_keys_managed]. The
group is empty; hosts join one at a time after the report.
- Removed the three dead key lists: plays/configure-ssh-keys.yml (no
pipeline ran it), common_ssh_public_keys (defined, never used) and
opentofu/locals.tf common_ssh_keys (no reference). They carried
krenner@fortress, ansible-runner@ci-new and the laptop id_ecdsa key.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Step A is verified: Secret gitea-ssh-ca holds exactly the CA key
(SHA256:xEhuqx..., one line, 81 bytes like OpenBao's content-length),
so the webhook provider passes a text/plain body through unchanged.
- StatefulSet: SSH_TRUSTED_USER_CA_KEYS from the Secret,
SSH_AUTHORIZED_PRINCIPALS_ALLOW=username. The built-in SSH server
looks a cert's principals up as principal keys, so each user
registers their username once as a principal.
- ExternalSecret: trim the trailing newline before it reaches app.ini.
- Reloader annotation: Gitea reads env only at start, so a CA rotation
restarts it. The step A comment claimed the refresh alone reaches
Gitea; corrected.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 3, step A of the SSH CA: Gitea's built-in SSH server needs the CA
key as SSH_TRUSTED_USER_CA_KEYS. A namespaced ESO webhook SecretStore
reads OpenBao's unauthenticated /v1/ssh-client-signer/public_key from
the in-cluster Service into Secret gitea-ssh-ca, so the key has one
source. Gitea is not wired to it yet; step B follows once the Secret
is verified to hold exactly the CA key (the ESO docs do not say how a
text/plain body without result.jsonPath is handled).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
netbird verified (Ubuntu, ssh.service): claude-bot cert login, sudo
journalctl allowed, systemctl restart refused. edge-1 is the last
Ansible-managed host (OpenSSH_10.6 banner like the proxies).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The netbird run (pipeline 14795) failed at the reload: 'Could not find
the requested service sshd.service'. Ubuntu's service_facts list
sshd.service as a not-found alias next to the real ssh.service
(running; disabled because ssh.socket activates it), and the role only
checked that the name exists.
The unit is now the one of sshd.service/ssh.service whose state is
running, asserted to be exactly one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
proxy-2 verified (claude-bot cert login, sudo allow/deny as on
proxy-1). netbird is the first Ubuntu host (OpenSSH_9.6p1
Ubuntu-3ubuntu13.20): unit ssh.service, which the role picks from
service_facts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
proxy-1 is verified: owner cert login as root (sk-ed25519 cert) and
claude-bot cert login with the expected sudo rules both work. proxy-2
is its twin (same OpenSSH_10.6 banner).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The second deploy-ssh-ca run on proxy-1 failed at the assert although
all three settings were active: OpenSSH 10.6 prints them as
TrustedUserCAKeys, AuthorizedPrincipalsFile and RevokedKeys, the check
expected lowercase keywords. The run stopped before the reload, so the
running sshd still has the old config.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first deploy to proxy-1 failed at the reload: 'Could not find the
requested service ssh'. proxy-1 is not Debian (banner OpenSSH_10.6
without distro suffix, root shell /usr/bin/bash), so the unit is
sshd.service; the handler only special-cased RedHat.
The unit is now picked from service_facts. The reload runs on every
run instead of via a handler: sshd -T only parses the files, so after
a failed reload a re-run with nothing changed would otherwise leave the
running daemon on the old config.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
claude-ws is decommissioned. proxy-1 resolves by name, answers on 22,
and has proxy-2 as its twin, so the public edge survives a bad run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New role ssh-ca with play configure-ssh-ca.yml (serial: 1) and pipeline
deploy-ssh-ca. Hosts in the new [ssh_ca] group get:
- TrustedUserCAKeys from OpenBao's public_key endpoint, fetched on the
controller at run time (no copy in git).
- AuthorizedPrincipalsFile: k.renner for root/ladmin (whichever
exists), claude-bot for Claude's own claude-bot account.
- RevokedKeys, always deployed: sshd refuses all key logins when the
file is set but missing.
- claude-bot sudo: read-only journalctl/systemctl status/ss/sshd -T,
restart/reload only for ssh_ca_claude_sudo_units; NOEXEC against
pager shell escapes; visudo-validated.
The drop-in is tested with sshd -t and removed again on failure; sshd -T
must show all three settings afterwards. authorized_keys is untouched,
so this only adds a login path. [ssh_ca] starts with claude-ws only.
ssh-sign-owner.hcl now also records the mount, CA and owner role
commands (review on #1466).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 1 of the SSH CA. The ssh-client-signer mount and its ed25519 CA
were set up by hand (root token); this records the roles and the
claude-bot signing policy.
- role owner: principal k.renner, 6-month certs. The mount was enabled
with -max-lease-ttl=4380h; the 768h system default would cap it.
- role claude: principal claude-bot, 1h certs, pty only.
- policy ssh-sign-claude for the claude-bot userpass user, next to its
existing claude-bot-write.
Verified: a YubiKey sk-ssh-ed25519 key signed on role owner yields a
sk-ssh-ed25519-cert-v01 cert, principal k.renner, valid to 2027-04-27.
CA fingerprint SHA256:xEhuqxqFVZ/ig6KpAtHOhLbwsIhyWpzXhvDcjIfM/Js,
served unauthenticated at /v1/ssh-client-signer/public_key.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 0 of the OpenBao SSH CA: humans log in to OpenBao with
Authentik instead of the root token.
- Authentik blueprint: group OpenBao Admins (k.renner), confidential
OAuth2 provider openbao (CLI + UI callbacks), application, group
binding. The client secret is generated by Authentik and only goes
into auth/oidc/config.
- openbao-authentik-egress: auth.renner.uno resolves to the Traefik
VIP in-cluster, which the existing world egress does not cover.
- Policy ssh-sign-owner with the manual bao commands for auth/oidc;
its sign path is enabled in phase 1.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Six retries at a fixed 10 s delay stopped after about 60 s, so
--retry-max-time 120 never bound. Twelve retries let the 120 s cap
be the limit, covering a Recreate rollout slowed by an image pull.
Retrying on 5xx is safe: POST /api/v1/sync is serialized by
SyncCoordinator and re-syncs the same state, the same as a webhook
landing mid-sync. --retry-max-time 120 keeps a long sync that fails
late from being repeated.
The wiki Deployment uses Recreate, so during an image rollout the Service
has no ready endpoint for ~10 s. wiki-sync-29884710 started 06:30:00Z, 4 s
before the new pod was Ready, failed with curl (7) and fired KubeJobFailed.
Retry up to 6 times, 10 s apart, on connection refused too, bounded to
120 s from the first attempt so a long sync that fails is not repeated.
Refs #1461
Chart, server/worker image and LDAP outpost image move together, as
the chart pin requires. Images built from authentik-samba-bridge
3c8f413 (PR #8), pipeline 43.
Supersedes #1347.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
aiosendspin starts its zeroconf on [bind_ip], or on InterfaceChoice.Default
when bind_ip is 0.0.0.0. Default egresses via the pod's main-table default
route (eth0); net1 routes sit in the sbr table. MA therefore never sent
_sendspin._tcp queries onto v30iot and only rediscovered Voice PEs when they
announced on their own, leaving them disconnected for hours after every MA
restart (e.g. 2026-10-26 13:56: 3 of 4 still offline at 15:05).
The reconcile initContainer now writes bind_ip = the live net1 address it
already computes for publish_ip. The address is read from the interface on
every boot, so the Sendspin no-bind-fallback failure (stale literal IP)
cannot recur.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Up Next payload carries no episode duration. Marking an episode played
in the UI reports the duration as the position, so the provider received 0
and took it for an explicit "mark unplayed": the episode stayed in Up Next
and was flagged unplayed.
The Up Next playlist now takes duration and playback status from the
(cached, throttled) episode details, falling back to the bare entry when
the details cannot be fetched.
Tested against the 2.10.7 source: 21/21 pocketcasts tests, ruff, mypy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Overlays the Pocket Casts provider with a patched __init__.py that adds
ProviderFeature.LIBRARY_PLAYLISTS and a read-only "Up Next" playlist of
podcast episodes, fetched live from /up_next/list in queue order.
An initContainer copies the image's provider into an emptyDir and swaps
in the patch only when the stock file's sha256 matches the version it was
written against (identical in 2.10.7 and 2.10.8). A later image bump that
changes the provider keeps the stock provider instead of breaking it.
Temporary until the change is upstreamed to music-assistant/server.
Tested against the 2.10.7 source: 18/18 pocketcasts tests, ruff, mypy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reverts #1451. The YouTube Music provider is not kept; the PO token
server and its image automation go with it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Renovate #1439 pinned a digest into the ImageRepository's spec.image,
which image-reflector rejects ("should not include a tag"), so Flux
stopped tracking gitea-mcp-server. The image was missing from the
renovate.json list of images Flux image automation already bumps; add
it there so Renovate leaves it alone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MA's YouTube Music provider needs a bgutil PO token server; MA installs
the matching yt-dlp plugin unpinned from PyPI (2.0.2 today), so the
server tracks 2.x via image automation. The MA docs' "1.2.1" predates
the 2.x plugin line.
Reachable only from the MA pod; egress limited to DNS and world:443.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Step 2 of 2. The coredns ConfigMap is a Talos bootstrap manifest; Flux
stops applying its own copy (prune is disabled since the previous
commit, so the live ConfigMap stays until talosctl upgrade-k8s writes
Talos's version). CoreDNS forwards to the node resolver, which asks
k8s-gateway first, so cluster DNS no longer depends on Flux.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Step 1 of 2 to hand the coredns ConfigMap back to Talos. It is a Talos
bootstrap manifest (talos-bootstrap-manifests-inventory) that Flux also
applies; talosctl upgrade-k8s reverts Flux's Corefile. Bootstrap already
runs on Talos's Corefile (forward to the node resolver, which asks
k8s-gateway first), and k8s-gateway carries serve_stale for every name,
so the Flux copy is redundant.
With prune disabled, step 2 can drop the file without Flux deleting the
live ConfigMap.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Since UniFi OS 6.0.15 / Network 11.0.97 (2026-10-25 18:42Z) the UDM's
DNS answers out of the interface a query arrived on. Pod traffic to
10.2.40.1 arrives on VLAN 40 (on-link route in table 47) while the pod
CIDRs are routed back over VLAN 47, so the UDM ARPs for the pod on
VLAN 40 and never answers (seen with talosctl pcap on k8s-3; TCP 443 to
the same address works, UDP and TCP 53 do not). fd00::1 was reached
over VLAN 47 and kept working.
k8s-gateway's upstream (k8s-gateway-site), the blackbox-dns probes and
the gatus UDM check now use the UDM's ingress-segment addresses,
fd02:47::1 and 10.2.47.1 (the egress-gw annotations); both answer from
a pod.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
syncthing's STUN NAT-type probe sends its change request to port 3479,
which syncthing-egress dropped (seen: 77.72.169.212:3479/UDP).
Loki tried to send usage reports to Grafana (34.49.170.206:443), denied
by its egress policy; switch reporting off instead of allowing it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
wiki-sync (every 30 min) and wiki-encyclopedia (06:00 Europe/Berlin) call
the service's API with a generated scheduler key; the service no longer
schedules anything itself (konairius/Wiki#18). Pods in ai already reach
the wiki through the namespace default policy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The controller (main-b1f294d, konairius/mqtt-account-controller#13)
delivers a password account to a namespace Secret. admin moves into the
register with its password in Secret mqtt-admin-password; Dynamic
Security is asked first, so it replaces the password_file login without
a broker restart. The controller gets get/create/patch/delete on that
Secret by name.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
shelly-manager resolves device *.local names via mDNS; the queries leave on
eth0 to 224.0.0.251:5353 and have been dropped since the egress lockdown
(#1432), failing with 'Name or service not known' since 15:07Z.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only members of the new Authentik group Wiki Readers may sign in to the wiki
OIDC application; membership is managed in Authentik, so the owner must be
added before merge. Encyclopedia is enabled with MaxArticles=0 so the first
nightly run only plans.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Since the broker restart on 2026-10-22 only register accounts connect; the
last legacy logins (homeassistant, sml2mqtt, open3e, zigbee2mqtt) were on
2026-10-08, and admin/evcc never logged in within the 30-day Loki window.
- Delete mosquitto-bootstrap.yaml (5 Jobs that apk add at runtime + RBAC).
- The broker's password_file/acl_file now hold only admin, the owner's
workstation login over the LoadBalancer. The controller has no delivery
mode that puts a password account's password into a Secret, and the cert
listener :8884 is in-cluster only, so admin cannot be a register account.
Its password comes from an ESO Password generator (CreatedOnce), the
pattern of apps/hermes/api-key.yaml, in Secret mosquitto-admin-password.
- Delete the evcc and health ExternalSecrets and the health rotator
CronJob/policy; nothing reads them.
- Delete PROPOSAL-mosquitto-credential-rotation.md; its subject is gone.
- MosquittoDown test masks with the dynsec-test pod instead of a
bootstrap Job pod; comments and docs follow.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Runtime package installs are not to be enabled by policy. The Jobs feed only
the unused legacy password_file and are removed in a separate PR, which must
merge before this one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Jobs run apk add curl jq on every execution; without world 443/80 they
fail and block the home-assistant Kustomization. Also correct the stale
renderer egress comment.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
edge-1.iot.renner.uno:8443 is the deliver.esp32 target in mqtt-accounts.yaml
(https+mTLS credential push). Matched by FQDN, not the current 10.2.30.108.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drop namespace-wide world egress and give each workload only the external
destinations it needs (FQDN where the host is in our config, port-scoped
world otherwise). Namespaces that had ingress-only policies get an
in-cluster + DNS egress baseline. CNPG instances get Hetzner S3 via one
clusterwide policy.
Out of scope: kube-system, flux-system, rook-ceph, woodpecker-ci,
music-assistant, evcc, openbao.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The embedded outpost is matched on the client too, so a filter the server ignored cannot select the LDAP outpost. A host named by two gated Ingresses is a GateError: the outpost finds providers by host, so it would serve one gate for both.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The gate is now declared on the app's own traefik Ingress: naming the
static Middleware authentik/gate (ForwardAuth against the embedded
outpost) in router.middlewares opts in, and its auth.renner.uno/*
annotations configure it. No generated route Ingress, ExternalName
Service or per-gate Middleware; the IngressClass authentik-gate is gone.
authentik-gate-sync keeps building the Authentik side (forward_single
provider on the embedded outpost, application, group policy) and the
backend CNP admitting Traefik, and sweeps the objects it generated
before. Traefik publishes the Ingress status itself.
The outpost's own paths (/outpost.goauthentik.io/) are one static
IngressRoute above the app routes. The embedded outpost answers 404 for
a host without a provider (checked live: auth/traefik with
X-Forwarded-Host no-such-gate.renner.uno), so a gate without its
provider stays closed.
docs.renner.uno points at paperless:80 behind redirect + gate (Paperless
Users) instead of the removed paperless-gate Service. Drops __pycache__
committed by mistake.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each gate was a single-replica proxy outpost in series with Traefik, so
any outpost restart, drain or node loss took its app down. Gates now
run as Traefik ForwardAuth against Authentik's embedded outpost inside
authentik-server: the request path is Traefik -> app, and the only auth
dependency is authentik-server, which every login already needs.
Per gated Ingress the controller now creates a forward_single provider
attached to the embedded outpost, a ForwardAuth Middleware first in the
route's chain, a /outpost.goauthentik.io route to authentik-server, and
a CNP admitting Traefik to the backend pods. It removes the per-gate
outposts and their egress CNPs.
Identity headers: authResponseHeadersRegex makes Traefik delete every
client-sent X-authentik-* header before copying the outpost's answer
(traefik pkg/middlewares/auth/forward.go), so a gated backend only ever
sees identity Authentik issued. Traefik answers 500 if the outpost is
unreachable and drops a router whose Middleware is missing: fail closed.
The blueprint no longer declares the embedded outpost: an entry must
carry providers, and re-applying [] would detach every gate. The
controller owns its providers and authentik_host/_browser.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The public Cloudflare wildcard CNAME *.renner.uno -> proxy.renner.uno and
its carve-out TXTs were removed on 2026-10-22. Public hosts now carry an
explicit CNAME to proxy.renner.uno created by external-dns from
dns.renner.uno/public; UDM records are opt-in via dns.renner.uno/internal.
Comment/doc text only; no config value changes:
- KumaMonitor, Ingress, HelmRelease and tofu comments that said hosts
resolve via the wildcard or that external-dns-unifi "skips" public hosts.
- PrometheusRule descriptions (k8s-gateway, dns-resolution) and the
matching promtool test strings.
- k8s-gateway cache comment: the cache tuning stays (dnsmasq refusal
bursts, 5s lease TTLs); the wildcard-trap rationale is gone.
- hermes skill references, claude-workstation runbook, crowdsec bouncer
defaults.
Not changed: kubernetes/apps/home-assistant/* (README rule 1 window) and
the wildcard sentence inside the CoreDNS Corefile data
(infrastructure/coredns-config), where an edit changes the ConfigMap.
Verified live before the edit: all 30 Kuma DNS monitors and all blackbox
DNS/HTTP probes green after the wildcard removal.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The controller's own NetBird client registered ~renner.uno in
systemd-resolved, pointing at the kube-dns-lb nameserver group, which is
unreachable from this peer. Every *.renner.uno lookup on the host returned
SERVFAIL, and Uptime Kuma, resolving through Docker's embedded DNS and the
host stub, marked all HTTP monitors DOWN after the 2026-10-22 update-os
reboot.
The role now reconnects the client with --disable-dns when its config has
DNS management enabled. The flag is only honoured on a fresh up, so the
task runs netbird down + up; it is a no-op once disableDns is true.
Applied live and verified 2026-10-22 before this commit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Step 2 of 2 of the label split (owner decision 2026-10-22). Requires
#1425 (step 1) to be applied first.
- external-dns-unifi: labelFilter dns.renner.uno/internal=true (opt-in).
Every object it publishes today carries the label since step 1, so the
UDM record set is unchanged.
- external-dns-cloudflare: labelFilter dns.renner.uno/public=true, default
target proxy.renner.uno (the cloudflare-lb-sync load balancer) instead of
the retired tunnel, no --cloudflare-proxied, policy sync. Each public
host gets a DNS-only CNAME to proxy.renner.uno plus its TXT registry
record; this is the prerequisite for removing the *.renner.uno wildcard.
- netbird-service-sync: reads netbird.renner.uno/public only. Every
dns-public Ingress carries it since step 1, so the 21 proxy services are
unchanged.
- git/gitea/gitea-mcp: add dns.renner.uno/public (public today via the
wildcard; keep their LAN record via dns.renner.uno/internal).
- Drop comments describing the old public!=true filter.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Step 1 of 2 of the label split, owner decision 2026-10-22:
dns.renner.uno/internal -> UniFi (LAN) DNS via external-dns-unifi
dns.renner.uno/public -> Cloudflare DNS via external-dns-cloudflare
netbird.renner.uno/public -> NetBird reverse-proxy service (netbird-service-sync)
Additive only; no consumer reads the new labels yet:
- dns.renner.uno/internal on every object external-dns-unifi publishes
today (24 Ingresses incl. Grafana via chart values, 6 LoadBalancer
Services = the 31 live UDM hostnames).
- netbird.renner.uno/public next to every dns.renner.uno/public label
(20 Ingresses). netbird-service-sync already ORs both labels, so its
desired set is unchanged.
authentik-gate-sync copies source labels to the generated *-gate Ingresses.
Step 2 flips external-dns-unifi to dns.renner.uno/internal=true,
netbird-service-sync to netbird.renner.uno/public only, and
external-dns-cloudflare to dns.renner.uno/public=true. It must merge only
after this change is applied, or external-dns-unifi (policy sync) would
delete LAN records.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#1423 shipped a placeholder tag (2026-10-22-0000) that was never pushed,
so the pod sits in ImagePullBackOff until image automation rescans.
2026-10-22-1904 is the tag pipeline 14713 pushed
(sha256:bef768a88c7b7bd30f7fcbf3f3fecfc58e8f7ff5e08e1c5f576f8043a9465f83).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New in-house controller (kuma-sync house pattern) that reconciles a
CloudflareLoadBalancer resource into a DNS-only Cloudflare load balancer,
its pool and its health monitor, using cloudflare-go v7.
proxy.renner.uno had round-robin A/AAAA records without health checks, so
every proxy reboot blackholed part of the public traffic. The owner set up
a DNS-only load balancer by hand (hostname origins proxy-1/proxy-2, HTTPS
monitor with Host: status.renner.uno); this moves it into git.
- Identity is the load balancer hostname; pool and monitor are found via
its references. Hand-made objects are adopted (the account allows only
2 origins, so a parallel pool would be rejected) and tagged in their
description; objects with another owner's tag are never written.
- Status shows per-origin health; resync every 2 minutes.
- Token from OpenBao homelab/infrastructure/k8s-cloudflare-lb via
ExternalSecret; egress limited to api.cloudflare.com:443.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CVE-2026-109618 and CVE-2026-109619 (BSI BITS-H 2026-297981-1132) allow
unauthenticated RCE on every PHP before 8.5.12/8.4.27/8.3.36/8.2.35;
mod_php is the highest-risk variant. The running pod reports PHP 8.3.35
and www.renner.uno serves / to the internet without the Authentik gate.
Switch to the php8.5-apache variant of the same WordPress release,
pinned to the index digest rebuilt on 2026-10-22 after the PHP fix.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review #1811: on Arch the reboot only fired when this run's pacman task
changed something, while the post-reboot check looks at host state. A
host whose newer kernel was installed by an interrupted earlier run (or
by hand) skipped the reboot, failed the kernel check and stopped the
whole run. The play now also reboots when /usr/lib/modules/$(uname -r)
is gone before the reboot step.
Also from the review:
- assert >= 1 GiB free on / before upgrading (edge-1 runs from a
6.6 GiB eMMC root, 2.1 GiB free on 2026-10-22)
- play comment names the groups it covers instead of 'every host'
- pipeline comment points at the README 02:00-05:00 window
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Since 52899ecb (#625) removed plays/update.yml and the update-all
pipeline together with the Proxmox fleet, nothing upgrades the remaining
Ansible hosts. proxy-1/2 and edge-1 (Arch) and netbird (Ubuntu) all run
kernels affected by CVE-2026-99999.
os-update role, per host:
- records failed systemd units and running docker containers
- Arch: archlinux-keyring first, then pacman -Syu
- Debian/Ubuntu: apt full-upgrade + autoremove
- reboots when pacman changed something / reboot-required is set
- asserts the running kernel is the installed one (Arch) or no reboot
is pending (Debian)
- waits until every previously running container is back and no new
failed units appear
update-os.yml runs netbird_proxy, then edge_host, then netbird_server,
serial: 1 with max_fail_percentage: 0, so the first failing host stops
the whole run before the next host is touched.
The pipeline is deployment-only (target update-os): edge-1 is the sole
NTP server and carries Z-Wave and the heating CAN bus, and netbird is
the control plane; neither may reboot on a cron.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fixes CVE-2026-99999 (ipv4: validate checksum_start before completing
checksum, CVSS 9.3). The nodes run 6.18.48-talos; the fix is in 6.18.57,
which v1.13.12 ships (siderolabs/pkgs@19e92dc).
Not applied to any node yet. Rollout is one node at a time: drain,
talosctl upgrade --image, wait for Ceph HEALTH_OK, starting with k8s-1.
Patch release within 1.13; Kubernetes stays v1.36.3. Schematic is
unchanged; factory installer manifest and ISO verified to exist for the
tag. talosctl checksum taken from the release's sha256sum.txt.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
www.multi-circuit-boards.eu (third-party Apache) occasionally resets a
single TLS handshake with ECONNRESET. Kuma 2.5.6 opens a fresh
connection for every check, so two such resets in a row were enough
for DOWN with retryInterval 60 / maxRetries 1: 14 false episodes since
2026-10-20 21:51 UTC. A 30-min probe from the cluster (559 requests at
3s) saw 557x 200 and 2 handshake resets, and got 200 during Kuma's
DOWN window.
With retryInterval 20 (CRD minimum) and maxRetries 3, DOWN now takes 4
failures in a row over at least 60s, so a real outage longer than
about 1 min still alerts. The other third-party monitors are left
unchanged.
Refs #1407
- predates() also guards approvals: an approval older than the card the PR
line names now (e.g. a follow-up whose expert rebutted without a push)
no longer completes that card on replay.
- predates() parses created_at as epoch or RFC 3339 and fails loudly when a
card has none (hermes kanban show --json gives epoch seconds, v0.22.2,
checked against the CLI), instead of silently never skipping.
- Closing one PR does not complete a card another open PR (stacked stage)
still names.
- Follow-up cards tell the worker to read every review and comment since the
previous card: only the newest finding is quoted, older ones that predate
the card are not added (deliberate, now pinned by a test).
review-bot 1802 on #1417 (blocking): marker dedupe was per card, but the PR's
Hermes-Card line moves (follow-up card, coordinator hand-off, gitea_pr.py
adopt), so a replay re-sent findings already handled on the earlier card to
the new one. A finding older than the card the line names now is skipped
(review submitted_at / comment created_at vs the card's created_at); the new
card's worker reads the PR itself.
Also from the review:
- PR merged or closed while its card is in review completes the card
(hook event pull_request_only, route event pull_request, closed hermes-bot
PRs of the replay window are replayed); a rebuttal without push otherwise
left the card in review forever once the owner merged.
- 404 on the PR (system hook fires for repos hermes-bot cannot read) is a
skip, not an error line per review.
- One route-script run per PR at a time (flock): a delivery and its replay
can overlap.
- find_review drops dismissed reviews like the replay does.
- Replay checks the run budget before listing; CronJob comment updated.
- Skill: stun-tls-probe.py runs via python3 (ConfigMap mode 0644); rebuttal
wording says the card waits for the owner.
Experts closed their card with kanban_complete as soon as the PR was open, and
nothing listened to Gitea reviews afterwards (#1412/#1414 REQUEST_CHANGES and
the review-bot suggestions on #1411/#1415 sat unanswered).
Owner decision: the card stays in review until the Gitea review is through; no
follow-up cards for a card that is still open.
- Herkunft: expert PRs carry `Hermes-Card: <card> (<profile>)`. The new helper
homelab-gitea-pr/scripts/gitea_pr.py sets it deterministically (open/adopt)
and refuses a second PR for the same branch or `Closes #N` (find/open).
The card id comes from --card, $HERMES_KANBAN_TASK or the scratch workspace
name: the terminal tool does not pass HERMES_KANBAN_TASK to commands.
- Events: the system hook adds pull_request_review and pull_request_comment
(Gitea 1.27.3 routers/api/v1/utils/hook.go; HasEvent maps the approved/
rejected/comment review types onto pull_request_review). Headers per
modules/webhook/type.go Event(): pull_request_approved/_rejected/_comment
for reviews, issue_comment for PR comments; the route lists exactly those.
- Routing in gitea_issues.py (no LLM, [SILENT], HMAC by the gateway as before):
the review payload carries no review id or commit, so the script reads the
reviews API and ignores reviews of an older head.
review-bot APPROVED on the head + card in review -> hermes kanban complete.
review-bot REQUEST_CHANGES/COMMENT, owner reviews with text, owner PR
comments -> card in review: hermes kanban reopen-review (back to the
implementer from review_requested, review quoted as untrusted text); card
open: comment; card done/missing: follow-up card for the same expert
(key <owner>/<repo>!<n>@after-<card>, needed because Hermes has no CLI to
reopen a done card) and the PR's line moves to it; PR without a line:
coordinator card (key ...@review). A marker per review/comment id makes
every replay a no-op.
`hermes kanban request-changes` is not usable here: it needs an active
reviewer run claimed from review, which only exists with review dispatch.
- kanban.review_dispatch=false on default (Hermes docs: "Set false for
human-only review boards"); otherwise every card in review would spawn an
sdlc-review LLM run next to review-bot.
- Replay step in gitea-webhook-sync: review-bot's latest review and owner
reviews of each open hermes-bot PR's head plus owner PR comments (48 h) are
re-delivered every 15 min. review-bot can approve before the expert calls
kanban_request_review; without the replay that card would stay in review.
- homelab-gitea-pr moves from the coordinator's PVC into git (experts never
had it): review-loop section, one-PR-per-cause rule, the helper. SOUL.md and
the bridge's card text say: PR cards end with kanban_request_review.
Addresses review-bot review 1792 on a1b52f5b:
- K8sGatewayProbesAbsent now fires per record (A, AAAA), so one broken
module no longer hides behind the other's series. New promtool case
(AAAA-only input) and mutation k8s-gateway-absent-aaaa-dropped.
- K8sGatewayIngressAnswerWrong description names the other cause: the
auth.renner.uno Ingress (authentik) being gone.
- The Traefik lb-ipam-ips annotation points back at the blackbox modules
that match the same VIP.
Folds the good parts of #1412 into this PR and addresses both review-bot
reviews (#1412 review 1789, #1414 review 1791):
- Everything after the OpenBao store is best-effort. Decode, reconnect,
listing and revokes run in reap_superseded(); reap_summary() turns any
error into "reap skipped" plus a warning, so rotate_ha_token returns Ok.
A failed run would start a second pod (backoffLimit: 1) that keeps the
token just stored and its successor and reaps the token consumers still
hold until ESO syncs, i.e. the #1081 outage. Two unit tests cover it.
- Warn when the replaced token is not named "<client_name> [<6 hex>]":
no later run reaps it, so it needs a manual revoke (from #1412).
- Staged image rollout (from #1412): only the three HA CronJobs carry the
flux-system:secret-rotator marker; the other 17 rotators stay on
2026-07-30 until a manual HA run validates the new binary, then a
follow-up adds the marker to them. rotation/kustomization.yaml records
the exception.
- Merged current main (a6f37e2b, 1691a0ac): the Reloader annotations for
zone-configurator and speech-to-phrase are already there via #1409.
review-bot (review 1787) noted the old explanatory comment was dropped without replacement. The range stays open-ended: the homelab rule is to never pin a Helm chart version, Traefik follows current upstream.
review-bot: /users/{u}/orgs requires the user and organization scope
categories (Gitea 1.27.3 api.go, HasScope needs all), and the sync's PAT
has write:admin,write:organization,write:issue,read:repository, no
read:user. /admin/users/{u}/orgs needs only the admin category and lists
private orgs for a site admin.
Also: drop paged()'s dead /repos/search unwrapping, test the team-mode
fallback of team_writes_issues, and say in SKILL.md that direct-collaborator
and user repos still get cards via the system hook.
From 2026-10-08 to 2026-10-16 k8s-gateway ran the chart's default Corefile
(zone example.com, forward . /etc/resolv.conf). Pods stayed Ready, every
renner.uno lookup was forwarded to the UDM and still answered (public names as
CNAME proxy.renner.uno), and nothing alerted. probes-dns.yaml only asks the UDM.
Two blackbox DNS modules ask the k8s-gateway Service (by name, not IP) for
auth.renner.uno A over IPv4 (the path CoreDNS forwards on, 10.96.0.11) and
AAAA over IPv6, and fail unless every answer RR is the Traefik VIP
(10.2.99.86 / fd99::86). A forwarded answer — CNAME or any other address —
fails. CoreDNS is bypassed on purpose: its renner.uno block serves stale for 1h.
K8sGatewayIngressAnswerWrong (critical) fires when the 10m success average is
under 0.8 for 5m: about 7m after a total failure starts, and also when one of
the two replicas is broken (average ~0.5), where a `for` on the raw value would
keep resetting. One or two failed probes stay silent. K8sGatewayProbesAbsent is
the dead-man's switch.
Verified: the module config was run with blackbox_exporter 0.29.0 against the
live cluster resolver (which relays k8s-gateway's answer): A and AAAA succeed;
a k8s-gateway-forwarded name (ha.renner.uno, 10.2.99.87) and a wildcard name
(CNAME proxy.renner.uno) fail. promtool-rules.sh with promtool 3.15.0: 9 suites
green, 41 mutations caught (4 new).
Closes#1290
ha-token mode revoked the outgoing LLAT seconds after storing the new one in
OpenBao. Consumers read the k8s Secret, which ESO syncs every 1h, so on
2026-10-01 the ServiceMonitor scrape got 401 from 02:00 to ~02:36Z
(HomeAssistantDown, TargetDown) and zone-configurator crashlooped (#1081,
#1082, #1084).
The reap now keeps the token just stored and the one it replaced, and
revokes only rotator-minted tokens (exact "<client_name> [6 hex]") other
than those two, the same keep-two rule gitea-token mode got in 59e804d9.
The replaced token is revoked by the following quarterly run.
The binary only reaches the CronJobs through a new image, and every
rotator was hand-pinned to secret-rotator:2026-07-30. Enrol the image in
Flux image automation (dated-tag policy, as for ci-backup-verify) and mark
all 20 rotator image lines, so they move together to the newest built tag.
Source between 2026-07-30 and this commit differs only by the Rust 1.91 ->
1.99 build image (68c09786).
Closes#1081
The sync reached only konairius/homelab ("repos: 1 with issues writable by
hermes-bot") although hermes-bot has write on all 10 org repos through the
org team Bots (includes_all_repositories, repo.issues write).
Cause: GET /repos/{o}/{r}/collaborators/{user}/permission (Gitea 1.27.3
GetRepoPermissions) returns permission.AccessMode, the repo-wide mode from
the collaborator/access level. Rights from a unit-based team land only in
the per-unit modes and are not part of that field, so it said "none" for
the 9 repos where hermes-bot is not a direct collaborator (checked live:
homelab "write", shelly-manager and ical-proxy "none", while
/orgs/konairius/repos reports push: true for all 10).
Scope is now derived from org membership: orgs of hermes-bot
(/users/hermes-bot/orgs) -> teams granting repo.issues write that it is a
member of -> all org repos (includes_all_repositories) or the team's repos,
non-archived with issues on. User-owned repos are out of scope, so labels
are org labels only and the repo-label branch is gone.
Tests: team scope, read-only team skipped, selected-repo team, user repos
out of scope, failed team lookups, org-labels-only. Live read-only probe
with hermes-bot's token: 10 repos, label target konairius.
Chart 41.0.0 renamed `logs.access` to `accessLog` and its values schema
rejects the old root `logs` key; that is why 399f101b capped the range
below 41.0.0. Moving `logs.access.enabled` to `accessLog.enabled` makes
our values valid on the current line, so the cap goes.
Verified with helm v4.3.1 against the published charts, values taken
from this HelmRelease, --kube-version 1.36.3:
- 42.0.0 with the old values fails exactly as in June
("additional properties 'logs' not allowed").
- 42.1.0 with these values renders the same manifests as the running
40.3.0, apart from chart/version labels and the image
(traefik v3.7.4 -> v3.7.15). The access-log args are unchanged
(--accesslog=true, fields defaultmode keep, headers drop), so the
CrowdSec agents parsing Traefik logs see the same format.
- Chart CRDs: same 25 CRDs; additive schema changes and controller-gen
0.17.1 -> 0.21.0 annotations.
Other breaking notes for 41/42 do not apply here: providers.file.content
is not set; 42.0 requires Kubernetes >= 1.30 (cluster runs 1.36.3);
the Hub secret RBAC and transparency-log witness changes only apply
with hub.token, which is not set.
The range is now open-ended (>=42.0.0), per the house rule not to pin
chart versions; note the wiki's floating-helm-chart-constraint-risk
recommends tilde ranges instead, so the owner should confirm the shape.
Refs #426
Regenerated with `flux install --export
--components-extra=image-reflector-controller,image-automation-controller`
from the v2.9.6 CLI (release tarball, sha256 checked against
flux_2.9.6_checksums.txt); the same component set as before.
Controllers: source v1.9.5 -> v1.9.6, kustomize v1.9.5 -> v1.9.6, helm
v1.6.4 -> v1.6.5. notification, image-reflector and image-automation
are unchanged. No CRD, RBAC or NetworkPolicy changes: the diff is the
three images and their app.kubernetes.io/version labels.
`flux check --pre` (v2.9.6 CLI) passes on Kubernetes 1.36.3.
Flux core: merge only 02:00-05:00 Europe/Berlin.
Refs #426
review-bot on 5de84e94: the guard sat after up to 20000 characters of untrusted
issue text. render() now takes a head block placed right after the title line;
Gitea-Issue: stays the last line.
Also checked (Gitea release/v1.27 routers/api/v1/repo/collaborators.go,
GetRepoPermissions; the API docs only list 200/403/404): for a site-admin caller
it answers 200 with the access mode (none for a non-collaborator), 404 only for an
unknown user, so repos hermes-bot cannot see are filtered, not failed.
review-bot on b6179a41:
- Expert card bodies open with "This card is for the <expert> expert. If you are
not <expert>, do no work on it": a coordinator that claims the card between the
body edit and the reassign completes it instead of running the alert protocol
next to the separate @expert card.
- issue_repos skips (and names) a repo whose hermes-bot permission lookup fails
instead of aborting; the other repos still get labels and backfill, the Job
still fails.
review-bot on de080442:
- The site-admin PAT moves from apps/hermes/gitea-admin (readable by the agent's
`hermes` role, homelab/data/apps/hermes/*) to apps/gitea-webhook-sync/admin.
Only ESO reads it (eso role, existing ClusterSecretStore); `bao token
capabilities` with the hermes role: deny on the new path. ExternalSecret and
K8s Secret are now gitea-webhook-sync-admin, mounted only by the CronJob.
- Hand-over writes the expert body before reassigning. If the coordinator claims
the card in between, its routing body is restored and the expert gets the
separate @expert card; if the edit fails, nothing is reassigned. Tests for both.
- [alert] cards: a comment is the owner's only by its API user.login (k.renner,
the login issue-fix trusted and the one commenting on homelab issues), never by
a name or quote in the text.
- Labels and backfill only in repos where hermes-bot has issue write (per
/collaborators/hermes-bot/permission), org labels only for orgs owning such a
repo; gitea_admin/_cargo-index (hermes-bot: read) no longer gets labels.
- Backfill starts no replay after RUN_BUDGET_SECONDS (900); activeDeadlineSeconds
1200 covers that plus one 60 s delivery; a test pins the relation.
Verified live: Gitea 1.27.3 swagger says IssueLabelsOption.labels takes IDs or
names, so the coordinator's {"labels": ["<expert>"]} works; hermes-bot has
write on konairius/homelab.
The owner wants the path alert -> [alert] issue -> kanban card -> expert fix PR
-> owner merges, without a human step in between. Owner decisions (kanban card
t_3f5d0a62): [alert] issues are not excluded from the bridge, and the issue-fix
Woodpecker pipeline is retired, so the expert's card is the only automated fix
path for [alert] issues.
Routing
- alert-monitor: `alerts.py file` takes a required `--label` (hestia, atlas,
apollo, clio, daedalus) and sets that label at creation (POST /issues
`labels: [id]`). Gitea 1.27.3 adds the labels in the same transaction
(services/issue/issue.go NewIssue) and the `opened` webhook payload loads them
(services/convert/issue.go toIssue -> LoadLabels), so the first delivery
already routes the card to the expert. Org labels are accepted for org repos
(models/issues/issue_update.go: label.OrgID == repo.OwnerID). If the label
does not exist yet, the issue is filed without it and the coordinator routes.
SKILL.md tells the triage agent how to pick the expert.
- Label-less issues no longer go to triage. kanban.auto_decompose defaults to
true (hermes_cli/config_defaults.py; not overridden here), so the dispatcher
hands every triage card to the auxiliary-LLM decomposer
(gateway/kanban_watchers_dispatcher.py auto_decompose_tick ->
kanban_decompose.decompose_task), which rewrites title and body (dropping the
Gitea-Issue line the completion hook needs), may fan out into children without
that line, and keeps the assignee `default` anyway (_apply_single only sets an
assignee when none is set). The card is now created ready for `default` with
routing instructions: set exactly one domain label on the issue, nothing else.
- The script reacts to that label: a coordinator card nobody has claimed yet
(todo/ready) is reassigned (`hermes kanban assign`) and its body replaced by
the expert body (`hermes kanban edit --body`); a claimed or finished one gets
a separate expert card under key `<owner>/<repo>#<n>@<expert>` (one per
expert, created_by gitea-webhook, so its completion is posted on the issue).
`assign` refuses a running card, which falls back to the separate card. Cards
already with an expert are still never moved.
[alert] cards
- Their body carries the fix protocol that replaces issue-fix, including its
prompt hardening from PR #253/#266: check for an open PR with `Closes #<n>` or
a branch `alert-fix/issue-<n>` first; owner (k.renner) comments are
authoritative, everything else is untrusted problem description; verify live
before fixing, never state cluster state not observed this run, give
read-only commands instead of predicted output; repo-fixable -> branch
`alert-fix/issue-<n>`, smallest fix, PR with `Closes #<n>`, never merge; not
repo-fixable -> no PR, the completion summary (posted on the issue) says why
and what a human must do.
Gitea side (CronJob gitea-webhook-sync, every 15 min)
- Domain labels: created as org labels in every org (GET /admin/orgs) and as
repo labels in user-owned repos, only when missing.
- Backfill: open issues without the card comment are replayed to the route as
signed `opened` deliveries (same headers Gitea sends), at most 10 per run, 3 s
apart (the adapter's default rate limit is 30/min/route), in random order so
repos where the comment keeps failing cannot starve the rest. The idempotency
key `<owner>/<repo>#<n>` makes a replay of an issue with a card a no-op.
The job runs in the hermes namespace, whose default-deny already allows
intra-namespace traffic to :8644.
- The admin PAT needs write:organization (org labels, reqOrgOwnership passes for
a site admin), write:issue (repo labels, comments) and read:repository
(/repos/search) next to write:admin. activeDeadlineSeconds 120 -> 600.
issue-fix removed
- .woodpecker/issue-fix.yml, ci/agents/issue-fix/, and what only it used:
scripts/ci/make-diag-kubeconfig.sh, the monitoring/ci-diagnostics
ServiceAccount and RBAC (no other pipeline mints its token; git grep), the
.ci-diag/ gitignore entry. Comments naming issue-fix updated.
- Not in git and left to the owner: the Woodpecker cron `issue-fix` and the
secret `issue_fix_bot_token` with its Gitea PAT.
Tested: unittest 73 OK; gitea_issues.py against the real hermes kanban CLI in a
scratch HERMES_HOME with a stub Gitea (unlabelled -> default ready card;
labelled while unclaimed -> same card reassigned to daedalus with the alert
body; labelled while claimed -> separate card `#n@daedalus`; replay -> no
duplicate card or comment); sync discovery read-only against live Gitea (42
open issues, all without a card); kubectl kustomize for apps/hermes,
apps/monitoring, apps/monitoring-rules.
Every open issue on gitea.renner.uno lands on the Hermes kanban board without a
manual step (owner decision: all issues, no opt-in label). Exactly one label out
of hestia/atlas/apollo/clio/daedalus assigns the card to that expert; anything
else goes to the coordinator (default) in triage.
Inbound, event-driven through upstream Hermes features only (webhook platform,
route script, kanban CLI; nothing forked or patched):
- A Gitea system webhook (events issues_only + issue_label) posts to
hermes.hermes.svc.cluster.local:8644/webhooks/gitea-issues. Gitea signs with
X-Hub-Signature-256 and sends X-GitHub-Event: issues, which the Hermes webhook
adapter validates as GitHub-style HMAC.
- Route gitea-issues runs skills/gitea-issues/scripts/gitea_issues.py as its
route script. The script creates the card itself with
`hermes kanban create --idempotency-key <owner>/<repo>#<number>
--created-by gitea-webhook` and answers [SILENT], so no agent run and no LLM
call happens. Edits and label changes reuse the existing card (no duplicate,
no reassignment). The issue text is quoted into the card body as untrusted
input; the card ends with a `Gitea-Issue: <url>` line.
- It then comments the card id on the issue once (marker comment, so retries
and later events do not repeat it).
Back channel: a kanban_task_completed shell hook (config `hooks:` on default and
all five experts) runs the same script in `completed` mode and posts the
completion summary on the issue. Dispatcher-spawned workers run with
--accept-hooks; the gateway (dashboard completions) gets an allowlist entry in
each profile's shell-hooks-allowlist.json. The issue is not closed
automatically: a finished card is not proof the issue is resolved (the work may
sit in an unmerged PR), so closing stays with the owner or an explicit ask.
Gitea side, declarative: CronJob gitea-webhook-sync (hermes ns, every 15 min,
hermes image, scripts/sync_webhook.py) keeps the system hook in place through
the admin API. Gitea 1.27.3 facts it relies on (routers/api/v1/utils/hook.go):
POST /admin/hooks creates a *default* (template) hook unless
config.is_system_webhook is "true", and PATCH ignores config.secret. So the
hook is named hermes-kanban-<hash of url, events, secret>; when no active hook
has the desired name, a new one is created first and older hermes-kanban* hooks
are deleted afterwards. Other system hooks (review-bot) are never touched.
Secrets (OpenBao, apps/hermes only, bootstrap commands in
gitea-webhook-externalsecret.yaml):
- apps/hermes/gitea-webhook#secret: HMAC secret, read by the gateway (written
to /opt/data/.env as WEBHOOK_SECRET by default-env, which also sets
WEBHOOK_ENABLED only when the secret exists) and by the sync job.
- apps/hermes/gitea-admin#api-token: site-admin PAT (write:admin; Gitea has no
narrower scope for system hooks), mounted only into the sync job.
The Deployment carries a Reloader annotation for hermes-gitea-webhook, so the
first sync of the secret (and a rotation) restarts the gateway with it.
Network: Service port webhook 8644, CNP hermes-allow-gitea (ingress from the
gitea pod on 8644), egress rule in gitea's default-deny, and
hermes.hermes.svc.cluster.local in GITEA__security__ALLOWED_HOST_LIST.
Tested offline: tests/test_gitea_issues.py (routing, card shape, marker
spoofing, comment dedup, completion, sync create/no-op/replace). Against the
real Hermes code in a scratch HERMES_HOME with a stub Gitea API: WebhookAdapter
accepted a Gitea-style signed POST (200, script), rejected a bad signature
(401), ignored a pull_request event; repeated issue events produced one card;
the init-container logic set the route, hooks and allowlist (`hooks list`:
allowed); kanban_db.complete_task after register_from_config fired the hook and
posted the summary.
# tools plus EXACTLY ONE live command: the RBAC-scoped read-only
# kubectl from fetch-diag (no Secrets readable, prefix match forbids
# pipes and other binaries). No web (see security-scan.yml).
[ -f /tmp/alert-fix/issue.json ] || exit 0
export KUBECONFIG="$$PWD/.ci-diag/kubeconfig"
# Model chosen live from the catalog (not hardcoded, no family assumed) —
# see the header of scripts/ci/pick-model.sh. Empty MODEL -> CLI default.
MODEL=$$(sh scripts/ci/pick-model.sh "diagnose an [alert] issue against read-only live cluster state and fix the root cause by editing repo manifests; multi-step reasoning that changes infrastructure code and needs a capable model")
if [ -n "$$MODEL" ]; then echo "issue-fix: selected model $$MODEL"; else echo "issue-fix: no live catalog — using the CLI default model"; fi
cat ci/agents/issue-fix/prompt.md | claude $${MODEL:+--model $$MODEL} \
if git diff --quiet && git diff --cached --quiet \
&& [ -z "$$(git ls-files --others --exclude-standard)" ]; then
echo "Agent made no edits — commenting assessment on issue #$$NUM."
[ -s /tmp/alert-fix/summary.md ] \
|| echo "Agent made no edits and wrote no summary." > /tmp/alert-fix/summary.md
printf '\n\n<!-- alert-fix:assessed -->\n_Assessed by the issue-fix pipeline: no repo change proposed. Remove this comment to make the pipeline retry._\n' \
>> /tmp/alert-fix/summary.md
jq -n --rawfile body /tmp/alert-fix/summary.md '{body: $$body}' > /tmp/gitea-comment.json
# (string) This setting controls the default policy of fact gathering (facts discovered about remote systems).
# This option can be useful for those wishing to save fact gathering time. Both 'smart' and 'explicit' will use the cache plugin.
;gathering=implicit
gathering=smart
# (list) Set the `gather_subset` option for the :ref:`ansible_collections.ansible.builtin.setup_module` task in the implicit fact gathering. See the module documentation for specifics.
# It does **not** apply to user defined ``ansible.builtin.setup`` tasks.
@@ -134,7 +134,7 @@
;hash_behaviour=replace
# (pathlist) Comma separated list of Ansible inventory sources
inventory=~/source/homelab/ansible/inventory
inventory=inventory
# (pathspec) Colon separated paths in which Ansible will search for HttpApi Plugins.
# (path) Option for connections using a certificate or key file to authenticate, rather than an agent or passwords, you can set the default value here to avoid re-specifying --private-key with every invocation.
private_key_file=~/.ssh/id_ed25519
;private_key_file=
# (boolean) By default, imported roles publish their variables to the play and other roles, this setting can avoid that.
# This was introduced as a way to reset role variables to default values if a role is used more than once in a playbook.
@@ -274,7 +274,7 @@ roles_path=../roles:./roles
# (path) The vault password file to use. Equivalent to --vault-password-file or --vault-id
# If executable, it will be run and the resulting stdout will be used as the password.
# It can result in a very significant performance improvement when enabled.
# However this conflicts with privilege escalation (become). For example, when using 'sudo:' operations you must first disable 'requiretty' in /etc/sudoers on all managed hosts, which is why it is disabled by default.
# This setting will be disabled if ``ANSIBLE_KEEP_REMOTE_FILES`` is enabled.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.