The archive was fetched into the controller's /tmp, so every run after a
/tmp cleanup downloaded it again and reported a change. It now lives in
~/.cache/ansible/crowdsec, like the netbird-client download.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- ssh-ca reloads sshd only when the CA key, the Include or the drop-in
changed; principals and revoked keys are read per login.
- On DSM the sudo rules are checked with visudo -c -f - on the
controller instead of a temporary file, which reported three changes
per run.
- Facts are read through ansible_facts (INJECT_FACTS_AS_VARS
deprecation) in common, qemu-guest-agent and keepalived-ntp.
- common masks systemd-timesyncd on guests only; a KVM host such as a
NAS running VMM has no such unit.
- netbird-client uses deb822_repository (apt_repository deprecation) and
removes the old one-line source.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
root on nas-1 accepts no owner key or certificate yet, so the first
run has to go through ladmin with its become password.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With the SSH CA in place, root accepts the owner certificate on DSM
(PermitRootLogin without-password), so runs need no become password,
as on every other host.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nas-1 is powered on again. It joins [synology_nas] and [ssh_ca] like
nas-2; the roles and plays dropped with it in 9c9e45d1 stay dropped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On DSM claude-bot is in administrators, whose sudo rule asks for a
password nobody knows. ssh_ca_claude_sudo_all (default false) adds a
passwordless rule; synology_nas sets it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nas-2 (Synology DSM 7.3) joins [ssh_ca] under a new [synology_nas] group,
so claude-bot reaches it with its 1h certificate like every other host.
DSM differs from the fleet in ways the ssh-ca role now handles:
- Python 3.8 only; plays/configure-synology-python.yml installs Synology's
Python3.9 package via raw, and group_vars point the interpreter at it.
- No getent: admin logins come from /etc/passwd on every host.
- No useradd: the account is made with synouser and added to
administrators (the only group DSM lets in over SSH). synogroup --member
replaces the list, so the current members are read and passed along,
and the run stops if the owner login is not among them.
- No visudo: the sudo rules are rendered and checked on the controller.
- sshd_config has no Include; the role puts it first.
- No sftp/scp for ladmin: transfers are piped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
App endpoints are <function>.<app>.renner.uno (ftp.paperless.renner.uno,
syncthing.paperless.renner.uno).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
[::]:8082 also listened on net1. POD_IPS (status.podIPs) lets the app bind
8080/8082/9090 to the pod IPv4 and IPv6 only; ASPNETCORE_URLS falls back
to the pod IPv4 for images that do not read POD_IPS.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Captured on k8s-1 (cilium-dbg monitor, endpoint 1708): an IPv6 SYN from
edge-1 (fd02:30::/64) to fd99::8a reached the pod on 8082, but no reply
left via eth0; the pod has on-link and default IPv6 routes on net1 from
RAs. sbr only confines IPv4. Turn RA processing off on net1 with the
tuning plugin (as matter-server already uses it) and roll the pod.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The auto-assigned IPv4 address 10.2.99.0 timed out from the LAN while
the pod answered on 8082 directly; pin 10.2.99.90/fd99::8a like the
other bgp-pool Services. Kestrel listened on 8082 only on the pod IPv4,
so the IPv6 endpoint had no listener; bind 8082 on [::].
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
shelly-manager now checks and rolls out Shelly firmware itself
(konairius/shelly-manager#121). Devices download from it on :8082 via a
BGP LoadBalancer named shelly-firmware.renner.uno; it reads Shelly's
index from *.shelly.cloud over HTTPS; /metrics on :9090 is scraped and
alerts on halted rollouts, a stale index and invalid rollout config.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pocket Casts now serves an Up Next recommendation row whose single item is
the Up Next playlist, titled with the episode at the head of the queue
(podcast as subtitle, podcast artwork). The frontend builds Top Picks for
You from the first items of the visible recommendation rows, so the head of
Up Next shows there; clicking or playing it loads the whole playlist. An
empty Up Next yields an empty row, which the frontend hides together with
the pick.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The k8s-outposts blueprint pinned the patched LDAP outpost image by hand.
Flux setters cannot patch text inside the blueprint ConfigMap string, and
Gitea's cleanup rule deletes a pinned tag once a newer build ages it out.
The image now lives in the authentik Flux Kustomization's
postBuild.substitute, with the authentik-samba-bridge-ldap-outpost
ImagePolicy marker, and the blueprint reads ${SAMBA_BRIDGE_LDAP_OUTPOST_IMAGE}.
kustomize-controller renders the literal before Authentik reads the file.
`flux build kustomization authentik` differs from the unsubstituted render
only in that line; kubernetes/apps/authentik has no other ${...} literals.
The SAMBA_BRIDGE_LDAP_OUTPOST_IMAGE env var in global.env, used by the
earlier !Env attempt, is removed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#1510 set the k8s-outposts blueprint's container_image through
`!Env SAMBA_BRIDGE_LDAP_OUTPOST_IMAGE`. It did not resolve to the patched
image: at 07:07:39Z the outpost controller recreated
ak-outpost-ldap-outpost with Authentik's default
ghcr.io/goauthentik/ldap:2026.8.4, although every worker carries the env
var. The upstream outpost lacks the sambaSamAccount/sambaDomain object
classes DSM's smbd queries for.
container_image is a literal again, on the samba-bridge outpost build that
exists now (2026-10-28-0655).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Gitea's konairius package cleanup rule keeps the most recent version and
counts every tag as a version. Builds that push several tags (build.<ts>,
sha, main, latest) keep only the last-pushed one, so the tag a workload
runs gets deleted: vsftpd kept only `main` and ImagePolicy vsftpd fails
with "version list argument cannot be empty" (#1509); ical-proxy's pinned
main-926ead0 is gone too.
Every konairius image is now pushed with exactly one tag, YYYY-MM-DD-HHMM
(UTC), the scheme review-bot and the gitea_admin images use, and every
consumer follows it through an ImagePolicy instead of a hand-written pin:
- hermes-image.yml: date tag only, no :latest.
- build-ci-images.yml: alpine-jq, cni-plugins, alpine-openssl tag
YYYY-MM-DD-HHMM; the konairius/ and ghcr copies drop :latest.
gitea_admin/ copies are unchanged.
- ImagePolicies for hermes, wiki, shelly-manager, vsftpd switch to the
date filter (alphabetical, accepts YYYY-MM-DD for the older utility
builds). New policies and markers: ical-proxy, mqtt-account-controller,
authentik-samba-bridge-server (HelmRelease tag), the samba-bridge LDAP
outpost, alpine-jq, alpine-openssl, cni-plugins.
- The outpost image moves out of the blueprint string, which Flux setters
cannot patch, into global.env SAMBA_BRIDGE_LDAP_OUTPOST_IMAGE, read by
the blueprint via !Env (same pattern as LDAP_OUTPOST_TLS_* and
AUTHENTIK_GATE_SYNC_TOKEN).
- renovate.json: the new Flux-tracked images join the disabled list.
- Hermes homelab-gitea-pr guide: date tag, one tag, no hand pins.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
shelly-manager's /api/v1/mqtt/push is being removed in favour of
group settings (konairius/shelly-manager group-settings M3).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner decision on #1455 (2026-10-27): package retention moves to Gitea's
built-in package cleanup rules (docs.gitea.com/usage/packages/storage),
created in the UI; Gitea 1.27.3 has no API for them.
Removed: package-retention.yaml (CronJob, ExternalSecret, ServiceAccount,
ClusterRole/Binding), package-retention.py and its configMapGenerator
entry, the gitea-package-retention-kube-apiserver CiliumNetworkPolicy, and
`gitea` from the openbao ClusterSecretStore allowlist. The retention
ExternalSecret was the only one in namespace gitea using that store;
gitea-ssh-ca uses its own namespaced SecretStore openbao-ssh-ca.
The job had failed nightly since 2026-10-18 with a dead token (hermes-bot
recreated, no longer site admin). Gitea's cleanup_packages cron in
statefulset.yaml stays: it runs the cleanup rules and frees blobs.
ImageRepository shelly-manager scans gitea.renner.uno/konairius/shelly-manager
and renovate's Flux-managed list names the new path. The Deployment line is
left to image automation: its marker is the full-reference form, so the first
konairius build.<ts> rewrites name and tag together. No manual tag.
konairius/shelly-manager already holds build.1791038240 (2026-10-03, older
than the running k.renner build.1792997278), so this must merge only after the
konairius/shelly-manager release pipeline has pushed a newer build there;
otherwise the policy would roll the Deployment back to the October image.
Owner decision on #1455: no packages under k.renner.
The bot posted COMMENT whenever the agent said comment, gave no
verdict, or approved with a red or yellow section. A COMMENT is no
review verdict, so the PR showed no approval or rejection by the bot.
Every review is now APPROVED or REQUEST_CHANGES. Only blocking findings
request changes: the verdict request-changes, a Blocking section, or an
empty change. Suggestions, nits and unverified points stay in the body
of an approval. An answer without a verdict line is not posted; the
review fails instead of guessing. The prompt offers only request-changes
and approve.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The sweep leaves queued and running reviews alone. Enqueueing one
marked it for another run when it ended, so a failing review was
chained back-to-back and FAILURE_COOLDOWN never applied.
- Mistral's read-only check accepts _compare_, so a re-review that
calls compare_commits is not rejected as a violation.
- SELECTOR_MIN_CONFIDENCE=NaN is rejected instead of disabling the
fallback.
- The prompt always lists recent base commits on the changed files, so
a change already on the base is still caught when mergeable is true.
- The requested-review search pages until an empty page.
- Doc comments describe the request-only, any-repository trigger.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
REPOS is gone. The webhook takes a review request from any repository,
and the sweep asks Gitea for every open pull request that requests a
review from the bot (issues/search review_requested=true), so a repo
needs no webhook and no config entry: requesting review-bot is enough.
The sweep runs every 5 minutes, since it is now one search plus two
reads per requested PR. A head whose review failed is left to the next
request or FAILURE_COOLDOWN (1h), so a failing review is not rerun on
every sweep.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reviews start only when review-bot is a requested reviewer: the
pull_request webhook with action review_requested for the bot, or the
sweep finding the bot in requested_reviewers. Gitea drops the request
when the bot posts its review, so each request yields one review. Pushes
and opened PRs alone start nothing.
The prompt reads the diff first, opens whole files only where the hunks
are not enough, reads the base side only when mergeable is false or the
base moved, and aims at 10-20 tool calls. Re-reviews read only the
commits since the last marker via compare_commits, now an enabled tool.
Every review logs and exports input, output and cache tokens
(review_bot_tokens_total, review_bot_tool_calls_total).
The selector falls back to the default model when it picks another one
with less than SELECTOR_MIN_CONFIDENCE (0.6), and asks for the cheapest
capable model. Sonnet is the default, Opus the exception, Haiku is gone
(41-50 tool calls on trivial PRs), Mistral stays as a separate budget.
Anthropic effort defaults to medium.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Shelly devices are not Ansible-managed. Drops
plays/configure-shelly-ntp.yml, its only helper files/discover_shelly.py,
and the mention in site.yml.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
install-ntp-ha.yml:
- 'chronyc clients | head -10' could not fail: without pipefail the rc
is head's, and the result was never used. 'chronyc -n tracking'
fails when chronyd does not answer.
- The udp/123 check had failed_when: false and only printed
LISTENING/NOT LISTENING. 'ss -Hlunp sport = :123' now fails unless
chronyd owns the socket.
- The v4/v6 VIP presence checks had failed_when: false, so an ip
error (e.g. a wrong interface name) read as 'BACKUP'. They now fail
on a non-zero rc; an empty result still means BACKUP.
update-dist-upgrade.yml: failed_when already passes 'No new release
found'; ignore_errors: true on top of it turned a failed
do-release-upgrade into 'ignored' and the play went on to reboot.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
chronyc -h <vip> talks to chronyd's command port, which chrony.conf binds
to loopback only, so it always returned '506 Cannot talk to daemon'. The
'|| echo' turned that into rc 0, so the task reported ok on every run.
chronyd -Q with a server directive on the command line does a one-shot
NTP query: no config file means no pidfile, and client-only mode zeroes
the NTP and command ports, so it runs alongside the live daemon. -t 10
makes it fail rather than hang when the VIP does not answer.
The VIP presence check used 'ip addr show | grep 10.2.40.4', which also
matches 10.2.40.40-49; it now uses 'ip -4 addr show to <vip>' like the
v6 check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The dashboard builds its Content-Security-Policy from AUTH_AUTHORITY, which
for Authentik is a path (https://auth.renner.uno/application/o/netbird/).
A CSP source with a path only matches URLs below it, so the browser blocked
the token endpoint (/application/o/token/): no code exchange, no refresh,
and the dashboard showed "Error: Unauthenticated" until site data was
cleared. Verified in Chromium against the live header; NETBIRD_CSP adds
https://auth.renner.uno, derived from netbird_auth_issuer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- crowdsec-edge-bouncer, netbird-client: the controller downloads also run
in check mode; they only write on the controller and the next step needs
the archive
- uptime-kuma: reads and writes OpenBao through its HTTP API (uri) instead
of community.hashi_vault, so the controller needs no hvac library. The
token comes from VAULT_TOKEN or ~/.vault-token, as with the bao CLI; the
owner OIDC role gets policy uptime-kuma-vps.
- deploy-uptime-kuma pipeline removed: its deploy step was gone, only the
renewal cron for the CI token remained.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Check mode skips command and uri tasks, so every assert or condition on a
read-only result failed or saw nothing (ssh-ca: "dict has no attribute
content" on the CA key). The tasks that only read state run in check mode
too (check_mode: false): the CA key fetch, sshd unit lookup, key
fingerprints, sshd listeners, netbird status, grep/pgrep/ip/chronyc/
timedatectl checks. The sshd -T assert is skipped in check mode, as sshd
reads the files check mode has not written.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
It was never installed on any managed host (/opt/node-exporter absent on
all four, site.yml --check) and Prometheus scrapes no :9100 target outside
the cluster. The role also downloaded from github.com on the host, which
proxy-1 (IPv6-only) cannot reach. Play and role removed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ansible-playbook site.yml, from ansible/, applies every state play in
dependency order; the inventory and vault password come from ansible.cfg,
now as paths relative to it. One-off actions (update-os,
update-dist-upgrade, rotate-netbird-secrets, configure-shelly-ntp) stay
separate.
- gathering=smart: facts once per host and run, not per play
- ControlPersist=30m so a run costs one YubiKey PIN per host;
plays/close-connections.yml ends site.yml and the action playbooks and
closes the connections
- qemu-guest-agent only on VMs (edge-1 is bare metal)
- removed: edge-mqtt-bridge.yml (install-edge-host.yml carries the role,
sml2mqtt is gone from edge-1) and the decommissioned claude-ws (inventory,
group_vars, role, play)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
service_facts took 3.5-11.7 s per host (profile over the four [ssh_ca]
hosts: 32 s of a 62 s run). The role only needs the running sshd unit,
which systemctl list-units answers in milliseconds; on Ubuntu the
sshd.service alias resolves to ssh.service.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Without it every task is a temp dir, an sftp upload, a chmod, the run and
a cleanup, each a separate SSH round trip. Measured over the mesh against
netbird.renner.uno with a warm ControlMaster: 1.85 s per command task
without pipelining, 0.85 s with it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
edge-1 and netbird accepted password authentication (OpenSSH default; no
config line sets it), the proxies did not. The ssh-ca drop-in turns off
password and keyboard-interactive authentication and the role asserts it
in sshd -T. The edge-host ISO renders the same drop-in.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replaces the docker-logs flag with one fixed set for every [ssh_ca] host:
- systemd-journal group instead of sudo journalctl/systemctl status, which
also allowed journalctl --vacuum-*/--rotate
- sudo: ss -tulpn, sshd -T, nft list ruleset (fixed args: nft joins argv
into one command line), netbird status [*], docker ps/logs/stats and
docker restart (no docker start: -a/-i attach to stdin)
- per-host restartable units: proxies netbird, netbird-proxy-flood-watchdog,
crowdsec-firewall-bouncer; netbird netbird; edge-1 mosquitto,
edge-mqtt-cred, edge-alloy, keepalived, chronyd
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The NetBird services on netbird.renner.uno log only to docker, which
claude-bot cannot reach. New role flag ssh_ca_claude_docker_logs adds
sudo for docker ps and docker logs; enabled for netbird.renner.uno.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
proxy-1 and proxy-2 were verified: only *%wt0:22 listens and the public
addresses refuse port 22.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From a workstation every new SSH login costs a YubiKey PIN and touch.
ssh-mesh-only reconnected on every run; the reconnect now sits in an
include that runs only when sshd is switched, since meta reset_connection
ignores when:. configure-ssh-ca is one play: with serial: 1, the second
play reached each host after its 60 s ControlPersist had expired, so it
logged in again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
proxy-2 was verified: only *%wt0:22 listens, the public IPv4 and IPv6
addresses refuse port 22, and mesh SSH works after a full reboot.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New role ssh-mesh-only and play configure-ssh-mesh-only.yml. sshd moves to a
systemd socket with BindToDevice=wt0: on Arch a new sshd.socket (Accept=yes,
packaged sshd@.service), on Ubuntu a drop-in for ssh.socket that masks the
sshd-socket-generator output, plus BindsTo= so ssh.service stops with it.
NetBird recreates wt0 on every engine restart; BindToDevice stops the socket
with the old interface and a udev SYSTEMD_WANTS rule starts it with the new
one. The socket is not in sockets.target, which would hold boot until wt0.
The switchover runs in a transient unit and arms a revert timer first; the
play cancels it only after logging in again through the new listener and
seeing no listener outside wt0. [ssh_mesh_only] starts with proxy-2.
ssh-ca reloads sshd only when one sshd unit is running, as there is none
with per-connection or stopped socket-activated sshd.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The build fetches the CA public key from the ssh-ca role URL and renders the
role drop-in, revocation list and root principal into the live system;
edge-install copies them to the installed system and checks sshd -T uses
the CA. Owner certificates log in as root from the first boot, with the
break-glass key as fallback.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The edge-host ISO carried krenner@fortress, ansible-runner@ci-new and
krenner@krenner-XPS-13-9310; it now carries only the break-glass key.
The netbird-server revoke task for netbird-backup@homelab-cluster goes:
the exclusive managed authorized_keys already excludes it. OpenTofu drops
the unused ssh_public_key/ssh_private_key variables and the Proxmox
provider ssh block.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The woodpecker-ci@homelab key is dropped from the managed authorized_keys
and from the edge-host ISO. The Ansible pipelines that used it are deleted;
plays run from a workstation. deploy-uptime-kuma keeps only the OpenBao
token renewal cron, and deploy-infrastructure no longer installs the key
(no root-module resource uses the Proxmox provider SSH).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
proxy-1, proxy-2 and netbird are addressed by their mesh names, so Ansible
for them runs from a mesh-connected workstation. CI cannot reach the mesh;
the pipelines targeting these hosts lose their push triggers and run only
on manual deployment.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first workstation run failed on proxy-1 with 'Network is
unreachable' fetching the release: proxy-1 is IPv6-only and github.com
has no IPv6 address. The tarball is now downloaded and checksum-verified
on the controller (~/.cache/ansible/netbird) and unarchive copies it to
the host, so the hosts need no route to GitHub.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the workstation each new SSH session asks for the YubiKey PIN and
a touch; with the default forks both proxies prompted at once. serial: 1
like the other plays that touch the proxies.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The play gathered facts on every inventory host before skipping those
without netbird_setup_key, so a workstation run opened an SSH session
(YubiKey PIN + touch) to edge-1 and netbird too, and timed out on the
decommissioned claude-ws. Facts are gathered after the skip now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ansible.cfg forced ~/.ssh/id_ed25519, which does not exist on the
owner's workstation: runs from there failed with 'no such identity'
and bypassed the ssh config that picks the plugged-in YubiKey's key
and certificate. CI never read this file ('world writable directory,
ignoring it as an ansible.cfg source'); its pipelines write the CI key
to ~/.ssh/id_ed25519, ssh's default identity, so they are unaffected.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Step 1 of SSH-only-over-NetBird: proxy-1/2 (Arch) need a host NetBird
client with wt0 before sshd can stop listening on their public
addresses. The role was apt-only.
- Arch: release tarball from GitHub, verified against the release's
checksums file (get_url checksum URL), binary to /usr/local/bin,
'netbird service install'. Version netbird_client_version, 0.75.0
like netbird-ingress; Debian/Ubuntu keep following the apt repo.
- netbird_disable_dns adds --disable-dns; set for [netbird_proxy],
whose resolv.conf the netbird-proxy role pins.
- The service is started before 'netbird up' (the Arch path has no
package that starts it).
- The connected check matched 'Connected' inside 'Disconnected'; it
now matches the 'Management: Connected' line.
- 'netbird up' runs with no_log: its argv holds the setup key.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review of #1481 (merged with it open):
- The Job printed the operator's streaming command with the pod's
HOME expanded (/work/.config/...), unusable on the workstation. The
defaults stay a literal ~ now; eval expands it on the workstation.
- 'Collecting secrets' captured stderr, where age-plugin-yubikey asks
for the PIN and touch: a silent hang. stderr stays on the terminal.
- Runbook/migration doc still named pass as the DR store in places;
tooling list gains age + age-plugin-yubikey; the file list gains
hetzner-cloud-api-secret.age.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The cold-rebuild secrets (seal key, Hetzner restore keys, GHCR pull
token) came from the pass store: one gpg key file on the owner's laptop,
readable by any process of that user while gpg-agent caches the
passphrase, and with no second copy.
bootstrap.sh now decrypts ~/.config/homelab-secrets/<name>.age, files
encrypted to both YubiKeys (touch) and an offline paper key; a copy is
on nas-2. HOMELAB_AGE_IDENTITY switches to the paper key when both
sticks are gone. The in-cluster Job path is unchanged: the printed
streaming command is the age one. Runbook and the two OpenBao alert
texts point at the new files.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
proxy-1 is verified after #1477/#1479: authorized_keys holds exactly
breakglass-neo-6532348 and the CI key, and CI still reaches it.
Removed with this run, per the [ssh_ca] report and the sshd journals
(no root login with any of them except id_ecdsa, the owner's old key):
- proxy-2: id_ecdsa, ansible-runner@ci-new
- netbird: id_ecdsa, ansible-runner@ci-new, krenner@fortress,
netbird-tunnel (SHA256:cNy8...), a second woodpecker-ci key
(SHA256:hQxz...); none used for a root login since 2026-08-29
- edge-1: id_ecdsa, ansible-runner@ci-new, krenner@fortress
The CI key keeps from= the pod network on edge-1 (LAN, CI arrives from
10.244.2.x) and has none on proxy-2 and netbird (CI arrives NATed).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replaces the per-service Music Assistant/Immich parser whitelists with
one postoverflow whitelist: http-crawl-non_statics overflows are dropped
unless at least half of the requests in the alert are 4xx.
Every self-ban in the alert history (MA 350/368/477, Immich 149/150) was
a burst of 200s from a signed-in UI. Every scanner ban on this scenario
was 404s (/.env, /.git/...). The per-service rules only covered apps
that had already failed and muted all scenarios on their paths; this
covers any image-heavy UI and touches only this scenario.
Verified in a crowdsecurity/crowdsec:v1.8.1 replay: a 60x200 burst is
whitelisted, a 60x404 burst with distinct paths still overflows and is
not whitelisted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#1477 locked CI out of proxy-1: the CI key got from= the pod network,
but CI reaches the internet hosts NATed from the home WAN address
(proxy-2 journal: root logins from 84.171.37.13). The from= came from
plays/configure-ssh-keys.yml, which no pipeline ever ran, so it was
never in effect on those hosts. LAN hosts do see the pod address
(edge-1 journal: 10.244.2.x).
ssh_ci_key_from is now per host: the pod CIDRs by default, empty for
[netbird_proxy] and netbird.renner.uno. The owner already removed the
from= on proxy-1 by hand; this keeps the next run from restoring it.
Restricting SSH on internet hosts to the NetBird mesh is the follow-up.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Scrolling the Immich timeline fetches a /api/timeline/bucket and then
every /api/assets/<uuid> in it; 46-54 requests in 3-9s fired
crowdsecurity/http-crawl-non_statics against a residential Telekom IPv6
(alerts 149/150, 2026-10-23).
Same parser-stage pattern as the Music Assistant whitelist, scoped to
the Immich Traefik router and those two path prefixes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The MA UI loads every cover through /imageproxy/<hash>?size=512 in one
burst; a library view (47 requests in 3s) fired
crowdsecurity/http-crawl-non_statics and banned a residential Telekom
IPv6 (alert 477, 2026-10-27).
Parser-stage whitelist scoped to the MA Traefik router and the
/imageproxy/ path. The Traefik parser sets no target_fqdn, so the
router name is used as host scope.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
First host in [ssh_keys_managed]. root on proxy-1 currently has
id_ecdsa (krenner@krenner-XPS-13-9310), woodpecker-ci@homelab
(SHA256:DQWPS...) and ansible-runner@ci-new (dead since #184). After
this run: breakglass-neo-6532348 and the same CI key with its from=
pod-network restriction; people log in with CA certificates. proxy-2
is the twin, so the public edge survives a mistake here.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 4 of the SSH CA, step 1. Nothing is removed from any host yet.
- ssh_admin_authorized_keys (group_vars/all) is the single list of
plain keys the admin logins may accept: the break-glass PIV key on
the offline YubiKey NEO and the CI deploy key (from= pod network).
- ssh-ca reports, on every [ssh_ca] host, the fingerprints of the
admin logins' current authorized_keys.
- configure-ssh-ca.yml gets a second play that sets the admin logins'
authorized_keys exclusively to that list on [ssh_keys_managed]. The
group is empty; hosts join one at a time after the report.
- Removed the three dead key lists: plays/configure-ssh-keys.yml (no
pipeline ran it), common_ssh_public_keys (defined, never used) and
opentofu/locals.tf common_ssh_keys (no reference). They carried
krenner@fortress, ansible-runner@ci-new and the laptop id_ecdsa key.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Step A is verified: Secret gitea-ssh-ca holds exactly the CA key
(SHA256:xEhuqx..., one line, 81 bytes like OpenBao's content-length),
so the webhook provider passes a text/plain body through unchanged.
- StatefulSet: SSH_TRUSTED_USER_CA_KEYS from the Secret,
SSH_AUTHORIZED_PRINCIPALS_ALLOW=username. The built-in SSH server
looks a cert's principals up as principal keys, so each user
registers their username once as a principal.
- ExternalSecret: trim the trailing newline before it reaches app.ini.
- Reloader annotation: Gitea reads env only at start, so a CA rotation
restarts it. The step A comment claimed the refresh alone reaches
Gitea; corrected.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 3, step A of the SSH CA: Gitea's built-in SSH server needs the CA
key as SSH_TRUSTED_USER_CA_KEYS. A namespaced ESO webhook SecretStore
reads OpenBao's unauthenticated /v1/ssh-client-signer/public_key from
the in-cluster Service into Secret gitea-ssh-ca, so the key has one
source. Gitea is not wired to it yet; step B follows once the Secret
is verified to hold exactly the CA key (the ESO docs do not say how a
text/plain body without result.jsonPath is handled).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
netbird verified (Ubuntu, ssh.service): claude-bot cert login, sudo
journalctl allowed, systemctl restart refused. edge-1 is the last
Ansible-managed host (OpenSSH_10.6 banner like the proxies).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The netbird run (pipeline 14795) failed at the reload: 'Could not find
the requested service sshd.service'. Ubuntu's service_facts list
sshd.service as a not-found alias next to the real ssh.service
(running; disabled because ssh.socket activates it), and the role only
checked that the name exists.
The unit is now the one of sshd.service/ssh.service whose state is
running, asserted to be exactly one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
proxy-2 verified (claude-bot cert login, sudo allow/deny as on
proxy-1). netbird is the first Ubuntu host (OpenSSH_9.6p1
Ubuntu-3ubuntu13.20): unit ssh.service, which the role picks from
service_facts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
proxy-1 is verified: owner cert login as root (sk-ed25519 cert) and
claude-bot cert login with the expected sudo rules both work. proxy-2
is its twin (same OpenSSH_10.6 banner).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The second deploy-ssh-ca run on proxy-1 failed at the assert although
all three settings were active: OpenSSH 10.6 prints them as
TrustedUserCAKeys, AuthorizedPrincipalsFile and RevokedKeys, the check
expected lowercase keywords. The run stopped before the reload, so the
running sshd still has the old config.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first deploy to proxy-1 failed at the reload: 'Could not find the
requested service ssh'. proxy-1 is not Debian (banner OpenSSH_10.6
without distro suffix, root shell /usr/bin/bash), so the unit is
sshd.service; the handler only special-cased RedHat.
The unit is now picked from service_facts. The reload runs on every
run instead of via a handler: sshd -T only parses the files, so after
a failed reload a re-run with nothing changed would otherwise leave the
running daemon on the old config.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
claude-ws is decommissioned. proxy-1 resolves by name, answers on 22,
and has proxy-2 as its twin, so the public edge survives a bad run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New role ssh-ca with play configure-ssh-ca.yml (serial: 1) and pipeline
deploy-ssh-ca. Hosts in the new [ssh_ca] group get:
- TrustedUserCAKeys from OpenBao's public_key endpoint, fetched on the
controller at run time (no copy in git).
- AuthorizedPrincipalsFile: k.renner for root/ladmin (whichever
exists), claude-bot for Claude's own claude-bot account.
- RevokedKeys, always deployed: sshd refuses all key logins when the
file is set but missing.
- claude-bot sudo: read-only journalctl/systemctl status/ss/sshd -T,
restart/reload only for ssh_ca_claude_sudo_units; NOEXEC against
pager shell escapes; visudo-validated.
The drop-in is tested with sshd -t and removed again on failure; sshd -T
must show all three settings afterwards. authorized_keys is untouched,
so this only adds a login path. [ssh_ca] starts with claude-ws only.
ssh-sign-owner.hcl now also records the mount, CA and owner role
commands (review on #1466).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 1 of the SSH CA. The ssh-client-signer mount and its ed25519 CA
were set up by hand (root token); this records the roles and the
claude-bot signing policy.
- role owner: principal k.renner, 6-month certs. The mount was enabled
with -max-lease-ttl=4380h; the 768h system default would cap it.
- role claude: principal claude-bot, 1h certs, pty only.
- policy ssh-sign-claude for the claude-bot userpass user, next to its
existing claude-bot-write.
Verified: a YubiKey sk-ssh-ed25519 key signed on role owner yields a
sk-ssh-ed25519-cert-v01 cert, principal k.renner, valid to 2027-04-27.
CA fingerprint SHA256:xEhuqxqFVZ/ig6KpAtHOhLbwsIhyWpzXhvDcjIfM/Js,
served unauthenticated at /v1/ssh-client-signer/public_key.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 0 of the OpenBao SSH CA: humans log in to OpenBao with
Authentik instead of the root token.
- Authentik blueprint: group OpenBao Admins (k.renner), confidential
OAuth2 provider openbao (CLI + UI callbacks), application, group
binding. The client secret is generated by Authentik and only goes
into auth/oidc/config.
- openbao-authentik-egress: auth.renner.uno resolves to the Traefik
VIP in-cluster, which the existing world egress does not cover.
- Policy ssh-sign-owner with the manual bao commands for auth/oidc;
its sign path is enabled in phase 1.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Six retries at a fixed 10 s delay stopped after about 60 s, so
--retry-max-time 120 never bound. Twelve retries let the 120 s cap
be the limit, covering a Recreate rollout slowed by an image pull.
Retrying on 5xx is safe: POST /api/v1/sync is serialized by
SyncCoordinator and re-syncs the same state, the same as a webhook
landing mid-sync. --retry-max-time 120 keeps a long sync that fails
late from being repeated.
The wiki Deployment uses Recreate, so during an image rollout the Service
has no ready endpoint for ~10 s. wiki-sync-29884710 started 06:30:00Z, 4 s
before the new pod was Ready, failed with curl (7) and fired KubeJobFailed.
Retry up to 6 times, 10 s apart, on connection refused too, bounded to
120 s from the first attempt so a long sync that fails is not repeated.
Refs #1461
Chart, server/worker image and LDAP outpost image move together, as
the chart pin requires. Images built from authentik-samba-bridge
3c8f413 (PR #8), pipeline 43.
Supersedes #1347.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
aiosendspin starts its zeroconf on [bind_ip], or on InterfaceChoice.Default
when bind_ip is 0.0.0.0. Default egresses via the pod's main-table default
route (eth0); net1 routes sit in the sbr table. MA therefore never sent
_sendspin._tcp queries onto v30iot and only rediscovered Voice PEs when they
announced on their own, leaving them disconnected for hours after every MA
restart (e.g. 2026-10-26 13:56: 3 of 4 still offline at 15:05).
The reconcile initContainer now writes bind_ip = the live net1 address it
already computes for publish_ip. The address is read from the interface on
every boot, so the Sendspin no-bind-fallback failure (stale literal IP)
cannot recur.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Up Next payload carries no episode duration. Marking an episode played
in the UI reports the duration as the position, so the provider received 0
and took it for an explicit "mark unplayed": the episode stayed in Up Next
and was flagged unplayed.
The Up Next playlist now takes duration and playback status from the
(cached, throttled) episode details, falling back to the bare entry when
the details cannot be fetched.
Tested against the 2.10.7 source: 21/21 pocketcasts tests, ruff, mypy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Overlays the Pocket Casts provider with a patched __init__.py that adds
ProviderFeature.LIBRARY_PLAYLISTS and a read-only "Up Next" playlist of
podcast episodes, fetched live from /up_next/list in queue order.
An initContainer copies the image's provider into an emptyDir and swaps
in the patch only when the stock file's sha256 matches the version it was
written against (identical in 2.10.7 and 2.10.8). A later image bump that
changes the provider keeps the stock provider instead of breaking it.
Temporary until the change is upstreamed to music-assistant/server.
Tested against the 2.10.7 source: 18/18 pocketcasts tests, ruff, mypy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reverts #1451. The YouTube Music provider is not kept; the PO token
server and its image automation go with it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Renovate #1439 pinned a digest into the ImageRepository's spec.image,
which image-reflector rejects ("should not include a tag"), so Flux
stopped tracking gitea-mcp-server. The image was missing from the
renovate.json list of images Flux image automation already bumps; add
it there so Renovate leaves it alone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MA's YouTube Music provider needs a bgutil PO token server; MA installs
the matching yt-dlp plugin unpinned from PyPI (2.0.2 today), so the
server tracks 2.x via image automation. The MA docs' "1.2.1" predates
the 2.x plugin line.
Reachable only from the MA pod; egress limited to DNS and world:443.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Step 2 of 2. The coredns ConfigMap is a Talos bootstrap manifest; Flux
stops applying its own copy (prune is disabled since the previous
commit, so the live ConfigMap stays until talosctl upgrade-k8s writes
Talos's version). CoreDNS forwards to the node resolver, which asks
k8s-gateway first, so cluster DNS no longer depends on Flux.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Step 1 of 2 to hand the coredns ConfigMap back to Talos. It is a Talos
bootstrap manifest (talos-bootstrap-manifests-inventory) that Flux also
applies; talosctl upgrade-k8s reverts Flux's Corefile. Bootstrap already
runs on Talos's Corefile (forward to the node resolver, which asks
k8s-gateway first), and k8s-gateway carries serve_stale for every name,
so the Flux copy is redundant.
With prune disabled, step 2 can drop the file without Flux deleting the
live ConfigMap.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Since UniFi OS 6.0.15 / Network 11.0.97 (2026-10-25 18:42Z) the UDM's
DNS answers out of the interface a query arrived on. Pod traffic to
10.2.40.1 arrives on VLAN 40 (on-link route in table 47) while the pod
CIDRs are routed back over VLAN 47, so the UDM ARPs for the pod on
VLAN 40 and never answers (seen with talosctl pcap on k8s-3; TCP 443 to
the same address works, UDP and TCP 53 do not). fd00::1 was reached
over VLAN 47 and kept working.
k8s-gateway's upstream (k8s-gateway-site), the blackbox-dns probes and
the gatus UDM check now use the UDM's ingress-segment addresses,
fd02:47::1 and 10.2.47.1 (the egress-gw annotations); both answer from
a pod.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
syncthing's STUN NAT-type probe sends its change request to port 3479,
which syncthing-egress dropped (seen: 77.72.169.212:3479/UDP).
Loki tried to send usage reports to Grafana (34.49.170.206:443), denied
by its egress policy; switch reporting off instead of allowing it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
wiki-sync (every 30 min) and wiki-encyclopedia (06:00 Europe/Berlin) call
the service's API with a generated scheduler key; the service no longer
schedules anything itself (konairius/Wiki#18). Pods in ai already reach
the wiki through the namespace default policy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The controller (main-b1f294d, konairius/mqtt-account-controller#13)
delivers a password account to a namespace Secret. admin moves into the
register with its password in Secret mqtt-admin-password; Dynamic
Security is asked first, so it replaces the password_file login without
a broker restart. The controller gets get/create/patch/delete on that
Secret by name.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
shelly-manager resolves device *.local names via mDNS; the queries leave on
eth0 to 224.0.0.251:5353 and have been dropped since the egress lockdown
(#1432), failing with 'Name or service not known' since 15:07Z.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only members of the new Authentik group Wiki Readers may sign in to the wiki
OIDC application; membership is managed in Authentik, so the owner must be
added before merge. Encyclopedia is enabled with MaxArticles=0 so the first
nightly run only plans.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Since the broker restart on 2026-10-22 only register accounts connect; the
last legacy logins (homeassistant, sml2mqtt, open3e, zigbee2mqtt) were on
2026-10-08, and admin/evcc never logged in within the 30-day Loki window.
- Delete mosquitto-bootstrap.yaml (5 Jobs that apk add at runtime + RBAC).
- The broker's password_file/acl_file now hold only admin, the owner's
workstation login over the LoadBalancer. The controller has no delivery
mode that puts a password account's password into a Secret, and the cert
listener :8884 is in-cluster only, so admin cannot be a register account.
Its password comes from an ESO Password generator (CreatedOnce), the
pattern of apps/hermes/api-key.yaml, in Secret mosquitto-admin-password.
- Delete the evcc and health ExternalSecrets and the health rotator
CronJob/policy; nothing reads them.
- Delete PROPOSAL-mosquitto-credential-rotation.md; its subject is gone.
- MosquittoDown test masks with the dynsec-test pod instead of a
bootstrap Job pod; comments and docs follow.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Runtime package installs are not to be enabled by policy. The Jobs feed only
the unused legacy password_file and are removed in a separate PR, which must
merge before this one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Jobs run apk add curl jq on every execution; without world 443/80 they
fail and block the home-assistant Kustomization. Also correct the stale
renderer egress comment.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
edge-1.iot.renner.uno:8443 is the deliver.esp32 target in mqtt-accounts.yaml
(https+mTLS credential push). Matched by FQDN, not the current 10.2.30.108.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drop namespace-wide world egress and give each workload only the external
destinations it needs (FQDN where the host is in our config, port-scoped
world otherwise). Namespaces that had ingress-only policies get an
in-cluster + DNS egress baseline. CNPG instances get Hetzner S3 via one
clusterwide policy.
Out of scope: kube-system, flux-system, rook-ceph, woodpecker-ci,
music-assistant, evcc, openbao.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# (string) This setting controls the default policy of fact gathering (facts discovered about remote systems).
# This option can be useful for those wishing to save fact gathering time. Both 'smart' and 'explicit' will use the cache plugin.
;gathering=implicit
gathering=smart
# (list) Set the `gather_subset` option for the :ref:`ansible_collections.ansible.builtin.setup_module` task in the implicit fact gathering. See the module documentation for specifics.
# It does **not** apply to user defined ``ansible.builtin.setup`` tasks.
@@ -134,7 +134,7 @@
;hash_behaviour=replace
# (pathlist) Comma separated list of Ansible inventory sources
inventory=~/source/homelab/ansible/inventory
inventory=inventory
# (pathspec) Colon separated paths in which Ansible will search for HttpApi Plugins.
# (path) Option for connections using a certificate or key file to authenticate, rather than an agent or passwords, you can set the default value here to avoid re-specifying --private-key with every invocation.
private_key_file=~/.ssh/id_ed25519
;private_key_file=
# (boolean) By default, imported roles publish their variables to the play and other roles, this setting can avoid that.
# This was introduced as a way to reset role variables to default values if a role is used more than once in a playbook.
@@ -274,7 +274,7 @@ roles_path=../roles:./roles
# (path) The vault password file to use. Equivalent to --vault-password-file or --vault-id
# If executable, it will be run and the resulting stdout will be used as the password.
# It can result in a very significant performance improvement when enabled.
# However this conflicts with privilege escalation (become). For example, when using 'sudo:' operations you must first disable 'requiretty' in /etc/sudoers on all managed hosts, which is why it is disabled by default.
# This setting will be disabled if ``ANSIBLE_KEEP_REMOTE_FILES`` is enabled.
# Alpine plus jq. Nothing application-specific — a general-purpose base for any
# workload that needs to read JSON from a shell script.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.