Compare commits

...
Author SHA1 Message Date
k.renner 970226d3be Merge pull request 'fix(ansible): drop node-exporter from the state' (#1497) from drop-node-exporter into main
Reviewed-on: #1497
2026-10-27 19:36:23 +00:00
claude-botandClaude Opus 5.5 5fdfbb6772 fix(ansible): drop node-exporter from the state
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
It was never installed on any managed host (/opt/node-exporter absent on
all four, site.yml --check) and Prometheus scrapes no :9100 target outside
the cluster. The role also downloaded from github.com on the host, which
proxy-1 (IPv6-only) cannot reach. Play and role removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 20:35:37 +01:00
k.renner 2ac5e705fd Merge pull request 'feat(ansible): one site.yml for the whole state' (#1496) from ansible-site into main
Reviewed-on: #1496
2026-10-27 19:32:56 +00:00
claude-botandClaude Opus 5.5 9e1dfc99bd feat(ansible): one site.yml for the whole state
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ansible-playbook site.yml, from ansible/, applies every state play in
dependency order; the inventory and vault password come from ansible.cfg,
now as paths relative to it. One-off actions (update-os,
update-dist-upgrade, rotate-netbird-secrets, configure-shelly-ntp) stay
separate.

- gathering=smart: facts once per host and run, not per play
- ControlPersist=30m so a run costs one YubiKey PIN per host;
  plays/close-connections.yml ends site.yml and the action playbooks and
  closes the connections
- qemu-guest-agent only on VMs (edge-1 is bare metal)
- removed: edge-mqtt-bridge.yml (install-edge-host.yml carries the role,
  sml2mqtt is gone from edge-1) and the decommissioned claude-ws (inventory,
  group_vars, role, play)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 20:29:33 +01:00
k.renner c2b0d9b6d9 Merge pull request 'perf(ssh-ca): find the sshd unit without service_facts' (#1495) from ssh-ca-no-service-facts into main
Reviewed-on: #1495
2026-10-27 19:16:37 +00:00
claude-botandClaude Opus 5.5 7ccdd8f405 perf(ssh-ca): find the sshd unit without service_facts
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
service_facts took 3.5-11.7 s per host (profile over the four [ssh_ca]
hosts: 32 s of a 62 s run). The role only needs the running sshd unit,
which systemctl list-units answers in milliseconds; on Ubuntu the
sshd.service alias resolves to ssh.service.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 20:15:40 +01:00
k.renner 36c958fb14 Merge pull request 'perf(ansible): enable pipelining' (#1494) from ansible-pipelining into main
Reviewed-on: #1494
2026-10-27 19:11:27 +00:00
claude-botandClaude Opus 5.5 0878054873 perf(ansible): enable pipelining
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
Without it every task is a temp dir, an sftp upload, a chmod, the run and
a cleanup, each a separate SSH round trip. Measured over the mesh against
netbird.renner.uno with a warm ControlMaster: 1.85 s per command task
without pipelining, 0.85 s with it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 20:10:40 +01:00
k.renner e502990db8 Merge pull request 'feat(ssh-ca): claude-bot rights for diagnosis and restarts' (#1493) from claude-bot-docker-logs into main
Reviewed-on: #1493
2026-10-27 19:02:32 +00:00
claude-botandClaude Opus 5.5 8832d9f23f fix(ssh-ca): no password logins
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
edge-1 and netbird accepted password authentication (OpenSSH default; no
config line sets it), the proxies did not. The ssh-ca drop-in turns off
password and keyboard-interactive authentication and the role asserts it
in sshd -T. The edge-host ISO renders the same drop-in.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 20:01:20 +01:00
claude-botandClaude Opus 5.5 e72c31f3e9 feat(ssh-ca): claude-bot rights for diagnosis and restarts
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
Replaces the docker-logs flag with one fixed set for every [ssh_ca] host:
- systemd-journal group instead of sudo journalctl/systemctl status, which
  also allowed journalctl --vacuum-*/--rotate
- sudo: ss -tulpn, sshd -T, nft list ruleset (fixed args: nft joins argv
  into one command line), netbird status [*], docker ps/logs/stats and
  docker restart (no docker start: -a/-i attach to stdin)
- per-host restartable units: proxies netbird, netbird-proxy-flood-watchdog,
  crowdsec-firewall-bouncer; netbird netbird; edge-1 mosquitto,
  edge-mqtt-cred, edge-alloy, keepalived, chronyd

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 19:57:00 +01:00
claude-botandClaude Opus 5.5 50b66682d3 feat(ssh-ca): claude-bot may read container logs on netbird
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
The NetBird services on netbird.renner.uno log only to docker, which
claude-bot cannot reach. New role flag ssh_ca_claude_docker_logs adds
sudo for docker ps and docker logs; enabled for netbird.renner.uno.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 19:50:21 +01:00
k.renner 1848079297 Merge pull request 'feat(ansible): netbird sshd listens on the NetBird interface only' (#1492) from ssh-mesh-only-netbird into main
Reviewed-on: #1492
2026-10-27 18:29:26 +00:00
claude-botandClaude Opus 5.5 48f1574d29 feat(ansible): netbird sshd listens on the NetBird interface only
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
proxy-1 and proxy-2 were verified: only *%wt0:22 listens and the public
addresses refuse port 22.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 19:26:30 +01:00
k.renner df6276101f Merge pull request 'fix(ansible): one SSH login per host for the SSH plays' (#1491) from ansible-fewer-logins into main
Reviewed-on: #1491
2026-10-27 18:25:31 +00:00
claude-botandClaude Opus 5.5 df0150d24a fix(ansible): one SSH login per host for the SSH plays
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
From a workstation every new SSH login costs a YubiKey PIN and touch.
ssh-mesh-only reconnected on every run; the reconnect now sits in an
include that runs only when sshd is switched, since meta reset_connection
ignores when:. configure-ssh-ca is one play: with serial: 1, the second
play reached each host after its 60 s ControlPersist had expired, so it
logged in again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 19:24:07 +01:00
k.renner 880bf00baa Merge pull request 'feat(ansible): proxy-1 sshd listens on the NetBird interface only' (#1490) from ssh-mesh-only-proxy-1 into main
Reviewed-on: #1490
2026-10-27 18:17:57 +00:00
claude-botandClaude Opus 5.5 6fbfb0ed03 feat(ansible): proxy-1 sshd listens on the NetBird interface only
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
proxy-2 was verified: only *%wt0:22 listens, the public IPv4 and IPv6
addresses refuse port 22, and mesh SSH works after a full reboot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 19:14:10 +01:00
k.renner 88c80ae21a Merge pull request 'feat(ansible): sshd listens on the NetBird interface only' (#1489) from ssh-mesh-only into main
Reviewed-on: #1489
2026-10-27 18:09:07 +00:00
claude-botandClaude Opus 5.5 12d11f8c50 feat(ansible): sshd listens on the NetBird interface only
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
New role ssh-mesh-only and play configure-ssh-mesh-only.yml. sshd moves to a
systemd socket with BindToDevice=wt0: on Arch a new sshd.socket (Accept=yes,
packaged sshd@.service), on Ubuntu a drop-in for ssh.socket that masks the
sshd-socket-generator output, plus BindsTo= so ssh.service stops with it.
NetBird recreates wt0 on every engine restart; BindToDevice stops the socket
with the old interface and a udev SYSTEMD_WANTS rule starts it with the new
one. The socket is not in sockets.target, which would hold boot until wt0.

The switchover runs in a transient unit and arms a revert timer first; the
play cancels it only after logging in again through the new listener and
seeing no listener outside wt0. [ssh_mesh_only] starts with proxy-2.

ssh-ca reloads sshd only when one sshd unit is running, as there is none
with per-connection or stopped socket-activated sshd.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 19:01:51 +01:00
k.renner 355b046abf Merge pull request 'feat(ansible): mesh inventory for the internet hosts, remove the Woodpecker CI SSH key' (#1488) from inventory-mesh-hosts into main
ci/woodpecker/push/deploy-infrastructure Pipeline was successful
Reviewed-on: #1488
2026-10-27 17:53:31 +00:00
claude-botandClaude Opus 5.5 b4933d65e5 feat(edge-host-iso): trust the SSH user CA
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
The build fetches the CA public key from the ssh-ca role URL and renders the
role drop-in, revocation list and root principal into the live system;
edge-install copies them to the installed system and checks sshd -T uses
the CA. Owner certificates log in as root from the first boot, with the
break-glass key as fallback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 18:29:57 +01:00
claude-botandClaude Opus 5.5 3187be1f62 chore: remove stale SSH keys
ci/woodpecker/pr/ci-scripts-test Pipeline failed
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
The edge-host ISO carried krenner@fortress, ansible-runner@ci-new and
krenner@krenner-XPS-13-9310; it now carries only the break-glass key.
The netbird-server revoke task for netbird-backup@homelab-cluster goes:
the exclusive managed authorized_keys already excludes it. OpenTofu drops
the unused ssh_public_key/ssh_private_key variables and the Proxmox
provider ssh block.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 18:20:48 +01:00
claude-botandClaude Opus 5.5 0c1fb0a229 feat: remove the Woodpecker CI SSH key
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was canceled
The woodpecker-ci@homelab key is dropped from the managed authorized_keys
and from the edge-host ISO. The Ansible pipelines that used it are deleted;
plays run from a workstation. deploy-uptime-kuma keeps only the OpenBao
token renewal cron, and deploy-infrastructure no longer installs the key
(no root-module resource uses the Proxmox provider SSH).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 18:18:15 +01:00
claude-botandClaude Opus 5.5 373b682902 feat(ansible): reach the internet hosts over the NetBird mesh
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
proxy-1, proxy-2 and netbird are addressed by their mesh names, so Ansible
for them runs from a mesh-connected workstation. CI cannot reach the mesh;
the pipelines targeting these hosts lose their push triggers and run only
on manual deployment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 16:45:46 +01:00
k.renner aad723252a Merge pull request 'fix(ansible): netbird-client downloads the release on the controller' (#1487) from netbird-client-controller-download into main
ci/woodpecker/push/deploy-netbird-client Pipeline failed
Reviewed-on: #1487
2026-10-27 15:35:13 +00:00
claude-botandClaude Opus 5.5 875fbfc730 fix(ansible): netbird-client downloads the release on the controller
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
The first workstation run failed on proxy-1 with 'Network is
unreachable' fetching the release: proxy-1 is IPv6-only and github.com
has no IPv6 address. The tarball is now downloaded and checksum-verified
on the controller (~/.cache/ansible/netbird) and unarchive copies it to
the host, so the hosts need no route to GitHub.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 16:34:37 +01:00
k.renner f56fc42c73 Merge pull request 'fix(ansible): install-netbird-client runs one host at a time' (#1486) from netbird-client-serial into main
ci/woodpecker/push/deploy-netbird-client Pipeline failed
Reviewed-on: #1486
2026-10-27 15:33:11 +00:00
79 changed files with 448 additions and 1595 deletions
-33
View File
@@ -1,33 +0,0 @@
when:
# Deploy only on main (merge) + manual — never auto-deploy from a branch.
- event: push
branch: main
path:
- "ansible/roles/claude-workstation/**"
- "ansible/plays/install-claude-workstation.yml"
- ".woodpecker/deploy-claude-workstation.yml"
- event: deployment
evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "deploy-claude-workstation"'
timeout: 30
steps:
- name: deploy-claude-workstation
image: gitea.renner.uno/gitea_admin/ci-ansible:2026-04-29
environment:
ANSIBLE_FORCE_COLOR: "true"
ANSIBLE_HOST_KEY_CHECKING: "false"
ANSIBLE_ROLES_PATH: "../roles:./roles"
SSH_PRIVATE_KEY:
from_secret: ssh_private_key
VAULT_PASS:
from_secret: vault_pass
commands:
- mkdir -p ~/.ssh
- echo "$SSH_PRIVATE_KEY" | base64 -d > ~/.ssh/id_ed25519
- chmod 600 ~/.ssh/id_ed25519
- printf "Host *\n\tStrictHostKeyChecking no\n\tUserKnownHostsFile /dev/null\n" > ~/.ssh/config
- cd ansible
- echo "$VAULT_PASS" > .vault_pass && chmod 600 .vault_pass
- export ANSIBLE_VAULT_PASSWORD_FILE=.vault_pass
- ansible-playbook -i inventory/ plays/install-claude-workstation.yml -v
-36
View File
@@ -1,36 +0,0 @@
when:
# Deploy only on main (merge) + manual — never auto-deploy from a branch.
- event: push
branch: main
path:
- "ansible/roles/mqtt-bridge/**"
- "ansible/plays/edge-mqtt-bridge.yml"
- "ansible/inventory/group_vars/edge_host/**"
- ".woodpecker/deploy-edge-mqtt-bridge.yml"
- event: deployment
evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "deploy-edge-mqtt-bridge"'
timeout: 30
steps:
- name: deploy-edge-mqtt-bridge
image: gitea.renner.uno/gitea_admin/ci-ansible:2026-04-29
environment:
ANSIBLE_FORCE_COLOR: "true"
ANSIBLE_HOST_KEY_CHECKING: "false"
ANSIBLE_ROLES_PATH: "../roles:./roles"
SSH_PRIVATE_KEY:
from_secret: ssh_private_key
VAULT_PASS:
from_secret: vault_pass
commands:
- mkdir -p ~/.ssh
- echo "$SSH_PRIVATE_KEY" | base64 -d > ~/.ssh/id_ed25519
- chmod 600 ~/.ssh/id_ed25519
- printf "Host *\n\tStrictHostKeyChecking no\n\tUserKnownHostsFile /dev/null\n" > ~/.ssh/config
- cd ansible
- echo "$VAULT_PASS" > .vault_pass && chmod 600 .vault_pass
- export ANSIBLE_VAULT_PASSWORD_FILE=.vault_pass
# Local MQTT bridge on the edge hosts only (REN-1307); restarts only the
# local mosquitto on edge-1, never the cluster broker.
- ansible-playbook -i inventory/ plays/edge-mqtt-bridge.yml -v
-7
View File
@@ -109,8 +109,6 @@ steps:
from_secret: hetzner_s3_access_key
TF_VAR_hetzner_s3_secret_key:
from_secret: hetzner_s3_secret_key
SSH_PRIVATE_KEY:
from_secret: ssh_private_key
PROXMOX_TOKEN_ID:
from_secret: proxmox_token_id
PROXMOX_TOKEN_SECRET:
@@ -121,11 +119,6 @@ steps:
ANSIBLE_ROLES_PATH: "../roles:./roles"
ANSIBLE_FORCE_COLOR: "true"
commands:
- mkdir -p ~/.ssh
- echo "$SSH_PRIVATE_KEY" | base64 -d > ~/.ssh/id_ed25519
- chmod 600 ~/.ssh/id_ed25519
- export TF_VAR_ssh_private_key="$(cat ~/.ssh/id_ed25519)"
- printf "Host *\n\tStrictHostKeyChecking no\n" > ~/.ssh/config
- echo "$VAULT_PASS" > /tmp/.vault_pass && chmod 600 /tmp/.vault_pass
- export ANSIBLE_VAULT_PASSWORD_FILE=/tmp/.vault_pass
- tofu init -upgrade -backend-config="access_key=$AWS_ACCESS_KEY_ID" -backend-config="secret_key=$AWS_SECRET_ACCESS_KEY"
-35
View File
@@ -1,35 +0,0 @@
when:
# Deploy only on main (merge) + manual — never auto-deploy from a branch.
- event: push
branch: main
path:
- "ansible/roles/netbird-client/**"
- "ansible/plays/install-netbird-client.yml"
- "ansible/inventory/group_vars/claude_workstation/**"
- ".woodpecker/deploy-netbird-client.yml"
- event: deployment
evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "deploy-netbird-client"'
timeout: 30
steps:
- name: deploy-netbird-client
image: gitea.renner.uno/gitea_admin/ci-ansible:2026-04-29
environment:
ANSIBLE_FORCE_COLOR: "true"
ANSIBLE_HOST_KEY_CHECKING: "false"
ANSIBLE_ROLES_PATH: "../roles:./roles"
SSH_PRIVATE_KEY:
from_secret: ssh_private_key
VAULT_PASS:
from_secret: vault_pass
commands:
- mkdir -p ~/.ssh
- echo "$SSH_PRIVATE_KEY" | base64 -d > ~/.ssh/id_ed25519
- chmod 600 ~/.ssh/id_ed25519
- printf "Host *\n\tStrictHostKeyChecking no\n\tUserKnownHostsFile /dev/null\n" > ~/.ssh/config
- cd ansible
- echo "$VAULT_PASS" > .vault_pass && chmod 600 .vault_pass
- export ANSIBLE_VAULT_PASSWORD_FILE=.vault_pass
# Enrol only the off-site Claude workstation into the Netbird mesh.
- ansible-playbook -i inventory/ plays/install-netbird-client.yml --limit claude_workstation -v
-54
View File
@@ -1,54 +0,0 @@
when:
# Deploy only on main (merge) + manual — never auto-deploy from a branch.
- event: push
branch: main
path:
- "ansible/roles/netbird-proxy/**"
# install-netbird-proxy.yml also applies the netbird-proxy-watchdog role,
# so changes to it must redeploy this host too.
- "ansible/roles/netbird-proxy-watchdog/**"
# install-netbird-proxy.yml also applies the edge bouncer.
- "ansible/roles/crowdsec-edge-bouncer/**"
- "ansible/plays/install-netbird-proxy.yml"
# The variables that steer those roles, not just the roles themselves.
# Without these two, a change that only touches inventory never deploys
# and nothing says so: arming the bouncer lives in host_vars, and its
# LAPI key lives in group_vars, so both would sit in git looking applied
# while the hosts kept running the previous configuration.
#
# pve_hosts.yml is deliberately NOT listed. It is the single inventory
# file for the whole fleet, so listing it would redeploy the proxies for
# any unrelated host edit. The cost is that a change to [netbird_proxy]
# group membership needs a manual deployment run.
- "ansible/inventory/host_vars/proxy-*.yml"
- "ansible/inventory/group_vars/netbird_proxy/**"
- ".woodpecker/deploy-netbird-proxy.yml"
- event: deployment
evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "deploy-netbird-proxy"'
timeout: 30
steps:
- name: deploy-netbird-proxy
image: gitea.renner.uno/gitea_admin/ci-ansible:2026-04-29
environment:
ANSIBLE_FORCE_COLOR: "true"
ANSIBLE_HOST_KEY_CHECKING: "false"
ANSIBLE_ROLES_PATH: "../roles:./roles"
SSH_PRIVATE_KEY:
from_secret: ssh_private_key
VAULT_PASS:
from_secret: vault_pass
PROXMOX_TOKEN_ID:
from_secret: proxmox_token_id
PROXMOX_TOKEN_SECRET:
from_secret: proxmox_token_secret
commands:
- mkdir -p ~/.ssh
- echo "$SSH_PRIVATE_KEY" | base64 -d > ~/.ssh/id_ed25519
- chmod 600 ~/.ssh/id_ed25519
- printf "Host *\n\tStrictHostKeyChecking no\n\tUserKnownHostsFile /dev/null\n" > ~/.ssh/config
- cd ansible
- echo "$VAULT_PASS" > .vault_pass && chmod 600 .vault_pass
- export ANSIBLE_VAULT_PASSWORD_FILE=.vault_pass
- ansible-playbook -i inventory/ plays/install-netbird-proxy.yml -v
-37
View File
@@ -1,37 +0,0 @@
when:
# Deploy only on main (merge) + manual — never auto-deploy from a branch.
- event: push
branch: main
path:
- "ansible/roles/netbird-server/**"
- "ansible/plays/install-netbird-server.yml"
- ".woodpecker/deploy-netbird-server.yml"
- event: deployment
evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "deploy-netbird-server"'
timeout: 30
steps:
- name: deploy-netbird-server
image: gitea.renner.uno/gitea_admin/ci-ansible:2026-04-29
environment:
ANSIBLE_FORCE_COLOR: "true"
ANSIBLE_HOST_KEY_CHECKING: "false"
ANSIBLE_ROLES_PATH: "../roles:./roles"
SSH_PRIVATE_KEY:
from_secret: ssh_private_key
VAULT_PASS:
from_secret: vault_pass
PROXMOX_TOKEN_ID:
from_secret: proxmox_token_id
PROXMOX_TOKEN_SECRET:
from_secret: proxmox_token_secret
commands:
- mkdir -p ~/.ssh
- echo "$SSH_PRIVATE_KEY" | base64 -d > ~/.ssh/id_ed25519
- chmod 600 ~/.ssh/id_ed25519
- printf "Host *\n\tStrictHostKeyChecking no\n\tUserKnownHostsFile /dev/null\n" > ~/.ssh/config
- cd ansible
- echo "$VAULT_PASS" > .vault_pass && chmod 600 .vault_pass
- export ANSIBLE_VAULT_PASSWORD_FILE=.vault_pass
- ansible-playbook -i inventory/ plays/install-netbird-server.yml -v
-37
View File
@@ -1,37 +0,0 @@
when:
# Deploy only on main (merge) + manual — never auto-deploy from a branch.
- event: push
branch: main
path:
- "ansible/roles/ssh-ca/**"
- "ansible/plays/configure-ssh-ca.yml"
- "ansible/inventory/pve_hosts.yml"
- "ansible/inventory/group_vars/all/**"
- "ansible/inventory/group_vars/netbird_proxy/vars.yml"
- "ansible/inventory/host_vars/netbird.renner.uno.yml"
- ".woodpecker/deploy-ssh-ca.yml"
- event: deployment
evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "deploy-ssh-ca"'
timeout: 30
steps:
- name: deploy-ssh-ca
image: gitea.renner.uno/gitea_admin/ci-ansible:2026-04-29
environment:
ANSIBLE_FORCE_COLOR: "true"
ANSIBLE_HOST_KEY_CHECKING: "false"
ANSIBLE_ROLES_PATH: "../roles:./roles"
SSH_PRIVATE_KEY:
from_secret: ssh_private_key
VAULT_PASS:
from_secret: vault_pass
commands:
- mkdir -p ~/.ssh
- echo "$SSH_PRIVATE_KEY" | base64 -d > ~/.ssh/id_ed25519
- chmod 600 ~/.ssh/id_ed25519
- printf "Host *\n\tStrictHostKeyChecking no\n\tUserKnownHostsFile /dev/null\n" > ~/.ssh/config
- cd ansible
- echo "$VAULT_PASS" > .vault_pass && chmod 600 .vault_pass
- export ANSIBLE_VAULT_PASSWORD_FILE=.vault_pass
- ansible-playbook -i inventory/ plays/configure-ssh-ca.yml -v
-33
View File
@@ -1,13 +1,4 @@
when:
# Deploy only on main (merge) + manual — never auto-deploy from a branch.
- event: push
branch: main
path:
- "ansible/roles/uptime-kuma/**"
- "ansible/plays/install-uptime-kuma.yml"
- ".woodpecker/deploy-uptime-kuma.yml"
- event: deployment
evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "deploy-uptime-kuma"'
- event: cron
cron: uptime-kuma-token-renew
@@ -21,27 +12,3 @@ steps:
from_secret: openbao_uptime_kuma_token
commands:
- 'curl -sSf -X POST -H "X-Vault-Token: $VAULT_TOKEN" https://openbao.renner.uno/v1/auth/token/renew-self -o /dev/null'
- name: deploy-uptime-kuma
image: gitea.renner.uno/gitea_admin/ci-ansible:2026-10-18
when:
- event: [push, deployment]
environment:
ANSIBLE_FORCE_COLOR: "true"
ANSIBLE_HOST_KEY_CHECKING: "false"
ANSIBLE_ROLES_PATH: "../roles:./roles"
SSH_PRIVATE_KEY:
from_secret: ssh_private_key
VAULT_PASS:
from_secret: vault_pass
VAULT_TOKEN:
from_secret: openbao_uptime_kuma_token
commands:
- mkdir -p ~/.ssh
- echo "$SSH_PRIVATE_KEY" | base64 -d > ~/.ssh/id_ed25519
- chmod 600 ~/.ssh/id_ed25519
- printf "Host *\n\tStrictHostKeyChecking no\n\tUserKnownHostsFile /dev/null\n" > ~/.ssh/config
- cd ansible
- echo "$VAULT_PASS" > .vault_pass && chmod 600 .vault_pass
- export ANSIBLE_VAULT_PASSWORD_FILE=.vault_pass
- ansible-playbook -i inventory/ plays/install-uptime-kuma.yml -v
-28
View File
@@ -1,28 +0,0 @@
when:
# Manual only: reboots edge-1 (Z-Wave, heating CAN, sole NTP server) and the
# NetBird control plane. Run inside the 02:00-05:00 Europe/Berlin window.
- event: deployment
evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "update-os"'
timeout: 120
steps:
- name: update-os
image: gitea.renner.uno/gitea_admin/ci-ansible:2026-04-29
environment:
ANSIBLE_FORCE_COLOR: "true"
ANSIBLE_HOST_KEY_CHECKING: "false"
ANSIBLE_ROLES_PATH: "../roles:./roles"
SSH_PRIVATE_KEY:
from_secret: ssh_private_key
VAULT_PASS:
from_secret: vault_pass
commands:
- mkdir -p ~/.ssh
- echo "$SSH_PRIVATE_KEY" | base64 -d > ~/.ssh/id_ed25519
- chmod 600 ~/.ssh/id_ed25519
- printf "Host *\n\tStrictHostKeyChecking no\n\tUserKnownHostsFile /dev/null\n" > ~/.ssh/config
- cd ansible
- echo "$VAULT_PASS" > .vault_pass && chmod 600 .vault_pass
- export ANSIBLE_VAULT_PASSWORD_FILE=.vault_pass
- ansible-playbook -i inventory/ plays/update-os.yml -v
+9 -4
View File
@@ -112,7 +112,7 @@
# (string) This setting controls the default policy of fact gathering (facts discovered about remote systems).
# This option can be useful for those wishing to save fact gathering time. Both 'smart' and 'explicit' will use the cache plugin.
;gathering=implicit
gathering=smart
# (list) Set the `gather_subset` option for the :ref:`ansible_collections.ansible.builtin.setup_module` task in the implicit fact gathering. See the module documentation for specifics.
# It does **not** apply to user defined ``ansible.builtin.setup`` tasks.
@@ -134,7 +134,7 @@
;hash_behaviour=replace
# (pathlist) Comma separated list of Ansible inventory sources
inventory=~/source/homelab/ansible/inventory
inventory=inventory
# (pathspec) Colon separated paths in which Ansible will search for HttpApi Plugins.
;httpapi_plugins={{ ANSIBLE_HOME ~ "/plugins/httpapi:/usr/share/ansible/plugins/httpapi" }}
@@ -274,7 +274,7 @@ roles_path=../roles:./roles
# (path) The vault password file to use. Equivalent to --vault-password-file or --vault-id
# If executable, it will be run and the resulting stdout will be used as the password.
vault_password_file=~/source/homelab/ansible/vault-password.sh
vault_password_file=vault-password.sh
# (integer) Sets the default verbosity, equivalent to the number of ``-v`` passed in the command line.
;verbosity=0
@@ -484,7 +484,7 @@ vault_password_file=~/source/homelab/ansible/vault-password.sh
# It can result in a very significant performance improvement when enabled.
# However this conflicts with privilege escalation (become). For example, when using 'sudo:' operations you must first disable 'requiretty' in /etc/sudoers on all managed hosts, which is why it is disabled by default.
# This setting will be disabled if ``ANSIBLE_KEEP_REMOTE_FILES`` is enabled.
;pipelining=False
pipelining=True
[colors]
@@ -694,3 +694,8 @@ vault_password_file=~/source/homelab/ansible/vault-password.sh
# (list) default list of tags to skip in your plays, has precedence over Run Tags
;skip=
[ssh_connection]
# Connections stay open across the plays of a run, so each host costs one
# YubiKey PIN; plays/close-connections.yml closes them at the end of a run.
ssh_args=-C -o ControlMaster=auto -o ControlPersist=30m
+4 -7
View File
@@ -41,13 +41,11 @@ Build and boot the dedicated installer image — see
4. Add the host to `[edge_host]` in `inventory/pve_hosts.yml`, by name.
That group is the base: every edge thin client belongs to it, and it
assumes no peripherals.
5. From the workstation:
5. From the workstation, in `ansible/`:
```
ansible-playbook -i inventory/pve_hosts.yml plays/install-edge-host.yml
ansible-playbook site.yml
```
One unrestricted run over the whole group. The play is self-contained --
it carries `certificates` and `common` itself, so there is no second
command and no `--limit`. Running it converges every edge host, which is
One unrestricted run, no `--limit`. It converges every host, which is
the point: a run that touches only the new box tells you nothing about
the others.
@@ -245,8 +243,7 @@ its own server key, nothing else. On a push it writes
`/etc/mosquitto/bridge.d/edge-1.conf`, deletes `bridge.d/legacy.conf` and
restarts the local mosquitto. The controller sets a new password whenever the
bridge reports "not connected" (also after a reinstall), so no play run is
needed for a rotation. `.woodpecker/deploy-edge-mqtt-bridge.yml` runs
`plays/edge-mqtt-bridge.yml` on merge. Diagnosis: `journalctl -u
needed for a rotation. Diagnosis: `journalctl -u
edge-mqtt-cred`, and `mosquitto_sub -t '$SYS/broker/connection/edge-1/state'`
(retained `1` = connected).
+2 -29
View File
@@ -5,37 +5,10 @@ pve_token_id: "ansible"
ansible_user: "ladmin"
ansible_python_interpreter: /usr/bin/python3
# The only plain keys the admin logins accept on [ssh_keys_managed] hosts
# The only plain key the admin logins accept on [ssh_keys_managed] hosts
# (ssh-ca role, tasks/authorized_keys.yml). People log in with OpenBao SSH CA
# certificates; these two are for when that path is unavailable.
# certificates; this one is for when that path is unavailable.
# breakglass-neo: PIV key (slot 9a, PIN) on the offline YubiKey NEO.
# ssh -I /usr/lib/opensc-pkcs11.so root@<host>
# woodpecker-ci: the CI deploy key, from ssh_ci_key_from where set.
ssh_admin_authorized_keys:
- key: "ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKvSl9LUP8u3S2Sq5YTWpW0Jxbue8cJhD7/LMFQbEsa3xo145HVR7pBzRxW71MUy+jgrgmpSlDhIb606Hswhvoo= breakglass-neo-6532348"
- key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICyNnX1I4W9jyDvCJ1DJauHSmbo1ZXSyBxVhSAwMWgen woodpecker-ci@homelab"
options: "{{ ('from=\"' ~ ssh_ci_key_from ~ '\"') if ssh_ci_key_from else '' }}"
# Source the CI key is accepted from. LAN hosts see the pod address; hosts on
# the internet see the home WAN address (NAT, dynamic) and set this empty in
# their group/host vars.
ssh_ci_key_from: "{{ ci_ssh_from_cidrs }}"
# Network-of-origin restriction for the woodpecker-ci@homelab CI key
# (issue #184). Used as the authorized_keys `from=` option so the CI key only
# authenticates from within the cluster's pod network — NOT from any
# host-pinned IP.
#
# The in-cluster Woodpecker step pods SSH to fleet hosts using their POD IP as
# source, not a node IP: Cilium runs `ipv4-native-routing-cidr=10.0.0.0/8` +
# `ipv6-native-routing-cidr=fd00::/8` with masquerade enabled, so pod egress to
# any 10.x / fd.. LAN destination is NOT SNAT'd, and each node's PodCIDR is
# advertised to the UDM over BGP (kubernetes/infrastructure/post-crds/
# cilium-bgp.yaml) so return traffic L3-routes back to the pod. Hosts therefore
# see the real pod source IP.
#
# The cluster pod subnets (kubernetes/talos/patches/cluster.yaml podSubnets)
# are the stable boundary — they survive node DHCP-lease churn and pod
# rescheduling across k8s-1/2/3, unlike pinning any node's current IP.
# v6 is included because some SSH targets (edge IoT hosts) are IPv6-only.
ci_ssh_from_cidrs: '10.244.0.0/16,fd10:244::/48'
@@ -1,16 +0,0 @@
$ANSIBLE_VAULT;1.1;AES256
35646538363164363061366433343963623166336630376233333033646637336266343337343230
3233356135323762636439336466663430366238363338650a633762623366643137303333383634
62623739363132303335636261626263333332623066303161633466306431623930666166323833
6439663964353338350a656435626664646666363831363164393335386666356637613037323838
63653461656366613331353064656161346263356131346437666632386166623037373363333534
37323435623366643534633333303534313434613135653237303966656162633266623433363934
36343133393834393765353338383764303664363431653865633764306138376366623832346262
36303766623331666632613639366139333661636665363631326565653938343637323861653064
37393465383263376337376639613162306563653365623036376662303030613264653534353332
66363030356363373439306164386561363734353631363738656236633164633062613437663366
62386430396262366336316166353366383233316333333234666538643962336361363630363762
31653963303032373531386566336632336130383034343430303964396538663237346434613737
39643765623766373639306132316635376434643232336163303066393736313130323065666232
39393038616162666666383837326137376233613437636361636238663566336365653863643933
353338646134383038323134323034653564
@@ -50,3 +50,11 @@ edge_mqtt_host: 127.0.0.1
edge_mqtt_port: 1883
edge_open3e_mqtt_user: open3e
edge_open3e_mqtt_topic: open3e
# Units claude-bot may restart (ssh-ca role); the containers go through docker.
ssh_ca_claude_sudo_units:
- mosquitto
- edge-mqtt-cred
- edge-alloy
- keepalived
- chronyd
@@ -1,8 +1,9 @@
---
# CI reaches these internet hosts NATed from the home WAN address, never from
# the pod network, so the CI key carries no from= restriction here.
ssh_ci_key_from: ""
# The host NetBird client (wt0) must not take over DNS: resolv.conf is pinned to
# the systemd-resolved stub by the netbird-proxy role.
netbird_disable_dns: true
ssh_ca_claude_sudo_units:
- netbird
- netbird-proxy-flood-watchdog
- crowdsec-firewall-bouncer
@@ -3,6 +3,5 @@
# Enable Docker monitoring (cAdvisor) for container metrics
docker_monitoring_enabled: true
# CI reaches this internet host NATed from the home WAN address, never from
# the pod network, so the CI key carries no from= restriction here.
ssh_ci_key_from: ""
ssh_ca_claude_sudo_units:
- netbird
+13 -13
View File
@@ -13,11 +13,11 @@
# Their hosts no longer resolve, except the three that resolve to Talos nodes.
[netbird_server]
netbird.renner.uno ansible_user=root
netbird.renner.uno ansible_host=netbird.nb.renner.uno ansible_user=root
[netbird_proxy]
proxy-1.renner.uno ansible_user=root
proxy-2.renner.uno ansible_user=root
proxy-1.renner.uno ansible_host=proxy-1.nb.renner.uno ansible_user=root
proxy-2.renner.uno ansible_host=proxy-2.nb.renner.uno ansible_user=root
# Every bare-metal edge thin client. Gets the host-generic roles only:
# flash-wear handling, the log shipper, the MQTT bridge, the VLAN
@@ -62,14 +62,6 @@ heating_can
[ntp_cluster:children]
edge_host
# Off-site Debian VM (Synology VMM on nas-2, 10.3.40.0/24) running Claude Code.
# Reached over the site-to-site tunnel at its current DHCP lease. Switch
# ansible_host to the stable Netbird mesh IP once the host is enrolled — the
# obg.renner.uno FQDN is split-horizon and resolves to the public proxy from
# home, so it must NOT be used as ansible_host.
[claude_workstation]
claude-ws ansible_host=10.3.40.78 ansible_user=root
# Hosts that trust the OpenBao SSH user CA (plays/configure-ssh-ca.yml).
# Rolled out one host at a time: add the next host after the previous one
# was verified.
@@ -79,11 +71,19 @@ proxy-2.renner.uno
netbird.renner.uno
edge-1.iot.renner.uno
# Hosts whose admin logins accept only ssh_admin_authorized_keys (break-glass
# + CI); everything else needs a CA certificate. Members must be in [ssh_ca].
# Hosts whose admin logins accept only ssh_admin_authorized_keys (break-glass);
# everything else needs a CA certificate. Members must be in [ssh_ca].
# Added one host at a time after the [ssh_ca] report was reviewed.
[ssh_keys_managed]
proxy-1.renner.uno
proxy-2.renner.uno
netbird.renner.uno
edge-1.iot.renner.uno
# Internet hosts whose sshd listens on the NetBird interface only
# (plays/configure-ssh-mesh-only.yml). Members need the host NetBird client.
# Added one host at a time after the previous one was verified.
[ssh_mesh_only]
proxy-1.renner.uno
proxy-2.renner.uno
netbird.renner.uno
+11
View File
@@ -0,0 +1,11 @@
---
# Last play of every playbook that is run directly (site.yml and the one-off
# actions). SSH connections stay open for 30 minutes (ansible.cfg) so a run
# costs one YubiKey PIN per host; closing them at the end leaves nothing that
# another process of the same user could reuse.
- name: Close the SSH connections
hosts: all
gather_facts: false
tasks:
- name: Close the connection
ansible.builtin.meta: reset_connection
+2 -1
View File
@@ -13,4 +13,5 @@
roles:
- common
- qemu-guest-agent
- role: qemu-guest-agent
when: ansible_facts['virtualization_role'] == 'guest'
+6 -10
View File
@@ -2,11 +2,12 @@
# Trust the OpenBao SSH user CA (ssh-client-signer) on the hosts in [ssh_ca]:
# owner certificates (principal k.renner) for root/ladmin, Claude's 1h
# certificates (principal claude-bot) for its own claude-bot account with
# limited sudo. The first play only reports the admin logins' authorized_keys;
# the second reduces them to ssh_admin_authorized_keys on [ssh_keys_managed].
# limited sudo. The role reports the admin logins' authorized_keys; on
# [ssh_keys_managed] they are then reduced to ssh_admin_authorized_keys.
#
# serial: 1 so a host whose sshd rejects the change stops the run before the
# next host is touched.
# next host is touched. One play, so each host needs one login (one YubiKey
# PIN from a workstation).
- name: Configure the SSH user CA
hosts: ssh_ca
become: true
@@ -14,14 +15,9 @@
serial: 1
roles:
- ssh-ca
- name: Reduce authorized_keys to break-glass and CI
hosts: ssh_keys_managed
become: true
gather_facts: false
serial: 1
tasks:
- name: Set the managed authorized_keys
- name: Reduce authorized_keys to the break-glass key
ansible.builtin.include_role:
name: ssh-ca
tasks_from: authorized_keys
when: inventory_hostname in groups['ssh_keys_managed']
+15
View File
@@ -0,0 +1,15 @@
---
# sshd on the hosts in [ssh_mesh_only] accepts connections only on the NetBird
# interface; nothing listens on their public addresses. Run from a workstation
# in the mesh. A switchover reverts by itself unless the play logs in again
# through the new listener.
#
# serial: 1 so a host that does not come back stops the run before the next
# host is touched.
- name: Restrict sshd to the mesh interface
hosts: ssh_mesh_only
become: true
gather_facts: true
serial: 1
roles:
- ssh-mesh-only
-33
View File
@@ -1,33 +0,0 @@
---
# Local MQTT bridge and its credential receiver only (REN-1307). Run by
# .woodpecker/deploy-edge-mqtt-bridge.yml on merge, so a change to this role
# reaches edge-1 without a play run by hand. The full base play
# (install-edge-host.yml) carries the same role.
- name: Converge the edge MQTT bridge
hosts: edge_host
become: true
# The SML power-meter reader (sml2mqtt) is retired for good: owner decision
# 2026-09-19, grid data comes from the E380 on the CAN bus. Its role and
# play are gone; this removes the leftover compose project from edge-1 on
# the next pipeline run. Idempotent: a no-op once the directory is gone.
pre_tasks:
- name: Check for the retired sml2mqtt compose project
ansible.builtin.stat:
path: /opt/edge-host/sml2mqtt/docker-compose.yml
register: edge_sml2mqtt_compose
- name: Stop the retired sml2mqtt container
ansible.builtin.command:
cmd: docker compose down --remove-orphans
chdir: /opt/edge-host/sml2mqtt
when: edge_sml2mqtt_compose.stat.exists
changed_when: true
- name: Remove the retired sml2mqtt directory
ansible.builtin.file:
path: /opt/edge-host/sml2mqtt
state: absent
roles:
- mqtt-bridge
@@ -1,20 +0,0 @@
---
# Off-site Debian VM running Claude Code — reattachable interactive sessions
# plus an unattended notes-distill timer. Host creation (Synology VMM), the
# Netbird dashboard route and secret material are manual prerequisites; see
# roles/claude-workstation/README.md and docs/claude-workstation.md.
#
# CI reaches this off-site host over the site-to-site tunnel (home -> off-site
# works), same as it reaches every other inventory host.
- name: Deploy the Claude Code workstation
hosts: claude_workstation
become: true
# qemu-guest-agent is applied
# to every inventory host by the umbrella plays/common.yml, same as for the
# netbird-proxy host — run that against this host too (see RUNBOOK Step 5).
roles:
- common
- role: netbird-client # mesh enrolment (needs netbird_setup_key in vault)
when: netbird_setup_key is defined
- claude-workstation
+1 -5
View File
@@ -6,11 +6,7 @@
# - host_vars/<hostname>/vars.yml - per-host key (overrides group)
# - group_vars/all/vault.yml - encrypted keys
#
# Usage:
# ansible-playbook -i inventory/ plays/setup-netbird-client.yml
#
# Limit to specific hosts:
# ansible-playbook -i inventory/ plays/setup-netbird-client.yml --limit caddy_servers
# Applied by site.yml.
- name: Install NetBird client on hosts with setup key defined
hosts: all
+1 -1
View File
@@ -1,6 +1,6 @@
---
# Deploy the NetBird reverse-proxy HA cluster members (the public edge).
# Usage: ansible-playbook -i inventory/ plays/install-netbird-proxy.yml
# Applied by site.yml.
#
# Prerequisites:
# - Proxy VPS provisioned (proxy-1 is Arch Linux), root SSH reachable, and
+1 -1
View File
@@ -1,6 +1,6 @@
---
# Deploy NetBird Server to VPS
# Usage: ansible-playbook -i inventory/ plays/install-netbird-server.yml
# Applied by site.yml.
#
# Prerequisites:
# - VPS provisioned with Ubuntu 22.04/24.04
-22
View File
@@ -1,22 +0,0 @@
---
# Install Node Exporter on all managed hosts
# Usage: ansible-playbook -i inventory/ plays/install-node-exporter.yml
- name: Install Node Exporter
hosts: all
become: yes
gather_facts: yes
roles:
- node-exporter
post_tasks:
- name: Verify node-exporter is running
uri:
url: "http://127.0.0.1:9100/metrics"
status_code: 200
register: exporter_health
retries: 3
delay: 5
until: exporter_health.status == 200
ignore_errors: yes
+1 -1
View File
@@ -1,6 +1,6 @@
---
# Deploy Uptime Kuma to the NetBird VPS
# Usage: ansible-playbook -i inventory/ plays/install-uptime-kuma.yml
# Applied by site.yml.
#
# Prerequisites:
# - netbird-server play applied (docker, Traefik on netbird_netbird)
+3
View File
@@ -248,3 +248,6 @@
- "Commit inventory/group_vars/all/netbird-rotatable.yml — it is encrypted."
- "NOT rotated: datastore_key (needs dump/restore), auth_client_secret (Authentik side)."
- "OpenBao is unaffected by design — its NetBird credentials are separate ones."
- name: Close the SSH connections
ansible.builtin.import_playbook: close-connections.yml
+3
View File
@@ -111,3 +111,6 @@
when:
- (ansible_distribution == 'Ubuntu' and reboot_required.stat.exists) or
(ansible_distribution == 'Debian')
- name: Close the SSH connections
ansible.builtin.import_playbook: close-connections.yml
+4 -1
View File
@@ -1,7 +1,7 @@
---
# Package upgrade + reboot for netbird_proxy, edge_host and netbird_server,
# one host at a time.
# Usage: ansible-playbook -i inventory/ plays/update-os.yml
# Usage (from ansible/): ansible-playbook plays/update-os.yml
#
# Plays run in order; a failed host stops the whole run (max_fail_percentage: 0).
@@ -28,3 +28,6 @@
become: true
roles:
- os-update
- name: Close the SSH connections
ansible.builtin.import_playbook: close-connections.yml
@@ -1,57 +0,0 @@
# claude-workstation
An always-on, **off-site** headless Claude Code host (Debian 12 VM under Synology
VMM on nas-2). It runs (a) reattachable interactive `tmux` sessions and (b) an
unattended notes-distill timer. It deliberately holds the **least** privilege
that still does the job.
## What lives here vs. what's manual
This role configures everything **inside** the VM. Three things are **manual**
(no IaC path) and precede/accompany it — see [`RUNBOOK.md`](RUNBOOK.md) for the
full step-by-step:
1. **VM creation** in Synology VMM (`ladmin` has no sudo; the DSM API is
root-only), with a cloud-init drive that installs the CI SSH key.
2. **Netbird dashboard route** advertising the k8s API to the mesh, and the
**site-to-site return route** on the off-site gateway (backup path).
3. **Secret material**: `claude setup-token` (subscription OAuth), the claude-bot
creds, and the read-only SA kubeconfig — encrypted into the group vault.
## Credential isolation
Two Unix users with separate credential stores so an unattended job cannot read
the interactive session's secrets:
| User | Holds | Runs |
|------|-------|------|
| `claude` | OAuth token, claude-bot creds, **read-only** SA kubeconfig | interactive tmux |
| `claude-job` | its **own** OAuth token, claude-bot creds (no kubeconfig) | notes-distill timer |
**No talosconfig, no SOPS age key** ever land here (Fable review #1). The
kubeconfig is a read-only `view` ServiceAccount (`kubernetes/apps/claude-workstation/`).
## Egress lockdown (staged)
`claude_ws_egress_enforce` ships **`false`** so a fresh host is never bricked.
The deny-default nftables ruleset, the DNS-resolver, and its refresh timer are
installed regardless. Flip to `true` (in host/group vars) only after confirming
`claude_ws_egress_allow_fqdns` / `claude_ws_egress_allow_cidrs*` cover real
traffic — watch the `claude-egress-dropped` counter first.
## Vault keys (encrypt into `inventory/group_vars/claude_workstation/vault.yml`)
```
netbird_setup_key # from the Netbird dashboard
claude_ws_oauth_token_interactive # `claude setup-token` (interactive user)
claude_ws_oauth_token_job # `claude setup-token` (job user, separate)
claude_ws_gitea_token # claude-bot PAT
claude_ws_gitea_git_credentials # https://claude-bot:<pat>@gitea.renner.uno
claude_ws_kubeconfig # read-only SA kubeconfig (literal YAML)
```
## Kill switch
Everything unattended hangs off systemd timers; `systemctl disable --now
claude-notes-distill.timer` (and the egress refresh) stops it. Every credential
is independently revocable (SA token, claude-bot PAT, OAuth token, Netbird peer).
-221
View File
@@ -1,221 +0,0 @@
# Runbook — off-site Claude Code workstation (nas-2)
Operator guide for the manual steps around the `claude-workstation` role. The
role automates everything **inside** the VM; this covers what it can't:
VM creation, networking, secret material, egress cutover, rotation and DR.
## Architecture (scope)
An always-on **Debian 13 VM under Synology VMM on the off-site nas-2**
(DS1522+, 10.3.40.0/24, reached over the site-to-site VPN, ~55 ms). It runs:
- **Reattachable interactive sessions** (`tmux` over SSH as user `claude`).
- **Unattended notes-distill timer** (as user `claude-job`).
- **A thin is-home-up signal** (the existing external Gatus on the Netbird VPS
already covers this — don't duplicate it).
**Out of scope (runs in-cluster instead):** scheduled repo-automation and
cluster-reports go in as Woodpecker cron / CronJobs with an in-cluster
ServiceAccount — *not* on this VM, so the VM holds no cluster-write creds.
Credentials on the VM are minimal: a **read-only** `view` SA kubeconfig,
claude-bot (non-admin) gitea creds, and a Claude subscription OAuth token.
**No talosconfig, no SOPS age key.**
---
## Step 1 — Create + install the VM (Synology VMM)
`ladmin` has no sudo and the DSM API is root-only, so VMM is driven through the
DSM **web UI**. Two hard facts dictate the method (all confirmed 2026-07-16, as
built): **VMM has no cloud-init**, and its ISO-install wizard only creates blank
disks. So the OS is installed from a **remastered Debian netinst ISO that
auto-boots a preseed** — no console interaction, no boot-menu race.
VMM gotchas that shape this (see the wiki build-record):
- VMM refuses to change/unmount a VM's ISO while it is **running** ("Failed to
replace ISO files"). Do all ISO/boot-device edits with the VM **off**.
- A **guest reboot does not re-read** VMM's boot-device/ISO config — only a full
VMM power-cycle applies it. So an auto-booting install ISO **loops** (reboot →
CD → reinstall, wiping the disk). Break it by preseeding the installer to
**power off** at the end: `d-i debian-installer/exit/poweroff boolean true`.
- Attach the NIC to the **untagged `Default VM Network`** — the VLAN-40
`40_Server` network is isolated (no DHCP, no network). The LAN has DHCP.
Procedure (build the ISO on a host with `xorriso`):
1. Download the current Debian **netinst** (`.../debian-cd/current/amd64/iso-cd/`).
2. Remaster it — add `/preseed.cfg` and replace `/isolinux/isolinux.cfg` with a
minimal one (`default preseed`, `timeout 1`, one label:
`kernel /install.amd/vmlinuz` /
`append auto=true priority=critical preseed/file=/cdrom/preseed.cfg vga=788 initrd=/install.amd/initrd.gz --- quiet`):
`xorriso -indev in.iso -outdev out.iso -boot_image any replay -map preseed.cfg /preseed.cfg -map isolinux.cfg /isolinux/isolinux.cfg`.
The preseed uses **DHCP** (`netcfg/dhcp_timeout 120` — the virtual-switch port
is slow to link), disk **`/dev/sda`** (VirtIO SCSI), **no user** + a locked
random root hash (key-only via `PermitRootLogin prohibit-password`), packages
`openssh-server sudo qemu-guest-agent python3`, and a `late_command` that
injects the SSH keys (`krenner@fortress`, `woodpecker-ci@homelab`, your
workstation key) into `/root/.ssh/authorized_keys`, plus the `poweroff` line.
3. Upload to nas-2 via `ssh 'cat > /volume1/Install/<iso>' < <iso>` — **scp /
sftp / rsync are blocked on DSM**. Then VMM → Image → Add → From Synology NAS
→ import (VMM **copies** it; re-import if you change the source file).
4. Create VM: 2 vCPU, 4 GB, 50 GB on `/volume1`, **Default VM Network**, Legacy
BIOS, **Autostart: Yes**; attach the ISO, **Boot from DVD/CD-ROM**, power on.
5. It installs unattended and **powers off**. Then (VM off) set **ISO =
Unmounted**, **Boot from = Virtual Disk**, power on → boots the installed
Debian. `qemu-guest-agent` reports the DHCP lease in VMM's IP column.
6. **Pin a DSM update window** (Control Panel → Update & Restore) so an
auto-reboot doesn't kill tmux/jobs.
> As built 2026-07-16: Debian 13 (trixie), got DHCP `10.3.40.78`, all three keys
> injected, packages present. Freeing RAM required powering off the pre-existing
> `pbs1` PBS VM (kept its 3.4 TB backup disk).
> **At-rest encryption (Fable #6).** Off-site physical custody is weaker than
> home. For true at-rest protection of the VM's secrets *with* unattended
> autostart, use **network-bound decryption** (clevis + a Tang server at home):
> a stolen disk off-site can't unlock because it can't reach Tang. Full-disk
> LUKS with a manual passphrase would break autostart. Until Tang is stood up,
> secrets rely on 0600 perms + VM access control — documented gap, not a fix.
---
## Step 2 — Networking (two paths)
**Netbird mesh (done via CI):** mint a setup key in the dashboard
(`https://netbird.renner.uno`), vault it as `netbird_setup_key` under
`group_vars/claude_workstation/`, and run `.woodpecker/deploy-netbird-client.yml`
(`install-netbird-client.yml --limit claude_workstation`). As built 2026-07-16
the VM enrolled as `claude-ws-190-112.nb.renner.uno` / `100.96.190.112` (+ IPv6).
> **Mesh reachability caveat.** The mesh IP (`100.96.x`) is only reachable from
> **other Netbird peers**. The home workstation and the Woodpecker CI agent are
> **not** peers — they reach the VM only via the site-to-site tunnel at its DHCP
> lease. So a *stable* mesh-based `ansible_host` needs CI (or a home host) on the
> mesh first, or a Netbird routing peer bridging tunnel↔mesh. Until then the
> inventory targets the tunnel DHCP IP.
> **DNS trap.** `claude-ws.obg.renner.uno` resolves to the VM's lease **only at
> the off-site DNS** (`@10.3.40.1`). From home this FQDN does not resolve (it
> exists only at the off-site DNS @10.3.40.1). Never use it as `ansible_host`.
**Optional — mesh route to the k8s API (redundant read path):** to reach the API
over the mesh too, add a Netbird dashboard **network route** advertising the
apiserver's ClusterIP to the `netbird-ingress` peer group, and add an egress
rule to the `netbird-ingress-egress` CiliumNetworkPolicy
(`kubernetes/infrastructure/network-policies/ingress-isolation.yaml`) permitting
`app: netbird-ingress` pods to reach `kube-apiserver:443`. Deferred from this PR
because it edits a security-critical CNP that can't be validated offline — do it
as a follow-up when you want the redundant API path.
---
## Step 3 — Read-only SA kubeconfig
After the `kubernetes/apps/claude-workstation/` PR merges and Flux reconciles:
```sh
NS=apps
TOKEN=$(kubectl -n $NS get secret claude-workstation-token -o jsonpath='{.data.token}' | base64 -d)
kubectl -n $NS get secret claude-workstation-token -o jsonpath='{.data.ca\.crt}' | base64 -d > /tmp/ca.crt
# server = the cluster's stable API endpoint (kubernetes/talos/patches/cluster.yaml).
# Use the name, not an address: it is what the API server certificate carries as a
# SAN, so an IP here fails TLS verification even when it is reachable. It must
# resolve from the workstation over the site-to-site route.
kubectl config set-cluster homelab --server=https://k8s.renner.uno:6443 \
--certificate-authority=/tmp/ca.crt --embed-certs=true --kubeconfig=/tmp/claude-ws.kubeconfig
kubectl config set-credentials claude-workstation --token="$TOKEN" --kubeconfig=/tmp/claude-ws.kubeconfig
kubectl config set-context homelab --cluster=homelab --user=claude-workstation --kubeconfig=/tmp/claude-ws.kubeconfig
kubectl config use-context homelab --kubeconfig=/tmp/claude-ws.kubeconfig
# verify read-only:
KUBECONFIG=/tmp/claude-ws.kubeconfig kubectl get nodes # works
KUBECONFIG=/tmp/claude-ws.kubeconfig kubectl get secrets -A # Forbidden (view excludes secrets)
```
The resulting file becomes the `claude_ws_kubeconfig` vault value.
---
## Step 4 — Secret material → vault
```sh
# Two independent OAuth tokens so job jobs can't impersonate the interactive user:
claude setup-token # -> claude_ws_oauth_token_interactive
claude setup-token # -> claude_ws_oauth_token_job (run again / separate session)
```
Encrypt into `ansible/inventory/group_vars/claude_workstation/vault.yml`
(ansible-vault; same VAULT_PASS the CI uses):
```yaml
netbird_setup_key: "..."
claude_ws_oauth_token_interactive: "..."
claude_ws_oauth_token_job: "..."
claude_ws_gitea_token: "..." # claude-bot PAT
claude_ws_gitea_git_credentials: "https://claude-bot:<PAT>@gitea.renner.uno"
claude_ws_kubeconfig: |
apiVersion: v1
... # the file from Step 3
```
> **Never run `claude` with `--bare`** here or on the VM — bare mode silently
> ignores `CLAUDE_CODE_OAUTH_TOKEN` and the jobs will fail auth.
---
## Step 5 — Deploy
Merge the PR, then trigger the pipeline (Woodpecker → Run → `PIPELINE ==
"deploy-claude-workstation"`), or push a change under the role's paths. CI SSHes
to `10.3.40.60` over the tunnel and applies `common`, `netbird-client`, and this
role. Also run the umbrella `plays/common.yml` against the new host once, so it
picks up `qemu-guest-agent` like every other inventory host.
---
## Step 6 — Egress cutover (staged)
The nftables allowlist ships **disabled**. To enforce:
1. Leave it off for a few days; watch what the box actually talks to.
2. Confirm `claude_ws_egress_allow_fqdns` + `claude_ws_egress_allow_cidrs*`
(home Prometheus/Loki/API) cover it.
3. Set `claude_ws_egress_enforce: true` in host/group vars, redeploy.
4. Watch the `claude-egress-dropped` nft counter (`nft list ruleset`) and Loki
for anything legitimate being dropped; widen the allowlist as needed.
---
## Step 7 — Notes (Syncthing pairing)
The role installs Syncthing for `claude-job` and starts
`syncthing@claude-job`. Pair it once (manual): from the VM's GUI (SSH tunnel to
`127.0.0.1:8384`) add your existing vault's device ID and accept the shared
`Notes` folder pointed at `{{ claude_ws_notes_dir }}`. The distill timer runs
daily at 04:30 and only ever **creates** new atomic notes.
---
## Rotation
- **OAuth token** (expires/can be revoked): re-run `claude setup-token` on your
laptop, update the two vault keys, redeploy. Headless auth failures surface in
Loki (the wrapper logs `result=failure`) — don't let them go quiet.
- **claude-bot PAT**: rotate in Gitea, update `claude_ws_gitea_*`, redeploy.
- **SA token**: `kubectl -n apps delete secret claude-workstation-token` (Flux
recreates it), re-assemble the kubeconfig (Step 3), update vault, redeploy.
## Kill switch
```sh
systemctl disable --now claude-notes-distill.timer claude-egress-refresh.timer
```
Every credential is independently revocable: delete the SA, revoke the PAT,
revoke the OAuth token, or remove the Netbird peer — no shared blast radius.
## Disaster recovery (cattle, not pet)
The VM holds no unique state — the only real state is the Notes replica, which
re-syncs from Syncthing. To rebuild: redo **Step 1** (VM + cloud-init), then
**Step 5** (pipeline). No VMM snapshots needed (they'd bloat the 86%-full
`/volume1`).
@@ -1,68 +0,0 @@
---
# claude-workstation role defaults
#
# An always-on headless host that runs Claude Code for (a) reattachable
# interactive sessions and (b) unattended notes/wiki timers. It deliberately
# holds the LEAST privilege that still lets it do its job: a read-only k8s
# ServiceAccount kubeconfig, claude-bot (non-admin) gitea creds, and a Claude
# subscription OAuth token — NO talosconfig and NO SOPS age key. See README.
# --- Users -----------------------------------------------------------------
# Two Unix users with separate credential stores so an unattended (timer) job
# cannot read the interactive session's OAuth token / kube / gitea creds.
claude_ws_interactive_user: claude # human tmux sessions
claude_ws_job_user: claude-job # unattended systemd-timer jobs
claude_ws_base_dir: /opt/claude-workstation
# --- Toolchain -------------------------------------------------------------
claude_ws_node_major: 22 # matches the ci-claude image
claude_ws_claude_code_pkg: "@anthropic-ai/claude-code"
claude_ws_base_packages:
- git
- tmux
- jq
- curl
- ca-certificates
- ripgrep
- unzip
- nftables
- python3
- python3-venv
# --- Egress allowlist ------------------------------------------------------
# Deny-default OUTPUT is the target end-state, but it can brick a fresh host if
# an allowed destination is missed, so it ships DISABLED and is tightened after
# observing real egress (staged rollout). Flip claude_ws_egress_enforce to true
# once the allowlist below is confirmed complete. See templates/nftables.conf.j2.
claude_ws_egress_enforce: false
# Static destinations the host must always reach. CDN-backed hosts (Anthropic,
# npm, apt) are matched by name at apply time via the fqdn set; home services
# are matched by IP. Tune in host_vars.
claude_ws_egress_allow_fqdns:
- api.anthropic.com
- statsig.anthropic.com
- gitea.renner.uno
- fsn1.your-objectstorage.com # Hetzner Object Storage (SOPS bundle)
- deb.debian.org
- security.debian.org
- deb.nodesource.com
- pkgs.netbird.io
- registry.npmjs.org
claude_ws_egress_allow_cidrs: [] # home Prometheus/Loki/API (v4), per-host
claude_ws_egress_allow_cidrs6: [] # same (v6) — IPv6-first, prefer these
# --- Notes / wiki pipeline -------------------------------------------------
# The Obsidian vault syncs onto this host via Syncthing (matches the user's
# existing multi-device setup). Unattended pipeline jobs must CREATE new atomic
# files rather than rewrite existing ones, to keep Syncthing conflicts rare.
claude_ws_notes_enabled: true
claude_ws_notes_dir: "/home/{{ claude_ws_job_user }}/Notes"
claude_ws_syncthing_gui_bind: "127.0.0.1:8384"
# --- Secrets (populated from the ansible-vault'd group_vars — see README) ---
# claude_ws_oauth_token_interactive : CLAUDE_CODE_OAUTH_TOKEN for the human user
# claude_ws_oauth_token_job : CLAUDE_CODE_OAUTH_TOKEN for timer jobs
# claude_ws_gitea_token : claude-bot PAT
# claude_ws_gitea_git_credentials : https://claude-bot:<pat>@gitea.renner.uno line
# claude_ws_kubeconfig : read-only SA kubeconfig (base64 or literal)
@@ -1,22 +0,0 @@
You are running unattended as a scheduled job on the claude-workstation host.
Your job is a conservative distillation pass over the Obsidian wiki vault.
Scope and rules (follow exactly — you have only Read, Write, Glob, Grep):
1. Look at `~/Notes/raw/` (ignore `~/Notes/raw/_processed/` and `~/Notes/_archive/`
entirely) for source material.
2. For each raw source, check whether `~/Notes/wiki/` already contains a note
grounded in it (Grep the wiki for the source filename / its `sources:` slug).
If a note already exists, SKIP that source — do not rewrite or duplicate it.
3. For raw sources with NO corresponding wiki note yet, create new atomic
concept notes in `~/Notes/wiki/`: one concept per file, self-contained,
tagged, cross-linked with `[[wikilinks]]`, every claim grounded in a
`sources:` entry. Never invent facts not present in the source.
4. CREATE new files only. Never rewrite, move, delete, or truncate an existing
file in `~/Notes/` — appending a missing cross-link to a note you created
THIS run is fine; touching pre-existing notes is not.
5. If there is nothing new to distill, do nothing and stop. Do not manufacture
work.
Preserve disagreement: if sources conflict, surface the conflict in the note
rather than silently picking a winner. Keep each note short and single-purpose.
@@ -1,8 +0,0 @@
# claude-workstation shared tmux config — reattachable long-running sessions.
set -g history-limit 100000
set -g mouse on
set -g base-index 1
setw -g pane-base-index 1
set -g status-right "#H %Y-%m-%d %H:%M"
# Keep sessions alive across client disconnects (the whole point of this host).
set -g destroy-unattached off
@@ -1,10 +0,0 @@
---
- name: Reload systemd
ansible.builtin.systemd:
daemon_reload: true
- name: Reload nftables
ansible.builtin.systemd:
name: nftables.service
state: reloaded
when: claude_ws_egress_enforce | bool
@@ -1,42 +0,0 @@
---
# Deny-default egress allowlist. Ships DISABLED (claude_ws_egress_enforce:
# false) so a fresh host is never bricked by a missed destination; the ruleset,
# the FQDN-resolver, and its refresh timer are installed regardless, and the
# operator flips the toggle to true once egress is confirmed complete against
# real traffic (staged rollout). CDN-backed hosts are matched via nft named
# sets that the refresh unit repopulates from DNS; static home services are
# matched by claude_ws_egress_allow_cidrs.
- name: Deploy the egress-refresh resolver
ansible.builtin.template:
src: claude-egress-refresh.sh.j2
dest: "{{ claude_ws_base_dir }}/claude-egress-refresh.sh"
mode: "0755"
- name: Deploy the egress-refresh service + timer
ansible.builtin.template:
src: "{{ item }}.j2"
dest: "/etc/systemd/system/{{ item }}"
mode: "0644"
loop:
- claude-egress-refresh.service
- claude-egress-refresh.timer
notify: Reload systemd
- name: Deploy the nftables ruleset
ansible.builtin.template:
src: nftables.conf.j2
dest: /etc/nftables.conf
mode: "0644"
validate: "nft -c -f %s"
notify: Reload nftables
- name: Enforce or relax egress lockdown
ansible.builtin.systemd:
name: "{{ item }}"
enabled: "{{ claude_ws_egress_enforce | bool }}"
state: "{{ 'started' if (claude_ws_egress_enforce | bool) else 'stopped' }}"
daemon_reload: true
loop:
- nftables.service
- claude-egress-refresh.timer
@@ -1,183 +0,0 @@
---
# claude-workstation: an always-on headless Claude Code host.
# Debian 12 VM on the off-site nas-2 (Synology VMM). See README for the manual
# bootstrap (VM creation, Netbird route, secret material) that precedes this.
#
# common.yml already applies `common` and `qemu-guest-agent`
# to every inventory host, so this role owns only the
# Claude-specific surface: toolchain, the two credential-isolated users, the
# read-only creds, notes sync, the unattended timers, and egress lockdown.
- name: Assert we are on Debian (role is apt-only)
ansible.builtin.assert:
that:
- ansible_facts['os_family'] == 'Debian'
fail_msg: "claude-workstation targets Debian; got {{ ansible_facts['os_family'] }}."
# --- Base packages ---------------------------------------------------------
- name: Install base packages
ansible.builtin.apt:
name: "{{ claude_ws_base_packages }}"
state: present
update_cache: true
# --- Node.js 22 + Claude Code (matches the ci-claude image toolchain) -------
- name: Add NodeSource signing key
ansible.builtin.get_url:
url: https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key
dest: /usr/share/keyrings/nodesource.gpg
mode: "0644"
force: false
- name: Add NodeSource apt repository
ansible.builtin.apt_repository:
repo: "deb [signed-by=/usr/share/keyrings/nodesource.gpg] https://deb.nodesource.com/node_{{ claude_ws_node_major }}.x nodistro main"
state: present
filename: nodesource
- name: Install Node.js
ansible.builtin.apt:
name: nodejs
state: present
update_cache: true
# NodeSource installs Node under /usr, so global npm modules land in
# /usr/lib/node_modules and `claude` is on PATH for every user/timer. `creates`
# keeps it idempotent (re-run `npm install -g` manually to bump the version).
- name: Install Claude Code globally
ansible.builtin.command:
cmd: "npm install -g {{ claude_ws_claude_code_pkg }}"
creates: /usr/lib/node_modules/@anthropic-ai/claude-code
# --- Credential-isolated users --------------------------------------------
- name: Create Claude users
ansible.builtin.user:
name: "{{ item }}"
shell: /bin/bash
create_home: true
loop:
- "{{ claude_ws_interactive_user }}"
- "{{ claude_ws_job_user }}"
- name: Create the role base directory (scripts, prompts)
ansible.builtin.file:
path: "{{ claude_ws_base_dir }}"
state: directory
mode: "0755"
# Let the interactive user's tmux server survive SSH logout / reboot.
- name: Enable lingering for the interactive user
ansible.builtin.command: "loginctl enable-linger {{ claude_ws_interactive_user }}"
args:
creates: "/var/lib/systemd/linger/{{ claude_ws_interactive_user }}"
- name: Deploy shared tmux config
ansible.builtin.copy:
src: tmux.conf
dest: "/home/{{ item }}/.tmux.conf"
owner: "{{ item }}"
group: "{{ item }}"
mode: "0644"
loop:
- "{{ claude_ws_interactive_user }}"
- "{{ claude_ws_job_user }}"
# --- Secrets: each user gets only what its role needs ----------------------
# The interactive user gets gitea + OAuth + read-only kubeconfig. The job user
# gets its OWN OAuth token + gitea (to open PRs) but NO kubeconfig.
- name: Create ~/.config/claude-bot for the interactive user
ansible.builtin.file:
path: "/home/{{ item }}/.config/claude-bot"
state: directory
owner: "{{ item }}"
group: "{{ item }}"
mode: "0700"
loop:
- "{{ claude_ws_interactive_user }}"
- "{{ claude_ws_job_user }}"
- name: Deploy claude-bot gitea token
ansible.builtin.copy:
content: "{{ claude_ws_gitea_token }}\n"
dest: "/home/{{ item }}/.config/claude-bot/gitea-token"
owner: "{{ item }}"
group: "{{ item }}"
mode: "0600"
loop:
- "{{ claude_ws_interactive_user }}"
- "{{ claude_ws_job_user }}"
no_log: true
- name: Deploy claude-bot git-credentials
ansible.builtin.copy:
content: "{{ claude_ws_gitea_git_credentials }}\n"
dest: "/home/{{ item }}/.config/claude-bot/git-credentials"
owner: "{{ item }}"
group: "{{ item }}"
mode: "0600"
loop:
- "{{ claude_ws_interactive_user }}"
- "{{ claude_ws_job_user }}"
no_log: true
# CLAUDE_CODE_OAUTH_TOKEN is delivered as an EnvironmentFile so both interactive
# shells and timer units can source it without it landing in shell history.
# NEVER run claude with --bare here: bare mode silently ignores this token.
- name: Deploy per-user Claude OAuth environment file
ansible.builtin.copy:
content: "CLAUDE_CODE_OAUTH_TOKEN={{ item.token }}\n"
dest: "/home/{{ item.user }}/.config/claude-code.env"
owner: "{{ item.user }}"
group: "{{ item.user }}"
mode: "0600"
loop:
- { user: "{{ claude_ws_interactive_user }}", token: "{{ claude_ws_oauth_token_interactive }}" }
- { user: "{{ claude_ws_job_user }}", token: "{{ claude_ws_oauth_token_job }}" }
loop_control:
label: "{{ item.user }}"
no_log: true
- name: Create ~/.kube for the interactive user
ansible.builtin.file:
path: "/home/{{ claude_ws_interactive_user }}/.kube"
state: directory
owner: "{{ claude_ws_interactive_user }}"
group: "{{ claude_ws_interactive_user }}"
mode: "0700"
- name: Deploy read-only SA kubeconfig (interactive user only)
ansible.builtin.copy:
content: "{{ claude_ws_kubeconfig }}"
dest: "/home/{{ claude_ws_interactive_user }}/.kube/config"
owner: "{{ claude_ws_interactive_user }}"
group: "{{ claude_ws_interactive_user }}"
mode: "0600"
no_log: true
# This VM's only git identity IS claude-bot (unlike the shared human
# workstation, where a global credential.helper is forbidden), so a per-user
# ~/.gitconfig pointing at each user's own claude-bot store is appropriate.
- name: Configure git identity + credential store for the Claude users
ansible.builtin.copy:
dest: "/home/{{ item }}/.gitconfig"
owner: "{{ item }}"
group: "{{ item }}"
mode: "0644"
content: |
[user]
name = Claude Bot
email = claude-bot@renner.uno
[credential]
helper = store --file=/home/{{ item }}/.config/claude-bot/git-credentials
loop:
- "{{ claude_ws_interactive_user }}"
- "{{ claude_ws_job_user }}"
# --- Notes / wiki pipeline (Syncthing + timers) ----------------------------
- name: Notes pipeline
ansible.builtin.include_tasks: notes.yml
when: claude_ws_notes_enabled | bool
# --- Egress lockdown -------------------------------------------------------
- name: Egress firewall
ansible.builtin.include_tasks: firewall.yml
@@ -1,48 +0,0 @@
---
# Notes/wiki pipeline: the Obsidian vault syncs onto this host via Syncthing
# (matching the user's existing multi-device setup), and a hardened, bounded
# systemd timer runs an unattended `claude -p` distill pass over it as the
# credential-isolated job user. Device pairing is a one-time manual step (see
# README) — this only installs Syncthing and the timer.
- name: Install Syncthing
ansible.builtin.apt:
name: syncthing
state: present
update_cache: true
- name: Enable Syncthing for the job user
ansible.builtin.systemd:
name: "syncthing@{{ claude_ws_job_user }}.service"
enabled: true
state: started
daemon_reload: true
- name: Deploy the notes-distill prompt
ansible.builtin.copy:
src: claude-notes-distill-prompt.md
dest: "{{ claude_ws_base_dir }}/claude-notes-distill-prompt.md"
mode: "0644"
- name: Deploy the notes-distill wrapper
ansible.builtin.template:
src: claude-notes-distill.sh.j2
dest: "{{ claude_ws_base_dir }}/claude-notes-distill.sh"
mode: "0755"
- name: Deploy the notes-distill service + timer
ansible.builtin.template:
src: "{{ item }}.j2"
dest: "/etc/systemd/system/{{ item }}"
mode: "0644"
loop:
- claude-notes-distill.service
- claude-notes-distill.timer
notify: Reload systemd
- name: Enable the notes-distill timer
ansible.builtin.systemd:
name: claude-notes-distill.timer
enabled: true
state: started
daemon_reload: true
@@ -1,10 +0,0 @@
[Unit]
# {{ ansible_managed }}
Description=Refresh claude-workstation egress allowlist from DNS
After=nftables.service network-online.target
Wants=network-online.target
Requires=nftables.service
[Service]
Type=oneshot
ExecStart={{ claude_ws_base_dir }}/claude-egress-refresh.sh
@@ -1,27 +0,0 @@
#!/usr/bin/env bash
# {{ ansible_managed }}
# Resolve the egress FQDN allowlist to current IPs and load them into the
# nftables @allow4/@allow6 sets. CDN IPs rotate, so this runs on a timer.
# No-op unless the claude_egress table is loaded (i.e. enforcement is on).
set -euo pipefail
TABLE="inet claude_egress"
FQDNS=(
{% for fqdn in claude_ws_egress_allow_fqdns %}
"{{ fqdn }}"
{% endfor %}
)
nft list table $TABLE >/dev/null 2>&1 || { echo "claude_egress table not loaded; enforcement off"; exit 0; }
v4=(); v6=()
for host in "${FQDNS[@]}"; do
while read -r ip; do [ -n "$ip" ] && v4+=("$ip"); done < <(getent ahostsv4 "$host" | awk '{print $1}' | sort -u)
while read -r ip; do [ -n "$ip" ] && v6+=("$ip"); done < <(getent ahostsv6 "$host" | awk '{print $1}' | sort -u)
done
nft flush set $TABLE allow4
nft flush set $TABLE allow6
[ ${#v4[@]} -gt 0 ] && nft add element $TABLE allow4 "{ $(IFS=,; echo "${v4[*]}") }"
[ ${#v6[@]} -gt 0 ] && nft add element $TABLE allow6 "{ $(IFS=,; echo "${v6[*]}") }"
echo "claude-egress-refresh: ${#v4[@]} v4 + ${#v6[@]} v6 addresses loaded"
@@ -1,10 +0,0 @@
[Unit]
# {{ ansible_managed }}
Description=Periodically refresh the claude-workstation egress allowlist
[Timer]
OnBootSec=30s
OnUnitActiveSec=15min
[Install]
WantedBy=timers.target
@@ -1,32 +0,0 @@
[Unit]
# {{ ansible_managed }}
Description=Claude Code unattended notes-distill pass
After=network-online.target syncthing@{{ claude_ws_job_user }}.service
Wants=network-online.target
[Service]
Type=oneshot
User={{ claude_ws_job_user }}
Group={{ claude_ws_job_user }}
EnvironmentFile=/home/{{ claude_ws_job_user }}/.config/claude-code.env
ExecStart={{ claude_ws_base_dir }}/claude-notes-distill.sh
# Hard wall-clock kill — the runaway/kill switch for this unattended agent.
TimeoutStartSec=1800
# --- Hardening: an unattended LLM job on internet-influenced input ----------
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ReadWritePaths=/home/{{ claude_ws_job_user }}
# Credential isolation: cannot read the interactive session's tokens/kubeconfig.
InaccessiblePaths=/home/{{ claude_ws_interactive_user }}
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictSUIDSGID=true
RestrictNamespaces=true
LockPersonality=true
MemoryDenyWriteExecute=false
[Install]
WantedBy=multi-user.target
@@ -1,32 +0,0 @@
#!/usr/bin/env bash
# {{ ansible_managed }}
# Unattended notes-distill pass. Prompt via stdin (never shell-substituted),
# tool-allowlisted, no --bare (bare silently drops CLAUDE_CODE_OAUTH_TOKEN).
# Bounded by the unit's TimeoutStartSec, not a CLI flag (--max-turns is absent
# in the installed Claude Code). Emits a terminal state line to the journal on
# EVERY outcome so a Loki alert can catch silent failures (dead-man switch).
set -euo pipefail
export HOME="/home/{{ claude_ws_job_user }}"
NOTES="{{ claude_ws_notes_dir }}"
PROMPT="{{ claude_ws_base_dir }}/claude-notes-distill-prompt.md"
log() { logger -t claude-notes-distill -- "$*"; echo "claude-notes-distill: $*"; }
if [ ! -d "$NOTES/raw" ]; then
log "result=skipped reason=no-vault path=$NOTES"
exit 0
fi
log "result=start"
if claude -p \
--add-dir "$NOTES" \
--allowedTools "Read,Write,Glob,Grep" \
--model sonnet \
< "$PROMPT"; then
log "result=success"
else
rc=$?
log "result=failure rc=$rc"
exit "$rc"
fi
@@ -1,11 +0,0 @@
[Unit]
# {{ ansible_managed }}
Description=Run the Claude Code notes-distill pass daily
[Timer]
OnCalendar=*-*-* 04:30:00
Persistent=true
RandomizedDelaySec=900
[Install]
WantedBy=timers.target
@@ -1,55 +0,0 @@
#!/usr/sbin/nft -f
# {{ ansible_managed }}
# Deny-default egress for the claude-workstation. CDN-backed destinations are
# matched via the @allow4/@allow6 named sets, repopulated from DNS by
# claude-egress-refresh. Static home services are matched by the @static* sets.
# Only loaded when claude_ws_egress_enforce is true.
flush ruleset
table inet claude_egress {
set allow4 { type ipv4_addr; flags interval; }
set allow6 { type ipv6_addr; flags interval; }
set static4 {
type ipv4_addr
flags interval
{% if (claude_ws_egress_allow_cidrs | default([])) | length > 0 %}
elements = { {{ claude_ws_egress_allow_cidrs | join(', ') }} }
{% endif %}
}
set static6 {
type ipv6_addr
flags interval
{% if (claude_ws_egress_allow_cidrs6 | default([])) | length > 0 %}
elements = { {{ claude_ws_egress_allow_cidrs6 | join(', ') }} }
{% endif %}
}
chain output {
type filter hook output priority 0; policy drop;
oif "lo" accept
ct state established,related accept
# DNS — required to resolve the allowlist itself.
udp dport 53 accept
tcp dport 53 accept
# Diagnostics + DHCP lease renewal on the bridged interface.
ip protocol icmp accept
ip6 nexthdr icmpv6 accept
udp dport { 67, 68 } accept
# NetBird / WireGuard control + data plane (initial handshakes).
udp dport { 3478, 51820 } accept
# Allowed destinations.
ip daddr @allow4 accept
ip6 daddr @allow6 accept
ip daddr @static4 accept
ip6 daddr @static6 accept
# Everything else is dropped and counted for inspection.
counter comment "claude-egress-dropped"
}
}
+2 -3
View File
@@ -140,9 +140,8 @@
Handing the address to keepalived while networkd also assigns it means
it would NOT be withdrawn when check_ntp fails -- the host would keep
serving time on an address VRRP believes it has released.
Re-apply the edge-vlan role first, whose template no longer carries the
[Address] block:
ansible-playbook -i inventory/pve_hosts.yml plays/install-edge-host.yml
Re-apply the edge-vlan role first (site.yml runs it before this play),
whose template no longer carries the [Address] block.
success_msg: >-
No static networkd assignment of {{ keepalived_ntp_vip6 }}; safe for
keepalived to own it.
@@ -11,6 +11,8 @@ netbird_management_url: "https://netbird.renner.uno"
# Debian/Ubuntu follow the apt repository instead.
netbird_client_version: "0.75.0"
netbird_release_url: "https://github.com/netbirdio/netbird/releases/download/v{{ netbird_client_version }}"
# On the controller; the archive is copied from there to the hosts.
netbird_download_dir: "{{ lookup('ansible.builtin.env', 'HOME') }}/.cache/ansible/netbird"
# Leave the host's DNS configuration alone (hosts with a pinned resolv.conf).
netbird_disable_dns: false
+18 -5
View File
@@ -33,22 +33,35 @@
update_cache: true
# Arch has no netbird package in its repositories; the release binary is
# verified against the release's checksums file.
# verified against the release's checksums file. It is downloaded on the
# controller and copied over: github.com has no IPv6 address, and proxy-1 is
# IPv6-only.
- name: Install NetBird from the release tarball
when: ansible_facts['os_family'] == 'Archlinux'
block:
- name: Download the NetBird release
- name: Create the controller download directory
ansible.builtin.file:
path: "{{ netbird_download_dir }}"
state: directory
mode: "0755"
delegate_to: localhost
become: false
run_once: true
- name: Download the NetBird release on the controller
ansible.builtin.get_url:
url: "{{ netbird_release_url }}/netbird_{{ netbird_client_version }}_linux_amd64.tar.gz"
dest: "/tmp/netbird_{{ netbird_client_version }}_linux_amd64.tar.gz"
dest: "{{ netbird_download_dir }}/netbird_{{ netbird_client_version }}_linux_amd64.tar.gz"
checksum: "sha256:{{ netbird_release_url }}/netbird_{{ netbird_client_version }}_checksums.txt"
mode: "0644"
delegate_to: localhost
become: false
run_once: true
- name: Install the NetBird binary
ansible.builtin.unarchive:
src: "/tmp/netbird_{{ netbird_client_version }}_linux_amd64.tar.gz"
src: "{{ netbird_download_dir }}/netbird_{{ netbird_client_version }}_linux_amd64.tar.gz"
dest: /usr/local/bin
remote_src: true
include:
- netbird
owner: root
+2 -12
View File
@@ -136,23 +136,13 @@
- /opt/netbird/proxy
- /opt/netbird/proxy-state
# The management store has no cluster-side backup. Pulling one put a root
# key on this host that the cluster held, and this server's own state is
# its own business to protect rather than the cluster's. Dropping the two
# tasks that placed them would only stop ansible managing the key and the
# script — the pair below revokes them instead, and can go once the role
# has run against every host that ever carried them.
# The management store has no cluster-side backup; the cluster's backup
# key is gone with the managed authorized_keys (ssh-ca role).
- name: Remove the backup dump script
ansible.builtin.file:
path: /usr/local/bin/netbird-backup-dump
state: absent
- name: Revoke the cluster backup key
ansible.posix.authorized_key:
user: root
key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFeW/L9bHMTSa5/Qb0ClppCf4Z0hL39JwzI6tBmKd2FG netbird-backup@homelab-cluster"
state: absent
- name: Wait for management container to be ready
ansible.builtin.pause:
seconds: 10
@@ -1,20 +0,0 @@
---
# Node Exporter defaults
node_exporter_version: "1.8.2"
node_exporter_port: 9100
node_exporter_user: node-exporter
node_exporter_group: node-exporter
# Installation paths
node_exporter_install_dir: /opt/node-exporter
node_exporter_binary: "{{ node_exporter_install_dir }}/node_exporter"
# Collectors to enable (defaults are usually sufficient)
node_exporter_enabled_collectors:
- systemd
- processes
- timex
# Collectors to disable
node_exporter_disabled_collectors: []
@@ -1,62 +0,0 @@
#!/bin/bash
# Generate Prometheus metrics from chronyc tracking output
# Runs as a cron job and outputs to textfile collector directory
OUTPUT_FILE="${1:-/var/lib/node-exporter/chrony.prom}"
# Get chrony tracking info
tracking=$(chronyc tracking 2>/dev/null)
if [ -z "$tracking" ]; then
echo "chrony_up 0" > "$OUTPUT_FILE.tmp"
mv "$OUTPUT_FILE.tmp" "$OUTPUT_FILE"
exit 0
fi
# Parse values
stratum=$(echo "$tracking" | grep "^Stratum" | awk '{print $3}')
system_time=$(echo "$tracking" | grep "^System time" | awk '{print $4}')
root_delay=$(echo "$tracking" | grep "^Root delay" | awk '{print $4}')
root_dispersion=$(echo "$tracking" | grep "^Root dispersion" | awk '{print $4}')
last_offset=$(echo "$tracking" | grep "^Last offset" | awk '{print $4}')
rms_offset=$(echo "$tracking" | grep "^RMS offset" | awk '{print $4}')
frequency=$(echo "$tracking" | grep "^Frequency" | awk '{print $3}')
ref_id=$(echo "$tracking" | grep "^Reference ID" | awk '{print $4}' | tr -d '()')
# Handle negative offsets (remove leading + or keep -)
system_time=$(echo "$system_time" | sed 's/^+//')
last_offset=$(echo "$last_offset" | sed 's/^+//')
rms_offset=$(echo "$rms_offset" | sed 's/^+//')
# Output Prometheus metrics
cat > "$OUTPUT_FILE.tmp" << EOF
# HELP chrony_stratum Current stratum of the NTP server
# TYPE chrony_stratum gauge
chrony_stratum ${stratum:-0}
# HELP chrony_system_time_offset_seconds System time offset from NTP time
# TYPE chrony_system_time_offset_seconds gauge
chrony_system_time_offset_seconds ${system_time:-0}
# HELP chrony_root_delay_seconds Total round-trip delay to the primary reference clock
# TYPE chrony_root_delay_seconds gauge
chrony_root_delay_seconds ${root_delay:-0}
# HELP chrony_root_dispersion_seconds Total dispersion to the primary reference clock
# TYPE chrony_root_dispersion_seconds gauge
chrony_root_dispersion_seconds ${root_dispersion:-0}
# HELP chrony_last_offset_seconds Estimated offset of the last update
# TYPE chrony_last_offset_seconds gauge
chrony_last_offset_seconds ${last_offset:-0}
# HELP chrony_rms_offset_seconds RMS offset
# TYPE chrony_rms_offset_seconds gauge
chrony_rms_offset_seconds ${rms_offset:-0}
# HELP chrony_frequency_ppm Frequency offset in parts per million
# TYPE chrony_frequency_ppm gauge
chrony_frequency_ppm ${frequency:-0}
# HELP chrony_up Whether chrony is running and synchronized
# TYPE chrony_up gauge
chrony_up 1
# HELP chrony_reference_info Reference clock information
# TYPE chrony_reference_info gauge
chrony_reference_info{reference="${ref_id:-unknown}"} 1
EOF
mv "$OUTPUT_FILE.tmp" "$OUTPUT_FILE"
@@ -1,9 +0,0 @@
---
- name: Reload systemd
systemd:
daemon_reload: yes
- name: Restart node-exporter
systemd:
name: node-exporter
state: restarted
@@ -1,88 +0,0 @@
---
# Install and configure Node Exporter
- name: Create node-exporter system group
group:
name: "{{ node_exporter_group }}"
system: yes
state: present
- name: Create node-exporter system user
user:
name: "{{ node_exporter_user }}"
group: "{{ node_exporter_group }}"
system: yes
shell: /usr/sbin/nologin
home: "{{ node_exporter_install_dir }}"
createhome: no
- name: Create installation directory
file:
path: "{{ node_exporter_install_dir }}"
state: directory
owner: "{{ node_exporter_user }}"
group: "{{ node_exporter_group }}"
mode: '0755'
- name: Check if node_exporter is already installed
stat:
path: "{{ node_exporter_binary }}"
register: node_exporter_installed
- name: Get installed version
command: "{{ node_exporter_binary }} --version"
register: installed_version
when: node_exporter_installed.stat.exists
changed_when: false
failed_when: false
- name: Download node_exporter
get_url:
url: "https://github.com/prometheus/node_exporter/releases/download/v{{ node_exporter_version }}/node_exporter-{{ node_exporter_version }}.linux-amd64.tar.gz"
dest: /tmp/node_exporter.tar.gz
mode: '0644'
when: not node_exporter_installed.stat.exists or (installed_version.stdout is defined and node_exporter_version not in installed_version.stdout)
- name: Extract node_exporter
unarchive:
src: /tmp/node_exporter.tar.gz
dest: /tmp
remote_src: yes
when: not node_exporter_installed.stat.exists or (installed_version.stdout is defined and node_exporter_version not in installed_version.stdout)
- name: Install node_exporter binary
copy:
src: "/tmp/node_exporter-{{ node_exporter_version }}.linux-amd64/node_exporter"
dest: "{{ node_exporter_binary }}"
owner: "{{ node_exporter_user }}"
group: "{{ node_exporter_group }}"
mode: '0755'
remote_src: yes
when: not node_exporter_installed.stat.exists or (installed_version.stdout is defined and node_exporter_version not in installed_version.stdout)
notify: Restart node-exporter
- name: Create systemd service
template:
src: node_exporter.service.j2
dest: /etc/systemd/system/node-exporter.service
owner: root
group: root
mode: '0644'
notify:
- Reload systemd
- Restart node-exporter
- name: Enable and start node-exporter service
systemd:
name: node-exporter
enabled: yes
state: started
daemon_reload: yes
- name: Clean up downloaded files
file:
path: "{{ item }}"
state: absent
loop:
- /tmp/node_exporter.tar.gz
- "/tmp/node_exporter-{{ node_exporter_version }}.linux-amd64"
@@ -1,38 +0,0 @@
[Unit]
Description=Prometheus Node Exporter
Documentation=https://github.com/prometheus/node_exporter
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User={{ node_exporter_user }}
Group={{ node_exporter_group }}
ExecStart={{ node_exporter_binary }} \
--web.listen-address=:{{ node_exporter_port }} \
{% for collector in node_exporter_enabled_collectors %}
--collector.{{ collector }} \
{% endfor %}
{% for collector in node_exporter_disabled_collectors %}
--no-collector.{{ collector }} \
{% endfor %}
{% if node_exporter_textfile_dir is defined %}
--collector.textfile \
--collector.textfile.directory={{ node_exporter_textfile_dir }} \
{% endif %}
--collector.filesystem.mount-points-exclude='^/(sys|proc|dev|host|etc)($$|/)'
Restart=on-failure
RestartSec=10s
# Security hardening
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
ReadOnlyPaths=/
{% if node_exporter_textfile_dir is defined %}
ReadWritePaths={{ node_exporter_textfile_dir }}
{% endif %}
[Install]
WantedBy=multi-user.target
+20 -10
View File
@@ -33,12 +33,15 @@
group: root
mode: "0644"
# systemd-journal: read every journal without sudo (see sudoers-claude.j2).
- name: Create the Claude account
ansible.builtin.user:
name: "{{ ssh_ca_claude_user }}"
shell: /bin/bash
create_home: true
password: "!"
groups: systemd-journal
append: true
- name: Look up the owner admin logins
ansible.builtin.getent:
@@ -125,29 +128,36 @@
- "'trustedusercakeys /etc/ssh/trusted_user_ca_keys' in ssh_ca_effective_lines"
- "'authorizedprincipalsfile /etc/ssh/auth_principals/%u' in ssh_ca_effective_lines"
- "'revokedkeys /etc/ssh/revoked_keys' in ssh_ca_effective_lines"
- "'passwordauthentication no' in ssh_ca_effective_lines"
- "'kbdinteractiveauthentication no' in ssh_ca_effective_lines"
fail_msg: "sshd -T does not show the SSH CA settings; another config line takes precedence"
# The unit is sshd.service on Arch and RedHat, ssh.service on Debian/Ubuntu.
# Ubuntu also lists sshd.service as a not-found alias, so the running one is
# picked. sshd -T only parses the files, so the running daemon is reloaded on
# every run; a reload keeps open sessions.
- name: Read the service list
ansible.builtin.service_facts:
# The unit is sshd.service on Arch and RedHat, ssh.service on Debian/Ubuntu,
# where sshd.service is an alias that resolves to it. sshd -T only parses the
# files, so the running daemon is reloaded on every run; a reload keeps open
# sessions. None runs when sshd is socket activated and stopped, or per
# connection (ssh-mesh-only role); the next start reads the config.
# systemctl directly: service_facts took 3-12 s per host.
- name: Find the running sshd unit
ansible.builtin.command: systemctl list-units --state=running --plain --no-legend --type=service sshd.service ssh.service
changed_when: false
register: ssh_ca_sshd_running
- name: Reload the running sshd unit
vars:
ssh_ca_sshd_units: "{{ ansible_facts.services | dict2items | selectattr('key', 'in', ['sshd.service', 'ssh.service']) | selectattr('value.state', 'equalto', 'running') | map(attribute='key') | list }}"
ssh_ca_sshd_units: "{{ ssh_ca_sshd_running.stdout_lines | map('split') | map('first') | unique | list }}"
block:
- name: Check exactly one sshd unit is running
- name: Check at most one sshd unit is running
ansible.builtin.assert:
that:
- ssh_ca_sshd_units | length == 1
fail_msg: "Expected one running sshd.service or ssh.service, found {{ ssh_ca_sshd_units }}"
- ssh_ca_sshd_units | length <= 1
fail_msg: "Expected at most one running sshd.service or ssh.service, found {{ ssh_ca_sshd_units }}"
- name: Reload sshd so the running daemon uses the CA config
ansible.builtin.systemd:
name: "{{ ssh_ca_sshd_units | first }}"
state: reloaded
when: ssh_ca_sshd_units | length == 1
- name: Report the admin logins' authorized keys
ansible.builtin.include_tasks: report.yml
@@ -2,3 +2,5 @@
TrustedUserCAKeys /etc/ssh/trusted_user_ca_keys
AuthorizedPrincipalsFile /etc/ssh/auth_principals/%u
RevokedKeys /etc/ssh/revoked_keys
PasswordAuthentication no
KbdInteractiveAuthentication no
@@ -1,6 +1,12 @@
# {{ ansible_managed }}
# NOEXEC: journalctl/systemctl open a pager, which must not spawn a shell.
{{ ssh_ca_claude_user }} ALL=(root) NOPASSWD:NOEXEC: /usr/bin/journalctl, /usr/bin/journalctl *, /usr/bin/systemctl status, /usr/bin/systemctl status *, /usr/bin/systemctl list-units, /usr/bin/systemctl list-units *, /usr/bin/ss -tulpn, /usr/sbin/sshd -T
# Logs and unit status come from the systemd-journal group, not sudo:
# journalctl as root would also allow --vacuum-*/--rotate.
# Commands are listed for both the Arch and the Debian/Ubuntu path.
# nft only with fixed arguments: it joins argv into one command line, so a
# wildcard would admit "; flush ruleset". No docker start: -a/-i attach to a
# container's stdin; docker restart also starts stopped containers.
{{ ssh_ca_claude_user }} ALL=(root) NOPASSWD:NOEXEC: /usr/bin/ss -tulpn, /usr/sbin/sshd -T, /usr/bin/nft list ruleset, /usr/sbin/nft list ruleset, /usr/bin/netbird status, /usr/bin/netbird status *, /usr/local/bin/netbird status, /usr/local/bin/netbird status *
{{ ssh_ca_claude_user }} ALL=(root) NOPASSWD:NOEXEC: /usr/bin/docker ps, /usr/bin/docker ps *, /usr/bin/docker logs *, /usr/bin/docker stats --no-stream, /usr/bin/docker restart *
{% for unit in ssh_ca_claude_sudo_units %}
{{ ssh_ca_claude_user }} ALL=(root) NOPASSWD:NOEXEC: /usr/bin/systemctl restart {{ unit }}, /usr/bin/systemctl reload {{ unit }}
{% endfor %}
@@ -0,0 +1,9 @@
---
# The only interface sshd accepts connections on.
ssh_mesh_interface: wt0
# Minutes until a switchover reverts to the distro listener, unless a new
# login through the mesh-only listener confirms it first.
ssh_mesh_revert_minutes: 5
ssh_mesh_udev_rule: /etc/udev/rules.d/90-ssh-mesh-only.rules
+113
View File
@@ -0,0 +1,113 @@
---
- name: Load the distro's sshd unit names
ansible.builtin.include_vars: "{{ ansible_facts['os_family'] }}.yml"
- name: Look for the mesh interface
ansible.builtin.stat:
path: "/sys/class/net/{{ ssh_mesh_interface }}"
register: ssh_mesh_iface
- name: Check the mesh interface exists
ansible.builtin.assert:
that:
- ssh_mesh_iface.stat.exists
fail_msg: "No {{ ssh_mesh_interface }}: sshd would be unreachable. Install the NetBird client first."
- name: Create the socket unit directory
ansible.builtin.file:
path: "{{ ssh_mesh_socket_file | dirname }}"
state: directory
owner: root
group: root
mode: "0755"
- name: Install the mesh-only socket
ansible.builtin.template:
src: "{{ ssh_mesh_socket_template }}"
dest: "{{ ssh_mesh_socket_file }}"
owner: root
group: root
mode: "0644"
- name: Bind the sshd service to the socket
when: ssh_mesh_service_dropin | length > 0
block:
- name: Create the service drop-in directory
ansible.builtin.file:
path: "{{ ssh_mesh_service_dropin | dirname }}"
state: directory
owner: root
group: root
mode: "0755"
- name: Install the service drop-in
ansible.builtin.template:
src: ssh.service-dropin.conf.j2
dest: "{{ ssh_mesh_service_dropin }}"
owner: root
group: root
mode: "0644"
- name: Install the udev rule that starts the socket with the interface
ansible.builtin.template:
src: udev.rules.j2
dest: "{{ ssh_mesh_udev_rule }}"
owner: root
group: root
mode: "0644"
register: ssh_mesh_udev
- name: Reload the udev rules
ansible.builtin.command: udevadm control --reload
when: ssh_mesh_udev is changed
changed_when: true
- name: Install the switch and revert scripts
ansible.builtin.template:
src: "{{ item }}.j2"
dest: "/usr/local/sbin/{{ item }}"
owner: root
group: root
mode: "0755"
loop:
- ssh-mesh-switch
- ssh-mesh-revert
- name: Reload systemd units
ansible.builtin.systemd:
daemon_reload: true
- name: Read the sshd listeners
ansible.builtin.command: ss -Htln 'sport = :22'
changed_when: false
register: ssh_mesh_listeners
# Each run that finds a listener outside the interface switches over. An
# include, because meta reset_connection ignores when: and a reconnect costs a
# YubiKey PIN from a workstation.
- name: Switch sshd to the mesh-only socket
ansible.builtin.include_tasks: switch.yml
when: >-
ssh_mesh_listeners.stdout_lines | length == 0 or
ssh_mesh_listeners.stdout_lines | reject('search', '%' ~ ssh_mesh_interface ~ ':22\s') | list | length > 0
- name: Check sshd listens on the mesh interface only
ansible.builtin.assert:
that:
- ssh_mesh_listeners.stdout_lines | length > 0
- ssh_mesh_listeners.stdout_lines | reject('search', '%' ~ ssh_mesh_interface ~ ':22\s') | list | length == 0
fail_msg: "Listeners outside {{ ssh_mesh_interface }}: {{ ssh_mesh_listeners.stdout_lines }}"
- name: Cancel the revert timer
ansible.builtin.command: systemctl stop ssh-mesh-revert.timer
changed_when: false
failed_when: false
- name: Check the public name refuses SSH
ansible.builtin.wait_for:
host: "{{ inventory_hostname }}"
port: 22
state: stopped
timeout: 15
delegate_to: localhost
become: false
@@ -0,0 +1,27 @@
---
# The revert timer runs in its own unit, so a switch that locks the play out is
# undone after ssh_mesh_revert_minutes. main.yml cancels it after checking the
# listeners read through the new one.
- name: Clear a revert timer left by an earlier run
ansible.builtin.command: systemctl stop ssh-mesh-revert.timer ssh-mesh-revert.service
changed_when: false
failed_when: false
- name: Arm the revert timer
ansible.builtin.command: >-
systemd-run --unit=ssh-mesh-revert --collect
--on-active={{ ssh_mesh_revert_minutes }}min
/usr/local/sbin/ssh-mesh-revert
changed_when: true
- name: Switch the listener
ansible.builtin.command: systemd-run --unit=ssh-mesh-switch --collect --wait /usr/local/sbin/ssh-mesh-switch
changed_when: true
- name: Reconnect through the new listener
ansible.builtin.meta: reset_connection
- name: Read the sshd listeners after reconnecting
ansible.builtin.command: ss -Htln 'sport = :22'
changed_when: false
register: ssh_mesh_listeners
@@ -0,0 +1,9 @@
#!/bin/sh
# {{ ansible_managed }}
# Restores the distro listener on all addresses. Armed as a timer by every
# switchover; the play cancels it after a login through the new listener.
systemctl stop {{ ssh_mesh_service }} {{ ssh_mesh_socket }}
rm -f {{ ssh_mesh_udev_rule }} {{ ssh_mesh_socket_file }}{{ (' ' ~ ssh_mesh_service_dropin) if ssh_mesh_service_dropin else '' }}
udevadm control --reload
systemctl daemon-reload
systemctl enable --now {{ ssh_mesh_distro_listener }}
@@ -0,0 +1,11 @@
#!/bin/sh
# {{ ansible_managed }}
# Replaces the distro listener with the {{ ssh_mesh_interface }}-only socket.
# Open sessions survive: they run in logind scopes and the sshd units use
# KillMode=process. Any failure restores the distro listener at once.
set -eu
trap '/usr/local/sbin/ssh-mesh-revert' EXIT
systemctl disable {{ ssh_mesh_disable_units }}
systemctl stop {{ ssh_mesh_service }} {{ ssh_mesh_socket }}
systemctl start {{ ssh_mesh_socket }}
trap - EXIT
@@ -0,0 +1,7 @@
# {{ ansible_managed }}
# sshd holds the socket's listener; it stops with the socket, so it never keeps
# a listener on a removed {{ ssh_mesh_interface }}. Replaces the Requires= that
# enabling {{ ssh_mesh_socket }} would add.
[Unit]
BindsTo={{ ssh_mesh_socket }}
After={{ ssh_mesh_socket }}
@@ -0,0 +1,8 @@
# {{ ansible_managed }}
# Listen on {{ ssh_mesh_interface }} only. The unit is disabled; the udev rule
# starts it whenever {{ ssh_mesh_interface }} appears.
[Socket]
ListenStream=
ListenStream=22
BindToDevice={{ ssh_mesh_interface }}
BindIPv6Only=both
@@ -0,0 +1,12 @@
# {{ ansible_managed }}
# Started by the udev rule whenever {{ ssh_mesh_interface }} appears; BindToDevice
# stops it when the interface goes away. No [Install]: waiting for the device
# from sockets.target would hold up boot until NetBird is up.
[Unit]
Description=OpenSSH on {{ ssh_mesh_interface }} only
[Socket]
ListenStream=22
BindToDevice={{ ssh_mesh_interface }}
BindIPv6Only=both
Accept=yes
@@ -0,0 +1,3 @@
# {{ ansible_managed }}
# NetBird deletes and recreates {{ ssh_mesh_interface }} on every engine restart.
SUBSYSTEM=="net", KERNEL=="{{ ssh_mesh_interface }}", ENV{SYSTEMD_WANTS}+="{{ ssh_mesh_socket }}"
@@ -0,0 +1,10 @@
---
# No packaged socket; sshd.socket (Accept=yes) pairs with the packaged
# per-connection sshd@.service, since sshd -D cannot take a listening fd.
ssh_mesh_socket: sshd.socket
ssh_mesh_service: sshd.service
ssh_mesh_distro_listener: sshd.service
ssh_mesh_disable_units: sshd.service
ssh_mesh_socket_file: /etc/systemd/system/sshd.socket
ssh_mesh_socket_template: sshd.socket.j2
ssh_mesh_service_dropin: ""
@@ -0,0 +1,10 @@
---
# Ubuntu's ssh.socket (Accept=no) hands its listener to ssh.service. The
# drop-in is named like sshd-socket-generator's output so it masks it.
ssh_mesh_socket: ssh.socket
ssh_mesh_service: ssh.service
ssh_mesh_distro_listener: ssh.socket
ssh_mesh_disable_units: ssh.service ssh.socket
ssh_mesh_socket_file: /etc/systemd/system/ssh.socket.d/addresses.conf
ssh_mesh_socket_template: ssh.socket-addresses.conf.j2
ssh_mesh_service_dropin: /etc/systemd/system/ssh.service.d/ssh-mesh-only.conf
+41
View File
@@ -0,0 +1,41 @@
---
# The whole defined state of every Ansible-managed host, in dependency order.
# Run from ansible/ on a mesh-connected workstation; the inventory comes from
# ansible.cfg:
#
# ansible-playbook site.yml
#
# One-off actions are not state and stay separate: plays/update-os.yml,
# plays/update-dist-upgrade.yml, plays/rotate-netbird-secrets.yml,
# plays/configure-shelly-ntp.yml (devices outside the inventory).
- name: Base system
ansible.builtin.import_playbook: plays/common.yml
- name: VM time sync
ansible.builtin.import_playbook: plays/configure-vm-time-sync.yml
- name: SSH user CA
ansible.builtin.import_playbook: plays/configure-ssh-ca.yml
- name: NetBird host clients
ansible.builtin.import_playbook: plays/install-netbird-client.yml
- name: sshd on the mesh interface only
ansible.builtin.import_playbook: plays/configure-ssh-mesh-only.yml
- name: NetBird server
ansible.builtin.import_playbook: plays/install-netbird-server.yml
- name: Uptime Kuma
ansible.builtin.import_playbook: plays/install-uptime-kuma.yml
- name: NetBird proxies
ansible.builtin.import_playbook: plays/install-netbird-proxy.yml
- name: Edge host base
ansible.builtin.import_playbook: plays/install-edge-host.yml
- name: Z-Wave gateway
ansible.builtin.import_playbook: plays/install-zwave-gateway.yml
- name: Heating CAN
ansible.builtin.import_playbook: plays/install-heating-can.yml
- name: NTP service
ansible.builtin.import_playbook: plays/install-ntp-ha.yml
- name: Close the SSH connections
ansible.builtin.import_playbook: plays/close-connections.yml
+10 -9
View File
@@ -12,7 +12,10 @@ The image is the official archiso **releng** profile plus:
- `edge-install` in the live environment — partitions the NVMe
(ESP + ext4 root, systemd-boot), pacstraps the base system incl.
docker/python, configures DHCP via systemd-networkd, enables sshd and
drops the common homelab SSH keys for root (key-only, no root password)
sets up root login (key-only, no root password) with owner certificates
from the OpenBao SSH CA, fetched at build time from the `ssh-ca` role's
URL, and the break-glass key (YubiKey NEO, PIV) as fallback:
`ssh -I /usr/lib/opensc-pkcs11.so root@<host>`
- the extra disk tools the installer needs
Division of labour: the ISO produces an **ansible-ready base system**;
@@ -30,7 +33,8 @@ sudo ./build.sh # ISO lands in ./out/
The profile is assembled at build time from `/usr/share/archiso/configs/releng`
plus the overlay in `airootfs/` — the repo carries only the delta, so
archiso updates flow in automatically.
archiso updates flow in automatically. The build needs openbao.renner.uno
for the CA key; rebuild the ISO after a CA rotation.
## Install
@@ -38,8 +42,7 @@ archiso updates flow in automatically.
2. Boot the target from it (UEFI, Secure Boot off; one-time F12/setup
visit if the box doesn't already boot USB first)
3. Either on the console (auto-login as root) or headless —
`ssh root@<dhcp-ip>` works out of the box, the live env carries the
homelab keys — run `edge-install`. It auto-detects the internal disk
`ssh root@<dhcp-ip>` with an owner certificate — run `edge-install`. It auto-detects the internal disk
(NVMe, else eMMC — covers the Wyse 3040 — else first non-USB disk),
shows the target and requires typing the hostname before wiping
anything. Override with `edge-install /dev/sdY`.
@@ -49,10 +52,8 @@ archiso updates flow in automatically.
`ansible/inventory/pve_hosts.yml` and, from the workstation:
```
cd ansible
ansible-playbook -i inventory/pve_hosts.yml plays/install-edge-host.yml
ansible-playbook site.yml
```
One run over the whole group, no `--limit`: the play carries
`certificates` and `common` itself, so there is no separate `common.yml`
step. Running it converges every edge host, which is the point — a run
that touches only the new box tells you nothing about the others.
One run, no `--limit`. It converges every host, which is the point — a
run that touches only the new box tells you nothing about the others.
5. Continue with the cutover checklist in `ansible/docs/edge-host.md`.
@@ -1,4 +1 @@
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPzFNBP3bq2m/6JqDqod9w+FP/uxXxHMolqhq3Ojflr1 krenner@fortress
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGfe3HJqFR+jIGrboweseAO7XjIVE24sje/b1HybgU9C ansible-runner@ci-new
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICyNnX1I4W9jyDvCJ1DJauHSmbo1ZXSyBxVhSAwMWgen woodpecker-ci@homelab
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB1OFjh6Q5xeioI6Cwfuv8Xka27veomYy9tKBbpMo8mcrsB4yIN4FX8HovBLtogetBhwBUefBar8nhA9GbRh6HQ= krenner@krenner-XPS-13-9310
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKvSl9LUP8u3S2Sq5YTWpW0Jxbue8cJhD7/LMFQbEsa3xo145HVR7pBzRxW71MUy+jgrgmpSlDhIb606Hswhvoo= breakglass-neo-6532348
@@ -1,4 +1 @@
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPzFNBP3bq2m/6JqDqod9w+FP/uxXxHMolqhq3Ojflr1 krenner@fortress
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGfe3HJqFR+jIGrboweseAO7XjIVE24sje/b1HybgU9C ansible-runner@ci-new
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICyNnX1I4W9jyDvCJ1DJauHSmbo1ZXSyBxVhSAwMWgen woodpecker-ci@homelab
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB1OFjh6Q5xeioI6Cwfuv8Xka27veomYy9tKBbpMo8mcrsB4yIN4FX8HovBLtogetBhwBUefBar8nhA9GbRh6HQ= krenner@krenner-XPS-13-9310
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKvSl9LUP8u3S2Sq5YTWpW0Jxbue8cJhD7/LMFQbEsa3xo145HVR7pBzRxW71MUy+jgrgmpSlDhIb606Hswhvoo= breakglass-neo-6532348
@@ -81,6 +81,17 @@ ln -sf /run/systemd/resolve/stub-resolv.conf /mnt/etc/resolv.conf
install -d -m700 /mnt/root/.ssh
install -m600 /root/edge-authorized_keys /mnt/root/.ssh/authorized_keys
# SSH user CA trust as baked into this ISO (owner certificates for root).
install -m644 -t /mnt/etc/ssh /etc/ssh/trusted_user_ca_keys /etc/ssh/revoked_keys
install -d -m755 /mnt/etc/ssh/auth_principals /mnt/etc/ssh/sshd_config.d
install -m644 /etc/ssh/auth_principals/root /mnt/etc/ssh/auth_principals/root
install -m644 /etc/ssh/sshd_config.d/50-ssh-ca.conf /mnt/etc/ssh/sshd_config.d/
# sshd -t/-T need host keys; sshdgenkeys would create them on first boot.
ssh-keygen -A -f /mnt
arch-chroot /mnt sshd -t
arch-chroot /mnt sshd -T | grep -qix 'trustedusercakeys /etc/ssh/trusted_user_ca_keys' \
|| { echo "installed sshd does not use the SSH CA"; exit 1; }
arch-chroot /mnt systemctl enable \
systemd-networkd systemd-resolved systemd-timesyncd sshd docker
@@ -104,8 +115,6 @@ echo
echo "Then, from the workstation:"
echo " 1. confirm 'ssh root@${HOST}' works (it must be reachable by name)"
echo " 2. add ${HOST} to [edge_host] in ansible/inventory/pve_hosts.yml"
echo " 3. cd ansible && ansible-playbook -i inventory/pve_hosts.yml \\"
echo " plays/install-edge-host.yml"
echo " 3. cd ansible && ansible-playbook site.yml"
echo
echo "One run, whole group, no --limit: the play carries certificates and"
echo "common itself, so there is no separate common.yml step."
echo "One run, all hosts, no --limit."
+17
View File
@@ -10,6 +10,23 @@ profile=/tmp/profile
cp -r /usr/share/archiso/configs/releng/. "$profile"
cp -r /src/airootfs/. "$profile/airootfs/"
# SSH user CA trust from the ssh-ca role (mounted at /ssh-ca): the live system
# and the installed one accept owner certificates for root, like the managed
# hosts. The CA public key is fetched from OpenBao at build time.
ca_url=$(sed -n 's/^ssh_ca_public_key_url: "\(.*\)"$/\1/p' /ssh-ca/defaults/main.yml)
principal=$(sed -n 's/^ssh_ca_owner_principal: \(.*\)$/\1/p' /ssh-ca/defaults/main.yml)
[ -n "$ca_url" ] && [ -n "$principal" ] || { echo "ssh-ca defaults not readable"; exit 1; }
ca_key=$(curl -sSf "$ca_url")
[[ "$ca_key" == "ssh-ed25519 AAAA"* ]] || { echo "unexpected CA key from $ca_url"; exit 1; }
sshd_conf=$(grep -vx '# {{ ansible_managed }}' /ssh-ca/templates/ssh-ca.conf.j2)
! grep -q '{[{%]' <<<"$sshd_conf" || { echo "ssh-ca.conf.j2 has template logic"; exit 1; }
ssh_dir="$profile/airootfs/etc/ssh"
install -d "$ssh_dir/sshd_config.d" "$ssh_dir/auth_principals"
printf '%s\n' "$ca_key" > "$ssh_dir/trusted_user_ca_keys"
cp /ssh-ca/files/revoked_keys "$ssh_dir/revoked_keys"
printf '%s\n' "$principal" > "$ssh_dir/auth_principals/root"
printf '%s\n' "$sshd_conf" > "$ssh_dir/sshd_config.d/50-ssh-ca.conf"
# Tools the installer script needs in the live environment (idempotent —
# skip entries releng already ships).
for pkg in gptfdisk dosfstools e2fsprogs; do
+1
View File
@@ -13,6 +13,7 @@ mkdir -p "$out"
docker run --rm --privileged \
-v "$here:/src:ro" \
-v "$here/../../ansible/roles/ssh-ca:/ssh-ca:ro" \
-v "$out:/out" \
archlinux:latest /src/build-inner.sh
-6
View File
@@ -43,12 +43,6 @@ provider "proxmox" {
endpoint = var.proxmox_api_url
api_token = "${var.proxmox_user}!${var.proxmox_token_id}=${var.proxmox_token_secret}"
insecure = var.proxmox_tls_insecure
ssh {
agent = false
username = "root"
private_key = var.ssh_private_key
}
}
provider "unifi" {
-12
View File
@@ -28,16 +28,4 @@ variable "proxmox_tls_insecure" {
default = false
}
variable "ssh_public_key" {
description = "Public SSH key to inject into containers"
type = string
default = null
}
variable "ssh_private_key" {
description = "SSH private key for Proxmox provider to import disk images"
type = string
sensitive = true
default = ""
}