Compare commits

..
Author SHA1 Message Date
claude-botandClaude Opus 5.5 4852fde2a2 Merge main into worktree-restore-test-real
Hermes Matrix settings moved to the hermes-matrix-config ConfigMap on main; its homeserver reads ${SITE_DOMAIN}. The Home Assistant MCP URL reads ${SITE_DOMAIN}, $${HA_TOKEN} is escaped for Flux. alert-monitor gets GITEA_API on the site domain, the Hermes probe and HermesDown read ${SITE_DOMAIN}.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 07:59:14 +02:00
claude-botandClaude Opus 5.5 d4860aae82 feat(site): Hermes reaches Home Assistant on the site domain
The ha-updates skill from #1400 calls Home Assistant at HA_URL; the
agent container now sets it to https://home.${SITE_DOMAIN}. The script's
fallback and the prose in SOUL.md and the skill stay literal and are
allow-listed: they are configMapGenerator files, as SOUL.md already is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 20:59:22 +02:00
claude-botandClaude Opus 5.5 1d5d1bd066 Merge main into worktree-restore-test-real
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 20:55:01 +02:00
claude-botandClaude Opus 5.5 7a7ff239ff Merge old worktree-restore-test-real; tree is feat/site-base-domain-stack-2
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 20:51:15 +02:00
claude-botandClaude Opus 5.5 e141ec3414 feat(site): NetBird network, router group and setup key per site
netbird-service-sync gets NETWORK_NAME and ROUTER_GROUP_NAME from
site-settings (homelab: the controller.py defaults, Ruling 11), the
netbird-ingress ExternalSecret reads ${SITE_NETBIRD_INGRESS_KEY}. The
restore test's setup key comes from a second rotator,
netbird-secret-rotate-rtest, writing homelab/infrastructure/netbird-ingress-rtest
in production OpenBao so the raft snapshot carries it into the test. It
ships suspended with NB_AUTO_GROUPS as a marked placeholder (Ruling 2)
until the owner creates the NetBird group k8s-ingress-router-rtest.

Owner steps, none automated: create the group and set its ID; seed the KV
path (bao kv put homelab/infrastructure/netbird-ingress-rtest
setup-key=placeholder, field name as netbird-ingress: the DaemonSet reads
setup-key and the rotator's netbird-key mode writes BAO_FIELD default
setup-key), because the Job patches and a patch needs an existing secret;
bao policy write rotator-netbird (the policy is not Flux-applied); one Job
from the CronJob after merge.

Expected --changed homelab differences against the previous PR: the new
suspended CronJob in infrastructure, and the two explicit env values on
netbird-service-sync (one pod restart, behaviour unchanged).

Precondition: SITE_NETBIRD_NETWORK, SITE_NETBIRD_ROUTER_GROUP and
SITE_NETBIRD_INGRESS_KEY exist in the live site-settings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 20:26:01 +02:00
claude-botandClaude Opus 5.5 5ae8da89e8 feat(site): service CIDR and DNS gateway address per site
serviceSubnets moves from talos/patches/cluster.yaml into each site patch
(homelab and qemu keep 10.96.0.0/12, hcloud gets 10.112.0.0/12; the IPv6
range fd10:96::/112 stays shared, Ruling 10). The hcloud resolver asks
10.112.0.11 first. k8s-gateway's clusterIP and the CoreDNS forward read
${SITE_DNS_GATEWAY_IP}, netbird-service-sync's SERVICE_CIDR reads
${SITE_SERVICE_CIDR}; infrastructure and infrastructure-inhouse substitute
since the wiring PR. bootstrap.sh needs no change: its k8s-gateway values
go through site-render.sh --subst since the base-domain PR, which now also
fills clusterIP.

Production: the merged Talos config is unchanged (the subnet only moves
between patch files) and --changed homelab against the previous PR is
clean (the CoreDNS comment names the variable, so it renders identically).

Precondition: SITE_SERVICE_CIDR and SITE_DNS_GATEWAY_IP exist in the live
site-settings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 20:26:01 +02:00
claude-botandClaude Opus 5.5 2a874cf31c feat(site): base domain and ACME server per site
Every name a site serves (Ingress and TLS hosts, external-dns hostnames
and domain filters, Certificates, app URLs, authentik redirect URIs and
hosts, monitoring probes and links, CoreDNS and k8s-gateway zones) reads
${SITE_DOMAIN}; the ClusterIssuers read ${SITE_ACME_SERVER}, and
ACME_SERVER leaves cluster-settings. The paperless docs redirect regex
uses ${SITE_DOMAIN_RE} (Ruling 9). Home Assistant's config-sync GITEA_URL
(from #1326) is a served name too, and so are Hermes' hosts, dashboard
URL, OIDC issuer, tea login and git credential URL and the wiki's OIDC
authority, with their authentik redirect URIs. Names nothing in the
cluster serves stay literal and are listed in
testdata/site-render/allowed-production-names.txt (now also kuma.renner.uno
on the NetBird VPS, the uptime.renner.uno API group, Element Call's
rtc/turn.matrix names, the konairius Flux source, and two prose lines of
Hermes' SOUL.md: it is a configMapGenerator file, and substituting it
would change the hashed ConfigMap name and restart production);
hcloud_has_no_production_names renders every Kustomization for hcloud and
fails on any other renner.uno. KumaMonitor objects are not checked: they
are the production Uptime Kuma's monitors and keep production URLs;
kuma-sync, which alone reads them, is suspended on test sites.

bootstrap.sh installs k8s-gateway before Flux from the HelmRelease values,
which now carry ${SITE_DOMAIN}: the values go through
site-render.sh --subst <site> kubernetes/infrastructure (the same
environment the infrastructure Kustomization gets, --strict), so no
${...} reaches helm.

Expected --changed homelab differences against the previous PR (Ruling 1,
one red site-render run on this PR): authentik gate-sync gains
COOKIE_DOMAIN=renner.uno, gitea gains GITEA__server__ROOT_URL=
https://gitea.renner.uno (both production's values), monitoring-rules
refolds two long strings (parsed values unchanged), and
infrastructure's cluster-settings loses ACME_SERVER.

Precondition: SITE_DOMAIN, SITE_DOMAIN_RE and SITE_ACME_SERVER exist in the
live site-settings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 20:26:01 +02:00
claude-botandClaude Opus 5.5 3f08b74d88 feat(flux): features a site lacks are suspended through site-settings
llama-server and wiki follow SITE_SUSPEND_IGPU, matter-server,
home-assistant-iot and music-assistant follow SITE_SUSPEND_IOT_VLAN,
external-dns-unifi follows SITE_SUSPEND_UNIFI, kuma-sync follows
SITE_SUSPEND_UPTIME_MONITORING (uptime monitoring does not run on test
sites; the KumaMonitor objects stay with their apps and are inert without
the controller). The root substitutes site-settings since the wiring PR,
so the values reach these Kustomization objects; homelab renders an
explicit suspend: false, which --changed treats as unchanged. Tests: the
seven follow each site, otbr stays suspended, and no unsuspended
Kustomization depends on a suspended one.

Precondition: SITE_SUSPEND_IGPU, SITE_SUSPEND_IOT_VLAN, SITE_SUSPEND_UNIFI
and SITE_SUSPEND_UPTIME_MONITORING exist in the live site-settings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 19:52:12 +02:00
claude-botandClaude Opus 5.5 40f263e25f feat(flux): site-settings wiring; nothing substituted yet
Every Kustomization that later carries a ${SITE_*} variable substitutes
site-settings from this commit on, before any variable reaches it: live
Flux substitutes without --strict, so a variable that lands before its
Kustomization substitutes is applied literally, and one whose ConfigMap is
missing renders empty. The root sync (flux-system, via the kustomize patch
in flux-system/kustomization.yaml) and the restore-test root substitute
too, for the suspend values of the next PR; kuma-sync substitutes for
SITE_SUSPEND_UPTIME_MONITORING (uptime monitoring does not run on test
sites).

The site-settings inline manifest (all 12 keys) goes into the three site
patches. Production precondition: the ConfigMap exists on the cluster
before merge; the procedure is in
docs/superpowers/plans/2026-10-18-site-base-domain-rollout.md.

Objects with unrelated ${...} in trees that now substitute carry
kustomize.toolkit.fluxcd.io/substitute: disabled (Ruling 8: metadata only,
no pod restart). Where a later PR also puts a site value into the same
object, the unrelated reference is escaped instead ($${1},
$${INFLUXDB_TOKEN}), which renders identically.

site-render --changed homelab against the previous PR shows only the 13
substitute-disabled annotations and the site-settings entry appended to
infrastructure-crds' substituteFrom; every other path, hermes and
kuma-sync included, renders identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 19:52:12 +02:00
claude-botandClaude Opus 5.5 8c862298ab ci(manifests): render check for per-site values
site-render.sh renders a path the way Flux does with postBuild: kustomize
(with Flux's generated kustomization for directories without one), then
flux envsubst --strict with the Kustomization's substituteFrom ConfigMaps
in listed order (site-settings from the site's Talos patch) and its
inline substitute last. --changed renders the root and every Kustomization
naming site-settings at the base ref and at HEAD with the same site values
and fails on any difference, so a PR that moves literals into ${SITE_*}
variables proves the homelab render unchanged.

Both yq flavours: mikefarah yq (Alpine's yq, the CI image) prints a ---
line between results of different documents and evaluates string
interpolation on empty input, so results go through yqr (drops the
separators) and values are joined with tostring. BusyBox: grep has no
--include and sed is not GNU sed, so neither is relied on. The
substitute: disabled annotation counts only on the object's own
metadata, not a pod template's. --subst substitutes stdin as a
Kustomization would, for tooling that installs values outside Flux.

The CI step keeps ci-k8s:2026-09-08 until the image with yq exists; the
tag bump follows that build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 19:52:12 +02:00
claude-botandClaude Opus 5.5 c0994341ac ci(images): yq in ci-k8s
The site-render check (next PR) reads Talos site patches and Flux
Kustomizations with yq; ci-k8s has flux and kubectl but no yq. Alpine's
yq package is mikefarah yq. Ships alone so build-ci-images produces the
dated tag before the check that needs it is wired into a pipeline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 19:52:12 +02:00
claude-botandClaude Opus 5.5 034b7aae9f docs: per-site base domain and service CIDR for the restore test (design + plan)
The restore-test site gets its own base domain (rtest.renner.uno), service
CIDR, NetBird identity and ACME server as site values in the Talos site
patches, and the features it lacks (iGPU, IoT VLAN, UniFi gateway) are
suspended there. Production renders unchanged. Approved by the owner
2026-10-18.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 19:52:12 +02:00
claude-botandClaude Opus 5.5 9c35f95fb3 Merge master into worktree-restore-test-real
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
review-bot review failed: context deadline exceeded
Conflict in kubernetes/apps/woodpecker/objectstore.yaml: master (#1295)
pointed wp-pg-store-source at wp-pg-woodpecker; this branch moved both
woodpecker ObjectStores to kubernetes/cnpg-object-stores/wp-pg.yaml, whose
source path comes from cnpg-paths and covers wp-pg-woodpecker through
cnpg.renner.uno/source-prefixes. Resolved by keeping the deletion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 10:28:34 +02:00
claude-botandClaude Opus 5.5 20b0621e2e fix(home-assistant): shelly-manager-auth stays with mqtt-account-controller
420bd35e moved shelly-manager-auth-externalsecret.yaml to home-assistant-iot
with the rest of shelly-manager. mqtt-account-controller, which stays in
home-assistant, mounts that Secret as well (volume shelly-manager-key, item
api-key, not optional). That made a cycle:

- mqtt-account-controller does not start without shelly-manager-auth;
- it creates the MQTT client certificates HA, mosquitto and zigbee2mqtt
  mount (mqtt-home-assistant-tls, mqtt-mosquitto-probe-tls,
  mqtt-zigbee2mqtt-bridge-tls), so they do not start either;
- home-assistant therefore never turns Ready, and home-assistant-iot, which
  would create shelly-manager-auth, depends on it.

Restore test run 29 (2026-10-16) sat in it: every other Secret in the
namespace existed, mqtt-account-controller reported
  MountVolume.SetUp failed for volume "shelly-manager-key" :
  secret "shelly-manager-auth" not found
and evcc, trmnl, zone-configurator, speech-to-phrase and home-assistant-iot
waited behind it.

The ExternalSecret goes back to home-assistant; shelly-manager in
home-assistant-iot runs after it and finds the Secret. Every Secret the
home-assistant render mounts now has its producer in the same Kustomization
(mqtt-dynsec-seed is optional).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 23:04:15 +02:00
claude-botandClaude Opus 5.5 afdeecc161 fix(monitoring-rules): drop the radius metrics Service; authentik's operator creates it
authentik-outpost-servicemonitors.yaml hand-wrote a headless Service
ak-outpost-local-radius-metrics (9300/TCP) and a ServiceMonitor for it,
because the radius outpost controller used not to expose its metrics port.
authentik 2026.8.3 does: the operator creates that same Service
(http-metrics 9300, managed-by goauthentik.io, service-type metrics) and its
own ServiceMonitor ak-outpost-local-radius selecting exactly those labels.

Restore test run 29 (2026-10-16): on a cold cluster the operator created the
Service first, with a ClusterIP. clusterIP is immutable, so the dry-run of
the hand-written clusterIP: None copy failed and took the whole
monitoring-rules Kustomization with it:

  Service/authentik/ak-outpost-local-radius-metrics dry-run failed (Invalid):
  spec.clusterIPs[0]: Invalid value: ["None"]: may not change once set

In production Flux had created it first; the operator adopted it, and Flux's
label managed-by: homelab now keeps the operator's ServiceMonitor from
matching. Removing the file lets the operator own both; when Flux prunes the
Service, the operator recreates it with its own labels.

No alert rule refers to the radius job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 22:51:46 +02:00
claude-botandClaude Opus 5.5 0933ccdf7b fix(restore-test): CNPG archives, backs up and prunes as in production
Patches 3c (isWALArchiver: false), 3d (no retentionPolicy) and 3e
(ScheduledBackups suspended) kept the test cluster from writing into the
production CNPG bucket. Since cffb7e02 it cannot: bootstrap.sh writes
cnpg-paths with every archive path in renner-uno-k8s-cnpg-rtest, the
cnpg-object-stores Kustomization fails rather than render without it, and
the source stores, the only ones in the production bucket, carry no
retentionPolicy.

The patches now only stopped the test from exercising what a real recovery
does next: archiving WAL into a new prefix and taking the immediate base
backup there. Removing them leaves three fewer deviations from production.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 22:19:54 +02:00
claude-botandClaude Opus 5.5 9b488b037e fix(flux): services that need OpenBao secrets apply after the secret store
crowdsec, external-dns-unifi, netbird-ingress and velero were in the
infrastructure Kustomization, while the ExternalSecrets that create their
Secrets were in infrastructure-crds, which depends on infrastructure. On a
cold start each was therefore installed before its Secret could exist: the
pods sat in CreateContainerConfigError until OpenBao was unsealed and ESO
synced. For the HelmReleases (crowdsec, external-dns-unifi, velero) Helm's
wait timed out meanwhile; with remediation.retries: 3 they give up after
four attempts and stay failed even once the Secret appears. Velero failing
that way stops every restore.

| workload | Secrets |
|---|---|
| crowdsec | crowdsec-bouncer-key (crowdsec, ingress), crowdsec-edge-bouncer-key |
| external-dns-unifi | external-dns-unifi-secret |
| netbird-ingress | netbird-ingress-setup-key |
| velero (+ backup-verify CronJob) | cloud-credentials, cloud-credentials-hetzner, hcloud-backup-verify |

Each now has a Kustomization of its own (clusters/homelab/
infrastructure-services.yaml), depending on infrastructure and
infrastructure-crds, with its ExternalSecrets beside it. The openbao
ClusterSecretStore in infrastructure-crds turns Ready only once OpenBao is
unsealed (a5c5d0e7), so the Secret exists before the consumer installs. One
Kustomization each so that a broken service holds back only itself.

Velero's Schedules need the velero.io CRDs from its HelmRelease, so they move
from post-crds to infrastructure/velero-schedules with a Kustomization after
velero. velero-backups-synced depends on velero (the BackupStorageLocation
and its credentials) instead of infrastructure-crds.

post-crds keeps cloudflare-api-token (consumed by its own ClusterIssuer) and
flux-git. The directories themselves stay where they are, so paths such as
the CI test infrastructure/velero/backup-verify/tests/run.sh are unchanged.

Verified: infrastructure, post-crds and the six service paths build;
infrastructure renders no ExternalSecret.

Production rollout: moving objects out of the pruned infrastructure and
infrastructure-crds Kustomizations is two-phase (add the new Kustomizations
first, then drop the old entries). Pruning the velero HelmRelease would
uninstall Velero.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 22:01:55 +02:00
claude-botandClaude Opus 5.5 12a0f8c000 fix(flux): renovate depends on gitea
Renovate's platform is Gitea: endpoint https://git.renner.uno/api/v1 in
apps/renovate/configmap.yaml, the same instance as gitea.renner.uno. It has
no function without it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 21:39:07 +02:00
claude-botandClaude Opus 5.5 9dc3f5b4a0 fix(home-assistant): 30m timeout, not 4h
The 4h was sized for a seed initContainer that waited up to 3h for an HAOS
backup to appear in Garage, then downloaded and staged it. That step no
longer exists: Velero restores the config PVC in restore-home-assistant
before this Kustomization runs, and Garage is retired. The HA container's
startupProbe allows 10 minutes.

The slowest part left is ha-pg recovering from barman: its newest base backup
(ha-pg/ha-pg/base/20261016T025000) is 1.7 GiB compressed, then WAL replay and
the second instance's clone. 30m, as the other CNPG apps have; the next
restore test measures it.

With wait: true, the 4h also held one of kustomize-controller's workers that
long on a failed cold start, and with it evcc, trmnl, zone-configurator and
speech-to-phrase.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 21:38:07 +02:00
claude-botandClaude Opus 5.5 4befc7c492 Revert "feat(talos): hcloud nodes carry ens19 as a dummy link"
This reverts commit ec8f496d. The v30iot workloads now select
net.renner.uno/iot-vlan, which no hcloud node carries, so nothing on that
site attaches to ens19 any more.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 21:36:09 +02:00
claude-botandClaude Opus 5.5 420bd35e7f feat(iot): v30iot workloads run only where a node declares the IoT VLAN; HA no longer waits on them
Five workloads attach a macvlan on ens19 (v30iot) with DHCP IPAM:
shelly-manager and the mDNS relay (inside the home-assistant Kustomization),
matter-server, music-assistant and otbr. On a node without that VLAN the CNI
add fails and the pod never starts.

- k8s-1..3 carry net.renner.uno/iot-vlan: "true" beside the GPU label; each
  of them declares ens19 in its VLANConfig. The five Deployments select on
  it. The comments they replace described the retired k8s-edge taint.
- shelly-manager and the mDNS relay move to apps/home-assistant-iot, with a
  Kustomization of their own (depends on restore-home-assistant, gitea,
  home-assistant and the infrastructure Kustomizations). home-assistant's
  wait: true no longer covers them, so HA and everything waiting on it
  (evcc, trmnl, zone-configurator, speech-to-phrase) are Ready without the
  VLAN, and a broken IoT bridge no longer holds them back in production
  either.

On a site without the label the IoT Kustomizations fail at their timeout,
naming the Pending pods; nothing else depends on them.

Verified: home-assistant, home-assistant-iot, matter-server,
music-assistant, otbr, authentik and monitoring-rules build; the
home-assistant render has no IoT attachment left; every IoT workload carries
the selector; k8s-1 with site-homelab passes talosctl validate --strict.

Production rollout:
1. Label the nodes first (talosctl patch the nodeLabels), or the five pods go
   Pending.
2. The move between Kustomizations is two-phase (add under
   home-assistant-iot, then drop from home-assistant). shelly-manager's PVC
   already carries prune: disabled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 21:36:01 +02:00
claude-botandClaude Opus 5.5 cffb7e027a feat(cnpg): every cluster incarnation archives to a new prefix; recovery reads the newest
A CNPG cluster restored from barman archived back into the location it
recovered from. plugin-barman-cloud's restore job checks the archive target
(plugins[] ObjectStore + serverName) and stops with "Expected empty archive"
when it holds WAL, which after a disaster it always does. It has no
exception for target == recovery source (internal/cnpgi/restore/restore.go,
v0.15). Run 28 hit it on wp-pg; the other six never got that far.

The check stays. Instead, archive and source become two ObjectStores whose
paths are chosen per incarnation:

- kubernetes/cnpg-object-stores/: all 14 ObjectStores (<x>-store archives,
  <x>-store-source is read by bootstrap.recovery), in a Kustomization of their
  own with postBuild.substituteFrom: cnpg-paths. Nothing else lives there:
  substitution blanks every ${...}, which is why it is not on the app
  Kustomizations (home-assistant's scripts, the Grafana dashboards). The
  ConfigMap is not optional; without it the paths would render empty.
- bootstrap.sh writes cnpg-paths before Flux starts, once per incarnation
  (an existing one is kept):
    CNPG_<X>_SOURCE   the prefix among <x>, <x>-g<14 digits> and the store's
                      cnpg.renner.uno/source-prefixes whose <x>/base/ holds
                      the newest base backup
    CNPG_<X>_ARCHIVE  <x>-g<UTC timestamp>, new and so empty
  Every ScheduledBackup has immediate: true, so a new incarnation has a base
  backup in its own prefix minutes after it starts, and a later recovery
  picks it. S3 access is curl --aws-sigv4 with the restore key (already in
  bootstrap.sh's secret list), so the CI image needs no new tool.
- The restore test archives to renner-uno-k8s-cnpg-rtest, created by the
  flux phase and emptied and deleted by hcloudclean. A test incarnation with a
  base backup in the production bucket would be its newest, and the next real
  recovery would read the test's data.
- The six clusters that recovered from their archive store now name
  <x>-store-source in externalClusters. serverName stays the cluster name
  everywhere. The app Kustomizations depend on cnpg-object-stores, which
  depends on infrastructure-cnpg-barman.

Verified against the bucket, read-only, 2026-10-16:
- the restore key lists renner-uno-k8s-cnpg and can create and delete a
  bucket (renner-uno-k8s-cnpg-rtest created and deleted again);
- the listing parser on real responses, top level and an encoded nested
  prefix; for wp-pg it picks wp-pg-woodpecker (newest 20261016T024000) over
  wp-pg (20260818T023502);
- leftover test serverNames (ak-pg-rtest, im-pg-rtest, ha-pg-restore-test,
  pprleval-pg) hold no base backups and match no candidate pattern.
shellcheck clean; all seven apps and cnpg-object-stores build; the render
holds exactly the 14 expected variables.

Production rollout, one time:
1. Create flux-system/cnpg-paths with the current prefixes as both source and
   archive (ak-pg, ha-pg, im-pg, mx-pg, pprl-pg, tr-pg; wp-pg-woodpecker), so
   the running clusters keep archiving where they do.
2. Move the ObjectStores in two phases (add in cnpg-object-stores, then drop
   from the apps). All 14 now carry prune: disabled; before this only ak, im
   and pprl did.
3. Whether changing externalClusters on a live Cluster restarts its pods is
   not verified; check on one cluster first.

Older generations are not cleaned up: retention applies only to the current
archive prefix. A cleanup script is planned separately.

RECOVERY.md: the table describes the new layout; the manual restore drill
reads <x>-store-source. That drill wrote 3 WAL files to ak-pg/ak-pg-rtest on
2026-07-25 despite isWALArchiver: false, so its "only ever reads" claim
does not hold; not changed here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 21:27:21 +02:00
claude-botandClaude Opus 5.5 43dfa6158c fix(woodpecker): recover wp-pg from the prefix it archives to
wp-pg-store-source, the recovery source of wp-pg's bootstrap.recovery,
pointed at s3://renner-uno-k8s-cnpg/wp-pg: the prefix of the retired
apps/wp-pg. Since the namespace move wp-pg archives to wp-pg-woodpecker
(wp-pg-store). A cold rebuild would therefore have restored woodpecker as of
2026-08-18 and reported success.

Bucket, listed read-only 2026-10-16:
- wp-pg/wp-pg/base: newest 20260818T023502
- wp-pg-woodpecker/wp-pg/base: 20260916T024000 .. 20261016T024000 (daily),
  wals/ through timeline 6 (00000006.history from 2026-09-23)

The source now reads wp-pg-woodpecker; serverName stays wp-pg, the name the
live cluster archives under. bootstrap.recovery is honoured only when PGDATA
is absent, so the running cluster is unaffected. The old prefix wp-pg is no
longer referenced and keeps its objects; nothing applies retention to it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 21:13:56 +02:00
claude-botandClaude Opus 5.5 cae4d83f1b fix(flux): trmnl depends on home-assistant, home-assistant on influxdb
- trmnl: the renderer reads HA's API
  (http://home-assistant-bgp.home-assistant.svc.cluster.local:8123).
- home-assistant: its InfluxDB integration writes to influxdb
  (influxdb_token in HA's secrets.yaml).

No cycle: influxdb depends only on its restore and infrastructure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 20:54:21 +02:00
claude-botandClaude Opus 5.5 7d9dadccba fix(flux): woodpecker depends on gitea
Gitea is woodpecker's forge and its OAuth login provider
(WOODPECKER_GITEA_URL https://gitea.renner.uno); the server has no function
without it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 20:51:28 +02:00
claude-botandClaude Opus 5.5 0486ea2d99 fix(flux): gitea-mcp depends on gitea
gitea-mcp is a front end to Gitea's API (-H https://gitea.renner.uno) and
does nothing without it. Its image is on docker.gitea.com, which is why the
in-house-image sweep did not add this edge. /healthz reports the process
only, so without the edge the Kustomization went Ready with no Gitea behind
it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 20:49:34 +02:00
claude-botandClaude Opus 5.5 abf549bb9a fix(scripts): flux-graph.sh draws what Flux actually checks
A Kustomization with healthChecks and no wait is health-checked (as
infrastructure is), and one with neither has no check at all, whatever its
healthCheckExprs say. Suspended Kustomizations are filled grey and labelled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 20:40:44 +02:00
claude-botandClaude Opus 5.5 813822462a fix(flux): every app Kustomization waits for its workloads; otbr suspended
With wait: false and no healthChecks, Ready means only that the manifests
were applied: dependents are released before the app runs, and a broken app
reports green. kustomize-controller's checkHealth returns before any check in
that case (len(Spec.HealthChecks) == 0 && !Spec.Wait), so healthCheckExprs
are not evaluated either.

- home-assistant: wait: true. The reason given (the USB meter, the canary)
  no longer holds; HA reads the meter over MQTT. evcc, zone-configurator and
  speech-to-phrase depend on it, and evcc needs mqtt-evcc-client-tls from it.
- llama-server, wiki: wait: true, reverting the wait: false of 7186e4d0. On a
  site without the iGPU they fail rather than report Ready.
- wordpress: wait: true. It set no wait, so its MariaDB healthCheckExprs
  were never evaluated.
- otbr: suspended. It is not in use.

infrastructure keeps wait: false with its explicit healthChecks, which is a
health gate on the operators, not the absence of one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 20:40:44 +02:00
claude-botandClaude Opus 5.5 09bf50731d feat(scripts): flux-graph.sh draws the Kustomization dependency graph
Reads every kustomize.toolkit.fluxcd.io/v1 Kustomization in
kubernetes/clusters/homelab and emits Graphviz DOT, one edge per dependsOn
entry, pointing from the dependency to its dependent. Each node carries its
wait mode and timeout. Infrastructure is green, the Velero restore chain
blue, wait: false dashed.

With no argument or a .dot path it writes the full graph; with a .svg or .png
path it renders the transitive reduction (tred), which drops edges implied by
a longer path.

Generated rather than checked in, so it cannot drift from the manifests.
Works with either yq (mikefarah's, as in the CI images, or the Python
wrapper): yq only converts YAML to JSON, jq builds the DOT.

Verified: the DOT matches the earlier hand-run yq output byte for byte;
.png and .svg render; shellcheck clean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 20:35:12 +02:00
claude-botandClaude Opus 5.5 26976414e4 fix(authentik): its ServiceMonitor lives in monitoring-rules
The authentik chart renders server/servicemonitor.yaml whenever
server.metrics.serviceMonitor.enabled is set, without checking
.Capabilities for monitoring.coreos.com (verified in chart 2026.8.3). The
authentik Kustomization does not depend on monitoring, so on an empty cluster
the Helm install fails until kube-prometheus-stack has installed the CRDs.

The chart's ServiceMonitor is switched off and declared standalone in
monitoring-rules, as cilium and velero already do. The metrics Service is
gated only on server.metrics.enabled and stays. The selector matches the live
Service authentik-server-metrics (port metrics, 9300).

The new name, authentik-server-metrics, differs from the chart's
authentik-server, so the two never contend for one object while production
rolls over.

Swept every manifest for other chart-rendered monitoring CRs outside the
monitoring Kustomization: cilium and velero already off, blackbox-exporter
renders none (no targets, selfMonitor off).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 20:26:28 +02:00
claude-botandClaude Opus 5.5 ec8f496d95 feat(talos): hcloud nodes carry ens19 as a dummy link
The v30iot NetworkAttachmentDefinitions of matter-server, music-assistant,
otbr, shelly-manager and the mdns relay name master: ens19. On the hcloud
site no such link existed, so macvlan creation failed outright. Each hcloud
node now declares ens19 as a DummyLinkConfig: no address, nothing behind it.

The NADs lease by DHCP, and nothing answers on a dummy link, so the pods get
their macvlan but the dhcp IPAM step still times out.

Validated: talosctl gen config (cluster + site-hcloud + hcloud-1) then
talosctl validate --mode cloud --strict.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 20:25:21 +02:00
claude-botandClaude Opus 5.5 7186e4d035 feat(gpu): iGPU workloads run only where a node declares the iGPU
llama-server, llama-embed, wiki-embed (gpu.intel.com/i915) and the wiki
behind them are non-essential and bound to the Intel iGPU. On a site
without one (Hetzner, restore rehearsal runs 27/28) their pods stayed
Pending, llama-server failed its health check and the Kustomizations held
kustomize-controller workers; the GPU plugin and exporter DaemonSets sat
on every node with nothing to serve.

The node patches k8s-1..3 label their machines
hardware.renner.uno/intel-gpu=true. The GPU plugin, the exporter, the
three llama Deployments and the wiki select on it, and llama-server and
wiki are no longer waited on. Elsewhere the DaemonSets have no pods and the
Deployments stay unscheduled without blocking anything.

Production rollout order: the label must be on the nodes (talosctl patch,
no reboot) before these manifests reconcile, or the GPU plugin and the
llama/wiki pods are removed or left Pending.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 12:51:49 +02:00
claude-botandClaude Opus 5.5 e9ca0c4e90 fix(mqtt-account-controller): start without the HA client certificate
The controller mounted mqtt-home-assistant-tls as a required volume, and is
the only creator of that Secret (it applies the cert-manager Certificates of
the register's cert accounts). On a new cluster - Velero restores volumes,
not Secrets - its pod could never start, and mosquitto, Home Assistant,
Zigbee2MQTT and evcc stayed in ContainerCreating behind the Secrets it would
have created.

Optional now. Verified in konairius/mqtt-account-controller main: the cert
dir is read only by the HA cert sync (02:00-05:00 window, errors logged);
the broker is dialled lazily; Secret.Read treats a missing Secret as empty;
Reconcile runs reconcileCerts even when the broker step fails. So the
controller starts, creates the Certificates, the consumers start, and the
next loop reconciles the broker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 12:44:26 +02:00
claude-botandClaude Opus 5.5 f3ba845e52 fix(restore-test): no writes to production NetBird from the test cluster
netbird-service-sync in the test cluster reconciled production NetBird
(netbird.renner.uno) with the PAT from the restored OpenBao: it rewrote the
public proxy services to the test cluster's Traefik ClusterIP, and
production's own controller re-created 14 and updated 5 of them every
minute from 09:53 to 10:39 UTC during run 28. The netbird-ingress routing
peers likewise joined the production mesh, announcing the same service and
LoadBalancer ranges as production.

Two safety patches in flux-root.yaml: netbird-service-sync runs with zero
replicas (3i), and netbird-ingress gets a nodeSelector no node carries (3j).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 12:43:19 +02:00
claude-botandClaude Opus 5.5 a02affc872 fix(velero): one restore Kustomization per schedule; each app waits for its own
Every app with backed-up volumes depended on one restore Kustomization that
was Ready only after all twelve Velero restores finished. authentik waited
for immich's photo library, home-assistant for paperless; in restore
rehearsal run 28 that was 45 minutes, during which the restore
Kustomization also held one of kustomize-controller's four workers.

kubernetes/restore/<schedule>/ now holds one Restore each, applied by a
restore-<schedule> Kustomization (clusters/homelab/restore.yaml) with the
same health check, after restore-gitea so Gitea's data still goes first.
Each app depends on its own: apps, audiobookshelf (libation through it),
authentik, home-assistant (matter-server, otbr, whisper too, same
namespace and schedule), immich, influxdb, matrix, music-assistant,
navidrome, paperless, trmnl, wordpress. Timeouts follow the data: immich
6h, audiobookshelf 3h, the rest 1h.

Not on production yet: the restore Kustomizations exist only on this
branch, so no Restore objects move between pruned Kustomizations there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 12:35:24 +02:00
claude-botandClaude Opus 5.5 0ac74b7b5b fix(flux): evcc depends on home-assistant
evcc reads its MQTT client certificate from mqtt-evcc-client-tls
(apps/evcc/deployment.yaml:48-59), which the mqtt-account-controller in
the home-assistant Kustomization delivers into the evcc namespace
(apps/home-assistant/mqtt-accounts.yaml:40-43), and connects to
mosquitto.home-assistant. On a cold cluster evcc started first and sat in
CreateContainerConfigError (restore rehearsal runs 27, 28). No cycle:
home-assistant does not reach evcc.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 11:57:46 +02:00
claude-botandClaude Opus 5.5 4ebc00e57a fix(bird): runs only where the site routes pods into table 47
bird fills routing table 47, which exists only where the site patch routes
pod traffic into it (site-homelab.yaml RoutingRuleConfig), and renders its
config from node annotations that only that site and the k8s-N patches set
(pod0, egress gateway). On Hetzner the template rendered <no value> and the
init container crash-looped ("syntax error, unexpected '<'", run 28).

site-homelab.yaml labels its nodes net.renner.uno/pod-table=47 and the
DaemonSet selects on it; on a site without table 47 it has no pods.

Production rollout order: the label must be on the nodes (talosctl patch,
no reboot) before the DaemonSet change reconciles, or bird is removed from
every node and pod traffic falls through to the main table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 11:57:25 +02:00
claude-botandClaude Opus 5.5 37352808c6 fix(flux): monitoring depends on influxdb
Grafana mounts the grafana-influxdb-token Secret, which a Job in the
influxdb app writes (apps/influxdb/grafana-token-job.yaml). On a cold
cluster monitoring started first; Grafana sat in CreateContainerConfigError
and the monitoring Kustomization held a kustomize-controller worker on its
health check (restore rehearsal run 28). No cycle: influxdb does not use
monitoring's CRDs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 11:48:16 +02:00
claude-botandClaude Opus 5.5 36b42677d6 fix(flux): in-house-image infrastructure waits for Gitea; fallback images don't
bird and netbird-service-sync run images from Gitea's registry with no
copy on ghcr.io. In the infrastructure Kustomization they started long
before the restored Gitea served them and sat in ImagePullBackOff (runs 27,
28). infrastructure cannot depend on gitea, which depends on it, so both
move to a new Kustomization, infrastructure-inhouse, that depends on
infrastructure, infrastructure-crds and gitea. Their namespace stays in
infrastructure.

Images published to ghcr.io too have a fallback in the nodes' registry
mirror and need no ordering: apps (ical-proxy: GitHub CI pushes
ghcr.io/konairius/ical-proxy main-<sha>) and trmnl (alpine-jq) lose the
gitea edge from the previous commit; multus cni-plugins stays in
infrastructure. wiki keeps it: no ghcr copy of konairius/wiki was found.

The secret-rotator and backup-verify CronJobs also use Gitea images but
only run on schedule; they are left in place.

Production rollout must be two-phase (moving objects between pruned
Kustomizations): first add infrastructure-inhouse with both paths while
infrastructure still lists them, then remove them from infrastructure.
Applied as one change, infrastructure prunes bird before the new
Kustomization re-creates it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 11:43:34 +02:00
claude-botandClaude Opus 5.5 abc2a1b0a5 fix(flux): apps with in-house images depend on gitea
Their images are pulled from Gitea's registry. On a cold cluster they
started before the restored Gitea served it and sat in ImagePullBackOff,
retried on the kubelet's back-off (restore rehearsal runs 27, 28). They now
wait for the gitea Kustomization: apps (ical-proxy), authentik (server,
LDAP outpost, gate-sync), home-assistant (shelly-manager,
mqtt-account-controller), paperless (ftp-scanner), review-bot, trmnl
(renderer), wiki.

No cycle: gitea depends only on restore-gitea and the infrastructure
layers. The infrastructure Kustomization itself also runs in-house images
(bird, netbird-service-sync, secret-rotator, backup-verify, cni-plugins);
it cannot depend on gitea, which depends on it, and is left for a
separate change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 11:40:25 +02:00
claude-botandClaude Opus 5.5 d25366a80c fix(bootstrap): read the hcloud node range from the machine document only
site-hcloud.yaml now holds a ResolverConfig document too; the unselected
query printed "10.90.0.0/16" and "null" and hcloud network create
rejected the range (run 28).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 11:12:18 +02:00
claude-botandClaude Opus 5.5 4a4390ef46 feat(dns): the nodes resolve through k8s-gateway at every site, from the first pod on
Node-level image pulls of gitea.renner.uno/* go through the node's resolver,
not the cluster's. Only site-homelab.yaml pointed it at k8s-gateway (PR
#1292); on Hetzner the nodes asked Hetzner's resolvers, got the public
proxy (blocked by the test firewall) and fell through to ghcr, which only
mirrors konairius/* - bird, netbird-service-sync and review-bot stayed in
ImagePullBackOff (restore rehearsal run 27) while the restored Gitea had
them.

PR #1292 also left a cold-start gap (run 23): until k8s-gateway exists the
node's first resolver does not answer, every lookup waits out the 2 s read
timeout, CoreDNS gives up first and pods resolve nothing - Flux included.

- Each site patch states its node resolver list (ResolverConfig: k8s-gateway
  first, then the site's resolver: UDM at home, Hetzner's on hcloud) and
  k8s-gateway's upstream as a k8s-gateway-site ConfigMap (deployment
  dnsPolicy/dnsConfig), read by the HelmRelease via valuesFrom. The Corefile
  forwards to the pod's resolv.conf, never to the node's resolver (loop).
- bootstrap.sh installs k8s-gateway right after the nodes are Ready, from
  the HelmRelease under the site values, as Flux merges them; Flux adopts
  the release. k8s-gateway answers before anything else needs DNS.

helm template (chart 4.0.1) renders forward . /etc/resolv.conf with the
site's nameservers in dnsConfig for homelab and hcloud. talosctl validate
passes for k8s-1 and hcloud-1.

Rollout order for production: the k8s-gateway-site ConfigMap must exist
before this HelmRelease reconciles (valuesFrom is not optional).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 11:04:05 +02:00
claude-botandClaude Opus 5.5 a5c5d0e79d fix(flux): infrastructure is Ready only once OpenBao is unsealed
The infrastructure Kustomization health-checked the openbao HelmRelease,
which is Ready as soon as the chart is installed, sealed or not. On a cold
cluster infrastructure-crds then applied the openbao ClusterSecretStore
while OpenBao was unreachable and then sealed ("503 Vault is sealed").
External Secrets backs the store off exponentially, so every
ExternalSecret stayed "ClusterSecretStore not ready" for minutes after
OpenBao was restored and unsealed; Velero's pod could not mount
cloud-credentials-hetzner meanwhile (restore rehearsal run 27; ~15 min in
run 22).

The openbao StatefulSet is now a health check too. Its readiness probe is
`bao status`, which fails while sealed, and podManagementPolicy is
OrderedReady, so 3/3 ready means every replica is unsealed. No cycle:
OpenBao's PVC comes from infrastructure-ceph-cluster, which does not
depend on infrastructure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 10:34:42 +02:00
claude-botandClaude Opus 5.5 ed8241384d feat(cilium): native-routing CIDRs are a site fact too
ipv4/ipv6NativeRoutingCIDR exempt traffic into those ranges from
masquerading (Cilium: handed to the kernel "without applying any SNAT").
At home that is the point: 10.0.0.0/8 and fd00::/8 let LAN services see
real pod IPs, routed back by the UDM via BGP. On Hetzner the same range
covers the node segment 10.90.0.0/16, whose gateway drops packets with a
source not assigned to the sending server (uRPF, Hetzner networks FAQ).

Run 26: a pod on hcloud-2 reached only its own node's kube-apiserver
(10.90.0.3:6443 -> 401); 10.90.0.2 and 10.90.0.4 timed out, the pod's SYNs
went to-stack with the pod address. Flux controllers on hcloud-2 crash-
looped and flux install failed.

The CIDRs move into each site's cilium-site values: homelab and qemu keep
10.0.0.0/8 and fd00::/8, hcloud exempts only the pod CIDRs. Production
still renders identically (helm template, TLS secrets excluded); talosctl
validate passes for k8s-1 and hcloud-1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 10:14:15 +02:00
claude-botandClaude Opus 5.5 16b6aee04d feat(cilium): pod routing is a site fact, from the site patch
routingMode, autoDirectNodeRoutes and nodePort.directRoutingDevice describe
the site's network, not the cluster: at home pod traffic crosses nodes
natively on pod0 (VLAN 48, one L2 segment). On Hetzner Cloud that cannot
work: networks are layer 3, route only server addresses, and carry IPv4
only (docs.hetzner.com networks FAQ), so cross-node pod traffic had no
path (run 25: every DNS query to a CoreDNS pod on another node timed out).

Each Talos site patch now carries a cilium-site ConfigMap as an inline
manifest with its part of Cilium's values; the HelmRelease reads it via
valuesFrom and bootstrap.sh merges it the same way (site values first,
HelmRelease values on top, as Flux merges them):
- homelab: native, autoDirectNodeRoutes, directRoutingDevice pod0
- hcloud: tunnel (VXLAN) over the node segment
- qemu: native on the provisioner's single L2 bridge

Production renders unchanged: helm template of cilium 1.20.3 with master's
values vs. homelab site values + the new values gives identical manifests
(random TLS secrets excluded). talosctl validate passes for k8s-1 (metal)
and hcloud-1 (cloud).

Rollout order for production: the ConfigMap must exist before this
HelmRelease reconciles (valuesFrom is not optional). Talos creates inline
manifests on a config change but never updates them; changes go through
talosctl upgrade-k8s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 09:59:21 +02:00
claude-botandClaude Opus 5.5 4922d8777e fix(bootstrap): split the hcloudtest tcp egress rule at 100 prefixes
Hetzner rejects a firewall rule with more than 100 destination prefixes
("rules[2].destination_ips: value required to be smaller"; the limit is
not in the API spec or the FAQ — measured on an unattached firewall: 100
accepted, 101 rejected). Run 24 failed on the single rule carrying all
245 complement prefixes. The tcp allow-list is now one rule per 100.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 09:19:52 +02:00
claude-botandClaude Opus 5.5 f58e06fc76 fix(bootstrap): create Cilium's CNI subdirectory and install Multus on a cold cluster
Cilium's cni.confPath is /etc/cni/net.d/multus, a directory Multus creates
(#832). On an empty cluster neither exists: /etc/cni is read-only to the
kubelet on Talos, so the agent's DirectoryOrCreate mount cannot make it,
and the install waits forever. Without Multus the kubelet reports "cni
plugin not initialized" and nothing schedules, Flux included.

A hostNetwork pod per node creates the directory (there is no pod network
before Cilium), and Multus is applied from the manifests Flux owns right
after Cilium. Found in the Hetzner restore rehearsal, where both steps ran
successfully in runs 22 and 23; this is the production bootstrap path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 09:07:00 +02:00
claude-botandClaude Opus 5.5 b550e84fcc feat(restore-test): Flux root with the test's deviations, and the egress check
flux-root.yaml applies the production tree from the GitHub mirror; its
patches are the source rewrite, no Ceph cluster network (Hetzner networks
are layer 3), and the safety set: Velero ReadOnly, no CNPG WAL archiving,
retention or base backups, no MariaDB dumps, rotators suspended, Gitea
without world egress.

egress-check.yaml asserts from a pod: public DNS, github.com https and ssh,
ghcr.io and object storage reachable; every proxy.renner.uno address closed
on 443. stderr is reported, not discarded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 09:05:13 +02:00
claude-botandClaude Opus 5.5 bb6874f59c feat(bootstrap): hcloudtest boots production's config, contained by egress firewall
hcloudtest runs SITE=hcloud on cpx52 with the OSD on the system disk, and
nothing stands in for the home network. Ported from worktree-hcloud-wg-site:
the Talos snapshot builder (removes its server on failure), this host's
inbound firewall replacing earlier runs', the Hetzner token from OpenBao, the
FLUX_SOURCE=restore-test phase (flux install, per-run read-only deploy key on
the GitHub mirror, restore-test-git Secret, flux-root.yaml on FLUX_BRANCH),
its safety prechecks, access files under ~/.local/state/homelab-hcloudtest
instead of access-staging, hcloudclean deleting ssh keys and restore-test
deploy keys, and the 30-minute OpenBao status wait.

Containment: an outbound firewall on the nodes, applied before any config,
admits udp and icmp anywhere and tcp to the complement of proxy.renner.uno's
addresses, resolved over DoH at run time (cidr_complement, in bash; fails
above Hetzner's 500 effective rules). The egress check pod runs at the end
of hcloudtest and again before the restore test writes any secret.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 09:05:13 +02:00
claude-botandClaude Opus 5.5 af7c48e29f fix(talos): etcd subnet in the site patches; hcloud OSD on the system disk
cluster.yaml named VLAN 46 as etcd's advertised subnet, a fact of the home
network. It moves unchanged to site-homelab.yaml (the rendered production
config is byte-identical); site-hcloud.yaml advertises etcd on the node
segment bootstrap.sh provisions, 10.90.0.0/16.

cpx52 carries 480 GB local disk and the project's volume quota cannot hold
three OSD volumes: EPHEMERAL is capped at 60GiB and osd-data takes at least
345GiB of the system disk (ported from 92c7ee73, 8d42d89c).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 08:58:41 +02:00
claude-botandClaude Opus 5.5 33077cd1bc fix(velero): restore Gitea first, the rest after it
Gitea is the registry for the in-house images. With one shared restore gate it waited for every volume, Immich's included, while everything pulling from it looped on image pull errors. restore-gitea restores only Gitea and the gitea Kustomization depends on it alone; the other Restores depend on it, so Velero moves Gitea's data first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 08:54:18 +02:00
claude-botandClaude Opus 5.5 99af3d6efc fix(rook): install the ceph-csi-drivers chart Rook requires
Since Rook 1.20 the CSI Driver and OperatorConfig resources come from the
ceph-csi-drivers chart (rook Documentation/Helm-Charts/csi-drivers-chart.md:
"Rook requires the installation of the Ceph-CSI Driver chart"). The chart
was never installed here: the live cluster's Drivers are the ones the
operator migrated once during the 1.19->1.20 upgrade (2026-05-09), and
PR #12 only rendered the ServiceAccounts and RBAC. A fresh cluster
therefore has no CSI driver and provisions no volume: found by the
Hetzner restore test, where every PVC stayed Pending ("Waiting for a
volume to be created either by the external provisioner
'rook-ceph.rbd.csi.ceph.com'").

Values: Rook's recommended values for 1.21.0 plus the live shape:
2 controller replicas with Recreate, the workload-class=pinned
toleration on the node plugins, a snapshotter for RBD (Velero's CSI
snapshots of ceph-block) and group snapshots for CephFS.

The chart's ServiceAccounts are named per driver
(rook-ceph-rbd-csi-ceph-com-ctrlplugin-sa, ...), so they do not collide
with the static ones; the static ServiceAccounts and RBAC stay until the
live CSI pods run on the chart's, and go in a second change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 08:54:18 +02:00
claude-botandClaude Opus 5.5 dcc034be92 feat(velero): the cluster restores its app volumes itself
One Velero Restore per Schedule (kubernetes/restore/restores.yaml),
applied by Flux, restoring the schedule's newest completed backup. Every
app Kustomization with backed-up volumes depends on the new `restore`
Kustomization, so on a fresh cluster no app starts on an empty volume.
Restores the capability 952e6150 removed without an owner decision.

Ordering and failure handling, from Velero release-1.17 source:
- A Restore whose schedule has no synced backup yet goes FailedValidation
  for good (restore_controller.go). `restore` therefore depends on
  `velero-backups-synced`, which is Ready once the BackupStorageLocation
  has completed a sync (status.lastSyncedTime).
- After that sync, FailedValidation with "No backups found for schedule"
  means there is nothing to restore and counts as done; any other
  FailedValidation, Failed or PartiallyFailed fails the Kustomization.
- A claim that already exists is skipped (csi pvc_action: "PVC already
  exists"), and the Restore completes with a warning. On the running
  cluster the Restores touch nothing.
- Velero deletes a Restore with the backup it came from (30-day TTL);
  Flux recreates it, and it again skips the existing claims.
- All StorageClasses bind Immediate, so the data mover does not wait for
  a consuming pod.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 08:54:18 +02:00
claude-botandClaude Opus 5.5 fb7c12a92a fix(gitea): no pinned volumeName on data-gitea-0
The claim named its production PV, which does not exist on a rebuilt
cluster: the claim stayed Pending there, and a restored claim bound to a
new volume made the gitea Kustomization fail on the immutable field.

volumeName leaves git and the gitea Kustomization ignores
/spec/volumeName on this claim. kustomize-controller is the field's
only manager on the live claim, so its ignore rule adopts the live value
instead of clearing it; size growth still applies.

To verify after merge: the gitea Kustomization reconciles Ready and the
live claim still names pvc-e7c56db2-481d-40cb-b94a-b264b7b4b5be.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 08:54:18 +02:00
claude-botandClaude Opus 5.5 5927e4bc0e fix(talos): IPv4 first in nodeIP.validSubnets, as the live nodes run
Talos keeps the validSubnets order (siderolabs/net FilterIPs) and the first
entry becomes the primary node IP. Git had fd02:46::/64 first since
be643533 while k8s-1 runs with 10.2.46.0/24 first. A cluster built from git
gets an IPv6 primary and kube-apiserver exits with
'service IP family "10.96.0.0/12" must match public address family
"fd02:46::11"' — reproduced on a Hetzner rebuild rehearsal 2026-10-15.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 08:54:18 +02:00
497 changed files with 7045 additions and 13753 deletions
+3
View File
@@ -20,5 +20,8 @@ kubernetes/access-staging/
trivy-iac.json
trivy-image-*.json
# CI diagnostic kubeconfig + kubectl for the agent steps (generated, never commit)
.ci-diag/
# Plaintext secret templates awaiting SOPS encryption (never commit)
*.plaintext.yaml
+12 -35
View File
@@ -243,8 +243,7 @@ steps:
- AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
- GH_AUTH=$(printf '%s:%s' "$GHCR_USER" "$GHCR_TOKEN" | base64 | tr -d '\n')
- printf '{"auths":{"gitea.renner.uno":{"auth":"%s"},"ghcr.io":{"auth":"%s"}}}' "$AUTH" "$GH_AUTH" > /kaniko/.docker/config.json
# konairius/ gets one tag per build: Gitea's cleanup rule counts tags.
- TAG=$(date +%Y-%m-%d-%H%M)
- TAG=$(date +%Y-%m-%d)
- |
/kaniko/executor \
--context . \
@@ -252,7 +251,9 @@ steps:
--destination gitea.renner.uno/gitea_admin/alpine-jq:$TAG \
--destination gitea.renner.uno/gitea_admin/alpine-jq:latest \
--destination gitea.renner.uno/konairius/alpine-jq:$TAG \
--destination ghcr.io/konairius/alpine-jq:$TAG
--destination gitea.renner.uno/konairius/alpine-jq:latest \
--destination ghcr.io/konairius/alpine-jq:$TAG \
--destination ghcr.io/konairius/alpine-jq:latest
# Runtime image, not CI tooling: upstream containernetworking/plugins binaries
# baked in, consumed by the multus cni-plugins-installer DaemonSet. Replaces a
@@ -294,8 +295,7 @@ steps:
- AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
- GH_AUTH=$(printf '%s:%s' "$GHCR_USER" "$GHCR_TOKEN" | base64 | tr -d '\n')
- printf '{"auths":{"gitea.renner.uno":{"auth":"%s"},"ghcr.io":{"auth":"%s"}}}' "$AUTH" "$GH_AUTH" > /kaniko/.docker/config.json
# konairius/ gets one tag per build: Gitea's cleanup rule counts tags.
- TAG=$(date +%Y-%m-%d-%H%M)
- TAG=$(date +%Y-%m-%d)
- |
/kaniko/executor \
--context . \
@@ -303,7 +303,9 @@ steps:
--destination gitea.renner.uno/gitea_admin/cni-plugins:$TAG \
--destination gitea.renner.uno/gitea_admin/cni-plugins:latest \
--destination gitea.renner.uno/konairius/cni-plugins:$TAG \
--destination ghcr.io/konairius/cni-plugins:$TAG
--destination gitea.renner.uno/konairius/cni-plugins:latest \
--destination ghcr.io/konairius/cni-plugins:$TAG \
--destination ghcr.io/konairius/cni-plugins:latest
# Runtime image, not CI tooling: alpine + the openssl CLI, for shell scripts
# needing s_client/x509. Named for what it is; first consumer is the paperless
@@ -334,8 +336,7 @@ steps:
- AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
- GH_AUTH=$(printf '%s:%s' "$GHCR_USER" "$GHCR_TOKEN" | base64 | tr -d '\n')
- printf '{"auths":{"gitea.renner.uno":{"auth":"%s"},"ghcr.io":{"auth":"%s"}}}' "$AUTH" "$GH_AUTH" > /kaniko/.docker/config.json
# konairius/ gets one tag per build: Gitea's cleanup rule counts tags.
- TAG=$(date +%Y-%m-%d-%H%M)
- TAG=$(date +%Y-%m-%d)
- |
/kaniko/executor \
--context . \
@@ -343,7 +344,9 @@ steps:
--destination gitea.renner.uno/gitea_admin/alpine-openssl:$TAG \
--destination gitea.renner.uno/gitea_admin/alpine-openssl:latest \
--destination gitea.renner.uno/konairius/alpine-openssl:$TAG \
--destination ghcr.io/konairius/alpine-openssl:$TAG
--destination gitea.renner.uno/konairius/alpine-openssl:latest \
--destination ghcr.io/konairius/alpine-openssl:$TAG \
--destination ghcr.io/konairius/alpine-openssl:latest
# Heating CAN readers: open3e (Vitocal/VX3) + E3onCAN (E380 meter), one image
# run as two services on edge-1. Replaces pip-into-venv on the host, where an
@@ -451,32 +454,6 @@ steps:
--destination gitea.renner.uno/gitea_admin/kuma-sync:$TAG \
--destination gitea.renner.uno/gitea_admin/kuma-sync:latest
# cloudflare-lb-sync: reconciles CloudflareLoadBalancer resources into
# Cloudflare. Its unit tests run as a stage of the image build.
- name: build-cloudflare-lb-sync
image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.19-debug@sha256:7795cec509a50a71378cc2c365f55c555ee3a05834d8eb1cd2552df5e66d44d8
directory: ci/images/cloudflare-lb-sync
when:
- event: deployment
- event: push
path: ["ci/images/cloudflare-lb-sync/**", ".woodpecker/build-ci-images.yml"]
environment:
GITEA_USER: gitea_admin
GITEA_TOKEN:
from_secret: ci_images_token
commands:
- mkdir -p /kaniko/.docker
- AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
- printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
# YYYY-MM-DD-HHMM: the image-automation ImagePolicy picks the newest.
- TAG=$(date +%Y-%m-%d-%H%M)
- |
/kaniko/executor \
--context . \
--dockerfile Containerfile \
--destination gitea.renner.uno/gitea_admin/cloudflare-lb-sync:$TAG \
--destination gitea.renner.uno/gitea_admin/cloudflare-lb-sync:latest
# bird: BGP daemon for the pod-network mesh over pod0 (#576). Runtime
# artifact like heating-can; same registry so the audit tracks it.
- name: build-bird
+21 -4
View File
@@ -37,6 +37,10 @@ when:
- ".woodpecker/**"
- "scripts/ci/promtool-rules.sh"
- "scripts/ci/yaml-citations.sh"
- "scripts/ci/site-render.sh"
- "scripts/ci/site-render-lib.sh"
- "scripts/ci/test-site-render.sh"
- "scripts/ci/testdata/site-render/**"
# THE THIRD DISCOVERY CHECK WENT TO A SCRIPT. kustomize-build and shellcheck
# both hand-roll the same shape — `find`, a floor guard against empty discovery,
@@ -66,9 +70,9 @@ steps:
# Verified green across every kustomization in the tree when this was added,
# so this gates new breakage rather than pre-existing debt.
#
# ci-k8s TAG SPLIT: this file pins a newer dated tag than e2e.yml and
# e2e-test.yml, which still sit on an older one. That is
# staleness in those two, not a deliberate split here — everything should be on
# ci-k8s TAG SPLIT: this file pins a newer dated tag than e2e.yml, e2e-test.yml
# and issue-fix.yml, which still sit on an older one. That is
# staleness in those three, not a deliberate split here — everything should be on
# the newest dated tag. They are not bumped here because none of them are
# triggered by this PR, so it would ship untested changes under cover of a green
# run that never touched them. Tracked in #461.
@@ -319,7 +323,7 @@ steps:
# writes its stripped mirror under mktemp and never into the workspace.
- name: promtool
depends_on: [] # see note above `steps:`
image: prom/prometheus:v3.15.1@sha256:fa3302279710be2f21a8301ec17a8b4f2301fd829bd113839b3b6b02b6f191a2
image: prom/prometheus:v3.15.0@sha256:efd719c99d83b060d9daefdcf00360461adf279f45ef5391f8d111892118753e
commands:
- sh scripts/ci/promtool-rules.sh
@@ -341,3 +345,16 @@ steps:
image: gitea.renner.uno/gitea_admin/ci-k8s:2026-09-08
commands:
- sh scripts/ci/yaml-citations.sh
# Per-site values (${SITE_*}, substituted by Flux from site-settings) must
# render, and must not change what the homelab site renders today.
- name: site-render
depends_on: [] # see note above `steps:`
image: gitea.renner.uno/gitea_admin/ci-k8s:2026-09-08
commands:
- bash scripts/ci/test-site-render.sh
- |
if [ "$CI_PIPELINE_EVENT" = "pull_request" ]; then
git fetch --depth=1 origin "$CI_COMMIT_TARGET_BRANCH"
bash scripts/ci/site-render.sh --changed homelab FETCH_HEAD
fi
+6 -7
View File
@@ -35,35 +35,34 @@ when:
steps:
- name: model-selection-tests
depends_on: []
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.311
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.303
commands:
- sh scripts/ci/test-model-selection.sh
- name: pr-review-mistral-tests
depends_on: []
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.311
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.303
commands:
- sh scripts/ci/test-pr-review-mistral.sh
- name: promtool-rules-guard-tests
depends_on: []
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.311
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.303
commands:
- sh scripts/ci/test-promtool-rules.sh
- name: yaml-citations-guard-tests
depends_on: []
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.311
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.303
commands:
- sh scripts/ci/test-yaml-citations.sh
# Offline unittests for the Hermes skill scripts
# (kubernetes/apps/hermes/skills/): the [alert] title, noise-log and blip
# fingerprint contract the alert-monitor dedup depends on, the issue->card
# bridge, and the secret guard.
# fingerprint contract issue-fix depends on, and the secret guard.
- name: hermes-skill-tests
depends_on: []
image: python:3.15-alpine@sha256:d44b1254cb92329418d164b22b20d86bbd3869d48e80e09e476531a409a728bb
image: python:3.14-alpine@sha256:f6a589d43c42b9e7f7dc67a12d37132491f362859a5d750607710cc56da3bc72
environment:
PYTHONDONTWRITEBYTECODE: "1"
commands:
+33
View File
@@ -0,0 +1,33 @@
when:
# Deploy only on main (merge) + manual — never auto-deploy from a branch.
- event: push
branch: main
path:
- "ansible/roles/claude-workstation/**"
- "ansible/plays/install-claude-workstation.yml"
- ".woodpecker/deploy-claude-workstation.yml"
- event: deployment
evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "deploy-claude-workstation"'
timeout: 30
steps:
- name: deploy-claude-workstation
image: gitea.renner.uno/gitea_admin/ci-ansible:2026-04-29
environment:
ANSIBLE_FORCE_COLOR: "true"
ANSIBLE_HOST_KEY_CHECKING: "false"
ANSIBLE_ROLES_PATH: "../roles:./roles"
SSH_PRIVATE_KEY:
from_secret: ssh_private_key
VAULT_PASS:
from_secret: vault_pass
commands:
- mkdir -p ~/.ssh
- echo "$SSH_PRIVATE_KEY" | base64 -d > ~/.ssh/id_ed25519
- chmod 600 ~/.ssh/id_ed25519
- printf "Host *\n\tStrictHostKeyChecking no\n\tUserKnownHostsFile /dev/null\n" > ~/.ssh/config
- cd ansible
- echo "$VAULT_PASS" > .vault_pass && chmod 600 .vault_pass
- export ANSIBLE_VAULT_PASSWORD_FILE=.vault_pass
- ansible-playbook -i inventory/ plays/install-claude-workstation.yml -v
+36
View File
@@ -0,0 +1,36 @@
when:
# Deploy only on main (merge) + manual — never auto-deploy from a branch.
- event: push
branch: main
path:
- "ansible/roles/mqtt-bridge/**"
- "ansible/plays/edge-mqtt-bridge.yml"
- "ansible/inventory/group_vars/edge_host/**"
- ".woodpecker/deploy-edge-mqtt-bridge.yml"
- event: deployment
evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "deploy-edge-mqtt-bridge"'
timeout: 30
steps:
- name: deploy-edge-mqtt-bridge
image: gitea.renner.uno/gitea_admin/ci-ansible:2026-04-29
environment:
ANSIBLE_FORCE_COLOR: "true"
ANSIBLE_HOST_KEY_CHECKING: "false"
ANSIBLE_ROLES_PATH: "../roles:./roles"
SSH_PRIVATE_KEY:
from_secret: ssh_private_key
VAULT_PASS:
from_secret: vault_pass
commands:
- mkdir -p ~/.ssh
- echo "$SSH_PRIVATE_KEY" | base64 -d > ~/.ssh/id_ed25519
- chmod 600 ~/.ssh/id_ed25519
- printf "Host *\n\tStrictHostKeyChecking no\n\tUserKnownHostsFile /dev/null\n" > ~/.ssh/config
- cd ansible
- echo "$VAULT_PASS" > .vault_pass && chmod 600 .vault_pass
- export ANSIBLE_VAULT_PASSWORD_FILE=.vault_pass
# Local MQTT bridge on the edge hosts only (REN-1307); restarts only the
# local mosquitto on edge-1, never the cluster broker.
- ansible-playbook -i inventory/ plays/edge-mqtt-bridge.yml -v
+8 -1
View File
@@ -13,7 +13,7 @@ when:
steps:
- name: validate-secrets
image: alpine:3.24.3@sha256:be2656061b536c1883c4034790e21d0a09c5aaaf51968b89368a3c0371ea2c9a
image: alpine:3.24.2@sha256:294b683cb724975bec92580e1e685676bd4b50bda910ddb8c51d4cabeaec77e6
environment:
TF_VAR_proxmox_api_url:
from_secret: proxmox_api_url
@@ -109,6 +109,8 @@ steps:
from_secret: hetzner_s3_access_key
TF_VAR_hetzner_s3_secret_key:
from_secret: hetzner_s3_secret_key
SSH_PRIVATE_KEY:
from_secret: ssh_private_key
PROXMOX_TOKEN_ID:
from_secret: proxmox_token_id
PROXMOX_TOKEN_SECRET:
@@ -119,6 +121,11 @@ steps:
ANSIBLE_ROLES_PATH: "../roles:./roles"
ANSIBLE_FORCE_COLOR: "true"
commands:
- mkdir -p ~/.ssh
- echo "$SSH_PRIVATE_KEY" | base64 -d > ~/.ssh/id_ed25519
- chmod 600 ~/.ssh/id_ed25519
- export TF_VAR_ssh_private_key="$(cat ~/.ssh/id_ed25519)"
- printf "Host *\n\tStrictHostKeyChecking no\n" > ~/.ssh/config
- echo "$VAULT_PASS" > /tmp/.vault_pass && chmod 600 /tmp/.vault_pass
- export ANSIBLE_VAULT_PASSWORD_FILE=/tmp/.vault_pass
- tofu init -upgrade -backend-config="access_key=$AWS_ACCESS_KEY_ID" -backend-config="secret_key=$AWS_SECRET_ACCESS_KEY"
+35
View File
@@ -0,0 +1,35 @@
when:
# Deploy only on main (merge) + manual — never auto-deploy from a branch.
- event: push
branch: main
path:
- "ansible/roles/netbird-client/**"
- "ansible/plays/install-netbird-client.yml"
- "ansible/inventory/group_vars/claude_workstation/**"
- ".woodpecker/deploy-netbird-client.yml"
- event: deployment
evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "deploy-netbird-client"'
timeout: 30
steps:
- name: deploy-netbird-client
image: gitea.renner.uno/gitea_admin/ci-ansible:2026-04-29
environment:
ANSIBLE_FORCE_COLOR: "true"
ANSIBLE_HOST_KEY_CHECKING: "false"
ANSIBLE_ROLES_PATH: "../roles:./roles"
SSH_PRIVATE_KEY:
from_secret: ssh_private_key
VAULT_PASS:
from_secret: vault_pass
commands:
- mkdir -p ~/.ssh
- echo "$SSH_PRIVATE_KEY" | base64 -d > ~/.ssh/id_ed25519
- chmod 600 ~/.ssh/id_ed25519
- printf "Host *\n\tStrictHostKeyChecking no\n\tUserKnownHostsFile /dev/null\n" > ~/.ssh/config
- cd ansible
- echo "$VAULT_PASS" > .vault_pass && chmod 600 .vault_pass
- export ANSIBLE_VAULT_PASSWORD_FILE=.vault_pass
# Enrol only the off-site Claude workstation into the Netbird mesh.
- ansible-playbook -i inventory/ plays/install-netbird-client.yml --limit claude_workstation -v
+54
View File
@@ -0,0 +1,54 @@
when:
# Deploy only on main (merge) + manual — never auto-deploy from a branch.
- event: push
branch: main
path:
- "ansible/roles/netbird-proxy/**"
# install-netbird-proxy.yml also applies the netbird-proxy-watchdog role,
# so changes to it must redeploy this host too.
- "ansible/roles/netbird-proxy-watchdog/**"
# install-netbird-proxy.yml also applies the edge bouncer.
- "ansible/roles/crowdsec-edge-bouncer/**"
- "ansible/plays/install-netbird-proxy.yml"
# The variables that steer those roles, not just the roles themselves.
# Without these two, a change that only touches inventory never deploys
# and nothing says so: arming the bouncer lives in host_vars, and its
# LAPI key lives in group_vars, so both would sit in git looking applied
# while the hosts kept running the previous configuration.
#
# pve_hosts.yml is deliberately NOT listed. It is the single inventory
# file for the whole fleet, so listing it would redeploy the proxies for
# any unrelated host edit. The cost is that a change to [netbird_proxy]
# group membership needs a manual deployment run.
- "ansible/inventory/host_vars/proxy-*.yml"
- "ansible/inventory/group_vars/netbird_proxy/**"
- ".woodpecker/deploy-netbird-proxy.yml"
- event: deployment
evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "deploy-netbird-proxy"'
timeout: 30
steps:
- name: deploy-netbird-proxy
image: gitea.renner.uno/gitea_admin/ci-ansible:2026-04-29
environment:
ANSIBLE_FORCE_COLOR: "true"
ANSIBLE_HOST_KEY_CHECKING: "false"
ANSIBLE_ROLES_PATH: "../roles:./roles"
SSH_PRIVATE_KEY:
from_secret: ssh_private_key
VAULT_PASS:
from_secret: vault_pass
PROXMOX_TOKEN_ID:
from_secret: proxmox_token_id
PROXMOX_TOKEN_SECRET:
from_secret: proxmox_token_secret
commands:
- mkdir -p ~/.ssh
- echo "$SSH_PRIVATE_KEY" | base64 -d > ~/.ssh/id_ed25519
- chmod 600 ~/.ssh/id_ed25519
- printf "Host *\n\tStrictHostKeyChecking no\n\tUserKnownHostsFile /dev/null\n" > ~/.ssh/config
- cd ansible
- echo "$VAULT_PASS" > .vault_pass && chmod 600 .vault_pass
- export ANSIBLE_VAULT_PASSWORD_FILE=.vault_pass
- ansible-playbook -i inventory/ plays/install-netbird-proxy.yml -v
+37
View File
@@ -0,0 +1,37 @@
when:
# Deploy only on main (merge) + manual — never auto-deploy from a branch.
- event: push
branch: main
path:
- "ansible/roles/netbird-server/**"
- "ansible/plays/install-netbird-server.yml"
- ".woodpecker/deploy-netbird-server.yml"
- event: deployment
evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "deploy-netbird-server"'
timeout: 30
steps:
- name: deploy-netbird-server
image: gitea.renner.uno/gitea_admin/ci-ansible:2026-04-29
environment:
ANSIBLE_FORCE_COLOR: "true"
ANSIBLE_HOST_KEY_CHECKING: "false"
ANSIBLE_ROLES_PATH: "../roles:./roles"
SSH_PRIVATE_KEY:
from_secret: ssh_private_key
VAULT_PASS:
from_secret: vault_pass
PROXMOX_TOKEN_ID:
from_secret: proxmox_token_id
PROXMOX_TOKEN_SECRET:
from_secret: proxmox_token_secret
commands:
- mkdir -p ~/.ssh
- echo "$SSH_PRIVATE_KEY" | base64 -d > ~/.ssh/id_ed25519
- chmod 600 ~/.ssh/id_ed25519
- printf "Host *\n\tStrictHostKeyChecking no\n\tUserKnownHostsFile /dev/null\n" > ~/.ssh/config
- cd ansible
- echo "$VAULT_PASS" > .vault_pass && chmod 600 .vault_pass
- export ANSIBLE_VAULT_PASSWORD_FILE=.vault_pass
- ansible-playbook -i inventory/ plays/install-netbird-server.yml -v
+47
View File
@@ -0,0 +1,47 @@
when:
# Deploy only on main (merge) + manual — never auto-deploy from a branch.
- event: push
branch: main
path:
- "ansible/roles/uptime-kuma/**"
- "ansible/plays/install-uptime-kuma.yml"
- ".woodpecker/deploy-uptime-kuma.yml"
- event: deployment
evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "deploy-uptime-kuma"'
- event: cron
cron: uptime-kuma-token-renew
timeout: 30
steps:
- name: renew-openbao-token
image: gitea.renner.uno/gitea_admin/ci-ansible:2026-10-18
environment:
VAULT_TOKEN:
from_secret: openbao_uptime_kuma_token
commands:
- 'curl -sSf -X POST -H "X-Vault-Token: $VAULT_TOKEN" https://openbao.renner.uno/v1/auth/token/renew-self -o /dev/null'
- name: deploy-uptime-kuma
image: gitea.renner.uno/gitea_admin/ci-ansible:2026-10-18
when:
- event: [push, deployment]
environment:
ANSIBLE_FORCE_COLOR: "true"
ANSIBLE_HOST_KEY_CHECKING: "false"
ANSIBLE_ROLES_PATH: "../roles:./roles"
SSH_PRIVATE_KEY:
from_secret: ssh_private_key
VAULT_PASS:
from_secret: vault_pass
VAULT_TOKEN:
from_secret: openbao_uptime_kuma_token
commands:
- mkdir -p ~/.ssh
- echo "$SSH_PRIVATE_KEY" | base64 -d > ~/.ssh/id_ed25519
- chmod 600 ~/.ssh/id_ed25519
- printf "Host *\n\tStrictHostKeyChecking no\n\tUserKnownHostsFile /dev/null\n" > ~/.ssh/config
- cd ansible
- echo "$VAULT_PASS" > .vault_pass && chmod 600 .vault_pass
- export ANSIBLE_VAULT_PASSWORD_FILE=.vault_pass
- ansible-playbook -i inventory/ plays/install-uptime-kuma.yml -v
+3 -3
View File
@@ -4,8 +4,8 @@
# Triggers:
# - cron : scoped to the named daily/weekly crons below. This used to be
# a bare `event: cron` (run on EVERY cron of this repo), which
# was harmless at daily/weekly cadence but not with hourly
# crons, so piggybacking must be opt-in.
# was harmless at daily/weekly cadence — but issue-fix.yml
# fires hourly, so piggybacking must be opt-in now.
# A new cron that SHOULD also run e2e gets its name added here.
# - deploy : `woodpecker-cli pipeline deploy konairius/homelab <n> e2e`, where
# <n> is an existing pipeline to promote.
@@ -26,7 +26,7 @@ steps:
# fetch-models.sh response mapping. No cluster, no secrets, no network — the
# ci-claude image just provides sh + jq. Keeps the flag-injection regression
# test from rotting (it exercises the exact injection shapes).
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.311
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.303
commands:
- sh scripts/ci/test-model-selection.sh
+4 -3
View File
@@ -1,5 +1,5 @@
# Builds ci/images/hermes (nousresearch/hermes-agent plus the agent's CLIs) as
# gitea.renner.uno/konairius/hermes:YYYY-MM-DD-HHMM, one tag per build.
# gitea.renner.uno/konairius/hermes:build.<unix-ts> and :latest.
when:
- event: push
branch: main
@@ -20,9 +20,10 @@ steps:
- mkdir -p /kaniko/.docker
- AUTH=$(printf '%s:%s' registry-bot "$GITEA_TOKEN" | base64 | tr -d '\n')
- printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
- TAG=$(date +%Y-%m-%d-%H%M)
- TAG="build.$(date +%s)"
- |
/kaniko/executor \
--context . \
--dockerfile Containerfile \
--destination gitea.renner.uno/konairius/hermes:$TAG
--destination gitea.renner.uno/konairius/hermes:$TAG \
--destination gitea.renner.uno/konairius/hermes:latest
+218
View File
@@ -0,0 +1,218 @@
# Issue fixer — hourly, pick the oldest open "[alert] ..." Gitea issue (filed
# by the Hermes alert-monitor job, kubernetes/apps/hermes/skills/alert-monitor/)
# that nobody has worked on
# yet, hand it to an AI agent, and open a PR with the fix. One issue per run
# keeps runs bounded and PRs reviewable.
#
# "Not worked on yet" means BOTH:
# - no alert-fix/issue-<n> branch exists on origin (a branch means a PR was
# already opened — or is being opened by a concurrent run), and
# - no issue comment contains "<!-- alert-fix:assessed -->" (the marker this
# pipeline leaves when the agent concluded the problem is NOT fixable in
# the repo, so we don't burn an agent run on it every hour) — UNLESS a
# TRUSTED_AUTHORS comment is newer than the latest marker: an owner reply
# re-opens the issue for the agent, with the reply passed to it as
# pipeline-verified guidance (see below).
#
# Trusted owner guidance: comments by TRUSTED_AUTHORS are appended to the
# agent's issue.md under "Verified owner guidance". The trust anchor is the
# hardcoded author list in THIS file (reviewed via PR; agents cannot edit
# .woodpecker/). Deliberately NOT the Gitea permission API: this pipeline's
# own comments post as issue-fix-bot, which is itself a write collaborator,
# so "has write access" cannot distinguish the human from the pipeline — and
# an injected instruction inside an agent-authored comment must never be able
# to masquerade as owner guidance (the #259 lesson, in both directions: the
# agent rightly refused unauthenticated body text, but then had no channel
# for the owner's authentic reply).
#
# Tokens: a single dedicated identity, issue-fix-bot, for both the issue
# comments and the push/PR. Its PAT is scoped to write:issue +
# write:repository and nothing else. It deliberately does NOT reuse
# security_bot_token (that identity belongs to the security-review pipeline)
# nor the admin-owned gitea_token — see #330.
#
# Triggering:
# - cron : a cron NAMED exactly "issue-fix":
# woodpecker repo cron add --repo konairius/homelab \
# --name issue-fix --branch main --schedule '41 * * * *'
# - deploy : woodpecker-cli pipeline deploy konairius/homelab <n> issue-fix
# where <n> is an existing pipeline to promote.
when:
- event: cron
cron: issue-fix
- event: deployment
evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "issue-fix"'
steps:
- name: fetch-diag
# Least-privilege live-diagnostics kubeconfig (+ kubectl binary) for the
# fix agent (scripts/ci/make-diag-kubeconfig.sh). Runs unconditionally
# (the issue-pick logic needs jq, which only the ci-claude image has);
# minting a 1h read-only token per run is cheap. Degrades to
# snapshot-only if the ci-diagnostics ServiceAccount isn't reconciled.
image: gitea.renner.uno/gitea_admin/ci-k8s:2026-09-08
environment:
SOPS_AGE_KEY:
from_secret: sops_age_key
S3_ACCESS_KEY_ID:
from_secret: hetzner_s3_access_key
S3_SECRET_ACCESS_KEY:
from_secret: hetzner_s3_secret_key
commands:
- bash scripts/ci/fetch-kubeconfig.sh
- bash scripts/ci/make-diag-kubeconfig.sh
- name: issue-fix-agent
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.303
environment:
CLAUDE_CODE_OAUTH_TOKEN:
from_secret: claude_code_oauth_token
# Dedicated identity: issue-fix-bot, PAT scoped to write:issue +
# write:repository (it comments, pushes a branch, and opens a PR).
# One identity for both roles — the previous split borrowed
# security_bot_token, which belongs to the security-review pipeline
# and has no business in this trust domain (#330).
GITEA_TOKEN:
from_secret: issue_fix_bot_token
PUSH_TOKEN:
from_secret: issue_fix_bot_token
commands:
- |
# Pick the oldest unhandled [alert] issue. The repo is public, so the
# candidate reads are unauthenticated; only writes need tokens.
set -eu
TRUSTED_AUTHORS="k.renner" # space-separated; the trust anchor (see header)
mkdir -p /tmp/alert-fix
API="https://gitea.renner.uno/api/v1/repos/konairius/homelab"
curl -sfL "$$API/issues?state=open&type=issues&limit=100" \
| jq '[.[] | select(.title | startswith("[alert] "))] | sort_by(.number)' \
> /tmp/alert-fix/candidates.json
COUNT=$$(jq length /tmp/alert-fix/candidates.json)
echo "Open [alert] issues: $$COUNT"
TRUSTED_JSON=$$(printf '%s\n' $$TRUSTED_AUTHORS | jq -R . | jq -s -c .)
i=0
while [ "$$i" -lt "$$COUNT" ]; do
NUM=$$(jq -r ".[$$i].number" /tmp/alert-fix/candidates.json)
curl -sfL "$$API/issues/$$NUM/comments" > /tmp/alert-fix/comments.json
# Eligibility in jq (ISO timestamps compare lexically; shell test
# has no portable string-< in busybox ash): eligible unless an
# assessed marker exists with no NEWER trusted-author comment.
STATE=$$(jq -r --argjson t "$$TRUSTED_JSON" '
([.[] | select(.body | contains("alert-fix:assessed")) | .created_at] | max // "") as $$mark
| ([.[] | select(.user.login as $$u | $$t | index($$u)) | .created_at] | max // "") as $$trust
| if $$mark == "" then "fresh"
elif $$trust > $$mark then "retry"
else "skip" end' /tmp/alert-fix/comments.json)
if git ls-remote --heads origin "alert-fix/issue-$$NUM" | grep -q .; then
echo "#$$NUM: branch alert-fix/issue-$$NUM already exists — skipping."
elif [ "$$STATE" = "skip" ]; then
echo "#$$NUM: assessed and no newer owner comment — skipping."
else
if [ "$$STATE" = "retry" ]; then echo "#$$NUM: owner replied after assessment — retrying with guidance."; fi
echo "Picking issue #$$NUM."
jq ".[$$i]" /tmp/alert-fix/candidates.json > /tmp/alert-fix/issue.json
{
printf '# %s\n\n' "$$(jq -r '.title' /tmp/alert-fix/issue.json)"
printf 'Issue number: %s\n\n' "$$NUM"
jq -r '.body' /tmp/alert-fix/issue.json
# Owner comments, authorship checked against TRUSTED_AUTHORS
# above. Everything in the section is authentic owner input;
# agent-authored comments never match (they post as
# issue-fix-bot) and marker comments are excluded regardless.
if [ "$$(jq -r --argjson t "$$TRUSTED_JSON" '[.[] | select((.user.login as $$u | $$t | index($$u)) and (.body | contains("alert-fix:") | not))] | length' /tmp/alert-fix/comments.json)" -gt 0 ]; then
printf '\n\n## Verified owner guidance (authorship checked by the pipeline)\n'
jq -r --argjson t "$$TRUSTED_JSON" '.[] | select((.user.login as $$u | $$t | index($$u)) and (.body | contains("alert-fix:") | not)) | "\n### \(.user.login), \(.created_at)\n\n\(.body)"' /tmp/alert-fix/comments.json
fi
} > /tmp/alert-fix/issue.md
break
fi
i=$$((i+1))
done
[ -f /tmp/alert-fix/issue.json ] || echo "Nothing to fix — all [alert] issues handled."
- |
# Branch + git identity. The bot user owns the commits so PRs are
# visibly machine-authored, and the identity names THIS pipeline —
# previously these commits and PRs were attributed to security-bot,
# which made alert fixes look like security-review output (#330).
[ -f /tmp/alert-fix/issue.json ] || exit 0
set -eu
NUM=$$(jq -r '.number' /tmp/alert-fix/issue.json)
BRANCH="alert-fix/issue-$$NUM"
git config user.email "issue-fix-bot@renner.uno"
git config user.name "Alert Fix Agent"
git checkout -b "$$BRANCH"
echo "$$BRANCH" > /tmp/alert-fix/branch
- |
# Headless fix run. Prompt piped via stdin, subscription OAuth, file
# tools plus EXACTLY ONE live command: the RBAC-scoped read-only
# kubectl from fetch-diag (no Secrets readable, prefix match forbids
# pipes and other binaries). No web (see security-scan.yml).
[ -f /tmp/alert-fix/issue.json ] || exit 0
export KUBECONFIG="$$PWD/.ci-diag/kubeconfig"
# Model chosen live from the catalog (not hardcoded, no family assumed) —
# see the header of scripts/ci/pick-model.sh. Empty MODEL -> CLI default.
MODEL=$$(sh scripts/ci/pick-model.sh "diagnose an [alert] issue against read-only live cluster state and fix the root cause by editing repo manifests; multi-step reasoning that changes infrastructure code and needs a capable model")
if [ -n "$$MODEL" ]; then echo "issue-fix: selected model $$MODEL"; else echo "issue-fix: no live catalog — using the CLI default model"; fi
cat ci/agents/issue-fix/prompt.md | claude $${MODEL:+--model $$MODEL} \
--permission-mode acceptEdits \
--allowedTools "Read,Edit,Write,Glob,Grep,Bash(./.ci-diag/kubectl:*)" \
--settings ci/agents/issue-fix/settings.json \
-p
- |
# No edits → the agent judged this not fixable in the repo. Post its
# summary (with the assessed-marker) as an issue comment so the next
# run skips this issue, and stop cleanly.
[ -f /tmp/alert-fix/issue.json ] || exit 0
set -eu
API="https://gitea.renner.uno/api/v1/repos/konairius/homelab"
NUM=$$(jq -r '.number' /tmp/alert-fix/issue.json)
if git diff --quiet && git diff --cached --quiet \
&& [ -z "$$(git ls-files --others --exclude-standard)" ]; then
echo "Agent made no edits — commenting assessment on issue #$$NUM."
[ -s /tmp/alert-fix/summary.md ] \
|| echo "Agent made no edits and wrote no summary." > /tmp/alert-fix/summary.md
printf '\n\n<!-- alert-fix:assessed -->\n_Assessed by the issue-fix pipeline: no repo change proposed. Remove this comment to make the pipeline retry._\n' \
>> /tmp/alert-fix/summary.md
jq -n --rawfile body /tmp/alert-fix/summary.md '{body: $$body}' > /tmp/gitea-comment.json
code=$$(curl -s -o /tmp/gitea-resp.json -w '%{http_code}' \
-X POST -H "Authorization: token $$GITEA_TOKEN" \
-H "Content-Type: application/json" --data @/tmp/gitea-comment.json \
"$$API/issues/$$NUM/comments")
echo "POST comment -> HTTP $$code"
[ "$$code" -ge 200 ] && [ "$$code" -lt 300 ] || { cat /tmp/gitea-resp.json; exit 1; }
touch /tmp/alert-fix/no-edits
fi
- |
# Edits → commit, push, open the PR (Closes #<n> so the merge closes
# the issue), and link the PR from the issue.
[ -f /tmp/alert-fix/issue.json ] || exit 0
if [ -f /tmp/alert-fix/no-edits ]; then exit 0; fi
set -eu
API="https://gitea.renner.uno/api/v1/repos/konairius/homelab"
NUM=$$(jq -r '.number' /tmp/alert-fix/issue.json)
BRANCH=$$(cat /tmp/alert-fix/branch)
TITLE="fix: $$(jq -r '.title' /tmp/alert-fix/issue.json) (#$$NUM)"
[ -s /tmp/alert-fix/summary.md ] \
|| echo "Agent-proposed fix (no summary written by the agent)." > /tmp/alert-fix/summary.md
printf '\n\nCloses #%s\n' "$$NUM" >> /tmp/alert-fix/summary.md
git add -A
git commit -m "$$TITLE"
git push "https://$$PUSH_TOKEN@gitea.renner.uno/konairius/homelab.git" HEAD:"$$BRANCH"
jq -n --arg title "$$TITLE" --rawfile body /tmp/alert-fix/summary.md \
--arg head "$$BRANCH" \
'{title: $$title, body: $$body, head: $$head, base: "main"}' > /tmp/gitea-pr.json
code=$$(curl -s -o /tmp/gitea-resp.json -w '%{http_code}' \
-X POST -H "Authorization: token $$PUSH_TOKEN" \
-H "Content-Type: application/json" --data @/tmp/gitea-pr.json \
"$$API/pulls")
echo "POST pull -> HTTP $$code"
[ "$$code" -ge 200 ] && [ "$$code" -lt 300 ] || { cat /tmp/gitea-resp.json; exit 1; }
jq '{number, html_url, title}' /tmp/gitea-resp.json
PR_URL=$$(jq -r '.html_url' /tmp/gitea-resp.json)
jq -n --arg body "🤖 Opened $$PR_URL with a proposed fix." '{body: $$body}' > /tmp/gitea-comment.json
code=$$(curl -s -o /tmp/gitea-resp.json -w '%{http_code}' \
-X POST -H "Authorization: token $$GITEA_TOKEN" \
-H "Content-Type: application/json" --data @/tmp/gitea-comment.json \
"$$API/issues/$$NUM/comments")
echo "POST comment -> HTTP $$code"
[ "$$code" -ge 200 ] && [ "$$code" -lt 300 ] || { cat /tmp/gitea-resp.json; exit 1; }
+1 -1
View File
@@ -6,7 +6,7 @@ when:
steps:
- name: go-test
image: golang:1.27.3-bookworm@sha256:972b4110ba0f47539a975c31e52c64765e1ff19f2d74469ae4a595c6676e80a5
image: golang:1.27.2-bookworm@sha256:5cf287a799e6b94384bad13d16b14904c531f51ba65792237e122ce42b392f61
commands:
- test -z "$(gofmt -l review-bot)"
- go -C review-bot vet ./...
+3 -3
View File
@@ -18,12 +18,12 @@ when:
steps:
- name: iac-scan
image: aquasec/trivy:latest@sha256:790fd66d1697395d8693d1acaa1ea702406a20f109b6fc7bac8db623eca6c06c
image: aquasec/trivy:latest@sha256:2416d1f74045b57274df1a340eee6b510cacd73c66a6ccb8da092b117eb7d222
commands:
- trivy fs --format json -o trivy-iac.json --severity CRITICAL,HIGH --ignore-unfixed .
- name: image-scans
image: aquasec/trivy:latest@sha256:790fd66d1697395d8693d1acaa1ea702406a20f109b6fc7bac8db623eca6c06c
image: aquasec/trivy:latest@sha256:2416d1f74045b57274df1a340eee6b510cacd73c66a6ccb8da092b117eb7d222
commands:
- trivy image --download-db-only
- |
@@ -70,7 +70,7 @@ steps:
# this step is skipped — otherwise we'd fork-bomb PRs.
when:
- branch: main
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.311
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.303
environment:
CLAUDE_CODE_OAUTH_TOKEN:
from_secret: claude_code_oauth_token
+1 -1
View File
@@ -30,7 +30,7 @@ when:
steps:
- name: update-audit-agent
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.311
image: gitea.renner.uno/gitea_admin/ci-claude:2.1.303
environment:
CLAUDE_CODE_OAUTH_TOKEN:
from_secret: claude_code_oauth_token
+5 -10
View File
@@ -112,7 +112,7 @@
# (string) This setting controls the default policy of fact gathering (facts discovered about remote systems).
# This option can be useful for those wishing to save fact gathering time. Both 'smart' and 'explicit' will use the cache plugin.
gathering=smart
;gathering=implicit
# (list) Set the `gather_subset` option for the :ref:`ansible_collections.ansible.builtin.setup_module` task in the implicit fact gathering. See the module documentation for specifics.
# It does **not** apply to user defined ``ansible.builtin.setup`` tasks.
@@ -134,7 +134,7 @@ gathering=smart
;hash_behaviour=replace
# (pathlist) Comma separated list of Ansible inventory sources
inventory=inventory
inventory=~/source/homelab/ansible/inventory
# (pathspec) Colon separated paths in which Ansible will search for HttpApi Plugins.
;httpapi_plugins={{ ANSIBLE_HOME ~ "/plugins/httpapi:/usr/share/ansible/plugins/httpapi" }}
@@ -206,7 +206,7 @@ inventory=inventory
;poll_interval=15
# (path) Option for connections using a certificate or key file to authenticate, rather than an agent or passwords, you can set the default value here to avoid re-specifying --private-key with every invocation.
;private_key_file=
private_key_file=~/.ssh/id_ed25519
# (boolean) By default, imported roles publish their variables to the play and other roles, this setting can avoid that.
# This was introduced as a way to reset role variables to default values if a role is used more than once in a playbook.
@@ -274,7 +274,7 @@ roles_path=../roles:./roles
# (path) The vault password file to use. Equivalent to --vault-password-file or --vault-id
# If executable, it will be run and the resulting stdout will be used as the password.
vault_password_file=vault-password.sh
vault_password_file=~/source/homelab/ansible/vault-password.sh
# (integer) Sets the default verbosity, equivalent to the number of ``-v`` passed in the command line.
;verbosity=0
@@ -484,7 +484,7 @@ vault_password_file=vault-password.sh
# It can result in a very significant performance improvement when enabled.
# However this conflicts with privilege escalation (become). For example, when using 'sudo:' operations you must first disable 'requiretty' in /etc/sudoers on all managed hosts, which is why it is disabled by default.
# This setting will be disabled if ``ANSIBLE_KEEP_REMOTE_FILES`` is enabled.
pipelining=True
;pipelining=False
[colors]
@@ -694,8 +694,3 @@ pipelining=True
# (list) default list of tags to skip in your plays, has precedence over Run Tags
;skip=
[ssh_connection]
# Connections stay open across the plays of a run, so each host costs one
# YubiKey PIN; plays/close-connections.yml closes them at the end of a run.
ssh_args=-C -o ControlMaster=auto -o ControlPersist=30m
+21 -17
View File
@@ -41,11 +41,13 @@ Build and boot the dedicated installer image — see
4. Add the host to `[edge_host]` in `inventory/pve_hosts.yml`, by name.
That group is the base: every edge thin client belongs to it, and it
assumes no peripherals.
5. From the workstation, in `ansible/`:
5. From the workstation:
```
ansible-playbook site.yml
ansible-playbook -i inventory/pve_hosts.yml plays/install-edge-host.yml
```
One unrestricted run, no `--limit`. It converges every host, which is
One unrestricted run over the whole group. The play is self-contained --
it carries `certificates` and `common` itself, so there is no second
command and no `--limit`. Running it converges every edge host, which is
the point: a run that touches only the new box tells you nothing about
the others.
@@ -70,14 +72,14 @@ none of these commands.
## Secrets (before first play run)
Create `inventory/group_vars/edge_host/vault.yml` with ansible-vault. The
cluster broker has no `open3e` user: edge-1 logs in only as the register
account `edge-1` (see "MQTT bridge" below), so
`vault_edge_open3e_mqtt_password` reaches only the loopback broker, which
allows anonymous clients; any random value works (`openssl rand -hex 24`).
The zwave-js value comes out of the cluster:
Create `inventory/group_vars/edge_host/vault.yml` with ansible-vault; the
current live values come straight out of the cluster:
```
# sml2mqtt broker password
kubectl -n home-assistant get secret mosquitto-sml2mqtt-credentials -o jsonpath='{.data.password}' | base64 -d
# open3e broker password
kubectl -n home-assistant get secret mosquitto-open3e-credentials -o jsonpath='{.data.password}' | base64 -d
# zwave-js session secret
kubectl -n home-assistant get secret zwave-js-session -o jsonpath='{.data.session-secret}' | base64 -d
```
@@ -228,14 +230,15 @@ They connect to a **local mosquitto on loopback** (`127.0.0.1:1883`,
broker is briefly unreachable, and flushes on reconnect — edge-1 no longer
hard-depends on cluster/network uptime for every publish.
The local broker is `allow_anonymous` on loopback — the trust boundary is the
host; nothing off-host can reach it.
Two bridge `connection`s preserve the existing cluster identities, so **no
cluster-side change was needed**: one authenticates as the `open3e` user
(carries `open3e/#` out + `open3e/cmnd` back), one as `sml2mqtt` (`sml2mqtt/#`
out). The local broker is `allow_anonymous` on loopback — the trust boundary
is the host; nothing off-host can reach it.
**Bridge credential (REN-1307).** One bridge `connection` logs in as the
register account `edge-1` (`kubernetes/apps/home-assistant/mqtt-accounts.yaml`)
and carries `open3e/#` out and `open3e/cmnd` back. The cluster broker does not
accept the vault-password bridges in `bridge.d/legacy.conf`, which exists only
until the first push. Its password is pushed
**Bridge credential (REN-1307).** The two vault-password bridges are being
replaced by one bridge as the register account `edge-1`
(`kubernetes/apps/home-assistant/mqtt-accounts.yaml`). Its password is pushed
by mqtt-account-controller to `edge-mqtt-cred` (`:8443`, the controller's
esp32 API over HTTPS). That endpoint accepts only a client certificate from
the MQTT CA with CN `mqtt-account-controller`; edge-1 holds the public CA and
@@ -243,7 +246,8 @@ its own server key, nothing else. On a push it writes
`/etc/mosquitto/bridge.d/edge-1.conf`, deletes `bridge.d/legacy.conf` and
restarts the local mosquitto. The controller sets a new password whenever the
bridge reports "not connected" (also after a reinstall), so no play run is
needed for a rotation. Diagnosis: `journalctl -u
needed for a rotation. `.woodpecker/deploy-edge-mqtt-bridge.yml` runs
`plays/edge-mqtt-bridge.yml` on merge. Diagnosis: `journalctl -u
edge-mqtt-cred`, and `mosquitto_sub -t '$SYS/broker/connection/edge-1/state'`
(retained `1` = connected).
+67
View File
@@ -0,0 +1,67 @@
#!/usr/bin/env python3
"""
Discover Shelly devices on the network via HTTP API
Returns JSON array of discovered devices
"""
import sys
import json
import ipaddress
import concurrent.futures
import urllib.request
import urllib.error
def check_shelly(ip):
"""Check if IP is a Shelly device and return info"""
try:
url = f"http://{ip}/shelly"
req = urllib.request.Request(url, headers={'User-Agent': 'Ansible-Shelly-Discovery'})
with urllib.request.urlopen(req, timeout=2) as response:
data = json.loads(response.read().decode())
# Gen1 devices don't have 'gen' field, default to 1
gen = data.get('gen', 1)
if gen is None:
gen = 1
return {
'ip': str(ip),
'type': data.get('type', 'unknown'),
'mac': data.get('mac', ''),
'gen': gen,
'name': data.get('name', data.get('id', str(ip))),
'fw_id': data.get('fw_id', data.get('fw', 'unknown'))
}
except (urllib.error.URLError, urllib.error.HTTPError, OSError, json.JSONDecodeError):
return None
except Exception:
return None
def main():
if len(sys.argv) < 2:
print("Usage: discover_shelly.py <network1> [network2] ...")
print("Example: discover_shelly.py 10.2.30.0/24 10.2.20.0/24")
sys.exit(1)
networks = sys.argv[1:]
devices = []
# Collect all IPs to scan
all_ips = []
for network in networks:
try:
all_ips.extend(ipaddress.IPv4Network(network).hosts())
except ValueError as e:
print(f"Invalid network: {network} - {e}", file=sys.stderr)
continue
# Scan in parallel
with concurrent.futures.ThreadPoolExecutor(max_workers=100) as executor:
results = executor.map(check_shelly, all_ips)
for result in results:
if result:
devices.append(result)
# Output JSON
print(json.dumps(devices, indent=2))
if __name__ == "__main__":
main()
+24 -7
View File
@@ -5,10 +5,27 @@ pve_token_id: "ansible"
ansible_user: "ladmin"
ansible_python_interpreter: /usr/bin/python3
# The only plain key the admin logins accept on [ssh_keys_managed] hosts
# (ssh-ca role, tasks/authorized_keys.yml). People log in with OpenBao SSH CA
# certificates; this one is for when that path is unavailable.
# breakglass-neo: PIV key (slot 9a, PIN) on the offline YubiKey NEO.
# ssh -I /usr/lib/opensc-pkcs11.so root@<host>
ssh_admin_authorized_keys:
- key: "ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKvSl9LUP8u3S2Sq5YTWpW0Jxbue8cJhD7/LMFQbEsa3xo145HVR7pBzRxW71MUy+jgrgmpSlDhIb606Hswhvoo= breakglass-neo-6532348"
# SSH Public Keys for Cloud-Init
# ansible-runner@ci-new removed (issue #184): dead key, no such host exists.
common_ssh_public_keys: |
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPzFNBP3bq2m/6JqDqod9w+FP/uxXxHMolqhq3Ojflr1 krenner@fortress
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICyNnX1I4W9jyDvCJ1DJauHSmbo1ZXSyBxVhSAwMWgen woodpecker-ci@homelab
# Network-of-origin restriction for the woodpecker-ci@homelab CI key
# (issue #184). Consumed by plays/configure-ssh-keys.yml as the authorized_key
# `from=` option so the CI key only authenticates from within the cluster's
# pod network — NOT from any host-pinned IP.
#
# The in-cluster Woodpecker step pods SSH to fleet hosts using their POD IP as
# source, not a node IP: Cilium runs `ipv4-native-routing-cidr=10.0.0.0/8` +
# `ipv6-native-routing-cidr=fd00::/8` with masquerade enabled, so pod egress to
# any 10.x / fd.. LAN destination is NOT SNAT'd, and each node's PodCIDR is
# advertised to the UDM over BGP (kubernetes/infrastructure/post-crds/
# cilium-bgp.yaml) so return traffic L3-routes back to the pod. Hosts therefore
# see the real pod source IP.
#
# The cluster pod subnets (kubernetes/talos/patches/cluster.yaml podSubnets)
# are the stable boundary — they survive node DHCP-lease churn and pod
# rescheduling across k8s-1/2/3, unlike pinning any node's current IP.
# v6 is included because some SSH targets (edge IoT hosts) are IPv6-only.
ci_ssh_from_cidrs: '10.244.0.0/16,fd10:244::/48'
@@ -0,0 +1,16 @@
$ANSIBLE_VAULT;1.1;AES256
35646538363164363061366433343963623166336630376233333033646637336266343337343230
3233356135323762636439336466663430366238363338650a633762623366643137303333383634
62623739363132303335636261626263333332623066303161633466306431623930666166323833
6439663964353338350a656435626664646666363831363164393335386666356637613037323838
63653461656366613331353064656161346263356131346437666632386166623037373363333534
37323435623366643534633333303534313434613135653237303966656162633266623433363934
36343133393834393765353338383764303664363431653865633764306138376366623832346262
36303766623331666632613639366139333661636665363631326565653938343637323861653064
37393465383263376337376639613162306563653365623036376662303030613264653534353332
66363030356363373439306164386561363734353631363738656236633164633062613437663366
62386430396262366336316166353366383233316333333234666538643962336361363630363762
31653963303032373531386566336632336130383034343430303964396538663237346434613737
39643765623766373639306132316635376434643232336163303066393736313130323065666232
39393038616162666666383837326137376233613437636361636238663566336365653863643933
353338646134383038323134323034653564
@@ -50,11 +50,3 @@ edge_mqtt_host: 127.0.0.1
edge_mqtt_port: 1883
edge_open3e_mqtt_user: open3e
edge_open3e_mqtt_topic: open3e
# Units claude-bot may restart (ssh-ca role); the containers go through docker.
ssh_ca_claude_sudo_units:
- mosquitto
- edge-mqtt-cred
- edge-alloy
- keepalived
- chronyd
@@ -1,9 +0,0 @@
---
# The host NetBird client (wt0) must not take over DNS: resolv.conf is pinned to
# the systemd-resolved stub by the netbird-proxy role.
netbird_disable_dns: true
ssh_ca_claude_sudo_units:
- netbird
- netbird-proxy-flood-watchdog
- crowdsec-firewall-bouncer
@@ -1,8 +0,0 @@
---
# DSM's own python3 is 3.8; plays/configure-synology-python.yml installs this.
ansible_python_interpreter: /var/packages/Python3.9/target/usr/bin/python3.9
# DSM offers no sftp/scp.
ansible_ssh_transfer_method: piped
# claude-bot is a DSM administrator here; its password is unknown.
ssh_ca_claude_sudo_all: true
@@ -2,6 +2,3 @@
# NetBird server - VPS not managed by Proxmox
# Enable Docker monitoring (cAdvisor) for container metrics
docker_monitoring_enabled: true
ssh_ca_claude_sudo_units:
- netbird
+12 -35
View File
@@ -8,14 +8,16 @@
# pve_cluster (jupiter/neptune/saturn) run Talos on bare metal; no SSH at all
# authentik_db / authentik_app authentik runs in-cluster (CNPG)
# grafana runs in-cluster
# synology_nas (nas-1) powered off; Garage and the PBS client
# it hosted are gone with it
# Their hosts no longer resolve, except the three that resolve to Talos nodes.
[netbird_server]
netbird.renner.uno ansible_host=netbird.nb.renner.uno ansible_user=root
netbird.renner.uno ansible_user=root
[netbird_proxy]
proxy-1.renner.uno ansible_host=proxy-1.nb.renner.uno ansible_user=root
proxy-2.renner.uno ansible_host=proxy-2.nb.renner.uno ansible_user=root
proxy-1.renner.uno ansible_user=root
proxy-2.renner.uno ansible_user=root
# Every bare-metal edge thin client. Gets the host-generic roles only:
# flash-wear handling, the log shipper, the MQTT bridge, the VLAN
@@ -23,11 +25,6 @@ proxy-2.renner.uno ansible_host=proxy-2.nb.renner.uno ansible_user=root
[edge_host]
edge-1.iot.renner.uno ansible_user=root
# Synology NAS (DSM): nas-1 RS1221+ on site, nas-2 DS1522+ off site.
[synology_nas]
nas-1.renner.uno ansible_user=root
nas-2.renner.uno ansible_user=root
# Capability groups. Membership states which hardware a host physically has,
# and each service playbook targets the group for its own concern. No flags and
# no `when:` on the roles -- moving a stick to another box is a one-line change
@@ -65,30 +62,10 @@ heating_can
[ntp_cluster:children]
edge_host
# Hosts that trust the OpenBao SSH user CA (plays/configure-ssh-ca.yml).
# Rolled out one host at a time: add the next host after the previous one
# was verified.
[ssh_ca]
proxy-1.renner.uno
proxy-2.renner.uno
netbird.renner.uno
edge-1.iot.renner.uno
nas-1.renner.uno
nas-2.renner.uno
# Hosts whose admin logins accept only ssh_admin_authorized_keys (break-glass);
# everything else needs a CA certificate. Members must be in [ssh_ca].
# Added one host at a time after the [ssh_ca] report was reviewed.
[ssh_keys_managed]
proxy-1.renner.uno
proxy-2.renner.uno
netbird.renner.uno
edge-1.iot.renner.uno
# Internet hosts whose sshd listens on the NetBird interface only
# (plays/configure-ssh-mesh-only.yml). Members need the host NetBird client.
# Added one host at a time after the previous one was verified.
[ssh_mesh_only]
proxy-1.renner.uno
proxy-2.renner.uno
netbird.renner.uno
# Off-site Debian VM (Synology VMM on nas-2, 10.3.40.0/24) running Claude Code.
# Reached over the site-to-site tunnel at its current DHCP lease. Switch
# ansible_host to the stable Netbird mesh IP once the host is enrolled — the
# obg.renner.uno FQDN is split-horizon and resolves to the public proxy from
# home, so it must NOT be used as ansible_host.
[claude_workstation]
claude-ws ansible_host=10.3.40.78 ansible_user=root
-11
View File
@@ -1,11 +0,0 @@
---
# Last play of every playbook that is run directly (site.yml and the one-off
# actions). SSH connections stay open for 30 minutes (ansible.cfg) so a run
# costs one YubiKey PIN per host; closing them at the end leaves nothing that
# another process of the same user could reuse.
- name: Close the SSH connections
hosts: all
gather_facts: false
tasks:
- name: Close the connection
ansible.builtin.meta: reset_connection
+1 -2
View File
@@ -13,5 +13,4 @@
roles:
- common
- role: qemu-guest-agent
when: ansible_facts['virtualization_role'] == 'guest'
- qemu-guest-agent
+125
View File
@@ -0,0 +1,125 @@
---
# Configure NTP server on all Shelly devices
# Discovers Shelly devices via network scan and configures them to use local NTP server
#
# Usage:
# ansible-playbook plays/configure-shelly-ntp.yml
# ansible-playbook plays/configure-shelly-ntp.yml -e "shelly_ntp_server=10.2.40.4"
# ansible-playbook plays/configure-shelly-ntp.yml -e "shelly_networks=10.2.30.0/24,10.2.20.0/24"
- name: Configure Shelly NTP Settings
hosts: localhost
connection: local
gather_facts: no
vars:
shelly_ntp_server: "ntp.renner.uno"
shelly_networks: "10.2.30.0/24 10.2.20.0/24"
shelly_timeout: 5
tasks:
- name: Discover Shelly devices on network
script: "{{ playbook_dir }}/../files/discover_shelly.py {{ shelly_networks }}"
args:
executable: python3
register: discovered_devices
changed_when: false
- name: Parse discovered devices
set_fact:
shelly_devices: "{{ discovered_devices.stdout | from_json }}"
- name: Display discovered Shelly devices
debug:
msg: "Found {{ shelly_devices | length }} Shelly devices"
- name: Show device list
debug:
msg: "{{ item.name }} ({{ item.type }}) - {{ item.ip }} - Gen{{ item.gen }}"
loop: "{{ shelly_devices }}"
when: shelly_devices | length > 0
- name: Configure NTP on Gen1 devices
uri:
url: "http://{{ item.ip }}/settings?sntp_server={{ shelly_ntp_server }}"
method: GET
timeout: "{{ shelly_timeout }}"
return_content: yes
loop: "{{ shelly_devices | selectattr('gen', 'equalto', 1) | list }}"
register: gen1_results
ignore_errors: yes
when: shelly_devices | length > 0
- name: Configure NTP on Gen2+ devices
uri:
url: "http://{{ item.ip }}/rpc/Sys.SetConfig"
method: POST
body_format: json
body:
config:
sntp:
server: "{{ shelly_ntp_server }}"
timeout: "{{ shelly_timeout }}"
return_content: yes
loop: "{{ shelly_devices | selectattr('gen', 'greaterthan', 1) | list }}"
register: gen2_results
ignore_errors: yes
when: shelly_devices | length > 0
- name: Verify Gen1 NTP configuration
uri:
url: "http://{{ item.ip }}/settings"
method: GET
timeout: "{{ shelly_timeout }}"
return_content: yes
loop: "{{ shelly_devices | selectattr('gen', 'equalto', 1) | list }}"
register: gen1_verify
ignore_errors: yes
when: shelly_devices | length > 0
- name: Verify Gen2+ NTP configuration
uri:
url: "http://{{ item.ip }}/rpc/Sys.GetConfig"
method: GET
timeout: "{{ shelly_timeout }}"
return_content: yes
loop: "{{ shelly_devices | selectattr('gen', 'greaterthan', 1) | list }}"
register: gen2_verify
ignore_errors: yes
when: shelly_devices | length > 0
- name: Summary - Gen1 devices configured
debug:
msg: "{{ item.item.name }} ({{ item.item.ip }}): NTP={{ item.json.sntp.server | default('unknown') }}"
loop: "{{ gen1_verify.results | default([]) }}"
when:
- gen1_verify is defined
- gen1_verify.results is defined
- item.json is defined
- item.json.sntp is defined
loop_control:
label: "{{ item.item.name | default(item.item.ip) }}"
- name: Summary - Gen2+ devices configured
debug:
msg: "{{ item.item.name }} ({{ item.item.ip }}): NTP={{ item.json.sntp.server | default('unknown') }}"
loop: "{{ gen2_verify.results | default([]) }}"
when:
- gen2_verify is defined
- gen2_verify.results is defined
- item.json is defined
- item.json.sntp is defined
loop_control:
label: "{{ item.item.name | default(item.item.ip) }}"
- name: Final summary
debug:
msg: |
==========================================
Shelly NTP Configuration Complete
==========================================
Total devices found: {{ shelly_devices | length }}
Gen1 devices: {{ shelly_devices | selectattr('gen', 'equalto', 1) | list | length }}
Gen2+ devices: {{ shelly_devices | selectattr('gen', 'greaterthan', 1) | list | length }}
NTP Server: {{ shelly_ntp_server }}
==========================================
-23
View File
@@ -1,23 +0,0 @@
---
# Trust the OpenBao SSH user CA (ssh-client-signer) on the hosts in [ssh_ca]:
# owner certificates (principal k.renner) for root/ladmin, Claude's 1h
# certificates (principal claude-bot) for its own claude-bot account with
# limited sudo. The role reports the admin logins' authorized_keys; on
# [ssh_keys_managed] they are then reduced to ssh_admin_authorized_keys.
#
# serial: 1 so a host whose sshd rejects the change stops the run before the
# next host is touched. One play, so each host needs one login (one YubiKey
# PIN from a workstation).
- name: Configure the SSH user CA
hosts: ssh_ca
become: true
gather_facts: true
serial: 1
roles:
- ssh-ca
tasks:
- name: Reduce authorized_keys to the break-glass key
ansible.builtin.include_role:
name: ssh-ca
tasks_from: authorized_keys
when: inventory_hostname in groups['ssh_keys_managed']
+48
View File
@@ -0,0 +1,48 @@
---
- name: Deploy authorized SSH keys to all hosts
hosts: all:!localhost
become: true
gather_facts: false
# Authorized keys deployed to root and ladmin on every managed host.
#
# - krenner@fortress (human): unrestricted.
# - woodpecker-ci@homelab (CI): restricted with from= to the cluster's pod
# network (ci_ssh_from_cidrs in group_vars/all/vars.yml). The in-cluster
# Woodpecker step pods SSH to fleet hosts presenting their POD IP as source
# (Cilium native-routing-cidr covers the LAN, so no SNAT; PodCIDRs are
# BGP-advertised). This is a network-of-origin restriction, NOT host-pinned,
# so it survives node DHCP churn and pod rescheduling. See vars.yml for the
# full derivation.
# - ansible-runner@ci-new: removed — dead key, no such host exists.
vars:
authorized_keys:
- key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPzFNBP3bq2m/6JqDqod9w+FP/uxXxHMolqhq3Ojflr1 krenner@fortress"
- key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICyNnX1I4W9jyDvCJ1DJauHSmbo1ZXSyBxVhSAwMWgen woodpecker-ci@homelab"
options: 'from="{{ ci_ssh_from_cidrs }}"'
tasks:
- name: Check SSH connectivity
ansible.builtin.wait_for_connection:
timeout: 5
ignore_errors: true
register: ssh_check
- name: Skip unreachable hosts
ansible.builtin.meta: end_host
when: ssh_check is failed
- name: Deploy authorized keys to root
ansible.builtin.authorized_key:
user: root
key: "{{ item.key }}"
key_options: "{{ item.options | default(omit) }}"
state: present
loop: "{{ authorized_keys }}"
- name: Deploy authorized keys to ladmin
ansible.builtin.authorized_key:
user: ladmin
key: "{{ item.key }}"
key_options: "{{ item.options | default(omit) }}"
state: present
loop: "{{ authorized_keys }}"
ignore_errors: true
-15
View File
@@ -1,15 +0,0 @@
---
# sshd on the hosts in [ssh_mesh_only] accepts connections only on the NetBird
# interface; nothing listens on their public addresses. Run from a workstation
# in the mesh. A switchover reverts by itself unless the play logs in again
# through the new listener.
#
# serial: 1 so a host that does not come back stops the run before the next
# host is touched.
- name: Restrict sshd to the mesh interface
hosts: ssh_mesh_only
become: true
gather_facts: true
serial: 1
roles:
- ssh-mesh-only
@@ -1,19 +0,0 @@
---
# Ansible needs Python >= 3.9 on the target; DSM ships 3.8. Synology's
# Python3.9 package installs beside it. raw, because the modules cannot run
# before it is there; raw is skipped in check mode.
- name: Install Python 3.9 on the Synology NAS
hosts: synology_nas
become: true
gather_facts: false
tasks:
- name: Install the Python3.9 package
ansible.builtin.raw: >-
if [ -x {{ ansible_python_interpreter }} ]; then echo present;
else /usr/syno/bin/synopkg install_from_server Python3.9; fi
register: synology_python_install
changed_when: "'present' not in synology_python_install.stdout"
- name: Check the interpreter exists
ansible.builtin.raw: test -x {{ ansible_python_interpreter }}
changed_when: false
-2
View File
@@ -29,7 +29,6 @@
- name: Check if systemd-timesyncd is installed
command: systemctl list-unit-files systemd-timesyncd.service
register: timesyncd_installed
check_mode: false
failed_when: false
changed_when: false
@@ -57,7 +56,6 @@
- name: Verify time sync status
command: timedatectl show --property=NTPSynchronized
register: timesync_status
check_mode: false
changed_when: false
failed_when: false
+33
View File
@@ -0,0 +1,33 @@
---
# Local MQTT bridge and its credential receiver only (REN-1307). Run by
# .woodpecker/deploy-edge-mqtt-bridge.yml on merge, so a change to this role
# reaches edge-1 without a play run by hand. The full base play
# (install-edge-host.yml) carries the same role.
- name: Converge the edge MQTT bridge
hosts: edge_host
become: true
# The SML power-meter reader (sml2mqtt) is retired for good: owner decision
# 2026-09-19, grid data comes from the E380 on the CAN bus. Its role and
# play are gone; this removes the leftover compose project from edge-1 on
# the next pipeline run. Idempotent: a no-op once the directory is gone.
pre_tasks:
- name: Check for the retired sml2mqtt compose project
ansible.builtin.stat:
path: /opt/edge-host/sml2mqtt/docker-compose.yml
register: edge_sml2mqtt_compose
- name: Stop the retired sml2mqtt container
ansible.builtin.command:
cmd: docker compose down --remove-orphans
chdir: /opt/edge-host/sml2mqtt
when: edge_sml2mqtt_compose.stat.exists
changed_when: true
- name: Remove the retired sml2mqtt directory
ansible.builtin.file:
path: /opt/edge-host/sml2mqtt
state: absent
roles:
- mqtt-bridge
@@ -0,0 +1,20 @@
---
# Off-site Debian VM running Claude Code — reattachable interactive sessions
# plus an unattended notes-distill timer. Host creation (Synology VMM), the
# Netbird dashboard route and secret material are manual prerequisites; see
# roles/claude-workstation/README.md and docs/claude-workstation.md.
#
# CI reaches this off-site host over the site-to-site tunnel (home -> off-site
# works), same as it reaches every other inventory host.
- name: Deploy the Claude Code workstation
hosts: claude_workstation
become: true
# qemu-guest-agent is applied
# to every inventory host by the umbrella plays/common.yml, same as for the
# netbird-proxy host — run that against this host too (see RUNBOOK Step 5).
roles:
- common
- role: netbird-client # mesh enrolment (needs netbird_setup_key in vault)
when: netbird_setup_key is defined
- claude-workstation
+11 -10
View File
@@ -6,25 +6,26 @@
# - host_vars/<hostname>/vars.yml - per-host key (overrides group)
# - group_vars/all/vault.yml - encrypted keys
#
# Applied by site.yml.
# Usage:
# ansible-playbook -i inventory/ plays/setup-netbird-client.yml
#
# Limit to specific hosts:
# ansible-playbook -i inventory/ plays/setup-netbird-client.yml --limit caddy_servers
- name: Install NetBird client on hosts with setup key defined
hosts: all
become: true
# One host at a time: from a workstation every new SSH session asks for the
# YubiKey PIN and a touch, and parallel hosts would prompt at once.
serial: 1
# Facts only after the skip: gathering them first opens an SSH session (a
# YubiKey touch from a workstation) to every host, including skipped ones.
gather_facts: false
gather_facts: true
pre_tasks:
- name: Check if netbird_setup_key is defined
ansible.builtin.debug:
msg: "NetBird setup key is defined for {{ inventory_hostname }}"
when: netbird_setup_key is defined
- name: Skip hosts without netbird_setup_key
ansible.builtin.meta: end_host
when: netbird_setup_key is not defined
- name: Gather facts
ansible.builtin.setup:
roles:
- netbird-client
+1 -1
View File
@@ -1,6 +1,6 @@
---
# Deploy the NetBird reverse-proxy HA cluster members (the public edge).
# Applied by site.yml.
# Usage: ansible-playbook -i inventory/ plays/install-netbird-proxy.yml
#
# Prerequisites:
# - Proxy VPS provisioned (proxy-1 is Arch Linux), root SSH reachable, and
+1 -1
View File
@@ -1,6 +1,6 @@
---
# Deploy NetBird Server to VPS
# Applied by site.yml.
# Usage: ansible-playbook -i inventory/ plays/install-netbird-server.yml
#
# Prerequisites:
# - VPS provisioned with Ubuntu 22.04/24.04
+22
View File
@@ -0,0 +1,22 @@
---
# Install Node Exporter on all managed hosts
# Usage: ansible-playbook -i inventory/ plays/install-node-exporter.yml
- name: Install Node Exporter
hosts: all
become: yes
gather_facts: yes
roles:
- node-exporter
post_tasks:
- name: Verify node-exporter is running
uri:
url: "http://127.0.0.1:9100/metrics"
status_code: 200
register: exporter_health
retries: 3
delay: 5
until: exporter_health.status == 200
ignore_errors: yes
+20 -44
View File
@@ -102,45 +102,32 @@
ansible.builtin.pause:
seconds: 5
- name: Verify chronyd answers
ansible.builtin.command:
argv:
- chronyc
- -n
- tracking
check_mode: false
- name: Verify chrony is serving NTP
ansible.builtin.shell: chronyc clients | head -10
register: chrony_clients
changed_when: false
- name: Verify chronyd listens on udp/123
ansible.builtin.command:
argv:
- ss
- -Hlunp
- sport = :123
- name: Check NTP port is listening
ansible.builtin.shell: ss -lun | grep ':123 '
register: ntp_port
check_mode: false
failed_when: false
changed_when: false
failed_when: "'chronyd' not in ntp_port.stdout"
- name: Display NTP port status
ansible.builtin.debug:
msg: "NTP port 123: {{ 'LISTENING' if ntp_port.rc == 0 else 'NOT LISTENING' }}"
- name: Check if this server has the VIP
ansible.builtin.command:
argv:
- ip
- -4
- addr
- show
- to
- "{{ ntp_vip }}"
ansible.builtin.shell: ip addr show | grep {{ ntp_vip }}
register: vip_check
check_mode: false
failed_when: vip_check.rc != 0
failed_when: false
changed_when: false
when: ntp_keepalived_enabled | default(false)
- name: Display VIP status
ansible.builtin.debug:
msg: >
{% if vip_check.stdout | default('') | length > 0 %}
{% if vip_check.rc == 0 %}
This server ({{ inventory_hostname }}) is MASTER and holds the VIP {{ ntp_vip }}
{% else %}
This server ({{ inventory_hostname }}) is BACKUP
@@ -172,8 +159,7 @@
- to
- "{{ ntp_vip6 }}"
register: vip6_check
check_mode: false
failed_when: vip6_check.rc != 0
failed_when: false
changed_when: false
when: ntp_keepalived_enabled | default(false)
@@ -198,28 +184,18 @@
(keepalived not enabled on this host).
when: not (ntp_keepalived_enabled | default(false))
# chronyc talks to the command port, which is loopback-only; this is a real
# NTP query. Directives on the command line mean no config file and no
# pidfile, and -Q opens no server or command port, so it runs alongside
# the live daemon.
- name: Query NTP through the VIP (only on master)
ansible.builtin.command:
argv:
- chronyd
- -Q
- -t
- "10"
- "server {{ ntp_vip }} iburst maxsamples 1"
- name: Test NTP via VIP (only on master)
ansible.builtin.shell: |
chronyc -h {{ ntp_vip }} tracking 2>&1 || echo "Cannot connect to VIP (may need to wait for failover)"
register: ntp_vip_test
check_mode: false
when:
- ntp_keepalived_enabled | default(false)
- vip_check.stdout | default('') | length > 0
- vip_check.rc == 0
changed_when: false
- name: Display VIP NTP test result
ansible.builtin.debug:
msg: "{{ ntp_vip_test.stderr_lines + ntp_vip_test.stdout_lines }}"
msg: "{{ ntp_vip_test.stdout_lines | default(['Not master, skipping VIP test']) }}"
when:
- ntp_keepalived_enabled | default(false)
- vip_check.stdout | default('') | length > 0
- vip_check.rc == 0
+2 -3
View File
@@ -1,14 +1,13 @@
---
# Deploy Uptime Kuma to the NetBird VPS
# Applied by site.yml.
# Usage: ansible-playbook -i inventory/ plays/install-uptime-kuma.yml
#
# Prerequisites:
# - netbird-server play applied (docker, Traefik on netbird_netbird)
# - OpenBao homelab/infrastructure/uptime-kuma with keys username, password
# - OpenBao homelab/infrastructure/uptime-kuma/smtp with keys server, port, username, token
# - homelab/infrastructure/uptime-kuma-metrics (key api_key) is written by the role
# - an OpenBao token with policy uptime-kuma-vps: `bao login -method=oidc`
# (role owner) or VAULT_TOKEN
# - VAULT_TOKEN in the environment (policy: uptime-kuma-vps)
# - DNS for kuma.renner.uno pointing at the VPS
- name: Deploy Uptime Kuma
-3
View File
@@ -248,6 +248,3 @@
- "Commit inventory/group_vars/all/netbird-rotatable.yml — it is encrypted."
- "NOT rotated: datastore_key (needs dump/restore), auth_client_secret (Authentik side)."
- "OpenBao is unaffected by design — its NetBird credentials are separate ones."
- name: Close the SSH connections
ansible.builtin.import_playbook: close-connections.yml
+1 -3
View File
@@ -43,6 +43,7 @@
- ubuntu_upgrade.rc != 0
- "'No new release found' not in ubuntu_upgrade.stdout"
- "'No new release found' not in ubuntu_upgrade.stderr"
ignore_errors: true
- name: Display Upgrade Result
ansible.builtin.debug:
@@ -110,6 +111,3 @@
when:
- (ansible_distribution == 'Ubuntu' and reboot_required.stat.exists) or
(ansible_distribution == 'Debian')
- name: Close the SSH connections
ansible.builtin.import_playbook: close-connections.yml
-33
View File
@@ -1,33 +0,0 @@
---
# Package upgrade + reboot for netbird_proxy, edge_host and netbird_server,
# one host at a time.
# Usage (from ansible/): ansible-playbook plays/update-os.yml
#
# Plays run in order; a failed host stops the whole run (max_fail_percentage: 0).
- name: Update NetBird proxies
hosts: netbird_proxy
serial: 1
max_fail_percentage: 0
become: true
roles:
- os-update
- name: Update edge hosts
hosts: edge_host
serial: 1
max_fail_percentage: 0
become: true
roles:
- os-update
- name: Update NetBird server
hosts: netbird_server
serial: 1
max_fail_percentage: 0
become: true
roles:
- os-update
- name: Close the SSH connections
ansible.builtin.import_playbook: close-connections.yml
@@ -18,7 +18,6 @@
- name: Get installed version
command: "{{ chrony_exporter_binary }} --version"
register: installed_version
check_mode: false
when: chrony_exporter_installed.stat.exists
changed_when: false
failed_when: false
@@ -0,0 +1,57 @@
# claude-workstation
An always-on, **off-site** headless Claude Code host (Debian 12 VM under Synology
VMM on nas-2). It runs (a) reattachable interactive `tmux` sessions and (b) an
unattended notes-distill timer. It deliberately holds the **least** privilege
that still does the job.
## What lives here vs. what's manual
This role configures everything **inside** the VM. Three things are **manual**
(no IaC path) and precede/accompany it — see [`RUNBOOK.md`](RUNBOOK.md) for the
full step-by-step:
1. **VM creation** in Synology VMM (`ladmin` has no sudo; the DSM API is
root-only), with a cloud-init drive that installs the CI SSH key.
2. **Netbird dashboard route** advertising the k8s API to the mesh, and the
**site-to-site return route** on the off-site gateway (backup path).
3. **Secret material**: `claude setup-token` (subscription OAuth), the claude-bot
creds, and the read-only SA kubeconfig — encrypted into the group vault.
## Credential isolation
Two Unix users with separate credential stores so an unattended job cannot read
the interactive session's secrets:
| User | Holds | Runs |
|------|-------|------|
| `claude` | OAuth token, claude-bot creds, **read-only** SA kubeconfig | interactive tmux |
| `claude-job` | its **own** OAuth token, claude-bot creds (no kubeconfig) | notes-distill timer |
**No talosconfig, no SOPS age key** ever land here (Fable review #1). The
kubeconfig is a read-only `view` ServiceAccount (`kubernetes/apps/claude-workstation/`).
## Egress lockdown (staged)
`claude_ws_egress_enforce` ships **`false`** so a fresh host is never bricked.
The deny-default nftables ruleset, the DNS-resolver, and its refresh timer are
installed regardless. Flip to `true` (in host/group vars) only after confirming
`claude_ws_egress_allow_fqdns` / `claude_ws_egress_allow_cidrs*` cover real
traffic — watch the `claude-egress-dropped` counter first.
## Vault keys (encrypt into `inventory/group_vars/claude_workstation/vault.yml`)
```
netbird_setup_key # from the Netbird dashboard
claude_ws_oauth_token_interactive # `claude setup-token` (interactive user)
claude_ws_oauth_token_job # `claude setup-token` (job user, separate)
claude_ws_gitea_token # claude-bot PAT
claude_ws_gitea_git_credentials # https://claude-bot:<pat>@gitea.renner.uno
claude_ws_kubeconfig # read-only SA kubeconfig (literal YAML)
```
## Kill switch
Everything unattended hangs off systemd timers; `systemctl disable --now
claude-notes-distill.timer` (and the egress refresh) stops it. Every credential
is independently revocable (SA token, claude-bot PAT, OAuth token, Netbird peer).
+222
View File
@@ -0,0 +1,222 @@
# Runbook — off-site Claude Code workstation (nas-2)
Operator guide for the manual steps around the `claude-workstation` role. The
role automates everything **inside** the VM; this covers what it can't:
VM creation, networking, secret material, egress cutover, rotation and DR.
## Architecture (scope)
An always-on **Debian 13 VM under Synology VMM on the off-site nas-2**
(DS1522+, 10.3.40.0/24, reached over the site-to-site VPN, ~55 ms). It runs:
- **Reattachable interactive sessions** (`tmux` over SSH as user `claude`).
- **Unattended notes-distill timer** (as user `claude-job`).
- **A thin is-home-up signal** (the existing external Gatus on the Netbird VPS
already covers this — don't duplicate it).
**Out of scope (runs in-cluster instead):** scheduled repo-automation and
cluster-reports go in as Woodpecker cron / CronJobs with an in-cluster
ServiceAccount — *not* on this VM, so the VM holds no cluster-write creds.
Credentials on the VM are minimal: a **read-only** `view` SA kubeconfig,
claude-bot (non-admin) gitea creds, and a Claude subscription OAuth token.
**No talosconfig, no SOPS age key.**
---
## Step 1 — Create + install the VM (Synology VMM)
`ladmin` has no sudo and the DSM API is root-only, so VMM is driven through the
DSM **web UI**. Two hard facts dictate the method (all confirmed 2026-07-16, as
built): **VMM has no cloud-init**, and its ISO-install wizard only creates blank
disks. So the OS is installed from a **remastered Debian netinst ISO that
auto-boots a preseed** — no console interaction, no boot-menu race.
VMM gotchas that shape this (see the wiki build-record):
- VMM refuses to change/unmount a VM's ISO while it is **running** ("Failed to
replace ISO files"). Do all ISO/boot-device edits with the VM **off**.
- A **guest reboot does not re-read** VMM's boot-device/ISO config — only a full
VMM power-cycle applies it. So an auto-booting install ISO **loops** (reboot →
CD → reinstall, wiping the disk). Break it by preseeding the installer to
**power off** at the end: `d-i debian-installer/exit/poweroff boolean true`.
- Attach the NIC to the **untagged `Default VM Network`** — the VLAN-40
`40_Server` network is isolated (no DHCP, no network). The LAN has DHCP.
Procedure (build the ISO on a host with `xorriso`):
1. Download the current Debian **netinst** (`.../debian-cd/current/amd64/iso-cd/`).
2. Remaster it — add `/preseed.cfg` and replace `/isolinux/isolinux.cfg` with a
minimal one (`default preseed`, `timeout 1`, one label:
`kernel /install.amd/vmlinuz` /
`append auto=true priority=critical preseed/file=/cdrom/preseed.cfg vga=788 initrd=/install.amd/initrd.gz --- quiet`):
`xorriso -indev in.iso -outdev out.iso -boot_image any replay -map preseed.cfg /preseed.cfg -map isolinux.cfg /isolinux/isolinux.cfg`.
The preseed uses **DHCP** (`netcfg/dhcp_timeout 120` — the virtual-switch port
is slow to link), disk **`/dev/sda`** (VirtIO SCSI), **no user** + a locked
random root hash (key-only via `PermitRootLogin prohibit-password`), packages
`openssh-server sudo qemu-guest-agent python3`, and a `late_command` that
injects the SSH keys (`krenner@fortress`, `woodpecker-ci@homelab`, your
workstation key) into `/root/.ssh/authorized_keys`, plus the `poweroff` line.
3. Upload to nas-2 via `ssh 'cat > /volume1/Install/<iso>' < <iso>` — **scp /
sftp / rsync are blocked on DSM**. Then VMM → Image → Add → From Synology NAS
→ import (VMM **copies** it; re-import if you change the source file).
4. Create VM: 2 vCPU, 4 GB, 50 GB on `/volume1`, **Default VM Network**, Legacy
BIOS, **Autostart: Yes**; attach the ISO, **Boot from DVD/CD-ROM**, power on.
5. It installs unattended and **powers off**. Then (VM off) set **ISO =
Unmounted**, **Boot from = Virtual Disk**, power on → boots the installed
Debian. `qemu-guest-agent` reports the DHCP lease in VMM's IP column.
6. **Pin a DSM update window** (Control Panel → Update & Restore) so an
auto-reboot doesn't kill tmux/jobs.
> As built 2026-07-16: Debian 13 (trixie), got DHCP `10.3.40.78`, all three keys
> injected, packages present. Freeing RAM required powering off the pre-existing
> `pbs1` PBS VM (kept its 3.4 TB backup disk).
> **At-rest encryption (Fable #6).** Off-site physical custody is weaker than
> home. For true at-rest protection of the VM's secrets *with* unattended
> autostart, use **network-bound decryption** (clevis + a Tang server at home):
> a stolen disk off-site can't unlock because it can't reach Tang. Full-disk
> LUKS with a manual passphrase would break autostart. Until Tang is stood up,
> secrets rely on 0600 perms + VM access control — documented gap, not a fix.
---
## Step 2 — Networking (two paths)
**Netbird mesh (done via CI):** mint a setup key in the dashboard
(`https://netbird.renner.uno`), vault it as `netbird_setup_key` under
`group_vars/claude_workstation/`, and run `.woodpecker/deploy-netbird-client.yml`
(`install-netbird-client.yml --limit claude_workstation`). As built 2026-07-16
the VM enrolled as `claude-ws-190-112.nb.renner.uno` / `100.96.190.112` (+ IPv6).
> **Mesh reachability caveat.** The mesh IP (`100.96.x`) is only reachable from
> **other Netbird peers**. The home workstation and the Woodpecker CI agent are
> **not** peers — they reach the VM only via the site-to-site tunnel at its DHCP
> lease. So a *stable* mesh-based `ansible_host` needs CI (or a home host) on the
> mesh first, or a Netbird routing peer bridging tunnel↔mesh. Until then the
> inventory targets the tunnel DHCP IP.
> **DNS trap.** `claude-ws.obg.renner.uno` resolves to the VM's lease **only at
> the off-site DNS** (`@10.3.40.1`). The home resolver catch-alls `*.renner.uno`
> to the public proxy (`162.55.50.97`, the Netbird VPS) — so from home the FQDN
> lands on `root@netbird`, **not** the VM. Never use it as `ansible_host`.
**Optional — mesh route to the k8s API (redundant read path):** to reach the API
over the mesh too, add a Netbird dashboard **network route** advertising the
apiserver's ClusterIP to the `netbird-ingress` peer group, and add an egress
rule to the `netbird-ingress-egress` CiliumNetworkPolicy
(`kubernetes/infrastructure/network-policies/ingress-isolation.yaml`) permitting
`app: netbird-ingress` pods to reach `kube-apiserver:443`. Deferred from this PR
because it edits a security-critical CNP that can't be validated offline — do it
as a follow-up when you want the redundant API path.
---
## Step 3 — Read-only SA kubeconfig
After the `kubernetes/apps/claude-workstation/` PR merges and Flux reconciles:
```sh
NS=apps
TOKEN=$(kubectl -n $NS get secret claude-workstation-token -o jsonpath='{.data.token}' | base64 -d)
kubectl -n $NS get secret claude-workstation-token -o jsonpath='{.data.ca\.crt}' | base64 -d > /tmp/ca.crt
# server = the cluster's stable API endpoint (kubernetes/talos/patches/cluster.yaml).
# Use the name, not an address: it is what the API server certificate carries as a
# SAN, so an IP here fails TLS verification even when it is reachable. It must
# resolve from the workstation over the site-to-site route.
kubectl config set-cluster homelab --server=https://k8s.renner.uno:6443 \
--certificate-authority=/tmp/ca.crt --embed-certs=true --kubeconfig=/tmp/claude-ws.kubeconfig
kubectl config set-credentials claude-workstation --token="$TOKEN" --kubeconfig=/tmp/claude-ws.kubeconfig
kubectl config set-context homelab --cluster=homelab --user=claude-workstation --kubeconfig=/tmp/claude-ws.kubeconfig
kubectl config use-context homelab --kubeconfig=/tmp/claude-ws.kubeconfig
# verify read-only:
KUBECONFIG=/tmp/claude-ws.kubeconfig kubectl get nodes # works
KUBECONFIG=/tmp/claude-ws.kubeconfig kubectl get secrets -A # Forbidden (view excludes secrets)
```
The resulting file becomes the `claude_ws_kubeconfig` vault value.
---
## Step 4 — Secret material → vault
```sh
# Two independent OAuth tokens so job jobs can't impersonate the interactive user:
claude setup-token # -> claude_ws_oauth_token_interactive
claude setup-token # -> claude_ws_oauth_token_job (run again / separate session)
```
Encrypt into `ansible/inventory/group_vars/claude_workstation/vault.yml`
(ansible-vault; same VAULT_PASS the CI uses):
```yaml
netbird_setup_key: "..."
claude_ws_oauth_token_interactive: "..."
claude_ws_oauth_token_job: "..."
claude_ws_gitea_token: "..." # claude-bot PAT
claude_ws_gitea_git_credentials: "https://claude-bot:<PAT>@gitea.renner.uno"
claude_ws_kubeconfig: |
apiVersion: v1
... # the file from Step 3
```
> **Never run `claude` with `--bare`** here or on the VM — bare mode silently
> ignores `CLAUDE_CODE_OAUTH_TOKEN` and the jobs will fail auth.
---
## Step 5 — Deploy
Merge the PR, then trigger the pipeline (Woodpecker → Run → `PIPELINE ==
"deploy-claude-workstation"`), or push a change under the role's paths. CI SSHes
to `10.3.40.60` over the tunnel and applies `common`, `netbird-client`, and this
role. Also run the umbrella `plays/common.yml` against the new host once, so it
picks up `qemu-guest-agent` like every other inventory host.
---
## Step 6 — Egress cutover (staged)
The nftables allowlist ships **disabled**. To enforce:
1. Leave it off for a few days; watch what the box actually talks to.
2. Confirm `claude_ws_egress_allow_fqdns` + `claude_ws_egress_allow_cidrs*`
(home Prometheus/Loki/API) cover it.
3. Set `claude_ws_egress_enforce: true` in host/group vars, redeploy.
4. Watch the `claude-egress-dropped` nft counter (`nft list ruleset`) and Loki
for anything legitimate being dropped; widen the allowlist as needed.
---
## Step 7 — Notes (Syncthing pairing)
The role installs Syncthing for `claude-job` and starts
`syncthing@claude-job`. Pair it once (manual): from the VM's GUI (SSH tunnel to
`127.0.0.1:8384`) add your existing vault's device ID and accept the shared
`Notes` folder pointed at `{{ claude_ws_notes_dir }}`. The distill timer runs
daily at 04:30 and only ever **creates** new atomic notes.
---
## Rotation
- **OAuth token** (expires/can be revoked): re-run `claude setup-token` on your
laptop, update the two vault keys, redeploy. Headless auth failures surface in
Loki (the wrapper logs `result=failure`) — don't let them go quiet.
- **claude-bot PAT**: rotate in Gitea, update `claude_ws_gitea_*`, redeploy.
- **SA token**: `kubectl -n apps delete secret claude-workstation-token` (Flux
recreates it), re-assemble the kubeconfig (Step 3), update vault, redeploy.
## Kill switch
```sh
systemctl disable --now claude-notes-distill.timer claude-egress-refresh.timer
```
Every credential is independently revocable: delete the SA, revoke the PAT,
revoke the OAuth token, or remove the Netbird peer — no shared blast radius.
## Disaster recovery (cattle, not pet)
The VM holds no unique state — the only real state is the Notes replica, which
re-syncs from Syncthing. To rebuild: redo **Step 1** (VM + cloud-init), then
**Step 5** (pipeline). No VMM snapshots needed (they'd bloat the 86%-full
`/volume1`).
@@ -0,0 +1,68 @@
---
# claude-workstation role defaults
#
# An always-on headless host that runs Claude Code for (a) reattachable
# interactive sessions and (b) unattended notes/wiki timers. It deliberately
# holds the LEAST privilege that still lets it do its job: a read-only k8s
# ServiceAccount kubeconfig, claude-bot (non-admin) gitea creds, and a Claude
# subscription OAuth token — NO talosconfig and NO SOPS age key. See README.
# --- Users -----------------------------------------------------------------
# Two Unix users with separate credential stores so an unattended (timer) job
# cannot read the interactive session's OAuth token / kube / gitea creds.
claude_ws_interactive_user: claude # human tmux sessions
claude_ws_job_user: claude-job # unattended systemd-timer jobs
claude_ws_base_dir: /opt/claude-workstation
# --- Toolchain -------------------------------------------------------------
claude_ws_node_major: 22 # matches the ci-claude image
claude_ws_claude_code_pkg: "@anthropic-ai/claude-code"
claude_ws_base_packages:
- git
- tmux
- jq
- curl
- ca-certificates
- ripgrep
- unzip
- nftables
- python3
- python3-venv
# --- Egress allowlist ------------------------------------------------------
# Deny-default OUTPUT is the target end-state, but it can brick a fresh host if
# an allowed destination is missed, so it ships DISABLED and is tightened after
# observing real egress (staged rollout). Flip claude_ws_egress_enforce to true
# once the allowlist below is confirmed complete. See templates/nftables.conf.j2.
claude_ws_egress_enforce: false
# Static destinations the host must always reach. CDN-backed hosts (Anthropic,
# npm, apt) are matched by name at apply time via the fqdn set; home services
# are matched by IP. Tune in host_vars.
claude_ws_egress_allow_fqdns:
- api.anthropic.com
- statsig.anthropic.com
- gitea.renner.uno
- fsn1.your-objectstorage.com # Hetzner Object Storage (SOPS bundle)
- deb.debian.org
- security.debian.org
- deb.nodesource.com
- pkgs.netbird.io
- registry.npmjs.org
claude_ws_egress_allow_cidrs: [] # home Prometheus/Loki/API (v4), per-host
claude_ws_egress_allow_cidrs6: [] # same (v6) — IPv6-first, prefer these
# --- Notes / wiki pipeline -------------------------------------------------
# The Obsidian vault syncs onto this host via Syncthing (matches the user's
# existing multi-device setup). Unattended pipeline jobs must CREATE new atomic
# files rather than rewrite existing ones, to keep Syncthing conflicts rare.
claude_ws_notes_enabled: true
claude_ws_notes_dir: "/home/{{ claude_ws_job_user }}/Notes"
claude_ws_syncthing_gui_bind: "127.0.0.1:8384"
# --- Secrets (populated from the ansible-vault'd group_vars — see README) ---
# claude_ws_oauth_token_interactive : CLAUDE_CODE_OAUTH_TOKEN for the human user
# claude_ws_oauth_token_job : CLAUDE_CODE_OAUTH_TOKEN for timer jobs
# claude_ws_gitea_token : claude-bot PAT
# claude_ws_gitea_git_credentials : https://claude-bot:<pat>@gitea.renner.uno line
# claude_ws_kubeconfig : read-only SA kubeconfig (base64 or literal)
@@ -0,0 +1,22 @@
You are running unattended as a scheduled job on the claude-workstation host.
Your job is a conservative distillation pass over the Obsidian wiki vault.
Scope and rules (follow exactly — you have only Read, Write, Glob, Grep):
1. Look at `~/Notes/raw/` (ignore `~/Notes/raw/_processed/` and `~/Notes/_archive/`
entirely) for source material.
2. For each raw source, check whether `~/Notes/wiki/` already contains a note
grounded in it (Grep the wiki for the source filename / its `sources:` slug).
If a note already exists, SKIP that source — do not rewrite or duplicate it.
3. For raw sources with NO corresponding wiki note yet, create new atomic
concept notes in `~/Notes/wiki/`: one concept per file, self-contained,
tagged, cross-linked with `[[wikilinks]]`, every claim grounded in a
`sources:` entry. Never invent facts not present in the source.
4. CREATE new files only. Never rewrite, move, delete, or truncate an existing
file in `~/Notes/` — appending a missing cross-link to a note you created
THIS run is fine; touching pre-existing notes is not.
5. If there is nothing new to distill, do nothing and stop. Do not manufacture
work.
Preserve disagreement: if sources conflict, surface the conflict in the note
rather than silently picking a winner. Keep each note short and single-purpose.
@@ -0,0 +1,8 @@
# claude-workstation shared tmux config — reattachable long-running sessions.
set -g history-limit 100000
set -g mouse on
set -g base-index 1
setw -g pane-base-index 1
set -g status-right "#H %Y-%m-%d %H:%M"
# Keep sessions alive across client disconnects (the whole point of this host).
set -g destroy-unattached off
@@ -0,0 +1,10 @@
---
- name: Reload systemd
ansible.builtin.systemd:
daemon_reload: true
- name: Reload nftables
ansible.builtin.systemd:
name: nftables.service
state: reloaded
when: claude_ws_egress_enforce | bool
@@ -0,0 +1,42 @@
---
# Deny-default egress allowlist. Ships DISABLED (claude_ws_egress_enforce:
# false) so a fresh host is never bricked by a missed destination; the ruleset,
# the FQDN-resolver, and its refresh timer are installed regardless, and the
# operator flips the toggle to true once egress is confirmed complete against
# real traffic (staged rollout). CDN-backed hosts are matched via nft named
# sets that the refresh unit repopulates from DNS; static home services are
# matched by claude_ws_egress_allow_cidrs.
- name: Deploy the egress-refresh resolver
ansible.builtin.template:
src: claude-egress-refresh.sh.j2
dest: "{{ claude_ws_base_dir }}/claude-egress-refresh.sh"
mode: "0755"
- name: Deploy the egress-refresh service + timer
ansible.builtin.template:
src: "{{ item }}.j2"
dest: "/etc/systemd/system/{{ item }}"
mode: "0644"
loop:
- claude-egress-refresh.service
- claude-egress-refresh.timer
notify: Reload systemd
- name: Deploy the nftables ruleset
ansible.builtin.template:
src: nftables.conf.j2
dest: /etc/nftables.conf
mode: "0644"
validate: "nft -c -f %s"
notify: Reload nftables
- name: Enforce or relax egress lockdown
ansible.builtin.systemd:
name: "{{ item }}"
enabled: "{{ claude_ws_egress_enforce | bool }}"
state: "{{ 'started' if (claude_ws_egress_enforce | bool) else 'stopped' }}"
daemon_reload: true
loop:
- nftables.service
- claude-egress-refresh.timer
@@ -0,0 +1,183 @@
---
# claude-workstation: an always-on headless Claude Code host.
# Debian 12 VM on the off-site nas-2 (Synology VMM). See README for the manual
# bootstrap (VM creation, Netbird route, secret material) that precedes this.
#
# common.yml already applies `common` and `qemu-guest-agent`
# to every inventory host, so this role owns only the
# Claude-specific surface: toolchain, the two credential-isolated users, the
# read-only creds, notes sync, the unattended timers, and egress lockdown.
- name: Assert we are on Debian (role is apt-only)
ansible.builtin.assert:
that:
- ansible_facts['os_family'] == 'Debian'
fail_msg: "claude-workstation targets Debian; got {{ ansible_facts['os_family'] }}."
# --- Base packages ---------------------------------------------------------
- name: Install base packages
ansible.builtin.apt:
name: "{{ claude_ws_base_packages }}"
state: present
update_cache: true
# --- Node.js 22 + Claude Code (matches the ci-claude image toolchain) -------
- name: Add NodeSource signing key
ansible.builtin.get_url:
url: https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key
dest: /usr/share/keyrings/nodesource.gpg
mode: "0644"
force: false
- name: Add NodeSource apt repository
ansible.builtin.apt_repository:
repo: "deb [signed-by=/usr/share/keyrings/nodesource.gpg] https://deb.nodesource.com/node_{{ claude_ws_node_major }}.x nodistro main"
state: present
filename: nodesource
- name: Install Node.js
ansible.builtin.apt:
name: nodejs
state: present
update_cache: true
# NodeSource installs Node under /usr, so global npm modules land in
# /usr/lib/node_modules and `claude` is on PATH for every user/timer. `creates`
# keeps it idempotent (re-run `npm install -g` manually to bump the version).
- name: Install Claude Code globally
ansible.builtin.command:
cmd: "npm install -g {{ claude_ws_claude_code_pkg }}"
creates: /usr/lib/node_modules/@anthropic-ai/claude-code
# --- Credential-isolated users --------------------------------------------
- name: Create Claude users
ansible.builtin.user:
name: "{{ item }}"
shell: /bin/bash
create_home: true
loop:
- "{{ claude_ws_interactive_user }}"
- "{{ claude_ws_job_user }}"
- name: Create the role base directory (scripts, prompts)
ansible.builtin.file:
path: "{{ claude_ws_base_dir }}"
state: directory
mode: "0755"
# Let the interactive user's tmux server survive SSH logout / reboot.
- name: Enable lingering for the interactive user
ansible.builtin.command: "loginctl enable-linger {{ claude_ws_interactive_user }}"
args:
creates: "/var/lib/systemd/linger/{{ claude_ws_interactive_user }}"
- name: Deploy shared tmux config
ansible.builtin.copy:
src: tmux.conf
dest: "/home/{{ item }}/.tmux.conf"
owner: "{{ item }}"
group: "{{ item }}"
mode: "0644"
loop:
- "{{ claude_ws_interactive_user }}"
- "{{ claude_ws_job_user }}"
# --- Secrets: each user gets only what its role needs ----------------------
# The interactive user gets gitea + OAuth + read-only kubeconfig. The job user
# gets its OWN OAuth token + gitea (to open PRs) but NO kubeconfig.
- name: Create ~/.config/claude-bot for the interactive user
ansible.builtin.file:
path: "/home/{{ item }}/.config/claude-bot"
state: directory
owner: "{{ item }}"
group: "{{ item }}"
mode: "0700"
loop:
- "{{ claude_ws_interactive_user }}"
- "{{ claude_ws_job_user }}"
- name: Deploy claude-bot gitea token
ansible.builtin.copy:
content: "{{ claude_ws_gitea_token }}\n"
dest: "/home/{{ item }}/.config/claude-bot/gitea-token"
owner: "{{ item }}"
group: "{{ item }}"
mode: "0600"
loop:
- "{{ claude_ws_interactive_user }}"
- "{{ claude_ws_job_user }}"
no_log: true
- name: Deploy claude-bot git-credentials
ansible.builtin.copy:
content: "{{ claude_ws_gitea_git_credentials }}\n"
dest: "/home/{{ item }}/.config/claude-bot/git-credentials"
owner: "{{ item }}"
group: "{{ item }}"
mode: "0600"
loop:
- "{{ claude_ws_interactive_user }}"
- "{{ claude_ws_job_user }}"
no_log: true
# CLAUDE_CODE_OAUTH_TOKEN is delivered as an EnvironmentFile so both interactive
# shells and timer units can source it without it landing in shell history.
# NEVER run claude with --bare here: bare mode silently ignores this token.
- name: Deploy per-user Claude OAuth environment file
ansible.builtin.copy:
content: "CLAUDE_CODE_OAUTH_TOKEN={{ item.token }}\n"
dest: "/home/{{ item.user }}/.config/claude-code.env"
owner: "{{ item.user }}"
group: "{{ item.user }}"
mode: "0600"
loop:
- { user: "{{ claude_ws_interactive_user }}", token: "{{ claude_ws_oauth_token_interactive }}" }
- { user: "{{ claude_ws_job_user }}", token: "{{ claude_ws_oauth_token_job }}" }
loop_control:
label: "{{ item.user }}"
no_log: true
- name: Create ~/.kube for the interactive user
ansible.builtin.file:
path: "/home/{{ claude_ws_interactive_user }}/.kube"
state: directory
owner: "{{ claude_ws_interactive_user }}"
group: "{{ claude_ws_interactive_user }}"
mode: "0700"
- name: Deploy read-only SA kubeconfig (interactive user only)
ansible.builtin.copy:
content: "{{ claude_ws_kubeconfig }}"
dest: "/home/{{ claude_ws_interactive_user }}/.kube/config"
owner: "{{ claude_ws_interactive_user }}"
group: "{{ claude_ws_interactive_user }}"
mode: "0600"
no_log: true
# This VM's only git identity IS claude-bot (unlike the shared human
# workstation, where a global credential.helper is forbidden), so a per-user
# ~/.gitconfig pointing at each user's own claude-bot store is appropriate.
- name: Configure git identity + credential store for the Claude users
ansible.builtin.copy:
dest: "/home/{{ item }}/.gitconfig"
owner: "{{ item }}"
group: "{{ item }}"
mode: "0644"
content: |
[user]
name = Claude Bot
email = claude-bot@renner.uno
[credential]
helper = store --file=/home/{{ item }}/.config/claude-bot/git-credentials
loop:
- "{{ claude_ws_interactive_user }}"
- "{{ claude_ws_job_user }}"
# --- Notes / wiki pipeline (Syncthing + timers) ----------------------------
- name: Notes pipeline
ansible.builtin.include_tasks: notes.yml
when: claude_ws_notes_enabled | bool
# --- Egress lockdown -------------------------------------------------------
- name: Egress firewall
ansible.builtin.include_tasks: firewall.yml
@@ -0,0 +1,48 @@
---
# Notes/wiki pipeline: the Obsidian vault syncs onto this host via Syncthing
# (matching the user's existing multi-device setup), and a hardened, bounded
# systemd timer runs an unattended `claude -p` distill pass over it as the
# credential-isolated job user. Device pairing is a one-time manual step (see
# README) — this only installs Syncthing and the timer.
- name: Install Syncthing
ansible.builtin.apt:
name: syncthing
state: present
update_cache: true
- name: Enable Syncthing for the job user
ansible.builtin.systemd:
name: "syncthing@{{ claude_ws_job_user }}.service"
enabled: true
state: started
daemon_reload: true
- name: Deploy the notes-distill prompt
ansible.builtin.copy:
src: claude-notes-distill-prompt.md
dest: "{{ claude_ws_base_dir }}/claude-notes-distill-prompt.md"
mode: "0644"
- name: Deploy the notes-distill wrapper
ansible.builtin.template:
src: claude-notes-distill.sh.j2
dest: "{{ claude_ws_base_dir }}/claude-notes-distill.sh"
mode: "0755"
- name: Deploy the notes-distill service + timer
ansible.builtin.template:
src: "{{ item }}.j2"
dest: "/etc/systemd/system/{{ item }}"
mode: "0644"
loop:
- claude-notes-distill.service
- claude-notes-distill.timer
notify: Reload systemd
- name: Enable the notes-distill timer
ansible.builtin.systemd:
name: claude-notes-distill.timer
enabled: true
state: started
daemon_reload: true
@@ -0,0 +1,10 @@
[Unit]
# {{ ansible_managed }}
Description=Refresh claude-workstation egress allowlist from DNS
After=nftables.service network-online.target
Wants=network-online.target
Requires=nftables.service
[Service]
Type=oneshot
ExecStart={{ claude_ws_base_dir }}/claude-egress-refresh.sh
@@ -0,0 +1,27 @@
#!/usr/bin/env bash
# {{ ansible_managed }}
# Resolve the egress FQDN allowlist to current IPs and load them into the
# nftables @allow4/@allow6 sets. CDN IPs rotate, so this runs on a timer.
# No-op unless the claude_egress table is loaded (i.e. enforcement is on).
set -euo pipefail
TABLE="inet claude_egress"
FQDNS=(
{% for fqdn in claude_ws_egress_allow_fqdns %}
"{{ fqdn }}"
{% endfor %}
)
nft list table $TABLE >/dev/null 2>&1 || { echo "claude_egress table not loaded; enforcement off"; exit 0; }
v4=(); v6=()
for host in "${FQDNS[@]}"; do
while read -r ip; do [ -n "$ip" ] && v4+=("$ip"); done < <(getent ahostsv4 "$host" | awk '{print $1}' | sort -u)
while read -r ip; do [ -n "$ip" ] && v6+=("$ip"); done < <(getent ahostsv6 "$host" | awk '{print $1}' | sort -u)
done
nft flush set $TABLE allow4
nft flush set $TABLE allow6
[ ${#v4[@]} -gt 0 ] && nft add element $TABLE allow4 "{ $(IFS=,; echo "${v4[*]}") }"
[ ${#v6[@]} -gt 0 ] && nft add element $TABLE allow6 "{ $(IFS=,; echo "${v6[*]}") }"
echo "claude-egress-refresh: ${#v4[@]} v4 + ${#v6[@]} v6 addresses loaded"
@@ -0,0 +1,10 @@
[Unit]
# {{ ansible_managed }}
Description=Periodically refresh the claude-workstation egress allowlist
[Timer]
OnBootSec=30s
OnUnitActiveSec=15min
[Install]
WantedBy=timers.target
@@ -0,0 +1,32 @@
[Unit]
# {{ ansible_managed }}
Description=Claude Code unattended notes-distill pass
After=network-online.target syncthing@{{ claude_ws_job_user }}.service
Wants=network-online.target
[Service]
Type=oneshot
User={{ claude_ws_job_user }}
Group={{ claude_ws_job_user }}
EnvironmentFile=/home/{{ claude_ws_job_user }}/.config/claude-code.env
ExecStart={{ claude_ws_base_dir }}/claude-notes-distill.sh
# Hard wall-clock kill — the runaway/kill switch for this unattended agent.
TimeoutStartSec=1800
# --- Hardening: an unattended LLM job on internet-influenced input ----------
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ReadWritePaths=/home/{{ claude_ws_job_user }}
# Credential isolation: cannot read the interactive session's tokens/kubeconfig.
InaccessiblePaths=/home/{{ claude_ws_interactive_user }}
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictSUIDSGID=true
RestrictNamespaces=true
LockPersonality=true
MemoryDenyWriteExecute=false
[Install]
WantedBy=multi-user.target
@@ -0,0 +1,32 @@
#!/usr/bin/env bash
# {{ ansible_managed }}
# Unattended notes-distill pass. Prompt via stdin (never shell-substituted),
# tool-allowlisted, no --bare (bare silently drops CLAUDE_CODE_OAUTH_TOKEN).
# Bounded by the unit's TimeoutStartSec, not a CLI flag (--max-turns is absent
# in the installed Claude Code). Emits a terminal state line to the journal on
# EVERY outcome so a Loki alert can catch silent failures (dead-man switch).
set -euo pipefail
export HOME="/home/{{ claude_ws_job_user }}"
NOTES="{{ claude_ws_notes_dir }}"
PROMPT="{{ claude_ws_base_dir }}/claude-notes-distill-prompt.md"
log() { logger -t claude-notes-distill -- "$*"; echo "claude-notes-distill: $*"; }
if [ ! -d "$NOTES/raw" ]; then
log "result=skipped reason=no-vault path=$NOTES"
exit 0
fi
log "result=start"
if claude -p \
--add-dir "$NOTES" \
--allowedTools "Read,Write,Glob,Grep" \
--model sonnet \
< "$PROMPT"; then
log "result=success"
else
rc=$?
log "result=failure rc=$rc"
exit "$rc"
fi
@@ -0,0 +1,11 @@
[Unit]
# {{ ansible_managed }}
Description=Run the Claude Code notes-distill pass daily
[Timer]
OnCalendar=*-*-* 04:30:00
Persistent=true
RandomizedDelaySec=900
[Install]
WantedBy=timers.target
@@ -0,0 +1,55 @@
#!/usr/sbin/nft -f
# {{ ansible_managed }}
# Deny-default egress for the claude-workstation. CDN-backed destinations are
# matched via the @allow4/@allow6 named sets, repopulated from DNS by
# claude-egress-refresh. Static home services are matched by the @static* sets.
# Only loaded when claude_ws_egress_enforce is true.
flush ruleset
table inet claude_egress {
set allow4 { type ipv4_addr; flags interval; }
set allow6 { type ipv6_addr; flags interval; }
set static4 {
type ipv4_addr
flags interval
{% if (claude_ws_egress_allow_cidrs | default([])) | length > 0 %}
elements = { {{ claude_ws_egress_allow_cidrs | join(', ') }} }
{% endif %}
}
set static6 {
type ipv6_addr
flags interval
{% if (claude_ws_egress_allow_cidrs6 | default([])) | length > 0 %}
elements = { {{ claude_ws_egress_allow_cidrs6 | join(', ') }} }
{% endif %}
}
chain output {
type filter hook output priority 0; policy drop;
oif "lo" accept
ct state established,related accept
# DNS — required to resolve the allowlist itself.
udp dport 53 accept
tcp dport 53 accept
# Diagnostics + DHCP lease renewal on the bridged interface.
ip protocol icmp accept
ip6 nexthdr icmpv6 accept
udp dport { 67, 68 } accept
# NetBird / WireGuard control + data plane (initial handshakes).
udp dport { 3478, 51820 } accept
# Allowed destinations.
ip daddr @allow4 accept
ip6 daddr @allow6 accept
ip daddr @static4 accept
ip6 daddr @static6 accept
# Everything else is dropped and counted for inspection.
counter comment "claude-egress-dropped"
}
}
+10 -7
View File
@@ -8,11 +8,12 @@
lineinfile:
path: /etc/hosts
regexp: '^127\.0\.1\.1\s+{{ inventory_hostname }}'
line: "{{ ansible_facts['default_ipv4']['address'] }} {{ inventory_hostname }} {{ inventory_hostname_short | default(inventory_hostname.split('.')[0]) }}"
line: "{{ ansible_default_ipv4.address }} {{ inventory_hostname }} {{ inventory_hostname_short | default(inventory_hostname.split('.')[0]) }}"
state: present
when:
- ansible_facts['virtualization_type'] == 'lxc'
- ansible_facts['default_ipv4']['address'] is defined
- ansible_virtualization_type is defined
- ansible_virtualization_type == 'lxc'
- ansible_default_ipv4.address is defined
# -------------------------------------------------------
# 1. Install Common Packages
@@ -50,8 +51,10 @@
# -------------------------------------------------------
# 4. SSH Configuration
# -------------------------------------------------------
# SSH access (CA trust, admin authorized_keys) is managed by the ssh-ca role
# via plays/configure-ssh-ca.yml.
# Authorized keys (human + woodpecker-ci, the latter from=-restricted) are
# managed by the dedicated ansible/plays/configure-ssh-keys.yml play. The old
# "Add Ansible Runner SSH Key" task deployed ansible-runner@ci-new, a dead key
# for a host that no longer exists — removed (issue #184).
# -------------------------------------------------------
# 5. Disable NTP on VMs/LXC (use host time)
@@ -63,8 +66,8 @@
enabled: no
masked: yes
when:
- ansible_facts['virtualization_role'] == 'guest'
- ansible_facts['virtualization_type'] in ['kvm', 'lxc']
- ansible_virtualization_type is defined
- ansible_virtualization_type in ['kvm', 'lxc']
ignore_errors: yes
# -------------------------------------------------------
@@ -3,7 +3,7 @@
#
# Enforces the decisions the in-cluster CrowdSec already detects, at the point
# where public traffic actually enters: these hosts terminate TLS for every
# public renner.uno name. A drop here never reaches the mesh, the relay or the
# *.renner.uno name. A drop here never reaches the mesh, the relay or the
# cluster.
#
# Detection stays in the cluster. The reverse-proxy produces no parseable
@@ -15,8 +15,6 @@
# amd64 asset for exactly this tag.
crowdsec_bouncer_version: v0.0.36
crowdsec_bouncer_sha256: f86e4b72693549d99f40a9402abefb894108f047a3fbe6e72fada25ee17ce88b
# Controller cache for the release archive; it outlives /tmp cleanups.
crowdsec_bouncer_download_dir: "{{ lookup('ansible.builtin.env', 'HOME') }}/.cache/ansible/crowdsec"
# The LAPI over the public path these hosts already serve, see
# kubernetes/infrastructure/crowdsec/lapi-ingress.yaml. The proxies cannot
@@ -29,26 +29,14 @@
# upstream ships no container image for this bouncer, which is expected —
# it manipulates host netfilter.
#
# The controller has v4, so it does the fetching. The checksum is verified
# there, before anything is copied to a host. Also in check mode: the fetch
# only writes on the controller, and the copy below needs the file.
- name: Create the controller download directory
ansible.builtin.file:
path: "{{ crowdsec_bouncer_download_dir }}"
state: directory
mode: "0755"
delegate_to: localhost
become: false
run_once: true
check_mode: false
# The CI runner has v4, so it does the fetching. The checksum is verified
# there, before anything is copied to a host.
- name: Fetch the pinned bouncer release on the controller
check_mode: false
ansible.builtin.get_url:
url: >-
https://github.com/crowdsecurity/cs-firewall-bouncer/releases/download/{{
crowdsec_bouncer_version }}/crowdsec-firewall-bouncer-linux-amd64.tgz
dest: "{{ crowdsec_bouncer_download_dir }}/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}.tgz"
dest: /tmp/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}.tgz
checksum: sha256:{{ crowdsec_bouncer_sha256 }}
mode: "0644"
delegate_to: localhost
@@ -63,7 +51,7 @@
# host.
- name: Copy the bouncer archive to the host
ansible.builtin.copy:
src: "{{ crowdsec_bouncer_download_dir }}/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}.tgz"
src: /tmp/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}.tgz
dest: /opt/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}.tgz
mode: "0644"
-1
View File
@@ -87,7 +87,6 @@
ansible.builtin.command:
argv: [pgrep, -af, "edge-host/(open3e|e3oncan)"]
register: heating_can_survivors
check_mode: false
changed_when: false
failed_when: false
+3 -4
View File
@@ -65,7 +65,6 @@
- /etc/network/interfaces
- /etc/network/interfaces.d/
register: keepalived_ntp_static6_ifupdown
check_mode: false
changed_when: false
failed_when: false
when:
@@ -124,7 +123,6 @@
- "{{ keepalived_ntp_vip6.split('/')[0] }}\\b"
- /etc/systemd/network/
register: keepalived_ntp_static6_networkd
check_mode: false
changed_when: false
failed_when: false
when:
@@ -142,8 +140,9 @@
Handing the address to keepalived while networkd also assigns it means
it would NOT be withdrawn when check_ntp fails -- the host would keep
serving time on an address VRRP believes it has released.
Re-apply the edge-vlan role first (site.yml runs it before this play),
whose template no longer carries the [Address] block.
Re-apply the edge-vlan role first, whose template no longer carries the
[Address] block:
ansible-playbook -i inventory/pve_hosts.yml plays/install-edge-host.yml
success_msg: >-
No static networkd assignment of {{ keepalived_ntp_vip6 }}; safe for
keepalived to own it.
@@ -2,7 +2,7 @@
# Managed by Ansible - Do not edit manually
global_defs {
router_id {{ ansible_facts['hostname'] }}_ntp
router_id {{ ansible_hostname }}_ntp
enable_script_security
script_user root
}
@@ -6,13 +6,3 @@ netbird_management_url: "https://netbird.renner.uno"
# Setup key - should be defined in group_vars or host_vars
# netbird_setup_key: ""
# Release installed on hosts without a NetBird package repository (Arch).
# Debian/Ubuntu follow the apt repository instead.
netbird_client_version: "0.75.0"
netbird_release_url: "https://github.com/netbirdio/netbird/releases/download/v{{ netbird_client_version }}"
# On the controller; the archive is copied from there to the hosts.
netbird_download_dir: "{{ lookup('ansible.builtin.env', 'HOME') }}/.cache/ansible/netbird"
# Leave the host's DNS configuration alone (hosts with a pinned resolv.conf).
netbird_disable_dns: false
+36 -99
View File
@@ -2,90 +2,47 @@
# NetBird Client Installation Role
# Installs and configures NetBird client on hosts with netbird_setup_key defined
- name: Install NetBird from the NetBird apt repository
when: ansible_facts['os_family'] == 'Debian'
block:
- name: Install required packages
ansible.builtin.apt:
name:
- curl
- ca-certificates
- gnupg
# deb822_repository
- python3-debian
state: present
update_cache: true
- name: Install required packages
ansible.builtin.apt:
name:
- curl
- ca-certificates
- gnupg
state: present
update_cache: true
- name: Add NetBird repository key
ansible.builtin.shell:
cmd: |
curl -sSL https://pkgs.netbird.io/debian/public.key | gpg --dearmor -o /usr/share/keyrings/netbird-archive-keyring.gpg
creates: /usr/share/keyrings/netbird-archive-keyring.gpg
- name: Add NetBird repository key
ansible.builtin.shell:
cmd: |
curl -sSL https://pkgs.netbird.io/debian/public.key | gpg --dearmor -o /usr/share/keyrings/netbird-archive-keyring.gpg
creates: /usr/share/keyrings/netbird-archive-keyring.gpg
- name: Add NetBird repository
ansible.builtin.deb822_repository:
name: netbird
types: deb
uris: https://pkgs.netbird.io/debian
suites: stable
components: main
signed_by: /usr/share/keyrings/netbird-archive-keyring.gpg
- name: Add NetBird repository
ansible.builtin.apt_repository:
repo: "deb [signed-by=/usr/share/keyrings/netbird-archive-keyring.gpg] https://pkgs.netbird.io/debian stable main"
state: present
filename: netbird
# A one-line netbird.list beside netbird.sources lists the repository twice.
- name: Remove the one-line NetBird repository
ansible.builtin.file:
path: /etc/apt/sources.list.d/netbird.list
state: absent
- name: Install NetBird client
ansible.builtin.apt:
name: netbird
state: present
update_cache: true
- name: Install NetBird client
ansible.builtin.apt:
name: netbird
state: present
update_cache: true
- name: Check if NetBird is already connected
ansible.builtin.command: netbird status
register: netbird_status
changed_when: false
failed_when: false
# Arch has no netbird package in its repositories; the release binary is
# verified against the release's checksums file. It is downloaded on the
# controller and copied over: github.com has no IPv6 address, and proxy-1 is
# IPv6-only. The controller steps also run in check mode: they only write on
# the controller, and the install step needs the archive.
- name: Install NetBird from the release tarball
when: ansible_facts['os_family'] == 'Archlinux'
block:
- name: Create the controller download directory
ansible.builtin.file:
path: "{{ netbird_download_dir }}"
state: directory
mode: "0755"
delegate_to: localhost
become: false
run_once: true
check_mode: false
- name: Download the NetBird release on the controller
check_mode: false
ansible.builtin.get_url:
url: "{{ netbird_release_url }}/netbird_{{ netbird_client_version }}_linux_amd64.tar.gz"
dest: "{{ netbird_download_dir }}/netbird_{{ netbird_client_version }}_linux_amd64.tar.gz"
checksum: "sha256:{{ netbird_release_url }}/netbird_{{ netbird_client_version }}_checksums.txt"
mode: "0644"
delegate_to: localhost
become: false
run_once: true
- name: Install the NetBird binary
ansible.builtin.unarchive:
src: "{{ netbird_download_dir }}/netbird_{{ netbird_client_version }}_linux_amd64.tar.gz"
dest: /usr/local/bin
include:
- netbird
owner: root
group: root
mode: "0755"
- name: Install the NetBird service
ansible.builtin.command: netbird service install
args:
creates: /etc/systemd/system/netbird.service
- name: Connect NetBird to management server
ansible.builtin.command: >
netbird up
--management-url {{ netbird_management_url }}
--setup-key {{ netbird_setup_key }}
when: "'Connected' not in netbird_status.stdout"
register: netbird_connect
changed_when: netbird_connect.rc == 0
- name: Enable NetBird service
ansible.builtin.service:
@@ -93,29 +50,9 @@
state: started
enabled: true
- name: Check if NetBird is already connected
ansible.builtin.command: netbird status
register: netbird_status
check_mode: false
changed_when: false
failed_when: false
# "Disconnected" contains "Connected", so match the management line.
- name: Connect NetBird to management server
ansible.builtin.command: >
netbird up
--management-url {{ netbird_management_url }}
--setup-key {{ netbird_setup_key }}
{{ '--disable-dns' if netbird_disable_dns else '' }}
when: "'Management: Connected' not in netbird_status.stdout"
register: netbird_connect
changed_when: netbird_connect.rc == 0
no_log: true
- name: Get NetBird status
ansible.builtin.command: netbird status
register: netbird_final_status
check_mode: false
changed_when: false
- name: Display NetBird status
+12 -26
View File
@@ -136,41 +136,27 @@
- /opt/netbird/proxy
- /opt/netbird/proxy-state
# The management store has no cluster-side backup; the cluster's backup
# key is gone with the managed authorized_keys (ssh-ca role).
# The management store has no cluster-side backup. Pulling one put a root
# key on this host that the cluster held, and this server's own state is
# its own business to protect rather than the cluster's. Dropping the two
# tasks that placed them would only stop ansible managing the key and the
# script — the pair below revokes them instead, and can go once the role
# has run against every host that ever carried them.
- name: Remove the backup dump script
ansible.builtin.file:
path: /usr/local/bin/netbird-backup-dump
state: absent
- name: Revoke the cluster backup key
ansible.posix.authorized_key:
user: root
key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFeW/L9bHMTSa5/Qb0ClppCf4Z0hL39JwzI6tBmKd2FG netbird-backup@homelab-cluster"
state: absent
- name: Wait for management container to be ready
ansible.builtin.pause:
seconds: 10
# The host's mesh client must not manage DNS: containers here resolve through
# the host stub, and the renner.uno nameserver group is unreachable from it.
- name: Check for the host NetBird client
ansible.builtin.stat:
path: /usr/bin/netbird
register: _netbird_client
- name: Read the host NetBird client config
ansible.builtin.command: netbird debug config
register: _netbird_client_config
check_mode: false
changed_when: false
when: _netbird_client.stat.exists
# disable-dns is only applied on a fresh `up`, not to a connected client.
- name: Reconnect the host NetBird client with DNS management disabled
ansible.builtin.shell: |
set -eu
netbird down
netbird up --disable-dns
when:
- _netbird_client.stat.exists
- not (_netbird_client_config.stdout | from_json).disableDns
- name: Display deployment status
ansible.builtin.debug:
msg: |
@@ -61,10 +61,6 @@ services:
- AUTH_AUDIENCE={{ netbird_auth_audience }}
- AUTH_CLIENT_ID={{ netbird_auth_client_id }}
- AUTH_AUTHORITY={{ netbird_auth_issuer }}
# The CSP allows AUTH_AUTHORITY only as a path (/application/o/netbird/);
# Authentik's token/userinfo endpoints sit outside it, so the browser
# would block the code exchange and every token refresh.
- NETBIRD_CSP={{ netbird_auth_issuer | urlsplit('scheme') }}://{{ netbird_auth_issuer | urlsplit('netloc') }}
- USE_AUTH0=false
- AUTH_SUPPORTED_SCOPES=openid profile email
- NETBIRD_TOKEN_SOURCE=idToken
@@ -0,0 +1,20 @@
---
# Node Exporter defaults
node_exporter_version: "1.8.2"
node_exporter_port: 9100
node_exporter_user: node-exporter
node_exporter_group: node-exporter
# Installation paths
node_exporter_install_dir: /opt/node-exporter
node_exporter_binary: "{{ node_exporter_install_dir }}/node_exporter"
# Collectors to enable (defaults are usually sufficient)
node_exporter_enabled_collectors:
- systemd
- processes
- timex
# Collectors to disable
node_exporter_disabled_collectors: []
@@ -0,0 +1,62 @@
#!/bin/bash
# Generate Prometheus metrics from chronyc tracking output
# Runs as a cron job and outputs to textfile collector directory
OUTPUT_FILE="${1:-/var/lib/node-exporter/chrony.prom}"
# Get chrony tracking info
tracking=$(chronyc tracking 2>/dev/null)
if [ -z "$tracking" ]; then
echo "chrony_up 0" > "$OUTPUT_FILE.tmp"
mv "$OUTPUT_FILE.tmp" "$OUTPUT_FILE"
exit 0
fi
# Parse values
stratum=$(echo "$tracking" | grep "^Stratum" | awk '{print $3}')
system_time=$(echo "$tracking" | grep "^System time" | awk '{print $4}')
root_delay=$(echo "$tracking" | grep "^Root delay" | awk '{print $4}')
root_dispersion=$(echo "$tracking" | grep "^Root dispersion" | awk '{print $4}')
last_offset=$(echo "$tracking" | grep "^Last offset" | awk '{print $4}')
rms_offset=$(echo "$tracking" | grep "^RMS offset" | awk '{print $4}')
frequency=$(echo "$tracking" | grep "^Frequency" | awk '{print $3}')
ref_id=$(echo "$tracking" | grep "^Reference ID" | awk '{print $4}' | tr -d '()')
# Handle negative offsets (remove leading + or keep -)
system_time=$(echo "$system_time" | sed 's/^+//')
last_offset=$(echo "$last_offset" | sed 's/^+//')
rms_offset=$(echo "$rms_offset" | sed 's/^+//')
# Output Prometheus metrics
cat > "$OUTPUT_FILE.tmp" << EOF
# HELP chrony_stratum Current stratum of the NTP server
# TYPE chrony_stratum gauge
chrony_stratum ${stratum:-0}
# HELP chrony_system_time_offset_seconds System time offset from NTP time
# TYPE chrony_system_time_offset_seconds gauge
chrony_system_time_offset_seconds ${system_time:-0}
# HELP chrony_root_delay_seconds Total round-trip delay to the primary reference clock
# TYPE chrony_root_delay_seconds gauge
chrony_root_delay_seconds ${root_delay:-0}
# HELP chrony_root_dispersion_seconds Total dispersion to the primary reference clock
# TYPE chrony_root_dispersion_seconds gauge
chrony_root_dispersion_seconds ${root_dispersion:-0}
# HELP chrony_last_offset_seconds Estimated offset of the last update
# TYPE chrony_last_offset_seconds gauge
chrony_last_offset_seconds ${last_offset:-0}
# HELP chrony_rms_offset_seconds RMS offset
# TYPE chrony_rms_offset_seconds gauge
chrony_rms_offset_seconds ${rms_offset:-0}
# HELP chrony_frequency_ppm Frequency offset in parts per million
# TYPE chrony_frequency_ppm gauge
chrony_frequency_ppm ${frequency:-0}
# HELP chrony_up Whether chrony is running and synchronized
# TYPE chrony_up gauge
chrony_up 1
# HELP chrony_reference_info Reference clock information
# TYPE chrony_reference_info gauge
chrony_reference_info{reference="${ref_id:-unknown}"} 1
EOF
mv "$OUTPUT_FILE.tmp" "$OUTPUT_FILE"
@@ -0,0 +1,9 @@
---
- name: Reload systemd
systemd:
daemon_reload: yes
- name: Restart node-exporter
systemd:
name: node-exporter
state: restarted
@@ -0,0 +1,88 @@
---
# Install and configure Node Exporter
- name: Create node-exporter system group
group:
name: "{{ node_exporter_group }}"
system: yes
state: present
- name: Create node-exporter system user
user:
name: "{{ node_exporter_user }}"
group: "{{ node_exporter_group }}"
system: yes
shell: /usr/sbin/nologin
home: "{{ node_exporter_install_dir }}"
createhome: no
- name: Create installation directory
file:
path: "{{ node_exporter_install_dir }}"
state: directory
owner: "{{ node_exporter_user }}"
group: "{{ node_exporter_group }}"
mode: '0755'
- name: Check if node_exporter is already installed
stat:
path: "{{ node_exporter_binary }}"
register: node_exporter_installed
- name: Get installed version
command: "{{ node_exporter_binary }} --version"
register: installed_version
when: node_exporter_installed.stat.exists
changed_when: false
failed_when: false
- name: Download node_exporter
get_url:
url: "https://github.com/prometheus/node_exporter/releases/download/v{{ node_exporter_version }}/node_exporter-{{ node_exporter_version }}.linux-amd64.tar.gz"
dest: /tmp/node_exporter.tar.gz
mode: '0644'
when: not node_exporter_installed.stat.exists or (installed_version.stdout is defined and node_exporter_version not in installed_version.stdout)
- name: Extract node_exporter
unarchive:
src: /tmp/node_exporter.tar.gz
dest: /tmp
remote_src: yes
when: not node_exporter_installed.stat.exists or (installed_version.stdout is defined and node_exporter_version not in installed_version.stdout)
- name: Install node_exporter binary
copy:
src: "/tmp/node_exporter-{{ node_exporter_version }}.linux-amd64/node_exporter"
dest: "{{ node_exporter_binary }}"
owner: "{{ node_exporter_user }}"
group: "{{ node_exporter_group }}"
mode: '0755'
remote_src: yes
when: not node_exporter_installed.stat.exists or (installed_version.stdout is defined and node_exporter_version not in installed_version.stdout)
notify: Restart node-exporter
- name: Create systemd service
template:
src: node_exporter.service.j2
dest: /etc/systemd/system/node-exporter.service
owner: root
group: root
mode: '0644'
notify:
- Reload systemd
- Restart node-exporter
- name: Enable and start node-exporter service
systemd:
name: node-exporter
enabled: yes
state: started
daemon_reload: yes
- name: Clean up downloaded files
file:
path: "{{ item }}"
state: absent
loop:
- /tmp/node_exporter.tar.gz
- "/tmp/node_exporter-{{ node_exporter_version }}.linux-amd64"
@@ -0,0 +1,38 @@
[Unit]
Description=Prometheus Node Exporter
Documentation=https://github.com/prometheus/node_exporter
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User={{ node_exporter_user }}
Group={{ node_exporter_group }}
ExecStart={{ node_exporter_binary }} \
--web.listen-address=:{{ node_exporter_port }} \
{% for collector in node_exporter_enabled_collectors %}
--collector.{{ collector }} \
{% endfor %}
{% for collector in node_exporter_disabled_collectors %}
--no-collector.{{ collector }} \
{% endfor %}
{% if node_exporter_textfile_dir is defined %}
--collector.textfile \
--collector.textfile.directory={{ node_exporter_textfile_dir }} \
{% endif %}
--collector.filesystem.mount-points-exclude='^/(sys|proc|dev|host|etc)($$|/)'
Restart=on-failure
RestartSec=10s
# Security hardening
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
ReadOnlyPaths=/
{% if node_exporter_textfile_dir is defined %}
ReadWritePaths={{ node_exporter_textfile_dir }}
{% endif %}
[Install]
WantedBy=multi-user.target
-142
View File
@@ -1,142 +0,0 @@
---
- name: Fail on unsupported distribution
ansible.builtin.fail:
msg: >-
os-update supports Debian and Archlinux only;
this host reports os_family={{ ansible_facts['os_family'] }}
when: ansible_facts['os_family'] not in ['Debian', 'Archlinux']
# --- Baseline: what is healthy before the upgrade must be healthy after ---
- name: Record failed units before upgrade
ansible.builtin.command: systemctl list-units --state=failed --plain --no-legend --no-pager
register: os_update_failed_before
changed_when: false
check_mode: false
- name: Check for docker
ansible.builtin.stat:
path: /usr/bin/docker
register: os_update_docker
- name: Record running containers before upgrade
ansible.builtin.command: docker ps --format "{{ '{{' }}.Names{{ '}}' }}"
register: os_update_containers_before
changed_when: false
check_mode: false
when: os_update_docker.stat.exists
- name: Assert at least 1 GiB free on /
ansible.builtin.assert:
that: >-
(ansible_facts['mounts'] | selectattr('mount', 'equalto', '/') | first)['size_available']
>= 1073741824
fail_msg: "Less than 1 GiB free on / -- free space before upgrading"
# --- Arch Linux ---
- name: Upgrade keyring first (Arch)
community.general.pacman:
name: archlinux-keyring
state: latest
update_cache: true
when: ansible_facts['os_family'] == 'Archlinux'
- name: Upgrade all packages (Arch)
community.general.pacman:
upgrade: true
register: os_update_pacman
when: ansible_facts['os_family'] == 'Archlinux'
# --- Debian/Ubuntu ---
- name: Upgrade all packages (Debian)
ansible.builtin.apt:
update_cache: true
upgrade: full
autoremove: true
environment:
DEBIAN_FRONTEND: noninteractive
when: ansible_facts['os_family'] == 'Debian'
- name: Check reboot-required flag (Debian)
ansible.builtin.stat:
path: /var/run/reboot-required
register: os_update_reboot_flag
when: ansible_facts['os_family'] == 'Debian'
# --- Reboot ---
- name: Read running kernel before reboot (Arch)
ansible.builtin.command: uname -r
register: os_update_kernel_before
changed_when: false
check_mode: false
when: ansible_facts['os_family'] == 'Archlinux'
# Missing modules dir: a newer kernel is installed but not booted, including
# from an earlier interrupted run where pacman now reports no change.
- name: Check running kernel is still installed (Arch)
ansible.builtin.stat:
path: "/usr/lib/modules/{{ os_update_kernel_before.stdout }}/vmlinuz"
register: os_update_kernel_present
when: ansible_facts['os_family'] == 'Archlinux'
- name: Reboot
ansible.builtin.reboot:
msg: "Rebooting after package upgrade"
reboot_timeout: 900
post_reboot_delay: 30
when: >-
(ansible_facts['os_family'] == 'Archlinux' and
(os_update_pacman is changed or not os_update_kernel_present.stat.exists)) or
(ansible_facts['os_family'] == 'Debian' and os_update_reboot_flag.stat.exists)
# --- Verify ---
- name: Read running kernel
ansible.builtin.command: uname -r
register: os_update_kernel
changed_when: false
check_mode: false
# Arch keeps only the installed kernel's modules; the directory existing means
# the host booted the kernel pacman installed.
- name: Assert running kernel is the installed one (Arch)
ansible.builtin.stat:
path: "/usr/lib/modules/{{ os_update_kernel.stdout }}/vmlinuz"
register: os_update_kernel_installed
failed_when: not os_update_kernel_installed.stat.exists
when: ansible_facts['os_family'] == 'Archlinux'
- name: Assert no reboot pending (Debian)
ansible.builtin.stat:
path: /var/run/reboot-required
register: os_update_reboot_flag_after
failed_when: os_update_reboot_flag_after.stat.exists
when: ansible_facts['os_family'] == 'Debian'
- name: Wait for previously running containers
ansible.builtin.command: docker ps --format "{{ '{{' }}.Names{{ '}}' }}"
register: os_update_containers_after
changed_when: false
check_mode: false
until: >-
os_update_containers_before.stdout_lines
| difference(os_update_containers_after.stdout_lines)
| length == 0
retries: 20
delay: 15
when: os_update_docker.stat.exists
- name: Wait for no new failed units
ansible.builtin.command: systemctl list-units --state=failed --plain --no-legend --no-pager
register: os_update_failed_after
changed_when: false
check_mode: false
until: >-
os_update_failed_after.stdout_lines
| map('split') | map('first')
| difference(os_update_failed_before.stdout_lines | map('split') | map('first'))
| length == 0
retries: 10
delay: 15
- name: Report kernel
ansible.builtin.debug:
msg: "{{ inventory_hostname }} runs {{ os_update_kernel.stdout }}"
@@ -6,15 +6,15 @@
apt:
name: qemu-guest-agent
state: present
when:
- ansible_facts['os_family'] == 'Debian'
when:
- ansible_os_family is defined and ansible_os_family == 'Debian'
- name: Install QEMU Guest Agent (RHEL/Rocky VMs)
dnf:
name: qemu-guest-agent
state: present
when:
- ansible_facts['os_family'] == 'RedHat'
when:
- ansible_os_family is defined and ansible_os_family == 'RedHat'
- name: Ensure qemu-guest-agent is started and enabled
service:
-21
View File
@@ -1,21 +0,0 @@
---
# SSH user CA (OpenBao ssh-client-signer). Hosts trust certificates signed by
# it; the public key is fetched from OpenBao on the controller at run time.
ssh_ca_public_key_url: "https://openbao.renner.uno/v1/ssh-client-signer/public_key"
# Owner certificates (role owner) carry this principal. It is accepted for
# whichever admin login the distro has; users that do not exist are skipped.
ssh_ca_owner_principal: k.renner
ssh_ca_owner_users:
- root
- ladmin
# Claude's own account. Certificates (role claude, 1h) carry the principal.
ssh_ca_claude_user: claude-bot
ssh_ca_claude_principal: claude-bot
# Units claude-bot may restart/reload via sudo on this host, e.g. [netbird].
ssh_ca_claude_sudo_units: []
# Full passwordless sudo for claude-bot on this host.
ssh_ca_claude_sudo_all: false
-2
View File
@@ -1,2 +0,0 @@
# Public keys whose certificates sshd must reject (one per line, OpenSSH format).
# Revoking a key also revokes every certificate issued for it.
@@ -1,15 +0,0 @@
---
# Replaces the admin logins' authorized_keys with ssh_admin_authorized_keys
# (break-glass + CI). Everything else logs in with a CA certificate.
# ssh_ca_admin_logins comes from main.yml, which runs first in the same play.
- name: Set the admin logins' authorized_keys to the managed list
ansible.posix.authorized_key:
user: "{{ item[0] }}"
key: |
{% for k in ssh_admin_authorized_keys %}
{{ (k.options ~ ' ') if (k.options | default('')) else '' }}{{ k.key }}
{% endfor %}
exclusive: true
loop: "{{ ssh_ca_admin_logins }}"
loop_control:
label: "{{ item[0] }}"
-276
View File
@@ -1,276 +0,0 @@
---
- name: Fetch the SSH CA public key from OpenBao
ansible.builtin.uri:
url: "{{ ssh_ca_public_key_url }}"
return_content: true
delegate_to: localhost
become: false
run_once: true
register: ssh_ca_key
check_mode: false
- name: Check the fetched CA key is an SSH public key
ansible.builtin.assert:
that:
- ssh_ca_key.content is match('^ssh-ed25519 AAAA')
fail_msg: "Unexpected CA key from {{ ssh_ca_public_key_url }}"
run_once: true
- name: Install the trusted user CA key
ansible.builtin.copy:
content: "{{ ssh_ca_key.content | trim }}\n"
dest: /etc/ssh/trusted_user_ca_keys
owner: root
group: root
mode: "0644"
register: ssh_ca_trusted_file
# sshd refuses all public key logins when RevokedKeys is set but unreadable,
# so the file is always deployed, even when empty.
- name: Install the revoked keys list
ansible.builtin.copy:
src: revoked_keys
dest: /etc/ssh/revoked_keys
owner: root
group: root
mode: "0644"
# /etc/passwd, not getent: DSM has no getent binary. The admin logins are
# local accounts. Each entry is the split passwd line (0 name, 5 home).
- name: Read the local accounts
ansible.builtin.slurp:
src: /etc/passwd
register: ssh_ca_passwd_file
- name: Find the existing admin logins
vars:
ssh_ca_passwd: "{{ ssh_ca_passwd_file.content | b64decode | trim | split('\n') | map('split', ':') | list }}"
ansible.builtin.set_fact:
ssh_ca_admin_logins: "{{ ssh_ca_passwd | selectattr('0', 'in', ssh_ca_owner_users) | list }}"
ssh_ca_claude_exists: "{{ ssh_ca_claude_user in ssh_ca_passwd | map('first') }}"
# DSM manages accounts with synouser/synogroup and has no useradd.
- name: Detect DSM
ansible.builtin.stat:
path: /usr/syno/sbin/synouser
register: ssh_ca_synouser
# systemd-journal: read every journal without sudo (see sudoers-claude.j2).
- name: Create the Claude account
ansible.builtin.user:
name: "{{ ssh_ca_claude_user }}"
shell: /bin/bash
create_home: true
password: "!"
groups: systemd-journal
append: true
when: not ssh_ca_synouser.stat.exists
- name: Create the Claude account on DSM
when: ssh_ca_synouser.stat.exists
block:
# synouser requires a password; nobody knows it and sshd refuses passwords.
- name: Create the DSM account
ansible.builtin.command:
argv:
- /usr/syno/sbin/synouser
- --add
- "{{ ssh_ca_claude_user }}"
- "{{ lookup('ansible.builtin.password', '/dev/null', length=40, chars=['ascii_letters', 'digits']) }}"
- Claude Bot
- "0"
- ""
- "0"
# command runs in check mode here.
when:
- not ssh_ca_claude_exists
- not ansible_check_mode
no_log: true
# Members are listed as "<n>:[<name>]".
- name: Read the administrators members
ansible.builtin.command: /usr/syno/sbin/synogroup --get administrators
register: ssh_ca_dsm_admins_out
changed_when: false
check_mode: false
- name: Find the administrators members
ansible.builtin.set_fact:
ssh_ca_dsm_admins: >-
{{ ssh_ca_dsm_admins_out.stdout_lines | select('match', '^[0-9]+:\[.*\]$')
| map('regex_replace', '^[0-9]+:\[(.*)\]$', '\1') | list }}
# synogroup --member replaces the member list; a parse that lost the owner
# admin login would drop it.
- name: Check the member list holds the owner admin login
ansible.builtin.assert:
that:
- ssh_ca_dsm_admins | intersect(ssh_ca_owner_users) | length > 0
fail_msg: "administrators members lack {{ ssh_ca_owner_users }}: {{ ssh_ca_dsm_admins }}"
# DSM allows SSH for administrators only.
- name: Add the Claude account to administrators
ansible.builtin.command:
argv: "{{ ['/usr/syno/sbin/synogroup', '--member', 'administrators'] + ssh_ca_dsm_admins + [ssh_ca_claude_user] }}"
when:
- ssh_ca_claude_user not in ssh_ca_dsm_admins
- not ansible_check_mode
- name: Create the principals directory
ansible.builtin.file:
path: /etc/ssh/auth_principals
state: directory
owner: root
group: root
mode: "0755"
- name: Accept the owner principal for the existing admin logins
ansible.builtin.copy:
content: "{{ ssh_ca_owner_principal }}\n"
dest: "/etc/ssh/auth_principals/{{ item[0] }}"
owner: root
group: root
mode: "0644"
loop: "{{ ssh_ca_admin_logins }}"
loop_control:
label: "{{ item[0] }}"
- name: Accept the Claude principal for the Claude account
ansible.builtin.copy:
content: "{{ ssh_ca_claude_principal }}\n"
dest: "/etc/ssh/auth_principals/{{ ssh_ca_claude_user }}"
owner: root
group: root
mode: "0644"
- name: Install the Claude sudo rules
ansible.builtin.template:
src: sudoers-claude.j2
dest: "/etc/sudoers.d/{{ ssh_ca_claude_user }}"
owner: root
group: root
mode: "0440"
validate: visudo -cf %s
when: not ssh_ca_synouser.stat.exists
# DSM has no visudo, so the rules are checked on the controller.
- name: Install the Claude sudo rules on DSM
when: ssh_ca_synouser.stat.exists
vars:
ssh_ca_sudoers: "{{ lookup('ansible.builtin.template', 'sudoers-claude.j2') | trim }}"
block:
- name: Check the sudo rules on the controller
ansible.builtin.command:
cmd: visudo -c -f -
stdin: "{{ ssh_ca_sudoers }}"
delegate_to: localhost
become: false
changed_when: false
check_mode: false
- name: Install the checked sudo rules
ansible.builtin.copy:
content: "{{ ssh_ca_sudoers }}\n"
dest: "/etc/sudoers.d/{{ ssh_ca_claude_user }}"
owner: root
group: root
mode: "0440"
- name: Create the sshd drop-in directory
ansible.builtin.file:
path: /etc/ssh/sshd_config.d
state: directory
owner: root
group: root
mode: "0755"
# DSM's sshd_config has no Include. sshd keeps the first value it reads, so
# the Include goes first for the drop-ins to win.
- name: Include the sshd drop-in directory
ansible.builtin.lineinfile:
path: /etc/ssh/sshd_config
line: Include /etc/ssh/sshd_config.d/*.conf
insertbefore: BOF
validate: sshd -t -f %s
register: ssh_ca_include_line
# A broken drop-in only bites on the next sshd restart, so it is tested here
# and removed again if the full config does not parse.
- name: Configure sshd for the user CA
block:
- name: Install the sshd drop-in
ansible.builtin.template:
src: ssh-ca.conf.j2
dest: /etc/ssh/sshd_config.d/50-ssh-ca.conf
owner: root
group: root
mode: "0644"
register: ssh_ca_dropin_file
- name: Test the full sshd config
ansible.builtin.command: sshd -t
changed_when: false
rescue:
- name: Remove the drop-in that broke sshd -t
ansible.builtin.file:
path: /etc/ssh/sshd_config.d/50-ssh-ca.conf
state: absent
- name: Fail the host
ansible.builtin.fail:
msg: "sshd -t failed with the SSH CA drop-in; it was removed again"
- name: Read the effective sshd settings
ansible.builtin.command: sshd -T
changed_when: false
register: ssh_ca_effective
# Keyword case in sshd -T output differs between OpenSSH versions (mixed case
# on 10.6), so the lines are compared lowercased. Not in check mode: sshd -T
# reads the files on disk, which check mode has not written.
- name: Check sshd uses the CA, principals and revocation list
when: not ansible_check_mode
vars:
ssh_ca_effective_lines: "{{ ssh_ca_effective.stdout_lines | map('lower') | list }}"
ansible.builtin.assert:
that:
- "'trustedusercakeys /etc/ssh/trusted_user_ca_keys' in ssh_ca_effective_lines"
- "'authorizedprincipalsfile /etc/ssh/auth_principals/%u' in ssh_ca_effective_lines"
- "'revokedkeys /etc/ssh/revoked_keys' in ssh_ca_effective_lines"
- "'passwordauthentication no' in ssh_ca_effective_lines"
- "'kbdinteractiveauthentication no' in ssh_ca_effective_lines"
fail_msg: "sshd -T does not show the SSH CA settings; another config line takes precedence"
# The unit is sshd.service on Arch and RedHat, ssh.service on Debian/Ubuntu,
# where sshd.service is an alias that resolves to it. The running daemon is
# reloaded when a file it reads at start changed; a reload keeps open
# sessions. None runs when sshd is socket activated and stopped, or per
# connection (ssh-mesh-only role); the next start reads the config.
# Principals and revoked keys are read per login and need no reload.
# systemctl directly: service_facts took 3-12 s per host.
- name: Find the running sshd unit
ansible.builtin.command: systemctl list-units --state=running --plain --no-legend --type=service sshd.service ssh.service
changed_when: false
register: ssh_ca_sshd_running
check_mode: false
- name: Reload the running sshd unit
vars:
ssh_ca_sshd_units: "{{ ssh_ca_sshd_running.stdout_lines | map('split') | map('first') | unique | list }}"
block:
- name: Check at most one sshd unit is running
ansible.builtin.assert:
that:
- ssh_ca_sshd_units | length <= 1
fail_msg: "Expected at most one running sshd.service or ssh.service, found {{ ssh_ca_sshd_units }}"
- name: Reload sshd so the running daemon uses the CA config
ansible.builtin.systemd:
name: "{{ ssh_ca_sshd_units | first }}"
state: reloaded
when:
- ssh_ca_sshd_units | length == 1
- ssh_ca_trusted_file is changed or ssh_ca_include_line is changed or ssh_ca_dropin_file is changed
- name: Report the admin logins' authorized keys
ansible.builtin.include_tasks: report.yml
-26
View File
@@ -1,26 +0,0 @@
---
# Read-only: what the admin logins accept besides certificates. Hosts in
# [ssh_keys_managed] have this reduced to ssh_admin_authorized_keys.
- name: Find the admin logins' authorized_keys
ansible.builtin.stat:
path: "{{ item[5] }}/.ssh/authorized_keys"
loop: "{{ ssh_ca_admin_logins }}"
loop_control:
label: "{{ item[0] }}"
register: ssh_ca_keyfiles
- name: Fingerprint the admin logins' authorized keys
ansible.builtin.command: ssh-keygen -lf {{ item.stat.path }}
loop: "{{ ssh_ca_keyfiles.results | selectattr('stat', 'defined') | selectattr('stat.exists') | list }}"
loop_control:
label: "{{ item.stat.path }}"
changed_when: false
register: ssh_ca_fingerprints
check_mode: false
- name: Report the admin logins' authorized keys
ansible.builtin.debug:
msg: "{{ item.item.stat.path }}: {{ item.stdout_lines }}"
loop: "{{ ssh_ca_fingerprints.results }}"
loop_control:
label: "{{ item.item.stat.path }}"
@@ -1,6 +0,0 @@
# {{ ansible_managed }}
TrustedUserCAKeys /etc/ssh/trusted_user_ca_keys
AuthorizedPrincipalsFile /etc/ssh/auth_principals/%u
RevokedKeys /etc/ssh/revoked_keys
PasswordAuthentication no
KbdInteractiveAuthentication no
@@ -1,15 +0,0 @@
# {{ ansible_managed }}
# Logs and unit status come from the systemd-journal group, not sudo:
# journalctl as root would also allow --vacuum-*/--rotate.
# Commands are listed for both the Arch and the Debian/Ubuntu path.
# nft only with fixed arguments: it joins argv into one command line, so a
# wildcard would admit "; flush ruleset". No docker start: -a/-i attach to a
# container's stdin; docker restart also starts stopped containers.
{{ ssh_ca_claude_user }} ALL=(root) NOPASSWD:NOEXEC: /usr/bin/ss -tulpn, /usr/sbin/sshd -T, /usr/bin/nft list ruleset, /usr/sbin/nft list ruleset, /usr/bin/netbird status, /usr/bin/netbird status *, /usr/local/bin/netbird status, /usr/local/bin/netbird status *
{{ ssh_ca_claude_user }} ALL=(root) NOPASSWD:NOEXEC: /usr/bin/docker ps, /usr/bin/docker ps *, /usr/bin/docker logs *, /usr/bin/docker stats --no-stream, /usr/bin/docker restart *
{% for unit in ssh_ca_claude_sudo_units %}
{{ ssh_ca_claude_user }} ALL=(root) NOPASSWD:NOEXEC: /usr/bin/systemctl restart {{ unit }}, /usr/bin/systemctl reload {{ unit }}
{% endfor %}
{% if ssh_ca_claude_sudo_all %}
{{ ssh_ca_claude_user }} ALL=(ALL) NOPASSWD: ALL
{% endif %}
@@ -1,9 +0,0 @@
---
# The only interface sshd accepts connections on.
ssh_mesh_interface: wt0
# Minutes until a switchover reverts to the distro listener, unless a new
# login through the mesh-only listener confirms it first.
ssh_mesh_revert_minutes: 5
ssh_mesh_udev_rule: /etc/udev/rules.d/90-ssh-mesh-only.rules
-114
View File
@@ -1,114 +0,0 @@
---
- name: Load the distro's sshd unit names
ansible.builtin.include_vars: "{{ ansible_facts['os_family'] }}.yml"
- name: Look for the mesh interface
ansible.builtin.stat:
path: "/sys/class/net/{{ ssh_mesh_interface }}"
register: ssh_mesh_iface
- name: Check the mesh interface exists
ansible.builtin.assert:
that:
- ssh_mesh_iface.stat.exists
fail_msg: "No {{ ssh_mesh_interface }}: sshd would be unreachable. Install the NetBird client first."
- name: Create the socket unit directory
ansible.builtin.file:
path: "{{ ssh_mesh_socket_file | dirname }}"
state: directory
owner: root
group: root
mode: "0755"
- name: Install the mesh-only socket
ansible.builtin.template:
src: "{{ ssh_mesh_socket_template }}"
dest: "{{ ssh_mesh_socket_file }}"
owner: root
group: root
mode: "0644"
- name: Bind the sshd service to the socket
when: ssh_mesh_service_dropin | length > 0
block:
- name: Create the service drop-in directory
ansible.builtin.file:
path: "{{ ssh_mesh_service_dropin | dirname }}"
state: directory
owner: root
group: root
mode: "0755"
- name: Install the service drop-in
ansible.builtin.template:
src: ssh.service-dropin.conf.j2
dest: "{{ ssh_mesh_service_dropin }}"
owner: root
group: root
mode: "0644"
- name: Install the udev rule that starts the socket with the interface
ansible.builtin.template:
src: udev.rules.j2
dest: "{{ ssh_mesh_udev_rule }}"
owner: root
group: root
mode: "0644"
register: ssh_mesh_udev
- name: Reload the udev rules
ansible.builtin.command: udevadm control --reload
when: ssh_mesh_udev is changed
changed_when: true
- name: Install the switch and revert scripts
ansible.builtin.template:
src: "{{ item }}.j2"
dest: "/usr/local/sbin/{{ item }}"
owner: root
group: root
mode: "0755"
loop:
- ssh-mesh-switch
- ssh-mesh-revert
- name: Reload systemd units
ansible.builtin.systemd:
daemon_reload: true
- name: Read the sshd listeners
ansible.builtin.command: ss -Htln 'sport = :22'
changed_when: false
register: ssh_mesh_listeners
check_mode: false
# Each run that finds a listener outside the interface switches over. An
# include, because meta reset_connection ignores when: and a reconnect costs a
# YubiKey PIN from a workstation.
- name: Switch sshd to the mesh-only socket
ansible.builtin.include_tasks: switch.yml
when: >-
ssh_mesh_listeners.stdout_lines | length == 0 or
ssh_mesh_listeners.stdout_lines | reject('search', '%' ~ ssh_mesh_interface ~ ':22\s') | list | length > 0
- name: Check sshd listens on the mesh interface only
ansible.builtin.assert:
that:
- ssh_mesh_listeners.stdout_lines | length > 0
- ssh_mesh_listeners.stdout_lines | reject('search', '%' ~ ssh_mesh_interface ~ ':22\s') | list | length == 0
fail_msg: "Listeners outside {{ ssh_mesh_interface }}: {{ ssh_mesh_listeners.stdout_lines }}"
- name: Cancel the revert timer
ansible.builtin.command: systemctl stop ssh-mesh-revert.timer
changed_when: false
failed_when: false
- name: Check the public name refuses SSH
ansible.builtin.wait_for:
host: "{{ inventory_hostname }}"
port: 22
state: stopped
timeout: 15
delegate_to: localhost
become: false
@@ -1,27 +0,0 @@
---
# The revert timer runs in its own unit, so a switch that locks the play out is
# undone after ssh_mesh_revert_minutes. main.yml cancels it after checking the
# listeners read through the new one.
- name: Clear a revert timer left by an earlier run
ansible.builtin.command: systemctl stop ssh-mesh-revert.timer ssh-mesh-revert.service
changed_when: false
failed_when: false
- name: Arm the revert timer
ansible.builtin.command: >-
systemd-run --unit=ssh-mesh-revert --collect
--on-active={{ ssh_mesh_revert_minutes }}min
/usr/local/sbin/ssh-mesh-revert
changed_when: true
- name: Switch the listener
ansible.builtin.command: systemd-run --unit=ssh-mesh-switch --collect --wait /usr/local/sbin/ssh-mesh-switch
changed_when: true
- name: Reconnect through the new listener
ansible.builtin.meta: reset_connection
- name: Read the sshd listeners after reconnecting
ansible.builtin.command: ss -Htln 'sport = :22'
changed_when: false
register: ssh_mesh_listeners
@@ -1,9 +0,0 @@
#!/bin/sh
# {{ ansible_managed }}
# Restores the distro listener on all addresses. Armed as a timer by every
# switchover; the play cancels it after a login through the new listener.
systemctl stop {{ ssh_mesh_service }} {{ ssh_mesh_socket }}
rm -f {{ ssh_mesh_udev_rule }} {{ ssh_mesh_socket_file }}{{ (' ' ~ ssh_mesh_service_dropin) if ssh_mesh_service_dropin else '' }}
udevadm control --reload
systemctl daemon-reload
systemctl enable --now {{ ssh_mesh_distro_listener }}
@@ -1,11 +0,0 @@
#!/bin/sh
# {{ ansible_managed }}
# Replaces the distro listener with the {{ ssh_mesh_interface }}-only socket.
# Open sessions survive: they run in logind scopes and the sshd units use
# KillMode=process. Any failure restores the distro listener at once.
set -eu
trap '/usr/local/sbin/ssh-mesh-revert' EXIT
systemctl disable {{ ssh_mesh_disable_units }}
systemctl stop {{ ssh_mesh_service }} {{ ssh_mesh_socket }}
systemctl start {{ ssh_mesh_socket }}
trap - EXIT
@@ -1,7 +0,0 @@
# {{ ansible_managed }}
# sshd holds the socket's listener; it stops with the socket, so it never keeps
# a listener on a removed {{ ssh_mesh_interface }}. Replaces the Requires= that
# enabling {{ ssh_mesh_socket }} would add.
[Unit]
BindsTo={{ ssh_mesh_socket }}
After={{ ssh_mesh_socket }}
@@ -1,8 +0,0 @@
# {{ ansible_managed }}
# Listen on {{ ssh_mesh_interface }} only. The unit is disabled; the udev rule
# starts it whenever {{ ssh_mesh_interface }} appears.
[Socket]
ListenStream=
ListenStream=22
BindToDevice={{ ssh_mesh_interface }}
BindIPv6Only=both
@@ -1,12 +0,0 @@
# {{ ansible_managed }}
# Started by the udev rule whenever {{ ssh_mesh_interface }} appears; BindToDevice
# stops it when the interface goes away. No [Install]: waiting for the device
# from sockets.target would hold up boot until NetBird is up.
[Unit]
Description=OpenSSH on {{ ssh_mesh_interface }} only
[Socket]
ListenStream=22
BindToDevice={{ ssh_mesh_interface }}
BindIPv6Only=both
Accept=yes
@@ -1,3 +0,0 @@
# {{ ansible_managed }}
# NetBird deletes and recreates {{ ssh_mesh_interface }} on every engine restart.
SUBSYSTEM=="net", KERNEL=="{{ ssh_mesh_interface }}", ENV{SYSTEMD_WANTS}+="{{ ssh_mesh_socket }}"
@@ -1,10 +0,0 @@
---
# No packaged socket; sshd.socket (Accept=yes) pairs with the packaged
# per-connection sshd@.service, since sshd -D cannot take a listening fd.
ssh_mesh_socket: sshd.socket
ssh_mesh_service: sshd.service
ssh_mesh_distro_listener: sshd.service
ssh_mesh_disable_units: sshd.service
ssh_mesh_socket_file: /etc/systemd/system/sshd.socket
ssh_mesh_socket_template: sshd.socket.j2
ssh_mesh_service_dropin: ""
@@ -1,10 +0,0 @@
---
# Ubuntu's ssh.socket (Accept=no) hands its listener to ssh.service. The
# drop-in is named like sshd-socket-generator's output so it masks it.
ssh_mesh_socket: ssh.socket
ssh_mesh_service: ssh.service
ssh_mesh_distro_listener: ssh.socket
ssh_mesh_disable_units: ssh.service ssh.socket
ssh_mesh_socket_file: /etc/systemd/system/ssh.socket.d/addresses.conf
ssh_mesh_socket_template: ssh.socket-addresses.conf.j2
ssh_mesh_service_dropin: /etc/systemd/system/ssh.service.d/ssh-mesh-only.conf
@@ -14,8 +14,3 @@ uptime_kuma_openbao_mount: homelab
uptime_kuma_openbao_path: infrastructure/uptime-kuma
uptime_kuma_openbao_smtp_path: infrastructure/uptime-kuma/smtp
uptime_kuma_openbao_metrics_path: infrastructure/uptime-kuma-metrics
# Token with policy uptime-kuma-vps, looked up like the bao CLI does:
# VAULT_TOKEN, else the token helper file written by `bao login -method=oidc`.
uptime_kuma_openbao_token: >-
{{ lookup('ansible.builtin.env', 'VAULT_TOKEN')
or (lookup('ansible.builtin.file', '~/.vault-token', errors='ignore') | default('', true)) }}
+36 -52
View File
@@ -2,58 +2,43 @@
# Admin bootstrap runs with the Traefik route disabled so /setup is never
# reachable from outside; the route is enabled once the admin exists.
# OpenBao's KV v2 HTTP API directly, so the controller needs no hvac library.
# Reads also run in check mode.
- name: Check for an OpenBao token
ansible.builtin.assert:
that:
- uptime_kuma_openbao_token | length > 0
fail_msg: "No OpenBao token: run 'bao login -method=oidc' (policy uptime-kuma-vps) or set VAULT_TOKEN."
run_once: true
- name: Read admin credentials from OpenBao
ansible.builtin.uri:
url: "{{ uptime_kuma_openbao_url }}/v1/{{ uptime_kuma_openbao_mount }}/data/{{ uptime_kuma_openbao_path }}"
headers:
X-Vault-Token: "{{ uptime_kuma_openbao_token }}"
community.hashi_vault.vault_kv2_get:
url: "{{ uptime_kuma_openbao_url }}"
engine_mount_point: "{{ uptime_kuma_openbao_mount }}"
path: "{{ uptime_kuma_openbao_path }}"
auth_method: token
delegate_to: localhost
become: false
run_once: true
check_mode: false
register: _uptime_kuma_secret_read
register: _uptime_kuma_secret
no_log: true
- name: Read SMTP credentials from OpenBao
ansible.builtin.uri:
url: "{{ uptime_kuma_openbao_url }}/v1/{{ uptime_kuma_openbao_mount }}/data/{{ uptime_kuma_openbao_smtp_path }}"
headers:
X-Vault-Token: "{{ uptime_kuma_openbao_token }}"
community.hashi_vault.vault_kv2_get:
url: "{{ uptime_kuma_openbao_url }}"
engine_mount_point: "{{ uptime_kuma_openbao_mount }}"
path: "{{ uptime_kuma_openbao_smtp_path }}"
auth_method: token
delegate_to: localhost
become: false
run_once: true
check_mode: false
register: _uptime_kuma_smtp_read
register: _uptime_kuma_smtp
no_log: true
# 404 until the role has stored a key for the first time.
- name: Read metrics API key from OpenBao
ansible.builtin.uri:
url: "{{ uptime_kuma_openbao_url }}/v1/{{ uptime_kuma_openbao_mount }}/data/{{ uptime_kuma_openbao_metrics_path }}"
headers:
X-Vault-Token: "{{ uptime_kuma_openbao_token }}"
status_code: [200, 404]
community.hashi_vault.vault_kv2_get:
url: "{{ uptime_kuma_openbao_url }}"
engine_mount_point: "{{ uptime_kuma_openbao_mount }}"
path: "{{ uptime_kuma_openbao_metrics_path }}"
auth_method: token
delegate_to: localhost
become: false
run_once: true
check_mode: false
register: _uptime_kuma_metrics_read
no_log: true
- name: Extract the OpenBao secrets
ansible.builtin.set_fact:
_uptime_kuma_secret: "{{ _uptime_kuma_secret_read.json.data.data }}"
_uptime_kuma_smtp: "{{ _uptime_kuma_smtp_read.json.data.data }}"
_uptime_kuma_metrics_key: "{{ _uptime_kuma_metrics_read.json.data.data.api_key if _uptime_kuma_metrics_read.status == 200 else '' }}"
register: _uptime_kuma_metrics_secret
failed_when:
- _uptime_kuma_metrics_secret.failed | default(false)
- "'Invalid or missing path' not in (_uptime_kuma_metrics_secret.msg | default(''))"
no_log: true
- name: Ensure Uptime Kuma directories exist
@@ -117,6 +102,8 @@
headers:
Authorization: "Basic {{ (':' ~ _uptime_kuma_metrics_key) | b64encode }}"
status_code: [200, 401]
vars:
_uptime_kuma_metrics_key: "{{ (_uptime_kuma_metrics_secret.secret | default({})).api_key | default('') }}"
when: _uptime_kuma_metrics_key | length > 0
register: _uptime_kuma_metrics_check
no_log: true
@@ -161,13 +148,13 @@
args:
chdir: "{{ uptime_kuma_dir }}"
environment:
KUMA_SETUP_USERNAME: "{{ _uptime_kuma_secret.username }}"
KUMA_SETUP_PASSWORD: "{{ _uptime_kuma_secret.password }}"
KUMA_SETUP_USERNAME: "{{ _uptime_kuma_secret.secret.username }}"
KUMA_SETUP_PASSWORD: "{{ _uptime_kuma_secret.secret.password }}"
KUMA_BASE_URL: "{{ uptime_kuma_base_url }}"
KUMA_SMTP_HOST: "{{ _uptime_kuma_smtp.server }}"
KUMA_SMTP_PORT: "{{ _uptime_kuma_smtp.port }}"
KUMA_SMTP_USERNAME: "{{ _uptime_kuma_smtp.username }}"
KUMA_SMTP_PASSWORD: "{{ _uptime_kuma_smtp.token }}"
KUMA_SMTP_HOST: "{{ _uptime_kuma_smtp.secret.server }}"
KUMA_SMTP_PORT: "{{ _uptime_kuma_smtp.secret.port }}"
KUMA_SMTP_USERNAME: "{{ _uptime_kuma_smtp.secret.username }}"
KUMA_SMTP_PASSWORD: "{{ _uptime_kuma_smtp.secret.token }}"
KUMA_SMTP_FROM: "{{ uptime_kuma_smtp_from }}"
KUMA_SMTP_TO: "{{ uptime_kuma_smtp_to }}"
KUMA_METRICS_ROTATE: "{{ 'false' if (_uptime_kuma_metrics_check.status | default(0)) == 200 else 'true' }}"
@@ -192,16 +179,13 @@
no_log: true
- name: Write the metrics API key to OpenBao
ansible.builtin.uri:
url: "{{ uptime_kuma_openbao_url }}/v1/{{ uptime_kuma_openbao_mount }}/data/{{ uptime_kuma_openbao_metrics_path }}"
method: POST
headers:
X-Vault-Token: "{{ uptime_kuma_openbao_token }}"
body_format: json
body:
data:
api_key: "{{ _uptime_kuma_new_metrics_key.content | b64decode }}"
status_code: [200, 204]
community.hashi_vault.vault_kv2_write:
url: "{{ uptime_kuma_openbao_url }}"
engine_mount_point: "{{ uptime_kuma_openbao_mount }}"
path: "{{ uptime_kuma_openbao_metrics_path }}"
auth_method: token
data:
api_key: "{{ _uptime_kuma_new_metrics_key.content | b64decode }}"
delegate_to: localhost
become: false
run_once: true

Some files were not shown because too many files have changed in this diff Show More