The `seed` initContainers in home-assistant and matter-server were
one-shot migration importers that streamed the HAOS backup from
s3://k8s-migration on the nas-1 Garage S3. The migration is done (live
pods carry the idempotency sentinels and skip the seed), nas-1 is now
off, and the k8s-migration bucket doesn't exist on Hetzner — so on a
fresh PVC the seed would hang the pod in Init. DR restore is via Velero.
Removed:
- home-assistant: seed initContainer.
- matter-server: seed initContainer + its import-script volume.
- garage-migration-creds Secret (migration-only S3 creds) + HA kustomization ref.
- matter-server-haos-import-script ConfigMap + matter kustomization ref.
- matter-server Flux Kustomization dependsOn: home-assistant (it existed
solely so the seed could mount garage-migration-creds; matter is now
independent).
Kept: home-assistant-backup-password Secret — it's the HA backup
encryption credential, not migration scaffolding.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
rebuild-k8s.yml was the one-shot PVE->K8s migration/rebuild pipeline; it
is no longer needed and still pointed at the retired Garage S3 on the
now-off nas-1. Remove it (DR restore is via Velero + flux bootstrap).
The live cluster-access path is kept and migrated off Garage onto Hetzner
Object Storage (renner-uno-k8s-secrets, fsn1.your-objectstorage.com,
region fsn1, hetzner_s3_* keypair):
- scripts/ci/fetch-kubeconfig.sh: Garage -> Hetzner; env vars renamed
GARAGE_SECRETS_* -> S3_ACCESS_KEY_ID/S3_SECRET_ACCESS_KEY.
- .woodpecker/e2e.yml, e2e-test.yml: pass hetzner_s3_access_key/secret_key
(the live cron e2e pipeline needs this — it was broken against dead nas-1).
- scripts/import-cluster-access.sh: Garage -> Hetzner endpoint/bucket/region.
NOTE: the workstation aws-cli "secrets" profile must now hold the Hetzner
Object Storage keypair.
Also: delete the orphaned test-garage-smoke play + pipeline, and fix the
two dangling rebuild-k8s.yml references (zfs-tuning play comment;
update-audit prompt Talos-pin file list).
Assumes cluster-access objects (kubeconfig.sops/talosconfig.sops) were
migrated into renner-uno-k8s-secrets; with rebuild-k8s gone, nothing
auto-refreshes them, which is fine for a stable cluster.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The netbird-tunnel.service.j2 reverse-SSH tunnel template was never
instantiated by any task in the role, and its target
netbird_auth_ingress_host=10.2.45.100 was the Caddy HA keepalived VIP,
destroyed in PR #70. The VPS reaches homelab auth over the NetBird mesh
now (netbird-ingress DaemonSet -> Traefik fd99::86), so the tunnel is
dead. Remove the template and its only-consumer vars
(netbird_auth_tunnel_port/ingress_host/ingress_port, netbird_tunnel_ssh_key).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Residue from the Caddy HA -> Traefik migration (Caddy LXCs destroyed in
PR #70):
- install-error-page.yml, install-gitea-proxy.yml: both target the
destroyed host caddy.srv.renner.uno (the gitea-proxy play also wrote a
Caddyfile reverse_proxy block + cloudflare import; gitea is now served
by a Traefik Ingress in kubernetes/apps/gitea).
- grafana-observability: drop the caddy-overview dashboard (monitored
caddy_http_* metrics from the dead Caddy HA) and its two deploy/cleanup
loop entries.
Note: a previously-provisioned Network/caddy-overview.json may still
linger in the live Grafana until the dashboards are re-synced or it's
removed in the UI; it just shows no data.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The netbird-ingress DaemonSet, netbird reverse-proxy, and Traefik have
fully superseded the old ingress tiers, which have been stopped (not
deleted) since the 2026-05-09 PVE cutover. Remove the dead code:
- Caddy HA: LXCs 232/233/234 (opentofu/ingress.tf), caddy-ha role,
install-caddy-ha + install-crowdsec-caddy plays, caddy cred vars,
proxmox_caddy_ha group_vars, build-caddy/deploy-caddy pipelines,
ci-caddy-builder image.
- cloudflared: LXC 117, cloudflared-lxc role, install-cloudflared play,
cloudflare_tunnel/variables_cloudflare disabled tofu stubs.
- netbird-gateway: LXC 260 (replaced by the in-cluster netbird-ingress
DaemonSet), netbird-gateway role + play.
- backup_jobs.tf: drop VMIDs 117/232/233/234/260 from the nightly vzdump
include list (those LXCs are being destroyed).
- netbird-server: point the setup-key hint at netbird-ingress/netbird-proxy.
Kept deliberately:
- crowdsec role (reusable; basis for the pending Traefik bouncer).
- netbird-client role + play (loki/prometheus LXCs are tagged
netbird_client, so it remains live architecture).
Note: applying the opentofu change will plan a DESTROY of stopped LXCs
117/232/233/234/260 — review the plan before apply.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
security_bot_token is scoped for code/PRs only; the issue POST in the
post step returned HTTP 403 'token does not have ... write:issue', so the
audit ran green but silently created no issue. Switch GITEA_TOKEN to the
gitea_token secret — the broader machine account already used by
build-caddy and build-ci-images for release-asset and registry writes —
which is the most likely existing secret to also hold write:issue.
No new token to provision; audit issues become attributed to gitea_token's
owner instead of the security bot.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Remove .woodpecker/migrate-tfstate.yml — the Garage(nas-1)->Hetzner state
migration it performed is complete and verified; it was a manual one-off.
- netbird-proxy no longer needs the /opt/netbird-proxy/certs and /state
bind-mount dirs (those are Docker named volumes now); only the compose dir
is required.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The management cached Authentik's JWKS at startup and never refreshed it, so
when Authentik rotated its JWT signing key the management could no longer
verify tokens ('unable to find appropriate key' -> 'token invalid'), breaking
the dashboard. Set IdpSignKeyRefreshEnabled=true so management re-fetches the
JWKS on an unknown key id instead of caching forever.
NOTE: deploying this redeploys VPS-1 (the IdP host) and restarts the
management container.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Same root-owned-bind-mount issue as /certs: the non-root proxy couldn't
write netbird state ('failed to write active profile state / geolocation:
permission denied'). Use a named volume for /var/lib/netbird as well.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The proxy runs non-root; a root-owned bind mount (/opt/netbird-proxy/certs)
was unwritable ('open lock file ...: permission denied'), which would block
cert issuance once the host gets traffic. Switch to a named volume like
VPS-1, which inherits the image's writable ownership.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Remote reverse-proxy instances (proxy-1) connect to the management server
over Traefik via the ProxyService gRPC path. Traefik only routed
ManagementService, so the remote proxy's mapping stream got Traefik's 404
HTML. Add a router for /management.ProxyService/ to the same management
backend (h2c:33073). The local VPS-1 proxy is unaffected (it uses
management:33073 directly).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On Arch /etc/docker does not exist until something writes there; the
daemon.json template failed with 'Destination directory does not exist'.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The deployed proxy on proxy-1 (v6-only host) couldn't connect to the
management server: 'dial tcp [2a01:...]:443 / 162.55.50.97:443: network is
unreachable'. The host reaches mgmt over v6 fine, but Docker had no IPv6
(no daemon.json), so the container had no v6 egress and no v4 exists.
Add /etc/docker/daemon.json (ipv6 + ULA fixed-cidr-v6 + ip6tables NAT),
flush it (restart docker) before the container deploys, and put the proxy
service on the default bridge (network_mode: bridge) so it inherits that v6.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
State already migrated (migrate-tfstate pipeline; 45 resources verified on
Hetzner). Repoint providers.tf backend (bucket/region/endpoint/path-style)
and the deploy-infrastructure backend creds from the nas-1 Garage S3 to
Hetzner. nas-1 is now fully decommissionable.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The pulled state is compact (one line), so grep -c returned 1. Count
occurrences instead.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The per-line commands were parsed as YAML maps because of colons in echo
strings. Move the whole script into a single block scalar.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Migrates the OpenTofu remote state to Hetzner Object Storage. Garage is
read-only (state pull only), so reversible. Manual trigger only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Target bucket for migrating the OpenTofu remote state off the nas-1 Garage
S3 (s3.nas-1.renner.uno) to Hetzner Object Storage. State copy + backend
repoint follow in subsequent steps.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The agents are unprivileged Proxmox LXCs whose ip_config had only an ipv4
block — no IPv6 at all. So no matter how Docker/Woodpecker were configured
for v6 (daemon.json ip6tables, WOODPECKER_BACKEND_DOCKER_ENABLE_IPV6), the
step containers had no v6 uplink to NAT onto, and deploys to v6-only hosts
(proxy-1) failed 'Network unreachable'. IPv4 NAT works in these LXCs, so v6
NAT will too once the LXC actually has v6.
Add an optional ipv6 block to the proxmox_lxc module (auto/dhcp/static) and
set ipv6_address = "auto" (SLAAC off v40srv) for the agents.
NOTE: `initialization` is in the module's ignore_changes, so this does NOT
apply to running agents on a normal apply — they must be RECREATED to pick
up v6 (they're tagged `transient`). Recreate one at a time to keep CI up,
then re-run install-woodpecker-agent.yml on each.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The self-deploy restarts were unreliable, so Docker may have the daemon.json
on disk but never reloaded it. Check docker bridge EnableIPv6 and, if not
active, schedule a deferred restart. Idempotent once v6 is live. The check
result also surfaces (in the deploy log) whether Docker had v6 at all.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The daemon.json (PR #48) only gives the *default bridge* IPv6 — but
Woodpecker runs each step on a per-pipeline network, which it creates
WITHOUT IPv6 unless told to. So step containers still had no v6 and the
deploy to v6-only proxy-1 kept failing 'Network unreachable'.
Set WOODPECKER_BACKEND_DOCKER_ENABLE_IPV6=true on the agent so it creates
v6-enabled pipeline networks (pairs with the daemon.json ipv6/ip6tables
NAT). Also defer the woodpecker-agent restart (systemd-run timer) so the
agent self-deploy doesn't interrupt its own running pipeline.
Ref: https://woodpecker-ci.org/docs/administration/configuration/backends/docker
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
deploy-woodpecker-agent runs on an agent; a synchronous 'systemctl restart
docker' kills the job's own container before the play finishes (pipeline
994 died at docker-install, before the daemon.json task even ran). Schedule
the restart via a transient systemd timer (systemd-run --on-active=15s) so
the playbook completes first, then docker restarts and picks up the IPv6
daemon.json.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The woodpecker-agent role had no deploy pipeline — so the Docker IPv6
daemon.json (PR #48) landed in git but never reached the agents. Mirror
the other deploy-* pipelines: master + manual trigger, runs
install-woodpecker-agent.yml in the ci-ansible runner.
Merging self-triggers on its own path and applies the agent config. It
deploys the agents from an agent, so the Docker restart kills this job
mid-run (reported as failure) but still applies on both agents; re-trigger
deploy-netbird-proxy after.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Drop the invented per-instance token: a shared NB_PROXY_TOKEN with the
same NB_PROXY_DOMAIN is a valid HA cluster, so proxy-1 just reuses the
existing vault_netbird_proxy_token. No token to generate, no new vault
secret — removes that deploy gate entirely.
- Scope the deploy-netbird-proxy pipeline to master + manual so it stops
auto-running (and failing) on feature-branch pushes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
k.rennerandClaude Opus 4.8