CI steps run inside Docker containers on the wp-agents, and Docker gives
containers no IPv6 by default — so deploys to IPv6-only hosts fail with
"Network unreachable" even when the agent host itself has v6 (confirmed:
the deploy-netbird-proxy pipeline can't SSH the v6-only proxy-1).
Write /etc/docker/daemon.json with ipv6 + a ULA fixed-cidr-v6 + ip6tables
(IPv6 NAT), so step containers get NAT'd v6 egress through the host.
Caveats:
- Requires Docker 27+ (ip6tables stable) and the agent HOSTS to have v6
egress themselves (the .srv VLAN must route v6).
- Applying notifies a docker restart, which interrupts running pipelines
(including the one applying it, if run via a pipeline on the agents).
Apply deliberately — out-of-band, or expect the triggering run to die on
the restart and re-run it.
Unblocks the deploy-netbird-proxy pipeline (PR #47) reaching proxy-1.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rather than reinventing docker install inside netbird-proxy, extend the
shared docker-install role to support Arch (proxy-1 runs Arch): guard the
existing apt/Docker-repo tasks with os_family == Debian and add an Arch
branch (package: docker, docker-compose). service enable + docker group
stay OS-agnostic. Debian hosts (netbird-server) are unaffected.
netbird-proxy drops its inline docker tasks; the play goes back to the
standard common + docker-install + netbird-proxy role stack.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
proxy-1 is Arch Linux, where the apt-based docker-install/common roles
fail. Make the netbird-proxy role self-contained: install Docker via the
OS package manager (pacman on Arch, apt on Debian) + enable the service,
and drop common/docker-install from the play.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Edge HA for the public ingress: run a second netbirdio/reverse-proxy as a
NetBird proxy-cluster member on proxy-1 (Netcup), so a proxy-host failure
no longer takes the public edge down. NetBird forms an HA cluster from
instances sharing the same NB_PROXY_DOMAIN; traffic is distributed by DNS
round-robin across the members.
https://docs.netbird.io/selfhosted/maintenance/scaling/multiple-proxy-instances
New `netbird-proxy` role (proxy-only host): just the reverse-proxy
container bound to :443 (no Traefik — nothing else competes for the port),
NB_PROXY_DOMAIN=proxy.renner.uno (matches VPS-1), its OWN per-instance
token, enrolling against https://netbird.renner.uno. Independent ACME
(option A) — each member issues its own tls-alpn-01 certs, so VPS-1's
working proxy is untouched. Reuses the docker-install role; deploy via the
deploy-netbird-proxy woodpecker pipeline.
Deployment is GATED on human prereqs (documented in the play):
1. Generate proxy-1's token on the management server and add
vault_netbird_proxy1_token to the ansible vault.
2. proxy-1.renner.uno: Ubuntu + root SSH, DNS for its own name -> its IP.
3. AFTER deploy + verify: repoint proxy.renner.uno from its CNAME to
netbird.renner.uno (management host) to A/AAAA records listing BOTH
proxy IPs, so traffic round-robins across the cluster.
netbird_version in the role MUST be kept in sync with netbird-server.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Smart Fades audio-analysis provider added in MA 2.9 loads an ML model
and reads the pod cgroup memory limit to gate startup. At the current 2Gi
limit it refuses to run:
Provider(instance) Smart Fades can not run on this system: at least 4GB
of RAM is required (2.0GB detected).
so the provider shows 'requires attention' / RECONFIGURE in the UI.
Loudness Analysis (lightweight) is unaffected. CPU/AVX2 (x86-64-v3) and the
2-core minimum are already met; RAM is the sole blocker.
Upstream recommends 6GB+ and warns the 4GB minimum risks OOM under load.
k8s-edge is a 12GB VM currently using ~4GB, so set the limit to the
recommended 6Gi (request bumped 512Mi->1Gi). On next reconcile MA restarts
and Smart Fades passes the requirement check.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Convert the single-replica Deployment to a DaemonSet so every node runs
its own rootless NetBird routing peer. With the mesh route/nameserver
pointed at the k8s-ingress-router group, NetBird distributes and fails the
route over across all per-node peers — no single point of failure if a
node or pod drops.
- kind Deployment -> DaemonSet (drop replicas/Recreate; RollingUpdate
maxUnavailable=1 so peers roll one node at a time)
- per-node peer name: NB_HOSTNAME=netbird-ingress-$(NODE_NAME) via the
downward API, so each peer shows its node in the dashboard
- securityContext/image/probes/egress CNP unchanged (rootless, capless)
NetBird-side: ensure the 10.2.99.0/24 + fd99::/64 route uses the
k8s-ingress-router GROUP as its routing peer (not a single peer) so all
DaemonSet members serve it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
All 8 public hostnames have moved off the tunnel to the NetBird reverse
proxy, so cloudflared routes nothing but a 404. Remove the component
entirely:
- delete kubernetes/infrastructure/cloudflared/ (Deployment, ConfigMap,
tunnel-credentials Secret, Kustomization) + drop it from
infrastructure/kustomization.yaml (Flux prunes the workload)
- remove the cloudflared image-automation policy + README row
- drop the cloudflared-to-ingress-only CiliumNetworkPolicy
- scrub stale "cloudflared" comments (netbird-ingress, k8s-gateway,
ingress-default-deny external-entry note)
Cloudflare is still used for DNS (external-dns-cloudflare) and
cert-manager DNS-01 — only the tunnel is retired. The Cloudflare-side
tunnel itself (Zero Trust) can be deleted separately.
MERGE LAST — only after every per-service proxy cutover is live and
verified; cloudflared already serves no routes, so this just removes the
dormant workload.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drop the external-dns/cloudflare label from the Authentik ingress (helm
values) and remove the auth.renner.uno cloudflared route. auth must be
served with NO proxy auth (it is the IdP every other SSO redirects to).
This is the last of the 8 public names; cloudflared now routes nothing
but a 404 catch-all.
MERGE ONLY AFTER the auth proxy service + DNS are live, then roll
cloudflared, or all SSO logins break in the gap.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drop the external-dns/cloudflare label from the gatus Ingress and remove
the status.renner.uno cloudflared route, moving the public edge to the
NetBird reverse proxy. Gatus stays public (no proxy auth — it's the
status page). LAN/mesh via status.k.renner.uno -> Traefik is unaffected.
MERGE ONLY AFTER the status proxy service + DNS are live, then roll
cloudflared, or status.renner.uno goes dark in the gap.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drop the external-dns/cloudflare label from the music-assistant Ingress
and remove the music.renner.uno cloudflared route, moving the public edge
to the NetBird reverse proxy. Music Assistant keeps its HA-delegated
login (no proxy auth added). LAN/mesh via music.k.renner.uno -> Traefik
is unaffected.
MERGE ONLY AFTER the music proxy service + DNS are live, then roll
cloudflared, or music.renner.uno goes dark in the gap.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drop the external-dns/cloudflare label from the HA Ingress and remove the
home.renner.uno cloudflared route, moving the public edge to the NetBird
reverse proxy. HA keeps its own login (no proxy auth added; the prior
Authentik forward-auth attempt stays reverted). LAN/mesh -> Traefik via
home.k.renner.uno is unaffected.
MERGE ONLY AFTER the home proxy service + DNS are live, then roll
cloudflared, or home.renner.uno goes dark in the gap.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drop the external-dns/cloudflare label from the immich Ingress and remove
the immich.renner.uno cloudflared tunnel route, moving the public edge to
the NetBird reverse proxy. immich keeps its own login (no proxy auth
added). The .k.renner.uno LAN alias and legacy photos.renner.uno were
already tunnel-404 via the shared label; LAN/mesh -> Traefik is unaffected.
MERGE ONLY AFTER the immich proxy service + DNS are live, then roll
cloudflared, or immich.renner.uno goes dark in the gap.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drop the external-dns/cloudflare label from the paperless and docs
Ingresses and remove their cloudflared tunnel routes, so the public edge
for paperless.renner.uno / docs.renner.uno moves to the NetBird reverse
proxy. Service behaviour (the paperless-gate Authentik outpost, the
docs->paperless redirect) is unchanged — this only swaps the public
ingress edge. LAN/mesh resolution -> Traefik is unaffected.
MERGE ONLY AFTER the paperless+docs proxy services + DNS are live, and
roll cloudflared after, or the public names go dark in the gap (as
audiobooks did when done out of order).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
audiobooks moved off the Cloudflare tunnel to the NetBird reverse proxy
(#36 removed the external-dns label). Remove the now-dead tunnel ingress
rule so cloudflared no longer advertises a route for it.
Note: cloudflared subPath-mounts this ConfigMap and does NOT auto-reload
— a rollout restart of the cloudflared Deployment is needed to apply.
Functionally moot once the orphaned Cloudflare DNS record is deleted.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Stop publishing audiobooks.renner.uno through the Cloudflare tunnel so
the public name can be repointed at the NetBird reverse proxy (which is
exposing it with Authentik SSO at the edge). LAN/mesh resolution (UniFi /
routing peer -> Traefik) is unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The first smoke run went green but created no issue: the post step used
`jq | curl -sf | jq`, so the pipe's exit status came from the trailing jq
and a failed POST was swallowed.
Rewrite the post step to capture each write's HTTP status explicitly via
`-w '%{http_code}'`, print it, and fail the step on any non-2xx (dumping the
Gitea error body). Payloads are written to files instead of piped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a Woodpecker cron pipeline that runs a read-only Claude Code agent to
audit the components NOT covered by existing automation, and posts the report
to a single persistent Gitea issue.
Scope (complements, does not duplicate, existing automation):
- Flux image-automation already bumps enrolled container tags.
- The daily security-scan agent already owns CVEs.
- This agent audits manually-pinned currency instead: Talos / Kubernetes /
Flux versions, Helm chart minor/major bumps held back by ~MAJOR.MINOR.0
constraints, images excluded from automation, and Ansible/OpenTofu drift.
The agent is read-only (allowedTools: Read,Glob,Grep,Write,WebFetch,WebSearch;
no Edit/Bash) so it can never mutate the repo. It writes a report to
/tmp/update-audit.md; the pipeline find-or-creates the "Infrastructure update
audit" Gitea issue and refreshes its body.
Cron uses Woodpecker's documented `cron:` name filter (rather than the injected
PIPELINE var the other pipelines use) so the schedule is CLI-creatable:
woodpecker repo cron add --repo gitea_admin/homelab \
--name update-audit --branch master --schedule '@weekly'
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The netbird-server role had no deploy pipeline, so pinning its image
versions (#32) landed in git but never reached the VPS — there was
nothing to run install-netbird-server.yml. Every other Ansible-managed
service (caddy-ha, …) has a path-triggered deploy-* pipeline that runs
its play in the ci-ansible runner; netbird-server was simply missing one.
This mirrors deploy-caddy.yml exactly: push-triggered on the role, the
play, and this file; manual via PIPELINE==deploy-netbird-server. Merging
this re-triggers on its own path and runs install-netbird-server.yml,
applying the pinned 0.73.2 / v2.80.0 images on netbird.renner.uno.
Note: a management bump runs a one-way store migration — snapshot the
VPS before merging if the live version is well behind 0.73.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The mesh control plane (management/signal/relay/reverse-proxy/dashboard)
ran on :latest — unpinned drift with no rollback and a non-reproducible
running version. Pin via two role vars:
netbird_version: 0.73.2 # mgmt/signal/relay/reverse-proxy
netbird_dashboard_version: v2.80.0 # dashboard (independent versioning)
0.73.2 is the current stable (0.75.0 is only -rc) and matches the agents
already on the mesh. >=0.71 is the floor that enables IPv6 overlay
addressing, which is the prerequisite for advertising fd99::/64 into the
mesh (IPv6-first ingress) — a follow-up, not done here.
Takes effect only when plays/install-netbird-server.yml is run on the
VPS. Before applying: confirm the currently-running version and SNAPSHOT
the host — a management version bump runs a one-way store migration.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The kernel-mode routing peer could not deliver mesh traffic to cluster
services: it L3-forwards packets to ClusterIP/LB VIPs, but Cilium
socket-LB only DNATs a pod's OWN socket connections, not forwarded
packets — so the VIP destination is blackholed (confirmed via conntrack:
the flow stays [UNREPLIED] while the pod's own socket to the same VIP
answers). It also needed NET_ADMIN/SYS_ADMIN, which Pod Security
baseline forbids in app namespaces.
The rootless image (userspace wireguard-go) fixes both: the netbird
process terminates the tunnel in userspace and reaches services by
opening its OWN sockets — which socket-LB DNATs correctly — and needs
no kernel WireGuard, so it runs fully unprivileged (drop ALL caps,
runAsNonRoot:65532). This is the documented community pattern for
running a NetBird routing peer in Kubernetes.
- image -> netbirdio/netbird:v0.73.2-rootless
- securityContext -> capless, runAsNonRoot, no privilege escalation
- add USER/HOME + fsGroup so the state emptyDir is writable by 65532
- image-automation policy now tracks vX.Y.Z-rootless tags
- existing setup-key secret, CNP, and NetBird-side routes/nameserver
are unchanged — they finally work now that forwarding is userspace
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Proof-of-concept for "direct mesh exposure" — the alternative to the
standalone routing peer, which can't deliver mesh traffic to cluster
LoadBalancer/ClusterIP VIPs (Cilium socket-LB only translates a pod's
OWN socket connections, not traffic a peer forwards on behalf of mesh
clients; confirmed via conntrack on netbird-ingress).
Here a non-privileged netbird sidecar shares the audiobookshelf pod's
network namespace, so the pod becomes its own mesh peer
(audiobookshelf.nb.renner.uno). Mesh clients connect straight to the
pod's mesh IP:80 — traffic is delivered locally inside the netns, so
there is no L3 forwarding and socket-LB is never in the path.
- non-privileged; NET_ADMIN/SYS_ADMIN/SYS_RESOURCE (WireGuard minimum)
- reuses the existing reusable setup key (new sops secret in this ns)
- existing audiobookshelf CNP already permits the sidecar's egress
(world + kube-dns), so no network-policy change needed
Validation: from a mesh peer, `dig audiobookshelf.nb.renner.uno` then
`curl http://audiobookshelf.nb.renner.uno` — expect the app directly,
no Traefik, no VIP. A NetBird access policy must allow the client's
group to reach this peer's group.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces the placeholder in secret.sops.yaml with the reusable+ephemeral
setup key created on netbird.renner.uno (auto-group k8s-ingress-router).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a second, parallel ingress path over the NetBird overlay mesh
(100.96.0.0/16), in addition to the existing Cloudflare path. A new
in-cluster NetBird routing peer enrolls with the self-hosted management
server (netbird.renner.uno) and advertises the Traefik VIP (10.2.99.86)
into the mesh, so every Traefik-fronted service becomes reachable from
mesh peers with no per-service work — mesh DNS (10.2.40.53) already
resolves <app>.renner.uno to that VIP; the only gap was that the
bgp-pool CIDR wasn't routable over the overlay.
Modeled on the cloudflared Deployment and held to the same in-cluster
firewall discipline: a dedicated CiliumNetworkPolicy
(netbird-ingress-egress) restricts egress to traefik + DNS + the NetBird
control/data plane only. Unlike the upstream NetBird operator the pod is
NOT privileged; the NET_ADMIN/SYS_ADMIN/SYS_RESOURCE caps are the
irreducible minimum for an L3 WireGuard routing peer.
- kubernetes/infrastructure/netbird-ingress/ (deployment, sops secret, kustomization)
- image-automation policy pinned to bare-numeric semver (excludes -rootless/-rc)
- netbird-ingress-egress CNP in network-policies/ingress-isolation.yaml
Requires one-time NetBird-side setup (dashboard/API): create a reusable
+ ephemeral setup key (auto-group k8s-ingress-router), replace the
placeholder in secret.sops.yaml, add a network route for 10.2.99.86/32
via that peer, and an access policy for the consuming mesh groups.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Finish line of the backup migration — nas-1's Garage is no longer a
cluster-backup dependency, so:
- drop the prometheus `garage` scrape job (s3.nas-1.renner.uno:39033) +
the garage-metrics-token secret (now orphaned)
- delete prometheusrule-garage.yaml (GarageDown / GarageScrapeFailing) —
it would fire every night once nas-1 powers off
Replace it with outage visibility for the new offsite target:
- add https://fsn1.your-objectstorage.com to the blackbox-http probe
(bare endpoint returns 200)
- new HetznerBackupEndpointDown alert on probe_success == 0 for 10m
Velero outages are already covered by VeleroBackupValidationFailure /
VeleroBackupStale. A granular CNPG "WAL archiving behind" alert needs
CNPG PodMonitors (enablePodMonitor=false today, metrics not scraped) —
left as a follow-up rather than shipping an alert on an absent metric.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Stage 2 of the offsite backup migration, applying the pattern validated
end-to-end on paperless (WAL archive + base backup + scratch restore).
CNPG (authentik ak-pg, immich im-pg):
- repoint ObjectStore -> s3://renner-uno-k8s-cnpg/<srv> @ fsn1
- creds barman-cloud-creds -> hetzner-s3-creds (wired into kustomizations)
- drop encryption: AES256 (Hetzner has no barman-usable SSE; see #646)
Velero:
- BSL -> renner-uno-k8s-velero @ fsn1, virtual-hosted (s3ForcePathStyle
false), region fsn1, checksumAlgorithm "" (required on Hetzner)
- credentials -> cloud-credentials-hetzner
- Kopia encrypts client-side, so velero keeps at-rest encryption.
Garage creds/secrets left in place for one-revert rollback. After this
deploys + validates, the Garage Prometheus scrape/alerts can be dropped
and nas-1 freed for nightly shutdown.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Hetzner cutover broke WAL archiving and base backups with
InvalidArgument: barman requests SSE-S3 (encryption: AES256), but
Hetzner Object Storage supports only SSE-C, which the barman-cloud
plugin cannot use (cnpg plugin-barman-cloud #646). Confirmed live —
aws-cli PUT --sse AES256 fails the same way, plain PUT succeeds.
Remove encryption from wal/data so backups use no SSE header. Trade-off
(accepted): CNPG backups are not encrypted at rest on Hetzner; transit
is TLS and the bucket is private. Revisit if/when the plugin gains SSE-C.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
k.rennerandClaude Opus 4.8
security-bot