feat(netbird): pull-based off-VPS backup of the controller store #417
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: konairius/homelab#417
Reference in New Issue
Block a user
Closes the last big SPOF-assessment item: the controller's management SQLite store (peers, setup keys, groups, routes) has zero backup coverage — losing the Hetzner VPS today means re-enrolling the entire mesh.
Why pull, not push
Verified from the VPS: both PBS names resolve to the public proxy (wildcard DNS) and nothing answers on 8007/443 — the established pbs-client pattern can't work there, and every push alternative needs a broad credential (Hetzner S3 keys or PBS token) living on an internet-facing host. Pulling inverts the trust: the only new secret is a dedicated ed25519 keypair, and its public half is pinned in the VPS's authorized_keys with
restrict,command="/usr/local/bin/netbird-backup-dump"— even a full cluster compromise gets exactly one capability: reading a backup dump.Pieces
sqlite3 .backup(online-backup API — consistent against the live writer, unlike copying store.db) +/opt/netbird/config, streams tar.gz to stdout; forced-command key entry.kubernetes/infrastructure/netbird-backup, new namespace): CronJob nightly 02:45 ssh-pulls onto a 5 Gi Ceph PVC, gzip-verifies before renaming (no truncated dumps), keeps 14; ExternalSecret syncs the key from OpenBao; restricted-profile pod (runs as 65534, TOFU known_hosts pinned on the PVC).netbird-backup-daily(07:45, PVC-only, snapshotMoveData, 30 d TTL — same pattern as the other schedules).NetbirdVpsBackupStale(critical >26 h) andNetbirdVpsBackupNeverRan(warning) — the backup cannot die silently, and the second alert also nags until initial provisioning is done.⚠️ One-time owner step (required before it works)
The private key is staged at
~/.config/netbird-backup-key.pending(0600) on the workstation. Store it and shred the staging copy:Until then the ExternalSecret can't sync and
NetbirdVpsBackupNeverRanfires (by design).Verification plan after merge + provisioning
Trigger a manual run (
kubectl -n netbird-backup create job --from=cronjob/netbird-vps-backup verify-1), confirm a valid tar.gz lands on the PVC and listsstore.db+config/, and confirm the Velero schedule picks the PVC up on its next run.Note: touches
monitoring-rules/kustomization.yaml, same as #415 — whichever merges second may need a trivial conflict resolution.🤖 Generated with Claude Code
Closes the last big SPOF-assessment item: the management SQLite store (peers, setup keys, groups, routes) had zero backup coverage — losing the Hetzner VPS meant re-enrolling the entire mesh. The VPS cannot reach any PBS (both PBS names resolve to the public proxy, ports closed), so the cluster pulls instead of the VPS pushing: - VPS (netbird-server role): a dump script takes a CONSISTENT snapshot via sqlite3 .backup (online-backup API, safe against the live writer) plus /opt/netbird/config and streams a tar.gz to stdout. A dedicated ed25519 key is authorized for root restricted to exactly that script (restrict,command=…) — the puller can run nothing else. - Cluster (kubernetes/infrastructure/netbird-backup): nightly CronJob (02:45) ssh-pulls the dump onto a 5Gi Ceph PVC, gzip-verifies before renaming, keeps 14 dumps; ExternalSecret syncs the private key from OpenBao (homelab/infrastructure/netbird-backup, one-time manual provisioning like the rotator bootstraps); restricted-profile pod. - Offsite: new Velero Schedule netbird-backup-daily (07:45, PVC-only, snapshotMoveData, 30 d TTL) ships the PVC to Hetzner Object Storage. - Alerts: NetbirdVpsBackupStale (critical, >26 h without success) and NetbirdVpsBackupNeverRan (warning — catches the unprovisioned-secret state so the backup cannot die silently). Restore: untar over /opt/netbird/{management,config} after the role rebuilds the stack. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>🤖 Automated review — model
CLI default(complexity: complex), commit37d9eafecf0661a3f59bf6520e5813f6af78d87d.Summary
This PR adds a pull-based, nightly SSH backup of the NetBird controller VPS's SQLite store and config, landing it on a new Ceph PVC (
netbird-backupnamespace) that a new Velero Schedule ships offsite, plus two staleness alerts. The design (restricted forced-command SSH key, onlinesqlite3 .backup, gzip-verify before rename, restricted pod security context) is sound and follows the repo's existing conventions closely. However, there is one concrete blocking gap that will prevent the backup from ever working, independent of the one-time OpenBao step called out in the PR description.🔴 Blocking
kubernetes/infrastructure/post-crds/openbao-clustersecretstore.yamlis not updated to allow the newnetbird-backupnamespace. TheClusterSecretStoreopenbaorestricts which namespaces may reference it viaspec.conditions.namespaces(lines 15-30), and the file's own maintenance comment (lines 12-14) states: "adding an ExternalSecret in a NEW namespace requires adding that namespace here first, or ESO reports 'not allowed to reference ClusterSecretStore' and the secret never syncs."kubernetes/infrastructure/netbird-backup/externalsecret.yaml:1-21creates anExternalSecretin the brand-newnetbird-backupnamespace referencing this same store, butnetbird-backupis absent from the allowlist. As it stands, theExternalSecretwill be rejected by ESO and will never sync — even after the owner completes thebao kv put …step the PR body describes as the only remaining blocker. This will surface asNetbirdVpsBackupNeverRanfiring forever, but the real root cause (missing allowlist entry) isn't mentioned anywhere in the PR body's verification plan, so it could easily be misdiagnosed as "provisioning still pending." Fix: addnetbird-backupto the sortednamespaceslist in that file (it's a one-line addition, same as every other consumer namespace there).🟡 Suggestions
ansible/roles/netbird-server/files/netbird-backup-dumpstreams the tar directly to the temp.partfile via shell redirection inkubernetes/infrastructure/netbird-backup/cronjob.yaml:201-208. Ifsshfails partway (e.g. connection drop mid-stream),set -eaborts before themv, but the partialnetbird-$ts.tar.gz.partis left behind on the PVC. Since the retention glob only matchesnetbird-*.tar.gz(not*.part), a persistently-failing day's.partfile is never cleaned up (it only gets overwritten if a later run happens to reuse the same$ts, i.e. same calendar day). Not urgent given the 5Gi headroom, but worth arm -f "*.part"at the top of the script for hygiene.ansible/roles/netbird-server/tasks/main.yml:95-99authorizes the backup key for therootuser on the VPS. The forcedcommand=option means the key can only ever run the dump script, so the blast radius is already tightly scoped — but since the dump script only needs read access to/opt/netbird/{management/store.db,config}, a lower-privileged local user (if one exists / is easy to add) would removerootfrom the equation entirely as defense-in-depth. Optional given therestrict+ forced-command mitigation already in place.🟢 Nits
kubernetes/infrastructure/netbird-backup/pvc.yamlomitsstorageClassName, relying on the comment "ceph-block (cluster default)" — this is correct (ceph-blockis annotatedstorageclass.kubernetes.io/is-default-class: "true"inrook-ceph-cluster/storageclass.yaml), but every other single-PVC manifest in this repo (e.g.kubernetes/apps/evcc/pvc.yaml,kubernetes/apps/libation/pvc.yaml) sets it explicitly. Purely cosmetic/consistency, not a defect.Tests
No automated tests exist for this kind of infrastructure change elsewhere in the repo (Ansible roles / Kustomize manifests / PrometheusRules), so the absence here is consistent with convention. The PR body's own "Verification plan after merge + provisioning" (manual CronJob trigger, checking the tarball, confirming the Velero schedule picks it up) is the right manual check for this kind of change — but per the blocking issue above, that plan will fail at the ExternalSecret-sync step even before the manual OpenBao step, since the namespace isn't yet allowlisted.
Recommendation
Request changes — 1 blocking issue.
Review follow-up (
10489eed) — the blocking issue was a genuine catch, thanks:netbird-backupis now allowlisted on the openbao ClusterSecretStore (sorted, with consumer comment) — without it the ExternalSecret would have sat unsyncable behind the NeverRan alert and been misdiagnosed as pending provisioning.Also addressed:
rm -f /backup/*.partat script start (aborted-run leftovers were outside the retention glob), and explicitstorageClassName: ceph-blockfor consistency with the repo's other PVC manifests.On the root-user suggestion: keeping the key on root with
restrict,command=is deliberate — the forced command is the actual privilege boundary (the key can only ever produce a dump; no shell, no forwarding, no other command), and a dedicated user would need read access to the root-owned live SQLite + config via group surgery on /opt/netbird or sudo rules — more moving parts for the same effective capability. Documented in the commit message.🤖 Automated review — model
CLI default(complexity: complex), commit10489eedd034f0f9a87388ff44ed1930b8b2ef8b.Summary
Adds a pull-based nightly backup of the NetBird controller VPS's management SQLite store: a forced-command SSH dump script + authorized_key on the VPS (Ansible), a CronJob that SSH-pulls the dump onto a Ceph PVC (in-cluster), an ExternalSecret for the private key, a Velero Schedule to ship the PVC offsite, and two Prometheus alerts for staleness/never-ran. I checked every piece against the existing conventions in this repo (ExternalSecret
dataFrom.extractpattern, theopenbao-raft-snapshotCronJob's retention idiom, theClusterSecretStorenamespace allowlist, VeleroSchedulestaggering, and thekube_cronjob_status_last_successful_timealert pattern already used for woodpecker-rotation) and everything is wired correctly — no blocking issues found.🟡 Suggestions
kubernetes/infrastructure/netbird-backup/cronjob.yaml:206-210):StrictHostKeyChecking=accept-newmeans the very first run silently trusts whatever host keynetbird.renner.unopresents, before it gets pinned into/backup/.known_hosts. Given the effort spent locking down the client credential (forced-command,restrict, single-purpose script), the server identity is comparatively unverified on that first connection. If you want to close the gap, seed.known_hostsonce by hand (or via a ConfigMap with the fingerprint captured during initial VPS provisioning) instead of relying on TOFU. Low real-world risk here (same LAN/WAN path Ansible already uses to deploy), so this is optional.ansible/roles/netbird-server/files/netbird-backup-dump:19, backing up/opt/netbird/config):turnserver.confcontainsuser=netbird:{{ netbird_turn_secret }}in plaintext (ansible/roles/netbird-server/templates/turnserver.conf.j2:15), so the TURN static secret now lands in every nightly tar.gz on thenetbird-backupPVC and then offsite via Velero. This is presumably intentional (needed for a full restore) and is no worse than how e.g. the OpenBao raft snapshot or Gitea/Woodpecker SQLite backups already carry sensitive data in this repo, but worth being deliberate about — the backup is only as protected as PVC/Velero access control, not separately encrypted.🟢 Nits
kubernetes/infrastructure/netbird-backup/cronjob.yaml:190pinsci-ansible:2026-04-29— same "shared CI image repurposed for a one-off job" pattern used byopenbao-raft-snapshot(which instead reuses the app's own image), just flagging that this tag will need a manual bump ifci-ansibleis rebuilt with breaking changes; not a new pattern this PR introduces, just worth knowing it's a manual dependency.Tests
No automated tests exist for this kind of change (Ansible role + Kubernetes manifests), consistent with the rest of the repo — there's no CI harness here that would exercise SSH pulls or CronJob semantics. The PR body already lays out a concrete manual verification plan (manual CronJob trigger, checking the tar.gz contents, confirming the Velero schedule picks up the PVC) which is the right way to validate this given the repo's conventions. I did not execute anything (no cluster/VPS access from this review); the design and wiring check out statically against the checkout.
Recommendation
Looks reasonable — no blocking issues; comments above are optional.
Follow-up (
c9009ffd) — both 🟡 addressed, each with the stronger fix:StrictHostKeyChecking=yes. A reprovisioned VPS makes the job fail loudly on key mismatch — which is the desired behavior; the pinned line carries a comment saying exactly that.store.dbonly —/opt/netbird/configis fully regenerable from the netbird-server role's templates + vault, so backing it up bought nothing except the plaintext TURN secret in every nightly tar.gz. Restore procedure updated accordingly (role run rebuilds config, untar restores the store).Image-pin nit acknowledged — known manual dependency, same pattern as openbao-raft-snapshot.
🤖 Automated review — model
CLI default(complexity: complex), commitc9009ffd3f1c3c3f999641ebd00d43c48df4c7fd.Summary
Adds a pull-based, off-VPS backup path for the NetBird controller's management SQLite store: a forced-command dump script + restricted SSH key on the VPS (netbird-server Ansible role), a nightly CronJob in a new
netbird-backupnamespace that ssh-pulls the dump onto a Ceph PVC, a Velero Schedule to ship that PVC offsite, and two staleness/never-ran Prometheus alerts. I checked this against the closest existing precedents in the repo (kubernetes/infrastructure/openbao/snapshot-cronjob.yaml,kubernetes/infrastructure/velero/restore-test-cronjob.yaml,kubernetes/apps/monitoring-rules/prometheusrule-woodpecker-rotation.yaml) and the design is consistent with them: correct SQLite backup API usage (.backup, not a raw file copy), a genuinely restricted forced-command key, gzip-verify-before-rename to avoid truncated dumps, pinned host key, and a two-alert design (stale + never-ran) that avoids false-firing before the first successful run. No blocking issues found.🟡 Suggestions
CiliumNetworkPolicyfor the newnetbird-backupnamespace. Every other namespace in the repo that runs an actual credentialed workload has a dedicateddefault-deny-ingresspolicy underkubernetes/infrastructure/network-policies/(e.g.openbao-ns.yaml,velero-ns.yaml,external-secrets-ns.yaml) — but nonetbird-backup-ns.yamlwas added here, andkubernetes/infrastructure/network-policies/kustomization.yamlwasn't touched. Without one, Cilium's default (no policy selects the pod) is allow-all egress/ingress for the CronJob pod. This pod mounts a private SSH key that authenticates asrooton an internet-facing VPS (kubernetes/infrastructure/netbird-backup/cronjob.yaml:242-244) — exactly the kind of sensitive-credential workload this repo otherwise locks down by default. It doesn't block the feature (the job will run fine either way, and the VPS-siderestrict,command=...forced-command already bounds what a compromised pod could do over that specific SSH connection), but it does undercut the PR's own stated threat model ("even a full cluster compromise gets exactly one capability") since the pod itself is left free to reach anything else in-cluster or on the internet. Worth adding a scoped policy (egress: DNS + world on 22/443, ingress: none needed since it's a Job) to match the pattern used forvelero-ns.yaml/openbao-ns.yaml.🟢 Nits
ansible/roles/netbird-server/files/netbird-backup-dump:20usesset -eu, while the CronJob's inline script (kubernetes/infrastructure/netbird-backup/cronjob.yaml:200) usesset -ec(no-u). Not a real risk here since no unset variables are referenced in the CronJob script, but worth aligning for consistency if either script grows.Tests
No automated tests exist for Ansible roles or Kubernetes manifests elsewhere in this repo (no
kustomize build/kubeconform/yamllintstep in.woodpecker/), so there's nothing to add here beyond what's already conventional. I traced the pieces that are checkable statically and they line up:store.dbpath (/opt/netbird/management/store.db) matches the bind mount indocker-compose.yml.j2(/opt/netbird/management:/var/lib/netbird), consistent with NetBird management's default sqlite location.ansible.posix.authorized_keyis already used by four other roles in this repo, so the collection dependency is proven to work in the existingci-ansibleimage.kube_cronjob_status_last_successful_time/kube_cronjob_createdmetrics are already relied on byprometheusrule-woodpecker-rotation.yaml, confirming kube-state-metrics exposes them in this cluster.openbao-raft-snapshot,restore-test) were presumably validated.Recommendation
Looks reasonable — no blocking issues; the network-policy suggestion above is worth a follow-up but doesn't need to hold up this PR.
🤖 Automated review — model
CLI default(complexity: complex), commitd70ebe9ce3f632faa07c9e33d10ae68347adc9e4.Summary
Adds a pull-based, off-VPS backup pipeline for the NetBird controller's SQLite store: a forced-command dump script + restricted SSH key on the VPS (Ansible), an in-cluster CronJob that SSH-pulls nightly onto a Ceph PVC, an ExternalSecret for the key, a Velero Schedule to ship the PVC offsite, and two staleness/never-ran alerts. I cross-checked every new/changed file against the rest of the repo (existing
dataFrom.extractExternalSecrets, theci-ansibleimage's actual toolset, theClusterSecretStorenamespace allowlist, the Velero schedule stagger, network-policy conventions, and the netbird-server role's existingsqlite3package +/opt/netbird/managementbind mount) and didn't find anything that would break at apply time. No secrets are committed — only the two public keys (backup key + VPS host key) appear in plaintext, which is expected. This is careful, well-reasoned infra work.No blocking issues found.
🟡 Suggestions
kubernetes/infrastructure/netbird-backup/cronjob.yaml:213— the VPS host key is hard-coded in the pod's inline shell script (pinned, no TOFU, by design per the comment). That's the right security tradeoff, but it means a legitimate VPS host-key rotation requires editing and redeploying this CronJob in lockstep with the Ansible side, with no automated signal beyond "the job starts failing." Worth a one-line pointer in the PR/runbook (or theNetbirdVpsBackupStalealert description) that aHost key verification failedjob failure means "check for an intentional VPS host key rotation," so a future on-call doesn't have to reverse-engineer that from the script comment.kubernetes/infrastructure/netbird-backup/cronjob.yaml— no dedicatedCiliumNetworkPolicyfor the newnetbird-backupnamespace, so it's fully unrestricted for ingress/egress by default. This matches the existingnetbird-ingressnamespace (also unpoliced), so it's consistent with current practice, not a regression — just flagging that it's the more permissive option among the two patterns present in the repo (most other namespaces get adefault-deny-ingressCNP undernetwork-policies/).Tests
No automated tests exist for this kind of change (Ansible role + K8s manifests), consistent with the rest of the repo. The PR body's "Verification plan after merge" (manual CronJob trigger, confirm tar.gz contents, confirm Velero schedule picks up the PVC) is the appropriate manual check here and covers the right things: dump script correctness, SSH pull path, and offsite shipping. I could not execute it since it depends on live infrastructure (VPS, OpenBao, cluster) outside this checkout.
Recommendation
Looks reasonable — no blocking issues; comments above are optional.