fix(netbird): let the nightly dump wait out the writer

The dump ran sqlite3's .backup with no busy handler. sqlite3 defaults its busy timeout to 0 ms, so the online-backup API gave up the first time the management container held the write lock instead of waiting for it, and set -eu turned that into a failed job. Four nightly runs failed that way before this, three attempts each, every one refused within seconds of the ssh handshake. NetBird sets no journal_mode on the store — its DSN carries only _busy_timeout=30000 and cache=shared — so store.db uses the default rollback journal, where a writer's lock blocks a reader outright. Under WAL the reader would have gone through and this would never have come up. .timeout arms the busy handler on the source connection, which is the contended side. The shell's .backup never arms one on the destination and does not need to: that file is created fresh in $tmp and nothing else touches it. VACUUM INTO replaces .backup because a busy timeout on its own only turns an instant failure into a slow one. .backup copies 100 pages per sqlite3_backup_step() and SQLite restarts the copy whenever the source is written between steps, so against a steady writer it returns only once the writes stop. Measured on a rollback-journal database of the same shape, with a writer committing continuously for 20 s: .backup returned after 19 s, VACUUM INTO returned in under a second, and integrity_check on its output passed. Both failed identically with no timeout at all, which is what reproduced the production error. VACUUM INTO also compacts, so the dump lands smaller than the live file. 30 s sits well inside the puller's 600 s activeDeadlineSeconds, and the read transaction VACUUM INTO holds is covered by NetBird's own 30 s busy timeout on the writing side. The script lives on the VPS at /usr/local/bin/netbird-backup-dump and is refreshed only when this role runs, so merging this to master is what deploys it — deploy-netbird-server triggers on ansible/roles/netbird-server/**. Closes #714. Refs #712. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-01 15:51:41 +02:00
1 changed files with 23 additions and 5 deletions
@@ -1,10 +1,10 @@
 #!/bin/sh
 # Forced-command target for the netbird-backup SSH key (see the
 # "Authorize the cluster backup key" task): takes a CONSISTENT snapshot
-# of the management SQLite store via the online-backup API (safe against
-# a live writer, unlike copying store.db) and streams a gzipped tar to
-# stdout. The key's authorized_keys entry is restricted to exactly this
-# script — the in-cluster puller cannot run anything else.
+# of the management SQLite store (safe against a live writer, unlike
+# copying store.db) and streams a gzipped tar to stdout. The key's
+# authorized_keys entry is restricted to exactly this script — the
+# in-cluster puller cannot run anything else.
 #
 # Deliberately store.db ONLY: /opt/netbird/config is fully regenerable
 # from the netbird-server role's templates + vault, and including it
@@ -14,5 +14,23 @@
 set -eu
 tmp=$(mktemp -d)
 trap 'rm -rf "$tmp"' EXIT
-sqlite3 /opt/netbird/management/store.db ".backup '$tmp/store.db'"
+# NetBird sets no journal_mode on the store (its DSN carries only
+# _busy_timeout and cache=shared), so store.db uses the default rollback
+# journal, where the management container's write lock blocks a reader
+# outright rather than letting it through as WAL would. Two things follow.
+#
+# .timeout arms a busy handler on the source connection, which is the
+# contended side; the destination is a fresh file in $tmp that nothing else
+# touches. Without it sqlite3 uses its 0 ms default and gives up the moment
+# it cannot take the lock. 30 s sits well inside the puller's 600 s
+# activeDeadlineSeconds.
+#
+# VACUUM INTO rather than .backup: the shell's .backup copies 100 pages per
+# sqlite3_backup_step(), and SQLite restarts the copy whenever the source is
+# written between steps, so a steady writer drags it out for as long as the
+# writes keep coming. VACUUM INTO reads one transaction in a single pass and
+# is documented as an alternative to the backup API for exactly this. It
+# also compacts, so the dump is smaller than the live file.
+sqlite3 -cmd '.timeout 30000' /opt/netbird/management/store.db \
+    "VACUUM INTO '$tmp/store.db'"
 tar -C "$tmp" -czf - store.db