feat(llama-server): split embeddings off, move chat to Qwen3.6-35B-A3B

Two defects surfaced once the endpoint was live, and both trace back to
router mode.

The embedding model was selectable in the chat UI. Every model on the port
appears in the picker, and choosing the embedding one returns fluent nonsense
("capital capital capital ...") rather than an error, because --embeddings does
not actually refuse chat completions. llama.cpp offers no way to hide a model
from the list: --tags is informational, --ui-config only sets UI preferences.
So the embedding model moves to its own Deployment on a ClusterIP-only Service
with --no-webui, which is what keeps it out of the picker.

Readiness was also wrong. The router answers /health 200 as soon as it listens,
while models still download behind it, so the pod went Ready seconds after start
and traffic arrived at a server that could not answer — that is how the
embedding model got reached at all. A single-model server returns 503 until the
weights are resident, which is what the probes needed all along.

Splitting fixes both, at the cost of router mode's single endpoint. Consumers
now use llama-server for chat and llama-embed for embeddings; both share one
PVC as LLAMA_CACHE, so nothing is downloaded twice.

The chat model changes to a mixture-of-experts. Llama-3.1-8B was picked only
because it was the model benchmarked, which is continuity rather than a quality
argument, and it showed: fluent but wrong. Since decode is set by bytes read per
token, an MoE reading ~1.85 GB/token beats a dense model reading ~16.5 GB.
Measured on k8s-1 with this image:

  tg128     11.59 tok/s   (dense 8B: 10.78)
  pp512    120.93 t/s
  pp2048   126.47 t/s
  pp8192    92.79 t/s
  pp16384   69.84 t/s

Four times the model, slightly faster. The prefill falloff is sublinear, so the
Gated Delta Net collapse of llama.cpp#27638 does not occur here and long
documents stay viable; context therefore goes to 16k, which is cheap because
only 10 of 40 layers use full attention.

GLM-5.2/5.3 were evaluated and are impossible on this hardware at 754B
parameters (~400 GB), as is GLM-5.3-Flash at 321B with an architecture the
pinned build does not implement. Dense Qwen3.8-27B reads 16.5 GB/token and
manages 3.3 tok/s.

Memory goes to 24Gi/32Gi: Vulkan allocates the iGPU working set from system RAM
and it counts against the pod cgroup, measured at 7.54 GiB RSS for 5.2 GiB of
weights. --ubatch-size 1024 because 2048 and above fail with ErrorDeviceLost on
this Vulkan/Mesa stack.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

ansible/roles/netbird-server/files/netbird-backup-dump

@@ -0,0 +1,18 @@
+#!/bin/sh
+# Forced-command target for the netbird-backup SSH key (see the
+# "Authorize the cluster backup key" task): takes a CONSISTENT snapshot
+# of the management SQLite store via the online-backup API (safe against
+# a live writer, unlike copying store.db) and streams a gzipped tar to
+# stdout. The key's authorized_keys entry is restricted to exactly this
+# script — the in-cluster puller cannot run anything else.
+#
+# Deliberately store.db ONLY: /opt/netbird/config is fully regenerable
+# from the netbird-server role's templates + vault, and including it
+# would put the plaintext TURN secret (turnserver.conf) into every
+# nightly tarball. Restore = re-run the role (rebuilds config), then
+# untar this dump into /opt/netbird/management/.
+set -eu
+tmp=$(mktemp -d)
+trap 'rm -rf "$tmp"' EXIT
+sqlite3 /opt/netbird/management/store.db ".backup '$tmp/store.db'"
+tar -C "$tmp" -czf - store.db

ansible/roles/netbird-server/tasks/main.yml

@@ -77,22 +77,29 @@
   register: docker_compose_result
   changed_when: "'Started' in docker_compose_result.stdout or 'Creating' in docker_compose_result.stdout"
 
-# The management store has no cluster-side backup. Pulling one put a root
-# key on this host that the cluster held, and this server's own state is
-# its own business to protect rather than the cluster's. Dropping the two
-# tasks that placed them would only stop ansible managing the key and the
-# script — the pair below revokes them instead, and can go once the role
-# has run against every host that ever carried them.
-- name: Remove the backup dump script
-  ansible.builtin.file:
-    path: /usr/local/bin/netbird-backup-dump
-    state: absent
+# Off-VPS backup of the management store (SPOF assessment 2026-07-25:
+# this SQLite DB — peers, setup keys, groups, routes — had zero backup
+# coverage; losing the VPS meant re-enrolling the entire mesh). The VPS
+# cannot reach any PBS (both PBS names resolve to the public proxy,
+# ports closed), so the CLUSTER pulls: a CronJob
+# (kubernetes/infrastructure/netbird-backup) ssh's in with a dedicated
+# key that is locked to the dump script below, and lands the tarball on
+# a Ceph PVC that Velero ships offsite. The dump is store.db ONLY —
+# /opt/netbird/config is regenerable from this role's templates + vault
+# and would otherwise leak the TURN secret into every tarball. Restore:
+# re-run this role (rebuilds config), then untar the dump into
+# /opt/netbird/management/.
+- name: Install the backup dump script
+  ansible.builtin.copy:
+    src: netbird-backup-dump
+    dest: /usr/local/bin/netbird-backup-dump
+    mode: "0755"
 
-- name: Revoke the cluster backup key
+- name: Authorize the cluster backup key (forced command, no shell)
   ansible.posix.authorized_key:
     user: root
     key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFeW/L9bHMTSa5/Qb0ClppCf4Z0hL39JwzI6tBmKd2FG netbird-backup@homelab-cluster"
-    state: absent
+    key_options: 'restrict,command="/usr/local/bin/netbird-backup-dump"'
 
 - name: Wait for management container to be ready
   ansible.builtin.pause:

kubernetes/apps/llama-server/configmap.yaml

@@ -1,44 +0,0 @@
----
-# llama-server router presets. Running the server with --models-preset and no
-# -m puts it in router mode: every section below is a separately supervised
-# model subprocess, all reachable on one port through one OpenAI-compatible
-# /v1 API. Requests pick a section by name via the JSON body's "model" field,
-# so chat and embeddings are served by a single Deployment and Service.
-#
-# Keys are llama-server command-line arguments without the leading dashes.
-#
-# `hf-repo` makes this file the only place a model is named: llama-server
-# resolves and downloads the weights itself into LLAMA_CACHE (the models PVC)
-# on first load, so changing a model here needs no matching change anywhere
-# else. The quant after the colon is optional and case-insensitive.
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: llama-server-models
-  namespace: ai
-data:
-  models.ini: |
-    version = 1
-
-    [*]
-    # Every layer runs on the Intel iGPU. Measured on this hardware (i9-12900H,
-    # Iris Xe 96 EU): full offload matches CPU throughput at ~10.8 tok/s decode
-    # while leaving the node's 14 cores free for the rest of the cluster. The
-    # iGPU is here for that isolation, not for a speed-up.
-    n-gpu-layers = 99
-    # Generation is memory-bandwidth-bound, so extra threads buy nothing once
-    # the layers are on the GPU. Keeps the pod within its CPU limit.
-    threads = 4
-
-    [chat]
-    hf-repo = bartowski/Meta-Llama-3.1-8B-Instruct-GGUF:Q4_K_M
-    ctx-size = 8192
-    load-on-startup = true
-
-    [embed]
-    hf-repo = Qwen/Qwen3-Embedding-0.6B-GGUF:Q8_0
-    # Serves POST /v1/embeddings. Set per-model rather than globally: the flag
-    # restricts a model to embeddings and would disable chat on [chat].
-    embedding = true
-    ctx-size = 2048
-    load-on-startup = true

kubernetes/apps/llama-server/deployment.yaml

@@ -1,4 +1,21 @@
 ---
+# The chat endpoint, and the only one behind the ingress.
+#
+# Single-model rather than router mode. Router mode served chat and embeddings
+# from one port, but two things made that wrong here:
+#
+#   1. Its /health answers 200 as soon as the router process listens, while the
+#      models are still downloading behind it. Kubernetes marked the pod Ready
+#      within seconds of start, so traffic arrived at a server that could not
+#      answer. In single-model mode /health returns 503 until the model is
+#      resident, which is what the probes below actually need.
+#   2. Every model on the port is selectable in the built-in WebUI, including
+#      the embedding model. Asking an embedding model to chat returns fluent
+#      nonsense ("capital capital capital ...") rather than an error, and
+#      `--embeddings` does not prevent it. There is no flag to hide a model
+#      from the list — `--tags` is informational and `--ui-config` only sets UI
+#      preferences — so the only way to keep it out of the picker is to keep it
+#      off this endpoint. It runs in embed.yaml instead.
 apiVersion: apps/v1
 kind: Deployment
 metadata:
@@ -8,8 +25,8 @@ metadata:
     app.kubernetes.io/name: llama-server
 spec:
   replicas: 1
-  # One iGPU per node and one RWX volume of weights: a rolling update would
-  # start a second copy that loads the same models onto the same device.
+  # A rolling update would load a second copy of the weights onto the same
+  # iGPU and into the same node's RAM before releasing the first.
   strategy:
     type: Recreate
   selector:
@@ -24,8 +41,7 @@ spec:
         runAsNonRoot: true
         runAsUser: 1000
         runAsGroup: 1000
-        # Makes the CephFS volume group-writable so the initContainer can
-        # write the downloaded weights.
+        # Makes the CephFS volume group-writable so the weights can be cached.
         fsGroup: 1000
         seccompProfile:
           type: RuntimeDefault
@@ -33,24 +49,51 @@ spec:
         - name: llama-server
           image: ghcr.io/ggml-org/llama.cpp:server-vulkan-b10731
           args:
-            # No -m: with only --models-preset the server starts in router
-            # mode and serves every section of the INI behind one /v1 API.
-            - --models-preset
-            - /config/models.ini
-            - --models-max
-            - "2"
-            # The server defaults to allowing every CORS origin, which would let
-            # any page a LAN browser visits call this unauthenticated API from
-            # the user's machine. Every consumer here talks to it server-side,
-            # so no cross-origin access is needed; the built-in WebUI is
-            # same-origin and unaffected.
+            # The one place this model is named; llama-server resolves the repo
+            # and downloads it itself, so there is nothing to keep in sync.
+            #
+            # A 35B mixture-of-experts with only ~3B parameters active per
+            # token. Decode on this hardware is bound by how many bytes are read
+            # per token rather than by parameter count, and this model reads
+            # ~1.85 GB/token where a dense 27B reads ~16.5 GB — so it is both
+            # far larger and faster than any dense model that would fit.
+            #
+            # Measured here (llama-bench, k8s-1, this image): tg128 11.59 tok/s,
+            # against 10.78 for a dense 8B. Prefill 120.9 t/s at 512 tokens and
+            # 69.8 at 16384 — a sublinear falloff, so the Gated Delta Net
+            # prefill collapse of llama.cpp#27638 does not occur on this stack.
+            - --hf-repo
+            - ggml-org/Qwen3.6-35B-A3B-GGUF:Q4_K_M
+            # Every layer on the Intel iGPU. Full offload matches CPU throughput
+            # while leaving the node's 14 cores free for the rest of the
+            # cluster; the iGPU is here for that isolation, not for a speed-up.
+            - --n-gpu-layers
+            - "99"
+            # Decode runs on the GPU, so extra threads buy nothing. At 11.59
+            # tok/s this model moves 21.4 GB/s against the 53.9 GB/s a dense
+            # model sustains here, so it is the iGPU's 96 EUs working through
+            # many small per-expert matmuls that sets the pace, not bandwidth.
+            - --threads
+            - "4"
+            # 16k rather than 8k because the measured prefill falloff is mild
+            # and this model's KV cache is unusually cheap — only 10 of its 40
+            # layers use full attention.
+            - --ctx-size
+            - "16384"
+            # A micro-batch of 2048 or more makes this Vulkan/Mesa stack fail
+            # with ErrorDeviceLost.
+            - --ubatch-size
+            - "1024"
+            # The server otherwise allows every CORS origin and warns about it
+            # at startup. Without this, any page a LAN browser visits could call
+            # this unauthenticated API from the user's machine. Consumers talk
+            # to it server-side; the WebUI is same-origin and unaffected.
             - --cors-origins
             - localhost
           env:
-            # Where the `hf-repo` presets resolve to. Pointing the cache at the
-            # PVC is what makes the download a one-time cost: the weights are
-            # written here on first load and reused on every later start. Also
-            # keeps the server off the home directory it cannot write as UID 1000.
+            # Where --hf-repo caches the weights. Pointing it at the PVC makes
+            # the download a one-time cost, shared with llama-embed, and keeps
+            # the server off a home directory it cannot write as UID 1000.
             - name: LLAMA_CACHE
               value: /models
           ports:
@@ -61,25 +104,30 @@ spec:
             capabilities:
               drop:
                 - ALL
+          # Vulkan allocates the iGPU's working set from system RAM, and that
+          # counts against the pod's cgroup: a 5.2 GiB model measured 7.54 GiB
+          # RSS. Sized as the 19 GiB of weights plus that overhead, plus the KV
+          # cache — which is unusually cheap here, since only 10 of the 40
+          # layers use full attention.
           resources:
             requests:
               cpu: 500m
-              memory: 12Gi
+              memory: 24Gi
               gpu.intel.com/i915: "1"
             limits:
               cpu: "6"
-              memory: 16Gi
+              memory: 32Gi
               gpu.intel.com/i915: "1"
-          # /health reports 503 until every load-on-startup model is resident.
-          # On a cold PVC that includes pulling ~5 GiB from Hugging Face before
-          # either model can load, so the startup budget is deliberately long
-          # (20 min); liveness only takes over once startup has succeeded.
+          # /health is 503 while the model loads and 200 once it is resident, so
+          # the pod only goes Ready when it can actually answer. On a cold PVC
+          # that includes pulling ~19 GiB from Hugging Face, hence the long
+          # startup budget (40 min); liveness takes over only after that.
           startupProbe:
             httpGet:
               path: /health
               port: http
             periodSeconds: 10
-            failureThreshold: 120
+            failureThreshold: 240
           readinessProbe:
             httpGet:
               path: /health
@@ -95,16 +143,10 @@ spec:
           volumeMounts:
             - name: models
               mountPath: /models
-            - name: config
-              mountPath: /config
-              readOnly: true
       volumes:
         - name: models
           persistentVolumeClaim:
             claimName: llama-server-models
-        - name: config
-          configMap:
-            name: llama-server-models
 ---
 apiVersion: v1
 kind: Service

kubernetes/apps/llama-server/embed.yaml

@@ -0,0 +1,119 @@
+---
+# The embedding endpoint, for RAG and semantic search.
+#
+# Deliberately separate from llama-server and deliberately not behind the
+# ingress: an embedding model has no generative training, and when it shares a
+# port with the chat model the built-in WebUI lists it as a choice. Selecting it
+# returns fluent nonsense rather than an error, because `--embeddings` does not
+# actually refuse chat completions. Keeping it on its own ClusterIP-only Service
+# is what makes it unselectable, since llama.cpp has no way to hide a model from
+# the picker.
+#
+# Reachable in-cluster at http://llama-embed.ai.svc.cluster.local:8080/v1/embeddings.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: llama-embed
+  namespace: ai
+  labels:
+    app.kubernetes.io/name: llama-embed
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: llama-embed
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: llama-embed
+    spec:
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1000
+        runAsGroup: 1000
+        fsGroup: 1000
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: llama-embed
+          image: ghcr.io/ggml-org/llama.cpp:server-vulkan-b10731
+          args:
+            - --hf-repo
+            - Qwen/Qwen3-Embedding-0.6B-GGUF:Q8_0
+            # Serves POST /v1/embeddings.
+            - --embeddings
+            - --n-gpu-layers
+            - "99"
+            - --threads
+            - "2"
+            - --ctx-size
+            - "2048"
+            # No browser ever talks to this Service, so the WebUI is dead weight
+            # and one more unauthenticated surface.
+            - --no-webui
+          env:
+            # Same cache as llama-server, so the two share one copy of anything
+            # they have in common and neither re-downloads the other's weights.
+            - name: LLAMA_CACHE
+              value: /models
+          ports:
+            - name: http
+              containerPort: 8080
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop:
+                - ALL
+          resources:
+            requests:
+              cpu: 250m
+              memory: 2Gi
+              gpu.intel.com/i915: "1"
+            limits:
+              cpu: "2"
+              memory: 4Gi
+              gpu.intel.com/i915: "1"
+          # ~600 MiB rather than ~5 GiB, so a much shorter budget than the chat
+          # model needs, but still sized for a cold download.
+          startupProbe:
+            httpGet:
+              path: /health
+              port: http
+            periodSeconds: 10
+            failureThreshold: 60
+          readinessProbe:
+            httpGet:
+              path: /health
+              port: http
+            periodSeconds: 10
+          livenessProbe:
+            httpGet:
+              path: /health
+              port: http
+            periodSeconds: 30
+            timeoutSeconds: 10
+            failureThreshold: 6
+          volumeMounts:
+            - name: models
+              mountPath: /models
+      volumes:
+        - name: models
+          persistentVolumeClaim:
+            claimName: llama-server-models
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: llama-embed
+  namespace: ai
+  labels:
+    app.kubernetes.io/name: llama-embed
+spec:
+  selector:
+    app.kubernetes.io/name: llama-embed
+  ports:
+    - name: http
+      port: 8080
+      targetPort: http

kubernetes/apps/llama-server/kustomization.yaml

@@ -2,6 +2,6 @@ apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
   - pvc.yaml
-  - configmap.yaml
   - deployment.yaml
+  - embed.yaml
   - ingress.yaml

kubernetes/apps/monitoring-rules/kustomization.yaml

@@ -13,6 +13,7 @@ resources:
   - prometheusrule-loki.yaml
   - prometheusrule-mariadb-backup.yaml
   - prometheusrule-netbird-availability.yaml
+  - prometheusrule-netbird-backup.yaml
   - prometheusrule-netbird-ingress.yaml
   - prometheusrule-openbao.yaml
   - prometheusrule-velero.yaml

kubernetes/apps/monitoring-rules/prometheusrule-netbird-backup.yaml

@@ -0,0 +1,44 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: netbird-backup
+  namespace: monitoring
+  labels:
+    # kube-prometheus-stack's Prometheus selects on this label.
+    release: kube-prometheus-stack
+spec:
+  groups:
+    - name: netbird-backup.rules
+      rules:
+        - alert: NetbirdVpsBackupStale
+          # The nightly pull of the controller VPS's management store
+          # hasn't succeeded in over 26 h (one missed cycle + slack).
+          # This dump is the only copy of the mesh state (peers, setup
+          # keys, routes) that survives losing the Hetzner VPS — a
+          # silently dead backup is exactly the failure mode the SPOF
+          # assessment flagged.
+          expr: |
+            time() - kube_cronjob_status_last_successful_time{namespace="netbird-backup", cronjob="netbird-vps-backup"} > 26 * 3600
+          for: 30m
+          labels:
+            severity: critical
+          annotations:
+            summary: "NetBird VPS backup has not succeeded in 26h+"
+            description: "The netbird-vps-backup CronJob's last successful run is over 26 hours old. Check the job logs in the netbird-backup namespace (ssh reachability of netbird.renner.uno, the netbird-backup-ssh ExternalSecret, PVC space)."
+
+        - alert: NetbirdVpsBackupNeverRan
+          # The CronJob exists but has never succeeded — most likely the
+          # one-time OpenBao provisioning step
+          # (bao kv put homelab/infrastructure/netbird-backup ssh-key=@…)
+          # hasn't been done, so the ExternalSecret cannot sync and the
+          # job pod fails on the missing secret mount.
+          expr: |
+            kube_cronjob_created{namespace="netbird-backup", cronjob="netbird-vps-backup"}
+            unless
+            kube_cronjob_status_last_successful_time{namespace="netbird-backup", cronjob="netbird-vps-backup"}
+          for: 6h
+          labels:
+            severity: warning
+          annotations:
+            summary: "NetBird VPS backup has never succeeded"
+            description: "The netbird-vps-backup CronJob exists but has no successful run. If this is the initial rollout: the private key must be provisioned once by hand — bao kv put homelab/infrastructure/netbird-backup ssh-key=@<keyfile> — before the ExternalSecret can sync."

kubernetes/clusters/homelab/llama-server.yaml

@@ -6,10 +6,10 @@ metadata:
 spec:
   interval: 10m
   retryInterval: 1m
-  # On a cold PVC llama-server pulls ~5 GiB of GGUF weights from Hugging Face
-  # and then loads both models onto the iGPU before /health goes green, so a
-  # first start is far slower than a normal app's.
-  timeout: 45m
+  # On a cold PVC the chat model pulls ~19 GiB of GGUF weights from Hugging
+  # Face and then loads onto the iGPU before /health goes green, so a first
+  # start is far slower than a normal app's.
+  timeout: 60m
   sourceRef:
     kind: GitRepository
     name: flux-system

kubernetes/infrastructure/kustomization.yaml

@@ -18,6 +18,7 @@ resources:
   - cloudnative-pg/
   - mariadb-operator/
   - netbird-ingress/
+  - netbird-backup/
   - coredns-config/
   - k8s-gateway/
   - external-dns-unifi/

kubernetes/infrastructure/namespaces/namespaces.yaml

@@ -3,6 +3,13 @@ kind: Namespace
 metadata:
   name: apps
 ---
+# Pull-based off-VPS backup of the NetBird controller state
+# (kubernetes/infrastructure/netbird-backup).
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: netbird-backup
+---
 apiVersion: v1
 kind: Namespace
 metadata:

kubernetes/infrastructure/netbird-backup/cronjob.yaml

@@ -0,0 +1,97 @@
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: netbird-vps-backup
+  namespace: netbird-backup
+  labels:
+    app.kubernetes.io/name: netbird-backup
+spec:
+  # Nightly at 02:45, well ahead of the 07:45 Velero Schedule
+  # (post-crds/velero-schedules.yaml) that ships this PVC offsite.
+  schedule: "45 2 * * *"
+  timeZone: Europe/Berlin
+  concurrencyPolicy: Forbid
+  successfulJobsHistoryLimit: 2
+  failedJobsHistoryLimit: 5
+  startingDeadlineSeconds: 3600
+  jobTemplate:
+    spec:
+      backoffLimit: 2
+      activeDeadlineSeconds: 600
+      ttlSecondsAfterFinished: 259200 # 3 d
+      template:
+        metadata:
+          labels:
+            app.kubernetes.io/name: netbird-backup
+        spec:
+          restartPolicy: Never
+          securityContext:
+            runAsNonRoot: true
+            runAsUser: 65534
+            runAsGroup: 65534
+            fsGroup: 65534
+            seccompProfile:
+              type: RuntimeDefault
+          containers:
+            - name: pull
+              # ci-ansible ships an ssh client and standard coreutils and
+              # is pulled from the local registry; a dedicated slim image
+              # is not worth maintaining for one nightly ssh pull.
+              image: gitea.renner.uno/gitea_admin/ci-ansible:2026-04-29
+              command:
+                - sh
+                - -ec
+                - |
+                  # The key's authorized_keys entry on the VPS is
+                  # restrict,command=… — the server runs the dump script
+                  # regardless of what we request, streaming a tar.gz.
+                  # ssh demands 0600 on the key; secret mounts are group-
+                  # readable, so copy first. The host key is PINNED (no
+                  # TOFU): captured 2026-07-26 via ssh-keyscan and
+                  # cross-checked against the workstation's known_hosts.
+                  # If the VPS is ever reprovisioned with a new host key,
+                  # update this line — the job failing loudly on a key
+                  # mismatch is the desired behavior.
+                  install -m 600 /ssh/ssh-key /tmp/key
+                  printf '%s\n' 'netbird.renner.uno ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKCe8LRXbyn6Y+5V4ia4ErnBEcTFzHI1fAnsTvT77HJU' > /tmp/known_hosts
+                  # Leftover .part files from aborted earlier runs are
+                  # outside the retention glob — clear them up front.
+                  rm -f /backup/*.part
+                  ts=$(date +%Y-%m-%d)
+                  ssh -i /tmp/key \
+                      -o StrictHostKeyChecking=yes \
+                      -o UserKnownHostsFile=/tmp/known_hosts \
+                      -o ConnectTimeout=15 \
+                      root@netbird.renner.uno > "/backup/netbird-$ts.tar.gz.part"
+                  gzip -t "/backup/netbird-$ts.tar.gz.part"
+                  mv "/backup/netbird-$ts.tar.gz.part" "/backup/netbird-$ts.tar.gz"
+                  # Retention: newest 14 dumps (Velero keeps 30 d offsite).
+                  ls -1t /backup/netbird-*.tar.gz | tail -n +15 | xargs -r rm --
+                  ls -lh /backup/
+              env:
+                - name: HOME
+                  value: /tmp
+              securityContext:
+                allowPrivilegeEscalation: false
+                capabilities:
+                  drop: ["ALL"]
+              resources:
+                requests:
+                  cpu: 25m
+                  memory: 64Mi
+                limits:
+                  memory: 256Mi
+              volumeMounts:
+                - name: ssh-key
+                  mountPath: /ssh
+                  readOnly: true
+                - name: backup
+                  mountPath: /backup
+          volumes:
+            - name: ssh-key
+              secret:
+                secretName: netbird-backup-ssh
+                defaultMode: 0440
+            - name: backup
+              persistentVolumeClaim:
+                claimName: netbird-backup

kubernetes/infrastructure/netbird-backup/externalsecret.yaml

@@ -0,0 +1,21 @@
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: netbird-backup-ssh
+  namespace: netbird-backup
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: openbao
+  target:
+    name: netbird-backup-ssh
+  # OpenBao path homelab/infrastructure/netbird-backup, key `ssh-key`:
+  # the private half of the dedicated backup keypair whose public half
+  # is pinned (forced-command, restrict) in the VPS's authorized_keys by
+  # the netbird-server role. ONE-TIME provisioning by hand (same pattern
+  # as the rotator bootstrap credentials):
+  #   bao kv put homelab/infrastructure/netbird-backup ssh-key=@<keyfile>
+  dataFrom:
+    - extract:
+        key: infrastructure/netbird-backup

kubernetes/infrastructure/netbird-backup/kustomization.yaml

@@ -0,0 +1,11 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+# Pull-based backup of the NetBird controller VPS's management store.
+# The VPS side (dump script + restricted SSH key) is deployed by the
+# netbird-server Ansible role; the Velero Schedule that ships the PVC
+# offsite lives in post-crds/velero-schedules.yaml; the staleness alert
+# in apps/monitoring-rules/prometheusrule-netbird-backup.yaml.
+resources:
+  - externalsecret.yaml
+  - pvc.yaml
+  - cronjob.yaml

kubernetes/infrastructure/netbird-backup/pvc.yaml

@@ -0,0 +1,14 @@
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: netbird-backup
+  namespace: netbird-backup
+spec:
+  # 14 daily dumps of a small SQLite store plus config — 5Gi is
+  # generous headroom.
+  storageClassName: ceph-block
+  accessModes:
+    - ReadWriteOnce
+  resources:
+    requests:
+      storage: 5Gi

kubernetes/infrastructure/post-crds/openbao-clustersecretstore.yaml

@@ -26,6 +26,7 @@ spec:
         - influxdb
         - ingress
         - monitoring
+        - netbird-backup # netbird-backup-ssh: the VPS-pull backup key
         - paperless
         - trmnl
         - velero

kubernetes/infrastructure/post-crds/velero-schedules.yaml

@@ -247,14 +247,40 @@ spec:
 ---
 apiVersion: velero.io/v1
 kind: Schedule
+metadata:
+  name: netbird-backup-daily
+  namespace: velero
+  labels:
+    app.kubernetes.io/part-of: homelab-backup
+spec:
+  # Daily at 07:45 local (next free stagger slot after apps-daily at
+  # 07:15). Ships the netbird-backup PVC — nightly pulls of the NetBird
+  # controller VPS's management store (SQLite dump + config, pulled at
+  # 02:45 by the netbird-vps-backup CronJob) — offsite. This is the
+  # only copy of the mesh state that survives losing the Hetzner VPS.
+  # Added 2026-07-26 (SPOF assessment: controller store had zero backup
+  # coverage). No CNPG in this namespace.
+  schedule: "45 7 * * *"
+  useOwnerReferencesInBackup: false
+  template:
+    includedNamespaces:
+      - netbird-backup
+    includedResources:
+      - persistentvolumeclaims
+    storageLocation: default
+    ttl: 720h0m0s
+    snapshotMoveData: true
+---
+apiVersion: velero.io/v1
+kind: Schedule
 metadata:
   name: trmnl-daily
   namespace: velero
   labels:
     app.kubernetes.io/part-of: homelab-backup
 spec:
-  # Daily at 08:15 local (next free stagger slot after apps-daily at
-  # 07:15). Selects the whole namespace minus the CNPG-labelled PVCs, so
+  # Daily at 08:15 local (next free stagger slot after netbird-backup-daily
+  # at 07:45). Selects the whole namespace minus the CNPG-labelled PVCs, so
   # it picks up terminus-data (device screens + synchronised fonts),
   # terminus-valkey-data and trmnl-renderer-data (the renderer's schedule
   # and rendered output) without needing to be edited per PVC. The tr-pg