feat(netbird): drop the cluster-side backup of the controller VPS

Backing up an external server is that server's own business. The pull path
made the cluster responsible for it and, worse, put a root key on the VPS
that the cluster held — a standing inbound grant that existed only to move
a SQLite file.

Removed cluster-side: the netbird-backup namespace and everything in it
(the nightly CronJob, the netbird-backup-ssh ExternalSecret, the 5Gi
ceph-block PVC), the namespace's entry in the OpenBao ClusterSecretStore
allow-list, the netbird-backup-daily Velero Schedule, and the
NetbirdVpsBackupStale / NetbirdVpsBackupNeverRan alerts.

Flux prunes the PVC, which destroys the 14 retained dumps (2026-08-15
through 08-28). That is intended. Velero's existing offsite copies age out
on their own under the schedule's 720h TTL; no cleanup is needed there.

Removed VPS-side: the dump script, and the forced-command key authorising
it. Deleting the two tasks that placed them would only stop ansible
managing them and leave both live on the host, so they are replaced with
state: absent tasks that actually revoke them. Those two can go once the
role has run against every host that ever carried them.

The trmnl-daily schedule's stagger comment pointed at netbird-backup-daily
for its slot; it now points at apps-daily, which is the schedule that
actually precedes it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

ansible/roles/netbird-server/files/netbird-backup-dump

@@ -1,18 +0,0 @@
-#!/bin/sh
-# Forced-command target for the netbird-backup SSH key (see the
-# "Authorize the cluster backup key" task): takes a CONSISTENT snapshot
-# of the management SQLite store via the online-backup API (safe against
-# a live writer, unlike copying store.db) and streams a gzipped tar to
-# stdout. The key's authorized_keys entry is restricted to exactly this
-# script — the in-cluster puller cannot run anything else.
-#
-# Deliberately store.db ONLY: /opt/netbird/config is fully regenerable
-# from the netbird-server role's templates + vault, and including it
-# would put the plaintext TURN secret (turnserver.conf) into every
-# nightly tarball. Restore = re-run the role (rebuilds config), then
-# untar this dump into /opt/netbird/management/.
-set -eu
-tmp=$(mktemp -d)
-trap 'rm -rf "$tmp"' EXIT
-sqlite3 /opt/netbird/management/store.db ".backup '$tmp/store.db'"
-tar -C "$tmp" -czf - store.db

ansible/roles/netbird-server/tasks/main.yml

@@ -77,29 +77,22 @@
   register: docker_compose_result
   changed_when: "'Started' in docker_compose_result.stdout or 'Creating' in docker_compose_result.stdout"
 
-# Off-VPS backup of the management store (SPOF assessment 2026-07-25:
-# this SQLite DB — peers, setup keys, groups, routes — had zero backup
-# coverage; losing the VPS meant re-enrolling the entire mesh). The VPS
-# cannot reach any PBS (both PBS names resolve to the public proxy,
-# ports closed), so the CLUSTER pulls: a CronJob
-# (kubernetes/infrastructure/netbird-backup) ssh's in with a dedicated
-# key that is locked to the dump script below, and lands the tarball on
-# a Ceph PVC that Velero ships offsite. The dump is store.db ONLY —
-# /opt/netbird/config is regenerable from this role's templates + vault
-# and would otherwise leak the TURN secret into every tarball. Restore:
-# re-run this role (rebuilds config), then untar the dump into
-# /opt/netbird/management/.
-- name: Install the backup dump script
-  ansible.builtin.copy:
-    src: netbird-backup-dump
-    dest: /usr/local/bin/netbird-backup-dump
-    mode: "0755"
+# The management store has no cluster-side backup. Pulling one put a root
+# key on this host that the cluster held, and this server's own state is
+# its own business to protect rather than the cluster's. Dropping the two
+# tasks that placed them would only stop ansible managing the key and the
+# script — the pair below revokes them instead, and can go once the role
+# has run against every host that ever carried them.
+- name: Remove the backup dump script
+  ansible.builtin.file:
+    path: /usr/local/bin/netbird-backup-dump
+    state: absent
 
-- name: Authorize the cluster backup key (forced command, no shell)
+- name: Revoke the cluster backup key
   ansible.posix.authorized_key:
     user: root
     key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFeW/L9bHMTSa5/Qb0ClppCf4Z0hL39JwzI6tBmKd2FG netbird-backup@homelab-cluster"
-    key_options: 'restrict,command="/usr/local/bin/netbird-backup-dump"'
+    state: absent
 
 - name: Wait for management container to be ready
   ansible.builtin.pause:

kubernetes/RUNBOOK-cold-rebuild.md

@@ -119,9 +119,11 @@ against do not exist, and the node ends up with no address on any interface.
 Beyond that it carries:
 
 - `siderolabs/intel-ucode` — no hypervisor supplies microcode here
-- `siderolabs/i915` — the Alder Lake-P Iris Xe. Nothing requests `/dev/dri`
-  today; present so wanting hardware transcoding later is a config change
-  rather than another rolling reinstall
+- `siderolabs/i915` — the Alder Lake-P Iris Xe. The Intel GPU device plugin
+  (`infrastructure/intel-gpu-plugin`) turns this into the schedulable resource
+  `gpu.intel.com/i915`, which `apps/llama-server` requests for LLM inference
+  and which any hardware-transcoding workload would use too. Without the
+  extension the plugin finds no `/dev/dri` and advertises nothing
 - `siderolabs/thunderbolt` — inert until `machine.kernel.modules` requests
   `thunderbolt`/`thunderbolt_net`, present for the same reason. Note its udev
   rule auto-authorises any Thunderbolt device attached once those modules load

kubernetes/apps/llama-server/configmap.yaml

@@ -0,0 +1,44 @@
+---
+# llama-server router presets. Running the server with --models-preset and no
+# -m puts it in router mode: every section below is a separately supervised
+# model subprocess, all reachable on one port through one OpenAI-compatible
+# /v1 API. Requests pick a section by name via the JSON body's "model" field,
+# so chat and embeddings are served by a single Deployment and Service.
+#
+# Keys are llama-server command-line arguments without the leading dashes.
+#
+# `hf-repo` makes this file the only place a model is named: llama-server
+# resolves and downloads the weights itself into LLAMA_CACHE (the models PVC)
+# on first load, so changing a model here needs no matching change anywhere
+# else. The quant after the colon is optional and case-insensitive.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: llama-server-models
+  namespace: ai
+data:
+  models.ini: |
+    version = 1
+
+    [*]
+    # Every layer runs on the Intel iGPU. Measured on this hardware (i9-12900H,
+    # Iris Xe 96 EU): full offload matches CPU throughput at ~10.8 tok/s decode
+    # while leaving the node's 14 cores free for the rest of the cluster. The
+    # iGPU is here for that isolation, not for a speed-up.
+    n-gpu-layers = 99
+    # Generation is memory-bandwidth-bound, so extra threads buy nothing once
+    # the layers are on the GPU. Keeps the pod within its CPU limit.
+    threads = 4
+
+    [chat]
+    hf-repo = bartowski/Meta-Llama-3.1-8B-Instruct-GGUF:Q4_K_M
+    ctx-size = 8192
+    load-on-startup = true
+
+    [embed]
+    hf-repo = Qwen/Qwen3-Embedding-0.6B-GGUF:Q8_0
+    # Serves POST /v1/embeddings. Set per-model rather than globally: the flag
+    # restricts a model to embeddings and would disable chat on [chat].
+    embedding = true
+    ctx-size = 2048
+    load-on-startup = true

kubernetes/apps/llama-server/deployment.yaml

@@ -0,0 +1,122 @@
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: llama-server
+  namespace: ai
+  labels:
+    app.kubernetes.io/name: llama-server
+spec:
+  replicas: 1
+  # One iGPU per node and one RWX volume of weights: a rolling update would
+  # start a second copy that loads the same models onto the same device.
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: llama-server
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: llama-server
+    spec:
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1000
+        runAsGroup: 1000
+        # Makes the CephFS volume group-writable so the initContainer can
+        # write the downloaded weights.
+        fsGroup: 1000
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: llama-server
+          image: ghcr.io/ggml-org/llama.cpp:server-vulkan-b10731
+          args:
+            # No -m: with only --models-preset the server starts in router
+            # mode and serves every section of the INI behind one /v1 API.
+            - --models-preset
+            - /config/models.ini
+            - --models-max
+            - "2"
+            # The server defaults to allowing every CORS origin, which would let
+            # any page a LAN browser visits call this unauthenticated API from
+            # the user's machine. Every consumer here talks to it server-side,
+            # so no cross-origin access is needed; the built-in WebUI is
+            # same-origin and unaffected.
+            - --cors-origins
+            - localhost
+          env:
+            # Where the `hf-repo` presets resolve to. Pointing the cache at the
+            # PVC is what makes the download a one-time cost: the weights are
+            # written here on first load and reused on every later start. Also
+            # keeps the server off the home directory it cannot write as UID 1000.
+            - name: LLAMA_CACHE
+              value: /models
+          ports:
+            - name: http
+              containerPort: 8080
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop:
+                - ALL
+          resources:
+            requests:
+              cpu: 500m
+              memory: 12Gi
+              gpu.intel.com/i915: "1"
+            limits:
+              cpu: "6"
+              memory: 16Gi
+              gpu.intel.com/i915: "1"
+          # /health reports 503 until every load-on-startup model is resident.
+          # On a cold PVC that includes pulling ~5 GiB from Hugging Face before
+          # either model can load, so the startup budget is deliberately long
+          # (20 min); liveness only takes over once startup has succeeded.
+          startupProbe:
+            httpGet:
+              path: /health
+              port: http
+            periodSeconds: 10
+            failureThreshold: 120
+          readinessProbe:
+            httpGet:
+              path: /health
+              port: http
+            periodSeconds: 10
+          livenessProbe:
+            httpGet:
+              path: /health
+              port: http
+            periodSeconds: 30
+            timeoutSeconds: 10
+            failureThreshold: 6
+          volumeMounts:
+            - name: models
+              mountPath: /models
+            - name: config
+              mountPath: /config
+              readOnly: true
+      volumes:
+        - name: models
+          persistentVolumeClaim:
+            claimName: llama-server-models
+        - name: config
+          configMap:
+            name: llama-server-models
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: llama-server
+  namespace: ai
+  labels:
+    app.kubernetes.io/name: llama-server
+spec:
+  selector:
+    app.kubernetes.io/name: llama-server
+  ports:
+    - name: http
+      port: 8080
+      targetPort: http

kubernetes/apps/llama-server/ingress.yaml

@@ -0,0 +1,28 @@
+---
+# LAN-only: no `dns.renner.uno/public` label, so the Cloudflare external-dns
+# instance skips it and only the UDM instance publishes the record. The API is
+# unauthenticated, so it must not be reachable from the internet.
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: llama-server
+  namespace: ai
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-cluster
+spec:
+  ingressClassName: traefik
+  tls:
+    - hosts:
+        - llm.renner.uno
+      secretName: llama-server-tls
+  rules:
+    - host: llm.renner.uno
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: llama-server
+                port:
+                  number: 8080

kubernetes/apps/llama-server/kustomization.yaml

@@ -0,0 +1,7 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - pvc.yaml
+  - configmap.yaml
+  - deployment.yaml
+  - ingress.yaml

kubernetes/apps/llama-server/pvc.yaml

@@ -0,0 +1,21 @@
+---
+# Holds the GGUF weights, downloaded once by the Deployment's initContainer.
+# Prune-disabled so retiring this Kustomization does not throw away several
+# GiB of model files that would have to be pulled from Hugging Face again.
+#
+# ReadWriteMany so a second replica or a one-off benchmark Job can mount the
+# same weights; the models are read-only at inference time.
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: llama-server-models
+  namespace: ai
+  annotations:
+    kustomize.toolkit.fluxcd.io/prune: disabled
+spec:
+  accessModes:
+    - ReadWriteMany
+  storageClassName: ceph-filesystem
+  resources:
+    requests:
+      storage: 40Gi

kubernetes/apps/monitoring-rules/kustomization.yaml

@@ -13,7 +13,6 @@ resources:
   - prometheusrule-loki.yaml
   - prometheusrule-mariadb-backup.yaml
   - prometheusrule-netbird-availability.yaml
-  - prometheusrule-netbird-backup.yaml
   - prometheusrule-netbird-ingress.yaml
   - prometheusrule-openbao.yaml
   - prometheusrule-velero.yaml

kubernetes/apps/monitoring-rules/prometheusrule-netbird-backup.yaml

@@ -1,44 +0,0 @@
-apiVersion: monitoring.coreos.com/v1
-kind: PrometheusRule
-metadata:
-  name: netbird-backup
-  namespace: monitoring
-  labels:
-    # kube-prometheus-stack's Prometheus selects on this label.
-    release: kube-prometheus-stack
-spec:
-  groups:
-    - name: netbird-backup.rules
-      rules:
-        - alert: NetbirdVpsBackupStale
-          # The nightly pull of the controller VPS's management store
-          # hasn't succeeded in over 26 h (one missed cycle + slack).
-          # This dump is the only copy of the mesh state (peers, setup
-          # keys, routes) that survives losing the Hetzner VPS — a
-          # silently dead backup is exactly the failure mode the SPOF
-          # assessment flagged.
-          expr: |
-            time() - kube_cronjob_status_last_successful_time{namespace="netbird-backup", cronjob="netbird-vps-backup"} > 26 * 3600
-          for: 30m
-          labels:
-            severity: critical
-          annotations:
-            summary: "NetBird VPS backup has not succeeded in 26h+"
-            description: "The netbird-vps-backup CronJob's last successful run is over 26 hours old. Check the job logs in the netbird-backup namespace (ssh reachability of netbird.renner.uno, the netbird-backup-ssh ExternalSecret, PVC space)."
-
-        - alert: NetbirdVpsBackupNeverRan
-          # The CronJob exists but has never succeeded — most likely the
-          # one-time OpenBao provisioning step
-          # (bao kv put homelab/infrastructure/netbird-backup ssh-key=@…)
-          # hasn't been done, so the ExternalSecret cannot sync and the
-          # job pod fails on the missing secret mount.
-          expr: |
-            kube_cronjob_created{namespace="netbird-backup", cronjob="netbird-vps-backup"}
-            unless
-            kube_cronjob_status_last_successful_time{namespace="netbird-backup", cronjob="netbird-vps-backup"}
-          for: 6h
-          labels:
-            severity: warning
-          annotations:
-            summary: "NetBird VPS backup has never succeeded"
-            description: "The netbird-vps-backup CronJob exists but has no successful run. If this is the initial rollout: the private key must be provisioned once by hand — bao kv put homelab/infrastructure/netbird-backup ssh-key=@<keyfile> — before the ExternalSecret can sync."

kubernetes/clusters/homelab/llama-server.yaml

@@ -0,0 +1,22 @@
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: llama-server
+  namespace: flux-system
+spec:
+  interval: 10m
+  retryInterval: 1m
+  # On a cold PVC llama-server pulls ~5 GiB of GGUF weights from Hugging Face
+  # and then loads both models onto the iGPU before /health goes green, so a
+  # first start is far slower than a normal app's.
+  timeout: 45m
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./kubernetes/apps/llama-server
+  prune: true
+  wait: true
+  dependsOn:
+    - name: infrastructure
+    - name: infrastructure-crds
+    - name: infrastructure-ceph-cluster

kubernetes/infrastructure/kustomization.yaml

@@ -18,7 +18,6 @@ resources:
   - cloudnative-pg/
   - mariadb-operator/
   - netbird-ingress/
-  - netbird-backup/
   - coredns-config/
   - k8s-gateway/
   - external-dns-unifi/

kubernetes/infrastructure/namespaces/namespaces.yaml

@@ -3,13 +3,6 @@ kind: Namespace
 metadata:
   name: apps
 ---
-# Pull-based off-VPS backup of the NetBird controller state
-# (kubernetes/infrastructure/netbird-backup).
-apiVersion: v1
-kind: Namespace
-metadata:
-  name: netbird-backup
----
 apiVersion: v1
 kind: Namespace
 metadata:
@@ -183,3 +176,15 @@ apiVersion: v1
 kind: Namespace
 metadata:
   name: woodpecker
+---
+# Local LLM inference (kubernetes/apps/llama-server). Declared here, like
+# `woodpecker` above, so the `infrastructure` KS creates it before
+# `infrastructure-crds` applies the namespace's CiliumNetworkPolicies.
+#
+# No pod-security labels: the workload needs no hostPath or privilege — the
+# Intel GPU device plugin passes the iGPU in through the device-plugin API —
+# so the cluster's default `baseline` enforcement is enough.
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: ai

kubernetes/infrastructure/netbird-backup/cronjob.yaml

@@ -1,97 +0,0 @@
-apiVersion: batch/v1
-kind: CronJob
-metadata:
-  name: netbird-vps-backup
-  namespace: netbird-backup
-  labels:
-    app.kubernetes.io/name: netbird-backup
-spec:
-  # Nightly at 02:45, well ahead of the 07:45 Velero Schedule
-  # (post-crds/velero-schedules.yaml) that ships this PVC offsite.
-  schedule: "45 2 * * *"
-  timeZone: Europe/Berlin
-  concurrencyPolicy: Forbid
-  successfulJobsHistoryLimit: 2
-  failedJobsHistoryLimit: 5
-  startingDeadlineSeconds: 3600
-  jobTemplate:
-    spec:
-      backoffLimit: 2
-      activeDeadlineSeconds: 600
-      ttlSecondsAfterFinished: 259200 # 3 d
-      template:
-        metadata:
-          labels:
-            app.kubernetes.io/name: netbird-backup
-        spec:
-          restartPolicy: Never
-          securityContext:
-            runAsNonRoot: true
-            runAsUser: 65534
-            runAsGroup: 65534
-            fsGroup: 65534
-            seccompProfile:
-              type: RuntimeDefault
-          containers:
-            - name: pull
-              # ci-ansible ships an ssh client and standard coreutils and
-              # is pulled from the local registry; a dedicated slim image
-              # is not worth maintaining for one nightly ssh pull.
-              image: gitea.renner.uno/gitea_admin/ci-ansible:2026-04-29
-              command:
-                - sh
-                - -ec
-                - |
-                  # The key's authorized_keys entry on the VPS is
-                  # restrict,command=… — the server runs the dump script
-                  # regardless of what we request, streaming a tar.gz.
-                  # ssh demands 0600 on the key; secret mounts are group-
-                  # readable, so copy first. The host key is PINNED (no
-                  # TOFU): captured 2026-07-26 via ssh-keyscan and
-                  # cross-checked against the workstation's known_hosts.
-                  # If the VPS is ever reprovisioned with a new host key,
-                  # update this line — the job failing loudly on a key
-                  # mismatch is the desired behavior.
-                  install -m 600 /ssh/ssh-key /tmp/key
-                  printf '%s\n' 'netbird.renner.uno ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKCe8LRXbyn6Y+5V4ia4ErnBEcTFzHI1fAnsTvT77HJU' > /tmp/known_hosts
-                  # Leftover .part files from aborted earlier runs are
-                  # outside the retention glob — clear them up front.
-                  rm -f /backup/*.part
-                  ts=$(date +%Y-%m-%d)
-                  ssh -i /tmp/key \
-                      -o StrictHostKeyChecking=yes \
-                      -o UserKnownHostsFile=/tmp/known_hosts \
-                      -o ConnectTimeout=15 \
-                      root@netbird.renner.uno > "/backup/netbird-$ts.tar.gz.part"
-                  gzip -t "/backup/netbird-$ts.tar.gz.part"
-                  mv "/backup/netbird-$ts.tar.gz.part" "/backup/netbird-$ts.tar.gz"
-                  # Retention: newest 14 dumps (Velero keeps 30 d offsite).
-                  ls -1t /backup/netbird-*.tar.gz | tail -n +15 | xargs -r rm --
-                  ls -lh /backup/
-              env:
-                - name: HOME
-                  value: /tmp
-              securityContext:
-                allowPrivilegeEscalation: false
-                capabilities:
-                  drop: ["ALL"]
-              resources:
-                requests:
-                  cpu: 25m
-                  memory: 64Mi
-                limits:
-                  memory: 256Mi
-              volumeMounts:
-                - name: ssh-key
-                  mountPath: /ssh
-                  readOnly: true
-                - name: backup
-                  mountPath: /backup
-          volumes:
-            - name: ssh-key
-              secret:
-                secretName: netbird-backup-ssh
-                defaultMode: 0440
-            - name: backup
-              persistentVolumeClaim:
-                claimName: netbird-backup

kubernetes/infrastructure/netbird-backup/externalsecret.yaml

@@ -1,21 +0,0 @@
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: netbird-backup-ssh
-  namespace: netbird-backup
-spec:
-  refreshInterval: 1h
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: netbird-backup-ssh
-  # OpenBao path homelab/infrastructure/netbird-backup, key `ssh-key`:
-  # the private half of the dedicated backup keypair whose public half
-  # is pinned (forced-command, restrict) in the VPS's authorized_keys by
-  # the netbird-server role. ONE-TIME provisioning by hand (same pattern
-  # as the rotator bootstrap credentials):
-  #   bao kv put homelab/infrastructure/netbird-backup ssh-key=@<keyfile>
-  dataFrom:
-    - extract:
-        key: infrastructure/netbird-backup

kubernetes/infrastructure/netbird-backup/kustomization.yaml

@@ -1,11 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-# Pull-based backup of the NetBird controller VPS's management store.
-# The VPS side (dump script + restricted SSH key) is deployed by the
-# netbird-server Ansible role; the Velero Schedule that ships the PVC
-# offsite lives in post-crds/velero-schedules.yaml; the staleness alert
-# in apps/monitoring-rules/prometheusrule-netbird-backup.yaml.
-resources:
-  - externalsecret.yaml
-  - pvc.yaml
-  - cronjob.yaml

kubernetes/infrastructure/netbird-backup/pvc.yaml

@@ -1,14 +0,0 @@
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: netbird-backup
-  namespace: netbird-backup
-spec:
-  # 14 daily dumps of a small SQLite store plus config — 5Gi is
-  # generous headroom.
-  storageClassName: ceph-block
-  accessModes:
-    - ReadWriteOnce
-  resources:
-    requests:
-      storage: 5Gi

kubernetes/infrastructure/network-policies/ai-ns.yaml

@@ -0,0 +1,96 @@
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: default-deny-ingress
+  namespace: ai
+spec:
+  endpointSelector: {}
+  ingress:
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": ai
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": ingress
+            app.kubernetes.io/name: traefik
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": monitoring
+    - fromEntities:
+        - host
+        - remote-node
+    # ICMP — all standard ICMPv4 + ICMPv6 from any cluster pod. See
+    # home-assistant-ns.yaml for the rationale (CT-RELATED path not
+    # passing related ICMP empirically; per-namespace shape, mirroring
+    # commit 3c517a0's verified-safe pattern with the 66dbe9f type set).
+    - fromEndpoints:
+        - matchExpressions:
+            - key: io.kubernetes.pod.namespace
+              operator: Exists
+      icmps:
+        - fields:
+            - { type: 0,   family: IPv4 }
+            - { type: 3,   family: IPv4 }
+            - { type: 8,   family: IPv4 }
+            - { type: 11,  family: IPv4 }
+            - { type: 12,  family: IPv4 }
+            - { type: 1,   family: IPv6 }
+            - { type: 2,   family: IPv6 }
+            - { type: 3,   family: IPv6 }
+            - { type: 4,   family: IPv6 }
+            - { type: 128, family: IPv6 }
+            - { type: 129, family: IPv6 }
+            - { type: 133, family: IPv6 }
+            - { type: 134, family: IPv6 }
+            - { type: 135, family: IPv6 }
+            - { type: 136, family: IPv6 }
+            - { type: 143, family: IPv6 }
+  egress:
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": ai
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": ingress
+            app.kubernetes.io/name: traefik
+    # rook-ceph: the model weights live on a CephFS RWX volume
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": rook-ceph
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": kube-system
+            k8s-app: kube-dns
+      toPorts:
+        - ports:
+            - port: "53"
+              protocol: UDP
+    # host/remote-node: kubelet probes + node-local reach (Ceph mons on host net)
+    # world: image pulls and llama-server's Hugging Face model download
+    - toEntities:
+        - host
+        - remote-node
+        - world
+    # ICMP — egress mirror; see ingress block above.
+    - toEndpoints:
+        - matchExpressions:
+            - key: io.kubernetes.pod.namespace
+              operator: Exists
+      icmps:
+        - fields:
+            - { type: 0,   family: IPv4 }
+            - { type: 3,   family: IPv4 }
+            - { type: 8,   family: IPv4 }
+            - { type: 11,  family: IPv4 }
+            - { type: 12,  family: IPv4 }
+            - { type: 1,   family: IPv6 }
+            - { type: 2,   family: IPv6 }
+            - { type: 3,   family: IPv6 }
+            - { type: 4,   family: IPv6 }
+            - { type: 128, family: IPv6 }
+            - { type: 129, family: IPv6 }
+            - { type: 133, family: IPv6 }
+            - { type: 134, family: IPv6 }
+            - { type: 135, family: IPv6 }
+            - { type: 136, family: IPv6 }
+            - { type: 143, family: IPv6 }

kubernetes/infrastructure/network-policies/kustomization.yaml

@@ -24,4 +24,5 @@ resources:
   - woodpecker-ns.yaml
   - trmnl-ns.yaml
   - wordpress-ns.yaml
+  - ai-ns.yaml
   - allow-cilium-health.yaml

kubernetes/infrastructure/post-crds/openbao-clustersecretstore.yaml

@@ -26,7 +26,6 @@ spec:
         - influxdb
         - ingress
         - monitoring
-        - netbird-backup # netbird-backup-ssh: the VPS-pull backup key
         - paperless
         - trmnl
         - velero

kubernetes/infrastructure/post-crds/velero-schedules.yaml

@@ -247,40 +247,14 @@ spec:
 ---
 apiVersion: velero.io/v1
 kind: Schedule
-metadata:
-  name: netbird-backup-daily
-  namespace: velero
-  labels:
-    app.kubernetes.io/part-of: homelab-backup
-spec:
-  # Daily at 07:45 local (next free stagger slot after apps-daily at
-  # 07:15). Ships the netbird-backup PVC — nightly pulls of the NetBird
-  # controller VPS's management store (SQLite dump + config, pulled at
-  # 02:45 by the netbird-vps-backup CronJob) — offsite. This is the
-  # only copy of the mesh state that survives losing the Hetzner VPS.
-  # Added 2026-07-26 (SPOF assessment: controller store had zero backup
-  # coverage). No CNPG in this namespace.
-  schedule: "45 7 * * *"
-  useOwnerReferencesInBackup: false
-  template:
-    includedNamespaces:
-      - netbird-backup
-    includedResources:
-      - persistentvolumeclaims
-    storageLocation: default
-    ttl: 720h0m0s
-    snapshotMoveData: true
----
-apiVersion: velero.io/v1
-kind: Schedule
 metadata:
   name: trmnl-daily
   namespace: velero
   labels:
     app.kubernetes.io/part-of: homelab-backup
 spec:
-  # Daily at 08:15 local (next free stagger slot after netbird-backup-daily
-  # at 07:45). Selects the whole namespace minus the CNPG-labelled PVCs, so
+  # Daily at 08:15 local (next free stagger slot after apps-daily at
+  # 07:15). Selects the whole namespace minus the CNPG-labelled PVCs, so
   # it picks up terminus-data (device screens + synchronised fonts),
   # terminus-valkey-data and trmnl-renderer-data (the renderer's schedule
   # and rendered output) without needing to be edited per PVC. The tr-pg