fix(llama-server): stable model alias, and thinking off by default

Two problems found by exercising the live endpoint after #727.

The model id was the full --hf-repo string, so callers had to ask for
"ggml-org/Qwen3.6-35B-A3B-GGUF:Q4_K_M". That name changes whenever the model
does, which would break every consumer's config on each swap. --alias chat
pins a stable name that survives model changes.

Qwen3.6 also enables thinking by default. A request with max_tokens 300 came
back with an empty `content` and finish_reason "length" — the entire budget had
gone into reasoning_content without ever reaching an answer. At ~11.6 tok/s that
is minutes of reasoning before a reply, which is wrong for voice and for short
answers generally. Verified against the live endpoint: the same prompt with
chat_template_kwargs {"enable_thinking": false} returns a correct two-sentence
German answer in 95 tokens at 12.0 tok/s.

Off by default; a client that wants reasoning can still ask for it per request.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

ansible/roles/netbird-server/files/netbird-backup-dump

@@ -1,10 +1,10 @@
 #!/bin/sh
 # Forced-command target for the netbird-backup SSH key (see the
 # "Authorize the cluster backup key" task): takes a CONSISTENT snapshot
-# of the management SQLite store via the online-backup API (safe against
-# a live writer, unlike copying store.db) and streams a gzipped tar to
-# stdout. The key's authorized_keys entry is restricted to exactly this
-# script — the in-cluster puller cannot run anything else.
+# of the management SQLite store (safe against a live writer, unlike
+# copying store.db) and streams a gzipped tar to stdout. The key's
+# authorized_keys entry is restricted to exactly this script — the
+# in-cluster puller cannot run anything else.
 #
 # Deliberately store.db ONLY: /opt/netbird/config is fully regenerable
 # from the netbird-server role's templates + vault, and including it
@@ -14,5 +14,23 @@
 set -eu
 tmp=$(mktemp -d)
 trap 'rm -rf "$tmp"' EXIT
-sqlite3 /opt/netbird/management/store.db ".backup '$tmp/store.db'"
+# NetBird sets no journal_mode on the store (its DSN carries only
+# _busy_timeout and cache=shared), so store.db uses the default rollback
+# journal, where the management container's write lock blocks a reader
+# outright rather than letting it through as WAL would. Two things follow.
+#
+# .timeout arms a busy handler on the source connection, which is the
+# contended side; the destination is a fresh file in $tmp that nothing else
+# touches. Without it sqlite3 uses its 0 ms default and gives up the moment
+# it cannot take the lock. 30 s sits well inside the puller's 600 s
+# activeDeadlineSeconds.
+#
+# VACUUM INTO rather than .backup: the shell's .backup copies 100 pages per
+# sqlite3_backup_step(), and SQLite restarts the copy whenever the source is
+# written between steps, so a steady writer drags it out for as long as the
+# writes keep coming. VACUUM INTO reads one transaction in a single pass and
+# is documented as an alternative to the backup API for exactly this. It
+# also compacts, so the dump is smaller than the live file.
+sqlite3 -cmd '.timeout 30000' /opt/netbird/management/store.db \
+    "VACUUM INTO '$tmp/store.db'"
 tar -C "$tmp" -czf - store.db

kubernetes/apps/llama-server/configmap.yaml

@@ -1,44 +0,0 @@
----
-# llama-server router presets. Running the server with --models-preset and no
-# -m puts it in router mode: every section below is a separately supervised
-# model subprocess, all reachable on one port through one OpenAI-compatible
-# /v1 API. Requests pick a section by name via the JSON body's "model" field,
-# so chat and embeddings are served by a single Deployment and Service.
-#
-# Keys are llama-server command-line arguments without the leading dashes.
-#
-# `hf-repo` makes this file the only place a model is named: llama-server
-# resolves and downloads the weights itself into LLAMA_CACHE (the models PVC)
-# on first load, so changing a model here needs no matching change anywhere
-# else. The quant after the colon is optional and case-insensitive.
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: llama-server-models
-  namespace: ai
-data:
-  models.ini: |
-    version = 1
-
-    [*]
-    # Every layer runs on the Intel iGPU. Measured on this hardware (i9-12900H,
-    # Iris Xe 96 EU): full offload matches CPU throughput at ~10.8 tok/s decode
-    # while leaving the node's 14 cores free for the rest of the cluster. The
-    # iGPU is here for that isolation, not for a speed-up.
-    n-gpu-layers = 99
-    # Generation is memory-bandwidth-bound, so extra threads buy nothing once
-    # the layers are on the GPU. Keeps the pod within its CPU limit.
-    threads = 4
-
-    [chat]
-    hf-repo = bartowski/Meta-Llama-3.1-8B-Instruct-GGUF:Q4_K_M
-    ctx-size = 8192
-    load-on-startup = true
-
-    [embed]
-    hf-repo = Qwen/Qwen3-Embedding-0.6B-GGUF:Q8_0
-    # Serves POST /v1/embeddings. Set per-model rather than globally: the flag
-    # restricts a model to embeddings and would disable chat on [chat].
-    embedding = true
-    ctx-size = 2048
-    load-on-startup = true

kubernetes/apps/llama-server/deployment.yaml

@@ -1,4 +1,21 @@
 ---
+# The chat endpoint, and the only one behind the ingress.
+#
+# Single-model rather than router mode. Router mode served chat and embeddings
+# from one port, but two things made that wrong here:
+#
+#   1. Its /health answers 200 as soon as the router process listens, while the
+#      models are still downloading behind it. Kubernetes marked the pod Ready
+#      within seconds of start, so traffic arrived at a server that could not
+#      answer. In single-model mode /health returns 503 until the model is
+#      resident, which is what the probes below actually need.
+#   2. Every model on the port is selectable in the built-in WebUI, including
+#      the embedding model. Asking an embedding model to chat returns fluent
+#      nonsense ("capital capital capital ...") rather than an error, and
+#      `--embeddings` does not prevent it. There is no flag to hide a model
+#      from the list — `--tags` is informational and `--ui-config` only sets UI
+#      preferences — so the only way to keep it out of the picker is to keep it
+#      off this endpoint. It runs in embed.yaml instead.
 apiVersion: apps/v1
 kind: Deployment
 metadata:
@@ -8,8 +25,8 @@ metadata:
     app.kubernetes.io/name: llama-server
 spec:
   replicas: 1
-  # One iGPU per node and one RWX volume of weights: a rolling update would
-  # start a second copy that loads the same models onto the same device.
+  # A rolling update would load a second copy of the weights onto the same
+  # iGPU and into the same node's RAM before releasing the first.
   strategy:
     type: Recreate
   selector:
@@ -24,8 +41,7 @@ spec:
         runAsNonRoot: true
         runAsUser: 1000
         runAsGroup: 1000
-        # Makes the CephFS volume group-writable so the initContainer can
-        # write the downloaded weights.
+        # Makes the CephFS volume group-writable so the weights can be cached.
         fsGroup: 1000
         seccompProfile:
           type: RuntimeDefault
@@ -33,24 +49,65 @@ spec:
         - name: llama-server
           image: ghcr.io/ggml-org/llama.cpp:server-vulkan-b10731
           args:
-            # No -m: with only --models-preset the server starts in router
-            # mode and serves every section of the INI behind one /v1 API.
-            - --models-preset
-            - /config/models.ini
-            - --models-max
-            - "2"
-            # The server defaults to allowing every CORS origin, which would let
-            # any page a LAN browser visits call this unauthenticated API from
-            # the user's machine. Every consumer here talks to it server-side,
-            # so no cross-origin access is needed; the built-in WebUI is
-            # same-origin and unaffected.
+            # The one place this model is named; llama-server resolves the repo
+            # and downloads it itself, so there is nothing to keep in sync.
+            #
+            # A 35B mixture-of-experts with only ~3B parameters active per
+            # token. Decode on this hardware is bound by how many bytes are read
+            # per token rather than by parameter count, and this model reads
+            # ~1.85 GB/token where a dense 27B reads ~16.5 GB — so it is both
+            # far larger and faster than any dense model that would fit.
+            #
+            # Measured here (llama-bench, k8s-1, this image): tg128 11.59 tok/s,
+            # against 10.78 for a dense 8B. Prefill 120.9 t/s at 512 tokens and
+            # 69.8 at 16384 — a sublinear falloff, so the Gated Delta Net
+            # prefill collapse of llama.cpp#27638 does not occur on this stack.
+            - --hf-repo
+            - ggml-org/Qwen3.6-35B-A3B-GGUF:Q4_K_M
+            # Every layer on the Intel iGPU. Full offload matches CPU throughput
+            # while leaving the node's 14 cores free for the rest of the
+            # cluster; the iGPU is here for that isolation, not for a speed-up.
+            - --n-gpu-layers
+            - "99"
+            # Decode runs on the GPU, so extra threads buy nothing. At 11.59
+            # tok/s this model moves 21.4 GB/s against the 53.9 GB/s a dense
+            # model sustains here, so it is the iGPU's 96 EUs working through
+            # many small per-expert matmuls that sets the pace, not bandwidth.
+            - --threads
+            - "4"
+            # 16k rather than 8k because the measured prefill falloff is mild
+            # and this model's KV cache is unusually cheap — only 10 of its 40
+            # layers use full attention.
+            - --ctx-size
+            - "16384"
+            # A micro-batch of 2048 or more makes this Vulkan/Mesa stack fail
+            # with ErrorDeviceLost.
+            - --ubatch-size
+            - "1024"
+            # The server otherwise allows every CORS origin and warns about it
+            # at startup. Without this, any page a LAN browser visits could call
+            # this unauthenticated API from the user's machine. Consumers talk
+            # to it server-side; the WebUI is same-origin and unaffected.
             - --cors-origins
             - localhost
+            # Stable name for the API, so consumers ask for "chat" rather than
+            # the full repo string. Without it the model id is the value of
+            # --hf-repo, and every consumer's config would break the next time
+            # the model is swapped.
+            - --alias
+            - chat
+            # Qwen3.6 enables thinking by default. At ~11.6 tok/s that spends
+            # minutes reasoning before answering, and a request with a modest
+            # max_tokens returns an empty `content` with finish_reason "length"
+            # because the whole budget went to reasoning_content. Off by default
+            # keeps answers usable; a client that wants it can send
+            # chat_template_kwargs per request.
+            - --chat-template-kwargs
+            - '{"enable_thinking": false}'
           env:
-            # Where the `hf-repo` presets resolve to. Pointing the cache at the
-            # PVC is what makes the download a one-time cost: the weights are
-            # written here on first load and reused on every later start. Also
-            # keeps the server off the home directory it cannot write as UID 1000.
+            # Where --hf-repo caches the weights. Pointing it at the PVC makes
+            # the download a one-time cost, shared with llama-embed, and keeps
+            # the server off a home directory it cannot write as UID 1000.
             - name: LLAMA_CACHE
               value: /models
           ports:
@@ -61,25 +118,30 @@ spec:
             capabilities:
               drop:
                 - ALL
+          # Vulkan allocates the iGPU's working set from system RAM, and that
+          # counts against the pod's cgroup: a 5.2 GiB model measured 7.54 GiB
+          # RSS. Sized as the 19 GiB of weights plus that overhead, plus the KV
+          # cache — which is unusually cheap here, since only 10 of the 40
+          # layers use full attention.
           resources:
             requests:
               cpu: 500m
-              memory: 12Gi
+              memory: 24Gi
               gpu.intel.com/i915: "1"
             limits:
               cpu: "6"
-              memory: 16Gi
+              memory: 32Gi
               gpu.intel.com/i915: "1"
-          # /health reports 503 until every load-on-startup model is resident.
-          # On a cold PVC that includes pulling ~5 GiB from Hugging Face before
-          # either model can load, so the startup budget is deliberately long
-          # (20 min); liveness only takes over once startup has succeeded.
+          # /health is 503 while the model loads and 200 once it is resident, so
+          # the pod only goes Ready when it can actually answer. On a cold PVC
+          # that includes pulling ~19 GiB from Hugging Face, hence the long
+          # startup budget (40 min); liveness takes over only after that.
           startupProbe:
             httpGet:
               path: /health
               port: http
             periodSeconds: 10
-            failureThreshold: 120
+            failureThreshold: 240
           readinessProbe:
             httpGet:
               path: /health
@@ -95,16 +157,10 @@ spec:
           volumeMounts:
             - name: models
               mountPath: /models
-            - name: config
-              mountPath: /config
-              readOnly: true
       volumes:
         - name: models
           persistentVolumeClaim:
             claimName: llama-server-models
-        - name: config
-          configMap:
-            name: llama-server-models
 ---
 apiVersion: v1
 kind: Service

kubernetes/apps/llama-server/embed.yaml

@@ -0,0 +1,119 @@
+---
+# The embedding endpoint, for RAG and semantic search.
+#
+# Deliberately separate from llama-server and deliberately not behind the
+# ingress: an embedding model has no generative training, and when it shares a
+# port with the chat model the built-in WebUI lists it as a choice. Selecting it
+# returns fluent nonsense rather than an error, because `--embeddings` does not
+# actually refuse chat completions. Keeping it on its own ClusterIP-only Service
+# is what makes it unselectable, since llama.cpp has no way to hide a model from
+# the picker.
+#
+# Reachable in-cluster at http://llama-embed.ai.svc.cluster.local:8080/v1/embeddings.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: llama-embed
+  namespace: ai
+  labels:
+    app.kubernetes.io/name: llama-embed
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: llama-embed
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: llama-embed
+    spec:
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1000
+        runAsGroup: 1000
+        fsGroup: 1000
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: llama-embed
+          image: ghcr.io/ggml-org/llama.cpp:server-vulkan-b10731
+          args:
+            - --hf-repo
+            - Qwen/Qwen3-Embedding-0.6B-GGUF:Q8_0
+            # Serves POST /v1/embeddings.
+            - --embeddings
+            - --n-gpu-layers
+            - "99"
+            - --threads
+            - "2"
+            - --ctx-size
+            - "2048"
+            # No browser ever talks to this Service, so the WebUI is dead weight
+            # and one more unauthenticated surface.
+            - --no-webui
+          env:
+            # Same cache as llama-server, so the two share one copy of anything
+            # they have in common and neither re-downloads the other's weights.
+            - name: LLAMA_CACHE
+              value: /models
+          ports:
+            - name: http
+              containerPort: 8080
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop:
+                - ALL
+          resources:
+            requests:
+              cpu: 250m
+              memory: 2Gi
+              gpu.intel.com/i915: "1"
+            limits:
+              cpu: "2"
+              memory: 4Gi
+              gpu.intel.com/i915: "1"
+          # ~600 MiB rather than ~5 GiB, so a much shorter budget than the chat
+          # model needs, but still sized for a cold download.
+          startupProbe:
+            httpGet:
+              path: /health
+              port: http
+            periodSeconds: 10
+            failureThreshold: 60
+          readinessProbe:
+            httpGet:
+              path: /health
+              port: http
+            periodSeconds: 10
+          livenessProbe:
+            httpGet:
+              path: /health
+              port: http
+            periodSeconds: 30
+            timeoutSeconds: 10
+            failureThreshold: 6
+          volumeMounts:
+            - name: models
+              mountPath: /models
+      volumes:
+        - name: models
+          persistentVolumeClaim:
+            claimName: llama-server-models
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: llama-embed
+  namespace: ai
+  labels:
+    app.kubernetes.io/name: llama-embed
+spec:
+  selector:
+    app.kubernetes.io/name: llama-embed
+  ports:
+    - name: http
+      port: 8080
+      targetPort: http

kubernetes/apps/llama-server/kustomization.yaml

@@ -2,6 +2,6 @@ apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
   - pvc.yaml
-  - configmap.yaml
   - deployment.yaml
+  - embed.yaml
   - ingress.yaml

kubernetes/clusters/homelab/llama-server.yaml

@@ -6,10 +6,10 @@ metadata:
 spec:
   interval: 10m
   retryInterval: 1m
-  # On a cold PVC llama-server pulls ~5 GiB of GGUF weights from Hugging Face
-  # and then loads both models onto the iGPU before /health goes green, so a
-  # first start is far slower than a normal app's.
-  timeout: 45m
+  # On a cold PVC the chat model pulls ~19 GiB of GGUF weights from Hugging
+  # Face and then loads onto the iGPU before /health goes green, so a first
+  # start is far slower than a normal app's.
+  timeout: 60m
   sourceRef:
     kind: GitRepository
     name: flux-system