Compare commits

..
Author SHA1 Message Date
claude-botandClaude Opus 5 1211d97851 fix(llama-server): stable model alias, and thinking off by default
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Two problems found by exercising the live endpoint after #727.

The model id was the full --hf-repo string, so callers had to ask for
"ggml-org/Qwen3.6-35B-A3B-GGUF:Q4_K_M". That name changes whenever the model
does, which would break every consumer's config on each swap. --alias chat
pins a stable name that survives model changes.

Qwen3.6 also enables thinking by default. A request with max_tokens 300 came
back with an empty `content` and finish_reason "length" — the entire budget had
gone into reasoning_content without ever reaching an answer. At ~11.6 tok/s that
is minutes of reasoning before a reply, which is wrong for voice and for short
answers generally. Verified against the live endpoint: the same prompt with
chat_template_kwargs {"enable_thinking": false} returns a correct two-sentence
German answer in 95 tokens at 12.0 tok/s.

Off by default; a client that wants reasoning can still ask for it per request.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-01 16:31:30 +02:00
k.renner 682af5df88 Merge pull request 'feat(llama-server): split embeddings off, move chat to Qwen3.6-35B-A3B' (#727) from llama-split-embed into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
Reviewed-on: k.renner/homelab#727
2026-09-01 14:26:44 +00:00
k.renner f9ca731d5d Merge pull request 'fix(netbird): let the nightly dump wait out the writer' (#724) from netbird-backup-vacuum-into into master
ci/woodpecker/push/deploy-netbird-server Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#724
2026-09-01 13:54:39 +00:00
claude-botandClaude Opus 5 8cde6d5127 fix(netbird): let the nightly dump wait out the writer
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The dump ran sqlite3's .backup with no busy handler. sqlite3 defaults its
busy timeout to 0 ms, so the online-backup API gave up the first time the
management container held the write lock instead of waiting for it, and
set -eu turned that into a failed job. Four nightly runs failed that way
before this, three attempts each, every one refused within seconds of the
ssh handshake.

NetBird sets no journal_mode on the store — its DSN carries only
_busy_timeout=30000 and cache=shared — so store.db uses the default
rollback journal, where a writer's lock blocks a reader outright. Under
WAL the reader would have gone through and this would never have come up.

.timeout arms the busy handler on the source connection, which is the
contended side. The shell's .backup never arms one on the destination and
does not need to: that file is created fresh in $tmp and nothing else
touches it.

VACUUM INTO replaces .backup because a busy timeout on its own only turns
an instant failure into a slow one. .backup copies 100 pages per
sqlite3_backup_step() and SQLite restarts the copy whenever the source is
written between steps, so against a steady writer it returns only once the
writes stop. Measured on a rollback-journal database of the same shape,
with a writer committing continuously for 20 s: .backup returned after
19 s, VACUUM INTO returned in under a second, and integrity_check on its
output passed. Both failed identically with no timeout at all, which is
what reproduced the production error. VACUUM INTO also compacts, so the
dump lands smaller than the live file.

30 s sits well inside the puller's 600 s activeDeadlineSeconds, and the
read transaction VACUUM INTO holds is covered by NetBird's own 30 s busy
timeout on the writing side.

The script lives on the VPS at /usr/local/bin/netbird-backup-dump and is
refreshed only when this role runs, so merging this to master is what
deploys it — deploy-netbird-server triggers on ansible/roles/netbird-server/**.

Closes #714. Refs #712.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-01 15:51:41 +02:00
2 changed files with 37 additions and 5 deletions
@@ -1,10 +1,10 @@
#!/bin/sh
# Forced-command target for the netbird-backup SSH key (see the
# "Authorize the cluster backup key" task): takes a CONSISTENT snapshot
# of the management SQLite store via the online-backup API (safe against
# a live writer, unlike copying store.db) and streams a gzipped tar to
# stdout. The key's authorized_keys entry is restricted to exactly this
# script — the in-cluster puller cannot run anything else.
# of the management SQLite store (safe against a live writer, unlike
# copying store.db) and streams a gzipped tar to stdout. The key's
# authorized_keys entry is restricted to exactly this script — the
# in-cluster puller cannot run anything else.
#
# Deliberately store.db ONLY: /opt/netbird/config is fully regenerable
# from the netbird-server role's templates + vault, and including it
@@ -14,5 +14,23 @@
set -eu
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
sqlite3 /opt/netbird/management/store.db ".backup '$tmp/store.db'"
# NetBird sets no journal_mode on the store (its DSN carries only
# _busy_timeout and cache=shared), so store.db uses the default rollback
# journal, where the management container's write lock blocks a reader
# outright rather than letting it through as WAL would. Two things follow.
#
# .timeout arms a busy handler on the source connection, which is the
# contended side; the destination is a fresh file in $tmp that nothing else
# touches. Without it sqlite3 uses its 0 ms default and gives up the moment
# it cannot take the lock. 30 s sits well inside the puller's 600 s
# activeDeadlineSeconds.
#
# VACUUM INTO rather than .backup: the shell's .backup copies 100 pages per
# sqlite3_backup_step(), and SQLite restarts the copy whenever the source is
# written between steps, so a steady writer drags it out for as long as the
# writes keep coming. VACUUM INTO reads one transaction in a single pass and
# is documented as an alternative to the backup API for exactly this. It
# also compacts, so the dump is smaller than the live file.
sqlite3 -cmd '.timeout 30000' /opt/netbird/management/store.db \
"VACUUM INTO '$tmp/store.db'"
tar -C "$tmp" -czf - store.db
@@ -90,6 +90,20 @@ spec:
# to it server-side; the WebUI is same-origin and unaffected.
- --cors-origins
- localhost
# Stable name for the API, so consumers ask for "chat" rather than
# the full repo string. Without it the model id is the value of
# --hf-repo, and every consumer's config would break the next time
# the model is swapped.
- --alias
- chat
# Qwen3.6 enables thinking by default. At ~11.6 tok/s that spends
# minutes reasoning before answering, and a request with a modest
# max_tokens returns an empty `content` with finish_reason "length"
# because the whole budget went to reasoning_content. Off by default
# keeps answers usable; a client that wants it can send
# chat_template_kwargs per request.
- --chat-template-kwargs
- '{"enable_thinking": false}'
env:
# Where --hf-repo caches the weights. Pointing it at the PVC makes
# the download a one-time cost, shared with llama-embed, and keeps