The LLM index rebuild ran for 79 minutes and then died:
input (1069 tokens) is too large to process.
increase the physical batch size (current batch size: 1024)
Setting the batch to Paperless's chunk size looked like an exact match
and was not. Paperless counts its 1024 tokens with llama_index's
tokenizer; this server measures the batch in bge-m3's, which is
XLM-RoBERTa and splits German into more pieces. A chunk Paperless
considered full arrived here 4.4% over the limit, and the request failed
rather than truncating.
1536 absorbs a tokenizer ratio rather than a rounding error. 2048 stays
out of reach: that is where the chat model hit ErrorDeviceLost on this
Vulkan stack, under a 20 GB model's memory pressure that a 600 MB one may
well not reproduce — but there is no reason to go looking.
The index is left partial by the failed run and has to be rebuilt with
`document_llmindex rebuild` once this is live.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The download moved to the controller in the previous fix and now works. The
extraction that followed it does not:
Failed to find handler for "/tmp/crowdsec-firewall-bouncer-v0.0.36.tgz".
Command "/bin/tar" detected as tar type None. GNU tar required.
The ci-ansible image ships a tar that ansible's unarchive rejects. Both host
distributions have GNU tar, and only the download ever needed to move — the
v6-only problem was reaching github.com, not opening a tarball.
So the controller now ships the archive rather than the unpacked binary, and
the host extracts it. That also keeps the artefact the checksum was verified
against intact all the way to its destination, instead of verifying an archive
and then copying something extracted from it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
First candidate to be carried through to an end-to-end measurement. The
embedding comparison on 400 real archive documents puts bge-m3 level with the
deployed Qwen3-Embedding-0.6B where it matters and ahead everywhere else:
correspondent 74.0% vs 75.0% (ceiling 82.5%, so 90% vs 91% of achievable)
document type 70.0% vs 68.2%
time 147s vs 380s for 400 documents
peak memory 1.6 GiB vs 10.6 GiB
Same 1024 dimensions, so nothing downstream changes shape.
Note the pooling type differs — bge-m3 wants CLS where the Qwen family wants
last-token. Read from the model's own 1_Pooling/config.json rather than assumed;
across the candidates measured, three different pooling types were required, and
a wrong one produces plausible but untrained vectors with no error anywhere.
The memory limit is now measured rather than estimated. It also supersedes #745,
which raised the limit for the model being replaced: the footprint is dominated
by four slots at 4096 context plus the Vulkan working set, not by the weights —
which is why several candidates were OOMKilled at limits that looked generous
against their model size.
This is not a final answer. Every candidate has to go through the end-to-end
test (scripts/paperless-eval.sh) before the embedding choice means anything,
because nothing has yet verified that the retrieval metric predicts what
Paperless actually suggests. bge-m3 is simply the one to start with.
Changing paperless-config is also what lets that test run at all: the AI
settings from #739 have never reached the running pod, which predates the
Reloader annotation from #741. This edit gives Reloader something real to react
to, rather than forcing a restart by hand or inventing a change to trigger one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The deploy failed on proxy-1:
Request failed: <urlopen error [Errno 101] Network is unreachable>
url: https://github.com/.../crowdsec-firewall-bouncer-linux-amd64.tgz
proxy-1 is IPv6-only, and neither github.com nor
objects.githubusercontent.com publishes an AAAA record. The host cannot reach
GitHub at all, so no retry or timeout would have helped.
Two alternatives do not work either. Mirroring into gitea.renner.uno is out:
its public records are the internal LB VIP (10.2.99.86 / fd99::86), so it is
unreachable from outside the LAN. And there is no upstream container image for
this bouncer, which is expected of something that manipulates host netfilter —
crowdsecurity publishes cloudflare, fastly, custom and others, but no firewall
bouncer.
So the controller fetches and pushes instead of the host pulling. The CI runner
has v4. The checksum is still verified, now before anything reaches a host,
which is the better place for it anyway.
The repo had told me twice that proxy-1 is the v6-only member — in
unifi_nat_hairpin.tf and in the netbird-proxy role's dhcpcd comment — and I
wrote it into the PR description myself. I never asked whether a v6-only host
can reach GitHub.
Nothing was deployed before this: the play runs serial: 1, proxy-1 went first
and failed, so proxy-2 never started.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Found by the pr-review bot on #751, and it is the exact failure this design
was written to avoid.
BOUNCER_KEY_edge is consumed as an env var from a secretKeyRef, and env is
fixed at pod start. The Reloader annotation listed only crowdsec-bouncer-key,
so a rotation of the edge key would update OpenBao, sync into the Secret, and
reach the hosts through the Ansible play — while the LAPI kept the old value
in its environment and went on trusting the old key. The bouncers would then
present the new one and get 401s, at exactly the moment the rotation was
supposed to be seamless.
The comment directly above that annotation already stated the rule — "it's an
env-var secret, so a restart is required to pick up a rotated key". I added a
second env-var secret and did not extend the list.
Comma-separated, matching shelly-manager-deployment.yaml.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Completes the pair: the ExternalSecret feeds BOUNCER_KEY_edge to the LAPI, this
feeds the same value to the bouncers that authenticate against it.
A single encrypted value rather than a whole-file vault like its siblings in
group_vars/all and group_vars/edge_host. Appending to one of those means
decrypting it first, which exposes every other secret in that file to add one
line. encrypt_string keeps the blast radius at this value, and the plaintext
never reaches a file: it goes from `bao kv get -field=key` through a pipe
straight into the encryption.
Scoped to netbird_proxy rather than all, because only those hosts run a
bouncer.
Not enrolled in the quarterly rotator, unlike crowdsec-bouncer-key. Nothing on
these hosts re-reads OpenBao on a schedule, so a rotation there would leave
them 401ing against the LAPI until someone ran the playbook. Rotating this one
means updating OpenBao and re-running plays/install-netbird-proxy.yml, in that
order.
Verified with `ansible-inventory --list`: the inventory parses and the value
decrypts.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Enforces the decisions the in-cluster CrowdSec already detects, at the point
where public traffic actually enters. These hosts terminate TLS for every
*.renner.uno name, so a drop here never reaches the mesh, the relay or the
cluster. Detection stays where it is: the reverse-proxy produces no parseable
access log and has no collection, while Traefik has a working one, and every
scenario in use needs HTTP detail that does not exist at this layer.
Deployed in dry-run. It pulls decisions and logs what it would block without
creating a single nftables rule. The first bouncer to arm anywhere enforces
~24k CAPI decisions at once on the only external entry path, so the
false-positive window has to come before the enforcement rather than after it.
Arming is flipping crowdsec_bouncer_mode per host.
The static release tarball, not a package. The AUR package
crowdsec-firewall-bouncer-nftables depends on `crowdsec` and would pull the
whole Security Engine onto hosts that need only the bouncer; the group is
mixed (proxy-1 Arch, VPS-1 Debian); and the AUR would be a pattern this repo
does not otherwise have. The tarball is the same artefact the AUR package
downloads, pinned by version and sha256, and only the binary is unpacked —
the install/upgrade/uninstall scripts beside it manage systemd and config
themselves, which would leave two owners for the same files.
retry_initial_connect is set, and it is not cosmetic. Without it the process
exits when the LAPI is unreachable at startup, and since a clean shutdown
deletes the nftables tables, the systemd restart loop would leave the host
with no blocklist at all until the first successful connect. The flag exists
in the code but not in the shipped example config.
nftables_hooks covers input and forward. input is the traffic this exists for,
because the reverse-proxy is host-networked and owns :443. forward covers the
bridge-networked http-redirect sidecar, whose packets never traverse INPUT —
the same Docker bridge that showed up as 172.18.0.3 in the Traefik access log.
Prometheus is enabled but bound to loopback. These hosts are public and have
no metrics collection path today; opening :60601 to the internet to gain a
scrape target is the wrong trade.
No CA file: Traefik serves Let's Encrypt for crowdsec.renner.uno, and
ca_cert_path is added to the system pool rather than replacing it.
Rollback is `systemctl stop` — a clean stop deletes both tables by itself.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The allowlist carried the published proxy addresses on the assumption that a
request from a proxy host arrives carrying that host's public address. It does
not.
The client connects to the local proxy on 127.0.0.1, the reverse-proxy writes
what it saw into X-Forwarded-For, and Traefik resolves that into ClientHost.
Measured against netbird-proxy-watchdog, which makes exactly this kind of
request every round:
172.18.0.3 - - [02/Sep/2026:10:12:01] "GET /" 200 websecure-immich-...
172.18.0.3 - - [02/Sep/2026:10:12:01] "GET /" 200 websecure-home-...
Two probes one second apart, one per watchdog target. 172.18.0.3 is a Docker
bridge address on the proxy side.
So the allowlist would have 403ed the bouncer. Allowlisting 172.18.0.0/16
instead is not a control: it is a private range on a host whose addressing we
do not own, and it would admit anything else that happens to originate there.
The bouncer key stays as the access control, which is what it always was. The
endpoint serves read-only decision streams to a valid key; the admin surface
is cscli, in-cluster. If the source address ever becomes deterministic, an
ipAllowList can go back on top.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Groundwork for enforcing CrowdSec decisions on proxy-1/proxy-2. Inert on its
own: an Ingress nobody calls yet, a bouncer key nobody uses yet, and a
whitelist that only ever narrows what may be banned.
The proxies cannot reach the LAPI over the mesh. The netbird-proxy container
runs non-root — that is what the ip_unprivileged_port_start task in the role
is for — so it has no NET_ADMIN and no /dev/net/tun, and its embedded NetBird
client terminates the tunnel in userspace. The tunnel belongs to that process,
not to the host netns, so a second process on the same host cannot route
through it. The alternative was a host-level NetBird client plus the LAPI on a
LoadBalancer VIP: two more peers, an ACL policy and a CNP rule. This reuses
the path every other service already takes.
Exposure is narrowed twice. The IPAllowList carries the three proxy addresses,
resolving them past the netbird-ingress pod IP through X-Forwarded-For the
same way the rate-limit middleware does — without that it would compare
against the shared forwarder IP, which is the gap behind the 2026-06-27 mass
ban. Behind it, the bouncer key. The LAPI serves read-only decision streams to
a valid key; the admin surface is cscli, in-cluster.
No CNP change is needed: crowdsec-ns.yaml already admits the traefik pods, and
ingress-default-deny carries no egress section.
BOUNCER_KEY_edge is separate from BOUNCER_KEY_ingress on purpose. The latter
rotates quarterly and both its consumers are in-cluster, so ESO propagates
within 10m. The edge hosts are Ansible-managed and read OpenBao on no
schedule, so a rotation would leave them 401ing until someone runs the
playbook. This key rotates with the playbook that carries it.
capi_whitelists.yaml is the only declarative allowlist the chart offers.
`cscli allowlists` objects live in the LAPI database, which runs without
persistence here deliberately, so anything created that way is gone at the
next restart. This file is a ConfigMap mount and survives by construction.
Upstream deprecates it; it is still the only thing that works against a LAPI
with no PVC.
It lists the proxy addresses and not the home WAN address, which is not
needed: unifi_nat_hairpin.tf DNATs the public proxy addresses to the Traefik
VIP on the UDM, so LAN clients reach the cluster directly with a LAN source
and never traverse the proxies.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The embedding model was picked from a recommendation and never compared. MTEB's
German retrieval results put the deployed Qwen3-Embedding-0.6B well down the
field, so a dozen candidates are being measured against the real archive with
scripts/embed-eval-real.sh — and they will not fit in 40Gi alongside the two
deployed models.
Growing the shared cache rather than using a scratch volume is deliberate: on a
slow connection the download is the long pole, and a shared cache lets the large
candidates pull in the background while the small ones are already being
measured. It also means nothing is fetched twice across runs, and a candidate
that wins is already local when it is promoted.
120Gi against 309 GiB currently free in cephfs-data0. CephFS has
allowVolumeExpansion, so this grows online without disturbing the running pods.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Immich's in-app migration to VectorChord is complete: both vector indexes,
face_index and clip_index, run on the vchordrq access method, and nothing uses
pgvecto.rs. The manifest still described a migration window.
`DROP EXTENSION vectors` was run against the immich database before this. It
was safe to do without CASCADE — nothing outside the extension depended on it,
which a rolled-back `BEGIN; DROP EXTENSION vectors; ROLLBACK;` confirmed
first. The embedding columns are `public.vector(512)`, which belongs to
pgvector, not to pgvecto.rs's type in the `vectors` schema. Afterwards both
vchordrq indexes are intact at 96 MB and 79 MB and the cluster stays healthy.
With the extension gone, vectors.so has nothing to serve and preloading it
only recreates pg_vectors in PGDATA. pgvector needs no preloading, so vchord.so
is the whole list.
instances goes back to 2. The blocker was never the extension as such — it was
that pg_basebackup's tar format cannot represent the long
pg_vectors/indexes/<uuid> paths pgvecto.rs creates, so a replica could not
join. That directory is now empty, 20 KiB in total, so the constraint is gone.
The image keeps its `:16-migration` tag. That is the upstream build's own name
for a multi-extension image, not a statement about this cluster, and it still
provides the two extensions in use. Moving to a different image is a separate
change with its own risk.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both documents described work as outstanding that the cluster shows finished.
A record that says "REMAINING" long after the fact is worse than no record:
it is read as current state, and the next person plans around it.
## Home Assistant recorder
Phase 2c is done, and done correctly. Against the live database:
statistics 11,092,586 2024-03-13 20:00 -> current
statistics_short_term 1,883,709 rolling ~7 days
statistics_meta 1,480
The 2024-03-13 origin is the part that matters. It is unchanged from the
10,793,712-row count recorded on 2026-08-19, so the history was copied across
engines rather than started from zero — the exact failure the document was
written to prevent. `states` carries current rows, so db_url points at
Postgres.
The three "INERT"/"PHASE 1" comments went with it. All three described a
cluster that nothing consumed; it is now the recorder's live database, and the
render-db-url init container is load-bearing rather than inert — HA will not
start without the secret it renders.
## Bare-metal conversion
`talosctl get systeminformation` reports Minisforum MS-01 hardware with
distinct chassis serials for all three nodes and no hypervisor, so saturn and
neptune are not outstanding. Node creationTimestamp is not evidence either way:
k8s-3 still carries its 2026-05-09 object and is on metal regardless.
The runbook stays as the procedure for a rebuild.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The AI settings from #739 applied cleanly to git and to the ConfigMap in the
cluster, and the application never saw them: the pod was 13 minutes old with no
PAPERLESS_AI_* in its environment at all.
paperless-config is consumed with envFrom, and Kubernetes does not restart a pod
when a ConfigMap read that way changes — the container keeps the environment it
started with. The Deployment carried a Reloader annotation for paperless-secrets
only, so secret rotations restarted it but settings changes did not.
Adding the ConfigMap to Reloader closes that, and makes the annotation pair
match how the ConfigMap is actually consumed rather than only the Secret.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Paperless 3.x has LLM suggestions built in, so this is configuration rather
than a sidecar: no extra image, no second database, no Paperless API token to
mint. Both endpoints are ClusterIP Services, so document text never leaves the
cluster and there is no key to rotate.
The make-or-break requirement was verified against the live endpoint before any
of this was written. Paperless's openai-like client calls chat_with_tools with
tool_required=True and error_on_no_tool_call=True — there is no free-text JSON
fallback, so the model must emit real tool_calls. It does: finish_reason
"tool_calls", a populated tool_calls array, empty content. A German utility
bill came back tagged Rechnung / Strom / Versorger.
That same probe showed the tags in German but document_types in English, which
is the documented weak spot and exactly what OUTPUT_LANGUAGE's second
localisation pass addresses. The value must be "de-de": it is matched against
Django's settings.LANGUAGES, where a bare "de" does not resolve.
llama-embed needed three fixes to serve as the RAG embedder:
- --alias, so the model is not advertised as its whole --hf-repo string.
That string would otherwise sit verbatim in this ConfigMap and break on the
next model swap.
- --pooling last. Qwen3-Embedding is trained for it; unspecified,
llama-server takes the GGUF default, which is invisible from outside. Set
now because nothing is indexed yet — changing pooling later invalidates
every stored vector.
- --batch-size/--ubatch-size 1024. The logs showed llama-server clamping both
to 512 under --embeddings, below Paperless's 1024-token chunk size. 2048 is
avoided deliberately: that is where this Vulkan stack fails with
ErrorDeviceLost.
Both network policies are scoped to the two pods and the one port on each side,
so neither grant widens if something else is placed in either namespace.
REQUEST_TIMEOUT goes to 300s. At ~11 tok/s a classification is 2-4k prompt
tokens plus a second round trip for localisation, and the 120s default is
marginal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-02 07:11:36 +02:00
23 changed files with 620 additions and 38 deletions
# Paperless calls both models for AI suggestions and RAG embeddings. The
# matching egress grant on its side is scoped to these two pods on :8080.
- fromEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": paperless
app.kubernetes.io/name:paperless
toPorts:
- ports:
- port:"8080"
protocol:TCP
- fromEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": monitoring
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
claude-botandClaude Opus 5
k.renner