The instance deployed and got nowhere. Both ExternalSecrets reported
"could not get secret data from provider", so hetzner-s3-creds never
existed, and the CNPG restore job died on it:
can't get credentials for cluster pprleval-pg: while getting secret
hetzner-s3-creds: secrets "hetzner-s3-creds" not found
The openbao ClusterSecretStore carries an explicit namespace allowlist in
spec.conditions, and paperless-eval was not on it. Nothing in the
manifests shows that — it is cluster state, which is why neither the
review nor `kubectl kustomize` caught it, and why the failure only
appeared on deploy.
The paperless container also gets production's securityContext back. The
review was right about this and it was dismissed as being about the
scanner and mail bridge sidecars, which those manifests were only cited
as evidence for the pattern. The paperless-ngx entrypoint starts as root,
chowns the mounted volumes and drops to uid 1000 via setgid/setuid, so
drop: ["ALL"] with nothing added fails that sequence with EPERM. It would
have bitten as soon as the restore succeeded and the pod actually tried
to start — and harder here than in production, whose volumes were chowned
once and kept it, while these are fresh on every rebuild.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The egress grant to llama-embed was there; the matching ingress in
ai-ns.yaml was not, so Cilium would have dropped every embedding call —
the one dependency an eval has outside its own pod. The same pairing was
done correctly for home-assistant in #759 and simply missed here.
Scoped to the namespace rather than an app label, unlike the grants
around it. That namespace exists only to run evaluations, and its
workload is jobs created per measurement whose labels vary with what is
being measured; a label match would present later as an embedding call
silently dropped.
Also from the review:
secret.reloader annotation alongside the configmap one, matching
production. The secret refreshes on the same 10m cycle, and without the
annotation a rotated key would sit unused until something else restarted
the pod.
The retention argument in cluster.yaml was too thin, and checking it made
it worse before it made it better: the plugin runs retention from the
instance sidecar on a timer
(instanceSidecarConfiguration.retentionPolicyIntervalSeconds, default
1800), not as part of a backup — so "this cluster never backs up" would
not have covered it.
What does cover it is a second property: retention runs
barman-cloud-backup-delete, which takes a server name and operates inside
<destinationPath>/<serverName>. The production store's status keys its
recovery window by "pprl-pg", confirming that layout; this cluster is
pprleval-pg, so even an accidental retention policy would address an
empty prefix.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The startupProbe was production's 10 minutes, and production sized that
budget for exactly this case — a from-scratch boot on the same ~3900
documents, per its own comment — so it would probably have held. The
difference is frequency, not size: production's volume is Retain and
keeps the index, making the expensive path a one-off, while this one is
ephemeral and rebuilds on every re-test. A killed pod restarts the work
rather than resuming, so a slow boot becomes a crash-loop that stalls
Flux behind wait: true. 60 minutes now, matching the Kustomization.
The three-hour LLM index is not in that budget and was never at risk: it
is a celery task, and boot only runs init-llmindex-migrate.
Also from the review:
The shared secret refreshes at 10m like production rather than 1h. The
point of sharing the key is that both hold the same value, which a slower
refresh would undo for an hour after a rotation.
The data PVC comment carries measured numbers instead of an unexplained
10Gi: 36 MiB search index and 125 MiB LLM index on production for the
same corpus. The margin is deliberate — phase two varies the embedding
model, and one with wider vectors than bge-m3's 1024 dimensions would
grow that index.
The ApplicationConfiguration warning now carries the query to check it
with. That row arrives in the restore and silently overrides the
ConfigMap, so an eval could measure production's settings while appearing
to measure the job's. Currently all four columns are null on production,
so the environment does apply — but that is a fact about today, not a
property of the design.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both blocking findings from the review, and both were already written
down in this repo.
kubernetes/RECOVERY.md:198-201 says the barman plugin MUST be declared
for bootstrap.recovery, with isWALArchiver: false to keep it read-only —
and notes that an earlier version of that runbook said to omit it, which
fails at instance start with "Unknown plugin:
barman-cloud.cloudnative-pg.io". externalClusters only says where the
backup is; it does not get the plugin injected into the pod. So the
read-only guarantee comes from the flag, not from leaving the plugin
out, and the previous commit would not have restored at all.
The namespace also never admitted the database namespace. The CNPG
operator dials each instance at https://<pod-ip>:8000/pg/status, and
without it the Cluster reports "Instance Status Extraction Error" while
Postgres serves normally (issue #433). Every other CNPG-backed namespace
policy here carries the rule; this one is scoped the same way, to the
pprleval-pg endpoints and that one port.
Storage moves to ceph-block-ephemeral for all five volumes while fixing
the first point, because RECOVERY.md makes the same choice for the same
reason: ceph-block is Retain, so a rebuilt-on-demand copy would leave
Released PVs and orphaned RBD images behind every time (issue #262).
The images keep their pinned tags without $imagepolicy markers, now with
a comment saying so: a measurement is comparable only against a fixed
version.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Choosing a chat model, and later an embedding model, means running the
same 50 documents through each candidate. That has been happening against
the live archive: the embedding model was swapped under it, the LLM index
rebuilt twice, the chat server broken for an afternoon, and the UI left
spinning while an eval held the only GPU slot. None of that belongs there.
The database is 90 MB and the AI path never opens a file — it reads
title, content and filename out of Postgres — so a copy costs a restore
and an empty media volume rather than a second copy of the documents.
Restores from paperless's backup store and archives nothing back. Three
omissions carry that, rather than any setting pointing somewhere safe:
no spec.plugins the cluster has no barman plugin at all, so no
WAL is archived
no retentionPolicy the source ObjectStore points at the production
prefix; with a retention policy this namespace
could expire paperless's backups
no ScheduledBackup nothing here is worth keeping
pg_basebackup would have avoided touching the store at all, but it needs
the production cluster's credentials, and those live in a Secret in the
paperless namespace that this one cannot read.
Everything that exists to serve people is left out: no ingress, no gate
outpost, no hostname, no remote-user auth, no Tika or Gotenberg, no mail
or FTP consumption. Nothing signs in and nothing is consumed, so the
instance is unreachable from outside the cluster and the documents are
not exposed a second time.
The deployment here is a smoke test — it proves the restore is usable and
that the search index builds. The evaluation runs as jobs that bring
their own paperless and llama-server, since a paperless instance can only
point at one chat endpoint and two candidates need to run at once.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both agents stopped reading on 2026-08-27 and had read nothing since:
k8s-1 at 12:15:41Z, k8s-2 at 19:14:27Z, each frozen at its line count
(35449 and 29917) while Traefik kept logging normally.
Each had hit its first kubelet log rotation after starting. The paths
under /var/log/containers are symlinks into /var/log/pods, and kubelet
rotates the target at containerLogMaxSize; the inotify tailer reopens
the path, waits for a file that is already present, and never returns.
Nothing about the result looks broken: pods Running and Ready, LAPI
answering, bouncers pulling, cumulative counters still showing large
numbers. The only symptom is that no scenario fires again, which reads
as quiet rather than blind.
So this was not a one-off. Rotation happens roughly every eight hours
here, and the agents die on the first one after every start, which
bounds local detection to a few hours at a time.
CrowdSec diagnoses this itself and names the fix, at agent startup:
File /var/log/containers/traefik-...log is a symlink, but inotify
polling is enabled. Crowdsec will not be able to detect rotation.
Consider setting poll_without_inotify to true in your configuration
Upstream carries the identical case as crowdsec#2449 (k3s + Traefik),
and the chart gained this field in helm-charts#109 to fix it. It does
not conflict with the force_inotify: true the chart hardcodes: that
governs the fsnotify watch on the directory, used to discover the log
of a replacement Traefik pod, while this governs how an already
discovered file is read. Verified by rendering the chart.
The acquisition ConfigMap is checksummed into the agent pod template,
so this rolls the DaemonSet and the stalled agents restart with it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four Paperless requests overlapped and all four died:
decode: failed to find free space in the KV cache
srv decode: Context size has been exceeded.
They wanted 6209, 6472, 4303 and 1271 tokens from a shared pool of
16384. Auto slot count opens four of them and is also what enables
--kv-unified, so they draw on one buffer rather than a quarter each --
generous for a single caller, and impossible for four.
They overlapped because a client timeout does not stop the generation
behind it. The eval abandons a request after 300 s and moves to the next
document; the abandoned one keeps consuming the budget the next one
needs, and one slow document is enough to start the pile-up. With one
slot they queue instead, which is what should happen: nothing calls this
server in parallel on purpose, and Paperless's celery worker runs at
concurrency 1.
Deliberately no liveness probe alongside this. /health answered 200 in
0.4 ms throughout, which looks like the probe failing at its job -- but
the server drained the batch and served again by itself, so a probe that
restarted the pod would have turned a self-healing incident into an
outage. With a single slot, a probe asking for a real completion would
also queue behind live work and kill healthy pods mid-document.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
#767 broke the model. Asked to classify a document and answer with JSON
only, the server returned 1025 tokens of "!" against a 650-token prompt.
That is the KV cache emitting numbers rather than language, and it takes
everything downstream with it: Paperless ran past its 300 s timeout on
every document, Home Assistant got an empty completion once and a
hallucinated list of devices that do not exist another time.
None of that looked like a backend fault from the outside. The server
starts, /props reports the requested size, /health passes, and the
OpenAI response is well-formed with finish_reason "stop". Only the
generated text gives it away, which is why three separate symptoms got
blamed on prompt size, on tokenizer behaviour and on the RAG context
before anyone read the output.
The boundary is between the two changes rather than at either: #765 left
16384 per slot and produced coherent answers; #767 raised it to 32768 and
produced this. The same Vulkan/Mesa stack already fails at --ubatch-size
2048 with ErrorDeviceLost, so a ceiling here is not a surprise -- only
that it degrades quietly instead of erroring.
--parallel goes with it. Unset, slots return to auto, which is what keeps
--kv-unified enabled ("if number of slots is auto"), so the auto slots
share one buffer and each may draw the full 16384.
Home Assistant needed 18910 tokens for one Assist request and cannot have
them at this ceiling. That path is closed, and Paperless -- which fits in
16k and never needed more -- is what this restores.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
proxy-2 has been enforcing since 14:21:54 without incident, so the role
default moves to nftables and the host_vars override goes away rather than
being duplicated for proxy-1. proxy-2's rendered config is unchanged by this,
so it will not restart; only proxy-1 does.
The staged rollout did its job and can be read off the deploy log:
proxy-1: deployed in mode "dry-run"
proxy-2: deployed in mode "nftables"
and only proxy-2 restarted, which showed up as a 1,887,882-byte startup pull
from its address alone.
The delivery path was tested in enforcing mode with a decision on 192.0.2.1
(TEST-NET-1): both hosts had the ban 3s later as a 205-byte delta, and the
revocation as a 203-byte delta 1-2s after deletion.
What is still not proven is that a packet is actually dropped, and the
measurement for that argues for finishing the rollout rather than waiting.
Requests from banned addresses reaching Traefik stopped after proxy-2 armed —
1 at 14:02, 3 at 14:07, 22 at 14:08, 1 at 14:18, none after — but 25 of those
27 arrived in a single burst beforehand, so at this volume that is consistent
with enforcement rather than evidence of it. With both hosts armed the claim
becomes falsifiable instead: no request from an address in the decision set
should reach Traefik at all, and a single one would prove otherwise.
dry-run remains available per host through inventory/host_vars, which is what
a newly added proxy should get before it enforces.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Arming the bouncer on proxy-2 (#766) merged and deployed nothing. The switch
lives in ansible/inventory/host_vars/proxy-2.renner.uno.yml, and the trigger
list covered ansible/roles/** and the play but no inventory path at all. The
change sat in git looking applied while the host kept running dry-run, and
nothing reported a problem — there was simply no pipeline run to fail.
The bouncer's LAPI key has the same shape: it lives in
group_vars/netbird_proxy/vault.yml, so a rotation would never have reached the
hosts either.
Both paths are now listed. pve_hosts.yml deliberately is not: it is the single
inventory file for the whole fleet, so listing it would redeploy the proxies
for any unrelated host edit. The cost is that a change to [netbird_proxy] group
membership needs a manual deployment run.
Merging this triggers the run itself, since .woodpecker/deploy-netbird-proxy.yml
is in its own trigger list — so the arming from #766 applies with it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`llama-server --help` documents both halves of what #765 got wrong, and
it ships in the image:
-np, --parallel N number of server slots (default: -1, -1 = auto)
-kvu, --kv-unified use single unified KV buffer shared across all
sequences (default: enabled if number of slots
is auto)
So leaving --parallel unset did not merely pick a slot count, it enabled
a shared KV buffer that every slot could draw the full --ctx-size from.
That is why this server reported n_ctx_slot 16384 across four slots, and
the startup log said so outright: kv_unified = 'true'. Naming a slot
count switches that off and splits the context.
The value stays 65536. Only the reasoning in the comment changes, from an
inference drawn off two /props readings to the documented behaviour.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
#765 claimed --ctx-size was per slot and halved the slot count to pay for
doubling it. Both halves of that were wrong. With --parallel set,
llama.cpp divides --ctx-size across the slots, so 32768 over two slots
gave 16384 each -- exactly what was there before -- and the Assist
request fails with the same message it did then:
request (18910 tokens) exceeds the available context size (16384)
The flag does behave differently without --parallel, which is where the
wrong reading came from: the four-slot server reported n_ctx_slot equal
to --ctx-size rather than a quarter of it. Adding --parallel changed
that, and the memory reading confirms it -- the pod fell from 24979 MiB
to 21601 MiB, so total KV halved rather than holding steady.
65536 across two slots is 32768 each, and puts total KV back where it was
before #765 along with the memory that came with it.
Verify rather than assume this time:
kubectl -n ai exec deploy/llama-server -- curl -sS \
http://127.0.0.1:8080/props | jq .default_generation_settings.n_ctx
must report 32768.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
First host goes from dry-run to enforcing. proxy-2 deliberately: it is the
dual-stack member, while proxy-1 is IPv6-only and the one where every
network-shaped problem has surfaced first — the dhcpcd/resolv.conf breakage,
and a GitHub download that could not work at all. host_vars keeps the change
to one host; proxy-1 stays on the role default.
Verified before committing: ansible-inventory resolves nftables for proxy-2
and nothing for proxy-1.
Adds netbird.renner.uno to capi_whitelists first, and that is not decoration.
The bouncer's chain sits on the INPUT hook, the proxy hosts run no other
firewall, and there is therefore no conntrack rule accepting established flows
ahead of it. A ban on the management address would drop the return traffic of
the proxies' own outbound control connection and take them off the mesh, with
no inbound attack involved at all. The same reasoning already covers the proxy
addresses, which were listed from the start.
Checked at the time of writing: none of these addresses appear in the current
decision set — searched all ~15k entries. That is the state today, not a
guarantee; they are rented addresses on a hoster whose ranges do end up on
community blocklists, which is the whole reason for listing them.
The delivery path was tested end to end beforehand with a decision on
192.0.2.1 (TEST-NET-1, so no real traffic could be affected). Both bouncers
received the ban within one poll cycle — a 206-byte delta against the usual
26 — and the revocation the same way, 203 bytes, ~5s after deletion. What that
did not test is whether a packet is actually dropped, which dry-run cannot
show. This commit is what makes that testable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Home Assistant conversation agent is configured and refuses every
request:
API error: request (18910 tokens) exceeds the available
context size (16384 tokens), try increasing it
18910 breaks down as ~12k for the exposed-entity overview and ~7k for
the base instructions and tool schemas, whose parameters enumerate every
exposed name, area and floor. Trimming the exposed set from 845 entities
to 666 already took the overview from 16045 tokens to 12048; the
remainder is not the sort of thing curation reaches.
--ctx-size is per slot rather than the total, which is what makes this
affordable: /props reports n_ctx_slot equal to the flag while running
four slots, where a total would have divided down to 4096. Doubling the
flag alone would double the KV cache, so the slot count halves in step
and the allocation lands where it already is -- 24979 MiB against the
32Gi limit, most of it the 19.5 GiB of weights.
Two slots matches the actual callers: one holds Home Assistant's prompt,
one holds Paperless's, and prompt caching keeps each warm. A cold prefill
of this prompt costs about two minutes, so a third caller queueing is a
better outcome than it evicting one of them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A dashboard first needed something to draw. Nothing scraped CrowdSec:
`up{namespace="crowdsec"}` returned zero series and Prometheus held no cs_*
metric at all, so the detection half of the ingress path was invisible —
including whether the agents were reading logs and whether any bouncer was
pulling decisions.
Both chart services already expose a `metrics` port on 6060 and the
crowdsec-ns CiliumNetworkPolicy already admits the monitoring namespace
without a port restriction. Only the ServiceMonitor was missing. One object
covers both services via matchExpressions, since they differ only in `app`.
Every panel query was written against label sets read off the live endpoints
rather than from the documentation:
cs_active_decisions{action,origin,reason}
cs_lapi_bouncer_requests_total{bouncer,method,route}
cs_bucket_overflowed_total{name}
cs_filesource_hits_total{acquis_type,datasource_type,source}
cs_node_wl_hits_ok_total{...,reason,...}
cs_parser_hits_ok_total / _ko_total
The bouncer panel is the one worth having: the LAPI counts requests per
bouncer, so a bouncer that stops pulling shows up as a line going to zero
rather than as nothing at all. It also makes the `<name>@<ip>` behaviour
visible — the LAPI registers a new entry per observed client address, and what
it observes is the netbird-ingress pod IP, so one logical bouncer appears
several times and gains new names as those pods roll.
The last panel is a text panel naming what the dashboard cannot show. The
bouncers' own metrics — fw_bouncer_active_decisions, dropped packets, their
view of the LAPI — sit on 127.0.0.1:60601 on public hosts with no metrics path
of any kind. So "Bouncer-Abholrate" proves a bouncer is asking, not that it
applied the answer. Better to say that on the dashboard than to let someone
read enforcement into a panel that does not measure it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The LLM index rebuild ran for 79 minutes and then died:
input (1069 tokens) is too large to process.
increase the physical batch size (current batch size: 1024)
Setting the batch to Paperless's chunk size looked like an exact match
and was not. Paperless counts its 1024 tokens with llama_index's
tokenizer; this server measures the batch in bge-m3's, which is
XLM-RoBERTa and splits German into more pieces. A chunk Paperless
considered full arrived here 4.4% over the limit, and the request failed
rather than truncating.
1536 absorbs a tokenizer ratio rather than a rounding error. 2048 stays
out of reach: that is where the chat model hit ErrorDeviceLost on this
Vulkan stack, under a 20 GB model's memory pressure that a 600 MB one may
well not reproduce — but there is no reason to go looking.
The index is left partial by the failed run and has to be rebuilt with
`document_llmindex rebuild` once this is live.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both bouncers reach the LAPI and pull every 10s, and every pull gets a 400:
2a03:4000:28:f6e:...:9a29 "GET /v1/decisions/stream?..." 400 48
2a03:4000:4b:c1e:...:8f85 "GET /v1/decisions/stream?..." 400 48
The body says what it is: "Client sent an HTTP request to an HTTPS server."
Traefik was talking plaintext to a backend that serves TLS only.
serversscheme and serverstransport were on the Ingress. Under the Kubernetes
Ingress provider `traefik.ingress.kubernetes.io/service.*` are SERVICE
annotations; on an Ingress they are silently ignored. Nothing warns, and the
failure surfaces as a 400 that reads like a malformed request rather than a
scheme mismatch. They move to the chart's Service via lapi.service.annotations.
With tls.enabled the LAPI serves HTTPS on 8080 and offers no plaintext port,
so https is mandatory here rather than hardening.
Also switches the ServersTransport from rootCAsSecrets to rootCAs, which is
what Traefik has been warning about on every reconcile:
RootCAsSecrets option is deprecated, please use the RootCA option instead.
Two of my own diagnostics were wrong along the way and both were measurement
artifacts. "Traefik sees no requests" came from a Loki query whose limit was
consumed by those deprecation warnings. And the source address that justified
dropping the IPAllowList was measured on the watchdog, which curls 127.0.0.1;
the bouncer resolves the public name and arrives on the proxy's public
address, as the log lines above show.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
HA 2026.8.3 ships a native `llama_cpp` integration, so an Assist
conversation agent backed by the cluster's own model needs no custom
component — only a path from the HA pod to llama-server, which two
default-deny policies currently block in both directions. Verified from
inside the pod: the call to llama-server.ai.svc.cluster.local:8080 times
out today.
Both grants are scoped to the single pod and port, mirroring the
paperless grant next to them, so neither widens if something else is
placed in the other namespace. llama-embed is deliberately excluded: the
conversation agent uses chat completions only.
Tool calling was verified against the running model before adding this —
the Assist API requires it, and llama.cpp only exposes it for some
builds. b10731 returns finish_reason=tool_calls and parses German
commands into the right arguments without --jinja.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The download moved to the controller in the previous fix and now works. The
extraction that followed it does not:
Failed to find handler for "/tmp/crowdsec-firewall-bouncer-v0.0.36.tgz".
Command "/bin/tar" detected as tar type None. GNU tar required.
The ci-ansible image ships a tar that ansible's unarchive rejects. Both host
distributions have GNU tar, and only the download ever needed to move — the
v6-only problem was reaching github.com, not opening a tarball.
So the controller now ships the archive rather than the unpacked binary, and
the host extracts it. That also keeps the artefact the checksum was verified
against intact all the way to its destination, instead of verifying an archive
and then copying something extracted from it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
First candidate to be carried through to an end-to-end measurement. The
embedding comparison on 400 real archive documents puts bge-m3 level with the
deployed Qwen3-Embedding-0.6B where it matters and ahead everywhere else:
correspondent 74.0% vs 75.0% (ceiling 82.5%, so 90% vs 91% of achievable)
document type 70.0% vs 68.2%
time 147s vs 380s for 400 documents
peak memory 1.6 GiB vs 10.6 GiB
Same 1024 dimensions, so nothing downstream changes shape.
Note the pooling type differs — bge-m3 wants CLS where the Qwen family wants
last-token. Read from the model's own 1_Pooling/config.json rather than assumed;
across the candidates measured, three different pooling types were required, and
a wrong one produces plausible but untrained vectors with no error anywhere.
The memory limit is now measured rather than estimated. It also supersedes #745,
which raised the limit for the model being replaced: the footprint is dominated
by four slots at 4096 context plus the Vulkan working set, not by the weights —
which is why several candidates were OOMKilled at limits that looked generous
against their model size.
This is not a final answer. Every candidate has to go through the end-to-end
test (scripts/paperless-eval.sh) before the embedding choice means anything,
because nothing has yet verified that the retrieval metric predicts what
Paperless actually suggests. bge-m3 is simply the one to start with.
Changing paperless-config is also what lets that test run at all: the AI
settings from #739 have never reached the running pod, which predates the
Reloader annotation from #741. This edit gives Reloader something real to react
to, rather than forcing a restart by hand or inventing a change to trigger one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The deploy failed on proxy-1:
Request failed: <urlopen error [Errno 101] Network is unreachable>
url: https://github.com/.../crowdsec-firewall-bouncer-linux-amd64.tgz
proxy-1 is IPv6-only, and neither github.com nor
objects.githubusercontent.com publishes an AAAA record. The host cannot reach
GitHub at all, so no retry or timeout would have helped.
Two alternatives do not work either. Mirroring into gitea.renner.uno is out:
its public records are the internal LB VIP (10.2.99.86 / fd99::86), so it is
unreachable from outside the LAN. And there is no upstream container image for
this bouncer, which is expected of something that manipulates host netfilter —
crowdsecurity publishes cloudflare, fastly, custom and others, but no firewall
bouncer.
So the controller fetches and pushes instead of the host pulling. The CI runner
has v4. The checksum is still verified, now before anything reaches a host,
which is the better place for it anyway.
The repo had told me twice that proxy-1 is the v6-only member — in
unifi_nat_hairpin.tf and in the netbird-proxy role's dhcpcd comment — and I
wrote it into the PR description myself. I never asked whether a v6-only host
can reach GitHub.
Nothing was deployed before this: the play runs serial: 1, proxy-1 went first
and failed, so proxy-2 never started.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Found by the pr-review bot on #751, and it is the exact failure this design
was written to avoid.
BOUNCER_KEY_edge is consumed as an env var from a secretKeyRef, and env is
fixed at pod start. The Reloader annotation listed only crowdsec-bouncer-key,
so a rotation of the edge key would update OpenBao, sync into the Secret, and
reach the hosts through the Ansible play — while the LAPI kept the old value
in its environment and went on trusting the old key. The bouncers would then
present the new one and get 401s, at exactly the moment the rotation was
supposed to be seamless.
The comment directly above that annotation already stated the rule — "it's an
env-var secret, so a restart is required to pick up a rotated key". I added a
second env-var secret and did not extend the list.
Comma-separated, matching shelly-manager-deployment.yaml.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Completes the pair: the ExternalSecret feeds BOUNCER_KEY_edge to the LAPI, this
feeds the same value to the bouncers that authenticate against it.
A single encrypted value rather than a whole-file vault like its siblings in
group_vars/all and group_vars/edge_host. Appending to one of those means
decrypting it first, which exposes every other secret in that file to add one
line. encrypt_string keeps the blast radius at this value, and the plaintext
never reaches a file: it goes from `bao kv get -field=key` through a pipe
straight into the encryption.
Scoped to netbird_proxy rather than all, because only those hosts run a
bouncer.
Not enrolled in the quarterly rotator, unlike crowdsec-bouncer-key. Nothing on
these hosts re-reads OpenBao on a schedule, so a rotation there would leave
them 401ing against the LAPI until someone ran the playbook. Rotating this one
means updating OpenBao and re-running plays/install-netbird-proxy.yml, in that
order.
Verified with `ansible-inventory --list`: the inventory parses and the value
decrypts.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Enforces the decisions the in-cluster CrowdSec already detects, at the point
where public traffic actually enters. These hosts terminate TLS for every
*.renner.uno name, so a drop here never reaches the mesh, the relay or the
cluster. Detection stays where it is: the reverse-proxy produces no parseable
access log and has no collection, while Traefik has a working one, and every
scenario in use needs HTTP detail that does not exist at this layer.
Deployed in dry-run. It pulls decisions and logs what it would block without
creating a single nftables rule. The first bouncer to arm anywhere enforces
~24k CAPI decisions at once on the only external entry path, so the
false-positive window has to come before the enforcement rather than after it.
Arming is flipping crowdsec_bouncer_mode per host.
The static release tarball, not a package. The AUR package
crowdsec-firewall-bouncer-nftables depends on `crowdsec` and would pull the
whole Security Engine onto hosts that need only the bouncer; the group is
mixed (proxy-1 Arch, VPS-1 Debian); and the AUR would be a pattern this repo
does not otherwise have. The tarball is the same artefact the AUR package
downloads, pinned by version and sha256, and only the binary is unpacked —
the install/upgrade/uninstall scripts beside it manage systemd and config
themselves, which would leave two owners for the same files.
retry_initial_connect is set, and it is not cosmetic. Without it the process
exits when the LAPI is unreachable at startup, and since a clean shutdown
deletes the nftables tables, the systemd restart loop would leave the host
with no blocklist at all until the first successful connect. The flag exists
in the code but not in the shipped example config.
nftables_hooks covers input and forward. input is the traffic this exists for,
because the reverse-proxy is host-networked and owns :443. forward covers the
bridge-networked http-redirect sidecar, whose packets never traverse INPUT —
the same Docker bridge that showed up as 172.18.0.3 in the Traefik access log.
Prometheus is enabled but bound to loopback. These hosts are public and have
no metrics collection path today; opening :60601 to the internet to gain a
scrape target is the wrong trade.
No CA file: Traefik serves Let's Encrypt for crowdsec.renner.uno, and
ca_cert_path is added to the system pool rather than replacing it.
Rollback is `systemctl stop` — a clean stop deletes both tables by itself.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The allowlist carried the published proxy addresses on the assumption that a
request from a proxy host arrives carrying that host's public address. It does
not.
The client connects to the local proxy on 127.0.0.1, the reverse-proxy writes
what it saw into X-Forwarded-For, and Traefik resolves that into ClientHost.
Measured against netbird-proxy-watchdog, which makes exactly this kind of
request every round:
172.18.0.3 - - [02/Sep/2026:10:12:01] "GET /" 200 websecure-immich-...
172.18.0.3 - - [02/Sep/2026:10:12:01] "GET /" 200 websecure-home-...
Two probes one second apart, one per watchdog target. 172.18.0.3 is a Docker
bridge address on the proxy side.
So the allowlist would have 403ed the bouncer. Allowlisting 172.18.0.0/16
instead is not a control: it is a private range on a host whose addressing we
do not own, and it would admit anything else that happens to originate there.
The bouncer key stays as the access control, which is what it always was. The
endpoint serves read-only decision streams to a valid key; the admin surface
is cscli, in-cluster. If the source address ever becomes deterministic, an
ipAllowList can go back on top.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Groundwork for enforcing CrowdSec decisions on proxy-1/proxy-2. Inert on its
own: an Ingress nobody calls yet, a bouncer key nobody uses yet, and a
whitelist that only ever narrows what may be banned.
The proxies cannot reach the LAPI over the mesh. The netbird-proxy container
runs non-root — that is what the ip_unprivileged_port_start task in the role
is for — so it has no NET_ADMIN and no /dev/net/tun, and its embedded NetBird
client terminates the tunnel in userspace. The tunnel belongs to that process,
not to the host netns, so a second process on the same host cannot route
through it. The alternative was a host-level NetBird client plus the LAPI on a
LoadBalancer VIP: two more peers, an ACL policy and a CNP rule. This reuses
the path every other service already takes.
Exposure is narrowed twice. The IPAllowList carries the three proxy addresses,
resolving them past the netbird-ingress pod IP through X-Forwarded-For the
same way the rate-limit middleware does — without that it would compare
against the shared forwarder IP, which is the gap behind the 2026-06-27 mass
ban. Behind it, the bouncer key. The LAPI serves read-only decision streams to
a valid key; the admin surface is cscli, in-cluster.
No CNP change is needed: crowdsec-ns.yaml already admits the traefik pods, and
ingress-default-deny carries no egress section.
BOUNCER_KEY_edge is separate from BOUNCER_KEY_ingress on purpose. The latter
rotates quarterly and both its consumers are in-cluster, so ESO propagates
within 10m. The edge hosts are Ansible-managed and read OpenBao on no
schedule, so a rotation would leave them 401ing until someone runs the
playbook. This key rotates with the playbook that carries it.
capi_whitelists.yaml is the only declarative allowlist the chart offers.
`cscli allowlists` objects live in the LAPI database, which runs without
persistence here deliberately, so anything created that way is gone at the
next restart. This file is a ConfigMap mount and survives by construction.
Upstream deprecates it; it is still the only thing that works against a LAPI
with no PVC.
It lists the proxy addresses and not the home WAN address, which is not
needed: unifi_nat_hairpin.tf DNATs the public proxy addresses to the Traefik
VIP on the UDM, so LAN clients reach the cluster directly with a LAN source
and never traverse the proxies.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The embedding model was picked from a recommendation and never compared. MTEB's
German retrieval results put the deployed Qwen3-Embedding-0.6B well down the
field, so a dozen candidates are being measured against the real archive with
scripts/embed-eval-real.sh — and they will not fit in 40Gi alongside the two
deployed models.
Growing the shared cache rather than using a scratch volume is deliberate: on a
slow connection the download is the long pole, and a shared cache lets the large
candidates pull in the background while the small ones are already being
measured. It also means nothing is fetched twice across runs, and a candidate
that wins is already local when it is promoted.
120Gi against 309 GiB currently free in cephfs-data0. CephFS has
allowVolumeExpansion, so this grows online without disturbing the running pods.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Immich's in-app migration to VectorChord is complete: both vector indexes,
face_index and clip_index, run on the vchordrq access method, and nothing uses
pgvecto.rs. The manifest still described a migration window.
`DROP EXTENSION vectors` was run against the immich database before this. It
was safe to do without CASCADE — nothing outside the extension depended on it,
which a rolled-back `BEGIN; DROP EXTENSION vectors; ROLLBACK;` confirmed
first. The embedding columns are `public.vector(512)`, which belongs to
pgvector, not to pgvecto.rs's type in the `vectors` schema. Afterwards both
vchordrq indexes are intact at 96 MB and 79 MB and the cluster stays healthy.
With the extension gone, vectors.so has nothing to serve and preloading it
only recreates pg_vectors in PGDATA. pgvector needs no preloading, so vchord.so
is the whole list.
instances goes back to 2. The blocker was never the extension as such — it was
that pg_basebackup's tar format cannot represent the long
pg_vectors/indexes/<uuid> paths pgvecto.rs creates, so a replica could not
join. That directory is now empty, 20 KiB in total, so the constraint is gone.
The image keeps its `:16-migration` tag. That is the upstream build's own name
for a multi-extension image, not a statement about this cluster, and it still
provides the two extensions in use. Moving to a different image is a separate
change with its own risk.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both documents described work as outstanding that the cluster shows finished.
A record that says "REMAINING" long after the fact is worse than no record:
it is read as current state, and the next person plans around it.
## Home Assistant recorder
Phase 2c is done, and done correctly. Against the live database:
statistics 11,092,586 2024-03-13 20:00 -> current
statistics_short_term 1,883,709 rolling ~7 days
statistics_meta 1,480
The 2024-03-13 origin is the part that matters. It is unchanged from the
10,793,712-row count recorded on 2026-08-19, so the history was copied across
engines rather than started from zero — the exact failure the document was
written to prevent. `states` carries current rows, so db_url points at
Postgres.
The three "INERT"/"PHASE 1" comments went with it. All three described a
cluster that nothing consumed; it is now the recorder's live database, and the
render-db-url init container is load-bearing rather than inert — HA will not
start without the secret it renders.
## Bare-metal conversion
`talosctl get systeminformation` reports Minisforum MS-01 hardware with
distinct chassis serials for all three nodes and no hypervisor, so saturn and
neptune are not outstanding. Node creationTimestamp is not evidence either way:
k8s-3 still carries its 2026-05-09 object and is on metal regardless.
The runbook stays as the procedure for a rebuild.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-02 08:37:19 +02:00
39 changed files with 1622 additions and 57 deletions
"description":"Aktive Ban-Entscheidungen in der LAPI, ganz überwiegend aus den CAPI-Community-Blocklisten. Das ist die Menge, die ein scharfgeschalteter Bouncer durchsetzt.",
"description":"Bouncer, die in den letzten 10 Minuten Decisions abgeholt haben. Fällt das unter die erwartete Zahl, holt ein Bouncer nichts mehr — und setzt damit einen veralteten Stand durch, ohne dass irgendetwas ausfällt.",
"description":"Vom Agent gelesene Logzeilen pro Sekunde. Steht das auf 0, detektiert CrowdSec nichts mehr — der häufigste stille Ausfall, weil alles andere weiterläuft.",
"description":"Abholrate je Bouncer. Der Firewall-Bouncer auf den Proxies fragt alle 10 s. Die LAPI legt bei jeder neuen beobachteten Client-Adresse einen Eintrag `<name>@<ip>` an, und beobachtet wird die Pod-IP des netbird-ingress-Peers — deshalb erscheint ein logischer Bouncer hier mehrfach und nach Pod-Rollen unter neuen Namen.",
"targets":[{"datasource":{"type":"prometheus","uid":"prometheus"},"expr":"sum by (bouncer) (rate(cs_lapi_bouncer_requests_total[5m]))","legendFormat":"{{bouncer}}","refId":"A"}],
"targets":[{"datasource":{"type":"prometheus","uid":"prometheus"},"expr":"sum by (reason) (cs_active_decisions)","legendFormat":"{{reason}}","refId":"A"}],
"targets":[{"datasource":{"type":"prometheus","uid":"prometheus"},"expr":"sum by (name) (increase(cs_bucket_overflowed_total[1h]))","legendFormat":"{{name}}","refId":"A"}],
"description":"Geparste gegen nicht geparste Zeilen. Ein Anstieg bei `ko` heißt meist, dass sich ein Logformat geändert hat und der Parser nicht mehr greift — CrowdSec liest dann weiter, erkennt aber nichts mehr.",
"targets":[{"datasource":{"type":"prometheus","uid":"prometheus"},"expr":"sum by (reason) (rate(cs_node_wl_hits_ok_total[5m]))","legendFormat":"{{reason}}","refId":"A"}],
"content":"## Was dieses Dashboard NICHT zeigt\n\nAlle Panels stammen aus der LAPI und den Agents im Cluster. Die **Bouncer auf proxy-1 und proxy-2 exportieren eigene Metriken**, die hier fehlen:\n\n- `fw_bouncer_active_decisions` — was tatsächlich in den nftables-Sets steht\n- `fw_bouncer_dropped_packets` / `_bytes` — was wirklich verworfen wurde\n- `lapi_requests_total` / `_failures_total` — aus Sicht des Bouncers\n\nSie liegen auf `127.0.0.1:60601` der Proxy-Hosts. Die Hosts sind öffentlich und haben bisher keinerlei Metrik-Pfad; den Port ins Internet zu öffnen, nur um ein Scrape-Ziel zu bekommen, wäre der falsche Tausch.\n\n**Konsequenz:** „Bouncer-Abholrate\" oben zeigt, dass ein Bouncer *fragt* — nicht, dass er die Antwort auch angewandt hat. Solange der Modus `dry-run` ist, wendet er ohnehin nichts an.\n\nEin Scrape-Pfad für die Hosts ist eigene Arbeit und Voraussetzung dafür, dass „was wird wirklich geblockt\" beantwortbar wird."
- paperless-eval# same apps/paperless key plus the shared barman S3 creds
- trmnl
- velero
- woodpecker# wp-pg barman creds; woodpecker moving out of `apps` (#433)
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
claude-botandClaude Opus 5
k.renner
security-bot