Compare commits

..
Author SHA1 Message Date
claude-botandClaude Opus 5 0a101eb8ee fix(llama-embed): size the batch for bge-m3's tokenizer, not Paperless's
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The LLM index rebuild ran for 79 minutes and then died:

    input (1069 tokens) is too large to process.
    increase the physical batch size (current batch size: 1024)

Setting the batch to Paperless's chunk size looked like an exact match
and was not. Paperless counts its 1024 tokens with llama_index's
tokenizer; this server measures the batch in bge-m3's, which is
XLM-RoBERTa and splits German into more pieces. A chunk Paperless
considered full arrived here 4.4% over the limit, and the request failed
rather than truncating.

1536 absorbs a tokenizer ratio rather than a rounding error. 2048 stays
out of reach: that is where the chat model hit ErrorDeviceLost on this
Vulkan stack, under a 20 GB model's memory pressure that a 600 MB one may
well not reproduce — but there is no reason to go looking.

The index is left partial by the failed run and has to be rebuilt with
`document_llmindex rebuild` once this is live.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-02 15:13:54 +02:00
7 changed files with 23 additions and 237 deletions
+19 -8
View File
@@ -64,16 +64,27 @@ spec:
- "99"
- --threads
- "2"
# Paperless chunks documents at 1024 tokens. The defaults are
# batch 2048 / ubatch 512, and with --embeddings llama-server
# clamps both down to 512 rather than run them mismatched — below
# the chunk size. Matching both to 1024 fits a whole chunk in one
# pass; 2048 is deliberately avoided, since that is where this
# Vulkan stack fails with ErrorDeviceLost.
# Paperless chunks at 1024 tokens, but counts them with its own
# tokenizer while this server measures the batch in bge-m3's.
# Those disagree: XLM-RoBERTa splits German into more pieces than
# llama_index's counter does, so a chunk Paperless considers full
# at 1024 arrived here as 1069 and the request failed outright —
# "input (1069 tokens) is too large to process".
#
# The headroom therefore has to absorb a tokenizer ratio, not just
# a rounding error, and 1.5x covers far more than the 1.044 that
# broke it. The defaults are batch 2048 / ubatch 512, and with
# --embeddings llama-server clamps both to the smaller rather than
# run them mismatched, so both are set.
#
# 2048 stays avoided: that is where the chat model met
# ErrorDeviceLost on this Vulkan stack. That failure came under a
# 20 GB model's memory pressure and may not bind a 600 MB one, but
# this does not need to find out.
- --batch-size
- "1024"
- "1536"
- --ubatch-size
- "1024"
- "1536"
# Room for a full chunk plus overhead rather than exactly one.
- --ctx-size
- "4096"
@@ -1,36 +0,0 @@
# Scrapes CrowdSec. Until this landed, nothing did: `up{namespace="crowdsec"}`
# returned zero series and Prometheus held no cs_* metric at all, so the
# detection side of the ingress path was invisible — including whether the
# agents were reading logs and whether any bouncer was pulling decisions.
#
# One monitor for both services. They carry the same shape (a `metrics` port on
# 6060) and differ only in `app`, so a matchExpressions selector keeps this to
# a single object rather than two near-identical ones:
#
# crowdsec-service the LAPI — decisions, bouncer + machine requests
# crowdsec-agent-service the agents — acquisition, parsers, scenarios
#
# The crowdsec-ns CiliumNetworkPolicy already admits the monitoring namespace
# without a port restriction, so no policy change is needed.
#
# `release: kube-prometheus-stack` is what the chart's Prometheus selects on.
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: crowdsec
namespace: crowdsec
labels:
app.kubernetes.io/name: crowdsec
release: kube-prometheus-stack
spec:
selector:
matchExpressions:
- key: app
operator: In
values:
- crowdsec-service
- crowdsec-agent-service
endpoints:
- port: metrics
path: /metrics
interval: 30s
@@ -23,7 +23,6 @@ resources:
- prometheusrule-cnpg-backup.yaml
- alloy-servicemonitor.yaml
- cnpg-podmonitors.yaml
- crowdsec-servicemonitor.yaml
- home-assistant-servicemonitor.yaml
- immich-servicemonitor.yaml
- loki-servicemonitor.yaml
@@ -1,163 +0,0 @@
{
"annotations": {"list": []},
"description": "CrowdSec: Erkennung im Cluster, Durchsetzung am NetBird-Edge",
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 1,
"id": null,
"links": [],
"panels": [
{
"datasource": {"type": "prometheus", "uid": "prometheus"},
"description": "Aktive Ban-Entscheidungen in der LAPI, ganz überwiegend aus den CAPI-Community-Blocklisten. Das ist die Menge, die ein scharfgeschalteter Bouncer durchsetzt.",
"fieldConfig": {
"defaults": {"color": {"mode": "thresholds"}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "text", "value": null}]}, "unit": "short"},
"overrides": []
},
"gridPos": {"h": 4, "w": 6, "x": 0, "y": 0},
"id": 1,
"options": {"colorMode": "value", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
"targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(cs_active_decisions)", "refId": "A"}],
"title": "Aktive Decisions",
"type": "stat"
},
{
"datasource": {"type": "prometheus", "uid": "prometheus"},
"description": "Bouncer, die in den letzten 10 Minuten Decisions abgeholt haben. Fällt das unter die erwartete Zahl, holt ein Bouncer nichts mehr — und setzt damit einen veralteten Stand durch, ohne dass irgendetwas ausfällt.",
"fieldConfig": {
"defaults": {"color": {"mode": "thresholds"}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "red", "value": null}, {"color": "green", "value": 1}]}, "unit": "short"},
"overrides": []
},
"gridPos": {"h": 4, "w": 6, "x": 6, "y": 0},
"id": 2,
"options": {"colorMode": "value", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
"targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "count(sum by (bouncer) (rate(cs_lapi_bouncer_requests_total[10m])) > 0)", "refId": "A"}],
"title": "Bouncer, die abholen",
"type": "stat"
},
{
"datasource": {"type": "prometheus", "uid": "prometheus"},
"description": "Vom Agent gelesene Logzeilen pro Sekunde. Steht das auf 0, detektiert CrowdSec nichts mehr — der häufigste stille Ausfall, weil alles andere weiterläuft.",
"fieldConfig": {
"defaults": {"color": {"mode": "thresholds"}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "red", "value": null}, {"color": "green", "value": 0.01}]}, "unit": "reqps"},
"overrides": []
},
"gridPos": {"h": 4, "w": 6, "x": 12, "y": 0},
"id": 3,
"options": {"colorMode": "value", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
"targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(rate(cs_filesource_hits_total[5m]))", "refId": "A"}],
"title": "Gelesene Logzeilen/s",
"type": "stat"
},
{
"datasource": {"type": "prometheus", "uid": "prometheus"},
"description": "Szenario-Overflows der letzten 24 h — also lokal erkannte Angriffe, im Gegensatz zu den von CAPI übernommenen.",
"fieldConfig": {
"defaults": {"color": {"mode": "thresholds"}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "text", "value": null}]}, "unit": "short"},
"overrides": []
},
"gridPos": {"h": 4, "w": 6, "x": 18, "y": 0},
"id": 4,
"options": {"colorMode": "value", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
"targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(increase(cs_bucket_overflowed_total[24h]))", "refId": "A"}],
"title": "Lokale Overflows (24 h)",
"type": "stat"
},
{
"datasource": {"type": "prometheus", "uid": "prometheus"},
"description": "Abholrate je Bouncer. Der Firewall-Bouncer auf den Proxies fragt alle 10 s. Die LAPI legt bei jeder neuen beobachteten Client-Adresse einen Eintrag `<name>@<ip>` an, und beobachtet wird die Pod-IP des netbird-ingress-Peers — deshalb erscheint ein logischer Bouncer hier mehrfach und nach Pod-Rollen unter neuen Namen.",
"fieldConfig": {
"defaults": {"color": {"mode": "palette-classic"}, "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]}, "unit": "reqps"},
"overrides": []
},
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 4},
"id": 5,
"options": {"legend": {"calcs": ["lastNotNull"], "displayMode": "table", "placement": "right", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
"targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum by (bouncer) (rate(cs_lapi_bouncer_requests_total[5m]))", "legendFormat": "{{bouncer}}", "refId": "A"}],
"title": "Bouncer-Abholrate",
"type": "timeseries"
},
{
"datasource": {"type": "prometheus", "uid": "prometheus"},
"description": "Aktive Decisions nach Grund. `http:scan` dominiert und stammt aus den CAPI-Blocklisten.",
"fieldConfig": {
"defaults": {"color": {"mode": "palette-classic"}, "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 20, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "normal"}, "thresholdsStyle": {"mode": "off"}}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]}, "unit": "short"},
"overrides": []
},
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 4},
"id": 6,
"options": {"legend": {"calcs": ["lastNotNull"], "displayMode": "table", "placement": "right", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
"targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum by (reason) (cs_active_decisions)", "legendFormat": "{{reason}}", "refId": "A"}],
"title": "Aktive Decisions nach Grund",
"type": "timeseries"
},
{
"datasource": {"type": "prometheus", "uid": "prometheus"},
"description": "Welche Szenarien lokal auslösen. Das ist das, was CrowdSec selbst aus den Traefik-Logs erkennt.",
"fieldConfig": {
"defaults": {"color": {"mode": "palette-classic"}, "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "bars", "fillOpacity": 70, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "normal"}, "thresholdsStyle": {"mode": "off"}}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]}, "unit": "short"},
"overrides": []
},
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 12},
"id": 7,
"options": {"legend": {"calcs": ["sum"], "displayMode": "table", "placement": "right", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
"targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum by (name) (increase(cs_bucket_overflowed_total[1h]))", "legendFormat": "{{name}}", "refId": "A"}],
"title": "Szenario-Overflows (je Stunde)",
"type": "timeseries"
},
{
"datasource": {"type": "prometheus", "uid": "prometheus"},
"description": "Geparste gegen nicht geparste Zeilen. Ein Anstieg bei `ko` heißt meist, dass sich ein Logformat geändert hat und der Parser nicht mehr greift — CrowdSec liest dann weiter, erkennt aber nichts mehr.",
"fieldConfig": {
"defaults": {"color": {"mode": "palette-classic"}, "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]}, "unit": "reqps"},
"overrides": []
},
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 12},
"id": 8,
"options": {"legend": {"calcs": ["lastNotNull"], "displayMode": "list", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
"targets": [
{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(rate(cs_parser_hits_ok_total[5m]))", "legendFormat": "geparst", "refId": "A"},
{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(rate(cs_parser_hits_ko_total[5m]))", "legendFormat": "nicht geparst", "refId": "B"}
],
"title": "Parser",
"type": "timeseries"
},
{
"datasource": {"type": "prometheus", "uid": "prometheus"},
"description": "Whitelist-Treffer nach Grund. Nützlich beim Scharfschalten: was hier zählt, wäre sonst ein Kandidat für einen Ban gewesen.",
"fieldConfig": {
"defaults": {"color": {"mode": "palette-classic"}, "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]}, "unit": "reqps"},
"overrides": []
},
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 20},
"id": 9,
"options": {"legend": {"calcs": ["lastNotNull"], "displayMode": "table", "placement": "right", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
"targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum by (reason) (rate(cs_node_wl_hits_ok_total[5m]))", "legendFormat": "{{reason}}", "refId": "A"}],
"title": "Whitelist-Treffer",
"type": "timeseries"
},
{
"datasource": {"type": "prometheus", "uid": "prometheus"},
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 20},
"id": 10,
"options": {
"code": {"language": "plaintext", "showLineNumbers": false, "showMiniMap": false},
"content": "## Was dieses Dashboard NICHT zeigt\n\nAlle Panels stammen aus der LAPI und den Agents im Cluster. Die **Bouncer auf proxy-1 und proxy-2 exportieren eigene Metriken**, die hier fehlen:\n\n- `fw_bouncer_active_decisions` — was tatsächlich in den nftables-Sets steht\n- `fw_bouncer_dropped_packets` / `_bytes` — was wirklich verworfen wurde\n- `lapi_requests_total` / `_failures_total` — aus Sicht des Bouncers\n\nSie liegen auf `127.0.0.1:60601` der Proxy-Hosts. Die Hosts sind öffentlich und haben bisher keinerlei Metrik-Pfad; den Port ins Internet zu öffnen, nur um ein Scrape-Ziel zu bekommen, wäre der falsche Tausch.\n\n**Konsequenz:** „Bouncer-Abholrate\" oben zeigt, dass ein Bouncer *fragt* — nicht, dass er die Antwort auch angewandt hat. Solange der Modus `dry-run` ist, wendet er ohnehin nichts an.\n\nEin Scrape-Pfad für die Hosts ist eigene Arbeit und Voraussetzung dafür, dass „was wird wirklich geblockt\" beantwortbar wird."
},
"title": "Blinder Fleck",
"type": "text"
}
],
"refresh": "30s",
"schemaVersion": 38,
"style": "dark",
"tags": ["homelab", "crowdsec", "security"],
"templating": {"list": []},
"time": {"from": "now-6h", "to": "now"},
"timepicker": {},
"timezone": "",
"title": "CrowdSec Overview",
"uid": "crowdsec-overview",
"version": 1,
"weekStart": ""
}
@@ -38,9 +38,6 @@ configMapGenerator:
- name: dashboard-blackbox-overview
files:
- blackbox-overview.json
- name: dashboard-crowdsec-overview
files:
- crowdsec-overview.json
- name: dashboard-disk-health
files:
- disk-health.json
@@ -108,24 +108,6 @@ spec:
lapi:
replicas: 1
service:
# Backend wiring for the public LAPI Ingress (lapi-ingress.yaml).
#
# These belong on the SERVICE, not on the Ingress: under the Kubernetes
# Ingress provider `traefik.ingress.kubernetes.io/service.*` are service
# annotations, and on an Ingress they are silently ignored. The symptom
# is Traefik speaking plaintext to a TLS-only backend, which the LAPI
# answers with "Client sent an HTTP request to an HTTPS server." and a
# 400 — an error that looks like a malformed request rather than a
# scheme mismatch.
#
# With tls.enabled the LAPI serves HTTPS on 8080 and offers no
# plaintext port, so https here is mandatory rather than a hardening
# choice. The ServersTransport carries the chart's own CA, since that
# certificate is self-signed.
annotations:
traefik.ingress.kubernetes.io/service.serversscheme: https
traefik.ingress.kubernetes.io/service.serverstransport: crowdsec-crowdsec-lapi@kubernetescrd
# Far below the chart default, which reserves half a core per LAPI for a
# workload that draws single-digit millicores — enough to leave the
# cluster unschedulable with one node out. Only the reservation shrinks;
@@ -44,8 +44,8 @@ metadata:
namespace: crowdsec
spec:
serverName: crowdsec-service.crowdsec.svc.cluster.local
rootCAs:
- secret: crowdsec-root-secret
rootCAsSecrets:
- crowdsec-root-secret
---
apiVersion: networking.k8s.io/v1
kind: Ingress
@@ -60,12 +60,8 @@ metadata:
cert-manager.io/cluster-issuer: letsencrypt-cluster
# NOT behind the Authentik gate: the client is a daemon with an API key,
# not a browser that can complete an OIDC flow.
#
# The backend scheme and ServersTransport are NOT set here. Under the
# Kubernetes Ingress provider, `traefik.ingress.kubernetes.io/service.*`
# are SERVICE annotations — on an Ingress they are silently ignored, and
# Traefik then talks plaintext to a TLS-only backend. They live on the
# chart's Service instead, via lapi.service.annotations in helmrelease.yaml.
traefik.ingress.kubernetes.io/service.serversscheme: https
traefik.ingress.kubernetes.io/service.serverstransport: crowdsec-crowdsec-lapi@kubernetescrd
spec:
ingressClassName: traefik
rules: