feat(monitoring): alert on CrowdSec, starting with the failure that hides

Three rules, and the first one is firing right now for a real reason.

CrowdsecNotReadingLogs. Both agents are frozen at their line counts —
29,917 and 35,449 — with zero increase over an hour, while Traefik keeps
logging and Loki keeps ingesting. cs_bucket_poured_total has not moved in two
hours either: nothing is flowing into any scenario. CrowdSec has not detected
anything since 2026-08-27.

Nothing else shows it. Pods are Running, the LAPI answers, the bouncers pull
and enforce. The line counters are cumulative, so `cscli metrics` reports a
large number and reads as healthy; only the rate distinguishes "has read" from
"is reading". I made exactly that mistake earlier in the day: I checked the
cumulative figure, concluded the pipeline was working, and dismissed a wiki
note about the tailer not reattaching after a log rotation as stale. It was
not stale. And the absence of alerts since 08-27, which I read as a quiet
period, was the symptom.

CrowdsecEdgeBouncerNotPulling counts rather than names. The LAPI registers a
new bouncer per observed client address, and what it observes is the
netbird-ingress pod IP, so `edge@10.244.0.113` and its successors come and go
as those pods roll — a name-based rule would decay. `or vector(0)` is there
because without it the total-failure case is the one the alert misses: all
bouncers silent means the inner filter is empty, count() returns no series,
and nothing fires.

CrowdsecTargetDown is the ordinary one.

Both expressions were run against live data before being written down: the
bouncer count returns 2, and the acquisition rate returns 0 — which is how the
outage was found in the first place.

This does not fix the stalled tailer. Restarting the agents picks up the
current file, but the reattach behaviour is the actual bug and belongs in its
own change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

.woodpecker/deploy-netbird-proxy.yml

@@ -10,6 +10,18 @@ when:
       # install-netbird-proxy.yml also applies the edge bouncer.
       - "ansible/roles/crowdsec-edge-bouncer/**"
       - "ansible/plays/install-netbird-proxy.yml"
+      # The variables that steer those roles, not just the roles themselves.
+      # Without these two, a change that only touches inventory never deploys
+      # and nothing says so: arming the bouncer lives in host_vars, and its
+      # LAPI key lives in group_vars, so both would sit in git looking applied
+      # while the hosts kept running the previous configuration.
+      #
+      # pve_hosts.yml is deliberately NOT listed. It is the single inventory
+      # file for the whole fleet, so listing it would redeploy the proxies for
+      # any unrelated host edit. The cost is that a change to [netbird_proxy]
+      # group membership needs a manual deployment run.
+      - "ansible/inventory/host_vars/proxy-*.yml"
+      - "ansible/inventory/group_vars/netbird_proxy/**"
       - ".woodpecker/deploy-netbird-proxy.yml"
   - event: deployment
     evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "deploy-netbird-proxy"'

ansible/roles/crowdsec-edge-bouncer/defaults/main.yml

@@ -28,14 +28,23 @@ crowdsec_bouncer_sha256: f86e4b72693549d99f40a9402abefb894108f047a3fbe6e72fada25
 crowdsec_bouncer_api_url: https://crowdsec.renner.uno/
 crowdsec_bouncer_api_key: "{{ vault_crowdsec_edge_bouncer_key }}"
 
-# STARTS INERT, and that is the point.
+# Enforcing. The bouncer creates two nftables tables (ip crowdsec, ip6
+# crowdsec6) holding the decisions it pulled, touches no existing chain and
+# sets no chain policy — SSH and :443 stay open unless the source is banned.
 #
-# dry-run pulls decisions and logs what it would block without creating a
-# single nftables rule. The first bouncer to arm anywhere enforces ~24k CAPI
-# decisions at once on the only external entry path, so the false-positive
-# window has to come before the enforcement, not after it. Flip to `nftables`
-# one host at a time once the log looks right.
-crowdsec_bouncer_mode: dry-run
+# `dry-run` is the other value: it pulls and logs what it would block without
+# creating a single rule. That is what a new host should be given before it
+# enforces, and what an existing one can be dropped back to, per host through
+# inventory/host_vars, without disarming the rest.
+#
+# Worth knowing before arming anything new: the first bouncer on a path
+# enforces the whole CAPI set at once — around 15k addresses today — so the
+# false-positive window belongs before the switch, not after it. And because
+# this chain sits on the INPUT hook while these hosts run no other firewall,
+# there is no conntrack rule accepting established flows ahead of it: a banned
+# address is dropped even when it is answering our own outbound connection.
+# That is why capi_whitelists.yaml carries the proxies and netbird.renner.uno.
+crowdsec_bouncer_mode: nftables
 
 crowdsec_bouncer_update_frequency: 10s
 crowdsec_bouncer_log_level: info

ansible/roles/crowdsec-edge-bouncer/tasks/main.yml

@@ -19,14 +19,41 @@
     state: directory
     mode: "0755"
 
-- name: Fetch the pinned bouncer release
+# Fetched on the CONTROLLER and pushed, not pulled by the host.
+#
+# proxy-1 is IPv6-only, and neither github.com nor objects.githubusercontent.com
+# publishes an AAAA record — a get_url on the host fails with "Network is
+# unreachable" and no amount of retrying fixes it. Mirroring into
+# gitea.renner.uno is no help either: its public records are the internal LB
+# VIP (10.2.99.86 / fd99::86), so it is unreachable from outside the LAN. And
+# upstream ships no container image for this bouncer, which is expected —
+# it manipulates host netfilter.
+#
+# The CI runner has v4, so it does the fetching. The checksum is verified
+# there, before anything is copied to a host.
+- name: Fetch the pinned bouncer release on the controller
   ansible.builtin.get_url:
     url: >-
       https://github.com/crowdsecurity/cs-firewall-bouncer/releases/download/{{
       crowdsec_bouncer_version }}/crowdsec-firewall-bouncer-linux-amd64.tgz
-    dest: /opt/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}.tgz
+    dest: /tmp/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}.tgz
     checksum: sha256:{{ crowdsec_bouncer_sha256 }}
     mode: "0644"
+  delegate_to: localhost
+  become: false
+  run_once: true
+
+# Only the download had to move to the controller — the extraction stays on
+# the host. The ci-ansible image ships a tar that ansible rejects ("detected as
+# tar type None. GNU tar required"), while both host distributions have GNU
+# tar. Shipping the archive rather than the unpacked binary also keeps the
+# artefact that the checksum was verified against intact all the way to the
+# host.
+- name: Copy the bouncer archive to the host
+  ansible.builtin.copy:
+    src: /tmp/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}.tgz
+    dest: /opt/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}.tgz
+    mode: "0644"
 
 # Only the binary. The tarball also carries install.sh / upgrade.sh /
 # uninstall.sh, which manage systemd and config themselves — that is this

kubernetes/apps/home-assistant/deployment.yaml

@@ -95,7 +95,7 @@ spec:
               memory: 32Mi
       containers:
         - name: home-assistant
-          image: ghcr.io/home-assistant/home-assistant:2026.8.3 # {"$imagepolicy": "flux-system:home-assistant"}
+          image: ghcr.io/home-assistant/home-assistant:2026.9.0 # {"$imagepolicy": "flux-system:home-assistant"}
           env:
             - name: TZ
               value: Europe/Berlin

kubernetes/apps/llama-server/deployment.yaml

@@ -75,11 +75,44 @@ spec:
             # many small per-expert matmuls that sets the pace, not bandwidth.
             - --threads
             - "4"
-            # 16k rather than 8k because the measured prefill falloff is mild
-            # and this model's KV cache is unusually cheap — only 10 of its 40
-            # layers use full attention.
+            # 16k, and going above it corrupts inference rather than failing.
+            # At 32768 per slot this server answered a 650-token classification
+            # prompt with 1025 tokens of "!" — the KV cache producing numbers
+            # rather than language. It does not crash, it does not warn, and
+            # the OpenAI response looks well-formed, so the damage only shows
+            # in what callers get back: Paperless ran past its 300 s timeout,
+            # and Home Assistant got an empty completion and a hallucinated
+            # device list.
+            #
+            # The same Vulkan/Mesa stack already fails hard at --ubatch-size
+            # 2048; this is the quiet version of that limit. Raise this only
+            # against a generation that is read, not against a server that
+            # merely starts.
+            #
             - --ctx-size
             - "16384"
+            # One slot, so the 16384 belongs to one request at a time.
+            #
+            # Left on auto, llama.cpp opens four slots and — because auto is
+            # also what enables --kv-unified — has them share a single buffer
+            # of this size. Four Paperless requests at once wanted 6209, 6472,
+            # 4303 and 1271 tokens against those 16384, and all four died with
+            # "Context size has been exceeded".
+            #
+            # They were concurrent because a client timeout does not stop the
+            # generation behind it: each abandoned request keeps consuming the
+            # shared budget the next one needs, so one slow document is enough
+            # to start the pile-up. Queueing is the behaviour worth having
+            # here — nothing calls this server in parallel on purpose, and
+            # Paperless's celery worker runs at concurrency 1.
+            #
+            # No liveness probe change goes with this. The server recovered on
+            # its own once the batch drained, so a probe that restarted the pod
+            # would have made the incident worse; with a single slot, one that
+            # asked for a real completion would also queue behind live work and
+            # kill healthy pods during a long document.
+            - --parallel
+            - "1"
             # A micro-batch of 2048 or more makes this Vulkan/Mesa stack fail
             # with ErrorDeviceLost.
             - --ubatch-size

kubernetes/apps/llama-server/embed.yaml

@@ -41,7 +41,7 @@ spec:
           image: ghcr.io/ggml-org/llama.cpp:server-vulkan-b10731
           args:
             - --hf-repo
-            - Qwen/Qwen3-Embedding-0.6B-GGUF:Q8_0
+            - gpustack/bge-m3-GGUF:Q8_0
             # Serves POST /v1/embeddings.
             - --embeddings
             # A short, stable id for consumers. Without it the model is
@@ -50,28 +50,41 @@ spec:
             # One name only: llama-server keeps aliases in a sorted set and
             # reports the first as the id, so a second name can steal it.
             - --alias
-            - qwen3-embedding-0.6b
-            # Qwen3-Embedding is trained for last-token pooling. Left
-            # unspecified llama-server takes the GGUF's default, which is not
-            # visible from outside and would silently produce vectors the model
-            # was not trained to emit. Set before anything is indexed: changing
-            # pooling later invalidates every stored embedding.
+            - bge-m3
+            # bge-m3 is trained for CLS pooling — a different type from the
+            # Qwen family this replaces, which uses last-token. Left unspecified
+            # llama-server takes the GGUF default, which is not visible from
+            # outside and would silently produce vectors the model was not
+            # trained to emit. Read from the model's own 1_Pooling/config.json
+            # rather than assumed; of the candidates measured, three different
+            # pooling types were needed.
             - --pooling
-            - last
+            - cls
             - --n-gpu-layers
             - "99"
             - --threads
             - "2"
-            # Paperless chunks documents at 1024 tokens. The defaults are
-            # batch 2048 / ubatch 512, and with --embeddings llama-server
-            # clamps both down to 512 rather than run them mismatched — below
-            # the chunk size. Matching both to 1024 fits a whole chunk in one
-            # pass; 2048 is deliberately avoided, since that is where this
-            # Vulkan stack fails with ErrorDeviceLost.
+            # Paperless chunks at 1024 tokens, but counts them with its own
+            # tokenizer while this server measures the batch in bge-m3's.
+            # Those disagree: XLM-RoBERTa splits German into more pieces than
+            # llama_index's counter does, so a chunk Paperless considers full
+            # at 1024 arrived here as 1069 and the request failed outright —
+            # "input (1069 tokens) is too large to process".
+            #
+            # The headroom therefore has to absorb a tokenizer ratio, not just
+            # a rounding error, and 1.5x covers far more than the 1.044 that
+            # broke it. The defaults are batch 2048 / ubatch 512, and with
+            # --embeddings llama-server clamps both to the smaller rather than
+            # run them mismatched, so both are set.
+            #
+            # 2048 stays avoided: that is where the chat model met
+            # ErrorDeviceLost on this Vulkan stack. That failure came under a
+            # 20 GB model's memory pressure and may not bind a 600 MB one, but
+            # this does not need to find out.
             - --batch-size
-            - "1024"
+            - "1536"
             - --ubatch-size
-            - "1024"
+            - "1536"
             # Room for a full chunk plus overhead rather than exactly one.
             - --ctx-size
             - "4096"
@@ -91,6 +104,14 @@ spec:
             capabilities:
               drop:
                 - ALL
+          # Measured, not estimated: embedding 400 real documents peaked at
+          # 1.6 GiB (Prometheus max_over_time on the working set). The limit is
+          # well above that because the footprint is dominated by the four slots
+          # at 4096 context and the Vulkan working set rather than by the
+          # weights — the Qwen model this replaces peaked at 10.6 GiB on the
+          # same settings for the same 600 MiB of weights, and several
+          # candidates were OOMKilled at limits that looked generous against
+          # their model size.
           resources:
             requests:
               cpu: 250m
@@ -98,7 +119,7 @@ spec:
               gpu.intel.com/i915: "1"
             limits:
               cpu: "2"
-              memory: 4Gi
+              memory: 8Gi
               gpu.intel.com/i915: "1"
           # ~600 MiB rather than ~5 GiB, so a much shorter budget than the chat
           # model needs, but still sized for a cold download.

kubernetes/apps/llama-server/pvc.yaml

@@ -1,10 +1,20 @@
 ---
-# Holds the GGUF weights, downloaded once by the Deployment's initContainer.
-# Prune-disabled so retiring this Kustomization does not throw away several
-# GiB of model files that would have to be pulled from Hugging Face again.
+# Holds the GGUF weights, downloaded once by llama-server itself via --hf-repo.
+# Prune-disabled so retiring this Kustomization does not throw away tens of GiB
+# of model files that would have to be pulled from Hugging Face again.
 #
-# ReadWriteMany so a second replica or a one-off benchmark Job can mount the
-# same weights; the models are read-only at inference time.
+# ReadWriteMany so a second replica, a one-off benchmark Job, or a candidate
+# model being evaluated can mount the same weights; the models are read-only at
+# inference time.
+#
+# Sized well beyond the two deployed models. The cache is shared by every
+# candidate under evaluation (scripts/embed-eval-real.sh), and on a slow
+# connection the download is the long pole — a shared cache lets the large
+# candidates pull in the background while the small ones are already being
+# measured, and means a model is never fetched twice across runs.
+#
+# CephFS supports online expansion (allowVolumeExpansion on the storage class),
+# so growing this does not disturb the running pods.
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
@@ -18,4 +28,4 @@ spec:
   storageClassName: ceph-filesystem
   resources:
     requests:
-      storage: 40Gi
+      storage: 120Gi

kubernetes/apps/monitoring-rules/crowdsec-servicemonitor.yaml

@@ -0,0 +1,36 @@
+# Scrapes CrowdSec. Until this landed, nothing did: `up{namespace="crowdsec"}`
+# returned zero series and Prometheus held no cs_* metric at all, so the
+# detection side of the ingress path was invisible — including whether the
+# agents were reading logs and whether any bouncer was pulling decisions.
+#
+# One monitor for both services. They carry the same shape (a `metrics` port on
+# 6060) and differ only in `app`, so a matchExpressions selector keeps this to
+# a single object rather than two near-identical ones:
+#
+#   crowdsec-service        the LAPI      — decisions, bouncer + machine requests
+#   crowdsec-agent-service  the agents    — acquisition, parsers, scenarios
+#
+# The crowdsec-ns CiliumNetworkPolicy already admits the monitoring namespace
+# without a port restriction, so no policy change is needed.
+#
+# `release: kube-prometheus-stack` is what the chart's Prometheus selects on.
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: crowdsec
+  namespace: crowdsec
+  labels:
+    app.kubernetes.io/name: crowdsec
+    release: kube-prometheus-stack
+spec:
+  selector:
+    matchExpressions:
+      - key: app
+        operator: In
+        values:
+          - crowdsec-service
+          - crowdsec-agent-service
+  endpoints:
+    - port: metrics
+      path: /metrics
+      interval: 30s

kubernetes/apps/monitoring-rules/kustomization.yaml

@@ -23,6 +23,8 @@ resources:
   - prometheusrule-cnpg-backup.yaml
   - alloy-servicemonitor.yaml
   - cnpg-podmonitors.yaml
+  - crowdsec-servicemonitor.yaml
+  - prometheusrule-crowdsec.yaml
   - home-assistant-servicemonitor.yaml
   - immich-servicemonitor.yaml
   - loki-servicemonitor.yaml

kubernetes/apps/monitoring-rules/prometheusrule-crowdsec.yaml

@@ -0,0 +1,81 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: crowdsec
+  namespace: monitoring
+  labels:
+    # kube-prometheus-stack's Prometheus selects on this label.
+    release: kube-prometheus-stack
+spec:
+  groups:
+    - name: crowdsec.rules
+      rules:
+        - alert: CrowdsecNotReadingLogs
+          # The failure that hides. Everything stays green — pods Running,
+          # LAPI answering, bouncers pulling, decisions being enforced — while
+          # the agents read nothing and therefore detect nothing. The counters
+          # are cumulative, so `cscli metrics` keeps showing a large number and
+          # looks healthy; only the rate tells the truth.
+          #
+          # This alert was written after finding exactly that state: both
+          # agents frozen at their line counts for hours while Traefik kept
+          # logging, and zero events poured into any scenario. It had been that
+          # way since 2026-08-27, and the only outward sign was an absence of
+          # alerts, which reads like quiet rather than blindness.
+          #
+          # Known cause per the wiki: the tailer does not reattach after the
+          # container log rotates.
+          expr: |
+            sum(rate(cs_filesource_hits_total[30m])) == 0
+          for: 30m
+          labels:
+            severity: critical
+          annotations:
+            summary: "CrowdSec liest seit 30min keine Logzeilen"
+            description: >-
+              Die Agents lesen nichts mehr, obwohl Traefik weiter loggt — es
+              wird also nichts erkannt. Prüfen: `cscli metrics` im Agent-Pod,
+              und ob die getailte Datei unter /var/log/containers noch dem
+              laufenden Traefik-Pod entspricht. Ein Neustart der Agents nimmt
+              die aktuelle Datei wieder auf.
+
+        - alert: CrowdsecEdgeBouncerNotPulling
+          # Both proxies must be fetching decisions. A bouncer that stops
+          # pulling keeps enforcing whatever it last held, and a bouncer whose
+          # process died enforces nothing at all — a clean shutdown deletes its
+          # nftables tables. Neither shows up anywhere else.
+          #
+          # Counted rather than named on purpose. The LAPI registers a new
+          # bouncer entry per observed client address, and what it observes is
+          # the netbird-ingress pod IP, so names like `edge@10.244.0.113` come
+          # and go as those pods roll. The count is stable; the names are not.
+          #
+          # `or vector(0)` matters: without it, all bouncers going silent makes
+          # the inner filter empty, count() returns no series, and the alert
+          # never fires — the total-failure case would be the one it misses.
+          expr: |
+            (count(sum by (bouncer) (rate(cs_lapi_bouncer_requests_total[15m])) > 0) or vector(0)) < 2
+          for: 15m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Weniger als zwei CrowdSec-Bouncer holen Decisions ab"
+            description: >-
+              Erwartet werden proxy-1 und proxy-2. Holt einer nichts mehr ab,
+              setzt er einen veralteten Stand durch; ist sein Dienst beendet,
+              setzt er gar nichts mehr durch, weil ein sauberer Stop die
+              nftables-Tabellen löscht. Prüfen: `systemctl status
+              crowdsec-firewall-bouncer` auf beiden Hosts.
+
+        - alert: CrowdsecTargetDown
+          expr: |
+            up{namespace="crowdsec"} == 0
+          for: 10m
+          labels:
+            severity: warning
+          annotations:
+            summary: "CrowdSec-Scrape-Target {{ $labels.pod }} antwortet nicht"
+            description: >-
+              Ein LAPI- oder Agent-Pod ist nicht erreichbar. Für sich noch kein
+              Ausfall der Durchsetzung — die Bouncer behalten ihren letzten
+              Stand —, aber solange das anhält, kommen keine neuen Decisions an.

kubernetes/apps/monitoring/dashboards/crowdsec-overview.json

@@ -0,0 +1,163 @@
+{
+  "annotations": {"list": []},
+  "description": "CrowdSec: Erkennung im Cluster, Durchsetzung am NetBird-Edge",
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 1,
+  "id": null,
+  "links": [],
+  "panels": [
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Aktive Ban-Entscheidungen in der LAPI, ganz überwiegend aus den CAPI-Community-Blocklisten. Das ist die Menge, die ein scharfgeschalteter Bouncer durchsetzt.",
+      "fieldConfig": {
+        "defaults": {"color": {"mode": "thresholds"}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "text", "value": null}]}, "unit": "short"},
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 6, "x": 0, "y": 0},
+      "id": 1,
+      "options": {"colorMode": "value", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
+      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(cs_active_decisions)", "refId": "A"}],
+      "title": "Aktive Decisions",
+      "type": "stat"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Bouncer, die in den letzten 10 Minuten Decisions abgeholt haben. Fällt das unter die erwartete Zahl, holt ein Bouncer nichts mehr — und setzt damit einen veralteten Stand durch, ohne dass irgendetwas ausfällt.",
+      "fieldConfig": {
+        "defaults": {"color": {"mode": "thresholds"}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "red", "value": null}, {"color": "green", "value": 1}]}, "unit": "short"},
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 6, "x": 6, "y": 0},
+      "id": 2,
+      "options": {"colorMode": "value", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
+      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "count(sum by (bouncer) (rate(cs_lapi_bouncer_requests_total[10m])) > 0)", "refId": "A"}],
+      "title": "Bouncer, die abholen",
+      "type": "stat"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Vom Agent gelesene Logzeilen pro Sekunde. Steht das auf 0, detektiert CrowdSec nichts mehr — der häufigste stille Ausfall, weil alles andere weiterläuft.",
+      "fieldConfig": {
+        "defaults": {"color": {"mode": "thresholds"}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "red", "value": null}, {"color": "green", "value": 0.01}]}, "unit": "reqps"},
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 6, "x": 12, "y": 0},
+      "id": 3,
+      "options": {"colorMode": "value", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
+      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(rate(cs_filesource_hits_total[5m]))", "refId": "A"}],
+      "title": "Gelesene Logzeilen/s",
+      "type": "stat"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Szenario-Overflows der letzten 24 h — also lokal erkannte Angriffe, im Gegensatz zu den von CAPI übernommenen.",
+      "fieldConfig": {
+        "defaults": {"color": {"mode": "thresholds"}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "text", "value": null}]}, "unit": "short"},
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 6, "x": 18, "y": 0},
+      "id": 4,
+      "options": {"colorMode": "value", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
+      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(increase(cs_bucket_overflowed_total[24h]))", "refId": "A"}],
+      "title": "Lokale Overflows (24 h)",
+      "type": "stat"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Abholrate je Bouncer. Der Firewall-Bouncer auf den Proxies fragt alle 10 s. Die LAPI legt bei jeder neuen beobachteten Client-Adresse einen Eintrag `<name>@<ip>` an, und beobachtet wird die Pod-IP des netbird-ingress-Peers — deshalb erscheint ein logischer Bouncer hier mehrfach und nach Pod-Rollen unter neuen Namen.",
+      "fieldConfig": {
+        "defaults": {"color": {"mode": "palette-classic"}, "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]}, "unit": "reqps"},
+        "overrides": []
+      },
+      "gridPos": {"h": 8, "w": 12, "x": 0, "y": 4},
+      "id": 5,
+      "options": {"legend": {"calcs": ["lastNotNull"], "displayMode": "table", "placement": "right", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum by (bouncer) (rate(cs_lapi_bouncer_requests_total[5m]))", "legendFormat": "{{bouncer}}", "refId": "A"}],
+      "title": "Bouncer-Abholrate",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Aktive Decisions nach Grund. `http:scan` dominiert und stammt aus den CAPI-Blocklisten.",
+      "fieldConfig": {
+        "defaults": {"color": {"mode": "palette-classic"}, "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 20, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "normal"}, "thresholdsStyle": {"mode": "off"}}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]}, "unit": "short"},
+        "overrides": []
+      },
+      "gridPos": {"h": 8, "w": 12, "x": 12, "y": 4},
+      "id": 6,
+      "options": {"legend": {"calcs": ["lastNotNull"], "displayMode": "table", "placement": "right", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum by (reason) (cs_active_decisions)", "legendFormat": "{{reason}}", "refId": "A"}],
+      "title": "Aktive Decisions nach Grund",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Welche Szenarien lokal auslösen. Das ist das, was CrowdSec selbst aus den Traefik-Logs erkennt.",
+      "fieldConfig": {
+        "defaults": {"color": {"mode": "palette-classic"}, "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "bars", "fillOpacity": 70, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "normal"}, "thresholdsStyle": {"mode": "off"}}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]}, "unit": "short"},
+        "overrides": []
+      },
+      "gridPos": {"h": 8, "w": 12, "x": 0, "y": 12},
+      "id": 7,
+      "options": {"legend": {"calcs": ["sum"], "displayMode": "table", "placement": "right", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum by (name) (increase(cs_bucket_overflowed_total[1h]))", "legendFormat": "{{name}}", "refId": "A"}],
+      "title": "Szenario-Overflows (je Stunde)",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Geparste gegen nicht geparste Zeilen. Ein Anstieg bei `ko` heißt meist, dass sich ein Logformat geändert hat und der Parser nicht mehr greift — CrowdSec liest dann weiter, erkennt aber nichts mehr.",
+      "fieldConfig": {
+        "defaults": {"color": {"mode": "palette-classic"}, "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]}, "unit": "reqps"},
+        "overrides": []
+      },
+      "gridPos": {"h": 8, "w": 12, "x": 12, "y": 12},
+      "id": 8,
+      "options": {"legend": {"calcs": ["lastNotNull"], "displayMode": "list", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(rate(cs_parser_hits_ok_total[5m]))", "legendFormat": "geparst", "refId": "A"},
+        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(rate(cs_parser_hits_ko_total[5m]))", "legendFormat": "nicht geparst", "refId": "B"}
+      ],
+      "title": "Parser",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Whitelist-Treffer nach Grund. Nützlich beim Scharfschalten: was hier zählt, wäre sonst ein Kandidat für einen Ban gewesen.",
+      "fieldConfig": {
+        "defaults": {"color": {"mode": "palette-classic"}, "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]}, "unit": "reqps"},
+        "overrides": []
+      },
+      "gridPos": {"h": 8, "w": 12, "x": 0, "y": 20},
+      "id": 9,
+      "options": {"legend": {"calcs": ["lastNotNull"], "displayMode": "table", "placement": "right", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum by (reason) (rate(cs_node_wl_hits_ok_total[5m]))", "legendFormat": "{{reason}}", "refId": "A"}],
+      "title": "Whitelist-Treffer",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "gridPos": {"h": 8, "w": 12, "x": 12, "y": 20},
+      "id": 10,
+      "options": {
+        "code": {"language": "plaintext", "showLineNumbers": false, "showMiniMap": false},
+        "content": "## Was dieses Dashboard NICHT zeigt\n\nAlle Panels stammen aus der LAPI und den Agents im Cluster. Die **Bouncer auf proxy-1 und proxy-2 exportieren eigene Metriken**, die hier fehlen:\n\n- `fw_bouncer_active_decisions` — was tatsächlich in den nftables-Sets steht\n- `fw_bouncer_dropped_packets` / `_bytes` — was wirklich verworfen wurde\n- `lapi_requests_total` / `_failures_total` — aus Sicht des Bouncers\n\nSie liegen auf `127.0.0.1:60601` der Proxy-Hosts. Die Hosts sind öffentlich und haben bisher keinerlei Metrik-Pfad; den Port ins Internet zu öffnen, nur um ein Scrape-Ziel zu bekommen, wäre der falsche Tausch.\n\n**Konsequenz:** „Bouncer-Abholrate\" oben zeigt, dass ein Bouncer *fragt* — nicht, dass er die Antwort auch angewandt hat. Solange der Modus `dry-run` ist, wendet er ohnehin nichts an.\n\nEin Scrape-Pfad für die Hosts ist eigene Arbeit und Voraussetzung dafür, dass „was wird wirklich geblockt\" beantwortbar wird."
+      },
+      "title": "Blinder Fleck",
+      "type": "text"
+    }
+  ],
+  "refresh": "30s",
+  "schemaVersion": 38,
+  "style": "dark",
+  "tags": ["homelab", "crowdsec", "security"],
+  "templating": {"list": []},
+  "time": {"from": "now-6h", "to": "now"},
+  "timepicker": {},
+  "timezone": "",
+  "title": "CrowdSec Overview",
+  "uid": "crowdsec-overview",
+  "version": 1,
+  "weekStart": ""
+}

kubernetes/apps/monitoring/dashboards/kustomization.yaml

@@ -38,6 +38,9 @@ configMapGenerator:
   - name: dashboard-blackbox-overview
     files:
       - blackbox-overview.json
+  - name: dashboard-crowdsec-overview
+    files:
+      - crowdsec-overview.json
   - name: dashboard-disk-health
     files:
       - disk-health.json

kubernetes/apps/paperless/paperless.yaml

@@ -82,7 +82,10 @@ data:
   # primarily English archives — the better fit for a German corpus.
   PAPERLESS_AI_LLM_EMBEDDING_BACKEND: "openai-like"
   PAPERLESS_AI_LLM_EMBEDDING_ENDPOINT: "http://llama-embed.ai.svc.cluster.local:8080/v1"
-  PAPERLESS_AI_LLM_EMBEDDING_MODEL: "qwen3-embedding-0.6b"
+  # Must match --alias in kubernetes/apps/llama-server/embed.yaml. Changing the
+  # embedding model invalidates every stored vector, so a swap means the LLM
+  # index has to be rebuilt.
+  PAPERLESS_AI_LLM_EMBEDDING_MODEL: "bge-m3"
   PAPERLESS_AI_LLM_EMBEDDING_CHUNK_SIZE: "1024"
 ---
 apiVersion: apps/v1

kubernetes/infrastructure/crowdsec/helmrelease.yaml

@@ -65,6 +65,21 @@ spec:
           - 45.157.177.26/32
           - 2a03:4000:4b:c1e:4474:43ff:fedf:8f85/128
           - 2a03:4000:28:f6e:a4c7:59ff:fe1e:9a29/128
+          # netbird.renner.uno — management, signal, relay and TURN.
+          #
+          # Not optional once a bouncer is armed. Its chain sits on the INPUT
+          # hook, the proxy hosts run no other firewall, and there is therefore
+          # no conntrack rule accepting established flows ahead of it. A ban on
+          # this address would drop the RETURN traffic of the proxies' own
+          # outbound control connection and take them off the mesh — with no
+          # inbound attack involved at all.
+          #
+          # Checked when this was added: none of these addresses appear in the
+          # current decision set. That is the state today, not a guarantee —
+          # these are rented addresses on a hoster whose ranges do end up on
+          # community blocklists.
+          - 162.55.50.97/32
+          - 2a01:4f8:c2c:4579::1/128
 
     # mTLS between agents and the LAPI. Do not switch back to password auth:
     # the LAPI runs without persistence, so its machine-registration DB is
@@ -108,6 +123,24 @@ spec:
 
     lapi:
       replicas: 1
+      service:
+        # Backend wiring for the public LAPI Ingress (lapi-ingress.yaml).
+        #
+        # These belong on the SERVICE, not on the Ingress: under the Kubernetes
+        # Ingress provider `traefik.ingress.kubernetes.io/service.*` are service
+        # annotations, and on an Ingress they are silently ignored. The symptom
+        # is Traefik speaking plaintext to a TLS-only backend, which the LAPI
+        # answers with "Client sent an HTTP request to an HTTPS server." and a
+        # 400 — an error that looks like a malformed request rather than a
+        # scheme mismatch.
+        #
+        # With tls.enabled the LAPI serves HTTPS on 8080 and offers no
+        # plaintext port, so https here is mandatory rather than a hardening
+        # choice. The ServersTransport carries the chart's own CA, since that
+        # certificate is self-signed.
+        annotations:
+          traefik.ingress.kubernetes.io/service.serversscheme: https
+          traefik.ingress.kubernetes.io/service.serverstransport: crowdsec-crowdsec-lapi@kubernetescrd
       # Far below the chart default, which reserves half a core per LAPI for a
       # workload that draws single-digit millicores — enough to leave the
       # cluster unschedulable with one node out. Only the reservation shrinks;

kubernetes/infrastructure/crowdsec/lapi-ingress.yaml

@@ -44,8 +44,8 @@ metadata:
   namespace: crowdsec
 spec:
   serverName: crowdsec-service.crowdsec.svc.cluster.local
-  rootCAsSecrets:
-    - crowdsec-root-secret
+  rootCAs:
+    - secret: crowdsec-root-secret
 ---
 apiVersion: networking.k8s.io/v1
 kind: Ingress
@@ -60,8 +60,12 @@ metadata:
     cert-manager.io/cluster-issuer: letsencrypt-cluster
     # NOT behind the Authentik gate: the client is a daemon with an API key,
     # not a browser that can complete an OIDC flow.
-    traefik.ingress.kubernetes.io/service.serversscheme: https
-    traefik.ingress.kubernetes.io/service.serverstransport: crowdsec-crowdsec-lapi@kubernetescrd
+    #
+    # The backend scheme and ServersTransport are NOT set here. Under the
+    # Kubernetes Ingress provider, `traefik.ingress.kubernetes.io/service.*`
+    # are SERVICE annotations — on an Ingress they are silently ignored, and
+    # Traefik then talks plaintext to a TLS-only backend. They live on the
+    # chart's Service instead, via lapi.service.annotations in helmrelease.yaml.
 spec:
   ingressClassName: traefik
   rules:

kubernetes/infrastructure/network-policies/ai-ns.yaml

@@ -34,6 +34,17 @@ spec:
         - ports:
             - port: "8080"
               protocol: TCP
+    # Home Assistant's llama.cpp conversation agent calls the chat model only —
+    # it has no use for the embedding endpoint, so unlike paperless above this
+    # grant does not extend to llama-embed.
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": home-assistant
+            app.kubernetes.io/name: home-assistant
+      toPorts:
+        - ports:
+            - port: "8080"
+              protocol: TCP
     - fromEndpoints:
         - matchLabels:
             "k8s:io.kubernetes.pod.namespace": monitoring

kubernetes/infrastructure/network-policies/home-assistant-ns.yaml

@@ -167,6 +167,18 @@ spec:
     - toEndpoints:
         - matchLabels:
             "k8s:io.kubernetes.pod.namespace": influxdb
+    # The cluster's own chat model, for the llama.cpp conversation agent that
+    # backs Assist. Scoped to the one pod and port rather than the ai namespace
+    # as a whole, so this grant cannot widen if something else is placed there —
+    # the embedding endpoint next to it is deliberately not included.
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": ai
+            app.kubernetes.io/name: llama-server
+      toPorts:
+        - ports:
+            - port: "8080"
+              protocol: TCP
     # traefik for east-west OIDC roundtrips that go through the
     # public hostname via k8s_gateway.
     - toEndpoints: