feat(monitoring): alert on CrowdSec, starting with the failure that hides

Three rules, and the first one is firing right now for a real reason.

CrowdsecNotReadingLogs. Both agents are frozen at their line counts —
29,917 and 35,449 — with zero increase over an hour, while Traefik keeps
logging and Loki keeps ingesting. cs_bucket_poured_total has not moved in two
hours either: nothing is flowing into any scenario. CrowdSec has not detected
anything since 2026-08-27.

Nothing else shows it. Pods are Running, the LAPI answers, the bouncers pull
and enforce. The line counters are cumulative, so `cscli metrics` reports a
large number and reads as healthy; only the rate distinguishes "has read" from
"is reading". I made exactly that mistake earlier in the day: I checked the
cumulative figure, concluded the pipeline was working, and dismissed a wiki
note about the tailer not reattaching after a log rotation as stale. It was
not stale. And the absence of alerts since 08-27, which I read as a quiet
period, was the symptom.

CrowdsecEdgeBouncerNotPulling counts rather than names. The LAPI registers a
new bouncer per observed client address, and what it observes is the
netbird-ingress pod IP, so `edge@10.244.0.113` and its successors come and go
as those pods roll — a name-based rule would decay. `or vector(0)` is there
because without it the total-failure case is the one the alert misses: all
bouncers silent means the inner filter is empty, count() returns no series,
and nothing fires.

CrowdsecTargetDown is the ordinary one.

Both expressions were run against live data before being written down: the
bouncer count returns 2, and the acquisition rate returns 0 — which is how the
outage was found in the first place.

This does not fix the stalled tailer. Restarting the agents picks up the
current file, but the reattach behaviour is the actual bug and belongs in its
own change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

ansible/inventory/host_vars/proxy-2.renner.uno.yml

@@ -1,19 +0,0 @@
----
-# Arms the CrowdSec firewall bouncer on THIS host only.
-#
-# The role defaults to dry-run for the whole netbird_proxy group, so the two
-# members can be armed one at a time. proxy-2 goes first deliberately: it is
-# the dual-stack member, while proxy-1 is IPv6-only and the one where every
-# network-shaped problem has surfaced first — the dhcpcd/resolv.conf breakage,
-# and the GitHub download that could not work at all.
-#
-# What flipping this does: the bouncer stops logging what it would block and
-# creates two nftables tables (ip crowdsec, ip6 crowdsec6) with the decisions
-# it holds. It touches no existing chain and sets no chain policy.
-#
-# Rollback is `systemctl stop crowdsec-firewall-bouncer` — a clean stop deletes
-# both tables itself. Setting this back to dry-run and re-running the play does
-# the same through the handler.
-#
-# Remove this file once proxy-1 follows and the role default moves to nftables.
-crowdsec_bouncer_mode: nftables

ansible/roles/crowdsec-edge-bouncer/defaults/main.yml

@@ -28,14 +28,23 @@ crowdsec_bouncer_sha256: f86e4b72693549d99f40a9402abefb894108f047a3fbe6e72fada25
 crowdsec_bouncer_api_url: https://crowdsec.renner.uno/
 crowdsec_bouncer_api_key: "{{ vault_crowdsec_edge_bouncer_key }}"
 
-# STARTS INERT, and that is the point.
+# Enforcing. The bouncer creates two nftables tables (ip crowdsec, ip6
+# crowdsec6) holding the decisions it pulled, touches no existing chain and
+# sets no chain policy — SSH and :443 stay open unless the source is banned.
 #
-# dry-run pulls decisions and logs what it would block without creating a
-# single nftables rule. The first bouncer to arm anywhere enforces ~24k CAPI
-# decisions at once on the only external entry path, so the false-positive
-# window has to come before the enforcement, not after it. Flip to `nftables`
-# one host at a time once the log looks right.
-crowdsec_bouncer_mode: dry-run
+# `dry-run` is the other value: it pulls and logs what it would block without
+# creating a single rule. That is what a new host should be given before it
+# enforces, and what an existing one can be dropped back to, per host through
+# inventory/host_vars, without disarming the rest.
+#
+# Worth knowing before arming anything new: the first bouncer on a path
+# enforces the whole CAPI set at once — around 15k addresses today — so the
+# false-positive window belongs before the switch, not after it. And because
+# this chain sits on the INPUT hook while these hosts run no other firewall,
+# there is no conntrack rule accepting established flows ahead of it: a banned
+# address is dropped even when it is answering our own outbound connection.
+# That is why capi_whitelists.yaml carries the proxies and netbird.renner.uno.
+crowdsec_bouncer_mode: nftables
 
 crowdsec_bouncer_update_frequency: 10s
 crowdsec_bouncer_log_level: info

kubernetes/apps/home-assistant/deployment.yaml

@@ -95,7 +95,7 @@ spec:
               memory: 32Mi
       containers:
         - name: home-assistant
-          image: ghcr.io/home-assistant/home-assistant:2026.8.3 # {"$imagepolicy": "flux-system:home-assistant"}
+          image: ghcr.io/home-assistant/home-assistant:2026.9.0 # {"$imagepolicy": "flux-system:home-assistant"}
           env:
             - name: TZ
               value: Europe/Berlin

kubernetes/apps/llama-server/deployment.yaml

@@ -75,27 +75,44 @@ spec:
             # many small per-expert matmuls that sets the pace, not bandwidth.
             - --threads
             - "4"
-            # 32k because Home Assistant's Assist agent measured 18910 tokens
-            # for a single request: ~12k of exposed-entity overview plus ~7k of
-            # base instructions and tool schemas, whose parameters enumerate
-            # every exposed name, area and floor. At 16384 the request was
-            # refused outright.
+            # 16k, and going above it corrupts inference rather than failing.
+            # At 32768 per slot this server answered a 650-token classification
+            # prompt with 1025 tokens of "!" — the KV cache producing numbers
+            # rather than language. It does not crash, it does not warn, and
+            # the OpenAI response looks well-formed, so the damage only shows
+            # in what callers get back: Paperless ran past its 300 s timeout,
+            # and Home Assistant got an empty completion and a hallucinated
+            # device list.
             #
-            # --ctx-size is per slot, not the total: /props reports
-            # n_ctx_slot equal to this value while running four slots, where a
-            # total would have divided down to 4096. Doubling it alone would
-            # therefore double the KV cache, so the slot count halves in step
-            # and the allocation stays where it is today — 24979 MiB of the
-            # 32Gi limit, most of it the 19.5 GiB of weights.
+            # The same Vulkan/Mesa stack already fails hard at --ubatch-size
+            # 2048; this is the quiet version of that limit. Raise this only
+            # against a generation that is read, not against a server that
+            # merely starts.
             #
-            # Two slots is what is actually in use: one holds Home Assistant's
-            # prompt, one holds Paperless's. A third caller would queue rather
-            # than evict either, which is the better failure mode given a cold
-            # prefill of this prompt costs around two minutes.
             - --ctx-size
-            - "32768"
+            - "16384"
+            # One slot, so the 16384 belongs to one request at a time.
+            #
+            # Left on auto, llama.cpp opens four slots and — because auto is
+            # also what enables --kv-unified — has them share a single buffer
+            # of this size. Four Paperless requests at once wanted 6209, 6472,
+            # 4303 and 1271 tokens against those 16384, and all four died with
+            # "Context size has been exceeded".
+            #
+            # They were concurrent because a client timeout does not stop the
+            # generation behind it: each abandoned request keeps consuming the
+            # shared budget the next one needs, so one slow document is enough
+            # to start the pile-up. Queueing is the behaviour worth having
+            # here — nothing calls this server in parallel on purpose, and
+            # Paperless's celery worker runs at concurrency 1.
+            #
+            # No liveness probe change goes with this. The server recovered on
+            # its own once the batch drained, so a probe that restarted the pod
+            # would have made the incident worse; with a single slot, one that
+            # asked for a real completion would also queue behind live work and
+            # kill healthy pods during a long document.
             - --parallel
-            - "2"
+            - "1"
             # A micro-batch of 2048 or more makes this Vulkan/Mesa stack fail
             # with ErrorDeviceLost.
             - --ubatch-size

kubernetes/apps/monitoring-rules/kustomization.yaml

@@ -24,6 +24,7 @@ resources:
   - alloy-servicemonitor.yaml
   - cnpg-podmonitors.yaml
   - crowdsec-servicemonitor.yaml
+  - prometheusrule-crowdsec.yaml
   - home-assistant-servicemonitor.yaml
   - immich-servicemonitor.yaml
   - loki-servicemonitor.yaml

kubernetes/apps/monitoring-rules/prometheusrule-crowdsec.yaml

@@ -0,0 +1,81 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: crowdsec
+  namespace: monitoring
+  labels:
+    # kube-prometheus-stack's Prometheus selects on this label.
+    release: kube-prometheus-stack
+spec:
+  groups:
+    - name: crowdsec.rules
+      rules:
+        - alert: CrowdsecNotReadingLogs
+          # The failure that hides. Everything stays green — pods Running,
+          # LAPI answering, bouncers pulling, decisions being enforced — while
+          # the agents read nothing and therefore detect nothing. The counters
+          # are cumulative, so `cscli metrics` keeps showing a large number and
+          # looks healthy; only the rate tells the truth.
+          #
+          # This alert was written after finding exactly that state: both
+          # agents frozen at their line counts for hours while Traefik kept
+          # logging, and zero events poured into any scenario. It had been that
+          # way since 2026-08-27, and the only outward sign was an absence of
+          # alerts, which reads like quiet rather than blindness.
+          #
+          # Known cause per the wiki: the tailer does not reattach after the
+          # container log rotates.
+          expr: |
+            sum(rate(cs_filesource_hits_total[30m])) == 0
+          for: 30m
+          labels:
+            severity: critical
+          annotations:
+            summary: "CrowdSec liest seit 30min keine Logzeilen"
+            description: >-
+              Die Agents lesen nichts mehr, obwohl Traefik weiter loggt — es
+              wird also nichts erkannt. Prüfen: `cscli metrics` im Agent-Pod,
+              und ob die getailte Datei unter /var/log/containers noch dem
+              laufenden Traefik-Pod entspricht. Ein Neustart der Agents nimmt
+              die aktuelle Datei wieder auf.
+
+        - alert: CrowdsecEdgeBouncerNotPulling
+          # Both proxies must be fetching decisions. A bouncer that stops
+          # pulling keeps enforcing whatever it last held, and a bouncer whose
+          # process died enforces nothing at all — a clean shutdown deletes its
+          # nftables tables. Neither shows up anywhere else.
+          #
+          # Counted rather than named on purpose. The LAPI registers a new
+          # bouncer entry per observed client address, and what it observes is
+          # the netbird-ingress pod IP, so names like `edge@10.244.0.113` come
+          # and go as those pods roll. The count is stable; the names are not.
+          #
+          # `or vector(0)` matters: without it, all bouncers going silent makes
+          # the inner filter empty, count() returns no series, and the alert
+          # never fires — the total-failure case would be the one it misses.
+          expr: |
+            (count(sum by (bouncer) (rate(cs_lapi_bouncer_requests_total[15m])) > 0) or vector(0)) < 2
+          for: 15m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Weniger als zwei CrowdSec-Bouncer holen Decisions ab"
+            description: >-
+              Erwartet werden proxy-1 und proxy-2. Holt einer nichts mehr ab,
+              setzt er einen veralteten Stand durch; ist sein Dienst beendet,
+              setzt er gar nichts mehr durch, weil ein sauberer Stop die
+              nftables-Tabellen löscht. Prüfen: `systemctl status
+              crowdsec-firewall-bouncer` auf beiden Hosts.
+
+        - alert: CrowdsecTargetDown
+          expr: |
+            up{namespace="crowdsec"} == 0
+          for: 10m
+          labels:
+            severity: warning
+          annotations:
+            summary: "CrowdSec-Scrape-Target {{ $labels.pod }} antwortet nicht"
+            description: >-
+              Ein LAPI- oder Agent-Pod ist nicht erreichbar. Für sich noch kein
+              Ausfall der Durchsetzung — die Bouncer behalten ihren letzten
+              Stand —, aber solange das anhält, kommen keine neuen Decisions an.