Compare commits

...
Author SHA1 Message Date
claude-botandClaude Opus 5 05cd7f3674 feat(crowdsec): arm the bouncer on proxy-2
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
First host goes from dry-run to enforcing. proxy-2 deliberately: it is the
dual-stack member, while proxy-1 is IPv6-only and the one where every
network-shaped problem has surfaced first — the dhcpcd/resolv.conf breakage,
and a GitHub download that could not work at all. host_vars keeps the change
to one host; proxy-1 stays on the role default.

Verified before committing: ansible-inventory resolves nftables for proxy-2
and nothing for proxy-1.

Adds netbird.renner.uno to capi_whitelists first, and that is not decoration.
The bouncer's chain sits on the INPUT hook, the proxy hosts run no other
firewall, and there is therefore no conntrack rule accepting established flows
ahead of it. A ban on the management address would drop the return traffic of
the proxies' own outbound control connection and take them off the mesh, with
no inbound attack involved at all. The same reasoning already covers the proxy
addresses, which were listed from the start.

Checked at the time of writing: none of these addresses appear in the current
decision set — searched all ~15k entries. That is the state today, not a
guarantee; they are rented addresses on a hoster whose ranges do end up on
community blocklists, which is the whole reason for listing them.

The delivery path was tested end to end beforehand with a decision on
192.0.2.1 (TEST-NET-1, so no real traffic could be affected). Both bouncers
received the ban within one poll cycle — a 206-byte delta against the usual
26 — and the revocation the same way, 203 bytes, ~5s after deletion. What that
did not test is whether a packet is actually dropped, which dry-run cannot
show. This commit is what makes that testable.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-02 16:13:11 +02:00
k.renner d655d65ad9 Merge pull request 'feat(llama-server): 32k context per slot for the Assist agent' (#765) from llama-ctx-assist into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#765
2026-09-02 14:09:07 +00:00
claude-botandClaude Opus 5 60a3da9447 feat(llama-server): 32k context per slot for the Assist agent
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The Home Assistant conversation agent is configured and refuses every
request:

    API error: request (18910 tokens) exceeds the available
    context size (16384 tokens), try increasing it

18910 breaks down as ~12k for the exposed-entity overview and ~7k for
the base instructions and tool schemas, whose parameters enumerate every
exposed name, area and floor. Trimming the exposed set from 845 entities
to 666 already took the overview from 16045 tokens to 12048; the
remainder is not the sort of thing curation reaches.

--ctx-size is per slot rather than the total, which is what makes this
affordable: /props reports n_ctx_slot equal to the flag while running
four slots, where a total would have divided down to 4096. Doubling the
flag alone would double the KV cache, so the slot count halves in step
and the allocation lands where it already is -- 24979 MiB against the
32Gi limit, most of it the 19.5 GiB of weights.

Two slots matches the actual callers: one holds Home Assistant's prompt,
one holds Paperless's, and prompt caching keeps each warm. A cold prefill
of this prompt costs about two minutes, so a third caller queueing is a
better outcome than it evicting one of them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-02 16:07:06 +02:00
k.renner 47086a5fe9 Merge pull request 'feat(monitoring): scrape CrowdSec and give it a dashboard' (#764) from crowdsec-metrics-dashboard into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#764
2026-09-02 13:59:21 +00:00
k.renner 412acf7cb4 Merge pull request 'feat(network-policies): let Home Assistant reach the chat model' (#759) from ha-llm-agent into master
Reviewed-on: k.renner/homelab#759
2026-09-02 13:59:00 +00:00
k.renner 75ecf020cc Merge pull request 'fix(llama-embed): size the batch for bge-m3 tokenizer, not Paperless' (#762) from embed-batch-fix into master
Reviewed-on: k.renner/homelab#762
2026-09-02 13:58:42 +00:00
claude-botandClaude Opus 5 6a65a5982a feat(monitoring): scrape CrowdSec and give it a dashboard
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
A dashboard first needed something to draw. Nothing scraped CrowdSec:
`up{namespace="crowdsec"}` returned zero series and Prometheus held no cs_*
metric at all, so the detection half of the ingress path was invisible —
including whether the agents were reading logs and whether any bouncer was
pulling decisions.

Both chart services already expose a `metrics` port on 6060 and the
crowdsec-ns CiliumNetworkPolicy already admits the monitoring namespace
without a port restriction. Only the ServiceMonitor was missing. One object
covers both services via matchExpressions, since they differ only in `app`.

Every panel query was written against label sets read off the live endpoints
rather than from the documentation:

  cs_active_decisions{action,origin,reason}
  cs_lapi_bouncer_requests_total{bouncer,method,route}
  cs_bucket_overflowed_total{name}
  cs_filesource_hits_total{acquis_type,datasource_type,source}
  cs_node_wl_hits_ok_total{...,reason,...}
  cs_parser_hits_ok_total / _ko_total

The bouncer panel is the one worth having: the LAPI counts requests per
bouncer, so a bouncer that stops pulling shows up as a line going to zero
rather than as nothing at all. It also makes the `<name>@<ip>` behaviour
visible — the LAPI registers a new entry per observed client address, and what
it observes is the netbird-ingress pod IP, so one logical bouncer appears
several times and gains new names as those pods roll.

The last panel is a text panel naming what the dashboard cannot show. The
bouncers' own metrics — fw_bouncer_active_decisions, dropped packets, their
view of the LAPI — sit on 127.0.0.1:60601 on public hosts with no metrics path
of any kind. So "Bouncer-Abholrate" proves a bouncer is asking, not that it
applied the answer. Better to say that on the dashboard than to let someone
read enforcement into a panel that does not measure it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-02 15:58:17 +02:00
k.renner 154d0328ce Merge pull request 'fix(crowdsec): move the LAPI backend annotations onto the Service' (#760) from crowdsec-lapi-service-annotations into master
Reviewed-on: k.renner/homelab#760
2026-09-02 13:48:51 +00:00
claude-botandClaude Opus 5 f86f19045e fix(crowdsec): move the LAPI backend annotations onto the Service
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Both bouncers reach the LAPI and pull every 10s, and every pull gets a 400:

  2a03:4000:28:f6e:...:9a29 "GET /v1/decisions/stream?..." 400 48
  2a03:4000:4b:c1e:...:8f85 "GET /v1/decisions/stream?..." 400 48

The body says what it is: "Client sent an HTTP request to an HTTPS server."
Traefik was talking plaintext to a backend that serves TLS only.

serversscheme and serverstransport were on the Ingress. Under the Kubernetes
Ingress provider `traefik.ingress.kubernetes.io/service.*` are SERVICE
annotations; on an Ingress they are silently ignored. Nothing warns, and the
failure surfaces as a 400 that reads like a malformed request rather than a
scheme mismatch. They move to the chart's Service via lapi.service.annotations.

With tls.enabled the LAPI serves HTTPS on 8080 and offers no plaintext port,
so https is mandatory here rather than hardening.

Also switches the ServersTransport from rootCAsSecrets to rootCAs, which is
what Traefik has been warning about on every reconcile:

  RootCAsSecrets option is deprecated, please use the RootCA option instead.

Two of my own diagnostics were wrong along the way and both were measurement
artifacts. "Traefik sees no requests" came from a Loki query whose limit was
consumed by those deprecation warnings. And the source address that justified
dropping the IPAllowList was measured on the watchdog, which curls 127.0.0.1;
the bouncer resolves the public name and arrives on the proxy's public
address, as the log lines above show.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-02 14:53:53 +02:00
claude-botandClaude Opus 5 7903c81c9a feat(network-policies): let Home Assistant reach the chat model
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
HA 2026.8.3 ships a native `llama_cpp` integration, so an Assist
conversation agent backed by the cluster's own model needs no custom
component — only a path from the HA pod to llama-server, which two
default-deny policies currently block in both directions. Verified from
inside the pod: the call to llama-server.ai.svc.cluster.local:8080 times
out today.

Both grants are scoped to the single pod and port, mirroring the
paperless grant next to them, so neither widens if something else is
placed in the other namespace. llama-embed is deliberately excluded: the
conversation agent uses chat completions only.

Tool calling was verified against the running model before adding this —
the Assist API requires it, and llama.cpp only exposes it for some
builds. b10731 returns finish_reason=tool_calls and parses German
commands into the right arguments without --jinja.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-02 14:18:30 +02:00
10 changed files with 306 additions and 8 deletions
@@ -0,0 +1,19 @@
---
# Arms the CrowdSec firewall bouncer on THIS host only.
#
# The role defaults to dry-run for the whole netbird_proxy group, so the two
# members can be armed one at a time. proxy-2 goes first deliberately: it is
# the dual-stack member, while proxy-1 is IPv6-only and the one where every
# network-shaped problem has surfaced first — the dhcpcd/resolv.conf breakage,
# and the GitHub download that could not work at all.
#
# What flipping this does: the bouncer stops logging what it would block and
# creates two nftables tables (ip crowdsec, ip6 crowdsec6) with the decisions
# it holds. It touches no existing chain and sets no chain policy.
#
# Rollback is `systemctl stop crowdsec-firewall-bouncer` — a clean stop deletes
# both tables itself. Setting this back to dry-run and re-running the play does
# the same through the handler.
#
# Remove this file once proxy-1 follows and the role default moves to nftables.
crowdsec_bouncer_mode: nftables
+20 -4
View File
@@ -75,11 +75,27 @@ spec:
# many small per-expert matmuls that sets the pace, not bandwidth.
- --threads
- "4"
# 16k rather than 8k because the measured prefill falloff is mild
# and this model's KV cache is unusually cheap — only 10 of its 40
# layers use full attention.
# 32k because Home Assistant's Assist agent measured 18910 tokens
# for a single request: ~12k of exposed-entity overview plus ~7k of
# base instructions and tool schemas, whose parameters enumerate
# every exposed name, area and floor. At 16384 the request was
# refused outright.
#
# --ctx-size is per slot, not the total: /props reports
# n_ctx_slot equal to this value while running four slots, where a
# total would have divided down to 4096. Doubling it alone would
# therefore double the KV cache, so the slot count halves in step
# and the allocation stays where it is today — 24979 MiB of the
# 32Gi limit, most of it the 19.5 GiB of weights.
#
# Two slots is what is actually in use: one holds Home Assistant's
# prompt, one holds Paperless's. A third caller would queue rather
# than evict either, which is the better failure mode given a cold
# prefill of this prompt costs around two minutes.
- --ctx-size
- "16384"
- "32768"
- --parallel
- "2"
# A micro-batch of 2048 or more makes this Vulkan/Mesa stack fail
# with ErrorDeviceLost.
- --ubatch-size
@@ -0,0 +1,36 @@
# Scrapes CrowdSec. Until this landed, nothing did: `up{namespace="crowdsec"}`
# returned zero series and Prometheus held no cs_* metric at all, so the
# detection side of the ingress path was invisible — including whether the
# agents were reading logs and whether any bouncer was pulling decisions.
#
# One monitor for both services. They carry the same shape (a `metrics` port on
# 6060) and differ only in `app`, so a matchExpressions selector keeps this to
# a single object rather than two near-identical ones:
#
# crowdsec-service the LAPI — decisions, bouncer + machine requests
# crowdsec-agent-service the agents — acquisition, parsers, scenarios
#
# The crowdsec-ns CiliumNetworkPolicy already admits the monitoring namespace
# without a port restriction, so no policy change is needed.
#
# `release: kube-prometheus-stack` is what the chart's Prometheus selects on.
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: crowdsec
namespace: crowdsec
labels:
app.kubernetes.io/name: crowdsec
release: kube-prometheus-stack
spec:
selector:
matchExpressions:
- key: app
operator: In
values:
- crowdsec-service
- crowdsec-agent-service
endpoints:
- port: metrics
path: /metrics
interval: 30s
@@ -23,6 +23,7 @@ resources:
- prometheusrule-cnpg-backup.yaml
- alloy-servicemonitor.yaml
- cnpg-podmonitors.yaml
- crowdsec-servicemonitor.yaml
- home-assistant-servicemonitor.yaml
- immich-servicemonitor.yaml
- loki-servicemonitor.yaml
@@ -0,0 +1,163 @@
{
"annotations": {"list": []},
"description": "CrowdSec: Erkennung im Cluster, Durchsetzung am NetBird-Edge",
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 1,
"id": null,
"links": [],
"panels": [
{
"datasource": {"type": "prometheus", "uid": "prometheus"},
"description": "Aktive Ban-Entscheidungen in der LAPI, ganz überwiegend aus den CAPI-Community-Blocklisten. Das ist die Menge, die ein scharfgeschalteter Bouncer durchsetzt.",
"fieldConfig": {
"defaults": {"color": {"mode": "thresholds"}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "text", "value": null}]}, "unit": "short"},
"overrides": []
},
"gridPos": {"h": 4, "w": 6, "x": 0, "y": 0},
"id": 1,
"options": {"colorMode": "value", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
"targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(cs_active_decisions)", "refId": "A"}],
"title": "Aktive Decisions",
"type": "stat"
},
{
"datasource": {"type": "prometheus", "uid": "prometheus"},
"description": "Bouncer, die in den letzten 10 Minuten Decisions abgeholt haben. Fällt das unter die erwartete Zahl, holt ein Bouncer nichts mehr — und setzt damit einen veralteten Stand durch, ohne dass irgendetwas ausfällt.",
"fieldConfig": {
"defaults": {"color": {"mode": "thresholds"}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "red", "value": null}, {"color": "green", "value": 1}]}, "unit": "short"},
"overrides": []
},
"gridPos": {"h": 4, "w": 6, "x": 6, "y": 0},
"id": 2,
"options": {"colorMode": "value", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
"targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "count(sum by (bouncer) (rate(cs_lapi_bouncer_requests_total[10m])) > 0)", "refId": "A"}],
"title": "Bouncer, die abholen",
"type": "stat"
},
{
"datasource": {"type": "prometheus", "uid": "prometheus"},
"description": "Vom Agent gelesene Logzeilen pro Sekunde. Steht das auf 0, detektiert CrowdSec nichts mehr — der häufigste stille Ausfall, weil alles andere weiterläuft.",
"fieldConfig": {
"defaults": {"color": {"mode": "thresholds"}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "red", "value": null}, {"color": "green", "value": 0.01}]}, "unit": "reqps"},
"overrides": []
},
"gridPos": {"h": 4, "w": 6, "x": 12, "y": 0},
"id": 3,
"options": {"colorMode": "value", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
"targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(rate(cs_filesource_hits_total[5m]))", "refId": "A"}],
"title": "Gelesene Logzeilen/s",
"type": "stat"
},
{
"datasource": {"type": "prometheus", "uid": "prometheus"},
"description": "Szenario-Overflows der letzten 24 h — also lokal erkannte Angriffe, im Gegensatz zu den von CAPI übernommenen.",
"fieldConfig": {
"defaults": {"color": {"mode": "thresholds"}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "text", "value": null}]}, "unit": "short"},
"overrides": []
},
"gridPos": {"h": 4, "w": 6, "x": 18, "y": 0},
"id": 4,
"options": {"colorMode": "value", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
"targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(increase(cs_bucket_overflowed_total[24h]))", "refId": "A"}],
"title": "Lokale Overflows (24 h)",
"type": "stat"
},
{
"datasource": {"type": "prometheus", "uid": "prometheus"},
"description": "Abholrate je Bouncer. Der Firewall-Bouncer auf den Proxies fragt alle 10 s. Die LAPI legt bei jeder neuen beobachteten Client-Adresse einen Eintrag `<name>@<ip>` an, und beobachtet wird die Pod-IP des netbird-ingress-Peers — deshalb erscheint ein logischer Bouncer hier mehrfach und nach Pod-Rollen unter neuen Namen.",
"fieldConfig": {
"defaults": {"color": {"mode": "palette-classic"}, "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]}, "unit": "reqps"},
"overrides": []
},
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 4},
"id": 5,
"options": {"legend": {"calcs": ["lastNotNull"], "displayMode": "table", "placement": "right", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
"targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum by (bouncer) (rate(cs_lapi_bouncer_requests_total[5m]))", "legendFormat": "{{bouncer}}", "refId": "A"}],
"title": "Bouncer-Abholrate",
"type": "timeseries"
},
{
"datasource": {"type": "prometheus", "uid": "prometheus"},
"description": "Aktive Decisions nach Grund. `http:scan` dominiert und stammt aus den CAPI-Blocklisten.",
"fieldConfig": {
"defaults": {"color": {"mode": "palette-classic"}, "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 20, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "normal"}, "thresholdsStyle": {"mode": "off"}}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]}, "unit": "short"},
"overrides": []
},
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 4},
"id": 6,
"options": {"legend": {"calcs": ["lastNotNull"], "displayMode": "table", "placement": "right", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
"targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum by (reason) (cs_active_decisions)", "legendFormat": "{{reason}}", "refId": "A"}],
"title": "Aktive Decisions nach Grund",
"type": "timeseries"
},
{
"datasource": {"type": "prometheus", "uid": "prometheus"},
"description": "Welche Szenarien lokal auslösen. Das ist das, was CrowdSec selbst aus den Traefik-Logs erkennt.",
"fieldConfig": {
"defaults": {"color": {"mode": "palette-classic"}, "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "bars", "fillOpacity": 70, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "normal"}, "thresholdsStyle": {"mode": "off"}}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]}, "unit": "short"},
"overrides": []
},
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 12},
"id": 7,
"options": {"legend": {"calcs": ["sum"], "displayMode": "table", "placement": "right", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
"targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum by (name) (increase(cs_bucket_overflowed_total[1h]))", "legendFormat": "{{name}}", "refId": "A"}],
"title": "Szenario-Overflows (je Stunde)",
"type": "timeseries"
},
{
"datasource": {"type": "prometheus", "uid": "prometheus"},
"description": "Geparste gegen nicht geparste Zeilen. Ein Anstieg bei `ko` heißt meist, dass sich ein Logformat geändert hat und der Parser nicht mehr greift — CrowdSec liest dann weiter, erkennt aber nichts mehr.",
"fieldConfig": {
"defaults": {"color": {"mode": "palette-classic"}, "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]}, "unit": "reqps"},
"overrides": []
},
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 12},
"id": 8,
"options": {"legend": {"calcs": ["lastNotNull"], "displayMode": "list", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
"targets": [
{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(rate(cs_parser_hits_ok_total[5m]))", "legendFormat": "geparst", "refId": "A"},
{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(rate(cs_parser_hits_ko_total[5m]))", "legendFormat": "nicht geparst", "refId": "B"}
],
"title": "Parser",
"type": "timeseries"
},
{
"datasource": {"type": "prometheus", "uid": "prometheus"},
"description": "Whitelist-Treffer nach Grund. Nützlich beim Scharfschalten: was hier zählt, wäre sonst ein Kandidat für einen Ban gewesen.",
"fieldConfig": {
"defaults": {"color": {"mode": "palette-classic"}, "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]}, "unit": "reqps"},
"overrides": []
},
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 20},
"id": 9,
"options": {"legend": {"calcs": ["lastNotNull"], "displayMode": "table", "placement": "right", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
"targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum by (reason) (rate(cs_node_wl_hits_ok_total[5m]))", "legendFormat": "{{reason}}", "refId": "A"}],
"title": "Whitelist-Treffer",
"type": "timeseries"
},
{
"datasource": {"type": "prometheus", "uid": "prometheus"},
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 20},
"id": 10,
"options": {
"code": {"language": "plaintext", "showLineNumbers": false, "showMiniMap": false},
"content": "## Was dieses Dashboard NICHT zeigt\n\nAlle Panels stammen aus der LAPI und den Agents im Cluster. Die **Bouncer auf proxy-1 und proxy-2 exportieren eigene Metriken**, die hier fehlen:\n\n- `fw_bouncer_active_decisions` — was tatsächlich in den nftables-Sets steht\n- `fw_bouncer_dropped_packets` / `_bytes` — was wirklich verworfen wurde\n- `lapi_requests_total` / `_failures_total` — aus Sicht des Bouncers\n\nSie liegen auf `127.0.0.1:60601` der Proxy-Hosts. Die Hosts sind öffentlich und haben bisher keinerlei Metrik-Pfad; den Port ins Internet zu öffnen, nur um ein Scrape-Ziel zu bekommen, wäre der falsche Tausch.\n\n**Konsequenz:** „Bouncer-Abholrate\" oben zeigt, dass ein Bouncer *fragt* — nicht, dass er die Antwort auch angewandt hat. Solange der Modus `dry-run` ist, wendet er ohnehin nichts an.\n\nEin Scrape-Pfad für die Hosts ist eigene Arbeit und Voraussetzung dafür, dass „was wird wirklich geblockt\" beantwortbar wird."
},
"title": "Blinder Fleck",
"type": "text"
}
],
"refresh": "30s",
"schemaVersion": 38,
"style": "dark",
"tags": ["homelab", "crowdsec", "security"],
"templating": {"list": []},
"time": {"from": "now-6h", "to": "now"},
"timepicker": {},
"timezone": "",
"title": "CrowdSec Overview",
"uid": "crowdsec-overview",
"version": 1,
"weekStart": ""
}
@@ -38,6 +38,9 @@ configMapGenerator:
- name: dashboard-blackbox-overview
files:
- blackbox-overview.json
- name: dashboard-crowdsec-overview
files:
- crowdsec-overview.json
- name: dashboard-disk-health
files:
- disk-health.json
@@ -65,6 +65,21 @@ spec:
- 45.157.177.26/32
- 2a03:4000:4b:c1e:4474:43ff:fedf:8f85/128
- 2a03:4000:28:f6e:a4c7:59ff:fe1e:9a29/128
# netbird.renner.uno — management, signal, relay and TURN.
#
# Not optional once a bouncer is armed. Its chain sits on the INPUT
# hook, the proxy hosts run no other firewall, and there is therefore
# no conntrack rule accepting established flows ahead of it. A ban on
# this address would drop the RETURN traffic of the proxies' own
# outbound control connection and take them off the mesh — with no
# inbound attack involved at all.
#
# Checked when this was added: none of these addresses appear in the
# current decision set. That is the state today, not a guarantee —
# these are rented addresses on a hoster whose ranges do end up on
# community blocklists.
- 162.55.50.97/32
- 2a01:4f8:c2c:4579::1/128
# mTLS between agents and the LAPI. Do not switch back to password auth:
# the LAPI runs without persistence, so its machine-registration DB is
@@ -108,6 +123,24 @@ spec:
lapi:
replicas: 1
service:
# Backend wiring for the public LAPI Ingress (lapi-ingress.yaml).
#
# These belong on the SERVICE, not on the Ingress: under the Kubernetes
# Ingress provider `traefik.ingress.kubernetes.io/service.*` are service
# annotations, and on an Ingress they are silently ignored. The symptom
# is Traefik speaking plaintext to a TLS-only backend, which the LAPI
# answers with "Client sent an HTTP request to an HTTPS server." and a
# 400 — an error that looks like a malformed request rather than a
# scheme mismatch.
#
# With tls.enabled the LAPI serves HTTPS on 8080 and offers no
# plaintext port, so https here is mandatory rather than a hardening
# choice. The ServersTransport carries the chart's own CA, since that
# certificate is self-signed.
annotations:
traefik.ingress.kubernetes.io/service.serversscheme: https
traefik.ingress.kubernetes.io/service.serverstransport: crowdsec-crowdsec-lapi@kubernetescrd
# Far below the chart default, which reserves half a core per LAPI for a
# workload that draws single-digit millicores — enough to leave the
# cluster unschedulable with one node out. Only the reservation shrinks;
@@ -44,8 +44,8 @@ metadata:
namespace: crowdsec
spec:
serverName: crowdsec-service.crowdsec.svc.cluster.local
rootCAsSecrets:
- crowdsec-root-secret
rootCAs:
- secret: crowdsec-root-secret
---
apiVersion: networking.k8s.io/v1
kind: Ingress
@@ -60,8 +60,12 @@ metadata:
cert-manager.io/cluster-issuer: letsencrypt-cluster
# NOT behind the Authentik gate: the client is a daemon with an API key,
# not a browser that can complete an OIDC flow.
traefik.ingress.kubernetes.io/service.serversscheme: https
traefik.ingress.kubernetes.io/service.serverstransport: crowdsec-crowdsec-lapi@kubernetescrd
#
# The backend scheme and ServersTransport are NOT set here. Under the
# Kubernetes Ingress provider, `traefik.ingress.kubernetes.io/service.*`
# are SERVICE annotations — on an Ingress they are silently ignored, and
# Traefik then talks plaintext to a TLS-only backend. They live on the
# chart's Service instead, via lapi.service.annotations in helmrelease.yaml.
spec:
ingressClassName: traefik
rules:
@@ -34,6 +34,17 @@ spec:
- ports:
- port: "8080"
protocol: TCP
# Home Assistant's llama.cpp conversation agent calls the chat model only —
# it has no use for the embedding endpoint, so unlike paperless above this
# grant does not extend to llama-embed.
- fromEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": home-assistant
app.kubernetes.io/name: home-assistant
toPorts:
- ports:
- port: "8080"
protocol: TCP
- fromEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": monitoring
@@ -167,6 +167,18 @@ spec:
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": influxdb
# The cluster's own chat model, for the llama.cpp conversation agent that
# backs Assist. Scoped to the one pod and port rather than the ai namespace
# as a whole, so this grant cannot widen if something else is placed there —
# the embedding endpoint next to it is deliberately not included.
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": ai
app.kubernetes.io/name: llama-server
toPorts:
- ports:
- port: "8080"
protocol: TCP
# traefik for east-west OIDC roundtrips that go through the
# public hostname via k8s_gateway.
- toEndpoints: