Compare commits

...
Author SHA1 Message Date
claude-botandClaude Opus 5 a88bebc76d fix(crowdsec): make Reloader watch the edge bouncer key too
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Found by the pr-review bot on #751, and it is the exact failure this design
was written to avoid.

BOUNCER_KEY_edge is consumed as an env var from a secretKeyRef, and env is
fixed at pod start. The Reloader annotation listed only crowdsec-bouncer-key,
so a rotation of the edge key would update OpenBao, sync into the Secret, and
reach the hosts through the Ansible play — while the LAPI kept the old value
in its environment and went on trusting the old key. The bouncers would then
present the new one and get 401s, at exactly the moment the rotation was
supposed to be seamless.

The comment directly above that annotation already stated the rule — "it's an
env-var secret, so a restart is required to pick up a rotated key". I added a
second env-var secret and did not extend the list.

Comma-separated, matching shelly-manager-deployment.yaml.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-02 13:08:16 +02:00
claude-botandClaude Opus 5 7897b50318 feat(crowdsec): vault the edge bouncer key for the proxy hosts
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Completes the pair: the ExternalSecret feeds BOUNCER_KEY_edge to the LAPI, this
feeds the same value to the bouncers that authenticate against it.

A single encrypted value rather than a whole-file vault like its siblings in
group_vars/all and group_vars/edge_host. Appending to one of those means
decrypting it first, which exposes every other secret in that file to add one
line. encrypt_string keeps the blast radius at this value, and the plaintext
never reaches a file: it goes from `bao kv get -field=key` through a pipe
straight into the encryption.

Scoped to netbird_proxy rather than all, because only those hosts run a
bouncer.

Not enrolled in the quarterly rotator, unlike crowdsec-bouncer-key. Nothing on
these hosts re-reads OpenBao on a schedule, so a rotation there would leave
them 401ing against the LAPI until someone ran the playbook. Rotating this one
means updating OpenBao and re-running plays/install-netbird-proxy.yml, in that
order.

Verified with `ansible-inventory --list`: the inventory parses and the value
decrypts.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-02 12:29:13 +02:00
claude-botandClaude Opus 5 22143c6a7c feat(crowdsec): firewall bouncer on the NetBird proxy hosts
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Enforces the decisions the in-cluster CrowdSec already detects, at the point
where public traffic actually enters. These hosts terminate TLS for every
*.renner.uno name, so a drop here never reaches the mesh, the relay or the
cluster. Detection stays where it is: the reverse-proxy produces no parseable
access log and has no collection, while Traefik has a working one, and every
scenario in use needs HTTP detail that does not exist at this layer.

Deployed in dry-run. It pulls decisions and logs what it would block without
creating a single nftables rule. The first bouncer to arm anywhere enforces
~24k CAPI decisions at once on the only external entry path, so the
false-positive window has to come before the enforcement rather than after it.
Arming is flipping crowdsec_bouncer_mode per host.

The static release tarball, not a package. The AUR package
crowdsec-firewall-bouncer-nftables depends on `crowdsec` and would pull the
whole Security Engine onto hosts that need only the bouncer; the group is
mixed (proxy-1 Arch, VPS-1 Debian); and the AUR would be a pattern this repo
does not otherwise have. The tarball is the same artefact the AUR package
downloads, pinned by version and sha256, and only the binary is unpacked —
the install/upgrade/uninstall scripts beside it manage systemd and config
themselves, which would leave two owners for the same files.

retry_initial_connect is set, and it is not cosmetic. Without it the process
exits when the LAPI is unreachable at startup, and since a clean shutdown
deletes the nftables tables, the systemd restart loop would leave the host
with no blocklist at all until the first successful connect. The flag exists
in the code but not in the shipped example config.

nftables_hooks covers input and forward. input is the traffic this exists for,
because the reverse-proxy is host-networked and owns :443. forward covers the
bridge-networked http-redirect sidecar, whose packets never traverse INPUT —
the same Docker bridge that showed up as 172.18.0.3 in the Traefik access log.

Prometheus is enabled but bound to loopback. These hosts are public and have
no metrics collection path today; opening :60601 to the internet to gain a
scrape target is the wrong trade.

No CA file: Traefik serves Let's Encrypt for crowdsec.renner.uno, and
ca_cert_path is added to the system pool rather than replacing it.

Rollback is `systemctl stop` — a clean stop deletes both tables by itself.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-02 12:19:09 +02:00
claude-botandClaude Opus 5 bc896e550c fix(crowdsec): drop the LAPI IPAllowList, it would have rejected the bouncer
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The allowlist carried the published proxy addresses on the assumption that a
request from a proxy host arrives carrying that host's public address. It does
not.

The client connects to the local proxy on 127.0.0.1, the reverse-proxy writes
what it saw into X-Forwarded-For, and Traefik resolves that into ClientHost.
Measured against netbird-proxy-watchdog, which makes exactly this kind of
request every round:

  172.18.0.3 - - [02/Sep/2026:10:12:01] "GET /" 200 websecure-immich-...
  172.18.0.3 - - [02/Sep/2026:10:12:01] "GET /" 200 websecure-home-...

Two probes one second apart, one per watchdog target. 172.18.0.3 is a Docker
bridge address on the proxy side.

So the allowlist would have 403ed the bouncer. Allowlisting 172.18.0.0/16
instead is not a control: it is a private range on a host whose addressing we
do not own, and it would admit anything else that happens to originate there.

The bouncer key stays as the access control, which is what it always was. The
endpoint serves read-only decision streams to a valid key; the admin surface
is cscli, in-cluster. If the source address ever becomes deterministic, an
ipAllowList can go back on top.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-02 12:13:27 +02:00
claude-botandClaude Opus 5 ea6e8e5083 feat(crowdsec): LAPI endpoint and key for the edge firewall bouncers
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Groundwork for enforcing CrowdSec decisions on proxy-1/proxy-2. Inert on its
own: an Ingress nobody calls yet, a bouncer key nobody uses yet, and a
whitelist that only ever narrows what may be banned.

The proxies cannot reach the LAPI over the mesh. The netbird-proxy container
runs non-root — that is what the ip_unprivileged_port_start task in the role
is for — so it has no NET_ADMIN and no /dev/net/tun, and its embedded NetBird
client terminates the tunnel in userspace. The tunnel belongs to that process,
not to the host netns, so a second process on the same host cannot route
through it. The alternative was a host-level NetBird client plus the LAPI on a
LoadBalancer VIP: two more peers, an ACL policy and a CNP rule. This reuses
the path every other service already takes.

Exposure is narrowed twice. The IPAllowList carries the three proxy addresses,
resolving them past the netbird-ingress pod IP through X-Forwarded-For the
same way the rate-limit middleware does — without that it would compare
against the shared forwarder IP, which is the gap behind the 2026-06-27 mass
ban. Behind it, the bouncer key. The LAPI serves read-only decision streams to
a valid key; the admin surface is cscli, in-cluster.

No CNP change is needed: crowdsec-ns.yaml already admits the traefik pods, and
ingress-default-deny carries no egress section.

BOUNCER_KEY_edge is separate from BOUNCER_KEY_ingress on purpose. The latter
rotates quarterly and both its consumers are in-cluster, so ESO propagates
within 10m. The edge hosts are Ansible-managed and read OpenBao on no
schedule, so a rotation would leave them 401ing until someone runs the
playbook. This key rotates with the playbook that carries it.

capi_whitelists.yaml is the only declarative allowlist the chart offers.
`cscli allowlists` objects live in the LAPI database, which runs without
persistence here deliberately, so anything created that way is gone at the
next restart. This file is a ConfigMap mount and survives by construction.
Upstream deprecates it; it is still the only thing that works against a LAPI
with no PVC.

It lists the proxy addresses and not the home WAN address, which is not
needed: unifi_nat_hairpin.tf DNATs the public proxy addresses to the Traefik
VIP on the UDM, so LAN clients reach the cluster directly with a LAN source
and never traverse the proxies.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-02 11:50:05 +02:00
k.renner bf2b520fc4 Merge pull request 'refactor(immich): finish the VectorChord move, restore the replica' (#747) from immich-drop-pgvectors into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#747
2026-09-02 07:44:49 +00:00
claude-botandClaude Opus 5 780401fe91 refactor(immich): finish the VectorChord move, restore the replica
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Immich's in-app migration to VectorChord is complete: both vector indexes,
face_index and clip_index, run on the vchordrq access method, and nothing uses
pgvecto.rs. The manifest still described a migration window.

`DROP EXTENSION vectors` was run against the immich database before this. It
was safe to do without CASCADE — nothing outside the extension depended on it,
which a rolled-back `BEGIN; DROP EXTENSION vectors; ROLLBACK;` confirmed
first. The embedding columns are `public.vector(512)`, which belongs to
pgvector, not to pgvecto.rs's type in the `vectors` schema. Afterwards both
vchordrq indexes are intact at 96 MB and 79 MB and the cluster stays healthy.

With the extension gone, vectors.so has nothing to serve and preloading it
only recreates pg_vectors in PGDATA. pgvector needs no preloading, so vchord.so
is the whole list.

instances goes back to 2. The blocker was never the extension as such — it was
that pg_basebackup's tar format cannot represent the long
pg_vectors/indexes/<uuid> paths pgvecto.rs creates, so a replica could not
join. That directory is now empty, 20 KiB in total, so the constraint is gone.

The image keeps its `:16-migration` tag. That is the upstream build's own name
for a multi-extension image, not a statement about this cluster, and it still
provides the two extensions in use. Moving to a different image is a separate
change with its own risk.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-02 09:41:48 +02:00
k.renner f9a9dc7ff8 Merge pull request 'docs: correct migration records that outlived their migrations' (#746) from fix-stale-migration-docs into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
Reviewed-on: k.renner/homelab#746
2026-09-02 06:48:52 +00:00
13 changed files with 423 additions and 13 deletions
+2
View File
@@ -7,6 +7,8 @@ when:
# install-netbird-proxy.yml also applies the netbird-proxy-watchdog role,
# so changes to it must redeploy this host too.
- "ansible/roles/netbird-proxy-watchdog/**"
# install-netbird-proxy.yml also applies the edge bouncer.
- "ansible/roles/crowdsec-edge-bouncer/**"
- "ansible/plays/install-netbird-proxy.yml"
- ".woodpecker/deploy-netbird-proxy.yml"
- event: deployment
@@ -0,0 +1,23 @@
---
# Bouncer key for crowdsec-edge-bouncer, scoped to the proxy hosts.
#
# A single encrypted VALUE rather than a whole-file vault like its siblings in
# group_vars/all and group_vars/edge_host. Deliberate: adding this variable to
# an existing whole-file vault means decrypting that file first, which exposes
# every other secret in it for the sake of appending one line. encrypt_string
# keeps the blast radius at this one value.
#
# Same value as OpenBao homelab/infrastructure/crowdsec-edge-bouncer-key, which
# the crowdsec-edge-bouncer-key ExternalSecret feeds to BOUNCER_KEY_edge on the
# LAPI. Both sides must move together — and unlike crowdsec-bouncer-key, this
# one is NOT in the quarterly rotator, because nothing on these hosts re-reads
# OpenBao on a schedule. Rotating it means updating OpenBao and re-running
# plays/install-netbird-proxy.yml.
vault_crowdsec_edge_bouncer_key: !vault |
$ANSIBLE_VAULT;1.1;AES256
32653764333536316163303838333839616562663862666637626165303837633866643033393239
3363376538656236643161386432326437633232643564630a633635306365323537346139626364
36613033343938363366306239363638383466396337623333623136313234656366333366623433
6437656632373364340a346566343866366433306533353431383064356362346536383136323663
39333864643534326433653233303137373162663066633132623931316135323733333863306330
6366623463666435636362376238313639666263323163303733
+4
View File
@@ -26,3 +26,7 @@
- docker-install
- netbird-proxy
- netbird-proxy-watchdog
# Enforces the cluster's CrowdSec decisions here, where public traffic
# enters. Deployed in dry-run: it logs what it would block and creates no
# nftables rules until crowdsec_bouncer_mode is flipped per host.
- crowdsec-edge-bouncer
@@ -0,0 +1,47 @@
---
# CrowdSec firewall bouncer on the NetBird proxy hosts.
#
# Enforces the decisions the in-cluster CrowdSec already detects, at the point
# where public traffic actually enters: these hosts terminate TLS for every
# *.renner.uno name. A drop here never reaches the mesh, the relay or the
# cluster.
#
# Detection stays in the cluster. The reverse-proxy produces no parseable
# access log and there is no CrowdSec collection for it, while Traefik has a
# working one — and every scenario in use needs HTTP detail (paths, status
# codes, user agents) that does not exist at this layer.
# Pinned. A release tarball URL is immutable, and this checksum is of the
# amd64 asset for exactly this tag.
crowdsec_bouncer_version: v0.0.36
crowdsec_bouncer_sha256: f86e4b72693549d99f40a9402abefb894108f047a3fbe6e72fada25ee17ce88b
# The LAPI over the public path these hosts already serve, see
# kubernetes/infrastructure/crowdsec/lapi-ingress.yaml. The proxies cannot
# reach it over the mesh: the reverse-proxy container runs non-root, so its
# embedded NetBird client terminates the tunnel in userspace and the tunnel
# belongs to that process rather than to the host netns.
#
# No ca_cert_path: Traefik serves a Let's Encrypt certificate for this name,
# and the bouncer adds ca_cert_path to the system pool rather than replacing
# it — the system pool already trusts LE.
crowdsec_bouncer_api_url: https://crowdsec.renner.uno/
crowdsec_bouncer_api_key: "{{ vault_crowdsec_edge_bouncer_key }}"
# STARTS INERT, and that is the point.
#
# dry-run pulls decisions and logs what it would block without creating a
# single nftables rule. The first bouncer to arm anywhere enforces ~24k CAPI
# decisions at once on the only external entry path, so the false-positive
# window has to come before the enforcement, not after it. Flip to `nftables`
# one host at a time once the log looks right.
crowdsec_bouncer_mode: dry-run
crowdsec_bouncer_update_frequency: 10s
crowdsec_bouncer_log_level: info
# Bound to loopback deliberately. These hosts are public and have no metrics
# collection path at all today — opening :60601 to the internet to gain a
# scrape target is the wrong trade. Revisit when host metrics exist.
crowdsec_bouncer_metrics_listen: 127.0.0.1
crowdsec_bouncer_metrics_port: 60601
@@ -0,0 +1,10 @@
---
# A restart drops both nftables tables and rebuilds them from the next
# successful LAPI pull, so there is a short window with no blocklist. That is
# the documented behaviour of a clean shutdown, not a defect — and it is why
# retry_initial_connect matters.
- name: Restart crowdsec firewall bouncer
ansible.builtin.systemd:
name: crowdsec-firewall-bouncer.service
state: restarted
daemon_reload: true
@@ -0,0 +1,84 @@
---
# See defaults/main.yml for why this exists and why it starts inert.
#
# The static release tarball, not a distribution package: the AUR package
# `crowdsec-firewall-bouncer-nftables` depends on `crowdsec` and would pull the
# whole Security Engine onto hosts that only need the bouncer, the group is
# mixed (proxy-1 Arch, VPS-1 Debian), and the AUR would be a new pattern this
# repo does not otherwise have. The tarball is the same artefact the AUR
# package downloads.
- name: Ensure nftables is present (the enforcing backend)
ansible.builtin.package:
name: nftables
state: present
- name: Ensure the bouncer config directory exists
ansible.builtin.file:
path: /etc/crowdsec/bouncers
state: directory
mode: "0755"
- name: Fetch the pinned bouncer release
ansible.builtin.get_url:
url: >-
https://github.com/crowdsecurity/cs-firewall-bouncer/releases/download/{{
crowdsec_bouncer_version }}/crowdsec-firewall-bouncer-linux-amd64.tgz
dest: /opt/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}.tgz
checksum: sha256:{{ crowdsec_bouncer_sha256 }}
mode: "0644"
# Only the binary. The tarball also carries install.sh / upgrade.sh /
# uninstall.sh, which manage systemd and config themselves — that is this
# role's job, and letting them run would leave two owners for the same files.
- name: Unpack the bouncer binary
ansible.builtin.unarchive:
src: /opt/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}.tgz
dest: /opt
remote_src: true
include:
- crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}/crowdsec-firewall-bouncer
creates: /opt/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}/crowdsec-firewall-bouncer
- name: Install the bouncer binary
ansible.builtin.copy:
src: /opt/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}/crowdsec-firewall-bouncer
dest: /usr/local/bin/crowdsec-firewall-bouncer
remote_src: true
mode: "0755"
notify: Restart crowdsec firewall bouncer
# 0600: this file carries the LAPI key.
- name: Deploy the bouncer configuration
ansible.builtin.template:
src: crowdsec-firewall-bouncer.yaml.j2
dest: /etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml
mode: "0600"
notify: Restart crowdsec firewall bouncer
- name: Deploy the bouncer systemd unit
ansible.builtin.template:
src: crowdsec-firewall-bouncer.service.j2
dest: /etc/systemd/system/crowdsec-firewall-bouncer.service
mode: "0644"
notify: Restart crowdsec firewall bouncer
- name: Enable and start the bouncer
ansible.builtin.systemd:
name: crowdsec-firewall-bouncer.service
enabled: true
state: started
daemon_reload: true
- name: Report how to read it
ansible.builtin.debug:
msg: |
crowdsec-firewall-bouncer deployed in mode "{{ crowdsec_bouncer_mode }}".
What it would block: journalctl -u crowdsec-firewall-bouncer -f
What it does block: nft list table ip crowdsec ; nft list table ip6 crowdsec6
Metrics: curl -s {{ crowdsec_bouncer_metrics_listen }}:{{ crowdsec_bouncer_metrics_port }}/metrics
Rollback: systemctl stop crowdsec-firewall-bouncer. A clean stop deletes
both nftables tables by itself. Only after a kill -9 do they need
removing by hand: nft delete table ip crowdsec ; nft delete table ip6 crowdsec6
@@ -0,0 +1,24 @@
# Managed by Ansible (crowdsec-edge-bouncer role) — do not edit.
#
# Upstream's shipped unit orders itself After=crowdsec.service. There is no
# local Security Engine on these hosts — the LAPI is in the cluster — so that
# dependency is dropped; nss-lookup instead, because api_url is a public name
# that has to resolve before the first connect.
[Unit]
Description=CrowdSec firewall bouncer (edge enforcement for the NetBird proxy)
After=network-online.target nss-lookup.target
Wants=network-online.target
[Service]
Type=notify
# -t validates the config and exits non-zero on a bad one, so a broken config
# fails the start instead of taking the running rules down with it.
ExecStartPre=/usr/local/bin/crowdsec-firewall-bouncer -c /etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml -t
ExecStart=/usr/local/bin/crowdsec-firewall-bouncer -c /etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml
Restart=always
RestartSec=10
LimitNOFILE=65536
KillMode=mixed
[Install]
WantedBy=multi-user.target
@@ -0,0 +1,55 @@
# Managed by Ansible (crowdsec-edge-bouncer role) — do not edit.
mode: {{ crowdsec_bouncer_mode }}
update_frequency: {{ crowdsec_bouncer_update_frequency }}
# stdout, so journald owns the log and there is no second rotation policy to
# keep in sync. The shipped example writes to /var/log; that only matters for
# hosts without journald.
log_mode: stdout
log_level: {{ crowdsec_bouncer_log_level }}
api_url: {{ crowdsec_bouncer_api_url }}
api_key: {{ crowdsec_bouncer_api_key }}
# Without this the process EXITS when the LAPI is unreachable at startup —
# there is no retry on the first connect. And because a clean shutdown deletes
# the nftables tables, the systemd restart loop would leave the host with no
# blocklist at all until the first successful connect. Present in the code,
# absent from the shipped example config.
retry_initial_connect: true
disable_ipv4: false
disable_ipv6: false
deny_action: DROP
supported_decisions_types:
- ban
# Separate tables per family; the bouncer never touches an existing table or
# chain, and never sets a chain policy. Priority -10 puts it ahead of anything
# at the default filter priority.
nftables:
ipv4:
enabled: true
set-only: false
table: crowdsec
chain: crowdsec-chain
priority: -10
ipv6:
enabled: true
set-only: false
table: crowdsec6
chain: crowdsec6-chain
priority: -10
# input is the traffic this exists for: the reverse-proxy is host-networked and
# owns :443, so its packets traverse INPUT. forward covers the bridge-networked
# http-redirect sidecar, whose packets never do.
nftables_hooks:
- input
- forward
prometheus:
enabled: true
listen_addr: {{ crowdsec_bouncer_metrics_listen }}
listen_port: {{ crowdsec_bouncer_metrics_port }}
+12 -10
View File
@@ -10,15 +10,15 @@ metadata:
# Kept permanently as a standing guard.
kustomize.toolkit.fluxcd.io/prune: disabled
spec:
# Single instance during pgvecto.rs → VectorChord migration window;
# pg_basebackup tar format chokes on the long pg_vectors/indexes/<uuid>
# paths, blocking replica join. Bump back to 2 once Immich's in-app
# vchord migration drops the vectors extension.
instances: 1
# See ak-pg cluster.yaml for rationale. Single-instance, so no failover, but
# the barman-plugin shutdown wedge (cloudnative-pg#9770) takes the whole DB
# down until SIGKILL — bounding terminationGracePeriod (stopDelay) matters
# even without a standby. smartShutdownTimeout keeps pod restarts quick.
# A replica is viable because nothing here writes pg_vectors any more. Both
# vector indexes run on vchordrq and the vectors extension is gone, so the
# long pg_vectors/indexes/<uuid> paths are absent — pg_basebackup's tar
# format cannot represent them, and that is what blocks a replica join.
instances: 2
# See ak-pg cluster.yaml for rationale. The barman-plugin shutdown wedge
# (cloudnative-pg#9770) holds an instance up until SIGKILL, so bounding
# terminationGracePeriod (stopDelay) is what keeps a failover from waiting
# on it. smartShutdownTimeout keeps pod restarts quick.
smartShutdownTimeout: 10
stopDelay: 60
imageName: ghcr.io/corentingiraud/cnpg-pgvector-vectorchord:16-migration
@@ -36,8 +36,10 @@ spec:
database: immich
owner: immich
postgresql:
# vchord only. The image also ships pgvecto.rs (vectors.so), but its
# extension is not installed and preloading it would recreate pg_vectors
# in PGDATA for nothing. pgvector needs no preloading.
shared_preload_libraries:
- vectors.so
- vchord.so
parameters:
shared_buffers: "512MB"
@@ -0,0 +1,27 @@
# Bouncer key for the firewall bouncers on proxy-1/proxy-2.
#
# Separate from crowdsec-bouncer-key ON PURPOSE. That key is rotated
# quarterly by kubernetes/infrastructure/openbao/rotation/crowdsec-bouncer-key.yaml,
# and both of its consumers live in the cluster, so ESO propagates a rotation
# within 10m. The edge bouncers are Ansible-managed hosts: nothing there
# re-reads OpenBao on a schedule, so a rotated key would leave them 401ing
# against the LAPI until someone runs the playbook. This key is therefore
# deliberately NOT enrolled in the rotator — it rotates when the playbook runs,
# and the playbook is what carries the new value to the hosts.
#
# The value must match ansible vault_crowdsec_edge_bouncer_key.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: crowdsec-edge-bouncer-key
namespace: crowdsec
spec:
refreshInterval: 10m
secretStoreRef:
kind: ClusterSecretStore
name: openbao
target:
name: crowdsec-edge-bouncer-key
dataFrom:
- extract:
key: infrastructure/crowdsec-edge-bouncer-key
@@ -36,6 +36,36 @@ spec:
values:
container_runtime: containerd
# Addresses a community blocklist must never be allowed to ban.
#
# This is the ONLY declarative allowlist the chart offers. `cscli
# allowlists` objects live in the LAPI database, and that database runs
# without persistence here on purpose (see lapi.persistentVolume below),
# so anything created through cscli is gone at the next restart. This file
# is a ConfigMap mount, so it survives by construction. Upstream marks it
# deprecated in favour of centralized allowlists; it is still the only
# thing that works against a LAPI with no PVC.
#
# Scope: CAPI decisions only. Locally generated decisions are not filtered
# here — they currently carry real client IPs (verified), because Traefik
# resolves X-Forwarded-For past the netbird-ingress pod IP.
#
# The home WAN address is deliberately absent and is not needed: LAN
# clients never present it. opentofu/unifi_nat_hairpin.tf DNATs the public
# proxy addresses to the Traefik VIP on the UDM, so on-LAN traffic reaches
# the cluster directly with a LAN source and never traverses the proxies.
# Private ranges cannot appear in a community blocklist either way.
config:
capi_whitelists.yaml: |
cidrs:
# The proxy hosts themselves. Same three addresses as the
# netbird-ingress-egress toCIDRSet and the hairpin endpoints —
# keep all three lists in sync. proxy-1 is the v6-only member,
# proxy-2 the dual-stack one.
- 45.157.177.26/32
- 2a03:4000:4b:c1e:4474:43ff:fedf:8f85/128
- 2a03:4000:28:f6e:a4c7:59ff:fe1e:9a29/128
# mTLS between agents and the LAPI. Do not switch back to password auth:
# the LAPI runs without persistence, so its machine-registration DB is
# wiped on every restart, and password-registered agents then get 401
@@ -86,10 +116,18 @@ spec:
requests:
cpu: 50m
podAnnotations:
# Reloader restarts the LAPI when crowdsec-bouncer-key changes (it's an
# env-var secret, so a restart is required to pick up a rotated key). See
# Reloader restarts the LAPI when either bouncer key changes. Both are
# consumed as env vars from a secretKeyRef, and env is fixed at pod
# start — without a restart the LAPI keeps trusting the old value while
# the bouncer already presents the new one, which is a 401 at exactly
# the moment a rotation is supposed to be seamless.
#
# crowdsec-bouncer-key rotates on a schedule, see
# kubernetes/infrastructure/openbao/rotation/crowdsec-bouncer-key.yaml.
secret.reloader.stakater.com/reload: crowdsec-bouncer-key
# crowdsec-edge-bouncer-key rotates by hand together with
# ansible/plays/install-netbird-proxy.yml; it still needs to be listed
# here, because the LAPI side of that rotation is this env var.
secret.reloader.stakater.com/reload: crowdsec-bouncer-key,crowdsec-edge-bouncer-key
persistentVolume:
data:
# Kept OFF deliberately. The #337 attempt to set this `true` (persist
@@ -118,6 +156,17 @@ spec:
secretKeyRef:
name: crowdsec-bouncer-key
key: key
# The firewall bouncers on proxy-1/proxy-2, over the public LAPI
# endpoint (lapi-ingress.yaml). The chart's entrypoint re-runs
# `register_bouncer` for every BOUNCER_KEY_* on each start and skips
# names that already exist, so this survives the LAPI's wiped
# registration DB the same way BOUNCER_KEY_ingress does — the env var
# is the source of truth, not the database.
- name: BOUNCER_KEY_edge
valueFrom:
secretKeyRef:
name: crowdsec-edge-bouncer-key
key: key
agent:
# Same reasoning as lapi.resources above, multiplied by three: this is a
@@ -2,4 +2,6 @@ apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- bouncer-key-externalsecret.yaml
- edge-bouncer-key-externalsecret.yaml
- lapi-ingress.yaml
- helmrelease.yaml
@@ -0,0 +1,81 @@
# Public LAPI endpoint for the firewall bouncers on proxy-1/proxy-2.
#
# WHY PUBLIC AT ALL. The bouncers run as host processes on the proxies, and
# the proxies have no mesh access: the netbird-proxy container runs non-root
# (see the ip_unprivileged_port_start task in the netbird-proxy role), so it
# has no NET_ADMIN and no /dev/net/tun, and its embedded NetBird client
# therefore terminates the tunnel in userspace. The tunnel belongs to that
# process, not to the host netns — a second process on the same host cannot
# route through it. The alternative was a host-level NetBird client plus the
# LAPI on a LoadBalancer VIP; this reuses the path every other service already
# takes instead of adding two peers and an ACL policy.
#
# NO IP ALLOWLIST, deliberately. The bouncer key is the access control.
#
# An allowlist was the obvious second factor and it does not work here. A
# request originating on a proxy host does not arrive carrying that host's
# public address: the client connects to the local proxy on 127.0.0.1, the
# reverse-proxy puts what it saw into X-Forwarded-For, and Traefik resolves
# that. Measured against the netbird-proxy-watchdog, which makes exactly this
# kind of request every round:
#
# 172.18.0.3 - - [02/Sep/2026:10:12:01] "GET /" 200 websecure-immich-...
# 172.18.0.3 - - [02/Sep/2026:10:12:01] "GET /" 200 websecure-home-...
#
# 172.18.0.3 is a Docker bridge address on the proxy side. Allowlisting the
# published proxy addresses would have rejected the bouncer with 403;
# allowlisting 172.18.0.0/16 would be a private range on a host we do not
# control the addressing of, which is not a security control.
#
# What remains is the API key, which is the real control anyway: this endpoint
# serves read-only decision streams to a valid bouncer key and nothing else.
# The admin surface is cscli, in-cluster. If the source address ever becomes
# deterministic, an ipAllowList can be added on top.
---
# The LAPI serves HTTPS only once tls.enabled is set — there is no plaintext
# fallback and no second port; 8080 is the TLS port. Its certificate comes
# from the chart's own self-signed CA, so Traefik needs that CA to verify the
# backend. SANs include crowdsec-service.crowdsec.svc.cluster.local, which is
# what the Service resolves to.
apiVersion: traefik.io/v1alpha1
kind: ServersTransport
metadata:
name: crowdsec-lapi
namespace: crowdsec
spec:
serverName: crowdsec-service.crowdsec.svc.cluster.local
rootCAsSecrets:
- crowdsec-root-secret
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: crowdsec-lapi
namespace: crowdsec
labels:
# Public DNS via Cloudflare; the proxies resolve it from outside. The UDM
# external-dns instance skips it, LAN reaches it via the hairpin NAT.
dns.renner.uno/public: "true"
annotations:
cert-manager.io/cluster-issuer: letsencrypt-cluster
# NOT behind the Authentik gate: the client is a daemon with an API key,
# not a browser that can complete an OIDC flow.
traefik.ingress.kubernetes.io/service.serversscheme: https
traefik.ingress.kubernetes.io/service.serverstransport: crowdsec-crowdsec-lapi@kubernetescrd
spec:
ingressClassName: traefik
rules:
- host: crowdsec.renner.uno
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: crowdsec-service
port:
number: 8080
tls:
- secretName: crowdsec-lapi-public-tls
hosts:
- crowdsec.renner.uno