fix(ci): deploy the proxies when their inventory changes, not just their roles

Arming the bouncer on proxy-2 (#766) merged and deployed nothing. The switch
lives in ansible/inventory/host_vars/proxy-2.renner.uno.yml, and the trigger
list covered ansible/roles/** and the play but no inventory path at all. The
change sat in git looking applied while the host kept running dry-run, and
nothing reported a problem — there was simply no pipeline run to fail.

The bouncer's LAPI key has the same shape: it lives in
group_vars/netbird_proxy/vault.yml, so a rotation would never have reached the
hosts either.

Both paths are now listed. pve_hosts.yml deliberately is not: it is the single
inventory file for the whole fleet, so listing it would redeploy the proxies
for any unrelated host edit. The cost is that a change to [netbird_proxy] group
membership needs a manual deployment run.

Merging this triggers the run itself, since .woodpecker/deploy-netbird-proxy.yml
is in its own trigger list — so the arming from #766 applies with it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

.woodpecker/deploy-netbird-proxy.yml

@@ -7,7 +7,21 @@ when:
       # install-netbird-proxy.yml also applies the netbird-proxy-watchdog role,
       # so changes to it must redeploy this host too.
       - "ansible/roles/netbird-proxy-watchdog/**"
+      # install-netbird-proxy.yml also applies the edge bouncer.
+      - "ansible/roles/crowdsec-edge-bouncer/**"
       - "ansible/plays/install-netbird-proxy.yml"
+      # The variables that steer those roles, not just the roles themselves.
+      # Without these two, a change that only touches inventory never deploys
+      # and nothing says so: arming the bouncer lives in host_vars, and its
+      # LAPI key lives in group_vars, so both would sit in git looking applied
+      # while the hosts kept running the previous configuration.
+      #
+      # pve_hosts.yml is deliberately NOT listed. It is the single inventory
+      # file for the whole fleet, so listing it would redeploy the proxies for
+      # any unrelated host edit. The cost is that a change to [netbird_proxy]
+      # group membership needs a manual deployment run.
+      - "ansible/inventory/host_vars/proxy-*.yml"
+      - "ansible/inventory/group_vars/netbird_proxy/**"
       - ".woodpecker/deploy-netbird-proxy.yml"
   - event: deployment
     evaluate: 'CI_PIPELINE_DEPLOY_TARGET == "deploy-netbird-proxy"'

ansible/inventory/group_vars/netbird_proxy/vault.yml

@@ -0,0 +1,23 @@
+---
+# Bouncer key for crowdsec-edge-bouncer, scoped to the proxy hosts.
+#
+# A single encrypted VALUE rather than a whole-file vault like its siblings in
+# group_vars/all and group_vars/edge_host. Deliberate: adding this variable to
+# an existing whole-file vault means decrypting that file first, which exposes
+# every other secret in it for the sake of appending one line. encrypt_string
+# keeps the blast radius at this one value.
+#
+# Same value as OpenBao homelab/infrastructure/crowdsec-edge-bouncer-key, which
+# the crowdsec-edge-bouncer-key ExternalSecret feeds to BOUNCER_KEY_edge on the
+# LAPI. Both sides must move together — and unlike crowdsec-bouncer-key, this
+# one is NOT in the quarterly rotator, because nothing on these hosts re-reads
+# OpenBao on a schedule. Rotating it means updating OpenBao and re-running
+# plays/install-netbird-proxy.yml.
+vault_crowdsec_edge_bouncer_key: !vault |
+          $ANSIBLE_VAULT;1.1;AES256
+          32653764333536316163303838333839616562663862666637626165303837633866643033393239
+          3363376538656236643161386432326437633232643564630a633635306365323537346139626364
+          36613033343938363366306239363638383466396337623333623136313234656366333366623433
+          6437656632373364340a346566343866366433306533353431383064356362346536383136323663
+          39333864643534326433653233303137373162663066633132623931316135323733333863306330
+          6366623463666435636362376238313639666263323163303733

ansible/inventory/host_vars/proxy-2.renner.uno.yml

@@ -0,0 +1,19 @@
+---
+# Arms the CrowdSec firewall bouncer on THIS host only.
+#
+# The role defaults to dry-run for the whole netbird_proxy group, so the two
+# members can be armed one at a time. proxy-2 goes first deliberately: it is
+# the dual-stack member, while proxy-1 is IPv6-only and the one where every
+# network-shaped problem has surfaced first — the dhcpcd/resolv.conf breakage,
+# and the GitHub download that could not work at all.
+#
+# What flipping this does: the bouncer stops logging what it would block and
+# creates two nftables tables (ip crowdsec, ip6 crowdsec6) with the decisions
+# it holds. It touches no existing chain and sets no chain policy.
+#
+# Rollback is `systemctl stop crowdsec-firewall-bouncer` — a clean stop deletes
+# both tables itself. Setting this back to dry-run and re-running the play does
+# the same through the handler.
+#
+# Remove this file once proxy-1 follows and the role default moves to nftables.
+crowdsec_bouncer_mode: nftables

ansible/plays/install-netbird-proxy.yml

@@ -26,3 +26,7 @@
     - docker-install
     - netbird-proxy
     - netbird-proxy-watchdog
+    # Enforces the cluster's CrowdSec decisions here, where public traffic
+    # enters. Deployed in dry-run: it logs what it would block and creates no
+    # nftables rules until crowdsec_bouncer_mode is flipped per host.
+    - crowdsec-edge-bouncer

ansible/roles/crowdsec-edge-bouncer/defaults/main.yml

@@ -0,0 +1,47 @@
+---
+# CrowdSec firewall bouncer on the NetBird proxy hosts.
+#
+# Enforces the decisions the in-cluster CrowdSec already detects, at the point
+# where public traffic actually enters: these hosts terminate TLS for every
+# *.renner.uno name. A drop here never reaches the mesh, the relay or the
+# cluster.
+#
+# Detection stays in the cluster. The reverse-proxy produces no parseable
+# access log and there is no CrowdSec collection for it, while Traefik has a
+# working one — and every scenario in use needs HTTP detail (paths, status
+# codes, user agents) that does not exist at this layer.
+
+# Pinned. A release tarball URL is immutable, and this checksum is of the
+# amd64 asset for exactly this tag.
+crowdsec_bouncer_version: v0.0.36
+crowdsec_bouncer_sha256: f86e4b72693549d99f40a9402abefb894108f047a3fbe6e72fada25ee17ce88b
+
+# The LAPI over the public path these hosts already serve, see
+# kubernetes/infrastructure/crowdsec/lapi-ingress.yaml. The proxies cannot
+# reach it over the mesh: the reverse-proxy container runs non-root, so its
+# embedded NetBird client terminates the tunnel in userspace and the tunnel
+# belongs to that process rather than to the host netns.
+#
+# No ca_cert_path: Traefik serves a Let's Encrypt certificate for this name,
+# and the bouncer adds ca_cert_path to the system pool rather than replacing
+# it — the system pool already trusts LE.
+crowdsec_bouncer_api_url: https://crowdsec.renner.uno/
+crowdsec_bouncer_api_key: "{{ vault_crowdsec_edge_bouncer_key }}"
+
+# STARTS INERT, and that is the point.
+#
+# dry-run pulls decisions and logs what it would block without creating a
+# single nftables rule. The first bouncer to arm anywhere enforces ~24k CAPI
+# decisions at once on the only external entry path, so the false-positive
+# window has to come before the enforcement, not after it. Flip to `nftables`
+# one host at a time once the log looks right.
+crowdsec_bouncer_mode: dry-run
+
+crowdsec_bouncer_update_frequency: 10s
+crowdsec_bouncer_log_level: info
+
+# Bound to loopback deliberately. These hosts are public and have no metrics
+# collection path at all today — opening :60601 to the internet to gain a
+# scrape target is the wrong trade. Revisit when host metrics exist.
+crowdsec_bouncer_metrics_listen: 127.0.0.1
+crowdsec_bouncer_metrics_port: 60601

ansible/roles/crowdsec-edge-bouncer/handlers/main.yml

@@ -0,0 +1,10 @@
+---
+# A restart drops both nftables tables and rebuilds them from the next
+# successful LAPI pull, so there is a short window with no blocklist. That is
+# the documented behaviour of a clean shutdown, not a defect — and it is why
+# retry_initial_connect matters.
+- name: Restart crowdsec firewall bouncer
+  ansible.builtin.systemd:
+    name: crowdsec-firewall-bouncer.service
+    state: restarted
+    daemon_reload: true

ansible/roles/crowdsec-edge-bouncer/tasks/main.yml

@@ -0,0 +1,111 @@
+---
+# See defaults/main.yml for why this exists and why it starts inert.
+#
+# The static release tarball, not a distribution package: the AUR package
+# `crowdsec-firewall-bouncer-nftables` depends on `crowdsec` and would pull the
+# whole Security Engine onto hosts that only need the bouncer, the group is
+# mixed (proxy-1 Arch, VPS-1 Debian), and the AUR would be a new pattern this
+# repo does not otherwise have. The tarball is the same artefact the AUR
+# package downloads.
+
+- name: Ensure nftables is present (the enforcing backend)
+  ansible.builtin.package:
+    name: nftables
+    state: present
+
+- name: Ensure the bouncer config directory exists
+  ansible.builtin.file:
+    path: /etc/crowdsec/bouncers
+    state: directory
+    mode: "0755"
+
+# Fetched on the CONTROLLER and pushed, not pulled by the host.
+#
+# proxy-1 is IPv6-only, and neither github.com nor objects.githubusercontent.com
+# publishes an AAAA record — a get_url on the host fails with "Network is
+# unreachable" and no amount of retrying fixes it. Mirroring into
+# gitea.renner.uno is no help either: its public records are the internal LB
+# VIP (10.2.99.86 / fd99::86), so it is unreachable from outside the LAN. And
+# upstream ships no container image for this bouncer, which is expected —
+# it manipulates host netfilter.
+#
+# The CI runner has v4, so it does the fetching. The checksum is verified
+# there, before anything is copied to a host.
+- name: Fetch the pinned bouncer release on the controller
+  ansible.builtin.get_url:
+    url: >-
+      https://github.com/crowdsecurity/cs-firewall-bouncer/releases/download/{{
+      crowdsec_bouncer_version }}/crowdsec-firewall-bouncer-linux-amd64.tgz
+    dest: /tmp/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}.tgz
+    checksum: sha256:{{ crowdsec_bouncer_sha256 }}
+    mode: "0644"
+  delegate_to: localhost
+  become: false
+  run_once: true
+
+# Only the download had to move to the controller — the extraction stays on
+# the host. The ci-ansible image ships a tar that ansible rejects ("detected as
+# tar type None. GNU tar required"), while both host distributions have GNU
+# tar. Shipping the archive rather than the unpacked binary also keeps the
+# artefact that the checksum was verified against intact all the way to the
+# host.
+- name: Copy the bouncer archive to the host
+  ansible.builtin.copy:
+    src: /tmp/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}.tgz
+    dest: /opt/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}.tgz
+    mode: "0644"
+
+# Only the binary. The tarball also carries install.sh / upgrade.sh /
+# uninstall.sh, which manage systemd and config themselves — that is this
+# role's job, and letting them run would leave two owners for the same files.
+- name: Unpack the bouncer binary
+  ansible.builtin.unarchive:
+    src: /opt/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}.tgz
+    dest: /opt
+    remote_src: true
+    include:
+      - crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}/crowdsec-firewall-bouncer
+    creates: /opt/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}/crowdsec-firewall-bouncer
+
+- name: Install the bouncer binary
+  ansible.builtin.copy:
+    src: /opt/crowdsec-firewall-bouncer-{{ crowdsec_bouncer_version }}/crowdsec-firewall-bouncer
+    dest: /usr/local/bin/crowdsec-firewall-bouncer
+    remote_src: true
+    mode: "0755"
+  notify: Restart crowdsec firewall bouncer
+
+# 0600: this file carries the LAPI key.
+- name: Deploy the bouncer configuration
+  ansible.builtin.template:
+    src: crowdsec-firewall-bouncer.yaml.j2
+    dest: /etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml
+    mode: "0600"
+  notify: Restart crowdsec firewall bouncer
+
+- name: Deploy the bouncer systemd unit
+  ansible.builtin.template:
+    src: crowdsec-firewall-bouncer.service.j2
+    dest: /etc/systemd/system/crowdsec-firewall-bouncer.service
+    mode: "0644"
+  notify: Restart crowdsec firewall bouncer
+
+- name: Enable and start the bouncer
+  ansible.builtin.systemd:
+    name: crowdsec-firewall-bouncer.service
+    enabled: true
+    state: started
+    daemon_reload: true
+
+- name: Report how to read it
+  ansible.builtin.debug:
+    msg: |
+      crowdsec-firewall-bouncer deployed in mode "{{ crowdsec_bouncer_mode }}".
+
+      What it would block:  journalctl -u crowdsec-firewall-bouncer -f
+      What it does block:   nft list table ip crowdsec ; nft list table ip6 crowdsec6
+      Metrics:              curl -s {{ crowdsec_bouncer_metrics_listen }}:{{ crowdsec_bouncer_metrics_port }}/metrics
+
+      Rollback: systemctl stop crowdsec-firewall-bouncer. A clean stop deletes
+      both nftables tables by itself. Only after a kill -9 do they need
+      removing by hand: nft delete table ip crowdsec ; nft delete table ip6 crowdsec6

ansible/roles/crowdsec-edge-bouncer/templates/crowdsec-firewall-bouncer.service.j2

@@ -0,0 +1,24 @@
+# Managed by Ansible (crowdsec-edge-bouncer role) — do not edit.
+#
+# Upstream's shipped unit orders itself After=crowdsec.service. There is no
+# local Security Engine on these hosts — the LAPI is in the cluster — so that
+# dependency is dropped; nss-lookup instead, because api_url is a public name
+# that has to resolve before the first connect.
+[Unit]
+Description=CrowdSec firewall bouncer (edge enforcement for the NetBird proxy)
+After=network-online.target nss-lookup.target
+Wants=network-online.target
+
+[Service]
+Type=notify
+# -t validates the config and exits non-zero on a bad one, so a broken config
+# fails the start instead of taking the running rules down with it.
+ExecStartPre=/usr/local/bin/crowdsec-firewall-bouncer -c /etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml -t
+ExecStart=/usr/local/bin/crowdsec-firewall-bouncer -c /etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml
+Restart=always
+RestartSec=10
+LimitNOFILE=65536
+KillMode=mixed
+
+[Install]
+WantedBy=multi-user.target

ansible/roles/crowdsec-edge-bouncer/templates/crowdsec-firewall-bouncer.yaml.j2

@@ -0,0 +1,55 @@
+# Managed by Ansible (crowdsec-edge-bouncer role) — do not edit.
+mode: {{ crowdsec_bouncer_mode }}
+update_frequency: {{ crowdsec_bouncer_update_frequency }}
+
+# stdout, so journald owns the log and there is no second rotation policy to
+# keep in sync. The shipped example writes to /var/log; that only matters for
+# hosts without journald.
+log_mode: stdout
+log_level: {{ crowdsec_bouncer_log_level }}
+
+api_url: {{ crowdsec_bouncer_api_url }}
+api_key: {{ crowdsec_bouncer_api_key }}
+
+# Without this the process EXITS when the LAPI is unreachable at startup —
+# there is no retry on the first connect. And because a clean shutdown deletes
+# the nftables tables, the systemd restart loop would leave the host with no
+# blocklist at all until the first successful connect. Present in the code,
+# absent from the shipped example config.
+retry_initial_connect: true
+
+disable_ipv4: false
+disable_ipv6: false
+deny_action: DROP
+
+supported_decisions_types:
+  - ban
+
+# Separate tables per family; the bouncer never touches an existing table or
+# chain, and never sets a chain policy. Priority -10 puts it ahead of anything
+# at the default filter priority.
+nftables:
+  ipv4:
+    enabled: true
+    set-only: false
+    table: crowdsec
+    chain: crowdsec-chain
+    priority: -10
+  ipv6:
+    enabled: true
+    set-only: false
+    table: crowdsec6
+    chain: crowdsec6-chain
+    priority: -10
+
+# input is the traffic this exists for: the reverse-proxy is host-networked and
+# owns :443, so its packets traverse INPUT. forward covers the bridge-networked
+# http-redirect sidecar, whose packets never do.
+nftables_hooks:
+  - input
+  - forward
+
+prometheus:
+  enabled: true
+  listen_addr: {{ crowdsec_bouncer_metrics_listen }}
+  listen_port: {{ crowdsec_bouncer_metrics_port }}

kubernetes/RUNBOOK-baremetal-conversion.md

@@ -2,7 +2,12 @@
 
 Converts one MS-01 from "Proxmox host running a Talos VM" to "Talos on bare
 metal", and swaps its Ceph OSD onto a Samsung PM983 at the same time. Written
-after doing jupiter on 2026-08-25; saturn and neptune remain.
+after doing jupiter on 2026-08-25.
+
+All three nodes are now on bare metal — `talosctl get systeminformation`
+reports Minisforum MS-01 hardware with distinct chassis serials for k8s-1,
+k8s-2 and k8s-3, with no hypervisor in between. Kept as the procedure for a
+rebuild, not as outstanding work.
 
 Everything here that reads like an over-specific warning is something that
 actually happened. The conversion itself is straightforward — the failures were

kubernetes/apps/home-assistant/MIGRATION-recorder-postgres.md

@@ -111,9 +111,23 @@ statistics             max_id=13323904    max_start_ts=1787162400.0
 statistics_short_term  max_id=159205762   max_start_ts=1787166000.0
 ```
 
-## Phase 2c — cutover (REMAINING)
+## Phase 2c — cutover: DONE
 
-Do not start until phase 1 is verified.
+Verified 2026-09-02 against the live database:
+
+| table | rows | span |
+|---|---|---|
+| `statistics` | 11,092,586 | 2024-03-13 20:00 → current |
+| `statistics_short_term` | 1,883,709 | rolling ~7 days |
+| `statistics_meta` | 1,480 | |
+
+The 2024-03-13 origin is the proof that matters. It is unchanged from the
+pre-migration count in "What is at risk" above, so the history was copied
+across engines rather than started from zero — which is the failure this
+document exists to prevent. `states` carries current rows, so `db_url` points
+at Postgres.
+
+The procedure below is kept as the record of what was done.
 
 **Migrate statistics only.** Copying all 18M `states` rows is slow, needs a long
 HA outage, and buys a table that is discarded within 7 days anyway.

kubernetes/apps/home-assistant/deployment.yaml

@@ -56,10 +56,9 @@ spec:
         # Re-rendered on every pod start, so a CNPG password rotation takes
         # effect rather than silently leaving a stale URI behind.
         #
-        # INERT until the recorder is repointed: nothing reads this key until
-        # `db_url: !secret recorder_db_url` lands in configuration.yaml in the
-        # homeassistant repo. That ordering is required — HA refuses to start
-        # when a referenced secret is missing.
+        # Load-bearing: `db_url: !secret recorder_db_url` in the homeassistant
+        # repo's configuration.yaml reads this key, so HA does not start
+        # without it. Removing this init container takes the recorder down.
         - name: render-db-url
           image: busybox:1.36
           command:

kubernetes/apps/home-assistant/ha-pg-cluster.yaml

@@ -18,12 +18,12 @@
 # That is the point of the move — it attacks the multiplier rather than trimming
 # what gets recorded, which would only hide the problem.
 #
-# STAGING
-# Phase 1 (this manifest): the cluster comes up EMPTY and HA keeps using SQLite.
-# Nothing depends on it, so a failure here is inert. Verify Ready, WAL archiving
-# to Hetzner, and a successful base backup before going further.
-# Phase 2: migrate the statistics tables and repoint recorder.db_url.
-# See MIGRATION-recorder-postgres.md for the sequence and the rollback.
+# This is the live recorder database. HA writes here; the SQLite file is no
+# longer the system of record. The 2.4 years of long-term statistics were
+# copied across engines rather than regenerated — HA does not migrate data
+# between database engines, so an empty Postgres would have silently started
+# from zero and taken the energy dashboard's history with it.
+# MIGRATION-recorder-postgres.md holds the sequence that was followed.
 apiVersion: postgresql.cnpg.io/v1
 kind: Cluster
 metadata:

kubernetes/apps/home-assistant/kustomization.yaml

@@ -60,9 +60,8 @@ resources:
   - zigbee2mqtt-gate-service.yaml
   - zigbee2mqtt-ingress.yaml
   # Postgres backend for the recorder (MIGRATION-recorder-postgres.md).
-  # PHASE 1: this cluster comes up EMPTY and nothing consumes it — HA is still
-  # on SQLite. Deploying it is therefore inert; phase 2 migrates the statistics
-  # tables and repoints recorder.db_url in the homeassistant repo.
+  # Live: recorder.db_url in the homeassistant repo points here, and the
+  # long-term statistics carry their full history from 2024.
   - ha-pg-hetzner-creds-externalsecret.yaml
   - ha-pg-objectstore.yaml
   - ha-pg-cluster.yaml

kubernetes/apps/immich/cluster.yaml

@@ -10,15 +10,15 @@ metadata:
     # Kept permanently as a standing guard.
     kustomize.toolkit.fluxcd.io/prune: disabled
 spec:
-  # Single instance during pgvecto.rs → VectorChord migration window;
-  # pg_basebackup tar format chokes on the long pg_vectors/indexes/<uuid>
-  # paths, blocking replica join. Bump back to 2 once Immich's in-app
-  # vchord migration drops the vectors extension.
-  instances: 1
-  # See ak-pg cluster.yaml for rationale. Single-instance, so no failover, but
-  # the barman-plugin shutdown wedge (cloudnative-pg#9770) takes the whole DB
-  # down until SIGKILL — bounding terminationGracePeriod (stopDelay) matters
-  # even without a standby. smartShutdownTimeout keeps pod restarts quick.
+  # A replica is viable because nothing here writes pg_vectors any more. Both
+  # vector indexes run on vchordrq and the vectors extension is gone, so the
+  # long pg_vectors/indexes/<uuid> paths are absent — pg_basebackup's tar
+  # format cannot represent them, and that is what blocks a replica join.
+  instances: 2
+  # See ak-pg cluster.yaml for rationale. The barman-plugin shutdown wedge
+  # (cloudnative-pg#9770) holds an instance up until SIGKILL, so bounding
+  # terminationGracePeriod (stopDelay) is what keeps a failover from waiting
+  # on it. smartShutdownTimeout keeps pod restarts quick.
   smartShutdownTimeout: 10
   stopDelay: 60
   imageName: ghcr.io/corentingiraud/cnpg-pgvector-vectorchord:16-migration
@@ -36,8 +36,10 @@ spec:
       database: immich
       owner: immich
   postgresql:
+    # vchord only. The image also ships pgvecto.rs (vectors.so), but its
+    # extension is not installed and preloading it would recreate pg_vectors
+    # in PGDATA for nothing. pgvector needs no preloading.
     shared_preload_libraries:
-      - vectors.so
       - vchord.so
     parameters:
       shared_buffers: "512MB"

kubernetes/apps/llama-server/deployment.yaml

@@ -75,11 +75,27 @@ spec:
             # many small per-expert matmuls that sets the pace, not bandwidth.
             - --threads
             - "4"
-            # 16k rather than 8k because the measured prefill falloff is mild
-            # and this model's KV cache is unusually cheap — only 10 of its 40
-            # layers use full attention.
+            # 32k because Home Assistant's Assist agent measured 18910 tokens
+            # for a single request: ~12k of exposed-entity overview plus ~7k of
+            # base instructions and tool schemas, whose parameters enumerate
+            # every exposed name, area and floor. At 16384 the request was
+            # refused outright.
+            #
+            # --ctx-size is per slot, not the total: /props reports
+            # n_ctx_slot equal to this value while running four slots, where a
+            # total would have divided down to 4096. Doubling it alone would
+            # therefore double the KV cache, so the slot count halves in step
+            # and the allocation stays where it is today — 24979 MiB of the
+            # 32Gi limit, most of it the 19.5 GiB of weights.
+            #
+            # Two slots is what is actually in use: one holds Home Assistant's
+            # prompt, one holds Paperless's. A third caller would queue rather
+            # than evict either, which is the better failure mode given a cold
+            # prefill of this prompt costs around two minutes.
             - --ctx-size
-            - "16384"
+            - "32768"
+            - --parallel
+            - "2"
             # A micro-batch of 2048 or more makes this Vulkan/Mesa stack fail
             # with ErrorDeviceLost.
             - --ubatch-size

kubernetes/apps/llama-server/embed.yaml

@@ -41,15 +41,53 @@ spec:
           image: ghcr.io/ggml-org/llama.cpp:server-vulkan-b10731
           args:
             - --hf-repo
-            - Qwen/Qwen3-Embedding-0.6B-GGUF:Q8_0
+            - gpustack/bge-m3-GGUF:Q8_0
             # Serves POST /v1/embeddings.
             - --embeddings
+            # A short, stable id for consumers. Without it the model is
+            # advertised as the whole --hf-repo string, which would end up
+            # verbatim in Paperless's config and break on the next model swap.
+            # One name only: llama-server keeps aliases in a sorted set and
+            # reports the first as the id, so a second name can steal it.
+            - --alias
+            - bge-m3
+            # bge-m3 is trained for CLS pooling — a different type from the
+            # Qwen family this replaces, which uses last-token. Left unspecified
+            # llama-server takes the GGUF default, which is not visible from
+            # outside and would silently produce vectors the model was not
+            # trained to emit. Read from the model's own 1_Pooling/config.json
+            # rather than assumed; of the candidates measured, three different
+            # pooling types were needed.
+            - --pooling
+            - cls
             - --n-gpu-layers
             - "99"
             - --threads
             - "2"
+            # Paperless chunks at 1024 tokens, but counts them with its own
+            # tokenizer while this server measures the batch in bge-m3's.
+            # Those disagree: XLM-RoBERTa splits German into more pieces than
+            # llama_index's counter does, so a chunk Paperless considers full
+            # at 1024 arrived here as 1069 and the request failed outright —
+            # "input (1069 tokens) is too large to process".
+            #
+            # The headroom therefore has to absorb a tokenizer ratio, not just
+            # a rounding error, and 1.5x covers far more than the 1.044 that
+            # broke it. The defaults are batch 2048 / ubatch 512, and with
+            # --embeddings llama-server clamps both to the smaller rather than
+            # run them mismatched, so both are set.
+            #
+            # 2048 stays avoided: that is where the chat model met
+            # ErrorDeviceLost on this Vulkan stack. That failure came under a
+            # 20 GB model's memory pressure and may not bind a 600 MB one, but
+            # this does not need to find out.
+            - --batch-size
+            - "1536"
+            - --ubatch-size
+            - "1536"
+            # Room for a full chunk plus overhead rather than exactly one.
             - --ctx-size
-            - "2048"
+            - "4096"
             # No browser ever talks to this Service, so the WebUI is dead weight
             # and one more unauthenticated surface.
             - --no-webui
@@ -66,6 +104,14 @@ spec:
             capabilities:
               drop:
                 - ALL
+          # Measured, not estimated: embedding 400 real documents peaked at
+          # 1.6 GiB (Prometheus max_over_time on the working set). The limit is
+          # well above that because the footprint is dominated by the four slots
+          # at 4096 context and the Vulkan working set rather than by the
+          # weights — the Qwen model this replaces peaked at 10.6 GiB on the
+          # same settings for the same 600 MiB of weights, and several
+          # candidates were OOMKilled at limits that looked generous against
+          # their model size.
           resources:
             requests:
               cpu: 250m
@@ -73,7 +119,7 @@ spec:
               gpu.intel.com/i915: "1"
             limits:
               cpu: "2"
-              memory: 4Gi
+              memory: 8Gi
               gpu.intel.com/i915: "1"
           # ~600 MiB rather than ~5 GiB, so a much shorter budget than the chat
           # model needs, but still sized for a cold download.

kubernetes/apps/llama-server/pvc.yaml

@@ -1,10 +1,20 @@
 ---
-# Holds the GGUF weights, downloaded once by the Deployment's initContainer.
-# Prune-disabled so retiring this Kustomization does not throw away several
-# GiB of model files that would have to be pulled from Hugging Face again.
+# Holds the GGUF weights, downloaded once by llama-server itself via --hf-repo.
+# Prune-disabled so retiring this Kustomization does not throw away tens of GiB
+# of model files that would have to be pulled from Hugging Face again.
 #
-# ReadWriteMany so a second replica or a one-off benchmark Job can mount the
-# same weights; the models are read-only at inference time.
+# ReadWriteMany so a second replica, a one-off benchmark Job, or a candidate
+# model being evaluated can mount the same weights; the models are read-only at
+# inference time.
+#
+# Sized well beyond the two deployed models. The cache is shared by every
+# candidate under evaluation (scripts/embed-eval-real.sh), and on a slow
+# connection the download is the long pole — a shared cache lets the large
+# candidates pull in the background while the small ones are already being
+# measured, and means a model is never fetched twice across runs.
+#
+# CephFS supports online expansion (allowVolumeExpansion on the storage class),
+# so growing this does not disturb the running pods.
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
@@ -18,4 +28,4 @@ spec:
   storageClassName: ceph-filesystem
   resources:
     requests:
-      storage: 40Gi
+      storage: 120Gi

kubernetes/apps/monitoring-rules/crowdsec-servicemonitor.yaml

@@ -0,0 +1,36 @@
+# Scrapes CrowdSec. Until this landed, nothing did: `up{namespace="crowdsec"}`
+# returned zero series and Prometheus held no cs_* metric at all, so the
+# detection side of the ingress path was invisible — including whether the
+# agents were reading logs and whether any bouncer was pulling decisions.
+#
+# One monitor for both services. They carry the same shape (a `metrics` port on
+# 6060) and differ only in `app`, so a matchExpressions selector keeps this to
+# a single object rather than two near-identical ones:
+#
+#   crowdsec-service        the LAPI      — decisions, bouncer + machine requests
+#   crowdsec-agent-service  the agents    — acquisition, parsers, scenarios
+#
+# The crowdsec-ns CiliumNetworkPolicy already admits the monitoring namespace
+# without a port restriction, so no policy change is needed.
+#
+# `release: kube-prometheus-stack` is what the chart's Prometheus selects on.
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: crowdsec
+  namespace: crowdsec
+  labels:
+    app.kubernetes.io/name: crowdsec
+    release: kube-prometheus-stack
+spec:
+  selector:
+    matchExpressions:
+      - key: app
+        operator: In
+        values:
+          - crowdsec-service
+          - crowdsec-agent-service
+  endpoints:
+    - port: metrics
+      path: /metrics
+      interval: 30s

kubernetes/apps/monitoring-rules/kustomization.yaml

@@ -23,6 +23,7 @@ resources:
   - prometheusrule-cnpg-backup.yaml
   - alloy-servicemonitor.yaml
   - cnpg-podmonitors.yaml
+  - crowdsec-servicemonitor.yaml
   - home-assistant-servicemonitor.yaml
   - immich-servicemonitor.yaml
   - loki-servicemonitor.yaml

kubernetes/apps/monitoring/dashboards/crowdsec-overview.json

@@ -0,0 +1,163 @@
+{
+  "annotations": {"list": []},
+  "description": "CrowdSec: Erkennung im Cluster, Durchsetzung am NetBird-Edge",
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 1,
+  "id": null,
+  "links": [],
+  "panels": [
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Aktive Ban-Entscheidungen in der LAPI, ganz überwiegend aus den CAPI-Community-Blocklisten. Das ist die Menge, die ein scharfgeschalteter Bouncer durchsetzt.",
+      "fieldConfig": {
+        "defaults": {"color": {"mode": "thresholds"}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "text", "value": null}]}, "unit": "short"},
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 6, "x": 0, "y": 0},
+      "id": 1,
+      "options": {"colorMode": "value", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
+      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(cs_active_decisions)", "refId": "A"}],
+      "title": "Aktive Decisions",
+      "type": "stat"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Bouncer, die in den letzten 10 Minuten Decisions abgeholt haben. Fällt das unter die erwartete Zahl, holt ein Bouncer nichts mehr — und setzt damit einen veralteten Stand durch, ohne dass irgendetwas ausfällt.",
+      "fieldConfig": {
+        "defaults": {"color": {"mode": "thresholds"}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "red", "value": null}, {"color": "green", "value": 1}]}, "unit": "short"},
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 6, "x": 6, "y": 0},
+      "id": 2,
+      "options": {"colorMode": "value", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
+      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "count(sum by (bouncer) (rate(cs_lapi_bouncer_requests_total[10m])) > 0)", "refId": "A"}],
+      "title": "Bouncer, die abholen",
+      "type": "stat"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Vom Agent gelesene Logzeilen pro Sekunde. Steht das auf 0, detektiert CrowdSec nichts mehr — der häufigste stille Ausfall, weil alles andere weiterläuft.",
+      "fieldConfig": {
+        "defaults": {"color": {"mode": "thresholds"}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "red", "value": null}, {"color": "green", "value": 0.01}]}, "unit": "reqps"},
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 6, "x": 12, "y": 0},
+      "id": 3,
+      "options": {"colorMode": "value", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
+      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(rate(cs_filesource_hits_total[5m]))", "refId": "A"}],
+      "title": "Gelesene Logzeilen/s",
+      "type": "stat"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Szenario-Overflows der letzten 24 h — also lokal erkannte Angriffe, im Gegensatz zu den von CAPI übernommenen.",
+      "fieldConfig": {
+        "defaults": {"color": {"mode": "thresholds"}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "text", "value": null}]}, "unit": "short"},
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 6, "x": 18, "y": 0},
+      "id": 4,
+      "options": {"colorMode": "value", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
+      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(increase(cs_bucket_overflowed_total[24h]))", "refId": "A"}],
+      "title": "Lokale Overflows (24 h)",
+      "type": "stat"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Abholrate je Bouncer. Der Firewall-Bouncer auf den Proxies fragt alle 10 s. Die LAPI legt bei jeder neuen beobachteten Client-Adresse einen Eintrag `<name>@<ip>` an, und beobachtet wird die Pod-IP des netbird-ingress-Peers — deshalb erscheint ein logischer Bouncer hier mehrfach und nach Pod-Rollen unter neuen Namen.",
+      "fieldConfig": {
+        "defaults": {"color": {"mode": "palette-classic"}, "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]}, "unit": "reqps"},
+        "overrides": []
+      },
+      "gridPos": {"h": 8, "w": 12, "x": 0, "y": 4},
+      "id": 5,
+      "options": {"legend": {"calcs": ["lastNotNull"], "displayMode": "table", "placement": "right", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum by (bouncer) (rate(cs_lapi_bouncer_requests_total[5m]))", "legendFormat": "{{bouncer}}", "refId": "A"}],
+      "title": "Bouncer-Abholrate",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Aktive Decisions nach Grund. `http:scan` dominiert und stammt aus den CAPI-Blocklisten.",
+      "fieldConfig": {
+        "defaults": {"color": {"mode": "palette-classic"}, "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 20, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "normal"}, "thresholdsStyle": {"mode": "off"}}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]}, "unit": "short"},
+        "overrides": []
+      },
+      "gridPos": {"h": 8, "w": 12, "x": 12, "y": 4},
+      "id": 6,
+      "options": {"legend": {"calcs": ["lastNotNull"], "displayMode": "table", "placement": "right", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum by (reason) (cs_active_decisions)", "legendFormat": "{{reason}}", "refId": "A"}],
+      "title": "Aktive Decisions nach Grund",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Welche Szenarien lokal auslösen. Das ist das, was CrowdSec selbst aus den Traefik-Logs erkennt.",
+      "fieldConfig": {
+        "defaults": {"color": {"mode": "palette-classic"}, "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "bars", "fillOpacity": 70, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "normal"}, "thresholdsStyle": {"mode": "off"}}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]}, "unit": "short"},
+        "overrides": []
+      },
+      "gridPos": {"h": 8, "w": 12, "x": 0, "y": 12},
+      "id": 7,
+      "options": {"legend": {"calcs": ["sum"], "displayMode": "table", "placement": "right", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum by (name) (increase(cs_bucket_overflowed_total[1h]))", "legendFormat": "{{name}}", "refId": "A"}],
+      "title": "Szenario-Overflows (je Stunde)",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Geparste gegen nicht geparste Zeilen. Ein Anstieg bei `ko` heißt meist, dass sich ein Logformat geändert hat und der Parser nicht mehr greift — CrowdSec liest dann weiter, erkennt aber nichts mehr.",
+      "fieldConfig": {
+        "defaults": {"color": {"mode": "palette-classic"}, "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]}, "unit": "reqps"},
+        "overrides": []
+      },
+      "gridPos": {"h": 8, "w": 12, "x": 12, "y": 12},
+      "id": 8,
+      "options": {"legend": {"calcs": ["lastNotNull"], "displayMode": "list", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(rate(cs_parser_hits_ok_total[5m]))", "legendFormat": "geparst", "refId": "A"},
+        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(rate(cs_parser_hits_ko_total[5m]))", "legendFormat": "nicht geparst", "refId": "B"}
+      ],
+      "title": "Parser",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Whitelist-Treffer nach Grund. Nützlich beim Scharfschalten: was hier zählt, wäre sonst ein Kandidat für einen Ban gewesen.",
+      "fieldConfig": {
+        "defaults": {"color": {"mode": "palette-classic"}, "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "off"}}, "mappings": [], "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]}, "unit": "reqps"},
+        "overrides": []
+      },
+      "gridPos": {"h": 8, "w": 12, "x": 0, "y": 20},
+      "id": 9,
+      "options": {"legend": {"calcs": ["lastNotNull"], "displayMode": "table", "placement": "right", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum by (reason) (rate(cs_node_wl_hits_ok_total[5m]))", "legendFormat": "{{reason}}", "refId": "A"}],
+      "title": "Whitelist-Treffer",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "gridPos": {"h": 8, "w": 12, "x": 12, "y": 20},
+      "id": 10,
+      "options": {
+        "code": {"language": "plaintext", "showLineNumbers": false, "showMiniMap": false},
+        "content": "## Was dieses Dashboard NICHT zeigt\n\nAlle Panels stammen aus der LAPI und den Agents im Cluster. Die **Bouncer auf proxy-1 und proxy-2 exportieren eigene Metriken**, die hier fehlen:\n\n- `fw_bouncer_active_decisions` — was tatsächlich in den nftables-Sets steht\n- `fw_bouncer_dropped_packets` / `_bytes` — was wirklich verworfen wurde\n- `lapi_requests_total` / `_failures_total` — aus Sicht des Bouncers\n\nSie liegen auf `127.0.0.1:60601` der Proxy-Hosts. Die Hosts sind öffentlich und haben bisher keinerlei Metrik-Pfad; den Port ins Internet zu öffnen, nur um ein Scrape-Ziel zu bekommen, wäre der falsche Tausch.\n\n**Konsequenz:** „Bouncer-Abholrate\" oben zeigt, dass ein Bouncer *fragt* — nicht, dass er die Antwort auch angewandt hat. Solange der Modus `dry-run` ist, wendet er ohnehin nichts an.\n\nEin Scrape-Pfad für die Hosts ist eigene Arbeit und Voraussetzung dafür, dass „was wird wirklich geblockt\" beantwortbar wird."
+      },
+      "title": "Blinder Fleck",
+      "type": "text"
+    }
+  ],
+  "refresh": "30s",
+  "schemaVersion": 38,
+  "style": "dark",
+  "tags": ["homelab", "crowdsec", "security"],
+  "templating": {"list": []},
+  "time": {"from": "now-6h", "to": "now"},
+  "timepicker": {},
+  "timezone": "",
+  "title": "CrowdSec Overview",
+  "uid": "crowdsec-overview",
+  "version": 1,
+  "weekStart": ""
+}

kubernetes/apps/monitoring/dashboards/kustomization.yaml

@@ -38,6 +38,9 @@ configMapGenerator:
   - name: dashboard-blackbox-overview
     files:
       - blackbox-overview.json
+  - name: dashboard-crowdsec-overview
+    files:
+      - crowdsec-overview.json
   - name: dashboard-disk-health
     files:
       - disk-health.json

kubernetes/apps/paperless/MIGRATION-data-to-block.md

@@ -1,97 +0,0 @@
-# paperless-data: CephFS → RBD
-
-## Why
-
-`paperless-data` is `ReadWriteMany` on `ceph-filesystem`, but **only the paperless
-Deployment mounts it**. Verified 2026-08-20:
-
-```
-$ pods mounting paperless-data
-paperless-68cb8b8744-7hjb8      <- one, and only one
-
-paperless-data     ReadWriteMany  ceph-filesystem
-paperless-media    ReadWriteMany  ceph-filesystem   <- genuinely shared (ftp-scanner)
-paperless-consume  ReadWriteMany  ceph-filesystem   <- genuinely shared (ftp-scanner)
-```
-
-Its contents are single-writer state — the 50 MB Whoosh index,
-`classification_model.pickle`, `celerybeat-schedule.db`, `log/`,
-`migration_lock`, `protonmail.pem`. 75 files. Nothing else reads or writes them.
-
-It is on CephFS by default, not by requirement.
-
-## What it costs
-
-Every access goes through the MDS. Measured with **zero CI pipelines running**:
-
-```
-paperless-data client   request_load_avg 22   (all other sessions 0-1)
-MDS over 180s           4,052 client requests  = 22.5/s
-                        2,718 journal events   = 15/s
-                            2 dir commits      <- almost none: not file writes
-```
-
-So it is cap/inode churn from repeated access, not data being written. That is
-most of the constant **7-8 GB/day** written to `cephfs-metadata`, and since that
-pool is `size: 3`, roughly **21-24 GB/day** reaches the drives.
-
-`cephfs-metadata` deliberately stays at size 3. Losing it destroys the entire
-filesystem rather than the objects on one OSD, and the 275 GiB in `cephfs-data0`
-depends on it. Cutting the writes at source is correct; cutting the replication
-is not.
-
-## Sequence
-
-Staged deliberately: the PVC lands first and empty, so merging it changes
-nothing. The copy and the repoint are separate.
-
-### Phase 1 — provision (this PR)
-
-`paperless-data-block` (RWO, `ceph-block`, 10Gi) is created and mounted by
-nothing. Inert.
-
-Verify: `kubectl -n paperless get pvc paperless-data-block` → Bound.
-
-### Phase 2 — copy
-
-The Whoosh index must come across intact. Paperless *can* rebuild it
-(`document_index reindex`), but that reprocesses every document, so copying is
-both faster and less disruptive.
-
-1. Scale paperless to 0. The source must be quiescent — `celerybeat-schedule.db`
-   and the Whoosh index are both live-written, and copying them under an active
-   writer gives a torn read.
-2. Run a one-off pod that mounts BOTH PVCs and copies:
-   ```
-   cp -a /src/. /dst/
-   ```
-   The RWO target pins the pod to one node; the RWX source will follow.
-3. Verify file count and index size match:
-   ```
-   find /src -type f | wc -l   ==   find /dst -type f | wc -l
-   du -sh /src/index           ==   du -sh /dst/index
-   ```
-
-### Phase 3 — repoint
-
-Change the Deployment's `data` volume to `claimName: paperless-data-block`,
-merge, let it roll.
-
-Verify afterwards:
-- paperless starts, search returns results (proves the Whoosh index survived)
-- `ceph tell mds.cephfs-b session ls` no longer shows a busy `paperless-data`
-  client
-- `rate(ceph_pool_wr_bytes{pool_id="3"})` drops from its 7-8 GB/day baseline
-
-**Do not delete `paperless-data` until search has been confirmed working.** It is
-the rollback, and it is `prune: disabled` so Flux will not remove it.
-
-## Expected effect
-
-~7-8 GB/day off `cephfs-metadata`, so ~21-24 GB/day off the drives once
-replication is counted. Against ~432 GB/day cluster-wide that is roughly 5% —
-worth doing, but it is a correctness fix (a single-writer volume does not belong
-on a shared filesystem) more than a large win.
-
-It also removes 22 req/sec of MDS load, which benefits the workloads that
-genuinely need CephFS — Woodpecker CI workspaces share the same MDS.

kubernetes/apps/paperless/network-policy.yaml

@@ -57,6 +57,20 @@ spec:
               protocol: UDP
             - port: "53"
               protocol: TCP
+    # The cluster's own LLM, for AI suggestions and RAG embeddings. Scoped to
+    # the two pods and the one port rather than the ai namespace as a whole, so
+    # this grant cannot widen if something else is ever placed there.
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": ai
+            app.kubernetes.io/name: llama-server
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": ai
+            app.kubernetes.io/name: llama-embed
+      toPorts:
+        - ports:
+            - port: "8080"
+              protocol: TCP
     # External HTTPS only — Tika model downloads, runtime image
     # fetches, etc. RFC1918 + ULA are excluded from `world` via the
     # CIDR-deny pattern. `world` covers public IPs / DNS-resolvable

kubernetes/apps/paperless/paperless.yaml

@@ -31,12 +31,14 @@ data:
   PAPERLESS_CONSUMER_DELETE_DUPLICATES: "true"
   PAPERLESS_CONSUMER_ENABLE_BARCODES: "true"
   PAPERLESS_CONSUMER_ENABLE_ASN_BARCODE: "true"
-  PAPERLESS_CONSUMER_BARCODE_SCANNER: "ZXING"
+  # No CONSUMER_BARCODE_SCANNER: zxing-cpp is the only backend, so the
+  # setting no longer exists. pyzbar and libzbar are gone from the image.
+  #
   # CephFS does not propagate inotify across nodes — files written
   # via the RWX consume PVC from a pod on a different node than
   # paperless never fire IN_CREATE in paperless's kernel. Polling
   # is the reliable backstop. See wiki: cephfs-inotify-no-cross-node-propagation.
-  PAPERLESS_CONSUMER_POLLING: "10"
+  PAPERLESS_CONSUMER_POLLING_INTERVAL: "10"
   PAPERLESS_REDIS: "redis://redis:6379"
   PAPERLESS_DBENGINE: "postgresql"
   PAPERLESS_DBHOST: "pprl-pg-rw"
@@ -44,6 +46,47 @@ data:
   PAPERLESS_TIKA_ENABLED: "1"
   PAPERLESS_TIKA_ENDPOINT: "http://tika:9998"
   PAPERLESS_TIKA_GOTENBERG_ENDPOINT: "http://gotenberg:3000"
+  # AI suggestions against the cluster's own LLM (kubernetes/apps/llama-server).
+  # Nothing leaves the cluster: both endpoints are ClusterIP Services, so
+  # document text is never sent to a third party and there is no API key to
+  # rotate — llama-server has no authentication, and the ai namespace's network
+  # policy is what restricts who may reach it.
+  #
+  # CAVEAT, and it is a real one: paperless reads these from an
+  # ApplicationConfiguration row in the database and falls back to the
+  # environment only when that row is empty. A value set in the admin UI
+  # therefore overrides this ConfigMap silently. Configure AI in ONE place —
+  # here — and leave those fields untouched in the UI.
+  PAPERLESS_AI_ENABLED: "true"
+  PAPERLESS_AI_LLM_BACKEND: "openai-like"
+  PAPERLESS_AI_LLM_ENDPOINT: "http://llama-server.ai.svc.cluster.local:8080/v1"
+  # Must match --alias in kubernetes/apps/llama-server/deployment.yaml. The
+  # server ignores the field on the wire, but paperless sends it and the value
+  # is what appears in its logs.
+  PAPERLESS_AI_LLM_MODEL: "qwen3.6-35b-a3b"
+  # ClusterIP addresses are private, and paperless blocks non-public endpoints
+  # unless this is set. Default is already true; explicit because the whole
+  # integration silently stops working if it ever flips.
+  PAPERLESS_AI_LLM_ALLOW_INTERNAL_ENDPOINTS: "true"
+  # Under llama-server's own 16384, leaving room for the response.
+  PAPERLESS_AI_LLM_CONTEXT_SIZE: "12288"
+  # The 120s default is too tight at ~11 tok/s: a classification is roughly
+  # 2-4k prompt tokens plus a second round trip for localisation.
+  PAPERLESS_AI_LLM_REQUEST_TIMEOUT: "300"
+  # Matched against Django's settings.LANGUAGES, so the region is required —
+  # "de" alone does not resolve. Without it the model returns German tags but
+  # English document types, which was reproducible against this endpoint.
+  PAPERLESS_AI_LLM_OUTPUT_LANGUAGE: "de-de"
+  # RAG embeddings from llama-embed. Its Qwen3-Embedding covers 100+ languages,
+  # where paperless's default all-MiniLM-L6-v2 is documented as suited to
+  # primarily English archives — the better fit for a German corpus.
+  PAPERLESS_AI_LLM_EMBEDDING_BACKEND: "openai-like"
+  PAPERLESS_AI_LLM_EMBEDDING_ENDPOINT: "http://llama-embed.ai.svc.cluster.local:8080/v1"
+  # Must match --alias in kubernetes/apps/llama-server/embed.yaml. Changing the
+  # embedding model invalidates every stored vector, so a swap means the LLM
+  # index has to be rebuilt.
+  PAPERLESS_AI_LLM_EMBEDDING_MODEL: "bge-m3"
+  PAPERLESS_AI_LLM_EMBEDDING_CHUNK_SIZE: "1024"
 ---
 apiVersion: apps/v1
 kind: Deployment
@@ -56,6 +99,13 @@ metadata:
     # Reloader restarts paperless when paperless-secrets changes (e.g. after a
     # secret-key rotation). See kubernetes/infrastructure/openbao/rotation/paperless.yaml.
     secret.reloader.stakater.com/reload: paperless-secrets
+    # And when paperless-config changes. The ConfigMap is consumed with
+    # envFrom, and Kubernetes does not restart a pod when a ConfigMap it reads
+    # that way is updated — the running container keeps the environment it
+    # started with. Without this, a settings change applies cleanly in git and
+    # in the cluster while the application never sees it, which is how the AI
+    # settings landed with no effect at all.
+    configmap.reloader.stakater.com/reload: paperless-config
 spec:
   replicas: 1
   strategy:
@@ -120,7 +170,7 @@ spec:
               drop: ["ALL"]
       containers:
         - name: paperless
-          image: ghcr.io/paperless-ngx/paperless-ngx:2.20.15 # {"$imagepolicy": "flux-system:paperless-ngx"}
+          image: ghcr.io/paperless-ngx/paperless-ngx:3.1.2 # {"$imagepolicy": "flux-system:paperless-ngx"}
           ports:
             - name: http
               containerPort: 8000
@@ -172,6 +222,25 @@ spec:
           # Safe against ALLOWED_HOSTS: the kubelet sends the pod IP as Host,
           # and PAPERLESS_ALLOWED_HOSTS is "*". Narrowing that value later
           # means giving these probes an explicit Host header.
+
+          # The container does not serve until its init sequence finishes:
+          # s6 makes svc-webserver depend on init-migrations and
+          # init-search-index. On the first 3.x start those are the SHA256
+          # checksum backfill, which opens and reads every original and every
+          # archive file, and a full Tantivy index build, since the Whoosh
+          # index cannot be converted. Both scale with the corpus.
+          #
+          # startupProbe holds liveness and readiness off until the webserver
+          # answers, giving that boot 30 * 20s = 10 minutes. Without it the
+          # 120s liveness delay plus three 30s periods kills the pod 210s in,
+          # part-way through a migration, and the next boot starts over.
+          startupProbe:
+            httpGet:
+              path: /accounts/login/
+              port: http
+            periodSeconds: 20
+            timeoutSeconds: 5
+            failureThreshold: 30
           livenessProbe:
             httpGet:
               path: /accounts/login/
@@ -220,22 +289,16 @@ spec:
             seccompProfile:
               type: RuntimeDefault
       volumes:
-        # paperless-data-block (RWO, ceph-block), NOT paperless-data
-        # (RWX, ceph-filesystem). Only this Deployment ever mounted the volume,
-        # and its contents are single-writer state — the Whoosh index,
-        # classification model, celerybeat schedule, logs. Being on CephFS meant
-        # every access went through the MDS: 22 requests/sec sustained with zero
-        # CI running, ~15 journal events/sec against 2 directory commits per
-        # 180s, i.e. cap/inode churn rather than file writes. That was most of
-        # the constant 7-8 GB/day on cephfs-metadata, and that pool is size 3.
+        # `data` is RWO on ceph-block, not RWX on ceph-filesystem. Only this
+        # Deployment mounts it and its contents are single-writer state — the
+        # search index, classification model, celerybeat schedule, logs. On
+        # CephFS every access went through the MDS: 22 requests/sec sustained
+        # with zero CI running, ~15 journal events/sec against 2 directory
+        # commits per 180s, i.e. cap/inode churn rather than file writes. That
+        # was most of the constant 7-8 GB/day on cephfs-metadata, a size 3 pool.
         #
         # `consume` and `media` below STAY on CephFS: the ftp-scanner writes to
         # those from another node, so they genuinely need RWX.
-        #
-        # paperless-data is retained (prune: disabled) as the rollback. Do not
-        # delete it until search has been confirmed working — that is what
-        # proves the Whoosh index survived the copy, not merely that paperless
-        # starts. Migration record: MIGRATION-data-to-block.md.
         - name: data
           persistentVolumeClaim:
             claimName: paperless-data-block

kubernetes/apps/paperless/pvcs.yaml

@@ -3,33 +3,18 @@
 # (e.g. when the seed initContainer or the Garage migration creds are
 # removed once we're done with the migration phase). Velero's
 # bootstrap-restore then owns their contents.
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: paperless-data
-  namespace: paperless
-  annotations:
-    kustomize.toolkit.fluxcd.io/prune: disabled
-spec:
-  accessModes:
-    - ReadWriteMany
-  storageClassName: ceph-filesystem
-  resources:
-    requests:
-      storage: 10Gi
----
-# Replacement for paperless-data, on RBD instead of CephFS.
+
+# paperless's single-writer state, on RBD rather than CephFS.
 #
-# WHY: paperless-data is declared ReadWriteMany on ceph-filesystem, but only ONE
-# pod ever mounts it — the paperless Deployment itself. Its contents are all
-# single-writer state: the 50 MB Whoosh index, classification_model.pickle,
-# celerybeat-schedule.db, logs, migration_lock. Nothing shares it. Only `consume`
-# and `media` genuinely need RWX, because the FTP scanner writes to those from a
-# different node.
+# WHY RWO: only ONE pod mounts it — the paperless Deployment itself. Its
+# contents are all single-writer state: the Tantivy search index,
+# classification_model.pickle, celerybeat-schedule.db, logs, migration_lock.
+# Nothing shares it. Only `consume` and `media` genuinely need RWX, because the
+# FTP scanner writes to those from a different node.
 #
-# COST OF LEAVING IT: every access goes through the MDS. Measured 2026-08-20 with
-# zero CI running, this volume's client was doing 22 MDS requests/sec sustained,
-# and the MDS logged ~15 journal events/sec against only 2 directory commits per
+# WHY NOT CEPHFS: every access goes through the MDS. With zero CI running, this
+# volume's client sustained 22 MDS requests/sec, and the MDS logged ~15 journal
+# events/sec against only 2 directory commits per
 # 180s — i.e. cap and inode churn from repeated access, not file writes. That is
 # most of the constant 7-8 GB/day written to cephfs-metadata, which is a size 3
 # pool, so ~21-24 GB/day actually hits the drives.
@@ -38,10 +23,6 @@ spec:
 # the metadata pool destroys the whole filesystem, not just the objects on one
 # OSD, so the 275 GiB in cephfs-data0 rides on it. Reducing the writes at source
 # is the right fix; reducing the replication is not.
-#
-# MIGRATION: this PVC is created EMPTY and nothing mounts it yet. The data copy
-# and the deployment repoint are a separate, staged step — see
-# MIGRATION-data-to-block.md. Merging this alone is inert.
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:

kubernetes/apps/whisper/deployment.yaml

@@ -18,7 +18,7 @@ spec:
     spec:
       containers:
         - name: whisper
-          image: rhasspy/wyoming-whisper:3.6.0 # {"$imagepolicy": "flux-system:wyoming-whisper"}
+          image: rhasspy/wyoming-whisper:3.7.0 # {"$imagepolicy": "flux-system:wyoming-whisper"}
           args:
             - --model=base-int8
             - --language=de

kubernetes/infrastructure/crowdsec/edge-bouncer-key-externalsecret.yaml

@@ -0,0 +1,27 @@
+# Bouncer key for the firewall bouncers on proxy-1/proxy-2.
+#
+# Separate from crowdsec-bouncer-key ON PURPOSE. That key is rotated
+# quarterly by kubernetes/infrastructure/openbao/rotation/crowdsec-bouncer-key.yaml,
+# and both of its consumers live in the cluster, so ESO propagates a rotation
+# within 10m. The edge bouncers are Ansible-managed hosts: nothing there
+# re-reads OpenBao on a schedule, so a rotated key would leave them 401ing
+# against the LAPI until someone runs the playbook. This key is therefore
+# deliberately NOT enrolled in the rotator — it rotates when the playbook runs,
+# and the playbook is what carries the new value to the hosts.
+#
+# The value must match ansible vault_crowdsec_edge_bouncer_key.
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: crowdsec-edge-bouncer-key
+  namespace: crowdsec
+spec:
+  refreshInterval: 10m
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: openbao
+  target:
+    name: crowdsec-edge-bouncer-key
+  dataFrom:
+    - extract:
+        key: infrastructure/crowdsec-edge-bouncer-key

kubernetes/infrastructure/crowdsec/helmrelease.yaml

@@ -36,6 +36,51 @@ spec:
   values:
     container_runtime: containerd
 
+    # Addresses a community blocklist must never be allowed to ban.
+    #
+    # This is the ONLY declarative allowlist the chart offers. `cscli
+    # allowlists` objects live in the LAPI database, and that database runs
+    # without persistence here on purpose (see lapi.persistentVolume below),
+    # so anything created through cscli is gone at the next restart. This file
+    # is a ConfigMap mount, so it survives by construction. Upstream marks it
+    # deprecated in favour of centralized allowlists; it is still the only
+    # thing that works against a LAPI with no PVC.
+    #
+    # Scope: CAPI decisions only. Locally generated decisions are not filtered
+    # here — they currently carry real client IPs (verified), because Traefik
+    # resolves X-Forwarded-For past the netbird-ingress pod IP.
+    #
+    # The home WAN address is deliberately absent and is not needed: LAN
+    # clients never present it. opentofu/unifi_nat_hairpin.tf DNATs the public
+    # proxy addresses to the Traefik VIP on the UDM, so on-LAN traffic reaches
+    # the cluster directly with a LAN source and never traverses the proxies.
+    # Private ranges cannot appear in a community blocklist either way.
+    config:
+      capi_whitelists.yaml: |
+        cidrs:
+          # The proxy hosts themselves. Same three addresses as the
+          # netbird-ingress-egress toCIDRSet and the hairpin endpoints —
+          # keep all three lists in sync. proxy-1 is the v6-only member,
+          # proxy-2 the dual-stack one.
+          - 45.157.177.26/32
+          - 2a03:4000:4b:c1e:4474:43ff:fedf:8f85/128
+          - 2a03:4000:28:f6e:a4c7:59ff:fe1e:9a29/128
+          # netbird.renner.uno — management, signal, relay and TURN.
+          #
+          # Not optional once a bouncer is armed. Its chain sits on the INPUT
+          # hook, the proxy hosts run no other firewall, and there is therefore
+          # no conntrack rule accepting established flows ahead of it. A ban on
+          # this address would drop the RETURN traffic of the proxies' own
+          # outbound control connection and take them off the mesh — with no
+          # inbound attack involved at all.
+          #
+          # Checked when this was added: none of these addresses appear in the
+          # current decision set. That is the state today, not a guarantee —
+          # these are rented addresses on a hoster whose ranges do end up on
+          # community blocklists.
+          - 162.55.50.97/32
+          - 2a01:4f8:c2c:4579::1/128
+
     # mTLS between agents and the LAPI. Do not switch back to password auth:
     # the LAPI runs without persistence, so its machine-registration DB is
     # wiped on every restart, and password-registered agents then get 401
@@ -78,6 +123,24 @@ spec:
 
     lapi:
       replicas: 1
+      service:
+        # Backend wiring for the public LAPI Ingress (lapi-ingress.yaml).
+        #
+        # These belong on the SERVICE, not on the Ingress: under the Kubernetes
+        # Ingress provider `traefik.ingress.kubernetes.io/service.*` are service
+        # annotations, and on an Ingress they are silently ignored. The symptom
+        # is Traefik speaking plaintext to a TLS-only backend, which the LAPI
+        # answers with "Client sent an HTTP request to an HTTPS server." and a
+        # 400 — an error that looks like a malformed request rather than a
+        # scheme mismatch.
+        #
+        # With tls.enabled the LAPI serves HTTPS on 8080 and offers no
+        # plaintext port, so https here is mandatory rather than a hardening
+        # choice. The ServersTransport carries the chart's own CA, since that
+        # certificate is self-signed.
+        annotations:
+          traefik.ingress.kubernetes.io/service.serversscheme: https
+          traefik.ingress.kubernetes.io/service.serverstransport: crowdsec-crowdsec-lapi@kubernetescrd
       # Far below the chart default, which reserves half a core per LAPI for a
       # workload that draws single-digit millicores — enough to leave the
       # cluster unschedulable with one node out. Only the reservation shrinks;
@@ -86,10 +149,18 @@ spec:
         requests:
           cpu: 50m
       podAnnotations:
-        # Reloader restarts the LAPI when crowdsec-bouncer-key changes (it's an
-        # env-var secret, so a restart is required to pick up a rotated key). See
+        # Reloader restarts the LAPI when either bouncer key changes. Both are
+        # consumed as env vars from a secretKeyRef, and env is fixed at pod
+        # start — without a restart the LAPI keeps trusting the old value while
+        # the bouncer already presents the new one, which is a 401 at exactly
+        # the moment a rotation is supposed to be seamless.
+        #
+        # crowdsec-bouncer-key rotates on a schedule, see
         # kubernetes/infrastructure/openbao/rotation/crowdsec-bouncer-key.yaml.
-        secret.reloader.stakater.com/reload: crowdsec-bouncer-key
+        # crowdsec-edge-bouncer-key rotates by hand together with
+        # ansible/plays/install-netbird-proxy.yml; it still needs to be listed
+        # here, because the LAPI side of that rotation is this env var.
+        secret.reloader.stakater.com/reload: crowdsec-bouncer-key,crowdsec-edge-bouncer-key
       persistentVolume:
         data:
           # Kept OFF deliberately. The #337 attempt to set this `true` (persist
@@ -118,6 +189,17 @@ spec:
             secretKeyRef:
               name: crowdsec-bouncer-key
               key: key
+        # The firewall bouncers on proxy-1/proxy-2, over the public LAPI
+        # endpoint (lapi-ingress.yaml). The chart's entrypoint re-runs
+        # `register_bouncer` for every BOUNCER_KEY_* on each start and skips
+        # names that already exist, so this survives the LAPI's wiped
+        # registration DB the same way BOUNCER_KEY_ingress does — the env var
+        # is the source of truth, not the database.
+        - name: BOUNCER_KEY_edge
+          valueFrom:
+            secretKeyRef:
+              name: crowdsec-edge-bouncer-key
+              key: key
 
     agent:
       # Same reasoning as lapi.resources above, multiplied by three: this is a

kubernetes/infrastructure/crowdsec/kustomization.yaml

@@ -2,4 +2,6 @@ apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
   - bouncer-key-externalsecret.yaml
+  - edge-bouncer-key-externalsecret.yaml
+  - lapi-ingress.yaml
   - helmrelease.yaml

kubernetes/infrastructure/crowdsec/lapi-ingress.yaml

@@ -0,0 +1,85 @@
+# Public LAPI endpoint for the firewall bouncers on proxy-1/proxy-2.
+#
+# WHY PUBLIC AT ALL. The bouncers run as host processes on the proxies, and
+# the proxies have no mesh access: the netbird-proxy container runs non-root
+# (see the ip_unprivileged_port_start task in the netbird-proxy role), so it
+# has no NET_ADMIN and no /dev/net/tun, and its embedded NetBird client
+# therefore terminates the tunnel in userspace. The tunnel belongs to that
+# process, not to the host netns — a second process on the same host cannot
+# route through it. The alternative was a host-level NetBird client plus the
+# LAPI on a LoadBalancer VIP; this reuses the path every other service already
+# takes instead of adding two peers and an ACL policy.
+#
+# NO IP ALLOWLIST, deliberately. The bouncer key is the access control.
+#
+# An allowlist was the obvious second factor and it does not work here. A
+# request originating on a proxy host does not arrive carrying that host's
+# public address: the client connects to the local proxy on 127.0.0.1, the
+# reverse-proxy puts what it saw into X-Forwarded-For, and Traefik resolves
+# that. Measured against the netbird-proxy-watchdog, which makes exactly this
+# kind of request every round:
+#
+#   172.18.0.3 - - [02/Sep/2026:10:12:01] "GET /" 200 websecure-immich-...
+#   172.18.0.3 - - [02/Sep/2026:10:12:01] "GET /" 200 websecure-home-...
+#
+# 172.18.0.3 is a Docker bridge address on the proxy side. Allowlisting the
+# published proxy addresses would have rejected the bouncer with 403;
+# allowlisting 172.18.0.0/16 would be a private range on a host we do not
+# control the addressing of, which is not a security control.
+#
+# What remains is the API key, which is the real control anyway: this endpoint
+# serves read-only decision streams to a valid bouncer key and nothing else.
+# The admin surface is cscli, in-cluster. If the source address ever becomes
+# deterministic, an ipAllowList can be added on top.
+---
+# The LAPI serves HTTPS only once tls.enabled is set — there is no plaintext
+# fallback and no second port; 8080 is the TLS port. Its certificate comes
+# from the chart's own self-signed CA, so Traefik needs that CA to verify the
+# backend. SANs include crowdsec-service.crowdsec.svc.cluster.local, which is
+# what the Service resolves to.
+apiVersion: traefik.io/v1alpha1
+kind: ServersTransport
+metadata:
+  name: crowdsec-lapi
+  namespace: crowdsec
+spec:
+  serverName: crowdsec-service.crowdsec.svc.cluster.local
+  rootCAs:
+    - secret: crowdsec-root-secret
+---
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: crowdsec-lapi
+  namespace: crowdsec
+  labels:
+    # Public DNS via Cloudflare; the proxies resolve it from outside. The UDM
+    # external-dns instance skips it, LAN reaches it via the hairpin NAT.
+    dns.renner.uno/public: "true"
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-cluster
+    # NOT behind the Authentik gate: the client is a daemon with an API key,
+    # not a browser that can complete an OIDC flow.
+    #
+    # The backend scheme and ServersTransport are NOT set here. Under the
+    # Kubernetes Ingress provider, `traefik.ingress.kubernetes.io/service.*`
+    # are SERVICE annotations — on an Ingress they are silently ignored, and
+    # Traefik then talks plaintext to a TLS-only backend. They live on the
+    # chart's Service instead, via lapi.service.annotations in helmrelease.yaml.
+spec:
+  ingressClassName: traefik
+  rules:
+    - host: crowdsec.renner.uno
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: crowdsec-service
+                port:
+                  number: 8080
+  tls:
+    - secretName: crowdsec-lapi-public-tls
+      hosts:
+        - crowdsec.renner.uno

kubernetes/infrastructure/image-automation/paperless-ngx.yaml

@@ -7,7 +7,7 @@ spec:
   image: ghcr.io/paperless-ngx/paperless-ngx
   interval: 1h
 ---
-# Allow only patch + minor bumps within the 2.x line. A 3.0 release
+# Allow only patch + minor bumps within the 3.x line. A 4.0 release
 # requires manual policy widening — major upgrades typically need a
 # migration plan and shouldn't land via auto-merge.
 apiVersion: image.toolkit.fluxcd.io/v1
@@ -20,4 +20,4 @@ spec:
     name: paperless-ngx
   policy:
     semver:
-      range: "2.x"
+      range: "3.x"

kubernetes/infrastructure/network-policies/ai-ns.yaml

@@ -24,6 +24,27 @@ spec:
         - ports:
             - port: "8080"
               protocol: TCP
+    # Paperless calls both models for AI suggestions and RAG embeddings. The
+    # matching egress grant on its side is scoped to these two pods on :8080.
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": paperless
+            app.kubernetes.io/name: paperless
+      toPorts:
+        - ports:
+            - port: "8080"
+              protocol: TCP
+    # Home Assistant's llama.cpp conversation agent calls the chat model only —
+    # it has no use for the embedding endpoint, so unlike paperless above this
+    # grant does not extend to llama-embed.
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": home-assistant
+            app.kubernetes.io/name: home-assistant
+      toPorts:
+        - ports:
+            - port: "8080"
+              protocol: TCP
     - fromEndpoints:
         - matchLabels:
             "k8s:io.kubernetes.pod.namespace": monitoring

kubernetes/infrastructure/network-policies/home-assistant-ns.yaml

@@ -167,6 +167,18 @@ spec:
     - toEndpoints:
         - matchLabels:
             "k8s:io.kubernetes.pod.namespace": influxdb
+    # The cluster's own chat model, for the llama.cpp conversation agent that
+    # backs Assist. Scoped to the one pod and port rather than the ai namespace
+    # as a whole, so this grant cannot widen if something else is placed there —
+    # the embedding endpoint next to it is deliberately not included.
+    - toEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": ai
+            app.kubernetes.io/name: llama-server
+      toPorts:
+        - ports:
+            - port: "8080"
+              protocol: TCP
     # traefik for east-west OIDC roundtrips that go through the
     # public hostname via k8s_gateway.
     - toEndpoints: