feat(netbird): carry the WAN client address through the passthrough edge

TLS-passthrough services hand Traefik a raw TCP stream, so every WAN
request now logs the routing peer's pod IP (10.244.4.64/.112 in the
access log) instead of the client. CrowdSec's detection parses exactly
that log; with the HTTP-mode services gone it would attribute every
external request to two pod IPs, and a re-armed bouncer would repeat the
2026-06-27 mass ban.

The controller now sets options.proxy_protocol on every target (PROXY
protocol v2, TCP/TLS modes only per NetBird) and treats a service
without it as out of date, so the eleven existing services are updated
on the next pass. Traefik's websecure entrypoint accepts the header from
the pod CIDRs, mirroring forwardedHeaders.trustedIPs. Traefik takes
connections with or without a header from a trusted IP, so mesh and LAN
traffic that carries none is unaffected and the Traefik change can land
before the controller image.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

.woodpecker/build-ci-images.yml

@@ -89,7 +89,7 @@ steps:
           --destination gitea.renner.uno/gitea_admin/ci-k8s:latest
 
   - name: build-vmtest
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
     directory: ci/images/vmtest
     depends_on: []
     when:
@@ -349,3 +349,30 @@ steps:
           --dockerfile Containerfile \
           --destination gitea.renner.uno/gitea_admin/heating-can:$TAG \
           --destination gitea.renner.uno/gitea_admin/heating-can:latest
+
+  # netbird-proxy-services: the controller that reconciles NetBird reverse-proxy
+  # services (public Ingress hosts -> TLS passthrough -> Traefik ClusterIP).
+  # Runtime artifact like heating-can; same registry so the audit tracks it.
+  - name: build-netbird-proxy-services
+    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
+    directory: ci/images/netbird-proxy-services
+    depends_on: []
+    when:
+      - event: deployment
+      - event: push
+        path: ["ci/images/netbird-proxy-services/**", ".woodpecker/build-ci-images.yml"]
+    environment:
+      GITEA_USER: gitea_admin
+      GITEA_TOKEN:
+        from_secret: ci_images_token
+    commands:
+      - mkdir -p /kaniko/.docker
+      - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
+      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
+      - TAG=$(date +%Y-%m-%d)
+      - |
+        /kaniko/executor \
+          --context . \
+          --dockerfile Containerfile \
+          --destination gitea.renner.uno/gitea_admin/netbird-proxy-services:$TAG \
+          --destination gitea.renner.uno/gitea_admin/netbird-proxy-services:latest

ansible/roles/netbird-proxy-watchdog/defaults/main.yml

@@ -32,8 +32,13 @@ netbird_watchdog_fail_threshold: 3
 
 # Minimum seconds between watchdog-issued restarts, so a genuinely broken
 # backend can't put the proxy into a restart loop. Shared by both the URL-probe
-# watchdog and the flood watchdog below.
-netbird_watchdog_cooldown_seconds: 600
+# watchdog and the flood watchdog below. One minute: the route-manager wedge
+# recurs on every routing-peer loss, and rolling node reboots lose one peer
+# every few minutes; a suppressed restart there leaves the wedge standing
+# until the URL probe's three rounds fire (3.5 min of 000 on every domain,
+# measured 2026-09-07). A minute still caps a loop at one restart per
+# probe interval.
+netbird_watchdog_cooldown_seconds: 60
 
 # Log fingerprint of the reverse-proxy route-manager wedge, matched as a plain
 # substring in the proxy's docker logs by the flood watchdog. On this build the

ci/images/netbird-proxy-services/Containerfile

@@ -0,0 +1,12 @@
+# netbird-proxy-services: reconciles NetBird reverse-proxy services (public
+# domains -> TLS passthrough -> Traefik ClusterIP) from Ingresses carrying the
+# public label. Standard library only, so the image is the interpreter plus one
+# file; runs unprivileged as its own user.
+FROM python:3.13-slim
+
+RUN useradd --system --no-create-home --shell /usr/sbin/nologin controller
+COPY controller.py /opt/controller.py
+
+ENV PYTHONUNBUFFERED=1
+USER controller
+ENTRYPOINT ["python3", "/opt/controller.py"]

ci/images/netbird-proxy-services/controller.py

@@ -0,0 +1,275 @@
+#!/usr/bin/env python3
+"""netbird-proxy-services: reconcile NetBird reverse-proxy services from Ingresses.
+
+Source of truth: every Ingress carrying the public label (dns.renner.uno/public).
+For each host on such an Ingress the NetBird management gets one reverse-proxy
+service in TLS-passthrough mode on :443, targeting Traefik's ClusterIP through a
+subnet resource that covers the cluster's service CIDR. SNI and Host reach
+Traefik untouched, the proxy terminates nothing, and no domain resource is
+involved -- the routing peers stay ordinary IP routers. The target sends a
+PROXY protocol v2 header so Traefik (and CrowdSec behind its access log) sees
+the WAN client address instead of the routing peer's pod IP.
+
+Ownership is derived from NetBird itself, not from local state: a service is
+managed iff its mode is "tls" and its target is this controller's subnet
+resource. Services that do not match (the hand-made HTTP/domain ones) are never
+touched; a desired host held by such a service is reported and skipped until
+that service is removed by hand.
+
+Credentials: the NetBird PAT is read from OpenBao on every pass through the
+Kubernetes auth role, exactly like the setup-key rotator; it never lands in a
+Kubernetes Secret or in a log line.
+"""
+
+import json
+import logging
+import os
+import sys
+import time
+import urllib.error
+import urllib.parse
+import urllib.request
+
+LOG = logging.getLogger("netbird-proxy-services")
+
+NB = os.environ.get("NB_MANAGEMENT_URL", "https://netbird.renner.uno").rstrip("/")
+BAO_ADDR = os.environ.get("BAO_ADDR", "http://openbao.openbao.svc.cluster.local:8200").rstrip("/")
+BAO_ROLE = os.environ.get("BAO_ROLE", "netbird-proxy-services")
+BAO_PAT_PATH = os.environ.get("BAO_PAT_PATH", "homelab/data/infrastructure/secret-rotators-bootstrap/netbird")
+BAO_PAT_KEY = os.environ.get("BAO_PAT_KEY", "pat")
+SERVICE_CIDR = os.environ.get("SERVICE_CIDR", "10.96.0.0/12")
+NETWORK_NAME = os.environ.get("NETWORK_NAME", "k8s public ingress")
+ROUTER_GROUP = os.environ.get("ROUTER_GROUP_NAME", "k8s-ingress-router")
+SOURCE_GROUP = os.environ.get("SOURCE_GROUP_NAME", "All")
+TRAEFIK_NS, TRAEFIK_SVC = os.environ.get("TRAEFIK_SERVICE", "ingress/traefik").split("/", 1)
+PUBLIC_LABEL = os.environ.get("PUBLIC_LABEL", "dns.renner.uno/public")
+LISTEN_PORT = int(os.environ.get("LISTEN_PORT", "443"))
+INTERVAL = int(os.environ.get("INTERVAL_SECONDS", "60"))
+HTTP_TIMEOUT = 20
+
+K8S_API = "https://kubernetes.default.svc"
+K8S_TOKEN_FILE = "/var/run/secrets/kubernetes.io/serviceaccount/token"
+K8S_CA_FILE = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
+
+
+class ApiError(Exception):
+    pass
+
+
+def http(method, url, body=None, headers=None, cafile=None):
+    data = None if body is None else json.dumps(body).encode()
+    req = urllib.request.Request(url, data=data, method=method)
+    req.add_header("Accept", "application/json")
+    if data is not None:
+        req.add_header("Content-Type", "application/json")
+    for k, v in (headers or {}).items():
+        req.add_header(k, v)
+    ctx = None
+    if cafile:
+        import ssl
+        ctx = ssl.create_default_context(cafile=cafile)
+    try:
+        with urllib.request.urlopen(req, timeout=HTTP_TIMEOUT, context=ctx) as resp:
+            raw = resp.read()
+            return json.loads(raw) if raw else None
+    except urllib.error.HTTPError as e:
+        detail = e.read().decode(errors="replace")[:300]
+        raise ApiError(f"{method} {url} -> {e.code}: {detail}") from None
+
+
+# --- Kubernetes -------------------------------------------------------------
+
+def k8s(path):
+    with open(K8S_TOKEN_FILE) as f:
+        token = f.read().strip()
+    return http("GET", K8S_API + path, headers={"Authorization": f"Bearer {token}"}, cafile=K8S_CA_FILE)
+
+
+def desired_hosts():
+    sel = urllib.parse.quote(f"{PUBLIC_LABEL}=true")
+    ingresses = k8s(f"/apis/networking.k8s.io/v1/ingresses?labelSelector={sel}")
+    hosts = {}
+    for ing in ingresses.get("items", []):
+        ref = f'{ing["metadata"]["namespace"]}/{ing["metadata"]["name"]}'
+        for rule in ing.get("spec", {}).get("rules", []):
+            host = rule.get("host")
+            if not host or host.startswith("*"):
+                continue
+            hosts.setdefault(host, ref)
+    return hosts
+
+
+def traefik_cluster_ip():
+    svc = k8s(f"/api/v1/namespaces/{TRAEFIK_NS}/services/{TRAEFIK_SVC}")
+    for ip in svc["spec"].get("clusterIPs", []):
+        if ":" not in ip:
+            return ip
+    raise ApiError(f"{TRAEFIK_NS}/{TRAEFIK_SVC} has no IPv4 ClusterIP")
+
+
+# --- OpenBao ----------------------------------------------------------------
+
+def netbird_pat():
+    with open(K8S_TOKEN_FILE) as f:
+        jwt = f.read().strip()
+    login = http("POST", f"{BAO_ADDR}/v1/auth/kubernetes/login", {"role": BAO_ROLE, "jwt": jwt})
+    bao_token = login["auth"]["client_token"]
+    kv = http("GET", f"{BAO_ADDR}/v1/{BAO_PAT_PATH}", headers={"X-Vault-Token": bao_token})
+    pat = kv["data"]["data"].get(BAO_PAT_KEY)
+    if not pat:
+        raise ApiError(f"OpenBao {BAO_PAT_PATH} has no key {BAO_PAT_KEY}")
+    return pat
+
+
+# --- NetBird management -----------------------------------------------------
+
+class NetBird:
+    def __init__(self, pat):
+        self.h = {"Authorization": f"Token {pat}"}
+
+    def get(self, path):
+        return http("GET", f"{NB}/api/{path}", headers=self.h)
+
+    def list(self, path):
+        # An empty collection comes back as an empty body / null, not [].
+        return self.get(path) or []
+
+    def post(self, path, body):
+        return http("POST", f"{NB}/api/{path}", body, headers=self.h)
+
+    def put(self, path, body):
+        return http("PUT", f"{NB}/api/{path}", body, headers=self.h)
+
+    def delete(self, path):
+        return http("DELETE", f"{NB}/api/{path}", headers=self.h)
+
+
+def group_id(nb, name):
+    for g in nb.list("groups"):
+        if g["name"] == name:
+            return g["id"]
+    raise ApiError(f"NetBird group {name!r} not found")
+
+
+def ensure_network(nb):
+    for n in nb.list("networks"):
+        if n["name"] == NETWORK_NAME:
+            return n["id"]
+    n = nb.post("networks", {"name": NETWORK_NAME,
+                             "description": "managed by netbird-proxy-services; public ingress via TLS passthrough"})
+    LOG.info("created network %s (%s)", NETWORK_NAME, n["id"])
+    return n["id"]
+
+
+def ensure_resource(nb, net):
+    for r in nb.list(f"networks/{net}/resources"):
+        if r.get("address") == SERVICE_CIDR:
+            return r["id"]
+    r = nb.post(f"networks/{net}/resources", {"name": "cluster service CIDR", "address": SERVICE_CIDR,
+                                              "description": "managed by netbird-proxy-services",
+                                              "enabled": True, "groups": []})
+    LOG.info("created subnet resource %s (%s)", SERVICE_CIDR, r["id"])
+    return r["id"]
+
+
+def ensure_router(nb, net, router_group):
+    for r in nb.list(f"networks/{net}/routers"):
+        if router_group in (r.get("peer_groups") or []):
+            return r["id"]
+    r = nb.post(f"networks/{net}/routers", {"peer_groups": [router_group], "metric": 9999,
+                                            "masquerade": True, "enabled": True})
+    LOG.info("created router on group %s (%s)", ROUTER_GROUP, r["id"])
+    return r["id"]
+
+
+def ensure_policy(nb, resource, source_group):
+    name = f"{NETWORK_NAME} access"
+    for p in nb.list("policies"):
+        if p["name"] == name:
+            return p["id"]
+    p = nb.post("policies", {"name": name, "enabled": True, "rules": [{
+        "name": name, "enabled": True, "action": "accept", "protocol": "tcp",
+        "ports": [str(LISTEN_PORT)], "bidirectional": False, "sources": [source_group],
+        "destinationResource": {"id": resource, "type": "subnet"}}]})
+    LOG.info("created policy %s (%s)", name, p["id"])
+    return p["id"]
+
+
+def service_body(host, resource, target_ip):
+    # proxy_protocol: PROXY v2 header carries the WAN client address to
+    # Traefik (proxyProtocol.trustedIPs on websecure); TCP/TLS modes only.
+    return {"name": host, "domain": host, "mode": "tls", "listen_port": LISTEN_PORT, "enabled": True,
+            "targets": [{"target_type": "subnet", "target_id": resource, "protocol": "tcp",
+                         "host": target_ip, "port": 443, "enabled": True,
+                         "options": {"proxy_protocol": True}}]}
+
+
+def is_managed(svc, resource):
+    return svc.get("mode") == "tls" and any(t.get("target_id") == resource for t in svc.get("targets") or [])
+
+
+def target_matches(svc, target_ip):
+    if svc.get("listen_port") != LISTEN_PORT or not svc.get("enabled"):
+        return False
+    ts = svc.get("targets") or []
+    return len(ts) == 1 and ts[0].get("host") == target_ip and ts[0].get("port") == 443 \
+        and ts[0].get("protocol") == "tcp" and ts[0].get("enabled") \
+        and (ts[0].get("options") or {}).get("proxy_protocol") is True
+
+
+def reconcile():
+    hosts = desired_hosts()
+    target_ip = traefik_cluster_ip()
+    nb = NetBird(netbird_pat())
+
+    router_group = group_id(nb, ROUTER_GROUP)
+    source_group = group_id(nb, SOURCE_GROUP)
+    net = ensure_network(nb)
+    resource = ensure_resource(nb, net)
+    ensure_router(nb, net, router_group)
+    ensure_policy(nb, resource, source_group)
+
+    by_domain = {s["domain"]: s for s in nb.list("reverse-proxies/services")}
+    created = updated = deleted = held = 0
+
+    for host, ref in sorted(hosts.items()):
+        svc = by_domain.get(host)
+        if svc is None:
+            nb.post("reverse-proxies/services", service_body(host, resource, target_ip))
+            LOG.info("created %s (from %s) -> %s:443", host, ref, target_ip)
+            created += 1
+        elif not is_managed(svc, resource):
+            LOG.warning("%s is held by unmanaged service %s (mode %s); remove it to migrate",
+                        host, svc["id"], svc.get("mode"))
+            held += 1
+        elif not target_matches(svc, target_ip):
+            nb.put(f"reverse-proxies/services/{svc['id']}", service_body(host, resource, target_ip))
+            LOG.info("updated %s -> %s:443", host, target_ip)
+            updated += 1
+
+    for domain, svc in by_domain.items():
+        if is_managed(svc, resource) and domain not in hosts:
+            nb.delete(f"reverse-proxies/services/{svc['id']}")
+            LOG.info("deleted %s (no public Ingress carries it any more)", domain)
+            deleted += 1
+
+    LOG.info("reconciled: %d hosts desired, %d created, %d updated, %d deleted, %d held by unmanaged services",
+             len(hosts), created, updated, deleted, held)
+
+
+def main():
+    logging.basicConfig(level=os.environ.get("LOG_LEVEL", "INFO"), stream=sys.stdout,
+                        format="%(asctime)s %(levelname)s %(message)s")
+    if os.environ.get("ONCE") == "1":
+        reconcile()
+        return
+    while True:
+        try:
+            reconcile()
+        except Exception as e:  # keep running; the next pass retries
+            LOG.error("reconcile failed: %s", e)
+        time.sleep(INTERVAL)
+
+
+if __name__ == "__main__":
+    main()

kubernetes/PROPOSAL-node-roll-downtime.md

@@ -1,304 +0,0 @@
-# Proposal — a node roll without the ingress hiccup
-
-Status: draft, grounded in one measured roll. On 2026-09-07 all three MS-01
-were drained, patched and rebooted back to back (06:52–07:08) for the SAN
-rollout. Everything below is read off that window: Traefik access logs and
-Cilium BGP logs in Loki, the drain logs, and the live cluster afterwards.
-
-The short version: there are two ingresses and they failed differently.
-**The LAN ingress (Traefik behind the BGP VIP) did not hiccup** — it
-answered every request; the 5xx it returned were for backends mid-move.
-**The public edge (proxies → NetBird mesh → Traefik) went dark for 40–80 s
-on two of the three rolls, and would have for 3½ minutes on the third had
-its routes not recovered by luck** — a known client bug the repo already
-documents and works around with a watchdog, whose 600 s restart cooldown is
-what turns a 3-second blip into minutes when rolls follow each other. So
-this is two projects: make evictions cheap, and make a peer loss on the
-mesh cost seconds, deterministically.
-
-## What was measured
-
-Traefik, 06:48–07:12, 24 minutes, 3 391 requests, of which 5xx:
-
-| router | status | count |
-|---|---|---|
-| gitea | 503 | 56 |
-| home-assistant | 503 | 36 |
-| shelly-manager | 502 | 25 |
-| immich | 503 | 19 |
-| openbao | 503 | 13 |
-| grafana | 503 | 10 |
-| music-assistant | 503 | 9 |
-| crowdsec-lapi | 503 | 7 |
-| wordpress | 503 | 6 |
-| woodpecker | 503 | 4 |
-
-The quiet 24 minutes before the roll had zero 5xx. Every entry above is a
-single-replica workload (or its single database) that was evicted and had to
-come up on another node — pod reschedule, RBD volume detach and re-attach,
-process start. That is 30–90 s of 503 per app per drain.
-
-Two things made it worse than one move per app:
-
-- **Double moves.** The drain of k8s-1 pushed pods onto k8s-2, which was
-  drained next: `ha-pg-1`, `ak-pg-1`, `im-pg-1`, `pprl-pg-2`, `tr-pg-1`,
-  `wp-pg-1`, `influxdb`, `protonmail-bridge`, `prometheus`, `wordpress-mariadb`
-  each appear in two drain logs. Half the downtime was paid twice.
-- **`--disable-eviction`.** Ten PodDisruptionBudgets allow zero disruptions
-  (every CNPG primary, gitea, influxdb, syncthing, protonmail-bridge,
-  woodpecker-server). A plain `kubectl drain` hangs on them, so the roll used
-  `--disable-eviction`, which deletes pods outright. For the two-instance CNPG
-  clusters (authentik, immich, paperless) that skipped the switchover CNPG
-  would have done: the operator's documented behaviour is "if a node is to be
-  drained and contains a cluster's primary instance, a switchover happens
-  ahead of the drain" — but only when the drain goes through the eviction API.
-  We turned a 2-second switchover into a 30–60 s primary restart.
-
-Beside the HTTP path, one reboot side effect took a dependency down for
-half an hour: openbao-0 came back **sealed**, and because the StatefulSet is
-`OrderedReady`, openbao-1/2 were never recreated after their eviction until
-openbao-0 was unsealed by hand. ESO could not refresh a secret for 35 minutes.
-
-### The public edge
-
-Public traffic reaches Traefik with client address `127.0.0.1` (the rootless
-netbird-ingress peer forwards from inside its pod through socket-LB), LAN
-traffic with `10.2.x`. That splits the access log by path. At that hour the
-only public traffic was the proxies' own watchdog — two hosts probed once a
-minute from each of two proxies, a flat 4 requests/min — which makes it a
-clean synthetic probe of the whole external chain:
-
-| minute (UTC) | LAN req | public req | what was happening |
-|---|---|---|---|
-| 04:55 | 128 | 4 | k8s-1 rebooting (04:55:10) |
-| 04:56 | 109 | 2 | k8s-1 back, its peer re-enrolled 04:56:24 |
-| 04:57 | 127 | **0** | — |
-| 04:58 | 126 | 4 | proxies restarted by their watchdog |
-| 05:00–05:03 | 186–233 | 4 | k8s-2 roll: **no dip** |
-| 05:06–05:07 | 97–135 | 2 | k8s-3 roll: half |
-
-The external chain was down for roughly two minutes at the k8s-1 roll, one
-at k8s-3, and not at all at k8s-2. The mechanism is in
-`ansible/roles/netbird-proxy-watchdog/defaults/main.yml`: the proxy's
-embedded NetBird client pins each domain route to one routing-peer
-*identity*; the netbird-ingress DaemonSet enrols with a setup key, so every
-pod start is a **new identity** (the k8s-1 pod registered fresh at
-04:56:24); and upstream's dns-interceptor never rebinds a route whose
-owning peer is gone ("already routed by peer … HA routing disabled"). The
-proxy then serves 502 for every domain until its container restarts —
-which the watchdog does after three failed one-minute probes, or at once
-when the flood watchdog sees the wedge in the proxy log. Which roll hurts
-depends on which of the three peers the proxies happened to pin. k8s-2's
-peer was pinned by neither.
-
-Note what did *not* help: the DaemonSet's preStop (`netbird down`) took the
-peer out of the mesh cleanly at 04:55:11, and the peer was back 73 s later.
-Neither matters to a client that will not rebind and will not recognise
-the returning pod as the same peer.
-
-## What was not the cause
-
-**BGP.** Per node the UDM sessions were down for 47–51 s (Peer Down at the
-Cilium shutdown, Peer Up ~50 s later). That gap did not carry traffic: every
-LoadBalancer here is `externalTrafficPolicy: Local`, so a VIP is only
-advertised from nodes with a local endpoint, and the drain had already
-evicted those endpoints — Cilium withdrew the routes while the session was
-still healthy (the "soft reset out" bursts during each drain). The 90 s hold
-time and 120 s graceful-restart window never came into play. Leave them.
-
-**Traefik.** Two replicas on k8s-1/k8s-2, one evicted per roll, the other
-carried the VIP alone. No PDB, no spread constraint, no preStop — which held
-by luck (the two pods happened to sit on different nodes), and is fixed below
-because luck is not a design.
-
-## Target state, in order of measured impact
-
-### 1. Drain in a way that lets operators do their job (procedure only)
-
-- **Cordon the next node before draining the current one.** Evicted pods
-  then land on the node that stays up, and nothing moves twice. Order:
-  cordon B → drain A → reboot A → uncordon A → cordon C → drain B → …
-- **Never `--disable-eviction`.** It exists in the roll only because ten
-  PDBs say "never". Stage 2 makes those PDBs honest, after which a plain
-  `kubectl drain --ignore-daemonsets --delete-emptydir-data` completes and
-  CNPG switches primaries over before its replicas are evicted.
-- **After every reboot, three gates before uncordon:** `00-multus.conf` on
-  the host and Multus + whereabouts Running (k8s-1's OSD came up
-  Cilium-only today because it beat the Multus DaemonSet by 30 s); OpenBao
-  unsealed (`bao status` on every pod that exists, then the ones the
-  StatefulSet creates after it); Ceph back at baseline.
-- This is `rollout-node.sh` from today's window, minus `--disable-eviction`,
-  plus the OpenBao gate, checked into `scripts/`. A reboot runbook that lives
-  in a scratchpad is a runbook nobody else can run.
-
-### 2. PodDisruptionBudgets that mean what they say
-
-A `minAvailable: 1` PDB on a single-replica workload does not protect
-anything; it only forces the operator to bypass all PDBs. Two changes:
-
-- **CNPG single-instance clusters** (grafana-db, home-assistant, trmnl,
-  woodpecker): `spec.enablePDB: false`. CNPG documents exactly this — "for
-  single-instance clusters the operator will prevent draining the node" and
-  recommends moving to direct control of disruption budgets. The instance
-  restarts on drain (it does today too, just through the back door), and the
-  multi-instance clusters keep their switchover-before-eviction behaviour.
-- **gitea, influxdb, syncthing, protonmail-bridge, woodpecker-server**: drop
-  the `minAvailable: 1` PDBs. They are single replicas with RWO volumes; the
-  PDB cannot buy availability, only stalls.
-
-### 3. The ingress itself: three replicas, one per node, drained gracefully
-
-Traefik goes to `replicas: 3` with a required `podAntiAffinity` on
-hostname, `PodDisruptionBudget maxUnavailable: 1`, and a preStop that lets
-in-flight requests finish before the endpoint is withdrawn:
-
-```yaml
-# chart keys as in traefik/values.yaml
-deployment:
-  replicas: 3
-  lifecycle:
-    preStop:
-      sleep:
-        seconds: 5
-affinity:
-  podAntiAffinity:
-    requiredDuringSchedulingIgnoredDuringExecution:
-      - topologyKey: kubernetes.io/hostname
-        labelSelector:
-          matchLabels:
-            app.kubernetes.io/name: traefik
-podDisruptionBudget:
-  enabled: true
-  maxUnavailable: 1
-ports:
-  websecure:
-    transport:
-      lifeCycle:
-        graceTimeOut: 15s
-```
-
-With `Local` traffic policy every node then advertises the VIP; a roll takes
-one of three ECMP paths away instead of one of two, and the UDM sees the
-withdrawal before the pod stops accepting. Also gives the UDM a route from
-k8s-3, which today advertises nothing for the ingress.
-
-### 4. Databases the apps actually wait on
-
-The 503 table is a priority list. Each of these is a single Postgres today:
-
-| app | 5xx | cluster | today |
-|---|---|---|---|
-| home-assistant | 36 | ha-pg | `instances: 1` |
-| grafana | 10 | gf-pg | `instances: 1` |
-| woodpecker | 4 | wp-pg | `instances: 1` |
-| trmnl | — | tr-pg | `instances: 1` |
-
-`instances: 2` for ha-pg and gf-pg turns a drain into a CNPG switchover
-(seconds). The cost is one more Postgres pod and a second RBD volume each.
-trmnl and woodpecker can stay at one: they were barely visible.
-
-gitea (56) is the largest number and has no cheap answer: one StatefulSet,
-one RWO volume, an application that does not run two instances against the
-same data directory without shared storage. It moves once per roll (with
-stage 1) and that is its floor unless its repositories move to CephFS.
-
-### 5. The public edge: what the proxies' own journals say
-
-This is the only *complete* outage in the window, so despite the numbering
-it ships right after stage 1. It was measured twice: from the proxies'
-watchdog journals for the morning roll, and in a controlled afternoon
-experiment (three routing-peer kills, probed from proxy-2 itself at 2 s —
-the LAN cannot probe this path, the UDM hairpin-DNATs the proxy addresses
-straight to the VIP). Proxy clocks run ~3 min behind the cluster.
-
-**What a peer loss does.** When the routing peer the proxy has pinned
-disappears, the proxy's route manager wedges (`couldn't fetch peer state`
-burst) and the *whole* process stops answering — every domain, and in the
-experiment also services that never used a domain route. The flood
-watchdog restarts the container within 2 s of the burst; the probe saw
-one failed round (≈3 s) at the first two kills. The third kill fell inside
-the flood watchdog's **600 s restart cooldown**: restart suppressed, wedge
-left standing, and the URL watchdog needed its three one-minute rounds —
-**3½ minutes of 000 on every public domain**. The morning roll had the
-same shape: k8s-1 (both proxies restarted, 40–80 s), k8s-2 (wedge at
-06:58, restart suppressed on both proxies, 278 s after the previous one —
-the routes happened to recover on their own), k8s-3 (restart, ~10 s).
-Three rolls in sixteen minutes guarantee the suppressed case.
-
-- **5a. Cooldown 600 s → 60 s** in
-  `ansible/roles/netbird-proxy-watchdog/defaults/main.yml`. The cooldown
-  exists so a genuinely broken backend cannot restart-loop the proxy; a
-  minute still does that (the URL watchdog needs three minutes to fire
-  anyway). With it, every peer loss — rolls back to back included — is a
-  ~3 s blip instead of a lottery between 3 s and 3½ min. Cheapest change
-  in this document, ships first.
-- **5b. Persist the peer identity.** Replace the DaemonSet with a
-  StatefulSet of three, required anti-affinity on hostname,
-  `PodDisruptionBudget maxUnavailable: 1`, and a small RBD volume per
-  replica holding NetBird's state directory. A drain then evicts the pod
-  through the PDB (the DaemonSet pod is not evicted at all today, it dies
-  with the reboot) and it returns within ~30 s **as the same peer**. Whether
-  the route manager then un-wedges without a restart is the hypothesis to
-  measure; the setup-key constraint (ephemeral peers reaped after >10 min
-  offline, NeedsLogin liveness restart) holds unchanged.
-- **5c. The structural exit: no domain routes at all.** The subnet-target
-  experiment settled what works: an HTTP-mode service with an IP target
-  forwards `Host: <ip>` and Traefik answers 404 (NetBird rejects a Host
-  override); an HTTP-mode service with a domain target needs mesh DNS on
-  the proxy, which answers SERVFAIL there; a **TLS-passthrough service
-  (`mode: tls`, `listen_port: 443`) with a subnet target works** — SNI and
-  Host reach Traefik untouched, it coexists with the HTTP listener on 443,
-  no `skip_tls_verify`. If every public service became such a passthrough
-  over a subnet resource covering the Cilium LB pool (`10.2.99.0/24`,
-  `fd99::/64`), there would be no domain resource left to wedge on, and
-  peer loss would be NetBird's documented HA failover (seconds). Costs: the
-  target's `host` is the VIP, so a reconciler (the external-dns-unifi
-  pattern, `PUT /api/reverse-proxies/services/{id}` from the Traefik
-  Service's `status.loadBalancer.ingress`) has to own it; the proxy's L7
-  features (auth, path rules) are lost — none are in use; and the pure
-  failover time is unmeasured, because today every kill wedged the domain
-  routes first. Decide after 5a and 5b have been measured on a real roll.
-- **Track the upstream bug.** The repo's note dates the dns-interceptor
-  behaviour to 2026-07; a search of NetBird's releases and issues turned up
-  no fix for the "already routed by peer / HA routing disabled" path.
-  Re-check on each NetBird bump.
-- **Side findings, each its own small change:** both proxy clocks are
-  ~3 min slow (the proxy logs SCT timestamps "in the future"; TLS
-  validity windows care) — NTP on proxy-1/proxy-2; a new service's
-  certificate reaches the IPv6-only member only after ~20 min of ACME
-  retries (Let's Encrypt validates over IPv4 and lands on the other member,
-  which has no token for it); proxy-1 and proxy-2 do not ship syslog to
-  Loki (edge-1 does), so every public-edge timeline is reconstructed over
-  SSH — point them at the alloy-syslog VIP.
-
-## Expected result
-
-| stage | removes |
-|---|---|
-| 1 | every double move (half of today's 5xx), the primary restarts, the 35-minute OpenBao/ESO outage, the Multus race |
-| 2 | the need for `--disable-eviction` at all |
-| 3 | the single-node ingress exposure; ECMP loses 1/3 instead of 1/2 |
-| 4 | home-assistant and grafana 503s entirely |
-| 5a | the 3½-minute public-edge blackout of a second peer loss within 10 min; every peer loss becomes a ~3 s restart |
-| 5b/5c | the restart itself, if the returning identity (5b) or the absence of domain routes (5c) proves to un-wedge — to be measured |
-
-What remains after all of it: one move per single-replica app per roll
-(gitea, immich, shelly-manager, music-assistant, wordpress), 30–60 s each,
-once, and a few seconds of public edge per roll. That is the honest floor
-for single-replica services on RWO storage and a mesh client that pins.
-
-## Transition
-
-1. Stage 5a is one Ansible default and a proxy deploy — ships first.
-2. Stage 1 is a script and a runbook change — no cluster change.
-3. Stage 5b's StatefulSet changes how the public edge enrols, so it gets
-   its own roll to measure, probed from a proxy.
-4. Stage 2 is one field on four CNPG clusters and five deleted PDBs; each
-   Kustomization applies without disruption.
-5. Stage 3 is a Traefik HelmRelease change; the rollout itself is a rolling
-   update under the new PDB.
-6. Stage 4 is `instances: 2` on two clusters; CNPG adds the replica online.
-7. Stage 5c only if 5a+5b leave a measurable public gap.
-
-Measure the next roll the same way: 5xx by router, the proxies' watchdog
-journals (or their syslog in Loki), BGP Down→Up gaps, pods per drain log.

kubernetes/infrastructure/crowdsec/lapi-ingress.yaml

@@ -31,22 +31,12 @@
 # serves read-only decision streams to a valid bouncer key and nothing else.
 # The admin surface is cscli, in-cluster. If the source address ever becomes
 # deterministic, an ipAllowList can be added on top.
----
+#
 # The LAPI serves HTTPS only once tls.enabled is set — there is no plaintext
 # fallback and no second port; 8080 is the TLS port. Its certificate comes
 # from the chart's own self-signed CA, so Traefik needs that CA to verify the
-# backend. SANs include crowdsec-service.crowdsec.svc.cluster.local, which is
-# what the Service resolves to.
-apiVersion: traefik.io/v1alpha1
-kind: ServersTransport
-metadata:
-  name: crowdsec-lapi
-  namespace: crowdsec
-spec:
-  serverName: crowdsec-service.crowdsec.svc.cluster.local
-  rootCAs:
-    - secret: crowdsec-root-secret
----
+# backend; the ServersTransport carrying it is a traefik.io CRD consumer and
+# lives in post-crds/crowdsec-lapi-serverstransport.yaml.
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:

kubernetes/infrastructure/kustomization.yaml

@@ -18,6 +18,7 @@ resources:
   - cloudnative-pg/
   - mariadb-operator/
   - netbird-ingress/
+  - netbird-proxy-services/
   - coredns-config/
   - k8s-gateway/
   - external-dns-unifi/

kubernetes/infrastructure/netbird-proxy-services/deployment.yaml

@@ -0,0 +1,104 @@
+# Reconciles NetBird reverse-proxy services from Ingresses that carry
+# dns.renner.uno/public: one TLS-passthrough service per host on :443,
+# targeting Traefik's ClusterIP through a subnet resource over the service
+# CIDR (ci/images/netbird-proxy-services/controller.py).
+#
+# openbao namespace, like the rotators: its default-deny already allows egress
+# to OpenBao, DNS, the apiserver and `world` (netbird.renner.uno), and the
+# NetBird PAT is read from OpenBao on every pass through the
+# netbird-proxy-services role — policies/netbird-proxy-services.hcl, applied
+# once by hand like the rotator roles. Until that role exists the pod logs an
+# auth error each minute and does nothing.
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: netbird-proxy-services
+  namespace: openbao
+  labels:
+    app.kubernetes.io/name: netbird-proxy-services
+---
+# Read-only, cluster-wide: public Ingresses live in many namespaces, the
+# Traefik Service in ingress/.
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: netbird-proxy-services-read
+rules:
+  - apiGroups: ["networking.k8s.io"]
+    resources: ["ingresses"]
+    verbs: ["get", "list", "watch"]
+  - apiGroups: [""]
+    resources: ["services"]
+    verbs: ["get"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: netbird-proxy-services-read
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: netbird-proxy-services-read
+subjects:
+  - kind: ServiceAccount
+    name: netbird-proxy-services
+    namespace: openbao
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: netbird-proxy-services
+  namespace: openbao
+  labels:
+    app.kubernetes.io/name: netbird-proxy-services
+spec:
+  replicas: 1
+  # One writer against the management API at a time.
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: netbird-proxy-services
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: netbird-proxy-services
+    spec:
+      serviceAccountName: netbird-proxy-services
+      securityContext:
+        runAsNonRoot: true
+        # Numeric, or the kubelet refuses the image's named user with
+        # "cannot verify user is non-root". The script needs no home and no
+        # writable path, so nobody/nogroup is enough.
+        runAsUser: 65534
+        runAsGroup: 65534
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: controller
+          image: gitea.renner.uno/gitea_admin/netbird-proxy-services:2026-09-07
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop: ["ALL"]
+          env:
+            - name: NB_MANAGEMENT_URL
+              value: https://netbird.renner.uno
+            - name: BAO_ADDR
+              value: http://openbao.openbao.svc.cluster.local:8200
+            - name: BAO_ROLE
+              value: netbird-proxy-services
+            # Must match cluster.network.serviceSubnets in talos/patches/cluster.yaml.
+            - name: SERVICE_CIDR
+              value: "10.96.0.0/12"
+            - name: TRAEFIK_SERVICE
+              value: ingress/traefik
+            - name: INTERVAL_SECONDS
+              value: "60"
+          resources:
+            requests:
+              cpu: 10m
+              memory: 32Mi
+            limits:
+              memory: 64Mi

kubernetes/infrastructure/netbird-proxy-services/kustomization.yaml

@@ -0,0 +1,4 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - deployment.yaml

kubernetes/infrastructure/openbao/policies/netbird-proxy-services.hcl

@@ -0,0 +1,23 @@
+# OpenBao policy: netbird-proxy-services
+#
+# The reverse-proxy-services controller (Deployment netbird-proxy-services in
+# the openbao namespace) reads the NetBird PAT on every reconcile pass and
+# calls the management API with it. Read-only on that one path; it shares the
+# bootstrap credential with rotator-netbird but not that role's PATCH rights.
+#
+# NOT applied by Flux. Apply ONCE with an admin token:
+#
+#   bao policy write netbird-proxy-services kubernetes/infrastructure/openbao/policies/netbird-proxy-services.hcl
+#   bao write auth/kubernetes/role/netbird-proxy-services \
+#     bound_service_account_names=netbird-proxy-services \
+#     bound_service_account_namespaces=openbao \
+#     policies=netbird-proxy-services ttl=15m
+#
+# The PAT itself is the one rotator-netbird already uses
+# (homelab/infrastructure/secret-rotators-bootstrap/netbird, key `pat`).
+#
+# Do NOT set token_no_default_policy=true (kv-v2 preflight needs the default policy).
+
+path "homelab/data/infrastructure/secret-rotators-bootstrap/netbird" {
+  capabilities = ["read"]
+}

kubernetes/infrastructure/post-crds/kustomization.yaml

@@ -31,7 +31,5 @@ resources:
   - traefik-crowdsec-middleware.yaml
   - traefik-rate-limit-middleware.yaml
   - traefik-security-headers-middleware.yaml
-  # Phase 1 of a two-Kustomization move: also still in crowdsec/lapi-ingress.yaml
-  # until this layer has applied it once, so prune never runs ahead of apply.
   - crowdsec-lapi-serverstransport.yaml
   - ../network-policies/

kubernetes/infrastructure/traefik/helmrelease.yaml

@@ -93,13 +93,17 @@ spec:
         # Real client-IP visibility. The netbird-ingress routing peer
         # re-originates every mesh + WAN-via-proxy connection from its own
         # pod IP (rootless userspace WireGuard), so Traefik's TCP peer for
-        # that traffic is always a pod IP (e.g. fd10:244::2de1) — collapsing
-        # all external clients to one source. The NetBird reverse-proxy
-        # (HTTP mode, on the VPS) sets X-Forwarded-For with the real WAN
-        # client IP; trust the pod CIDRs so Traefik reads it instead. Without
-        # this, a CrowdSec bouncer bans the shared pod IP and 403s everyone
-        # (the 2026-06-27 IPv6 outage). Harmless no-op when no XFF is present
-        # (direct LAN/mesh) — those keep their RemoteAddr.
+        # that traffic is always a pod IP — collapsing all external clients
+        # to one source. The NetBird reverse-proxy services are TLS
+        # passthrough and carry the WAN client address in a PROXY protocol
+        # v2 header (options.proxy_protocol, set by netbird-proxy-services);
+        # X-Forwarded-For stays trusted for HTTP-mode services. Both are
+        # accepted from the pod CIDRs only; connections without a header
+        # (direct LAN/mesh) keep their RemoteAddr.
+        proxyProtocol:
+          trustedIPs:
+            - "10.244.0.0/16"
+            - "fd10:244::/56"
         forwardedHeaders:
           trustedIPs:
             - "10.244.0.0/16"