docs(proposal): public edge measured from the proxies — cooldown is the lever

Watchdog journals on proxy-1/proxy-2 for the morning roll, plus a
controlled experiment (three routing-peer kills probed from proxy-2 at
2 s; the LAN cannot probe this path, the UDM hairpin-DNATs the proxy
addresses to the VIP):

- a pinned-peer loss wedges the whole proxy process; the flood watchdog
  restarts it within 2 s, the probe sees one failed round (~3 s)
- inside the flood watchdog 600 s cooldown the restart is suppressed and
  the URL watchdog needs three one-minute rounds: 3.5 min of 000 on every
  public domain, measured; the morning k8s-2 roll hit the same suppression
- subnet-target services: HTTP mode + IP target -> Host header is the IP,
  Traefik 404; HTTP mode + domain target -> SERVFAIL, no mesh DNS on the
  proxy; TLS passthrough (mode tls, listen_port 443) + subnet target works
  end to end and coexists on 443 -- but dies with the wedge as long as any
  domain resource exists

Stage 5 rewritten: 5a cooldown 600->60 s (ships first), 5b identity
persistence as a hypothesis to measure, 5c all-passthrough as the
structural exit with its VIP reconciler cost. Side findings: proxy clocks
3 min slow, ACME on the v6-only member ~20 min late, no proxy syslog in
Loki.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

.woodpecker/build-ci-images.yml

@@ -89,7 +89,7 @@ steps:
           --destination gitea.renner.uno/gitea_admin/ci-k8s:latest
 
   - name: build-vmtest
-    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
+    image: gcr.io/kaniko-project/executor:v1.23.2-debug
     directory: ci/images/vmtest
     depends_on: []
     when:
@@ -349,30 +349,3 @@ steps:
           --dockerfile Containerfile \
           --destination gitea.renner.uno/gitea_admin/heating-can:$TAG \
           --destination gitea.renner.uno/gitea_admin/heating-can:latest
-
-  # netbird-proxy-services: the controller that reconciles NetBird reverse-proxy
-  # services (public Ingress hosts -> TLS passthrough -> Traefik ClusterIP).
-  # Runtime artifact like heating-can; same registry so the audit tracks it.
-  - name: build-netbird-proxy-services
-    image: registry.gitlab.com/gitlab-ci-utils/container-images/kaniko:v1.25.18-debug
-    directory: ci/images/netbird-proxy-services
-    depends_on: []
-    when:
-      - event: deployment
-      - event: push
-        path: ["ci/images/netbird-proxy-services/**", ".woodpecker/build-ci-images.yml"]
-    environment:
-      GITEA_USER: gitea_admin
-      GITEA_TOKEN:
-        from_secret: ci_images_token
-    commands:
-      - mkdir -p /kaniko/.docker
-      - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
-      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
-      - TAG=$(date +%Y-%m-%d)
-      - |
-        /kaniko/executor \
-          --context . \
-          --dockerfile Containerfile \
-          --destination gitea.renner.uno/gitea_admin/netbird-proxy-services:$TAG \
-          --destination gitea.renner.uno/gitea_admin/netbird-proxy-services:latest

ansible/roles/netbird-proxy-watchdog/defaults/main.yml

@@ -32,13 +32,8 @@ netbird_watchdog_fail_threshold: 3
 
 # Minimum seconds between watchdog-issued restarts, so a genuinely broken
 # backend can't put the proxy into a restart loop. Shared by both the URL-probe
-# watchdog and the flood watchdog below. One minute: the route-manager wedge
-# recurs on every routing-peer loss, and rolling node reboots lose one peer
-# every few minutes; a suppressed restart there leaves the wedge standing
-# until the URL probe's three rounds fire (3.5 min of 000 on every domain,
-# measured 2026-09-07). A minute still caps a loop at one restart per
-# probe interval.
-netbird_watchdog_cooldown_seconds: 60
+# watchdog and the flood watchdog below.
+netbird_watchdog_cooldown_seconds: 600
 
 # Log fingerprint of the reverse-proxy route-manager wedge, matched as a plain
 # substring in the proxy's docker logs by the flood watchdog. On this build the

ci/images/netbird-proxy-services/Containerfile

@@ -1,12 +0,0 @@
-# netbird-proxy-services: reconciles NetBird reverse-proxy services (public
-# domains -> TLS passthrough -> Traefik ClusterIP) from Ingresses carrying the
-# public label. Standard library only, so the image is the interpreter plus one
-# file; runs unprivileged as its own user.
-FROM python:3.13-slim
-
-RUN useradd --system --no-create-home --shell /usr/sbin/nologin controller
-COPY controller.py /opt/controller.py
-
-ENV PYTHONUNBUFFERED=1
-USER controller
-ENTRYPOINT ["python3", "/opt/controller.py"]

ci/images/netbird-proxy-services/controller.py

@@ -1,275 +0,0 @@
-#!/usr/bin/env python3
-"""netbird-proxy-services: reconcile NetBird reverse-proxy services from Ingresses.
-
-Source of truth: every Ingress carrying the public label (dns.renner.uno/public).
-For each host on such an Ingress the NetBird management gets one reverse-proxy
-service in TLS-passthrough mode on :443, targeting Traefik's ClusterIP through a
-subnet resource that covers the cluster's service CIDR. SNI and Host reach
-Traefik untouched, the proxy terminates nothing, and no domain resource is
-involved -- the routing peers stay ordinary IP routers. The target sends a
-PROXY protocol v2 header so Traefik (and CrowdSec behind its access log) sees
-the WAN client address instead of the routing peer's pod IP.
-
-Ownership is derived from NetBird itself, not from local state: a service is
-managed iff its mode is "tls" and its target is this controller's subnet
-resource. Services that do not match (the hand-made HTTP/domain ones) are never
-touched; a desired host held by such a service is reported and skipped until
-that service is removed by hand.
-
-Credentials: the NetBird PAT is read from OpenBao on every pass through the
-Kubernetes auth role, exactly like the setup-key rotator; it never lands in a
-Kubernetes Secret or in a log line.
-"""
-
-import json
-import logging
-import os
-import sys
-import time
-import urllib.error
-import urllib.parse
-import urllib.request
-
-LOG = logging.getLogger("netbird-proxy-services")
-
-NB = os.environ.get("NB_MANAGEMENT_URL", "https://netbird.renner.uno").rstrip("/")
-BAO_ADDR = os.environ.get("BAO_ADDR", "http://openbao.openbao.svc.cluster.local:8200").rstrip("/")
-BAO_ROLE = os.environ.get("BAO_ROLE", "netbird-proxy-services")
-BAO_PAT_PATH = os.environ.get("BAO_PAT_PATH", "homelab/data/infrastructure/secret-rotators-bootstrap/netbird")
-BAO_PAT_KEY = os.environ.get("BAO_PAT_KEY", "pat")
-SERVICE_CIDR = os.environ.get("SERVICE_CIDR", "10.96.0.0/12")
-NETWORK_NAME = os.environ.get("NETWORK_NAME", "k8s public ingress")
-ROUTER_GROUP = os.environ.get("ROUTER_GROUP_NAME", "k8s-ingress-router")
-SOURCE_GROUP = os.environ.get("SOURCE_GROUP_NAME", "All")
-TRAEFIK_NS, TRAEFIK_SVC = os.environ.get("TRAEFIK_SERVICE", "ingress/traefik").split("/", 1)
-PUBLIC_LABEL = os.environ.get("PUBLIC_LABEL", "dns.renner.uno/public")
-LISTEN_PORT = int(os.environ.get("LISTEN_PORT", "443"))
-INTERVAL = int(os.environ.get("INTERVAL_SECONDS", "60"))
-HTTP_TIMEOUT = 20
-
-K8S_API = "https://kubernetes.default.svc"
-K8S_TOKEN_FILE = "/var/run/secrets/kubernetes.io/serviceaccount/token"
-K8S_CA_FILE = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
-
-
-class ApiError(Exception):
-    pass
-
-
-def http(method, url, body=None, headers=None, cafile=None):
-    data = None if body is None else json.dumps(body).encode()
-    req = urllib.request.Request(url, data=data, method=method)
-    req.add_header("Accept", "application/json")
-    if data is not None:
-        req.add_header("Content-Type", "application/json")
-    for k, v in (headers or {}).items():
-        req.add_header(k, v)
-    ctx = None
-    if cafile:
-        import ssl
-        ctx = ssl.create_default_context(cafile=cafile)
-    try:
-        with urllib.request.urlopen(req, timeout=HTTP_TIMEOUT, context=ctx) as resp:
-            raw = resp.read()
-            return json.loads(raw) if raw else None
-    except urllib.error.HTTPError as e:
-        detail = e.read().decode(errors="replace")[:300]
-        raise ApiError(f"{method} {url} -> {e.code}: {detail}") from None
-
-
-# --- Kubernetes -------------------------------------------------------------
-
-def k8s(path):
-    with open(K8S_TOKEN_FILE) as f:
-        token = f.read().strip()
-    return http("GET", K8S_API + path, headers={"Authorization": f"Bearer {token}"}, cafile=K8S_CA_FILE)
-
-
-def desired_hosts():
-    sel = urllib.parse.quote(f"{PUBLIC_LABEL}=true")
-    ingresses = k8s(f"/apis/networking.k8s.io/v1/ingresses?labelSelector={sel}")
-    hosts = {}
-    for ing in ingresses.get("items", []):
-        ref = f'{ing["metadata"]["namespace"]}/{ing["metadata"]["name"]}'
-        for rule in ing.get("spec", {}).get("rules", []):
-            host = rule.get("host")
-            if not host or host.startswith("*"):
-                continue
-            hosts.setdefault(host, ref)
-    return hosts
-
-
-def traefik_cluster_ip():
-    svc = k8s(f"/api/v1/namespaces/{TRAEFIK_NS}/services/{TRAEFIK_SVC}")
-    for ip in svc["spec"].get("clusterIPs", []):
-        if ":" not in ip:
-            return ip
-    raise ApiError(f"{TRAEFIK_NS}/{TRAEFIK_SVC} has no IPv4 ClusterIP")
-
-
-# --- OpenBao ----------------------------------------------------------------
-
-def netbird_pat():
-    with open(K8S_TOKEN_FILE) as f:
-        jwt = f.read().strip()
-    login = http("POST", f"{BAO_ADDR}/v1/auth/kubernetes/login", {"role": BAO_ROLE, "jwt": jwt})
-    bao_token = login["auth"]["client_token"]
-    kv = http("GET", f"{BAO_ADDR}/v1/{BAO_PAT_PATH}", headers={"X-Vault-Token": bao_token})
-    pat = kv["data"]["data"].get(BAO_PAT_KEY)
-    if not pat:
-        raise ApiError(f"OpenBao {BAO_PAT_PATH} has no key {BAO_PAT_KEY}")
-    return pat
-
-
-# --- NetBird management -----------------------------------------------------
-
-class NetBird:
-    def __init__(self, pat):
-        self.h = {"Authorization": f"Token {pat}"}
-
-    def get(self, path):
-        return http("GET", f"{NB}/api/{path}", headers=self.h)
-
-    def list(self, path):
-        # An empty collection comes back as an empty body / null, not [].
-        return self.get(path) or []
-
-    def post(self, path, body):
-        return http("POST", f"{NB}/api/{path}", body, headers=self.h)
-
-    def put(self, path, body):
-        return http("PUT", f"{NB}/api/{path}", body, headers=self.h)
-
-    def delete(self, path):
-        return http("DELETE", f"{NB}/api/{path}", headers=self.h)
-
-
-def group_id(nb, name):
-    for g in nb.list("groups"):
-        if g["name"] == name:
-            return g["id"]
-    raise ApiError(f"NetBird group {name!r} not found")
-
-
-def ensure_network(nb):
-    for n in nb.list("networks"):
-        if n["name"] == NETWORK_NAME:
-            return n["id"]
-    n = nb.post("networks", {"name": NETWORK_NAME,
-                             "description": "managed by netbird-proxy-services; public ingress via TLS passthrough"})
-    LOG.info("created network %s (%s)", NETWORK_NAME, n["id"])
-    return n["id"]
-
-
-def ensure_resource(nb, net):
-    for r in nb.list(f"networks/{net}/resources"):
-        if r.get("address") == SERVICE_CIDR:
-            return r["id"]
-    r = nb.post(f"networks/{net}/resources", {"name": "cluster service CIDR", "address": SERVICE_CIDR,
-                                              "description": "managed by netbird-proxy-services",
-                                              "enabled": True, "groups": []})
-    LOG.info("created subnet resource %s (%s)", SERVICE_CIDR, r["id"])
-    return r["id"]
-
-
-def ensure_router(nb, net, router_group):
-    for r in nb.list(f"networks/{net}/routers"):
-        if router_group in (r.get("peer_groups") or []):
-            return r["id"]
-    r = nb.post(f"networks/{net}/routers", {"peer_groups": [router_group], "metric": 9999,
-                                            "masquerade": True, "enabled": True})
-    LOG.info("created router on group %s (%s)", ROUTER_GROUP, r["id"])
-    return r["id"]
-
-
-def ensure_policy(nb, resource, source_group):
-    name = f"{NETWORK_NAME} access"
-    for p in nb.list("policies"):
-        if p["name"] == name:
-            return p["id"]
-    p = nb.post("policies", {"name": name, "enabled": True, "rules": [{
-        "name": name, "enabled": True, "action": "accept", "protocol": "tcp",
-        "ports": [str(LISTEN_PORT)], "bidirectional": False, "sources": [source_group],
-        "destinationResource": {"id": resource, "type": "subnet"}}]})
-    LOG.info("created policy %s (%s)", name, p["id"])
-    return p["id"]
-
-
-def service_body(host, resource, target_ip):
-    # proxy_protocol: PROXY v2 header carries the WAN client address to
-    # Traefik (proxyProtocol.trustedIPs on websecure); TCP/TLS modes only.
-    return {"name": host, "domain": host, "mode": "tls", "listen_port": LISTEN_PORT, "enabled": True,
-            "targets": [{"target_type": "subnet", "target_id": resource, "protocol": "tcp",
-                         "host": target_ip, "port": 443, "enabled": True,
-                         "options": {"proxy_protocol": True}}]}
-
-
-def is_managed(svc, resource):
-    return svc.get("mode") == "tls" and any(t.get("target_id") == resource for t in svc.get("targets") or [])
-
-
-def target_matches(svc, target_ip):
-    if svc.get("listen_port") != LISTEN_PORT or not svc.get("enabled"):
-        return False
-    ts = svc.get("targets") or []
-    return len(ts) == 1 and ts[0].get("host") == target_ip and ts[0].get("port") == 443 \
-        and ts[0].get("protocol") == "tcp" and ts[0].get("enabled") \
-        and (ts[0].get("options") or {}).get("proxy_protocol") is True
-
-
-def reconcile():
-    hosts = desired_hosts()
-    target_ip = traefik_cluster_ip()
-    nb = NetBird(netbird_pat())
-
-    router_group = group_id(nb, ROUTER_GROUP)
-    source_group = group_id(nb, SOURCE_GROUP)
-    net = ensure_network(nb)
-    resource = ensure_resource(nb, net)
-    ensure_router(nb, net, router_group)
-    ensure_policy(nb, resource, source_group)
-
-    by_domain = {s["domain"]: s for s in nb.list("reverse-proxies/services")}
-    created = updated = deleted = held = 0
-
-    for host, ref in sorted(hosts.items()):
-        svc = by_domain.get(host)
-        if svc is None:
-            nb.post("reverse-proxies/services", service_body(host, resource, target_ip))
-            LOG.info("created %s (from %s) -> %s:443", host, ref, target_ip)
-            created += 1
-        elif not is_managed(svc, resource):
-            LOG.warning("%s is held by unmanaged service %s (mode %s); remove it to migrate",
-                        host, svc["id"], svc.get("mode"))
-            held += 1
-        elif not target_matches(svc, target_ip):
-            nb.put(f"reverse-proxies/services/{svc['id']}", service_body(host, resource, target_ip))
-            LOG.info("updated %s -> %s:443", host, target_ip)
-            updated += 1
-
-    for domain, svc in by_domain.items():
-        if is_managed(svc, resource) and domain not in hosts:
-            nb.delete(f"reverse-proxies/services/{svc['id']}")
-            LOG.info("deleted %s (no public Ingress carries it any more)", domain)
-            deleted += 1
-
-    LOG.info("reconciled: %d hosts desired, %d created, %d updated, %d deleted, %d held by unmanaged services",
-             len(hosts), created, updated, deleted, held)
-
-
-def main():
-    logging.basicConfig(level=os.environ.get("LOG_LEVEL", "INFO"), stream=sys.stdout,
-                        format="%(asctime)s %(levelname)s %(message)s")
-    if os.environ.get("ONCE") == "1":
-        reconcile()
-        return
-    while True:
-        try:
-            reconcile()
-        except Exception as e:  # keep running; the next pass retries
-            LOG.error("reconcile failed: %s", e)
-        time.sleep(INTERVAL)
-
-
-if __name__ == "__main__":
-    main()

kubernetes/PROPOSAL-node-roll-downtime.md

@@ -0,0 +1,304 @@
+# Proposal — a node roll without the ingress hiccup
+
+Status: draft, grounded in one measured roll. On 2026-09-07 all three MS-01
+were drained, patched and rebooted back to back (06:52–07:08) for the SAN
+rollout. Everything below is read off that window: Traefik access logs and
+Cilium BGP logs in Loki, the drain logs, and the live cluster afterwards.
+
+The short version: there are two ingresses and they failed differently.
+**The LAN ingress (Traefik behind the BGP VIP) did not hiccup** — it
+answered every request; the 5xx it returned were for backends mid-move.
+**The public edge (proxies → NetBird mesh → Traefik) went dark for 40–80 s
+on two of the three rolls, and would have for 3½ minutes on the third had
+its routes not recovered by luck** — a known client bug the repo already
+documents and works around with a watchdog, whose 600 s restart cooldown is
+what turns a 3-second blip into minutes when rolls follow each other. So
+this is two projects: make evictions cheap, and make a peer loss on the
+mesh cost seconds, deterministically.
+
+## What was measured
+
+Traefik, 06:48–07:12, 24 minutes, 3 391 requests, of which 5xx:
+
+| router | status | count |
+|---|---|---|
+| gitea | 503 | 56 |
+| home-assistant | 503 | 36 |
+| shelly-manager | 502 | 25 |
+| immich | 503 | 19 |
+| openbao | 503 | 13 |
+| grafana | 503 | 10 |
+| music-assistant | 503 | 9 |
+| crowdsec-lapi | 503 | 7 |
+| wordpress | 503 | 6 |
+| woodpecker | 503 | 4 |
+
+The quiet 24 minutes before the roll had zero 5xx. Every entry above is a
+single-replica workload (or its single database) that was evicted and had to
+come up on another node — pod reschedule, RBD volume detach and re-attach,
+process start. That is 30–90 s of 503 per app per drain.
+
+Two things made it worse than one move per app:
+
+- **Double moves.** The drain of k8s-1 pushed pods onto k8s-2, which was
+  drained next: `ha-pg-1`, `ak-pg-1`, `im-pg-1`, `pprl-pg-2`, `tr-pg-1`,
+  `wp-pg-1`, `influxdb`, `protonmail-bridge`, `prometheus`, `wordpress-mariadb`
+  each appear in two drain logs. Half the downtime was paid twice.
+- **`--disable-eviction`.** Ten PodDisruptionBudgets allow zero disruptions
+  (every CNPG primary, gitea, influxdb, syncthing, protonmail-bridge,
+  woodpecker-server). A plain `kubectl drain` hangs on them, so the roll used
+  `--disable-eviction`, which deletes pods outright. For the two-instance CNPG
+  clusters (authentik, immich, paperless) that skipped the switchover CNPG
+  would have done: the operator's documented behaviour is "if a node is to be
+  drained and contains a cluster's primary instance, a switchover happens
+  ahead of the drain" — but only when the drain goes through the eviction API.
+  We turned a 2-second switchover into a 30–60 s primary restart.
+
+Beside the HTTP path, one reboot side effect took a dependency down for
+half an hour: openbao-0 came back **sealed**, and because the StatefulSet is
+`OrderedReady`, openbao-1/2 were never recreated after their eviction until
+openbao-0 was unsealed by hand. ESO could not refresh a secret for 35 minutes.
+
+### The public edge
+
+Public traffic reaches Traefik with client address `127.0.0.1` (the rootless
+netbird-ingress peer forwards from inside its pod through socket-LB), LAN
+traffic with `10.2.x`. That splits the access log by path. At that hour the
+only public traffic was the proxies' own watchdog — two hosts probed once a
+minute from each of two proxies, a flat 4 requests/min — which makes it a
+clean synthetic probe of the whole external chain:
+
+| minute (UTC) | LAN req | public req | what was happening |
+|---|---|---|---|
+| 04:55 | 128 | 4 | k8s-1 rebooting (04:55:10) |
+| 04:56 | 109 | 2 | k8s-1 back, its peer re-enrolled 04:56:24 |
+| 04:57 | 127 | **0** | — |
+| 04:58 | 126 | 4 | proxies restarted by their watchdog |
+| 05:00–05:03 | 186–233 | 4 | k8s-2 roll: **no dip** |
+| 05:06–05:07 | 97–135 | 2 | k8s-3 roll: half |
+
+The external chain was down for roughly two minutes at the k8s-1 roll, one
+at k8s-3, and not at all at k8s-2. The mechanism is in
+`ansible/roles/netbird-proxy-watchdog/defaults/main.yml`: the proxy's
+embedded NetBird client pins each domain route to one routing-peer
+*identity*; the netbird-ingress DaemonSet enrols with a setup key, so every
+pod start is a **new identity** (the k8s-1 pod registered fresh at
+04:56:24); and upstream's dns-interceptor never rebinds a route whose
+owning peer is gone ("already routed by peer … HA routing disabled"). The
+proxy then serves 502 for every domain until its container restarts —
+which the watchdog does after three failed one-minute probes, or at once
+when the flood watchdog sees the wedge in the proxy log. Which roll hurts
+depends on which of the three peers the proxies happened to pin. k8s-2's
+peer was pinned by neither.
+
+Note what did *not* help: the DaemonSet's preStop (`netbird down`) took the
+peer out of the mesh cleanly at 04:55:11, and the peer was back 73 s later.
+Neither matters to a client that will not rebind and will not recognise
+the returning pod as the same peer.
+
+## What was not the cause
+
+**BGP.** Per node the UDM sessions were down for 47–51 s (Peer Down at the
+Cilium shutdown, Peer Up ~50 s later). That gap did not carry traffic: every
+LoadBalancer here is `externalTrafficPolicy: Local`, so a VIP is only
+advertised from nodes with a local endpoint, and the drain had already
+evicted those endpoints — Cilium withdrew the routes while the session was
+still healthy (the "soft reset out" bursts during each drain). The 90 s hold
+time and 120 s graceful-restart window never came into play. Leave them.
+
+**Traefik.** Two replicas on k8s-1/k8s-2, one evicted per roll, the other
+carried the VIP alone. No PDB, no spread constraint, no preStop — which held
+by luck (the two pods happened to sit on different nodes), and is fixed below
+because luck is not a design.
+
+## Target state, in order of measured impact
+
+### 1. Drain in a way that lets operators do their job (procedure only)
+
+- **Cordon the next node before draining the current one.** Evicted pods
+  then land on the node that stays up, and nothing moves twice. Order:
+  cordon B → drain A → reboot A → uncordon A → cordon C → drain B → …
+- **Never `--disable-eviction`.** It exists in the roll only because ten
+  PDBs say "never". Stage 2 makes those PDBs honest, after which a plain
+  `kubectl drain --ignore-daemonsets --delete-emptydir-data` completes and
+  CNPG switches primaries over before its replicas are evicted.
+- **After every reboot, three gates before uncordon:** `00-multus.conf` on
+  the host and Multus + whereabouts Running (k8s-1's OSD came up
+  Cilium-only today because it beat the Multus DaemonSet by 30 s); OpenBao
+  unsealed (`bao status` on every pod that exists, then the ones the
+  StatefulSet creates after it); Ceph back at baseline.
+- This is `rollout-node.sh` from today's window, minus `--disable-eviction`,
+  plus the OpenBao gate, checked into `scripts/`. A reboot runbook that lives
+  in a scratchpad is a runbook nobody else can run.
+
+### 2. PodDisruptionBudgets that mean what they say
+
+A `minAvailable: 1` PDB on a single-replica workload does not protect
+anything; it only forces the operator to bypass all PDBs. Two changes:
+
+- **CNPG single-instance clusters** (grafana-db, home-assistant, trmnl,
+  woodpecker): `spec.enablePDB: false`. CNPG documents exactly this — "for
+  single-instance clusters the operator will prevent draining the node" and
+  recommends moving to direct control of disruption budgets. The instance
+  restarts on drain (it does today too, just through the back door), and the
+  multi-instance clusters keep their switchover-before-eviction behaviour.
+- **gitea, influxdb, syncthing, protonmail-bridge, woodpecker-server**: drop
+  the `minAvailable: 1` PDBs. They are single replicas with RWO volumes; the
+  PDB cannot buy availability, only stalls.
+
+### 3. The ingress itself: three replicas, one per node, drained gracefully
+
+Traefik goes to `replicas: 3` with a required `podAntiAffinity` on
+hostname, `PodDisruptionBudget maxUnavailable: 1`, and a preStop that lets
+in-flight requests finish before the endpoint is withdrawn:
+
+```yaml
+# chart keys as in traefik/values.yaml
+deployment:
+  replicas: 3
+  lifecycle:
+    preStop:
+      sleep:
+        seconds: 5
+affinity:
+  podAntiAffinity:
+    requiredDuringSchedulingIgnoredDuringExecution:
+      - topologyKey: kubernetes.io/hostname
+        labelSelector:
+          matchLabels:
+            app.kubernetes.io/name: traefik
+podDisruptionBudget:
+  enabled: true
+  maxUnavailable: 1
+ports:
+  websecure:
+    transport:
+      lifeCycle:
+        graceTimeOut: 15s
+```
+
+With `Local` traffic policy every node then advertises the VIP; a roll takes
+one of three ECMP paths away instead of one of two, and the UDM sees the
+withdrawal before the pod stops accepting. Also gives the UDM a route from
+k8s-3, which today advertises nothing for the ingress.
+
+### 4. Databases the apps actually wait on
+
+The 503 table is a priority list. Each of these is a single Postgres today:
+
+| app | 5xx | cluster | today |
+|---|---|---|---|
+| home-assistant | 36 | ha-pg | `instances: 1` |
+| grafana | 10 | gf-pg | `instances: 1` |
+| woodpecker | 4 | wp-pg | `instances: 1` |
+| trmnl | — | tr-pg | `instances: 1` |
+
+`instances: 2` for ha-pg and gf-pg turns a drain into a CNPG switchover
+(seconds). The cost is one more Postgres pod and a second RBD volume each.
+trmnl and woodpecker can stay at one: they were barely visible.
+
+gitea (56) is the largest number and has no cheap answer: one StatefulSet,
+one RWO volume, an application that does not run two instances against the
+same data directory without shared storage. It moves once per roll (with
+stage 1) and that is its floor unless its repositories move to CephFS.
+
+### 5. The public edge: what the proxies' own journals say
+
+This is the only *complete* outage in the window, so despite the numbering
+it ships right after stage 1. It was measured twice: from the proxies'
+watchdog journals for the morning roll, and in a controlled afternoon
+experiment (three routing-peer kills, probed from proxy-2 itself at 2 s —
+the LAN cannot probe this path, the UDM hairpin-DNATs the proxy addresses
+straight to the VIP). Proxy clocks run ~3 min behind the cluster.
+
+**What a peer loss does.** When the routing peer the proxy has pinned
+disappears, the proxy's route manager wedges (`couldn't fetch peer state`
+burst) and the *whole* process stops answering — every domain, and in the
+experiment also services that never used a domain route. The flood
+watchdog restarts the container within 2 s of the burst; the probe saw
+one failed round (≈3 s) at the first two kills. The third kill fell inside
+the flood watchdog's **600 s restart cooldown**: restart suppressed, wedge
+left standing, and the URL watchdog needed its three one-minute rounds —
+**3½ minutes of 000 on every public domain**. The morning roll had the
+same shape: k8s-1 (both proxies restarted, 40–80 s), k8s-2 (wedge at
+06:58, restart suppressed on both proxies, 278 s after the previous one —
+the routes happened to recover on their own), k8s-3 (restart, ~10 s).
+Three rolls in sixteen minutes guarantee the suppressed case.
+
+- **5a. Cooldown 600 s → 60 s** in
+  `ansible/roles/netbird-proxy-watchdog/defaults/main.yml`. The cooldown
+  exists so a genuinely broken backend cannot restart-loop the proxy; a
+  minute still does that (the URL watchdog needs three minutes to fire
+  anyway). With it, every peer loss — rolls back to back included — is a
+  ~3 s blip instead of a lottery between 3 s and 3½ min. Cheapest change
+  in this document, ships first.
+- **5b. Persist the peer identity.** Replace the DaemonSet with a
+  StatefulSet of three, required anti-affinity on hostname,
+  `PodDisruptionBudget maxUnavailable: 1`, and a small RBD volume per
+  replica holding NetBird's state directory. A drain then evicts the pod
+  through the PDB (the DaemonSet pod is not evicted at all today, it dies
+  with the reboot) and it returns within ~30 s **as the same peer**. Whether
+  the route manager then un-wedges without a restart is the hypothesis to
+  measure; the setup-key constraint (ephemeral peers reaped after >10 min
+  offline, NeedsLogin liveness restart) holds unchanged.
+- **5c. The structural exit: no domain routes at all.** The subnet-target
+  experiment settled what works: an HTTP-mode service with an IP target
+  forwards `Host: <ip>` and Traefik answers 404 (NetBird rejects a Host
+  override); an HTTP-mode service with a domain target needs mesh DNS on
+  the proxy, which answers SERVFAIL there; a **TLS-passthrough service
+  (`mode: tls`, `listen_port: 443`) with a subnet target works** — SNI and
+  Host reach Traefik untouched, it coexists with the HTTP listener on 443,
+  no `skip_tls_verify`. If every public service became such a passthrough
+  over a subnet resource covering the Cilium LB pool (`10.2.99.0/24`,
+  `fd99::/64`), there would be no domain resource left to wedge on, and
+  peer loss would be NetBird's documented HA failover (seconds). Costs: the
+  target's `host` is the VIP, so a reconciler (the external-dns-unifi
+  pattern, `PUT /api/reverse-proxies/services/{id}` from the Traefik
+  Service's `status.loadBalancer.ingress`) has to own it; the proxy's L7
+  features (auth, path rules) are lost — none are in use; and the pure
+  failover time is unmeasured, because today every kill wedged the domain
+  routes first. Decide after 5a and 5b have been measured on a real roll.
+- **Track the upstream bug.** The repo's note dates the dns-interceptor
+  behaviour to 2026-07; a search of NetBird's releases and issues turned up
+  no fix for the "already routed by peer / HA routing disabled" path.
+  Re-check on each NetBird bump.
+- **Side findings, each its own small change:** both proxy clocks are
+  ~3 min slow (the proxy logs SCT timestamps "in the future"; TLS
+  validity windows care) — NTP on proxy-1/proxy-2; a new service's
+  certificate reaches the IPv6-only member only after ~20 min of ACME
+  retries (Let's Encrypt validates over IPv4 and lands on the other member,
+  which has no token for it); proxy-1 and proxy-2 do not ship syslog to
+  Loki (edge-1 does), so every public-edge timeline is reconstructed over
+  SSH — point them at the alloy-syslog VIP.
+
+## Expected result
+
+| stage | removes |
+|---|---|
+| 1 | every double move (half of today's 5xx), the primary restarts, the 35-minute OpenBao/ESO outage, the Multus race |
+| 2 | the need for `--disable-eviction` at all |
+| 3 | the single-node ingress exposure; ECMP loses 1/3 instead of 1/2 |
+| 4 | home-assistant and grafana 503s entirely |
+| 5a | the 3½-minute public-edge blackout of a second peer loss within 10 min; every peer loss becomes a ~3 s restart |
+| 5b/5c | the restart itself, if the returning identity (5b) or the absence of domain routes (5c) proves to un-wedge — to be measured |
+
+What remains after all of it: one move per single-replica app per roll
+(gitea, immich, shelly-manager, music-assistant, wordpress), 30–60 s each,
+once, and a few seconds of public edge per roll. That is the honest floor
+for single-replica services on RWO storage and a mesh client that pins.
+
+## Transition
+
+1. Stage 5a is one Ansible default and a proxy deploy — ships first.
+2. Stage 1 is a script and a runbook change — no cluster change.
+3. Stage 5b's StatefulSet changes how the public edge enrols, so it gets
+   its own roll to measure, probed from a proxy.
+4. Stage 2 is one field on four CNPG clusters and five deleted PDBs; each
+   Kustomization applies without disruption.
+5. Stage 3 is a Traefik HelmRelease change; the rollout itself is a rolling
+   update under the new PDB.
+6. Stage 4 is `instances: 2` on two clusters; CNPG adds the replica online.
+7. Stage 5c only if 5a+5b leave a measurable public gap.
+
+Measure the next roll the same way: 5xx by router, the proxies' watchdog
+journals (or their syslog in Loki), BGP Down→Up gaps, pods per drain log.

kubernetes/infrastructure/crowdsec/lapi-ingress.yaml

@@ -31,12 +31,22 @@
 # serves read-only decision streams to a valid bouncer key and nothing else.
 # The admin surface is cscli, in-cluster. If the source address ever becomes
 # deterministic, an ipAllowList can be added on top.
-#
+---
 # The LAPI serves HTTPS only once tls.enabled is set — there is no plaintext
 # fallback and no second port; 8080 is the TLS port. Its certificate comes
 # from the chart's own self-signed CA, so Traefik needs that CA to verify the
-# backend; the ServersTransport carrying it is a traefik.io CRD consumer and
-# lives in post-crds/crowdsec-lapi-serverstransport.yaml.
+# backend. SANs include crowdsec-service.crowdsec.svc.cluster.local, which is
+# what the Service resolves to.
+apiVersion: traefik.io/v1alpha1
+kind: ServersTransport
+metadata:
+  name: crowdsec-lapi
+  namespace: crowdsec
+spec:
+  serverName: crowdsec-service.crowdsec.svc.cluster.local
+  rootCAs:
+    - secret: crowdsec-root-secret
+---
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:

kubernetes/infrastructure/kustomization.yaml

@@ -18,7 +18,6 @@ resources:
   - cloudnative-pg/
   - mariadb-operator/
   - netbird-ingress/
-  - netbird-proxy-services/
   - coredns-config/
   - k8s-gateway/
   - external-dns-unifi/

kubernetes/infrastructure/netbird-proxy-services/deployment.yaml

@@ -1,106 +0,0 @@
-# Reconciles NetBird reverse-proxy services from Ingresses that carry
-# dns.renner.uno/public: one TLS-passthrough service per host on :443,
-# targeting Traefik's ClusterIP through a subnet resource over the service
-# CIDR (ci/images/netbird-proxy-services/controller.py).
-#
-# openbao namespace, like the rotators: its default-deny already allows egress
-# to OpenBao, DNS, the apiserver and `world` (netbird.renner.uno), and the
-# NetBird PAT is read from OpenBao on every pass through the
-# netbird-proxy-services role — policies/netbird-proxy-services.hcl, applied
-# once by hand like the rotator roles. Until that role exists the pod logs an
-# auth error each minute and does nothing.
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: netbird-proxy-services
-  namespace: openbao
-  labels:
-    app.kubernetes.io/name: netbird-proxy-services
----
-# Read-only, cluster-wide: public Ingresses live in many namespaces, the
-# Traefik Service in ingress/.
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRole
-metadata:
-  name: netbird-proxy-services-read
-rules:
-  - apiGroups: ["networking.k8s.io"]
-    resources: ["ingresses"]
-    verbs: ["get", "list", "watch"]
-  - apiGroups: [""]
-    resources: ["services"]
-    verbs: ["get"]
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRoleBinding
-metadata:
-  name: netbird-proxy-services-read
-roleRef:
-  apiGroup: rbac.authorization.k8s.io
-  kind: ClusterRole
-  name: netbird-proxy-services-read
-subjects:
-  - kind: ServiceAccount
-    name: netbird-proxy-services
-    namespace: openbao
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: netbird-proxy-services
-  namespace: openbao
-  labels:
-    app.kubernetes.io/name: netbird-proxy-services
-spec:
-  replicas: 1
-  # One writer against the management API at a time.
-  strategy:
-    type: Recreate
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: netbird-proxy-services
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/name: netbird-proxy-services
-    spec:
-      serviceAccountName: netbird-proxy-services
-      securityContext:
-        runAsNonRoot: true
-        # Numeric, or the kubelet refuses the image's named user with
-        # "cannot verify user is non-root". The script needs no home and no
-        # writable path, so nobody/nogroup is enough.
-        runAsUser: 65534
-        runAsGroup: 65534
-        seccompProfile:
-          type: RuntimeDefault
-      containers:
-        - name: controller
-          # Pinned by digest: the date tag is rebuilt on every push of the
-          # image directory, so the tag alone cannot express a new build.
-          image: gitea.renner.uno/gitea_admin/netbird-proxy-services:2026-09-07@sha256:0d266b0288aec692c698ff5c05f76c826e247bfef365e0306a0b9a6541ebf180
-          securityContext:
-            allowPrivilegeEscalation: false
-            readOnlyRootFilesystem: true
-            capabilities:
-              drop: ["ALL"]
-          env:
-            - name: NB_MANAGEMENT_URL
-              value: https://netbird.renner.uno
-            - name: BAO_ADDR
-              value: http://openbao.openbao.svc.cluster.local:8200
-            - name: BAO_ROLE
-              value: netbird-proxy-services
-            # Must match cluster.network.serviceSubnets in talos/patches/cluster.yaml.
-            - name: SERVICE_CIDR
-              value: "10.96.0.0/12"
-            - name: TRAEFIK_SERVICE
-              value: ingress/traefik
-            - name: INTERVAL_SECONDS
-              value: "60"
-          resources:
-            requests:
-              cpu: 10m
-              memory: 32Mi
-            limits:
-              memory: 64Mi

kubernetes/infrastructure/netbird-proxy-services/kustomization.yaml

@@ -1,4 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-resources:
-  - deployment.yaml

kubernetes/infrastructure/openbao/policies/netbird-proxy-services.hcl

@@ -1,23 +0,0 @@
-# OpenBao policy: netbird-proxy-services
-#
-# The reverse-proxy-services controller (Deployment netbird-proxy-services in
-# the openbao namespace) reads the NetBird PAT on every reconcile pass and
-# calls the management API with it. Read-only on that one path; it shares the
-# bootstrap credential with rotator-netbird but not that role's PATCH rights.
-#
-# NOT applied by Flux. Apply ONCE with an admin token:
-#
-#   bao policy write netbird-proxy-services kubernetes/infrastructure/openbao/policies/netbird-proxy-services.hcl
-#   bao write auth/kubernetes/role/netbird-proxy-services \
-#     bound_service_account_names=netbird-proxy-services \
-#     bound_service_account_namespaces=openbao \
-#     policies=netbird-proxy-services ttl=15m
-#
-# The PAT itself is the one rotator-netbird already uses
-# (homelab/infrastructure/secret-rotators-bootstrap/netbird, key `pat`).
-#
-# Do NOT set token_no_default_policy=true (kv-v2 preflight needs the default policy).
-
-path "homelab/data/infrastructure/secret-rotators-bootstrap/netbird" {
-  capabilities = ["read"]
-}

kubernetes/infrastructure/post-crds/kustomization.yaml

@@ -31,5 +31,7 @@ resources:
   - traefik-crowdsec-middleware.yaml
   - traefik-rate-limit-middleware.yaml
   - traefik-security-headers-middleware.yaml
+  # Phase 1 of a two-Kustomization move: also still in crowdsec/lapi-ingress.yaml
+  # until this layer has applied it once, so prune never runs ahead of apply.
   - crowdsec-lapi-serverstransport.yaml
   - ../network-policies/

kubernetes/infrastructure/traefik/helmrelease.yaml

@@ -93,17 +93,13 @@ spec:
         # Real client-IP visibility. The netbird-ingress routing peer
         # re-originates every mesh + WAN-via-proxy connection from its own
         # pod IP (rootless userspace WireGuard), so Traefik's TCP peer for
-        # that traffic is always a pod IP — collapsing all external clients
-        # to one source. The NetBird reverse-proxy services are TLS
-        # passthrough and carry the WAN client address in a PROXY protocol
-        # v2 header (options.proxy_protocol, set by netbird-proxy-services);
-        # X-Forwarded-For stays trusted for HTTP-mode services. Both are
-        # accepted from the pod CIDRs only; connections without a header
-        # (direct LAN/mesh) keep their RemoteAddr.
-        proxyProtocol:
-          trustedIPs:
-            - "10.244.0.0/16"
-            - "fd10:244::/56"
+        # that traffic is always a pod IP (e.g. fd10:244::2de1) — collapsing
+        # all external clients to one source. The NetBird reverse-proxy
+        # (HTTP mode, on the VPS) sets X-Forwarded-For with the real WAN
+        # client IP; trust the pod CIDRs so Traefik reads it instead. Without
+        # this, a CrowdSec bouncer bans the shared pod IP and 403s everyone
+        # (the 2026-06-27 IPv6 outage). Harmless no-op when no XFF is present
+        # (direct LAN/mesh) — those keep their RemoteAddr.
         forwardedHeaders:
           trustedIPs:
             - "10.244.0.0/16"