docs(proposal): public edge measured from the proxies — cooldown is the lever

Watchdog journals on proxy-1/proxy-2 for the morning roll, plus a
controlled experiment (three routing-peer kills probed from proxy-2 at
2 s; the LAN cannot probe this path, the UDM hairpin-DNATs the proxy
addresses to the VIP):

- a pinned-peer loss wedges the whole proxy process; the flood watchdog
  restarts it within 2 s, the probe sees one failed round (~3 s)
- inside the flood watchdog 600 s cooldown the restart is suppressed and
  the URL watchdog needs three one-minute rounds: 3.5 min of 000 on every
  public domain, measured; the morning k8s-2 roll hit the same suppression
- subnet-target services: HTTP mode + IP target -> Host header is the IP,
  Traefik 404; HTTP mode + domain target -> SERVFAIL, no mesh DNS on the
  proxy; TLS passthrough (mode tls, listen_port 443) + subnet target works
  end to end and coexists on 443 -- but dies with the wedge as long as any
  domain resource exists

Stage 5 rewritten: 5a cooldown 600->60 s (ships first), 5b identity
persistence as a hypothesis to measure, 5c all-passthrough as the
structural exit with its VIP reconciler cost. Side findings: proxy clocks
3 min slow, ACME on the v6-only member ~20 min late, no proxy syslog in
Loki.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

kubernetes/PROPOSAL-node-roll-downtime.md

@@ -0,0 +1,304 @@
+# Proposal — a node roll without the ingress hiccup
+
+Status: draft, grounded in one measured roll. On 2026-09-07 all three MS-01
+were drained, patched and rebooted back to back (06:52–07:08) for the SAN
+rollout. Everything below is read off that window: Traefik access logs and
+Cilium BGP logs in Loki, the drain logs, and the live cluster afterwards.
+
+The short version: there are two ingresses and they failed differently.
+**The LAN ingress (Traefik behind the BGP VIP) did not hiccup** — it
+answered every request; the 5xx it returned were for backends mid-move.
+**The public edge (proxies → NetBird mesh → Traefik) went dark for 40–80 s
+on two of the three rolls, and would have for 3½ minutes on the third had
+its routes not recovered by luck** — a known client bug the repo already
+documents and works around with a watchdog, whose 600 s restart cooldown is
+what turns a 3-second blip into minutes when rolls follow each other. So
+this is two projects: make evictions cheap, and make a peer loss on the
+mesh cost seconds, deterministically.
+
+## What was measured
+
+Traefik, 06:48–07:12, 24 minutes, 3 391 requests, of which 5xx:
+
+| router | status | count |
+|---|---|---|
+| gitea | 503 | 56 |
+| home-assistant | 503 | 36 |
+| shelly-manager | 502 | 25 |
+| immich | 503 | 19 |
+| openbao | 503 | 13 |
+| grafana | 503 | 10 |
+| music-assistant | 503 | 9 |
+| crowdsec-lapi | 503 | 7 |
+| wordpress | 503 | 6 |
+| woodpecker | 503 | 4 |
+
+The quiet 24 minutes before the roll had zero 5xx. Every entry above is a
+single-replica workload (or its single database) that was evicted and had to
+come up on another node — pod reschedule, RBD volume detach and re-attach,
+process start. That is 30–90 s of 503 per app per drain.
+
+Two things made it worse than one move per app:
+
+- **Double moves.** The drain of k8s-1 pushed pods onto k8s-2, which was
+  drained next: `ha-pg-1`, `ak-pg-1`, `im-pg-1`, `pprl-pg-2`, `tr-pg-1`,
+  `wp-pg-1`, `influxdb`, `protonmail-bridge`, `prometheus`, `wordpress-mariadb`
+  each appear in two drain logs. Half the downtime was paid twice.
+- **`--disable-eviction`.** Ten PodDisruptionBudgets allow zero disruptions
+  (every CNPG primary, gitea, influxdb, syncthing, protonmail-bridge,
+  woodpecker-server). A plain `kubectl drain` hangs on them, so the roll used
+  `--disable-eviction`, which deletes pods outright. For the two-instance CNPG
+  clusters (authentik, immich, paperless) that skipped the switchover CNPG
+  would have done: the operator's documented behaviour is "if a node is to be
+  drained and contains a cluster's primary instance, a switchover happens
+  ahead of the drain" — but only when the drain goes through the eviction API.
+  We turned a 2-second switchover into a 30–60 s primary restart.
+
+Beside the HTTP path, one reboot side effect took a dependency down for
+half an hour: openbao-0 came back **sealed**, and because the StatefulSet is
+`OrderedReady`, openbao-1/2 were never recreated after their eviction until
+openbao-0 was unsealed by hand. ESO could not refresh a secret for 35 minutes.
+
+### The public edge
+
+Public traffic reaches Traefik with client address `127.0.0.1` (the rootless
+netbird-ingress peer forwards from inside its pod through socket-LB), LAN
+traffic with `10.2.x`. That splits the access log by path. At that hour the
+only public traffic was the proxies' own watchdog — two hosts probed once a
+minute from each of two proxies, a flat 4 requests/min — which makes it a
+clean synthetic probe of the whole external chain:
+
+| minute (UTC) | LAN req | public req | what was happening |
+|---|---|---|---|
+| 04:55 | 128 | 4 | k8s-1 rebooting (04:55:10) |
+| 04:56 | 109 | 2 | k8s-1 back, its peer re-enrolled 04:56:24 |
+| 04:57 | 127 | **0** | — |
+| 04:58 | 126 | 4 | proxies restarted by their watchdog |
+| 05:00–05:03 | 186–233 | 4 | k8s-2 roll: **no dip** |
+| 05:06–05:07 | 97–135 | 2 | k8s-3 roll: half |
+
+The external chain was down for roughly two minutes at the k8s-1 roll, one
+at k8s-3, and not at all at k8s-2. The mechanism is in
+`ansible/roles/netbird-proxy-watchdog/defaults/main.yml`: the proxy's
+embedded NetBird client pins each domain route to one routing-peer
+*identity*; the netbird-ingress DaemonSet enrols with a setup key, so every
+pod start is a **new identity** (the k8s-1 pod registered fresh at
+04:56:24); and upstream's dns-interceptor never rebinds a route whose
+owning peer is gone ("already routed by peer … HA routing disabled"). The
+proxy then serves 502 for every domain until its container restarts —
+which the watchdog does after three failed one-minute probes, or at once
+when the flood watchdog sees the wedge in the proxy log. Which roll hurts
+depends on which of the three peers the proxies happened to pin. k8s-2's
+peer was pinned by neither.
+
+Note what did *not* help: the DaemonSet's preStop (`netbird down`) took the
+peer out of the mesh cleanly at 04:55:11, and the peer was back 73 s later.
+Neither matters to a client that will not rebind and will not recognise
+the returning pod as the same peer.
+
+## What was not the cause
+
+**BGP.** Per node the UDM sessions were down for 47–51 s (Peer Down at the
+Cilium shutdown, Peer Up ~50 s later). That gap did not carry traffic: every
+LoadBalancer here is `externalTrafficPolicy: Local`, so a VIP is only
+advertised from nodes with a local endpoint, and the drain had already
+evicted those endpoints — Cilium withdrew the routes while the session was
+still healthy (the "soft reset out" bursts during each drain). The 90 s hold
+time and 120 s graceful-restart window never came into play. Leave them.
+
+**Traefik.** Two replicas on k8s-1/k8s-2, one evicted per roll, the other
+carried the VIP alone. No PDB, no spread constraint, no preStop — which held
+by luck (the two pods happened to sit on different nodes), and is fixed below
+because luck is not a design.
+
+## Target state, in order of measured impact
+
+### 1. Drain in a way that lets operators do their job (procedure only)
+
+- **Cordon the next node before draining the current one.** Evicted pods
+  then land on the node that stays up, and nothing moves twice. Order:
+  cordon B → drain A → reboot A → uncordon A → cordon C → drain B → …
+- **Never `--disable-eviction`.** It exists in the roll only because ten
+  PDBs say "never". Stage 2 makes those PDBs honest, after which a plain
+  `kubectl drain --ignore-daemonsets --delete-emptydir-data` completes and
+  CNPG switches primaries over before its replicas are evicted.
+- **After every reboot, three gates before uncordon:** `00-multus.conf` on
+  the host and Multus + whereabouts Running (k8s-1's OSD came up
+  Cilium-only today because it beat the Multus DaemonSet by 30 s); OpenBao
+  unsealed (`bao status` on every pod that exists, then the ones the
+  StatefulSet creates after it); Ceph back at baseline.
+- This is `rollout-node.sh` from today's window, minus `--disable-eviction`,
+  plus the OpenBao gate, checked into `scripts/`. A reboot runbook that lives
+  in a scratchpad is a runbook nobody else can run.
+
+### 2. PodDisruptionBudgets that mean what they say
+
+A `minAvailable: 1` PDB on a single-replica workload does not protect
+anything; it only forces the operator to bypass all PDBs. Two changes:
+
+- **CNPG single-instance clusters** (grafana-db, home-assistant, trmnl,
+  woodpecker): `spec.enablePDB: false`. CNPG documents exactly this — "for
+  single-instance clusters the operator will prevent draining the node" and
+  recommends moving to direct control of disruption budgets. The instance
+  restarts on drain (it does today too, just through the back door), and the
+  multi-instance clusters keep their switchover-before-eviction behaviour.
+- **gitea, influxdb, syncthing, protonmail-bridge, woodpecker-server**: drop
+  the `minAvailable: 1` PDBs. They are single replicas with RWO volumes; the
+  PDB cannot buy availability, only stalls.
+
+### 3. The ingress itself: three replicas, one per node, drained gracefully
+
+Traefik goes to `replicas: 3` with a required `podAntiAffinity` on
+hostname, `PodDisruptionBudget maxUnavailable: 1`, and a preStop that lets
+in-flight requests finish before the endpoint is withdrawn:
+
+```yaml
+# chart keys as in traefik/values.yaml
+deployment:
+  replicas: 3
+  lifecycle:
+    preStop:
+      sleep:
+        seconds: 5
+affinity:
+  podAntiAffinity:
+    requiredDuringSchedulingIgnoredDuringExecution:
+      - topologyKey: kubernetes.io/hostname
+        labelSelector:
+          matchLabels:
+            app.kubernetes.io/name: traefik
+podDisruptionBudget:
+  enabled: true
+  maxUnavailable: 1
+ports:
+  websecure:
+    transport:
+      lifeCycle:
+        graceTimeOut: 15s
+```
+
+With `Local` traffic policy every node then advertises the VIP; a roll takes
+one of three ECMP paths away instead of one of two, and the UDM sees the
+withdrawal before the pod stops accepting. Also gives the UDM a route from
+k8s-3, which today advertises nothing for the ingress.
+
+### 4. Databases the apps actually wait on
+
+The 503 table is a priority list. Each of these is a single Postgres today:
+
+| app | 5xx | cluster | today |
+|---|---|---|---|
+| home-assistant | 36 | ha-pg | `instances: 1` |
+| grafana | 10 | gf-pg | `instances: 1` |
+| woodpecker | 4 | wp-pg | `instances: 1` |
+| trmnl | — | tr-pg | `instances: 1` |
+
+`instances: 2` for ha-pg and gf-pg turns a drain into a CNPG switchover
+(seconds). The cost is one more Postgres pod and a second RBD volume each.
+trmnl and woodpecker can stay at one: they were barely visible.
+
+gitea (56) is the largest number and has no cheap answer: one StatefulSet,
+one RWO volume, an application that does not run two instances against the
+same data directory without shared storage. It moves once per roll (with
+stage 1) and that is its floor unless its repositories move to CephFS.
+
+### 5. The public edge: what the proxies' own journals say
+
+This is the only *complete* outage in the window, so despite the numbering
+it ships right after stage 1. It was measured twice: from the proxies'
+watchdog journals for the morning roll, and in a controlled afternoon
+experiment (three routing-peer kills, probed from proxy-2 itself at 2 s —
+the LAN cannot probe this path, the UDM hairpin-DNATs the proxy addresses
+straight to the VIP). Proxy clocks run ~3 min behind the cluster.
+
+**What a peer loss does.** When the routing peer the proxy has pinned
+disappears, the proxy's route manager wedges (`couldn't fetch peer state`
+burst) and the *whole* process stops answering — every domain, and in the
+experiment also services that never used a domain route. The flood
+watchdog restarts the container within 2 s of the burst; the probe saw
+one failed round (≈3 s) at the first two kills. The third kill fell inside
+the flood watchdog's **600 s restart cooldown**: restart suppressed, wedge
+left standing, and the URL watchdog needed its three one-minute rounds —
+**3½ minutes of 000 on every public domain**. The morning roll had the
+same shape: k8s-1 (both proxies restarted, 40–80 s), k8s-2 (wedge at
+06:58, restart suppressed on both proxies, 278 s after the previous one —
+the routes happened to recover on their own), k8s-3 (restart, ~10 s).
+Three rolls in sixteen minutes guarantee the suppressed case.
+
+- **5a. Cooldown 600 s → 60 s** in
+  `ansible/roles/netbird-proxy-watchdog/defaults/main.yml`. The cooldown
+  exists so a genuinely broken backend cannot restart-loop the proxy; a
+  minute still does that (the URL watchdog needs three minutes to fire
+  anyway). With it, every peer loss — rolls back to back included — is a
+  ~3 s blip instead of a lottery between 3 s and 3½ min. Cheapest change
+  in this document, ships first.
+- **5b. Persist the peer identity.** Replace the DaemonSet with a
+  StatefulSet of three, required anti-affinity on hostname,
+  `PodDisruptionBudget maxUnavailable: 1`, and a small RBD volume per
+  replica holding NetBird's state directory. A drain then evicts the pod
+  through the PDB (the DaemonSet pod is not evicted at all today, it dies
+  with the reboot) and it returns within ~30 s **as the same peer**. Whether
+  the route manager then un-wedges without a restart is the hypothesis to
+  measure; the setup-key constraint (ephemeral peers reaped after >10 min
+  offline, NeedsLogin liveness restart) holds unchanged.
+- **5c. The structural exit: no domain routes at all.** The subnet-target
+  experiment settled what works: an HTTP-mode service with an IP target
+  forwards `Host: <ip>` and Traefik answers 404 (NetBird rejects a Host
+  override); an HTTP-mode service with a domain target needs mesh DNS on
+  the proxy, which answers SERVFAIL there; a **TLS-passthrough service
+  (`mode: tls`, `listen_port: 443`) with a subnet target works** — SNI and
+  Host reach Traefik untouched, it coexists with the HTTP listener on 443,
+  no `skip_tls_verify`. If every public service became such a passthrough
+  over a subnet resource covering the Cilium LB pool (`10.2.99.0/24`,
+  `fd99::/64`), there would be no domain resource left to wedge on, and
+  peer loss would be NetBird's documented HA failover (seconds). Costs: the
+  target's `host` is the VIP, so a reconciler (the external-dns-unifi
+  pattern, `PUT /api/reverse-proxies/services/{id}` from the Traefik
+  Service's `status.loadBalancer.ingress`) has to own it; the proxy's L7
+  features (auth, path rules) are lost — none are in use; and the pure
+  failover time is unmeasured, because today every kill wedged the domain
+  routes first. Decide after 5a and 5b have been measured on a real roll.
+- **Track the upstream bug.** The repo's note dates the dns-interceptor
+  behaviour to 2026-07; a search of NetBird's releases and issues turned up
+  no fix for the "already routed by peer / HA routing disabled" path.
+  Re-check on each NetBird bump.
+- **Side findings, each its own small change:** both proxy clocks are
+  ~3 min slow (the proxy logs SCT timestamps "in the future"; TLS
+  validity windows care) — NTP on proxy-1/proxy-2; a new service's
+  certificate reaches the IPv6-only member only after ~20 min of ACME
+  retries (Let's Encrypt validates over IPv4 and lands on the other member,
+  which has no token for it); proxy-1 and proxy-2 do not ship syslog to
+  Loki (edge-1 does), so every public-edge timeline is reconstructed over
+  SSH — point them at the alloy-syslog VIP.
+
+## Expected result
+
+| stage | removes |
+|---|---|
+| 1 | every double move (half of today's 5xx), the primary restarts, the 35-minute OpenBao/ESO outage, the Multus race |
+| 2 | the need for `--disable-eviction` at all |
+| 3 | the single-node ingress exposure; ECMP loses 1/3 instead of 1/2 |
+| 4 | home-assistant and grafana 503s entirely |
+| 5a | the 3½-minute public-edge blackout of a second peer loss within 10 min; every peer loss becomes a ~3 s restart |
+| 5b/5c | the restart itself, if the returning identity (5b) or the absence of domain routes (5c) proves to un-wedge — to be measured |
+
+What remains after all of it: one move per single-replica app per roll
+(gitea, immich, shelly-manager, music-assistant, wordpress), 30–60 s each,
+once, and a few seconds of public edge per roll. That is the honest floor
+for single-replica services on RWO storage and a mesh client that pins.
+
+## Transition
+
+1. Stage 5a is one Ansible default and a proxy deploy — ships first.
+2. Stage 1 is a script and a runbook change — no cluster change.
+3. Stage 5b's StatefulSet changes how the public edge enrols, so it gets
+   its own roll to measure, probed from a proxy.
+4. Stage 2 is one field on four CNPG clusters and five deleted PDBs; each
+   Kustomization applies without disruption.
+5. Stage 3 is a Traefik HelmRelease change; the rollout itself is a rolling
+   update under the new PDB.
+6. Stage 4 is `instances: 2` on two clusters; CNPG adds the replica online.
+7. Stage 5c only if 5a+5b leave a measurable public gap.
+
+Measure the next roll the same way: 5xx by router, the proxies' watchdog
+journals (or their syslog in Loki), BGP Down→Up gaps, pods per drain log.