Compare commits

...
Author SHA1 Message Date
claude-botandClaude Fable 5.1 8e9eedfd16 fix(netbird-proxy): watchdog restart cooldown 600 s -> 60 s
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Every loss of the routing peer a proxy has pinned wedges its route
manager; the flood watchdog restarts the container within 2 s and the
public edge is back after one failed probe round. Inside the 600 s
cooldown the restart is suppressed and the URL watchdog three one-minute
rounds are the only recovery: 3.5 min of 000 on every public domain,
measured 2026-09-07 10:28 on proxy-2 ("restart suppressed, last restart
567s ago"). The morning node roll hit the same suppression on both
proxies at 06:58 and survived only because the routes recovered by luck.

Rolling reboots lose one peer every few minutes by design, so with 600 s
the second roll is always the unlucky one. A minute keeps the loop
protection (one restart per probe interval at most).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-07 10:38:28 +02:00
k.renner 57b2485487 Merge pull request 'fix(ceph): state the cluster CIDR — host-device leaves Rook's canary no NIC' (#807) from ceph-address-ranges into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#807
2026-09-07 05:22:30 +00:00
k.renner cbd47596bf Merge pull request 'fix(talos): answer ARP only on the link that owns the address' (#804) from fix/arp-flux-second-nic into master
Reviewed-on: k.renner/homelab#804
2026-09-07 05:17:09 +00:00
claude-botandClaude Fable 5.1 4d629bfd66 fix(talos): answer ARP only on the link that owns the address
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Every node has a second, address-less NIC (enp87s0) up on the untagged
fallback network, which shares the mgmt VLAN's broadcast domain. With
the Linux default arp_ignore=0 the host answers ARP for the node IP from
that NIC too, so the UDM can learn 10.2.40.x against enp87s0's MAC.
Inbound IPv4 then arrives on a device Cilium is not attached to, never
reaches cil_from_netdev, and is routed straight back to the UDM by the
default route — a loop that made every IPv4 LoadBalancer VIP on the
affected node unreachable for 13.5 hours. IPv6 was untouched because
NDP only answers on the interface that owns the address.

The flux was measurable on all three nodes: an ARP request for a node
IP from a neighbour returned replies from two MACs. Which node's entry
had actually flipped at the UDM was a race, and it moved during the
diagnosis.

arp_ignore=1 answers only when the target address is configured on the
receiving interface; arp_announce=2 stops the host from sourcing ARP
from an interface that does not own the address. Applied live with
talosctl on all three nodes before this commit and verified with
enp87s0 up: each node now answers from its mgmt0 MAC only.

A node property, so it lives next to the existing enp87s0 sysctls in
each machine patch rather than in the site patch.

Refs #793, #576

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-06 23:46:51 +02:00
4 changed files with 24 additions and 2 deletions
@@ -32,8 +32,13 @@ netbird_watchdog_fail_threshold: 3
# Minimum seconds between watchdog-issued restarts, so a genuinely broken
# backend can't put the proxy into a restart loop. Shared by both the URL-probe
# watchdog and the flood watchdog below.
netbird_watchdog_cooldown_seconds: 600
# watchdog and the flood watchdog below. One minute: the route-manager wedge
# recurs on every routing-peer loss, and rolling node reboots lose one peer
# every few minutes; a suppressed restart there leaves the wedge standing
# until the URL probe's three rounds fire (3.5 min of 000 on every domain,
# measured 2026-09-07). A minute still caps a loop at one restart per
# probe interval.
netbird_watchdog_cooldown_seconds: 60
# Log fingerprint of the reverse-proxy route-manager wedge, matched as a plain
# substring in the proxy's docker logs by the flood watchdog. On this build the
+7
View File
@@ -14,6 +14,13 @@ machine:
# Disabling v6 on a VLAN parent does not affect the VLAN child.
net.ipv6.conf.enp2s0f0np0.disable_ipv6: "1"
net.ipv6.conf.enp87s0.disable_ipv6: "1"
# That network also carries the mgmt VLAN's ARP. Answer ARP only on the
# link that owns the address, or the UDM learns the node IP against
# enp87s0's MAC and delivers IPv4 to a NIC Cilium is not attached to (#793).
net.ipv4.conf.all.arp_ignore: "1"
net.ipv4.conf.default.arp_ignore: "1"
net.ipv4.conf.all.arp_announce: "2"
net.ipv4.conf.default.arp_announce: "2"
install:
# Select disks by serial only. A broad selector such as
# `disk.transport == 'nvme'` matches every drive in the machine, and the
+5
View File
@@ -12,6 +12,11 @@ machine:
# cluster IPv6 egress.
net.ipv6.conf.enp2s0f0np0.disable_ipv6: "1"
net.ipv6.conf.enp87s0.disable_ipv6: "1"
# Answer ARP only on the link that owns the address — see k8s-1.yaml (#793).
net.ipv4.conf.all.arp_ignore: "1"
net.ipv4.conf.default.arp_ignore: "1"
net.ipv4.conf.all.arp_announce: "2"
net.ipv4.conf.default.arp_announce: "2"
install:
# Select the installer target by serial so it cannot land on the OSD drive
# or on EPHEMERAL, whatever order the kernel enumerates them in.
+5
View File
@@ -8,6 +8,11 @@ machine:
# k8s-1.yaml.
net.ipv6.conf.enp2s0f0np0.disable_ipv6: "1"
net.ipv6.conf.enp87s0.disable_ipv6: "1"
# Answer ARP only on the link that owns the address — see k8s-1.yaml (#793).
net.ipv4.conf.all.arp_ignore: "1"
net.ipv4.conf.default.arp_ignore: "1"
net.ipv4.conf.all.arp_announce: "2"
net.ipv4.conf.default.arp_announce: "2"
install:
# Select the installer target by serial so it cannot land on the OSD drive
# or on EPHEMERAL, whatever order the kernel enumerates them in.