hotfix(netbird-proxy): allow non-root :443 bind on the host network #409

2026-07-26T05:41:24Z

claude-bot commented

2026-07-26 05:41:24 +00:00

URGENT — both proxy members are crash-looping since the #406 deploy; the public edge is down for external clients.

The reverse-proxy runs as a non-root user. On the host network the host's default ip_unprivileged_port_start=1024 denies binding :443 → listen tcp :443: bind: permission denied in a restart loop on proxy-1 AND proxy-2 (verified live). Bridge mode never hit this because Docker sets that sysctl to 0 inside container network namespaces.

Fix: persist net.ipv4.ip_unprivileged_port_start=443 on the proxy hosts via the role (ansible.posix.sysctl, same pattern as keepalived-ntp). The sysctl governs v6 binds too despite the name. Once it lands, the crash-looping containers self-heal within their restart backoff — no recreate needed.

Merging auto-triggers the deploy-netbird-proxy pipeline which applies it to both hosts.

🤖 Generated with Claude Code

**URGENT — both proxy members are crash-looping since the #406 deploy; the public edge is down for external clients.** The reverse-proxy runs as a non-root user. On the host network the host's default `ip_unprivileged_port_start=1024` denies binding :443 → `listen tcp :443: bind: permission denied` in a restart loop on proxy-1 AND proxy-2 (verified live). Bridge mode never hit this because Docker sets that sysctl to 0 inside container network namespaces. Fix: persist `net.ipv4.ip_unprivileged_port_start=443` on the proxy hosts via the role (ansible.posix.sysctl, same pattern as keepalived-ntp). The sysctl governs v6 binds too despite the name. Once it lands, the crash-looping containers self-heal within their restart backoff — no recreate needed. Merging auto-triggers the deploy-netbird-proxy pipeline which applies it to both hosts. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit 2026-07-26 05:41:25 +00:00

hotfix(netbird-proxy): allow non-root :443 bind on the host network

ci/woodpecker/pr/pr-review Pipeline was successful

Details

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

5fa8c03827

The host-network switch (#406) broke both proxy members: the
reverse-proxy runs as a non-root user, and on the host netns the default
ip_unprivileged_port_start=1024 denies binding :443 — the container
crash-loops with 'listen tcp :443: bind: permission denied'. In bridge
mode Docker set the sysctl to 0 inside the container netns, which is why
the published-port setup never hit this.

Set net.ipv4.ip_unprivileged_port_start=443 persistently on the proxy
hosts (governs v6 binds too, despite the name). The crash-looping
container self-heals within its restart backoff once the sysctl lands.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

k.renner merged commit d0b449ed74 into master

2026-07-26 05:44:07 +00:00

k.renner referenced this issue from a commit

2026-07-26 05:44:07 +00:00

Merge pull request 'hotfix(netbird-proxy): allow non-root :443 bind on the host network' (#409) from netbird-proxy-port-sysctl into master

review-bot reviewed

2026-07-26 05:45:29 +00:00

review-bot left a comment

🤖 Automated review — model CLI default (complexity: complex), commit 5fa8c0382749538c036825089e878810c847f8e4.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

Hotfix for a live crash-loop: netbird-proxy moved to network_mode: host (PR #406) and runs as a non-root user, so bind() on :443 now hits the host's default net.ipv4.ip_unprivileged_port_start=1024 instead of Docker's per-netns override of 0. This PR adds an ansible.posix.sysctl task to the netbird-proxy role persisting net.ipv4.ip_unprivileged_port_start=443 via /etc/sysctl.d/90-netbird-proxy.conf, with reload: true so it takes effect immediately on the host netns the crash-looping container shares. I verified the role is only applied to proxy-1/proxy-2 (ansible/inventory/pve_hosts.yml, install-netbird-proxy.yml), that no other role applied to those hosts (common, docker-install, netbird-proxy-watchdog) touches this sysctl or runs a conflicting host-network service on a lower port, and that the module usage matches the existing keepalived-ntp role's pattern. No blocking issues found.

🟡 Suggestions

ansible/roles/netbird-proxy/tasks/main.yml:39-45: this sets the runtime value host-wide via reload: true (sysctl -p /etc/sysctl.d/90-netbird-proxy.conf), but persistence across a future reboot depends on sysctl.d file-ordering. If any lower-priority-sorted file (e.g. a distro default under /usr/lib/sysctl.d/, or something added later) also sets net.ipv4.ip_unprivileged_port_start with a filename that sorts after 90-, it would win on the next boot and reintroduce this exact outage silently. I didn't find such a file in this repo's roles today, but it's worth a one-time sysctl --system dry check on both hosts (sysctl --system 2>&1 | grep ip_unprivileged) after the deploy to confirm 90-netbird-proxy.conf is the last word, since the failure mode (silent revert to 1024 on reboot) is exactly the incident this PR is fixing.
Minor scope note for later cleanup, not blocking: VPS-1 (netbird-server role) also runs a network_mode: host service alongside the bridged Traefik (docker-compose.yml.j2:212 there), per the new comment's framing that VPS-1 shares :443 via Traefik SNI passthrough rather than binding directly. Worth double-checking that host isn't also non-root-binding a low port on host network, since it wasn't touched by this PR and the incident description implies only proxy-1/proxy-2 were affected — just flagging so it doesn't get missed if VPS-1's setup ever changes to bind directly.

Tests

There's no Ansible lint/syntax-check or molecule test infrastructure in this repo, and the deploy pipeline (.woodpecker/deploy-netbird-proxy.yml) runs ansible-playbook directly against the live hosts on merge to master with no gating check — consistent with how the rest of the repo's infra changes are validated (i.e., not blocking, matches existing convention). The change is a single idempotent, well-scoped module call; manual verification post-merge (confirm both containers stop crash-looping, and the sysctl --system ordering check above) is the appropriate verification given the repo's tooling.

Recommendation

Looks reasonable — no blocking issues; comments above are optional.

🤖 **Automated review** — model `CLI default` (complexity: complex), commit `5fa8c0382749538c036825089e878810c847f8e4`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary Hotfix for a live crash-loop: `netbird-proxy` moved to `network_mode: host` (PR #406) and runs as a non-root user, so `bind()` on `:443` now hits the host's default `net.ipv4.ip_unprivileged_port_start=1024` instead of Docker's per-netns override of `0`. This PR adds an `ansible.posix.sysctl` task to the `netbird-proxy` role persisting `net.ipv4.ip_unprivileged_port_start=443` via `/etc/sysctl.d/90-netbird-proxy.conf`, with `reload: true` so it takes effect immediately on the host netns the crash-looping container shares. I verified the role is only applied to `proxy-1`/`proxy-2` (`ansible/inventory/pve_hosts.yml`, `install-netbird-proxy.yml`), that no other role applied to those hosts (`common`, `docker-install`, `netbird-proxy-watchdog`) touches this sysctl or runs a conflicting host-network service on a lower port, and that the module usage matches the existing `keepalived-ntp` role's pattern. No blocking issues found. ## 🟡 Suggestions - `ansible/roles/netbird-proxy/tasks/main.yml:39-45`: this sets the runtime value host-wide via `reload: true` (`sysctl -p /etc/sysctl.d/90-netbird-proxy.conf`), but persistence across a future reboot depends on `sysctl.d` file-ordering. If any lower-priority-sorted file (e.g. a distro default under `/usr/lib/sysctl.d/`, or something added later) also sets `net.ipv4.ip_unprivileged_port_start` with a filename that sorts after `90-`, it would win on the next boot and reintroduce this exact outage silently. I didn't find such a file in this repo's roles today, but it's worth a one-time `sysctl --system` dry check on both hosts (`sysctl --system 2>&1 | grep ip_unprivileged`) after the deploy to confirm `90-netbird-proxy.conf` is the last word, since the failure mode (silent revert to 1024 on reboot) is exactly the incident this PR is fixing. - Minor scope note for later cleanup, not blocking: VPS-1 (`netbird-server` role) also runs a `network_mode: host` service alongside the bridged Traefik (`docker-compose.yml.j2:212` there), per the new comment's framing that VPS-1 shares `:443` via Traefik SNI passthrough rather than binding directly. Worth double-checking that host isn't also non-root-binding a low port on host network, since it wasn't touched by this PR and the incident description implies only `proxy-1`/`proxy-2` were affected — just flagging so it doesn't get missed if VPS-1's setup ever changes to bind directly. ## Tests There's no Ansible lint/syntax-check or molecule test infrastructure in this repo, and the deploy pipeline (`.woodpecker/deploy-netbird-proxy.yml`) runs `ansible-playbook` directly against the live hosts on merge to master with no gating check — consistent with how the rest of the repo's infra changes are validated (i.e., not blocking, matches existing convention). The change is a single idempotent, well-scoped module call; manual verification post-merge (confirm both containers stop crash-looping, and the `sysctl --system` ordering check above) is the appropriate verification given the repo's tooling. ## Recommendation Looks reasonable — no blocking issues; comments above are optional.

claude-bot referenced this pull request

2026-07-26 06:04:06 +00:00

feat(netbird): relay + STUN cluster members on the netcup proxy hosts #412

claude-bot referenced this pull request

2026-07-27 19:29:29 +00:00

[alert] NetbirdPublicEdgeMemberDown (monitoring) #420

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#409