claude-bot
853928b6b1
Both of these were learned the expensive way today and existed only in commit
messages and PR descriptions, which is not where anyone looks.
providers.tf carried a warning that a UniFi PUT may require _id in the body,
which restapi does not send, and that a 400 should be answered by tainting the
object so CI recreates it. That is wrong on this controller. Tested against a
live rule with exactly the body tofu sends: HTTP 200, object returned intact,
rule byte-identical before and after, 60 rules before and after. Confirmed at
scale by apply 5675 — all 60 modified, no 400s, ci.renner.uno and
immich.renner.uno still 200.
That wrong warning had a cost. A note saying an apply might break firewall
rules is one nobody argues with, so the apply did not get run, and the
renner-uno-k8s-mariadb bucket added in b976a67b sat uncreated for three days
while the WordPress backup failed nightly with "The specified bucket does not
exist". The warning is replaced with the evidence that disproves it.
unifi_nat_hairpin.tf now records that the 60-rule `- _id` diff is permanent and
cosmetic, and lists the four attempts to remove it with the plan number that
disproved each: ignore_changes_to (5661), ignore_all_server_changes (5662),
restapi 3.x ignore_server_additions (5667), and create_returns_object (5677).
The first three cannot work in principle — `data` is a configured attribute, so
Terraform core diffs it against state before provider-side ignore logic runs.
The fourth was the better theory, that PUT responses re-polluted `data`, and it
is wrong too: the diff returned on the plan immediately after the apply. That
means the response is written into `data` on the read path, and settling it
needs the provider source rather than more guesses at its schema.
No behaviour change. Documentation only, so the next person inherits the
findings instead of the dead ends.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
174 lines
7.9 KiB
Terraform
174 lines
7.9 KiB
Terraform
# Hairpin DNAT: LAN clients reaching the public NetBird proxy addresses get
|
|
# rewritten to the cluster Traefik VIP, so public DNS answers work on-LAN
|
|
# without the traffic leaving via the WAN. Companion of the external-dns
|
|
# labelFilter change (kubernetes/infrastructure/external-dns-unifi): resources
|
|
# labeled dns.renner.uno/public=true register publicly instead of on the UDM,
|
|
# so phones roaming wifi<->cellular always hold a publicly-valid answer.
|
|
#
|
|
# Managed via Mastercard/restapi against the UDM's v2 NAT endpoint
|
|
# (/proxy/network/v2/api/site/default/nat) because no UniFi Terraform
|
|
# provider models NAT translation fields as of 2026-07 (checked:
|
|
# ubiquiti-community 0.55, filipowm #104 open, resnickio shell-only).
|
|
#
|
|
# Every rule uses the same shape regardless of address family: destination
|
|
# filter ADDRESS_AND_PORT with a specific proxy address and a single port.
|
|
# Group-based (FIREWALL_GROUPS) destinations are deliberately not used:
|
|
# on Network 10.5.62 the IPv6 variant silently drops the address match in
|
|
# ip6tables and DNATs ALL v6 traffic on the listed ports (verified live,
|
|
# 2026-07-23 — this broke the network twice). Controller constraints:
|
|
# - ADDRESS_AND_PORT may not carry firewall_group_ids (400 otherwise).
|
|
# - The port field takes a single port or a range, so ports 80/443 are
|
|
# separate rules.
|
|
# - rule_index must be explicit and unique across NAT rules (400 on
|
|
# duplicates); indices here are static per network+endpoint+port so
|
|
# adding a VLAN never renumbers existing rules.
|
|
# - Rules take ~60-90s after apply before the dataplane enforces them;
|
|
# verify no earlier than that.
|
|
|
|
locals {
|
|
# Public addresses of proxy.renner.uno (NetBird reverse-proxy cluster) and
|
|
# the cluster-ingress VIP (Traefik LB) each family translates to. Keep the
|
|
# addresses in sync with the proxy hosts' DNS records. slot orders the
|
|
# rule_index derivation — never reuse a freed slot for a new endpoint.
|
|
hairpin_endpoints = {
|
|
"proxy v4" = {
|
|
address = "45.157.177.26"
|
|
ip_version = "IPV4"
|
|
target = "10.2.99.86"
|
|
slot = 0
|
|
}
|
|
# Attribution corrected 2026-07-27: these two addresses were previously
|
|
# assigned to the OPPOSITE members. DNS is the operational truth —
|
|
# `getent ahosts proxy-1.renner.uno` returns only 2a03:4000:28:f6e:…:9a29
|
|
# (the v6-only member), and proxy-2.renner.uno returns
|
|
# 2a03:4000:4b:c1e:…:8f85 plus 45.157.177.26 (the dual-stack member).
|
|
# Purely a labelling fix: both entries DNAT to the same target, so the
|
|
# hairpin behaved identically before and after. Slots are unchanged — the
|
|
# "never reuse a freed slot" rule above applies to the keys, not the values.
|
|
"proxy-1 v6" = {
|
|
address = "2a03:4000:28:f6e:a4c7:59ff:fe1e:9a29"
|
|
ip_version = "IPV6"
|
|
target = "fd99::86"
|
|
slot = 1
|
|
}
|
|
"proxy-2 v6" = {
|
|
address = "2a03:4000:4b:c1e:4474:43ff:fedf:8f85"
|
|
ip_version = "IPV6"
|
|
target = "fd99::86"
|
|
slot = 2
|
|
}
|
|
}
|
|
|
|
hairpin_ports = {
|
|
"80" = 0
|
|
"443" = 1
|
|
}
|
|
|
|
# Networks that get hairpin rules. in_interface wants the network _id;
|
|
# idx is a static per-network slot for rule_index derivation — never
|
|
# reuse a freed idx for a new network.
|
|
#
|
|
# Deliberately absent: (40) Server idx=4, (42) Ingress idx=5, and
|
|
# (49) k8s Ceph idx=6. The cluster nodes have interfaces on those VLANs,
|
|
# so a DNATed flow's reply would go directly over the shared L2 and
|
|
# bypass the UDM's conntrack (asymmetric, connection drops) — and
|
|
# clients there are the nodes themselves, whose DNS chain (CoreDNS ->
|
|
# k8s_gateway) already resolves the public names in-cluster. Their idx
|
|
# slots stay reserved.
|
|
hairpin_networks = {
|
|
"Default" = { id = "5d4fba77ce63ee614841d53d", idx = 0 }
|
|
"(10) Management" = { id = "600334abf9ee0c059da6be8f", idx = 1 }
|
|
"(20) Video" = { id = "61d461ddf9ee0c1243264e2f", idx = 2 }
|
|
"(30) IOT" = { id = "60a0d9c1f9ee0c0571928a79", idx = 3 }
|
|
"(50) Privat" = { id = "6259dd6c1f9a5409193f4167", idx = 7 }
|
|
"(60) Phone" = { id = "69e0d3d00217ad9fadb26fff", idx = 8 }
|
|
"(70) Access" = { id = "66729b669eac7341fe2fc8eb", idx = 9 }
|
|
"(122) Multi Transfer" = { id = "6346887e55fba60ba53dacba", idx = 10 }
|
|
"(123) Multi LAN" = { id = "6346890d55fba60ba53dacbe", idx = 11 }
|
|
"(200) Mutti" = { id = "6236c1b6f9ee0c05a16732d4", idx = 12 }
|
|
}
|
|
|
|
# network x endpoint x port, uniform across families.
|
|
# rule_index = 100 + idx*6 + slot*2 + port_offset (13 idx slots -> 100..177)
|
|
hairpin_rules = merge([
|
|
for net_name, net in local.hairpin_networks : {
|
|
for combo in setproduct(keys(local.hairpin_endpoints), keys(local.hairpin_ports)) :
|
|
"${combo[0]} ${net_name} ${combo[1]}" => {
|
|
network_id = net.id
|
|
endpoint = local.hairpin_endpoints[combo[0]]
|
|
port = combo[1]
|
|
rule_index = 100 + net.idx * 6 + local.hairpin_endpoints[combo[0]].slot * 2 + local.hairpin_ports[combo[1]]
|
|
}
|
|
}
|
|
]...)
|
|
}
|
|
|
|
resource "restapi_object" "hairpin" {
|
|
for_each = local.hairpin_rules
|
|
|
|
path = "/nat"
|
|
|
|
# KNOWN PERMANENT DIFF — cosmetic. Do not try to fix it with a provider
|
|
# setting; four have been tried and none work. Recorded here so the next
|
|
# person does not spend a day rediscovering that.
|
|
#
|
|
# Every plan shows all 60 of these as "will be updated in-place" with the
|
|
# entire diff being one line:
|
|
#
|
|
# ~ data = jsonencode( ~ { - _id = "6a61c7d7bbd7b2ca90981f51"
|
|
# # (15 unchanged attributes hidden) } )
|
|
#
|
|
# The UDM assigns _id and returns it; the provider puts the response into
|
|
# `data`; the `data` below never sets it. So state carries a field the config
|
|
# does not. It survives an apply: 5675 rewrote all 60, and the very next plan
|
|
# (5677) showed the same 60 again.
|
|
#
|
|
# Tried, with the plan that disproved each — do not repeat these:
|
|
#
|
|
# ignore_changes_to = ["_id"] 5661 60 remained
|
|
# ignore_all_server_changes = true 5662 60 remained
|
|
# restapi 3.x + ignore_server_additions = true 5667 60 remained, and
|
|
# added an api_data diff
|
|
# write_returns_object -> create_returns_object 5677 60 remained
|
|
#
|
|
# The first three cannot work in principle: `data` is a configured attribute,
|
|
# so Terraform core diffs config against state before any provider-side ignore
|
|
# logic runs. The fourth was the better theory — that PUT responses were
|
|
# re-polluting `data` — and it is wrong too, which means the response is
|
|
# written into `data` on the READ path. Settling it properly needs the
|
|
# provider source, not more guesses at the schema.
|
|
#
|
|
# It is safe to apply through. Verified twice: a single PUT without _id
|
|
# returns 200 and leaves the rule byte-identical (see providers.tf), and
|
|
# apply 5675 modified all 60 with no errors and no service impact.
|
|
data = jsonencode({
|
|
description = "Hairpin ${each.key}"
|
|
type = "DNAT"
|
|
enabled = true
|
|
exclude = false
|
|
in_interface = each.value.network_id
|
|
ip_address = each.value.endpoint.target
|
|
ip_version = each.value.endpoint.ip_version
|
|
protocol = "tcp_udp"
|
|
rule_index = each.value.rule_index
|
|
logging = false
|
|
is_predefined = false
|
|
pppoe_use_base_interface = false
|
|
setting_preference = "manual"
|
|
destination_filter = {
|
|
filter_type = "ADDRESS_AND_PORT"
|
|
address = each.value.endpoint.address
|
|
port = each.value.port
|
|
firewall_group_ids = []
|
|
invert_address = false
|
|
invert_port = false
|
|
}
|
|
source_filter = {
|
|
filter_type = "NONE"
|
|
firewall_group_ids = []
|
|
invert_address = false
|
|
invert_port = false
|
|
}
|
|
})
|
|
}
|