ETP=Local on Traefik exposed real LAN client IPs to backends. Home
Assistant has ip_ban_enabled + a 50-entry ip_bans.yaml accumulated
from failed login attempts; previously these bans were smeared across
the handful of cluster node IPs (masquerade) and rarely matched a
live client. With real source IPs reaching HA, individual LAN clients
match their own ban entries and get 403 on every request.
The other six BGP-pool Services keep ETP=Local — they don't have the
trusted_proxies / X-Forwarded-For / login-attempt-ban pattern that
ingress-via-Traefik does. Re-enabling ETP=Local on Traefik is gated
on tightening HA's HTTP config (and any other Traefik backend with
strict source-IP checks).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cilium BGPv2 advertises ETP=Local Service IPs only from nodes that host
a Ready endpoint, so the UDM stops ECMPing across non-backend nodes and
the in-cluster forwarding hop disappears. Multi-replica services
(traefik, kube-dns-lb, alloy-syslog, mosquitto-iot) keep one path per
replica-hosting node. Single-replica services (vscode, ftp-scanner,
syncthing-sync) lose nothing — they already had a single point of
failure on pod loss.
Stripped stale L2-era ETP=Cluster comments from ftp-scanner, mosquitto-iot,
and alloy-syslog. Those warnings cited the ARP-cache trap
(cilium-l2-etp-local-l3-routing-trap) and presumed an L2-announce
ingress topology; they don't apply to BGP-advertised VIPs because
Cilium's BGP control plane only originates the prefix from
endpoint-hosting nodes (verified against upstream docs for v1.19).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Constrain kubelet auto-detected node IP to 10.2.40.0/24 so it picks
eth0 (mgmt LAN) instead of landing on eth1/v30iot. Cluster-wide
patch — applies to all 4 nodes on next bootstrap.
Already live via talosctl patch on all 4 nodes; this commit puts
the setting in the bootstrap input layer so future regenerations
of controlplane.yaml / worker.yaml include it.
Both controlplane.yaml and worker.yaml are listed in .gitignore as
generated bootstrap artifacts. controlplane.yaml had been tracked
since before the gitignore rule was added (gitignore doesn't remove
tracked files). Untracking it matches the design intent — these
configs are generated during cluster bootstrap, not edited in repo.
Kubelet's auto-detected node IP was landing on eth1/v30iot
(10.2.30.0/24) instead of eth0/mgmt LAN (10.2.40.0/24). The repo's
network-design memory note explicitly states control-plane traffic
+ hostNetwork pod IPs must stay on mgmt LAN. Adding
machine.kubelet.nodeIP.validSubnets pins it.
Already live-applied via talosctl patch + kubelet restart on all 4
nodes. This commit persists the change for future re-applies.
This reverts c7d6524 and 6b4610d.
The dummy0-based local-binding approach broke node InternalIP
detection. Cilium walks interfaces in index order and dummy0
(index 3) is lower than eth0/eth1, so once dummy0 had global-scope
LB IPs Cilium re-detected node IPs from there. This cascaded to
apiserver→kubelet routing breakage, pod IPs being allocated from
the wrong range, and eBPF state corruption that persisted past
agent restarts.
hostNetwork pods don't go through CoreDNS by default, so the
kubernetes.default.svc hostname doesn't resolve and the reconcile
loop silently failed on the curl. The auto-injected
KUBERNETES_SERVICE_HOST/PORT env vars give the apiserver Service
IP+port directly — Cilium kpr DNATs the ClusterIP to a live
apiserver endpoint.
Binds every LoadBalancer Service's external IP as a /32 (IPv4) or
/128 (IPv6) on each node's dummy0 interface. Cilium's eBPF kpr
intercepts TCP/UDP for these IPs at the TC ingress hook before the
kernel's local-input decision, so local binding doesn't affect data
flow — it only makes the kernel respond to ICMP echo locally.
Why: Cilium L2 announcements and BGP both publish "I am this IP" to
the network without binding the IP locally on any host. ICMP echo
to LB IPs loops via the default route until TTL exhausts. kube-proxy
IPVS used to bind LB IPs on kube-ipvs0; we run full kpr instead.
Verified the binding+kpr composition works: bound 10.2.99.81 on
k8s-edge's dummy0 manually, ping returned 0% loss, TCP to 1883/8883
still hit the mosquitto backend, Shelly MQTT session unaffected.
Upstream Cilium PRs #37623/#40565 proposed an in-Cilium fix but
neither merged.
Ingress-only ns. Same pattern as crowdsec (9908183). Affects only
the pod-network components (operator, ceph-csi-controller-manager,
CSI ctrlplugins) — Ceph daemons run on hostNetwork and are outside
Cilium endpoint policy.
Ingress-only ns. ICMP allow added to the existing ingress section
only — adding an egress section to an ingress-only CNP transitions
it to egress-enforced mode and was the cause of attempt 66dbe9f's
DNS/apiserver/world-egress breakage.
Same pattern as home-assistant (d61b24d). Only modifies the
default-deny-ingress CNP; the allow-ftp-scanner and allow-syncthing-sync
sidecar policies are left untouched.
Adds an explicit ICMP allow (all standard ICMPv4 + ICMPv6 types, both
directions) inside HA's existing default-deny CNP. Hubble was showing
~115 ICMPv6 DestinationUnreachable POLICY_DENIED drops per 2 minutes
from HA to coredns, despite the Cilium docs claiming "ICMP related to
a connection otherwise allowed by the policy" should pass automatically.
The CT-RELATED path isn't working empirically.
Per-namespace shape (proven safe in 3c517a0); type enumeration from
66dbe9f. NOT a CCNP — prior CCNP attempts (7f95486 / 66dbe9f / 7da87bf
/ 4fc52db) all broke unrelated traffic in this cluster.
Starting with home-assistant only as a smoke-test for the wider
roll-out across other default-deny namespaces.
Today's Garage outage at 02:00-04:00 UTC silently lost 3 of 6 daily
backups (paperless, audiobookshelf, authentik all FailedValidation
because BackupStorageLocation was Unavailable). GarageDown alert
fired, but nothing surfaced per-schedule backup loss. Add three
alerts covering validation failure, failed/partial backup, and
30h+ stale schedule.
ha.renner.uno and grafana.renner.uno don't exist as Ingress hosts —
Home Assistant lives at home.renner.uno, Grafana at dash.renner.uno
(already in list). NXDOMAIN every 30s was the cluster's top error
generator (721 lines/hr in Loki). Rename ha → home, drop grafana
duplicate.
Chart ships no defaults for singleBinary.resources, which let the kernel
OOM-kill loki-0 on 2026-05-14 (exit 137 during initial install). Pin
requests at 100m / 512Mi and limit memory to 2Gi (~4x steady-state RSS
of 540Mi at current 37 lines/sec ingest). CPU stays unburstable.
Loki had no Prometheus scrape config — no loki_* metrics, no way to
alert on the log stack itself going sideways. ServiceMonitor covers
the single-binary + canary services, alerts cover Down / 5xx /
panics / canary-detected log loss.
The CCNP `allow-icmp-default-deny-namespaces` (added in 66dbe9f, then
shrunk in 7da87bf to ingress-only) was breaking *all* traefik->backend
routing — even after the egress section was removed.
Symptom: traefik logs filled with 504 / 499 / 30s upstream timeouts to
every backend in the matched namespaces (authentik, home-assistant,
audiobookshelf, music-assistant, monitoring/grafana, ...). Removing
the CCNP from the cluster restored ingress immediately.
I don't have a full root-cause yet for why an ingress-only CCNP with
matchExpressions In on a list of namespaces causes Cilium to drop
established traefik->pod TCP flows, but empirically: this shape of
policy is incompatible with the existing per-namespace CNPs in this
cluster. Falling back to per-namespace ICMP allows is the next attempt
(in a separate commit, after the cluster has stabilised).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The egress section transitioned namespaces that previously had
ingress-only CNPs (velero, monitoring, rook-ceph CSI) into
egress-policy-enforced mode with only ICMP egress allowed — breaking
DNS lookups to coredns, kube-apiserver access, and world egress.
The user-facing ICMP problem was coredns→HA ICMP DU drops, which is
ingress on HA's side. The ingress rule alone is sufficient. ICMP
egress in the per-namespace egress-enforced CNPs already flows via the
broad toEntities rules (host/remote-node/world don't constrain
protocol so ICMP is included).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces both the per-namespace ICMP-error addition (3c517a0) and the
broad CCNP that broke kube-system (reverted in 3c517a0). Adds a single
CCNP that:
* Scopes its endpointSelector via `matchExpressions In` to the explicit
list of namespaces that already opted into default-deny — so it only
widens what's already enforced and CANNOT transition kube-system or
any other unmanaged namespace into policy-enforced mode.
* Allows the full standard ICMPv4 (0-30) and ICMPv6 (1-4, 128-153) type
set in both directions. Cilium's ICMPField requires an explicit type
per entry; there is no wildcard, so the list is enumerated.
This covers Echo Request/Reply (ping), Destination Unreachable / Port
Unreachable, Time Exceeded (traceroute), Packet Too Big (PMTUD), and
the full ND/RA/RS set so v6 neighbour discovery works.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The CCNP added in 7f95486 used `endpointSelector: matchExpressions Exists`
on `io.kubernetes.pod.namespace`, which matched every pod in the cluster
including kube-system. That transitioned pods like metrics-server (which
previously had NO policies and was default-allow) into policy-enforced
mode where only ICMP errors were allowed, breaking apiserver:6443 and
kubelet:10250 egress and likely impacting other kube-system workloads.
Reverts the CCNP and adds the ICMP-error ingress allow only inside
home-assistant's existing default-deny CNP — i.e. only in namespaces
that already opted into policy enforcement. Will extend to other
namespaces individually if Hubble shows ICMP-error drops there.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the `policy-verdict:none ... DENIED (ICMPv4
DestinationUnreachable(Port))` drops Hubble was showing for
home-assistant <-> coredns and similar. When a UDP query lands on a
closed coredns port (SO_REUSEPORT race, draining pod), the receiver
sends ICMP DU back; without this CCNP the sender's namespace
default-deny CNP drops the ICMP error and the kernel just keeps
retransmitting.
Scoped via matchExpressions Exists on `io.kubernetes.pod.namespace`
so reserved-identity endpoints (host, remote-node, world, health,
kube-apiserver) are excluded — only real pods are widened. Allows
only the error class (DU + TimeExceeded, v4 + v6); ICMP Echo is
NOT allowed by this policy.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a cluster-wide policy scoped strictly to the `health` reserved
identity that permits ingress on TCP/4240 + ICMP echo from `host` and
`remote-node`, and matching egress. This is required so the per-node
cilium-health endpoints can be probed across nodes; without it Hubble
shows `policy-verdict:none ... DENIED` between `remote-node` and
`health` and `cilium-health status` reports 0/N reachable.
Scoped via matchLabels: reserved.health, so it cannot affect any
user pod — only the Cilium-internal health endpoints.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The CCNP with endpointSelector: {} isolated previously-unprotected pods
cluster-wide, including the reserved 'health' identity used by cilium-
health endpoints. Effect: cilium-health TCP/4240 and ICMP EchoRequest
between nodes started getting denied, and the policy did NOT clear
the original HA→coredns ICMP NACK drops anyway (those are an
artifact of Cilium not conntrack-correlating ICMP error messages to
the originating UDP flow — a known interaction, not fixable with a
naive cluster-wide allow rule).
Reverting to think this through properly. Per-namespace ICMP allows
that scope by label (not cluster-wide endpoint capture) is the right
direction.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previous version had ingress/egress with only icmps: block, no
from/to source selector. Cilium requires a source/destination match
on every ingress/egress rule; without it the rule is incomplete and
never matches — verified empirically (Hubble kept logging the same
DENIED ICMPv4/v6 DestinationUnreachable drops after the CCNP applied).
Source/destination entities: cluster (other pods), host + remote-node
(kubelet/control plane), world (external routers emitting PMTU and
Fragmentation-Needed). ICMP error class is a trusted control protocol;
wide allow is appropriate.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The per-namespace default-deny CNPs only allow TCP/UDP on specific
ports. ICMP errors arrive as unrelated flows from Cilium policy's
L3/L4 perspective — they're not conntrack-correlated to the
originating UDP/TCP — so without an explicit allow they get DENIED.
That breaks:
v4 type 3 (Dest Unreachable) — UDP port-unreachable + Path MTU
v4 type 11 (Time Exceeded) — traceroute, TTL-loop visibility
v6 type 1 (Dest Unreachable) — same as v4 type 3
v6 type 2 (Packet Too Big) — IPv6 PMTU Discovery silent blackholes
v6 type 3 (Time Exceeded) — same as v4 type 11
Hubble started spamming `home-assistant → coredns DENIED ICMPv4
DestinationUnreachable(Port)` as soon as a broad-DROP monitor was
armed; the ICMP NACK was getting dropped at the policy boundary.
CCNP allows these specific types in both ingress and egress for every
endpoint cluster-wide. Echo (ping) not included — diagnostics-only,
add per-namespace when needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
pbs.obg.renner.uno resolves to 10.3.40.96 — an offsite Proxmox Backup
Server on a remote network reached over WAN (~50ms latency). Checking
it from inside the cluster doesn't measure what it should: it conflates
WAN reachability, in-cluster DNS path stability (CoreDNS → k8s-gateway
→ proxmox-dns LXC → UDM), and the PBS itself. The intermittent UDP
loss on the DNS path produced spammy i/o timeout logs and false-
positive gatus failures.
Monitor offsite PBS from the offsite side, or via a check that doesn't
loop through cluster DNS.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two CNP gaps caught by the post-rollout audit:
1. alloy-syslog drops UDP from world-ipv4. ETP=Cluster on the LB
SNATs to the receiving node's cilium_host IP, which Cilium classifies
as world-ipv4 (not host) since cilium_host isn't a known endpoint.
The broad monitoring CNP's `fromEntities: host` misses this.
New scoped CNP `allow-syslog-world-ingress` opens world ingress
only on the actual syslog ports (514/UDP+TCP, 1514/UDP, 5514/UDP,
6514/TCP) and only for the alloy-syslog pod.
2. influxdb-ns missed `apps` ingress, breaking gatus' services_influxdb
healthcheck (success=false on every probe). Added apps with the
`app: gatus` label so the allow stays narrow.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Ceph storage layer must never be killed by kubelet eviction or
the Linux OOM killer. Pre-this commit, every Ceph daemon (mon, mgr,
osd, mds, rgw) ran with no priorityClassName and Burstable QoS —
which means they'd be picked early by both the eviction algorithm
under node pressure AND the kernel OOM killer.
Three changes:
cephcluster.yaml: spec.priorityClassNames.all = system-node-critical
(PriorityValue 2000001000 — the highest built-in class). Applies
to mon, mgr, osd, cleanup daemons. Memory requests bumped to equal
limits for mon/osd/mgr so they land in Guaranteed QoS; the kernel
then clamps oom_score_adj as low as possible.
storageclass.yaml (CephFilesystem MDS): same priority class. Losing
the active MDS hangs every CephFS mount until standby promotes —
worth pinning. memory bumped to 4Gi/4Gi for Guaranteed.
objectstore.yaml (CephObjectStore RGW): system-cluster-critical
rather than -node-critical. Object storage outage is recoverable
and blocks nothing in-cluster, so less aggressive priority is
appropriate, but still above app pods. memory 512Mi/512Mi for
Guaranteed.
CPU limits intentionally NOT added on any daemon — Ceph benefits
from CPU bursting during recovery/backfill, and the cluster runs at
~15% actual CPU so contention isn't real.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two Alertmanager alerts cleared at the source:
1. KubeDeploymentRolloutStuck / KubeContainerWaiting / KubePodNotReady
on apps/gatus. The deployment's RWO PVC (gatus-data, ceph-block)
can't be mounted twice. Default RollingUpdate strategy tries to
start the new pod BEFORE terminating the old one — surge=25% of
1 replica rounds up to 1 surge pod. The new pod hangs in
ContainerCreating indefinitely (no event, just a stuck volume
attachment) and ProgressDeadlineExceeded fires.
Recreate strategy guarantees the old pod releases the volume first.
Same fix the home-assistant Deployment already uses for the same
reason.
2. TargetDown blackbox-tcp / monitoring. The only target was the
keepalived VIP for the legacy authentik-HA Postgres LXC pair at
10.2.40.135:5432. That LXC stack was decommissioned when authentik
moved to in-cluster CNPG (ak-pg). Probe deleted; the comment
block in the file documents what's no longer there.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Hubble drops on alertmanager-kube-prometheus-stack-alertmanager:9093
from flux-system/notification-controller within 2 min of the previous
CNP landing. Flux's alerting path posts to Alertmanager; without this
rule Flux reconciliation alerts stop firing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
monitoring: intra + traefik (Grafana/Loki ingress) + database (gf-pg
CNPG operator) + host/remote-node (kubelet probes, kube-prometheus-
stack admission webhooks, alloy-syslog SNAT-to-node from ETP=Cluster).
rook-ceph: intra + monitoring (Prometheus scrape of pod-network CSI
metrics) + host/remote-node (kubelet drives CSI RPCs + rook admission
webhook). Note: only pod-network components in this namespace are
gated by this CNP — every Ceph daemon (mons, OSDs, MDSs, RGW, mgr,
CSI nodeplugins, exporter, crashcollector) runs on hostNetwork and is
outside Cilium endpoint policy entirely. Those are firewalled by the
UDM at the VLAN boundary, not here.
Still unprotected after this commit:
music-assistant -- entirely hostNetwork, CNP can't apply
security, storage -- empty namespaces
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Missed in the previous batch. im-pg lives in this namespace, but the
CNPG operator that reconciles it lives in the database namespace.
Without this rule, the operator's health/failover/role queries hit
the new default-deny and im-pg eventually stops being managed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
With pod CIDR (10.244.0.0/16) BGP-advertised to the UDM and the
v30iot → pod-CIDR firewall rule in place for return-path traffic,
every pod in every namespace without a CNP is reachable from any
device VLAN that can route. Defense-in-depth: lock down the
namespaces that don't already have one.
Five new CNPs, all the same minimal shape (intra-namespace + monitoring
scrape + host/remote-node + any specific cross-namespace consumer):
database : webhook ingress from kube-apiserver (host) only.
No app pods land here — the actual postgres Clusters
live in each consumer's namespace.
influxdb : home-assistant (HACS writes) + grafana queries
immich : traefik (web UI) + velero (PVC backup mount)
velero : no cross-namespace ingress, just probes
crowdsec : traefik bouncer middleware (LAPI :8080)
Not in this commit, addressed separately:
monitoring, rook-ceph -- extensive cross-ns flows, deserves audit
music-assistant -- hostNetwork, CNP doesn't apply
security, storage -- empty namespaces
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Final cleanup of the BGP rollout. Every Service that used to live in
default-pool (10.2.40.64-95 / fd40::40-ff via L2 announce) has been
migrated to bgp-pool (10.2.99.0/24 / fd99::/64 via BGP), so the
default-pool definition + its L2 announce policy are dead weight.
iot-pool + the `iot` L2 announce policy stay. They exist specifically
so home-assistant-iot lives on v30iot itself — IoT devices reach HA
on-link via L2, no UDM L3 routing, no inter-VLAN firewall rule. That
is the intentional design and the BGP rollout doesn't change it.
bgp-pool comment trimmed of the phase-1/2/3 rollout description that
referred to default-pool — now the end state.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Ceph object storage has no business with a routable LB IP. The
LoadBalancer Service was a relic of the Garage→Ceph backfill, where the
migration tooling lived off-cluster and needed direct RGW access. In-
cluster consumers (OBCs, S3 clients) reach the daemon via the existing
ClusterIP `rook-ceph-rgw-migration`, which is unchanged.
Drops the Service entirely. Cilium releases 10.2.99.91 + fd99::91 back
to the bgp-pool; BGP withdraws the /32 advertisement. Hubble showed no
external traffic to the LB for at least 30min before deletion (only
internal 127.0.0.1 health-checks in the rgw pod logs).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Missed in the previous lb-pool batch. vsftpd advertises this address
to clients for passive-mode data connections — if it doesn't match the
LB IP the scanner reaches HA on, every transfer hangs at LIST/RETR.
10.2.40.82 -> 10.2.99.82, matching the Service annotation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Five Services move from default-pool (10.2.40.64-95, L2-announced) to
bgp-pool (10.2.99.0/24, BGP-routed off-link). Same pattern as
ha-vscode / traefik / mosquitto-iot: `lb-pool: bgp` label + IP repinned.
IP mapping (last octet preserved where possible — no client of any of
these hardcodes the IP, but consistent numbering helps when reading
DNS lookups vs. live state):
kube-dns-lb 10.2.40.83 + fd40::83 → 10.2.99.83 + fd99::83
alloy-syslog 10.2.40.84 → 10.2.99.84
syncthing-sync 10.2.40.85 → 10.2.99.85
ftp-scanner 10.2.40.82 → 10.2.99.82
rgw-migration-lb 10.2.40.81 + fd40::81 → 10.2.99.91 + fd99::91
(.81 is mosquitto in the
new pool; bump to .91)
external-dns auto-republishes the matching hostnames
(syslog.k.renner.uno, syncthing.paperless.renner.uno,
ftp.paperless.renner.uno) on its next 60s reconcile. kube-dns-lb has
no hostname annotation — clients reference it directly by IP via
k8s_gateway upstream config.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Same pattern as ha-vscode and traefik: `lb-pool: bgp` label, off-link
IP from 10.2.99.0/24. Devices reach the broker via UDM BGP routes
instead of v30iot ARP. mqtt.renner.uno A record follows automatically
via external-dns-unifi.
Shelly fleet gets reconfigured via shelly-manager's /api/v1/mqtt/push
to use the hostname `mqtt.renner.uno:1883` instead of the bare IP, so
future IP changes are a DNS-only operation — no re-push sweep needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Traefik's LoadBalancer moves from the default-pool (10.2.40.86 / fd40::86,
L2-announced) to the BGP-routed off-link pool. Every hostname behind
traefik — every *.home.renner.uno, vscode.home.renner.uno, the cluster
status page, paperless, immich, etc. — now reaches the ingress via the
UDM's BGP-learned routes instead of L2 ARP on the mgmt LAN.
This is the lever that takes vscode "fully working via the routed
network" — the TLS + DNS access path goes through the same BGP route
as the direct bgp-pool LB IPs. Same migration covers every other
traefik-fronted service in one step.
Companion updates: explicit `external-dns.alpha.kubernetes.io/target`
annotations on the pve and test-browser IngressRoutes (Traefik doesn't
populate IngressRoute.status.loadBalancer, so the target is hardcoded
and has to track the LB IP); the bootstrap.sh / e2e-test.sh smoke-test
IPs; a stale code comment in cilium-l2.yaml.
The label/annotation pattern matches the ha-vscode migration: chart's
`service.labels` sets `lb-pool: bgp` (single Service handles both TCP
and UDP for HTTP/3, so one label is enough), and the
`io.cilium/lb-ipam-ips` annotation pins the IPs deterministically.
external-dns publishes the new IPs to Cloudflare + UniFi DNS providers
on its next 60s reconcile cycle after the Service IP flips.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`talosctl patch mc --patch @cilium-cni.yaml` was failing with
`inline manifest name "cilium" is duplicate` because Talos strategic
merge appends to `cluster.inlineManifests` instead of replacing named
entries (per the upstream docs at /configure-your-talos-cluster/
system-configuration/patching).
Documented fix: a 2-doc patch — first doc deletes the existing entry
via `$patch: delete`, second doc adds the new one. At bootstrap via
`talosctl gen config --config-patch @cilium-cni.yaml` the delete is a
no-op against the empty initial config; the add inserts as before.
Same file therefore serves both paths.
Applied via `talosctl patch mc -n <each-node> --patch @cilium-cni.yaml`
to all 4 nodes. No reboot. BGP sessions stayed Established throughout;
live ConfigMap retained enable-bgp-control-plane + ipv4/ipv6-native-
routing-cidr. Source-of-truth (this file) and Talos stored state are
now in sync.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ipv4-native-routing-cidr=10.0.0.0/8 and ipv6-native-routing-cidr=fd00::/8.
Pod traffic to LAN destinations (any 10.x.x.x or ULA) now egresses with
the pod's real source IP — no SNAT. The UDM has BGP-learned routes
back to the per-node pod /24s (verified after the phase 1 + ha-vscode
migration smoke test), so return packets follow the same path home.
Internet egress (outside these CIDRs) keeps masquerading to the node IP
because pod IPs aren't globally routable. IPv6: pod CIDR is ULA only,
so pod→GUA still SNATs to the node's GUA — that's correct, ULA isn't
internet-reachable anyway.
This is the change that actually addresses the v30iot SNAT/conntrack
flake — the original motivator for the entire BGP rollout. The
persistent-TCP integrations (ESPHome, MQTT) were surviving via long-
lived conntrack; the polling/UDP ones (Siku, Philips CoAP, AirGradient
HTTP, Shelly HTTP) should now go from "intermittent" to "works".
Requires a cilium-agent DaemonSet rollout to pick up the new keys.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
k.rennerandClaude Opus 4.7
security-bot