fix(cilium): stop the endpoint policy map filling — trim ICMP set, limit identity labels, size + alert #495
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: konairius/homelab#495
Reference in New Issue
Block a user
Closes #451. Closes #454.
What is wrong
Both issues are one fault. A pod's dedicated
cilium_policy_v3_<id>BPF map runs out of space, endpoint regeneration fails withno space left on device, the CNI ADD call never returns, and the pod hangs in sandbox creation. For paperless (replicas: 1,strategy: Recreate) that was a 41-minute outage on 2026-07-30.It is still happening. Verified against live state today:
cilium_bpf_map_pressure{map_name="cilium_policy_v3_*"}max_over_time(...[30d])authentik/ak-outpost-ldap-outpostandhome-assistant/esphomebpf-policy-map-maxincilium-configPressure measured 0.854 at the start of this investigation and 0.867 forty minutes later. It is trending up, not stable.
Changes
bpf.policyMapMax: 65536— 4x the chart default, matching the cluster's existingbpf-lb-map-max. Takes steady-state pressure from ~87% to ~22%. Key verified againsthelm show values cilium --version 1.20.0; flag confirmed live asbpf-policy-map-maxin thecilium-configConfigMap.">=1.19.0"→">=1.20.0 <1.21.0"— the unbounded range is what let Flux take 1.19.6 → 1.20.0 unattended on its 30m interval, 45 minutes before the first outage. Patches still flow; a minor now needs a reviewed diff.PrometheusRule/cilium— warning above 80% for 15m, critical at 100%. The metric was already scraped (landed via #452) but nothing alerted on it, so the first signal was a downstreamKubeDeploymentReplicasMismatch.Verification done
kubectl kustomizebuilds clean for bothinfrastructure/ciliumandapps/monitoring-rules.kubectl apply --dry-run=serveraccepts the PrometheusRule.Read this before merging
Merging rolls the CNI.
updateStrategy.rollingUpdate.maxUnavailable: 2means two of three agents restart at once, and the map is only resized at agent start. Worth doing at a chosen moment rather than whenever Flux next reconciles.Expect
CiliumBPFMapPressureHighto fire in the window between the rule loading and the agents restarting — the map really is 87% full until they do.Deliberately not done
Why the busiest endpoint needs ~14k entries is unexplained. Only 178 CiliumIdentities exist and no CiliumNetworkPolicy declares more than 9 ports, so the count is not obviously a function of policy shape. Settling it needs
cilium-dbg bpf policy get --allon an agent. This PR buys headroom and visibility; it does not claim to have found why demand is that high.🤖 Automated review — model
CLI default(complexity: complex), commit8f45e2901b9179ce90bf66a42b96984805787095.Summary
This PR sizes up the Cilium per-endpoint policy BPF map (
bpf.policyMapMax: 65536), pins the chart to a patch-only range, and adds aPrometheusRulethat alerts oncilium_bpf_map_pressurebefore the map fills. The change is well-investigated (live pressure numbers, Loki evidence, a real incident) and the new alert/version-pin follow existing repo conventions closely. However, the fix only updates the Flux-managedHelmRelease; it misses a second, actively-used source of the same setting, so the bug it closes can be silently reintroduced on the bootstrap path.🔴 Blocking
kubernetes/talos/patches/cilium-cni.yaml:324still hardcodesbpf-policy-map-max: "16384"— the old, insufficient value — inside the Cilium inline-manifest Talos config patch. This file is not incidental:kubernetes/bootstrap.sh:86requires it to exist andkubernetes/bootstrap.sh:123-128passes it as--config-patchtotalosctl apply-configfor every node (k8s-1,k8s-2,k8s-3) during a full bootstrap/cold rebuild. Its own header comment also says it's reused for livetalosctl patch mc --patch @<file>updates. That means:RUNBOOK-cold-rebuild.md/FLUX_SOURCE=githubbreak-glass path inbootstrap.shexists for) installs Cilium with the pre-fix 16384-entry policy map, reproducing the "no space left on device" / stuck-sandbox-creation failure this PR closes, until Flux'sciliumHelmReleasereconciles and the agents restart (up to the 30mspec.intervalinhelmrelease.yaml, or whenever it's next triggered).bpf-lb-map-max: "65536"atcilium-cni.yaml:330already matches the current live/HelmRelease value, showing this repo's established practice is to keep this rendered bootstrap manifest in sync withbpf.*values inhelmrelease.yaml. This PR breaks that sync for the new key.bpf-policy-map-maxto"65536"inkubernetes/talos/patches/cilium-cni.yaml(and regenerate/updatekubernetes/talos/cilium.yamltoo, see Suggestions) so bootstrap and steady state agree.🟡 Suggestions
kubernetes/talos/cilium.yaml:306has the same stalebpf-policy-map-max: "16384". Unlike the patch file above, nothing inbootstrap.shorRUNBOOK-cold-rebuild.mdreferences this path directly (onlytalos/patches/cilium-cni.yamlis used) — it looks like an orphaned rendered-manifest snapshot. Worth either deleting it if it's dead, or updating it for consistency if it's kept as a reference/diff baseline — right now it's silently out of sync with both the fix and the actively-used patch file.helmrelease.yaml'sbpf:block (or a repo-level note) pointing atkubernetes/talos/patches/cilium-cni.yamlas a value that must be updated in lockstep, so this class of drift doesn't recur the next time abpf-*-map-maxvalue changes.🟢 Nits
None.
Tests
No automated tests apply to Kustomize/HelmRelease/PrometheusRule YAML in this repo; the PR body documents manual verification (
kubectl kustomizebuild,kubectl apply --dry-run=serverfor thePrometheusRule, and both alert expressions run against live Prometheus) which is the right verification for this kind of change and I have no reason to doubt it. I could not independently re-runkubectl kustomize/dry-runin this review sandbox (no cluster/tool access), but manual read-through ofprometheusrule-cilium.yamlagainst sibling rule files (prometheusrule-loki.yaml,prometheusrule-netbird-availability.yaml) shows matching structure:release: kube-prometheus-stacklabel,<name>.rulesgroup naming, andfor: 0mused elsewhere for immediate-critical alerts (LokiPanics), so nothing here looks malformed. What's untested (and probably untestable outside a real cluster) is the bootstrap-path regression above — that would only surface on an actual cold rebuild ortalosctl apply-configre-run.Recommendation
Request changes — 1 blocking issue.
The blocking finding is correct and is now fixed in
621e4e25.I verified it independently rather than taking it on faith:
kubernetes/talos/patches/cilium-cni.yaml:324did holdbpf-policy-map-max: "16384", andkubernetes/bootstrap.sh:126passes that file totalosctl apply-configfor every node. A cold rebuild would have come up with exactly the map size this PR exists to raise.bpf-lb-map-maxis"65536"in both talos files and matches the live agent config.All three sites are now
65536, and each names the other two so the lockstep is written where someone changing it will actually hit it.On the suggestion about
kubernetes/talos/cilium.yaml: I updated it, and I can confirm it is orphaned —grep -rn "cilium.yaml" . --exclude-dir=.gitreturns no reference to it anywhere in the repo (onlypatches/cilium-cni.yaml" is wired into bootstrap). Its owncni-exclusive` comment at :397 does claim a lockstep contract with the patch file and the HelmRelease, which is why I synced it rather than deleted it. Whether an unreferenced rendered snapshot should be kept at all is a real question, but deleting it is a scope decision for the owner, not something to fold into this fix.One thing the review did not raise, noted for the record and deliberately not changed:
enable-endpoint-lockdown-on-policy-overflowis"false"in both talos manifests. That is why an overflow presents as a stuck sandbox rather than a fail-closed endpoint. Flipping it trades one failure mode for another and belongs in its own discussion.🤖 Automated review — model
CLI default(complexity: complex), commit621e4e25bffa645ef9623e757db2a5fba9ba0d8b.Summary
This PR raises the Cilium per-endpoint policy BPF map from 16384 to 65536 entries in all three places it needs to move together (Helm values, the live
cilium-configConfigMap render, and the Talos bootstrap inline manifest), pins the Cilium chart to a patch-only range to stop unattended minor bumps, and adds aPrometheusRulethat alerts oncilium_bpf_map_pressure. All three copies ofbpf-policy-map-maxare updated consistently (verified via grep acrosskubernetes/), the new rule follows this repo's existingmonitoring-rulesconventions exactly (label selector, alert/description style, alphabetical ordering inkustomization.yaml), and the reasoning in the comments is grounded and specific. I found no blocking issues.🟡 Suggestions
kubernetes/apps/monitoring-rules/prometheusrule-cilium.yaml:38and:49—cilium_bpf_map_pressureis not policy-map-specific; it's emitted per BPF map type (policy, CT, NAT, LB, etc., distinguished by themap_namelabel used in the annotations). Both alerts will legitimately fire for any Cilium map crossing the threshold, not justcilium_policy_v3_*. That's arguably the right generic behavior (and the annotation text is already written generically — "Raise the corresponding size... (values.bpf)"), but it's worth a one-line comment noting the alert is intentionally map-agnostic, since the PR title/body frame this narrowly around the policy map and a future reader may assume it's policy-map-only.kubernetes/talos/cilium.yaml— this file doesn't appear to be referenced or applied anywhere in the repo (kubernetes/bootstrap.shonly appliespatches/cilium-cni.yaml; nothing sourcestalos/cilium.yaml). It looks like a static rendered snapshot kept as documentation/diff-reference, and this PR correctly keeps it in sync with the same lockstep comment. Since it's a third copy of the same value with no automated enforcement tying it to the other two, consider whether it's worth keeping at all, or note in the file header what regenerates/reviews it — otherwise it's a fourth place a future change can silently drift from (the PR body's own "lockstep" comments only mention two other files, not this one, so the cross-referencing is one-directional).🟢 Nits
kubernetes/infrastructure/cilium/helmrelease.yaml:63—policyMapMax: 65536is unquoted while the rendered ConfigMap value is a quoted string ("65536"). This works because Helm/sprig coerces the value downstream, matching the pattern of the chart's likely upstream default, but if the chart'svalues.yamldocuments this key as a string, an explicit"65536"would remove any ambiguity for the next reader.Tests
No automated tests apply here — this is GitOps config (Helm values, a Talos machine-config patch, and a
PrometheusRule). The PR body documents concrete verification instead:kubectl kustomizebuilds for both changed Kustomizations,kubectl apply --dry-run=serveraccepts the newPrometheusRule, and both alert expressions were run against live Prometheus. That's the right substitute for tests in this repo and is more thorough than most PRs of this kind — I did not re-run these dry-runs myself sincekubectl/kustomizearen't available in this review environment, but I independently confirmed all threebpf-policy-map-max/policyMapMaxoccurrences were updated to the same value and that the new rule file matches the structure of every other file inmonitoring-rules/.Recommendation
Looks reasonable — no blocking issues; comments above are optional.
Second review is clean — no blocking issues. Two of the three optional points taken in
9371f188, one declined.Taken. The alert's map scope is now stated in the rule file: it is map-agnostic on purpose, since conntrack/NAT/LB exhaustion degrades the datapath too, and the policy map is just the one that has bitten us. Also noted there that Cilium only exports the metric above
policyMapPressureMetricsThreshold(chart default 0.1), so quiet maps produce no series rather than near-zero noise.Taken. The cross-references were one-directional, as the review says —
helmrelease.yamlandpatches/cilium-cni.yamleach named one other site whiletalos/cilium.yamlnamed both. All three now name the other two.Declined — the nit is wrong. Quoting
policyMapMaxas"65536"would not be the safer choice. The chart declares it as an integer in its values schema:(
helm show values cilium --repo https://helm.cilium.io --version 1.20.0.) An unquoted integer matches the schema; a quoted string is what would risk failing validation. The quoted"16384"/"65536"in the Talos files are a different thing — those are Cilium ConfigMap keys, where every value is a string by definition.On whether
kubernetes/talos/cilium.yamlshould exist at all: agreed it is a third unenforced copy, and I have confirmed nothing in the repo references it. Still leaving that decision to the owner rather than deleting a file as a side effect of a map-sizing fix.🤖 Automated review — model
CLI default(complexity: complex), commit9371f188b67f76613fabf29a20c0db9360224668.Summary
This PR resizes the per-endpoint Cilium policy BPF map from 16384 to 65536 entries in all three places it's duplicated (Helm values, the Talos bootstrap inline manifest, and the reference copy of that manifest), pins the Cilium chart to a
1.20.x-only range instead of an unbounded>=1.19.0, and adds aPrometheusRulealerting oncilium_bpf_map_pressure. I cross-checked all three map-size copies, the alert against siblingPrometheusRulefiles' conventions, and the alertmanager routing tree — everything is internally consistent and I found no blocking issues.🟡 Suggestions
kubernetes/apps/monitoring-rules/prometheusrule-cilium.yaml:51,62— both alertdescriptionannotations reference{{ $labels.node }}. I could not confirm from the static checkout thatcilium_bpf_map_pressureactually carries anodelabel (the CiliumServiceMonitor/relabeling config is applied live via the Helm release, not present in the pinned bootstrap manifests, so there's nothing in-repo to grep for it). The PR body says both expressions were run against live Prometheus and returned the expected series, which is good evidence, but it's worth a last glance at the actual label set in Prometheus (cilium_bpf_map_pressuretarget labels) right before merge, since a missingnodelabel would just render as an empty string in the alert text rather than fail loudly.kubernetes/talos/cilium.yamlisn't referenced by any script or tooling in this repo (confirmed via grep) — it's a hand-maintained third copy purely for readability next to the harder-to-diff inline manifest inpatches/cilium-cni.yaml. The PR already discloses this ("Nothing enforces any of this; grep bpf-policy-map-max before changing it."), so this isn't new debt, just flagging that the honesty of that comment is accurate — a future editor really can update two of the three and nothing will catch it.Tests
No automated tests exist for this class of change in the repo (Kustomize/Helm/Talos manifests), consistent with how similar
PrometheusRuleandhelmrelease.yamlchanges have been made before. The PR body documents manual verification instead:kubectl kustomizebuild,kubectl apply --dry-run=serverfor thePrometheusRule, and both alert expressions run against live Prometheus. That's the right verification method for this kind of change and matches repo convention — I have no way to re-run the live checks myself from this checkout, but the static structure (YAML validity, alphabetical resource ordering inkustomization.yaml, label conventions matching sibling rule files, all threebpf-policy-map-maxcopies actually landing on 65536) all check out.Recommendation
Looks reasonable — no blocking issues; comments above are optional.
The map is not legitimately full — this PR as written would mask a bug
I said in the PR body that the ~14k entry count was unexplained. I have now explained it, and it changes what this PR should be.
Right now on k8s-1, 8 endpoint maps sit at exactly 16384 — full. Not near-full.
cilium-dbg map listshows a bimodal distribution: a handful of small maps (10–229 entries) and then a cluster at ~8,500 and a cluster pinned at the 16384 ceiling. The split tracks policy direction:Enabled/Disabledendpoints land at ~8,500,Enabled/Enabledat 16384. So one direction costs ~8,500 entries.The affected endpoints are unrelated to each other — cnpg databases, matter-server, spegel, ftp-scanner, openwakeword, zone-configurator, velero, mariadb. This is systemic, not one bad workload.
What is in the map
Dumping velero's endpoint 2328 (8,499 entries, ingress only):
woodpecker-ci-step217 identities × 39 ≈ 8,460 of 8,499. The map is ~99% ICMP.
Two compounding causes
1. The ICMP rule in every
default-deny-ingress.kubernetes/infrastructure/network-policies/*-ns.yaml— 19 namespaces — each carry:The rationale in
home-assistant-ns.yaml:144-152is sound — Cilium blocks ICMP once any L4 policy exists, CT-RELATED was not passing ICMPv6 DestinationUnreachable, andICMPFieldhas no wildcard so types must be enumerated. But it notes it was "widened from the error class to the full standard type set", and that widening is the cost: the set includes echo, router discovery, MLD, inverse ND and node-information types (139–153) alongside the error types that were actually needed. Every type costs one entry per identity, per endpoint, per direction, in 19 namespaces.2. Woodpecker mints an identity per build step.
cilium-confighas nolabelskey, so Cilium derives identity from all pod labels. Step pods carrywoodpecker-ci.org/task-uuid=6361andstep=clone, so every step of every build is a distinct security identity. There are 96woodpecker-ci-stepidentities right now with zero pods in the namespace — all created within the last hour. The operator's GC is working and actively marking them; it simply cannot keep up with the mint rate.That is why the overflow is intermittent and why it appeared when it did: the entry count is a function of CI load.
Why 65536 is the wrong headline fix
It works arithmetically (17k → 26% of 65536) but leaves the growth term untouched. A busier CI window still overflows, just later. Fixing either cause is worth more:
Cilium documents the first as the standard remedy for exactly this shape — labels carrying uuids or hashes causing identity proliferation — via the
labelsHelm value with!prefixexclusions.What I propose for this PR
The version pin and the PrometheusRule are unaffected and should land either way — the alert is what surfaced this at all. The
policyMapMaxraise I would keep only as headroom, not as the fix, and say so in the commit.The two real fixes touch a security policy replicated across 19 namespaces and cluster-wide identity computation. Both are owner calls, not something to fold into this PR unannounced. Holding for direction.
fix(cilium): size the endpoint policy map and alert before it fillsto fix(cilium): stop the endpoint policy map filling — trim ICMP set, limit identity labels, size + alertBoth root-cause fixes are in
25c8c806. The PR now removes the demand rather than raising the ceiling;policyMapMaxstays as headroom only.1. ICMP set trimmed 39 → 16, across 19 files
Kept: the error types that motivated the rule (v4 3/11/12, v6 1/2/3/4 — PMTUD is the documented need), echo (v4 0/8, v6 128/129), and the IPv6 ND/RA/MLDv2 types observed actually in flight via
hubble observe— 133, 135, 143.That last point changed the shape of the trim. My first instinct was error-types-only, which would have dropped 133, 135 and 143 — all three of which are live on the wire. Sampling before cutting was worth it.
Dropped, with reasons: v4 4 (Source Quench, deprecated by RFC 6633), v4 5 / v6 137 (Redirect — router-only), v4 9/10, 13–18, 30 (obsolete or infoleak-prone), v6 130–132 (MLDv1, superseded by the 143 we see), v6 138, 141/142, 148/149, 151–153 (never deployed).
Corroboration I did not expect to find:
openbao-ns.yamlalready runs an error-only set of just 7 types in production. So 16 is on the conservative side of practice that already exists in this repo.2. Identity label filter
labels: '!woodpecker-ci\.org/.* !step$'on the HelmRelease and both Talos manifests. Syntax verified against the v1.20 docs — patterns are start-anchored regexes,!excludes while keeping the default set.Checked before cutting: no CiliumNetworkPolicy selects on either label, and no pod outside woodpecker carries a bare
steplabel. This collapses 78 current identities to one.Projected effect
That fits inside the old 16384 with room to spare, which is the point: the growth term tied to CI load is gone.
Verification
kubectl kustomizebuilds clean fornetwork-policies,ciliumandmonitoring-rules.kubectl apply --dry-run=serveraccepts all 25 rendered policies and the PrometheusRule.labels,policyMapMaxand the version range.Two things to know before merging
Per Cilium's docs, a
labelschange does not recompute existing identities — new ones use the filter and old ones age out via normal GC. So the map drains over roughly an hour after the agents restart, not instantly. Don't read the first post-merge reading as a failure.There is a further lever I did not pull: the ICMP rule still matches
namespace: Exists, so cost stays linear in identity count. Allowing ICMP from a wildcard source instead would collapse it to one entry per type regardless of identity count — but that widens the source beyond cluster pods, which is a security tradeoff and a separate decision.🤖 Automated review — model
CLI default(complexity: complex), commit25c8c806b4d0397713c6c4a19dcdb49e37834c00.Summary
This PR addresses the recurring
cilium_policy_v3_<id>BPF policy-map exhaustion (#451, #454) three ways: quadruplesbpf.policyMapMaxto 65536 (kept in sync across the three copies —helmrelease.yaml,kubernetes/talos/cilium.yaml,kubernetes/talos/patches/cilium-cni.yaml), pins the Cilium chart to a patch-only range, and adds aPrometheusRuleso map pressure is alertable. It also trims the ICMP allow-list in 17CiliumNetworkPolicyfiles and adds an identity-label exclude filter for Woodpecker CI pod labels, both aimed at reducing the number of policy-map entries an endpoint needs. I checked all threebpf-policy-map-max/policyMapMaxcopies and all threelabelscopies — they're byte-identical, and I diffed the post-trim ICMP type sets across all 17 network-policy files — they're uniform (IPv4: 0,3,8,11,12; IPv6: 1,2,3,4,128,129,133,134,135,136,143), keeping the PMTUD-relevant and neighbor-discovery types and dropping obsolete/rare ones (source quench, address-mask, MLD, inverse-ND, etc.). No blocking issues found.🟡 Suggestions
kubernetes/infrastructure/cilium/helmrelease.yaml:45(and the two mirrored copies) — the identity-label filterlabels: '!woodpecker-ci\.org/.* !step$'is the one piece of this PR whose effect isn't verified against live state the way the map-size and alert-expression changes are (the PR body's "Verification done" and "Deliberately not done" sections don't mention it, and it's a plausible partial explanation for "why the busiest endpoint needs ~14k entries" that the author says is still unexplained). A wrong regex here fails silently — no dry-run or kustomize build catches it, it just doesn't reduce identity churn. Worth confirming post-rollout, e.g.cilium-dbg identity listfor a woodpecker-ci step pod before/after an agent restart, to check CI steps actually collapse onto one identity.kubernetes/apps/monitoring-rules/prometheusrule-cilium.yaml:32-44— both alerts key their summary/description on{{ $labels.node }}, whereas the other rules in this same directory (prometheusrule-loki.yaml,prometheusrule-netbird-availability.yaml) use$labels.instance/$labels.job. The PR body states the expressions were run against live Prometheus and returned results for all three nodes, which is good evidencecilium_bpf_map_pressuredoes carry anodelabel in this cluster's setup (likely via the chart's ServiceMonitor relabeling__meta_kubernetes_pod_node_name), so this isn't a correctness concern — just flagging the label-naming inconsistency with the rest of the directory for future reference.🟢 Nits
home-assistant-ns.yamlkeeps per-type ICMP comments (e.g.# Destination Unreachable — PMTUD) after the trim; the other 16 files'icmps.fieldslists are bare numbers both before and after this change, so there's no rationale recorded next to them for why e.g. type 4 or type 30 IPv4 were dropped. Not introduced by this PR, but since this PR is the one making the trim deliberate, it'd be a good place to add a one-line pointer (e.g. "see home-assistant-ns.yaml for the full type table") so a future editor doesn't have to reverse-engineer the surviving set.Tests
This is a GitOps config repo with no unit-test suite; the appropriate verification is what the PR describes:
kubectl kustomizebuild and--dry-run=serverapply for the newPrometheusRule, both of which the author ran, and which.woodpecker/ci-manifests-test.ymlalso exercises in CI. The two alert expressions were run against live Prometheus per the PR body. I didn't find anything in the diff that this level of testing would fail to catch.Recommendation
Looks reasonable — no blocking issues; comments above are optional.