claude-botandClaude Opus 5.5
c7b504a3f5
- The k8s-gateway values were read with `yq -y`, a flag only the Python yq has. The vmtest image installs Alpine's `yq`, which is yq-go (mikefarah), so the talos phase would fail there. Without the flag both print a document helm accepts, as the Cilium values line already relies on. - A cni-confdir pod left from an earlier run (Failed or Succeeded) made `kubectl apply` a no-op and the wait time out. The step deletes it first. - The Cilium HelmRelease comment named itself the only place Cilium is configured; the site patch's cilium-site ConfigMap is the other. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
195 lines
8.4 KiB
YAML
195 lines
8.4 KiB
YAML
apiVersion: source.toolkit.fluxcd.io/v1
|
|
kind: HelmRepository
|
|
metadata:
|
|
name: cilium
|
|
namespace: kube-system
|
|
spec:
|
|
interval: 24h
|
|
url: https://helm.cilium.io/
|
|
---
|
|
apiVersion: helm.toolkit.fluxcd.io/v2
|
|
kind: HelmRelease
|
|
metadata:
|
|
name: cilium
|
|
namespace: kube-system
|
|
spec:
|
|
interval: 30m
|
|
chart:
|
|
spec:
|
|
chart: cilium
|
|
# Patch-only range: Flux reconciles every 30m and would otherwise take a
|
|
# CNI minor or major unattended. Widening this is a deliberate, reviewed
|
|
# act — a minor bump changes datapath behaviour and rolls 2 of 3 agents
|
|
# at once.
|
|
version: ">=1.20.0 <1.21.0"
|
|
sourceRef:
|
|
kind: HelmRepository
|
|
name: cilium
|
|
namespace: kube-system
|
|
install:
|
|
remediation:
|
|
retries: 3
|
|
upgrade:
|
|
remediation:
|
|
retries: 3
|
|
values:
|
|
# Cilium is configured here and, for what differs per site, in the
|
|
# `cilium-site` ConfigMap of the site patch (valuesFrom, merged under these
|
|
# values). Talos provides only `cni: none` and `proxy: disabled`; a cold
|
|
# rebuild installs Cilium from both with helm before Flux takes over.
|
|
#
|
|
# Rolls the agent DaemonSet when cilium-config changes. Cilium reads
|
|
# datapath options once at agent start and the chart puts no config
|
|
# checksum on the pod template, so without this a changed ConfigMap leaves
|
|
# every agent on the previous datapath indefinitely (#592).
|
|
#
|
|
# The trade is that any cilium-config change now rolls the CNI node by
|
|
# node — which is what a datapath change required anyway, only now it is
|
|
# not forgotten.
|
|
annotations:
|
|
configmap.reloader.stakater.com/reload: cilium-config
|
|
# Which pod labels feed security-identity computation. Patterns are
|
|
# start-anchored regexes; a leading "!" excludes while keeping the default
|
|
# set. Woodpecker stamps every step pod with woodpecker-ci.org/task-uuid
|
|
# and step=<name>, so without this each build step is its own identity —
|
|
# and every identity costs one entry per allowed (direction, proto, port)
|
|
# in every endpoint's policy map, cluster-wide. Nothing selects on these
|
|
# labels in policy, so they buy identity churn and nothing else.
|
|
labels: '!woodpecker-ci\.org/.* !step$'
|
|
ipv6:
|
|
enabled: true
|
|
ipam:
|
|
mode: kubernetes
|
|
kubeProxyReplacement: true
|
|
socketLB:
|
|
enabled: true
|
|
prometheus:
|
|
# Without agent metrics, BPF map utilisation is invisible: a full policy
|
|
# map only shows up as "no space left on device" in the log while pod
|
|
# sandbox creation fails (#452). cilium_bpf_map_pressure makes it
|
|
# alertable beforehand.
|
|
enabled: true
|
|
# Renders the headless `cilium-agent` metrics Service. The chart gates
|
|
# that Service on serviceMonitor.enabled OR metricsService; with neither
|
|
# set, no Service existed and cilium_* had zero series (REN-147). This
|
|
# file owns the Service now; apps/monitoring-rules/cilium-servicemonitor.yaml
|
|
# no longer declares its own, and its ServiceMonitor selects this one.
|
|
metricsService: true
|
|
# Stays false. The chart aborts rendering, not skips, when
|
|
# monitoring.coreos.com/v1 is missing, and cilium is one of the
|
|
# infrastructure Kustomization's healthChecks: on an empty cluster the CNI
|
|
# would wait on monitoring, which waits on the CNI. The ServiceMonitor is
|
|
# declared in cilium-servicemonitor.yaml instead, as velero does.
|
|
serviceMonitor:
|
|
enabled: false
|
|
bpf:
|
|
# Per-endpoint policy map size: one entry per (identity, direction,
|
|
# protocol, port) allowed. When it fills, endpoint regeneration fails
|
|
# with "no space left on device" and the pod hangs in sandbox creation.
|
|
# Keep it comfortably above the busiest endpoint, watched by
|
|
# cilium_bpf_map_pressure. Sized at map creation, so it takes an agent
|
|
# restart.
|
|
#
|
|
policyMapMax: 65536
|
|
# VLAN ids the eBPF datapath is allowed to pass. Without this it drops
|
|
# VLAN-tagged frames, accounted as "VLAN traffic disallowed by VLAN
|
|
# filter" in cilium_drop_count_total.
|
|
#
|
|
# Only VLAN 30 (v30iot) needs listing. Cilium attaches to devices that
|
|
# carry an address, so it sees mgmt0's frames already untagged by the
|
|
# 8021q layer — nothing for a VLAN filter to match. ens19 is deliberately
|
|
# address-less, so its frames only meet the filter on the shared parent,
|
|
# still tagged. Adding 40 would change nothing and would touch the
|
|
# datapath carrying etcd and the API.
|
|
#
|
|
# The symptom when this is wrong is not a DHCP timeout: a tc-layer drop
|
|
# reaches an AF_PACKET sender as ENOBUFS, so the CNI dhcp plugin reports
|
|
# "no buffer space available" (#592).
|
|
#
|
|
# Same three-copy problem as policyMapMax above. Takes an agent restart.
|
|
vlanBypass: [30]
|
|
# Let pods reach the cluster's own LoadBalancer external IPs; Cilium
|
|
# redirects the socket directly to the backend pod instead of sending
|
|
# traffic out to the VIP and back (which fails with L2 announcements).
|
|
lbExternalClusterIP: true
|
|
cni:
|
|
# Multus is the only writer of /etc/cni/net.d (#832): Cilium's conflist
|
|
# goes to /etc/cni/net.d/multus/, which the container runtime does not
|
|
# read, and Multus delegates to it via clusterNetwork. Without this a
|
|
# pod created after a reboot, before Multus rewrote 00-multus.conf, came
|
|
# up with Cilium alone. See kubernetes/infrastructure/multus.
|
|
confPath: /etc/cni/net.d/multus
|
|
# The default `exclusive: true` renames other configs in the conf dir to
|
|
# *.cilium_bak on every reconcile, which kept nuking multus.
|
|
exclusive: false
|
|
# One agent at a time: the chart default of 2 restarts two of three
|
|
# nodes' datapath at once.
|
|
updateStrategy:
|
|
type: RollingUpdate
|
|
rollingUpdate:
|
|
maxUnavailable: 1
|
|
# Talos-specific settings
|
|
securityContext:
|
|
capabilities:
|
|
ciliumAgent:
|
|
- CHOWN
|
|
- KILL
|
|
- NET_ADMIN
|
|
- NET_RAW
|
|
- IPC_LOCK
|
|
- SYS_ADMIN
|
|
- SYS_RESOURCE
|
|
- DAC_OVERRIDE
|
|
- FOWNER
|
|
- SETGID
|
|
- SETUID
|
|
cleanCiliumState:
|
|
- NET_ADMIN
|
|
- SYS_ADMIN
|
|
- SYS_RESOURCE
|
|
cgroup:
|
|
autoMount:
|
|
enabled: false
|
|
hostRoot: /sys/fs/cgroup
|
|
k8sServiceHost: localhost
|
|
k8sServicePort: 7445
|
|
# Hubble itself is on in every agent (chart default); the relay aggregates
|
|
# the three nodes' flow buffers behind one gRPC endpoint so `hubble observe`
|
|
# works without exec into kube-system (REN-487). Renders no change to
|
|
# cilium-config or the agent DaemonSet, so the reloader annotation above
|
|
# does not roll the CNI. The relay has no client auth; who may reach it is
|
|
# network-policies/kube-system-hubble-relay.yaml.
|
|
hubble:
|
|
relay:
|
|
enabled: true
|
|
l2announcements:
|
|
enabled: true
|
|
externalIPs:
|
|
enabled: true
|
|
# BGP to the UDM. The peering itself has always been in git as CRDs in
|
|
# post-crds/cilium-bgp.yaml — but the switch that starts the control plane
|
|
# was not, and neither were the CIDRs below. They were applied by hand and
|
|
# held by the `kubectl-patch` field manager on cilium-config. Server-side
|
|
# apply left them alone, so nothing broke and nothing noticed; a cold
|
|
# rebuild would have come up with BGP off while the CRDs said otherwise.
|
|
#
|
|
# This is not a free change of ownership. Enabling bgpControlPlane through
|
|
# the chart also emits companion keys the hand-patch never set —
|
|
# bgp-secrets-namespace, bgp-router-id-allocation-mode and -ip-pool,
|
|
# enable-bgp-control-plane-status-report, enable-bgp-legacy-origin-attribute
|
|
# — so Cilium has been running on its own defaults for those. Router-id
|
|
# allocation is the one to watch: if the chart's default disagrees with
|
|
# Cilium's, the sessions re-establish.
|
|
bgpControlPlane:
|
|
enabled: true
|
|
# What the site's network can carry decides how pod traffic crosses between
|
|
# nodes and which destinations see real pod IPs: routingMode,
|
|
# autoDirectNodeRoutes, the direct-routing device and the native-routing
|
|
# (masquerade-exempt) CIDRs come from the cilium-site ConfigMap that each
|
|
# Talos site patch carries (talos/patches/site-*.yaml). They must not be set
|
|
# in values above, which would override it.
|
|
valuesFrom:
|
|
- kind: ConfigMap
|
|
name: cilium-site
|
|
valuesKey: values.yaml
|