Compare commits
3
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
6ffbc468d7 | ||
|
|
8c7fd25e03 | ||
|
|
7f11867407 |
@@ -33,22 +33,14 @@ spec:
|
||||
remediation:
|
||||
retries: 3
|
||||
values:
|
||||
# Roll the agent DaemonSet when cilium-config changes. Cilium reads
|
||||
# datapath options once, at agent start, and the chart puts no config
|
||||
# checksum on the pod template — so a changed ConfigMap leaves every agent
|
||||
# running the previous datapath until something restarts it. Nothing did.
|
||||
# Rolls the agent DaemonSet when cilium-config changes. Cilium reads
|
||||
# datapath options once at agent start and the chart puts no config
|
||||
# checksum on the pod template, so without this a changed ConfigMap leaves
|
||||
# every agent on the previous datapath indefinitely (#592).
|
||||
#
|
||||
# This is not hypothetical. vlan-bpf-bypass reached cilium-config and every
|
||||
# v30iot macvlan workload stayed broken until the agents were deleted by
|
||||
# hand, one node at a time (issue #592). policyMapMax has the same
|
||||
# exposure, and its comment below says "takes effect on agent restart" —
|
||||
# documentation standing in for automation.
|
||||
#
|
||||
# The trade: any cilium-config change now rolls the CNI cluster-wide,
|
||||
# node by node. That is disruptive-ish, but it is what a datapath change
|
||||
# already required — the difference is that it now happens on its own
|
||||
# instead of being forgotten. Reloader is already deployed for exactly
|
||||
# this pattern; named per-ConfigMap rather than blanket, as elsewhere.
|
||||
# The trade is that any cilium-config change now rolls the CNI node by
|
||||
# node — which is what a datapath change required anyway, only now it is
|
||||
# not forgotten.
|
||||
annotations:
|
||||
configmap.reloader.stakater.com/reload: cilium-config
|
||||
# Which pod labels feed security-identity computation. Patterns are
|
||||
@@ -69,57 +61,42 @@ spec:
|
||||
socketLB:
|
||||
enabled: true
|
||||
prometheus:
|
||||
# The agent exported no metrics at all, so BPF map utilisation was
|
||||
# invisible: when per-endpoint policy maps (cilium_policy_v3_*) run out
|
||||
# of space the agent only says "no space left on device" in its log and
|
||||
# pod sandbox creation fails (issue #452). cilium_bpf_map_pressure makes
|
||||
# that observable — and alertable — before it stalls a rollout.
|
||||
# Without agent metrics, BPF map utilisation is invisible: a full policy
|
||||
# map only shows up as "no space left on device" in the log while pod
|
||||
# sandbox creation fails (#452). cilium_bpf_map_pressure makes it
|
||||
# alertable beforehand.
|
||||
enabled: true
|
||||
serviceMonitor:
|
||||
enabled: true
|
||||
bpf:
|
||||
# Size of each endpoint's own cilium_policy_v3_<id> map, holding one
|
||||
# entry per (identity, direction, protocol, port) the endpoint is
|
||||
# allowed. When it fills, endpoint regeneration fails with "no space
|
||||
# left on device", the CNI ADD call never returns, and the pod is stuck
|
||||
# in sandbox creation — so this must stay comfortably above the busiest
|
||||
# endpoint's entry count, tracked by cilium_bpf_map_pressure.
|
||||
# Takes effect on agent restart; the map is sized at creation.
|
||||
# Bootstrap installs Cilium from the inline manifest in
|
||||
# kubernetes/talos/patches/cilium-cni.yaml, so bpf-policy-map-max there
|
||||
# must move with this value or a cold rebuild starts with the old size.
|
||||
# kubernetes/talos/cilium.yaml carries a third copy of the same key.
|
||||
# Nothing enforces any of this; grep bpf-policy-map-max before changing it.
|
||||
# Per-endpoint policy map size: one entry per (identity, direction,
|
||||
# protocol, port) allowed. When it fills, endpoint regeneration fails
|
||||
# with "no space left on device" and the pod hangs in sandbox creation.
|
||||
# Keep it comfortably above the busiest endpoint, watched by
|
||||
# cilium_bpf_map_pressure. Sized at map creation, so it takes an agent
|
||||
# restart.
|
||||
#
|
||||
# THREE COPIES of this key exist — here, talos/patches/cilium-cni.yaml
|
||||
# and talos/cilium.yaml. The latter two are what a cold rebuild installs
|
||||
# from. Nothing enforces agreement; grep bpf-policy-map-max before
|
||||
# changing it.
|
||||
policyMapMax: 65536
|
||||
# VLAN ids the eBPF datapath is allowed to pass. Without this it drops
|
||||
# VLAN-tagged frames, accounted as "VLAN traffic disallowed by VLAN
|
||||
# filter" in cilium_drop_count_total.
|
||||
#
|
||||
# 30 is v30iot, the macvlan parent for the shelly-manager, matter-server,
|
||||
# otbr and music-assistant attachments. On bare metal it is a VLAN
|
||||
# sub-interface of enp2s0f0np0; inside the former Proxmox guests it was
|
||||
# an untagged virtio NIC, which is why this was not needed before.
|
||||
# Only VLAN 30 (v30iot) needs listing. Cilium attaches to devices that
|
||||
# carry an address, so it sees mgmt0's frames already untagged by the
|
||||
# 8021q layer — nothing for a VLAN filter to match. ens19 is deliberately
|
||||
# address-less, so its frames only meet the filter on the shared parent,
|
||||
# still tagged. Adding 40 would change nothing and would touch the
|
||||
# datapath carrying etcd and the API.
|
||||
#
|
||||
# VLAN 40 (mgmt0) is deliberately NOT listed, and the asymmetry is the
|
||||
# interesting part. Cilium attaches to devices that carry an address —
|
||||
# `cilium-dbg status --verbose` lists enp87s0, enp2s0f0np0, enp2s0f1np1
|
||||
# and mgmt0, but NOT ens19. mgmt0 holds the node IP, so Cilium owns it as
|
||||
# a device and sees frames the 8021q layer has already untagged; nothing
|
||||
# for a VLAN filter to match. ens19 is deliberately address-less (see
|
||||
# talos/patches/k8s-1.yaml — host addresses on that L2 make node replies
|
||||
# bypass the UDM), so Cilium never attaches to it and VLAN 30 frames only
|
||||
# meet the filter on the shared parent, still tagged.
|
||||
# The symptom when this is wrong is not a DHCP timeout: a tc-layer drop
|
||||
# reaches an AF_PACKET sender as ENOBUFS, so the CNI dhcp plugin reports
|
||||
# "no buffer space available" (#592).
|
||||
#
|
||||
# So the fix belongs on 30 alone. Listing 40 would change nothing, and
|
||||
# would touch the datapath carrying etcd and the API for no reason.
|
||||
#
|
||||
# A tc-layer drop surfaces to an AF_PACKET sender as ENOBUFS, so the
|
||||
# symptom is the CNI dhcp plugin reporting "no buffer space available"
|
||||
# rather than a DHCP timeout. See issue #592.
|
||||
#
|
||||
# Same three-copy problem as policyMapMax above: vlan-bpf-bypass in
|
||||
# kubernetes/talos/cilium.yaml and kubernetes/talos/patches/cilium-cni.yaml
|
||||
# must move with this list. Takes effect on agent restart.
|
||||
# Same three-copy problem as policyMapMax above. Takes an agent restart.
|
||||
vlanBypass: [30]
|
||||
# Let pods reach the cluster's own LoadBalancer external IPs; Cilium
|
||||
# redirects the socket directly to the backend pod instead of sending
|
||||
|
||||
@@ -36,25 +36,16 @@ spec:
|
||||
values:
|
||||
container_runtime: containerd
|
||||
|
||||
# --- mTLS between agents and the LAPI: the durable fix for the recurring
|
||||
# KubeDaemonSetRolloutStuck auth loop (#340) ---
|
||||
# mTLS between agents and the LAPI. Do not switch back to password auth:
|
||||
# the LAPI runs without persistence, so its machine-registration DB is
|
||||
# wiped on every restart, and password-registered agents then get 401
|
||||
# "machine not found" and crashloop until re-registered by hand — which
|
||||
# wedges the whole `infrastructure` Kustomization behind a HelmRelease
|
||||
# that never goes Ready (#340).
|
||||
#
|
||||
# The LAPI runs WITHOUT persistence (see persistentVolume note below), so
|
||||
# its sqlite machine-registration DB is ephemeral and every LAPI pod restart
|
||||
# wiped it. Under the old password flow the agents registered exactly ONCE
|
||||
# (init container running `cscli lapi register --token`) and cached a
|
||||
# password the wiped LAPI no longer recognised -> 401 "machine not found"
|
||||
# -> CrashLoopBackOff on all three agents -> DaemonSet never Ready -> the
|
||||
# crowdsec HelmRelease upgrade `--wait` times out forever -> the whole
|
||||
# `infrastructure` Kustomization (and every app depending on it) stays
|
||||
# unhealthy. That is the #340 outage.
|
||||
#
|
||||
# With client-cert auth the agent init container no longer registers at all
|
||||
# (verified via `helm template`: the `cscli lapi register` step is gone).
|
||||
# The agent presents a cert whose OU (agent-ou) the LAPI trusts, and a
|
||||
# freshly restarted LAPI with an EMPTY DB auto-recreates each machine record
|
||||
# from the presented cert on connect. So the system self-heals across LAPI
|
||||
# restarts with no PVC, no Ceph, and no manual re-registration.
|
||||
# With client certs the agent never registers: a restarted LAPI with an
|
||||
# empty DB recreates each machine record from the presented cert, so this
|
||||
# self-heals with no PVC and no intervention.
|
||||
#
|
||||
# cert-manager (already reconciled earlier in this same `infrastructure`
|
||||
# stage) provisions a fully self-contained self-signed CA (selfSigned
|
||||
@@ -87,13 +78,10 @@ spec:
|
||||
|
||||
lapi:
|
||||
replicas: 1
|
||||
# The chart defaults request 500m. Measured steady-state draw is 2m, and
|
||||
# the request is what the scheduler reserves, so the default parks half a
|
||||
# core of unusable capacity per LAPI. That is affordable on three nodes
|
||||
# and is not on two: with one node out for maintenance the remaining pair
|
||||
# sit at 100% of requested CPU while actually using ~22%, and nothing new
|
||||
# can be placed. The limit stays at the chart default so a log flood can
|
||||
# still burst; only the reservation shrinks.
|
||||
# Far below the chart default, which reserves half a core per LAPI for a
|
||||
# workload that draws single-digit millicores — enough to leave the
|
||||
# cluster unschedulable with one node out. Only the reservation shrinks;
|
||||
# the limit stays at the chart default so a log flood can still burst.
|
||||
resources:
|
||||
requests:
|
||||
cpu: 50m
|
||||
|
||||
@@ -6,19 +6,12 @@
|
||||
# fails the route over across all per-node peers (no single point of
|
||||
# failure if a node/pod drops).
|
||||
#
|
||||
# Uses the ROOTLESS image (userspace wireguard-go). The netbird process
|
||||
# terminates the WireGuard tunnel in userspace and reaches cluster
|
||||
# services by opening its OWN sockets to them — NOT by L3-forwarding
|
||||
# packets. That distinction is the whole point:
|
||||
# - Cilium socket-LB DNATs the netbird process's own connections, so
|
||||
# forwarding to ClusterIP / LB VIPs actually works. A kernel L3
|
||||
# routing peer fails here: socket-LB never sees forwarded packets,
|
||||
# so they hit the raw VIP and are blackholed (confirmed via conntrack).
|
||||
# - No NET_ADMIN / no kernel WireGuard → runs fully unprivileged
|
||||
# (drop ALL caps, runAsNonRoot), so it satisfies Pod Security too.
|
||||
#
|
||||
# Flow: mesh peer (100.96.x.x) --wg--> a per-node pod (userspace netbird)
|
||||
# --own socket (socket-LB DNAT)--> service ClusterIP/LB VIP
|
||||
# Must stay on the ROOTLESS image (userspace wireguard-go). The process
|
||||
# terminates the tunnel in userspace and reaches services by opening its own
|
||||
# sockets, which Cilium's socket-LB can DNAT. A kernel WireGuard peer
|
||||
# L3-forwards instead, socket-LB never sees those packets, and they are
|
||||
# blackholed at the raw VIP. Rootless also means no NET_ADMIN, so the pod
|
||||
# runs fully unprivileged.
|
||||
#
|
||||
# Egress is locked down by the netbird-ingress-egress CiliumNetworkPolicy
|
||||
# (network-policies/ingress-isolation.yaml).
|
||||
|
||||
@@ -1,12 +1,5 @@
|
||||
---
|
||||
# Baseline ingress policy for the `woodpecker` namespace (wp-pg now; server and
|
||||
# agent arrive in stage 2).
|
||||
#
|
||||
# Unlike the `apps` baseline this replaces, every allow here lands on a
|
||||
# namespace holding one service, so a namespace-scoped grant is not a blanket
|
||||
# grant across unrelated workloads. That is the point of the move: in `apps`,
|
||||
# letting the CNPG operator reach wp-pg would also have opened gitea, gatus,
|
||||
# external-dns and ical-proxy to the whole database namespace (issue #433).
|
||||
# Baseline ingress policy for the `woodpecker` namespace.
|
||||
#
|
||||
# CAUTION when adding rules here: this policy has `endpointSelector: {}`, so
|
||||
# every rule below applies to EVERY pod in the namespace. A `toPorts` block
|
||||
@@ -15,11 +8,7 @@
|
||||
# belongs in its own policy with its own selector, as
|
||||
# `wp-pg-allow-cnpg-operator` below does.
|
||||
#
|
||||
# No egress section, so egress is unrestricted. Deliberate for now and revisited
|
||||
# in stage 2: wp-pg needs DNS plus `world` for WAL archiving to Hetzner S3, and
|
||||
# the server/agent add gitea, the Kubernetes API and outbound plugin pulls.
|
||||
# Writing that ruleset once, with the full workload set present, beats guessing
|
||||
# it here and silently breaking the WAL archiving this stage exists to verify.
|
||||
# There is no egress section, so egress is unrestricted. Still open (#433).
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
|
||||
@@ -5,25 +5,15 @@ kind: Kustomization
|
||||
# secret. The matching OpenBao policy/role for each lives in ../policies/ and is
|
||||
# bootstrapped manually (not GitOps-managed).
|
||||
#
|
||||
# IMAGE PINNING: every rotator here uses a DATED TAG of secret-rotator
|
||||
# (`:YYYY-MM-DD`), never a digest and never `:latest`. This is a registry-local
|
||||
# image, so it does not need digest pinning, and a dated tag makes "what actually
|
||||
# runs" a reviewable diff.
|
||||
# IMAGE PINNING: dated tags (`:YYYY-MM-DD`), never a digest and never
|
||||
# `:latest`. Do NOT switch to digests — untagged manifests in this registry are
|
||||
# garbage-collected once the tags move past them, so a pinned digest silently
|
||||
# stops resolving after a few unrelated rebuilds of build-ci-images.yml. These
|
||||
# jobs are quarterly, so the resulting ImagePullBackOff looks like nothing
|
||||
# happening rather than like a failure (#458).
|
||||
#
|
||||
# Digests were tried and actively broke: untagged manifests are garbage-collected
|
||||
# once :latest and the dated tags move past them, so a digest pinned today stops
|
||||
# resolving after a few rebuilds. That decay is driven by edits to
|
||||
# .woodpecker/build-ci-images.yml — a shared file — so adding an UNRELATED image
|
||||
# republishes secret-rotator and orphans the manifest every rotator here was
|
||||
# pinned to. 11 of these were pinned to digests that no longer existed and would
|
||||
# have ImagePullBackOff'd on their first scheduled run (issue #458); because they
|
||||
# are quarterly, that would have looked like nothing happening rather than like a
|
||||
# failure.
|
||||
#
|
||||
# Keep them all on one tag: it is a single binary containing every mode
|
||||
# (random | ha-token | cloudflare-roll | netbird-key | proxmox-token |
|
||||
# gitea-token | woodpecker-secret), so a split serves no purpose and just makes
|
||||
# the next bump partial.
|
||||
# Keep every rotator on the same tag: it is one binary containing all modes, so
|
||||
# splitting only makes the next bump partial.
|
||||
#
|
||||
# ADDING A ROTATOR: copy an existing file, and check you did not inherit a
|
||||
# digest pin from an older example — nothing enforces this automatically, so it
|
||||
|
||||
@@ -24,12 +24,10 @@ spec:
|
||||
- start: "fd99::"
|
||||
stop: "fd99::ff"
|
||||
---
|
||||
# Cilium BGPv2 control-plane config peering each k8s node with the UDM
|
||||
# over its MGMT LAN interface — NOT over v30iot. Control-plane traffic
|
||||
# belongs on the management network; the IoT VLAN is the noisy untrusted
|
||||
# L2 segment we're explicitly trying to keep cluster infrastructure off
|
||||
# of. Each node sources from its eth0 (10.2.40.x / fd40::<host>) via
|
||||
# the kernel routing decision; no explicit localAddress override needed.
|
||||
# Cilium BGPv2 control-plane config peering each node with the UDM over the
|
||||
# management LAN — never over v30iot, which is the untrusted IoT segment.
|
||||
# The source address comes from the kernel routing decision, so no explicit
|
||||
# localAddress override is needed.
|
||||
#
|
||||
# Two parallel sessions, one per address family:
|
||||
# - v4 session: peer 10.2.40.1, family IPv4 unicast only
|
||||
|
||||
@@ -1,22 +1,7 @@
|
||||
---
|
||||
# The Ceph cluster network as a Multus attachment, so OSDs reach VLAN 49
|
||||
# without every Ceph daemon being put on host networking to get there.
|
||||
#
|
||||
# WHAT THIS REPLACES, AND WHY
|
||||
#
|
||||
# `addressRanges` only takes effect under `provider: host` — Rook ignores it
|
||||
# for the default CNI provider. So carrying replication over the dedicated
|
||||
# 10G link meant setting provider: host, which is not per-daemon: it puts
|
||||
# mons on host networking too. A mon's address is then the node's DHCP lease,
|
||||
# and when that lease moves the mon cannot bind and fails. Worse, three mons
|
||||
# on three nodes each hold the mon host ports, so Rook cannot schedule a
|
||||
# replacement anywhere and the failover it starts cannot finish.
|
||||
#
|
||||
# None of that has anything to do with the goal. Per the Ceph documentation
|
||||
# the cluster network carries OSD heartbeat, replication and recovery only —
|
||||
# "Ceph Monitors always operate on the public network". Attaching it here
|
||||
# gives the OSDs the link and leaves everything else on the pod network,
|
||||
# where Rook manages identity itself.
|
||||
# The Ceph cluster network as a Multus attachment, so the OSDs reach VLAN 49
|
||||
# without every Ceph daemon landing on host networking to get there. The
|
||||
# alternative and why it was rejected is in cephcluster.yaml.
|
||||
#
|
||||
# ISOLATION
|
||||
#
|
||||
|
||||
@@ -7,11 +7,8 @@ spec:
|
||||
interval: 24h
|
||||
url: https://traefik.github.io/charts
|
||||
---
|
||||
# Traefik runs alongside ingress-nginx during the migration off the
|
||||
# archived kubernetes/ingress-nginx project. Shares the `ingress` namespace
|
||||
# so the post-migration topology is just "ingress namespace, one controller".
|
||||
# Pilots: shelly-manager.home.renner.uno. ingress-nginx remains the default
|
||||
# IngressClass; this controller is opt-in via `ingressClassName: traefik`.
|
||||
# The cluster's only ingress controller, and the only IngressClass. It
|
||||
# replaced ingress-nginx, which was retired after that project was archived.
|
||||
apiVersion: helm.toolkit.fluxcd.io/v2
|
||||
kind: HelmRelease
|
||||
metadata:
|
||||
@@ -44,27 +41,18 @@ spec:
|
||||
values:
|
||||
deployment:
|
||||
replicas: 2
|
||||
# Entrypoint-default Middlewares for websecure: baseline security
|
||||
# response headers (HSTS/CSP/X-Frame-Options/etc.) and a generous
|
||||
# abuse/brute-force rate limit, both applied to every router on this
|
||||
# entrypoint without per-Ingress annotations. See
|
||||
# security-headers-middleware.yaml / rate-limit-middleware.yaml for the
|
||||
# actual settings and rationale. This is the same entrypoint-default
|
||||
# wiring mechanism the (currently detached) crowdsec-bouncer Middleware
|
||||
# documents re-enabling via — independent of that CrowdSec enforcement
|
||||
# decision, which stays off pending the client-IP-visibility fix
|
||||
# referenced below.
|
||||
# Entrypoint-default Middlewares for websecure: security response headers
|
||||
# and a rate limit, applied to every router on this entrypoint without
|
||||
# per-Ingress annotations. Settings live in
|
||||
# security-headers-middleware.yaml and rate-limit-middleware.yaml.
|
||||
additionalArguments:
|
||||
- --entryPoints.websecure.http.middlewares=ingress-security-headers@kubernetescrd,ingress-rate-limit@kubernetescrd
|
||||
# CrowdSec bouncer plugin (Yaegi), loaded but NOT globally attached.
|
||||
# Enforcement was disabled after it caused an IPv6 ingress outage on
|
||||
# first deploy: Traefik sees a single shared/SNAT'd source IP for v6
|
||||
# ingress (no real per-client IP), so one http-admin-interface-probing
|
||||
# hit (the e2e scan) banned that shared IP and 403'd all v6 clients.
|
||||
# Re-attach the bouncer (additionalArguments entrypoint middleware ->
|
||||
# ingress-crowdsec-bouncer@kubernetescrd) only after fixing client-IP
|
||||
# visibility (forwardedHeaders/trustedIPs or the v6 masquerade) and
|
||||
# adding an internal/LAN whitelist. The plugin + crowdsec-middleware.yaml
|
||||
# CrowdSec bouncer plugin (Yaegi), loaded but deliberately NOT attached.
|
||||
# Traefik sees one shared SNAT'd source for IPv6 ingress, so a single
|
||||
# probe banned that address and 403'd every v6 client. Do not re-attach
|
||||
# it until client-IP visibility is fixed (forwardedHeaders/trustedIPs or
|
||||
# the v6 masquerade) and a LAN whitelist exists.
|
||||
# The plugin + crowdsec-middleware.yaml
|
||||
# stay in place so re-enabling is a one-line change. Detection (the
|
||||
# crowdsec agents parsing Traefik logs) remains on and is unaffected.
|
||||
experimental:
|
||||
|
||||
Reference in New Issue
Block a user