Compare commits

...
Author SHA1 Message Date
claude-bot 6ffbc468d7 docs(infra): trim the remaining oversized comments
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
crowdsec, netbird-ingress, woodpecker-ns, san-network-attachment, cilium-bgp
and the openbao rotation kustomization. Comments only; the diff filtered for
non-comment lines is empty and infrastructure/ still builds.

Every instruction survives, because that is the half worth having: do not
switch crowdsec back to password auth, or a wiped LAPI crashloops the agents
and wedges the Kustomization; do not move netbird-ingress off the rootless
image, or socket-LB stops seeing its traffic and the VIPs blackhole; do not
pin the rotators by digest, or a quarterly job fails to pull a manifest that
was garbage-collected months earlier and looks like it simply did not run.

What went is the incident each was learned from, and one duplicate: the
san-network-attachment header re-derived the provider: host argument that
cephcluster.yaml already makes, so it now points there instead.

Two more stale claims found on the way through:

  - woodpecker-ns.yaml described the namespace as holding wp-pg alone with
    'server and agent arriving in stage 2'. Both have been running there for
    some time.
  - cilium-bgp.yaml said each node sources its BGP session from eth0. The
    management interface is mgmt0; eth0 was the VM-era name.

infrastructure/ is now 2553 comment lines.
2026-08-28 06:50:54 +02:00
claude-bot 8c7fd25e03 docs(infra): trim the cilium and traefik helm release comments
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
cilium 86 -> 63 comment lines, traefik 71 -> 59. Comments only; the diff
filtered for non-comment lines is empty and both files still parse.

Cilium keeps everything that is a trap rather than a story: that bpf-policy-map-max
and vlan-bpf-bypass each exist in three files with nothing enforcing agreement,
that both take an agent restart, that a full policy map surfaces as a pod stuck
in sandbox creation, and why only VLAN 30 belongs in the bypass list. What went
is how each was diagnosed — the Hubble traces, the manual agent deletions, the
device-by-device walk through `cilium-dbg status`.

Traefik keeps the version ceiling and the reason it exists, and keeps the
CrowdSec bouncer note as an instruction: do not re-attach until client-IP
visibility is fixed and a LAN whitelist exists. It loses the retelling of the
outage that produced that instruction.

The traefik header was outright wrong and is the reason this file was worth
opening. It described Traefik as running "alongside ingress-nginx during the
migration", called ingress-nginx "the default IngressClass" and Traefik
"opt-in via ingressClassName: traefik". There is one IngressClass in the
cluster and it is traefik; ingress-nginx has no deployment and no class. A
reader following that comment would have added an ingressClassName they do not
need and looked for a controller that is not there.
2026-08-28 06:47:39 +02:00
k.renner 7f11867407 Merge pull request 'docs(netpol): deduplicate the home-assistant policy comments' (#684) from trim-comments-netpol into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#684
2026-08-28 04:45:23 +00:00
8 changed files with 80 additions and 172 deletions
@@ -33,22 +33,14 @@ spec:
remediation:
retries: 3
values:
# Roll the agent DaemonSet when cilium-config changes. Cilium reads
# datapath options once, at agent start, and the chart puts no config
# checksum on the pod template — so a changed ConfigMap leaves every agent
# running the previous datapath until something restarts it. Nothing did.
# Rolls the agent DaemonSet when cilium-config changes. Cilium reads
# datapath options once at agent start and the chart puts no config
# checksum on the pod template, so without this a changed ConfigMap leaves
# every agent on the previous datapath indefinitely (#592).
#
# This is not hypothetical. vlan-bpf-bypass reached cilium-config and every
# v30iot macvlan workload stayed broken until the agents were deleted by
# hand, one node at a time (issue #592). policyMapMax has the same
# exposure, and its comment below says "takes effect on agent restart" —
# documentation standing in for automation.
#
# The trade: any cilium-config change now rolls the CNI cluster-wide,
# node by node. That is disruptive-ish, but it is what a datapath change
# already required — the difference is that it now happens on its own
# instead of being forgotten. Reloader is already deployed for exactly
# this pattern; named per-ConfigMap rather than blanket, as elsewhere.
# The trade is that any cilium-config change now rolls the CNI node by
# node — which is what a datapath change required anyway, only now it is
# not forgotten.
annotations:
configmap.reloader.stakater.com/reload: cilium-config
# Which pod labels feed security-identity computation. Patterns are
@@ -69,57 +61,42 @@ spec:
socketLB:
enabled: true
prometheus:
# The agent exported no metrics at all, so BPF map utilisation was
# invisible: when per-endpoint policy maps (cilium_policy_v3_*) run out
# of space the agent only says "no space left on device" in its log and
# pod sandbox creation fails (issue #452). cilium_bpf_map_pressure makes
# that observable — and alertable — before it stalls a rollout.
# Without agent metrics, BPF map utilisation is invisible: a full policy
# map only shows up as "no space left on device" in the log while pod
# sandbox creation fails (#452). cilium_bpf_map_pressure makes it
# alertable beforehand.
enabled: true
serviceMonitor:
enabled: true
bpf:
# Size of each endpoint's own cilium_policy_v3_<id> map, holding one
# entry per (identity, direction, protocol, port) the endpoint is
# allowed. When it fills, endpoint regeneration fails with "no space
# left on device", the CNI ADD call never returns, and the pod is stuck
# in sandbox creation — so this must stay comfortably above the busiest
# endpoint's entry count, tracked by cilium_bpf_map_pressure.
# Takes effect on agent restart; the map is sized at creation.
# Bootstrap installs Cilium from the inline manifest in
# kubernetes/talos/patches/cilium-cni.yaml, so bpf-policy-map-max there
# must move with this value or a cold rebuild starts with the old size.
# kubernetes/talos/cilium.yaml carries a third copy of the same key.
# Nothing enforces any of this; grep bpf-policy-map-max before changing it.
# Per-endpoint policy map size: one entry per (identity, direction,
# protocol, port) allowed. When it fills, endpoint regeneration fails
# with "no space left on device" and the pod hangs in sandbox creation.
# Keep it comfortably above the busiest endpoint, watched by
# cilium_bpf_map_pressure. Sized at map creation, so it takes an agent
# restart.
#
# THREE COPIES of this key exist — here, talos/patches/cilium-cni.yaml
# and talos/cilium.yaml. The latter two are what a cold rebuild installs
# from. Nothing enforces agreement; grep bpf-policy-map-max before
# changing it.
policyMapMax: 65536
# VLAN ids the eBPF datapath is allowed to pass. Without this it drops
# VLAN-tagged frames, accounted as "VLAN traffic disallowed by VLAN
# filter" in cilium_drop_count_total.
#
# 30 is v30iot, the macvlan parent for the shelly-manager, matter-server,
# otbr and music-assistant attachments. On bare metal it is a VLAN
# sub-interface of enp2s0f0np0; inside the former Proxmox guests it was
# an untagged virtio NIC, which is why this was not needed before.
# Only VLAN 30 (v30iot) needs listing. Cilium attaches to devices that
# carry an address, so it sees mgmt0's frames already untagged by the
# 8021q layer — nothing for a VLAN filter to match. ens19 is deliberately
# address-less, so its frames only meet the filter on the shared parent,
# still tagged. Adding 40 would change nothing and would touch the
# datapath carrying etcd and the API.
#
# VLAN 40 (mgmt0) is deliberately NOT listed, and the asymmetry is the
# interesting part. Cilium attaches to devices that carry an address —
# `cilium-dbg status --verbose` lists enp87s0, enp2s0f0np0, enp2s0f1np1
# and mgmt0, but NOT ens19. mgmt0 holds the node IP, so Cilium owns it as
# a device and sees frames the 8021q layer has already untagged; nothing
# for a VLAN filter to match. ens19 is deliberately address-less (see
# talos/patches/k8s-1.yaml — host addresses on that L2 make node replies
# bypass the UDM), so Cilium never attaches to it and VLAN 30 frames only
# meet the filter on the shared parent, still tagged.
# The symptom when this is wrong is not a DHCP timeout: a tc-layer drop
# reaches an AF_PACKET sender as ENOBUFS, so the CNI dhcp plugin reports
# "no buffer space available" (#592).
#
# So the fix belongs on 30 alone. Listing 40 would change nothing, and
# would touch the datapath carrying etcd and the API for no reason.
#
# A tc-layer drop surfaces to an AF_PACKET sender as ENOBUFS, so the
# symptom is the CNI dhcp plugin reporting "no buffer space available"
# rather than a DHCP timeout. See issue #592.
#
# Same three-copy problem as policyMapMax above: vlan-bpf-bypass in
# kubernetes/talos/cilium.yaml and kubernetes/talos/patches/cilium-cni.yaml
# must move with this list. Takes effect on agent restart.
# Same three-copy problem as policyMapMax above. Takes an agent restart.
vlanBypass: [30]
# Let pods reach the cluster's own LoadBalancer external IPs; Cilium
# redirects the socket directly to the backend pod instead of sending
@@ -36,25 +36,16 @@ spec:
values:
container_runtime: containerd
# --- mTLS between agents and the LAPI: the durable fix for the recurring
# KubeDaemonSetRolloutStuck auth loop (#340) ---
# mTLS between agents and the LAPI. Do not switch back to password auth:
# the LAPI runs without persistence, so its machine-registration DB is
# wiped on every restart, and password-registered agents then get 401
# "machine not found" and crashloop until re-registered by hand — which
# wedges the whole `infrastructure` Kustomization behind a HelmRelease
# that never goes Ready (#340).
#
# The LAPI runs WITHOUT persistence (see persistentVolume note below), so
# its sqlite machine-registration DB is ephemeral and every LAPI pod restart
# wiped it. Under the old password flow the agents registered exactly ONCE
# (init container running `cscli lapi register --token`) and cached a
# password the wiped LAPI no longer recognised -> 401 "machine not found"
# -> CrashLoopBackOff on all three agents -> DaemonSet never Ready -> the
# crowdsec HelmRelease upgrade `--wait` times out forever -> the whole
# `infrastructure` Kustomization (and every app depending on it) stays
# unhealthy. That is the #340 outage.
#
# With client-cert auth the agent init container no longer registers at all
# (verified via `helm template`: the `cscli lapi register` step is gone).
# The agent presents a cert whose OU (agent-ou) the LAPI trusts, and a
# freshly restarted LAPI with an EMPTY DB auto-recreates each machine record
# from the presented cert on connect. So the system self-heals across LAPI
# restarts with no PVC, no Ceph, and no manual re-registration.
# With client certs the agent never registers: a restarted LAPI with an
# empty DB recreates each machine record from the presented cert, so this
# self-heals with no PVC and no intervention.
#
# cert-manager (already reconciled earlier in this same `infrastructure`
# stage) provisions a fully self-contained self-signed CA (selfSigned
@@ -87,13 +78,10 @@ spec:
lapi:
replicas: 1
# The chart defaults request 500m. Measured steady-state draw is 2m, and
# the request is what the scheduler reserves, so the default parks half a
# core of unusable capacity per LAPI. That is affordable on three nodes
# and is not on two: with one node out for maintenance the remaining pair
# sit at 100% of requested CPU while actually using ~22%, and nothing new
# can be placed. The limit stays at the chart default so a log flood can
# still burst; only the reservation shrinks.
# Far below the chart default, which reserves half a core per LAPI for a
# workload that draws single-digit millicores — enough to leave the
# cluster unschedulable with one node out. Only the reservation shrinks;
# the limit stays at the chart default so a log flood can still burst.
resources:
requests:
cpu: 50m
@@ -6,19 +6,12 @@
# fails the route over across all per-node peers (no single point of
# failure if a node/pod drops).
#
# Uses the ROOTLESS image (userspace wireguard-go). The netbird process
# terminates the WireGuard tunnel in userspace and reaches cluster
# services by opening its OWN sockets to them — NOT by L3-forwarding
# packets. That distinction is the whole point:
# - Cilium socket-LB DNATs the netbird process's own connections, so
# forwarding to ClusterIP / LB VIPs actually works. A kernel L3
# routing peer fails here: socket-LB never sees forwarded packets,
# so they hit the raw VIP and are blackholed (confirmed via conntrack).
# - No NET_ADMIN / no kernel WireGuard → runs fully unprivileged
# (drop ALL caps, runAsNonRoot), so it satisfies Pod Security too.
#
# Flow: mesh peer (100.96.x.x) --wg--> a per-node pod (userspace netbird)
# --own socket (socket-LB DNAT)--> service ClusterIP/LB VIP
# Must stay on the ROOTLESS image (userspace wireguard-go). The process
# terminates the tunnel in userspace and reaches services by opening its own
# sockets, which Cilium's socket-LB can DNAT. A kernel WireGuard peer
# L3-forwards instead, socket-LB never sees those packets, and they are
# blackholed at the raw VIP. Rootless also means no NET_ADMIN, so the pod
# runs fully unprivileged.
#
# Egress is locked down by the netbird-ingress-egress CiliumNetworkPolicy
# (network-policies/ingress-isolation.yaml).
@@ -1,12 +1,5 @@
---
# Baseline ingress policy for the `woodpecker` namespace (wp-pg now; server and
# agent arrive in stage 2).
#
# Unlike the `apps` baseline this replaces, every allow here lands on a
# namespace holding one service, so a namespace-scoped grant is not a blanket
# grant across unrelated workloads. That is the point of the move: in `apps`,
# letting the CNPG operator reach wp-pg would also have opened gitea, gatus,
# external-dns and ical-proxy to the whole database namespace (issue #433).
# Baseline ingress policy for the `woodpecker` namespace.
#
# CAUTION when adding rules here: this policy has `endpointSelector: {}`, so
# every rule below applies to EVERY pod in the namespace. A `toPorts` block
@@ -15,11 +8,7 @@
# belongs in its own policy with its own selector, as
# `wp-pg-allow-cnpg-operator` below does.
#
# No egress section, so egress is unrestricted. Deliberate for now and revisited
# in stage 2: wp-pg needs DNS plus `world` for WAL archiving to Hetzner S3, and
# the server/agent add gitea, the Kubernetes API and outbound plugin pulls.
# Writing that ruleset once, with the full workload set present, beats guessing
# it here and silently breaking the WAL archiving this stage exists to verify.
# There is no egress section, so egress is unrestricted. Still open (#433).
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
@@ -5,25 +5,15 @@ kind: Kustomization
# secret. The matching OpenBao policy/role for each lives in ../policies/ and is
# bootstrapped manually (not GitOps-managed).
#
# IMAGE PINNING: every rotator here uses a DATED TAG of secret-rotator
# (`:YYYY-MM-DD`), never a digest and never `:latest`. This is a registry-local
# image, so it does not need digest pinning, and a dated tag makes "what actually
# runs" a reviewable diff.
# IMAGE PINNING: dated tags (`:YYYY-MM-DD`), never a digest and never
# `:latest`. Do NOT switch to digests — untagged manifests in this registry are
# garbage-collected once the tags move past them, so a pinned digest silently
# stops resolving after a few unrelated rebuilds of build-ci-images.yml. These
# jobs are quarterly, so the resulting ImagePullBackOff looks like nothing
# happening rather than like a failure (#458).
#
# Digests were tried and actively broke: untagged manifests are garbage-collected
# once :latest and the dated tags move past them, so a digest pinned today stops
# resolving after a few rebuilds. That decay is driven by edits to
# .woodpecker/build-ci-images.yml — a shared file — so adding an UNRELATED image
# republishes secret-rotator and orphans the manifest every rotator here was
# pinned to. 11 of these were pinned to digests that no longer existed and would
# have ImagePullBackOff'd on their first scheduled run (issue #458); because they
# are quarterly, that would have looked like nothing happening rather than like a
# failure.
#
# Keep them all on one tag: it is a single binary containing every mode
# (random | ha-token | cloudflare-roll | netbird-key | proxmox-token |
# gitea-token | woodpecker-secret), so a split serves no purpose and just makes
# the next bump partial.
# Keep every rotator on the same tag: it is one binary containing all modes, so
# splitting only makes the next bump partial.
#
# ADDING A ROTATOR: copy an existing file, and check you did not inherit a
# digest pin from an older example — nothing enforces this automatically, so it
@@ -24,12 +24,10 @@ spec:
- start: "fd99::"
stop: "fd99::ff"
---
# Cilium BGPv2 control-plane config peering each k8s node with the UDM
# over its MGMT LAN interface — NOT over v30iot. Control-plane traffic
# belongs on the management network; the IoT VLAN is the noisy untrusted
# L2 segment we're explicitly trying to keep cluster infrastructure off
# of. Each node sources from its eth0 (10.2.40.x / fd40::<host>) via
# the kernel routing decision; no explicit localAddress override needed.
# Cilium BGPv2 control-plane config peering each node with the UDM over the
# management LAN — never over v30iot, which is the untrusted IoT segment.
# The source address comes from the kernel routing decision, so no explicit
# localAddress override is needed.
#
# Two parallel sessions, one per address family:
# - v4 session: peer 10.2.40.1, family IPv4 unicast only
@@ -1,22 +1,7 @@
---
# The Ceph cluster network as a Multus attachment, so OSDs reach VLAN 49
# without every Ceph daemon being put on host networking to get there.
#
# WHAT THIS REPLACES, AND WHY
#
# `addressRanges` only takes effect under `provider: host` — Rook ignores it
# for the default CNI provider. So carrying replication over the dedicated
# 10G link meant setting provider: host, which is not per-daemon: it puts
# mons on host networking too. A mon's address is then the node's DHCP lease,
# and when that lease moves the mon cannot bind and fails. Worse, three mons
# on three nodes each hold the mon host ports, so Rook cannot schedule a
# replacement anywhere and the failover it starts cannot finish.
#
# None of that has anything to do with the goal. Per the Ceph documentation
# the cluster network carries OSD heartbeat, replication and recovery only —
# "Ceph Monitors always operate on the public network". Attaching it here
# gives the OSDs the link and leaves everything else on the pod network,
# where Rook manages identity itself.
# The Ceph cluster network as a Multus attachment, so the OSDs reach VLAN 49
# without every Ceph daemon landing on host networking to get there. The
# alternative and why it was rejected is in cephcluster.yaml.
#
# ISOLATION
#
@@ -7,11 +7,8 @@ spec:
interval: 24h
url: https://traefik.github.io/charts
---
# Traefik runs alongside ingress-nginx during the migration off the
# archived kubernetes/ingress-nginx project. Shares the `ingress` namespace
# so the post-migration topology is just "ingress namespace, one controller".
# Pilots: shelly-manager.home.renner.uno. ingress-nginx remains the default
# IngressClass; this controller is opt-in via `ingressClassName: traefik`.
# The cluster's only ingress controller, and the only IngressClass. It
# replaced ingress-nginx, which was retired after that project was archived.
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
@@ -44,27 +41,18 @@ spec:
values:
deployment:
replicas: 2
# Entrypoint-default Middlewares for websecure: baseline security
# response headers (HSTS/CSP/X-Frame-Options/etc.) and a generous
# abuse/brute-force rate limit, both applied to every router on this
# entrypoint without per-Ingress annotations. See
# security-headers-middleware.yaml / rate-limit-middleware.yaml for the
# actual settings and rationale. This is the same entrypoint-default
# wiring mechanism the (currently detached) crowdsec-bouncer Middleware
# documents re-enabling via — independent of that CrowdSec enforcement
# decision, which stays off pending the client-IP-visibility fix
# referenced below.
# Entrypoint-default Middlewares for websecure: security response headers
# and a rate limit, applied to every router on this entrypoint without
# per-Ingress annotations. Settings live in
# security-headers-middleware.yaml and rate-limit-middleware.yaml.
additionalArguments:
- --entryPoints.websecure.http.middlewares=ingress-security-headers@kubernetescrd,ingress-rate-limit@kubernetescrd
# CrowdSec bouncer plugin (Yaegi), loaded but NOT globally attached.
# Enforcement was disabled after it caused an IPv6 ingress outage on
# first deploy: Traefik sees a single shared/SNAT'd source IP for v6
# ingress (no real per-client IP), so one http-admin-interface-probing
# hit (the e2e scan) banned that shared IP and 403'd all v6 clients.
# Re-attach the bouncer (additionalArguments entrypoint middleware ->
# ingress-crowdsec-bouncer@kubernetescrd) only after fixing client-IP
# visibility (forwardedHeaders/trustedIPs or the v6 masquerade) and
# adding an internal/LAN whitelist. The plugin + crowdsec-middleware.yaml
# CrowdSec bouncer plugin (Yaegi), loaded but deliberately NOT attached.
# Traefik sees one shared SNAT'd source for IPv6 ingress, so a single
# probe banned that address and 403'd every v6 client. Do not re-attach
# it until client-IP visibility is fixed (forwardedHeaders/trustedIPs or
# the v6 masquerade) and a LAN whitelist exists.
# The plugin + crowdsec-middleware.yaml
# stay in place so re-enabling is a one-line change. Detection (the
# crowdsec agents parsing Traefik logs) remains on and is unaffected.
experimental: