crowdsec, external-dns-unifi, netbird-ingress and velero were in the
infrastructure Kustomization, while the ExternalSecrets that create their
Secrets were in infrastructure-crds, which depends on infrastructure. On a
cold start each was therefore installed before its Secret could exist: the
pods sat in CreateContainerConfigError until OpenBao was unsealed and ESO
synced. For the HelmReleases (crowdsec, external-dns-unifi, velero) Helm's
wait timed out meanwhile; with remediation.retries: 3 they give up after
four attempts and stay failed even once the Secret appears. Velero failing
that way stops every restore.
| workload | Secrets |
|---|---|
| crowdsec | crowdsec-bouncer-key (crowdsec, ingress), crowdsec-edge-bouncer-key |
| external-dns-unifi | external-dns-unifi-secret |
| netbird-ingress | netbird-ingress-setup-key |
| velero (+ backup-verify CronJob) | cloud-credentials, cloud-credentials-hetzner, hcloud-backup-verify |
Each now has a Kustomization of its own (clusters/homelab/
infrastructure-services.yaml), depending on infrastructure and
infrastructure-crds, with its ExternalSecrets beside it. The openbao
ClusterSecretStore in infrastructure-crds turns Ready only once OpenBao is
unsealed, so the Secret exists before the consumer installs. One
Kustomization each so that a broken service holds back only itself.
Velero's Schedules need the velero.io CRDs from its HelmRelease, so they move
from post-crds to infrastructure/velero-schedules with a Kustomization after
velero.
post-crds keeps cloudflare-api-token (consumed by its own ClusterIssuer) and
flux-git. The directories themselves stay where they are, so paths such as
the CI test infrastructure/velero/backup-verify/tests/run.sh are unchanged.
netbird-ingress gets 10m to roll its DaemonSet over three nodes
(maxUnavailable 1, minReadySeconds 45). The crowdsec comments name the
Kustomization it now runs in, and flux-graph.sh colours the new nodes.
Production rollout: infrastructure and infrastructure-crds must run with
prune: false (the stack's first PR) when this merges. Pruned, the velero
HelmRelease would be uninstalled, and the moved ExternalSecrets would take
their Secrets with them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 1 (#802) put the ServersTransport under post-crds and
infrastructure-crds has applied it. The copy in crowdsec/lapi-ingress.yaml
goes; the Ingress stays (core kind). With it, the infrastructure apply set
holds no traefik.io consumer any more, which is what a cold rebuild needs.
Prune: infrastructure reconciles first and skips the object if the last
applier label names infrastructure-crds; otherwise it deletes and
infrastructure-crds recreates it seconds later. Only the public LAPI
ingress used by the external bouncers is behind that transport.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
One unknown kind fails a server-side dry-run for the WHOLE apply set,
including the very HelmRelease that would have installed the missing
CRDs. On a running cluster every CRD exists, so this never showed; a
cold rebuild deadlocks on it immediately.
Three groups move, content unchanged:
ExternalSecrets and Traefik Middlewares out of infrastructure into
post-crds, which reconciles after the operators. The flux-git
ExternalSecret also changes creationPolicy Merge -> Orphan with
deletionPolicy Retain: Merge never CREATES the target secret, so on a
fresh cluster the Gitea credential handover silently did not happen.
ServiceMonitors/PodMonitors out of their charts and app sets into
apps/monitoring-rules, which applies after kube-prometheus-stack owns
the CRDs. Rook additionally stops creating its own two monitors
(spec.monitoring.enabled: false) — creating them is fatal to its
node-daemon reconcile when the CRD is missing, which held the
CephCluster in Progressing and with it the whole apps layer. Both
objects are reproduced field for field off the live cluster; the
ceph-exporter DaemonSet and the metrics endpoints are untouched.
Cilium is the worst case: its chart aborts rendering outright on the
missing CRD, and cilium is one of the seven infrastructure
healthChecks — the lowest layer in the cluster waiting on the highest.
Steady-state effect: object ownership moves between Kustomizations;
specs are identical. Verified on local vmtest runs and the Hetzner
throwaway cluster.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both agents stopped reading on 2026-08-27 and had read nothing since:
k8s-1 at 12:15:41Z, k8s-2 at 19:14:27Z, each frozen at its line count
(35449 and 29917) while Traefik kept logging normally.
Each had hit its first kubelet log rotation after starting. The paths
under /var/log/containers are symlinks into /var/log/pods, and kubelet
rotates the target at containerLogMaxSize; the inotify tailer reopens
the path, waits for a file that is already present, and never returns.
Nothing about the result looks broken: pods Running and Ready, LAPI
answering, bouncers pulling, cumulative counters still showing large
numbers. The only symptom is that no scenario fires again, which reads
as quiet rather than blind.
So this was not a one-off. Rotation happens roughly every eight hours
here, and the agents die on the first one after every start, which
bounds local detection to a few hours at a time.
CrowdSec diagnoses this itself and names the fix, at agent startup:
File /var/log/containers/traefik-...log is a symlink, but inotify
polling is enabled. Crowdsec will not be able to detect rotation.
Consider setting poll_without_inotify to true in your configuration
Upstream carries the identical case as crowdsec#2449 (k3s + Traefik),
and the chart gained this field in helm-charts#109 to fix it. It does
not conflict with the force_inotify: true the chart hardcodes: that
governs the fsnotify watch on the directory, used to discover the log
of a replacement Traefik pod, while this governs how an already
discovered file is read. Verified by rendering the chart.
The acquisition ConfigMap is checksummed into the agent pod template,
so this rolls the DaemonSet and the stalled agents restart with it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
First host goes from dry-run to enforcing. proxy-2 deliberately: it is the
dual-stack member, while proxy-1 is IPv6-only and the one where every
network-shaped problem has surfaced first — the dhcpcd/resolv.conf breakage,
and a GitHub download that could not work at all. host_vars keeps the change
to one host; proxy-1 stays on the role default.
Verified before committing: ansible-inventory resolves nftables for proxy-2
and nothing for proxy-1.
Adds netbird.renner.uno to capi_whitelists first, and that is not decoration.
The bouncer's chain sits on the INPUT hook, the proxy hosts run no other
firewall, and there is therefore no conntrack rule accepting established flows
ahead of it. A ban on the management address would drop the return traffic of
the proxies' own outbound control connection and take them off the mesh, with
no inbound attack involved at all. The same reasoning already covers the proxy
addresses, which were listed from the start.
Checked at the time of writing: none of these addresses appear in the current
decision set — searched all ~15k entries. That is the state today, not a
guarantee; they are rented addresses on a hoster whose ranges do end up on
community blocklists, which is the whole reason for listing them.
The delivery path was tested end to end beforehand with a decision on
192.0.2.1 (TEST-NET-1, so no real traffic could be affected). Both bouncers
received the ban within one poll cycle — a 206-byte delta against the usual
26 — and the revocation the same way, 203 bytes, ~5s after deletion. What that
did not test is whether a packet is actually dropped, which dry-run cannot
show. This commit is what makes that testable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both bouncers reach the LAPI and pull every 10s, and every pull gets a 400:
2a03:4000:28:f6e:...:9a29 "GET /v1/decisions/stream?..." 400 48
2a03:4000:4b:c1e:...:8f85 "GET /v1/decisions/stream?..." 400 48
The body says what it is: "Client sent an HTTP request to an HTTPS server."
Traefik was talking plaintext to a backend that serves TLS only.
serversscheme and serverstransport were on the Ingress. Under the Kubernetes
Ingress provider `traefik.ingress.kubernetes.io/service.*` are SERVICE
annotations; on an Ingress they are silently ignored. Nothing warns, and the
failure surfaces as a 400 that reads like a malformed request rather than a
scheme mismatch. They move to the chart's Service via lapi.service.annotations.
With tls.enabled the LAPI serves HTTPS on 8080 and offers no plaintext port,
so https is mandatory here rather than hardening.
Also switches the ServersTransport from rootCAsSecrets to rootCAs, which is
what Traefik has been warning about on every reconcile:
RootCAsSecrets option is deprecated, please use the RootCA option instead.
Two of my own diagnostics were wrong along the way and both were measurement
artifacts. "Traefik sees no requests" came from a Loki query whose limit was
consumed by those deprecation warnings. And the source address that justified
dropping the IPAllowList was measured on the watchdog, which curls 127.0.0.1;
the bouncer resolves the public name and arrives on the proxy's public
address, as the log lines above show.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Found by the pr-review bot on #751, and it is the exact failure this design
was written to avoid.
BOUNCER_KEY_edge is consumed as an env var from a secretKeyRef, and env is
fixed at pod start. The Reloader annotation listed only crowdsec-bouncer-key,
so a rotation of the edge key would update OpenBao, sync into the Secret, and
reach the hosts through the Ansible play — while the LAPI kept the old value
in its environment and went on trusting the old key. The bouncers would then
present the new one and get 401s, at exactly the moment the rotation was
supposed to be seamless.
The comment directly above that annotation already stated the rule — "it's an
env-var secret, so a restart is required to pick up a rotated key". I added a
second env-var secret and did not extend the list.
Comma-separated, matching shelly-manager-deployment.yaml.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The allowlist carried the published proxy addresses on the assumption that a
request from a proxy host arrives carrying that host's public address. It does
not.
The client connects to the local proxy on 127.0.0.1, the reverse-proxy writes
what it saw into X-Forwarded-For, and Traefik resolves that into ClientHost.
Measured against netbird-proxy-watchdog, which makes exactly this kind of
request every round:
172.18.0.3 - - [02/Sep/2026:10:12:01] "GET /" 200 websecure-immich-...
172.18.0.3 - - [02/Sep/2026:10:12:01] "GET /" 200 websecure-home-...
Two probes one second apart, one per watchdog target. 172.18.0.3 is a Docker
bridge address on the proxy side.
So the allowlist would have 403ed the bouncer. Allowlisting 172.18.0.0/16
instead is not a control: it is a private range on a host whose addressing we
do not own, and it would admit anything else that happens to originate there.
The bouncer key stays as the access control, which is what it always was. The
endpoint serves read-only decision streams to a valid key; the admin surface
is cscli, in-cluster. If the source address ever becomes deterministic, an
ipAllowList can go back on top.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Groundwork for enforcing CrowdSec decisions on proxy-1/proxy-2. Inert on its
own: an Ingress nobody calls yet, a bouncer key nobody uses yet, and a
whitelist that only ever narrows what may be banned.
The proxies cannot reach the LAPI over the mesh. The netbird-proxy container
runs non-root — that is what the ip_unprivileged_port_start task in the role
is for — so it has no NET_ADMIN and no /dev/net/tun, and its embedded NetBird
client terminates the tunnel in userspace. The tunnel belongs to that process,
not to the host netns, so a second process on the same host cannot route
through it. The alternative was a host-level NetBird client plus the LAPI on a
LoadBalancer VIP: two more peers, an ACL policy and a CNP rule. This reuses
the path every other service already takes.
Exposure is narrowed twice. The IPAllowList carries the three proxy addresses,
resolving them past the netbird-ingress pod IP through X-Forwarded-For the
same way the rate-limit middleware does — without that it would compare
against the shared forwarder IP, which is the gap behind the 2026-06-27 mass
ban. Behind it, the bouncer key. The LAPI serves read-only decision streams to
a valid key; the admin surface is cscli, in-cluster.
No CNP change is needed: crowdsec-ns.yaml already admits the traefik pods, and
ingress-default-deny carries no egress section.
BOUNCER_KEY_edge is separate from BOUNCER_KEY_ingress on purpose. The latter
rotates quarterly and both its consumers are in-cluster, so ESO propagates
within 10m. The edge hosts are Ansible-managed and read OpenBao on no
schedule, so a rotation would leave them 401ing until someone runs the
playbook. This key rotates with the playbook that carries it.
capi_whitelists.yaml is the only declarative allowlist the chart offers.
`cscli allowlists` objects live in the LAPI database, which runs without
persistence here deliberately, so anything created that way is gone at the
next restart. This file is a ConfigMap mount and survives by construction.
Upstream deprecates it; it is still the only thing that works against a LAPI
with no PVC.
It lists the proxy addresses and not the home WAN address, which is not
needed: unifi_nat_hairpin.tf DNATs the public proxy addresses to the Traefik
VIP on the UDM, so LAN clients reach the cluster directly with a LAN source
and never traverse the proxies.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
crowdsec, netbird-ingress, woodpecker-ns, san-network-attachment, cilium-bgp
and the openbao rotation kustomization. Comments only; the diff filtered for
non-comment lines is empty and infrastructure/ still builds.
Every instruction survives, because that is the half worth having: do not
switch crowdsec back to password auth, or a wiped LAPI crashloops the agents
and wedges the Kustomization; do not move netbird-ingress off the rootless
image, or socket-LB stops seeing its traffic and the VIPs blackhole; do not
pin the rotators by digest, or a quarterly job fails to pull a manifest that
was garbage-collected months earlier and looks like it simply did not run.
What went is the incident each was learned from, and one duplicate: the
san-network-attachment header re-derived the provider: host argument that
cephcluster.yaml already makes, so it now points there instead.
Two more stale claims found on the way through:
- woodpecker-ns.yaml described the namespace as holding wp-pg alone with
'server and agent arriving in stage 2'. Both have been running there for
some time.
- cilium-bgp.yaml said each node sources its BGP session from eth0. The
management interface is mgmt0; eth0 was the VM-era name.
infrastructure/ is now 2553 comment lines.
No Proxmox remains. jupiter, saturn and neptune all run Talos on bare metal,
so everything that scraped, proxied, alerted on or rotated credentials for the
PVE API now points at hosts that answer nothing on :8006.
Two of these were not merely dead but actively failing on a schedule:
gatus an Infrastructure check on https://pve.renner.uno, every
60s, which can now only ever fail
openbao rotation a Class-B CronJob rolling the pve-exporter@pve API token
against pve.apps.svc.cluster.local:8006
Removed:
apps/pve/ Service, EndpointSlice (10.2.40.201-203),
IngressRoute for the PVE web UIs
monitoring/pve-exporter-* Deployment and its config ExternalSecret
prometheus scrape configs job_name pve and pve-node
dashboards/proxmox-cluster.json and its ConfigMap generator entry
image-automation/pve-exporter image policy for an image nothing runs
openbao/rotation/pve-exporter the rotation CronJob
openbao/policies/rotator-proxmox.hcl
crowdsec pve-ui-polling-whitelist, which suppressed false bans on PVE UI
API polling that no longer happens
Kept, with comments corrected: the insecure-backend ServersTransport, still
used by apps/file for the NAS web UI.
The gatus check leaves a numbering gap at "Infrastructure 2" rather than
renumbering 3 and 4. The labels are deliberately opaque and renumbering would
relabel the surviving checks, discarding their history for cosmetics.
Not touched: the opentofu Proxmox provider, VM and LXC modules,
vm_152_k8s_3.tf, sync_state_from_pve.py, and the ansible PVE inventory. Those
carry state implications and want a deliberate pass rather than folding into a
manifest cleanup.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Draining k8s-1 for the jupiter conversion left three pods unschedulable:
k8s-2 and k8s-3 both sat at 100% of requested CPU while actually using 21-23%.
Cluster-wide the gap is 19,680m requested against 7,957m used.
crowdsec is the clearest single contributor. The chart defaults reserve 500m
for each agent and for the LAPI; measured draw is 1-2m. As a DaemonSet the
agent alone parks half a core on every node. Nothing here had ever set
resources, so this was an inherited default rather than a decision.
Only the request changes. Limits stay at the chart default, so a log flood can
still burst to 500m; what shrinks is the reservation the scheduler subtracts
from every node whether or not the process uses it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mTLS fix in #343 failed to apply: chart 0.24.0's agent/bouncer/appsec
Certificate templates emit `secretTemplate:` unconditionally while its
child blocks are guarded, so with nothing configured it rendered as null
and cert-manager's webhook rejected it (spec.secretTemplate must be of
type object). The helm upgrade failed and rolled back, so agents stayed
on the fragile register-based path.
Set tls.certManager.secretTemplate.labels so secretTemplate renders as a
valid object. Verified via kubectl apply --dry-run=server against the
live cert-manager webhook: all 4 Certificates now accepted.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The crowdsec-agent DaemonSet CrashLoops whenever the LAPI pod restarts.
Root cause: the LAPI has no persistence (PVCs disabled in 629a303d to
break the Ceph bootstrap circular-dependency), so its sqlite machine-
registration DB is ephemeral. Agents register once (init container) and
cache a password; a wiped LAPI no longer knows it -> 401 "machine not
found" -> CrashLoop -> DaemonSet never Ready -> the crowdsec HelmRelease
upgrade --wait times out -> the whole `infrastructure` Kustomization and
every app that depends on it stays unhealthy.
Enable chart mTLS (tls.enabled + tls.agent.tlsClientAuth, cert-manager
self-signed CA). With client-cert auth the agent no longer runs
`cscli lapi register`; the LAPI auto-recreates each machine record from
the presented cert on connect, so a wiped/restarted LAPI self-heals with
no PVC and no manual re-registration. Verified via `helm template`
against chart 0.24.0: register step gone, 4 certs emitted, zero PVCs.
Supersede the #337 persistence attempt (revert lapi.persistentVolume.
data.enabled to false): it reintroduced the Ceph circular-dependency and
did not heal the agents (fresh volume = empty DB). mTLS makes the
ephemeral DB harmless instead of trying to persist it.
Pin the chart to 0.24.x: the floating ">=0.12.0" silently upgraded to
0.24.0 (changed persistence defaults), part of this incident.
Note in traefik/crowdsec-middleware.yaml that the inert ingress bouncer
now needs https + CA trust before it can be re-enabled.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rotates crowdsec-bouncer-key:key quarterly + on-demand via the openbao-ns pattern (per-secret rotator-crowdsec-bouncer role/policy, no netpol). Self-sync: the single KV path feeds both the crowdsec + ingress ns Secrets; Reloader restarts crowdsec-lapi. Both ExternalSecrets refreshInterval 1h->10m.
CAVEAT handled: traefik/crowdsec-middleware.yaml has a THIRD hardcoded copy of this key (not Secret-sourced) — that middleware is inert, and a loud guard comment now warns it must be refreshed from OpenBao before re-enabling. Requires the one-time OpenBao bootstrap.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
bouncer-key.yaml committed the CrowdSec<->ingress-nginx bouncer key in
cleartext across two namespaces, outside the ESO/OpenBao pattern used
elsewhere in the cluster. Replace both Secrets with ExternalSecrets
pulling the same value from a single OpenBao KV path
(infrastructure/crowdsec-bouncer-key), matching the existing
apps/infrastructure ExternalSecret convention.
Requires the owner to populate OpenBao before merge:
bao kv put -mount=homelab infrastructure/crowdsec-bouncer-key key=<value>
Fixes#212
The agent's acquisition still targeted podName ingress-nginx-controller-*
in the ingress namespace — a selector that has matched no pod since
ingress-nginx was retired (2026-05-17), so the agents have been parsing
nothing and no decisions were being generated.
Repoint to the Traefik controller (podName traefik-*, program traefik)
and swap the nginx collection/parser for crowdsecurity/traefik (which
bundles its own log parser). This restores attack detection. Enforcement
(a Traefik bouncer acting on the decisions) is a separate change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Proxmox web UI polls /api2/json/cluster/* and per-guest status/current
endpoints every ~2s; CrowdSec's http-probing scenario flagged the
pattern as scanning and banned residential IPv6 clients via the
ingress-nginx Lua bouncer (bare 403 across every host).
Adds a parser-stage whitelist matching only those polling paths on
pve.renner.uno.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CrowdSec is in infrastructure stage which must complete before
Ceph cluster deploys. CrowdSec PVCs need Ceph. Disable PVCs
for now — CrowdSec state is ephemeral (bouncer decisions rebuild
quickly from the crowd).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace the fragile kubectl-exec Job with a simple static Secret
in both crowdsec and ingress namespaces. CrowdSec LAPI pre-registers
the bouncer via BOUNCER_KEY_ingress env var. The key is for internal
cluster communication only.
Also removes RBAC resources that were needed for the Job.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move CrowdSec to infrastructure stage with a Job that:
1. Waits for CrowdSec LAPI to be ready
2. Generates a bouncer API key via cscli
3. Creates a K8s Secret in the ingress namespace
ingress-nginx HelmRelease depends on CrowdSec, uses the CrowdSec
controller image, and reads the bouncer key from the provisioned
secret. No SealedSecrets or hardcoded keys needed.
Flow: CrowdSec deploys → Job creates bouncer key → ingress-nginx
starts with bouncer plugin authenticated to LAPI.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
claude-botandClaude Opus 5.5
issue-fix-bot
k.rennerandClaude Opus 4.8