From the update audit (#426), which flags this as the one item warranting
prompt action. Confirmed live: woodpecker-server runs v3.17.0 while the lint
step still pinned the v3.16.0 CLI.
That gap is the whole point of the step. The CLI compiles pipeline configs
against its own schema, so one behind the server can pass a config the server
then rejects — the gate stops predicting what the server will do, which is the
only thing it exists to do.
Verified the bump does not trade one problem for another: pulled
v3.17.0-alpine and ran the step's own lint loop over all 18 files in
.woodpecker/. Every one passes, so v3.17's schema does not reject anything
currently committed.
The drift is structural, not an oversight, and the comment now says so instead
of implying the two versions track each other. The server is a Kubernetes
manifest and image-automation bumps it; this pipeline file is plain YAML with
no apiVersion/kind, which Flux cannot patch (fluxcd/flux2#2826, recorded in
infrastructure/image-automation/README.md). So the server always moves first
and this always trails until someone bumps it by hand. Caught here by the
audit rather than by CI, which is the arrangement working as designed.
Also drops two stale "v3.16.0" references in the surrounding commentary that
would have aged into the same confusion.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Closes#433.
Deletes the four apps-namespace manifests, promotes the migration-* set onto
their names, and gives the surviving Cluster the prune: disabled data guard it
was deliberately denied while a rollback still existed. After this the
kustomization renders nothing into `apps` at all.
Merging this deletes nothing. The old resources carry prune: disabled, so
dropping them from git only stops Flux managing them — they keep running and
keep their PVC. The teardown is a separate, deliberate act, and the runbook
carries the commands plus the precondition to check first.
That precondition is already satisfied: the new cluster took its own base
backup to the wp-pg-woodpecker prefix (wp-pg-daily-20260818205831, completed)
and reports Initialized, ConsistentSystemID, Ready, ContinuousArchiving and
LastBackupSucceeded all True. Recovery no longer depends on the retired
cluster's data.
Two things kept rather than cleaned up, both deliberate:
- bootstrap.recovery and its externalCluster stay on the Cluster. CNPG honours
bootstrap only when PGDATA is absent, so it is inert on a live cluster — and
keeping it means a cold rebuild restores from backup instead of coming up
empty, which is the gap RECOVERY.md exists to close for the other clusters.
- wp-pg-store-source stays, pointing read-only at the old prefix. Nothing
writes to it, and it is a second independent restore point predating the
move — worth having until the new prefix builds real backup history. It is
also the actual safety net for the teardown, rather than the running cluster.
The hetzner-creds comment now records the openbao ClusterSecretStore allowlist
requirement that cost stage 1 a failed restore (#505), so the next namespace
move does not rediscover it.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
The stage-1 verification section said row counts "will not match exactly if
builds ran after the backup the restore used — that is expected, and stage 2
quiesces the server before the final cutover".
Stage 2 does not quiesce. Flux prunes the apps workloads and creates the
woodpecker ones; nothing stops writes to the old database in between. The
sentence described an intent that was never implemented, which is worse than
saying nothing.
Measured on the live cutover rather than estimated: the restore captured
through pipeline 9488; 9491, 9492, 9494 and 9496 landed in apps/wp-pg before
the swap; the new cluster resumed at 9521. Four pipeline records and their log
entries exist only in the old cluster. No ID collision and no functional
breakage — cluster.yaml itself calls this database small and non-critical with
largely reconstructable state — but it is real history loss, and it should be a
decision rather than something discovered later from a row count.
Documents how to avoid it next time (suspend the KS, scale the old server and
agent to zero, re-run the restore) and notes the rows stay recoverable while
the old cluster keeps its prune guard, which is until stage 3.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Follow-up to #506, which merged before this landed. Addresses the review's
stale-comment suggestion, and a real over-grant found while checking it.
The comment above woodpecker-allow-rotators still explained the rule in terms
of the `apps` baseline and allow-ingress.yaml. Chasing that down showed
allow-ingress.yaml is hard-scoped to `namespace: apps` and matches traefik by
label — so it does not apply in the new namespace, and the traefik admit I put
in the woodpecker baseline instead was broader than the thing it replaced:
endpointSelector:{} plus a whole-namespace source, where the `ingress`
namespace also holds netbird-ingress. That let those pods reach wp-pg and the
agent, not just the server.
Replaced with woodpecker-allow-traefik, scoped to traefik and to the server
pod, matching how the rotator and CI-step rules are already written.
It lives in network-policies/woodpecker-ns.yaml rather than beside the other
workload policies, deliberately: the baseline admit is withdrawn in the same
file, so both halves ship in the infrastructure-crds Kustomization and swap in
one reconcile. Split across Kustomizations they would not — infrastructure-crds
reconciles before the `woodpecker` KS, so the admit would be withdrawn minutes
before its replacement arrived and ci.renner.uno would 503 in the gap.
Confirmed safe against certificate renewal: letsencrypt-cluster solves DNS-01
through Cloudflare, so there is no in-cluster ACME solver pod needing ingress.
With an HTTP-01 issuer this scoping would have broken renewals silently, 90
days later.
Runbook: records that the stage-2 outage outlasts pods becoming Ready, because
certificates are namespace-scoped and cert-manager must reissue woodpecker-tls.
Observed during the live cutover as a TLS SNI failure with no HTTP status at
all, which reads like a routing fault and is not one.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Addresses #433. This is the downtime window: Flux prunes the apps copies and
creates the woodpecker ones, so ci.renner.uno is unavailable in between and
in-flight pipelines die with their agent.
Moved into `woodpecker`: server, agent, service, ingress, pdb, the agent
ServiceAccount, the woodpecker-secret ExternalSecret, and the rotator ingress
policy. The server's DSN now points at wp-pg-rw.woodpecker.svc and the agent's
gRPC at woodpecker.woodpecker.svc:9000.
Stage 1's restore is verified before doing this, not assumed: the new cluster
reports Ready=True and ConsistentSystemID=True (both False on the apps copy for
three weeks), its row counts match the live database exactly — 6 repos, 6125
pipelines, 2 users, 659 MB — and cnpg_collector_up now has a woodpecker series
for the first time.
Cross-namespace edges that had to move with it:
- rbac.yaml's RoleBinding subject. The Role and RoleBinding stay in
woodpecker-ci, because that is where they grant pod create/delete; only the
subject namespace changes.
- The openbao rotators' egress, which targeted `namespace: apps` + name
woodpecker on :8000.
New: woodpecker-allow-ci-steps grants woodpecker-ci → server :8000, scoped to
the SERVER pod. Stage 1 left this out on purpose after review found the same
rule on the namespace baseline would have handed arbitrary build steps a route
to wp-pg, since the CNPG instance manager serves its admin API on :8000 too.
Post-stage-1 the datapath confirms step-pod identities reach wp-pg on ICMP types
only; this rule must not change that.
Deliberately NOT touched: cluster.yaml, objectstore.yaml, scheduled-backup.yaml
and hetzner-creds-externalsecret.yaml still describe the original apps cluster.
They are the rollback, and they keep their prune guards. Stage 3 retires them
and renames the migration-* set onto their names.
Also corrects the runbook. It told the next person to verify policy semantics
with `cilium-dbg policy trace`, which does not exist in Cilium 1.20 — the
subcommand was removed, leaving only get/selectors/subject-selectors/wait.
Replaced with reading the realized BPF policy map via `bpf policy get
<endpoint-id>`, which is better evidence anyway: it is the datapath as
programmed rather than a simulation, and it carries live byte/packet counters.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Stage 1 of #433 is failing in the cluster. The wp-pg restore job errors
with:
can't get credentials for cluster wp-pg: while getting secret
hetzner-s3-creds: secrets "hetzner-s3-creds" not found
and the ExternalSecret behind it reports SecretSyncedError:
using cluster store "openbao" is not allowed from namespace
"woodpecker": denied by spec.condition
The ClusterSecretStore carries an explicit namespace allowlist, and the new
namespace was not on it. Its own header says so — "adding an ExternalSecret
in a NEW namespace requires adding that namespace here first, or ESO
reports 'not allowed to reference ClusterSecretStore' and the secret never
syncs" — so the requirement was documented and simply missed.
The reference sweep for #504 looked for everything that referenced
woodpecker or resolved *.apps.svc. It never asked the other question a
namespace move raises: what allowlists must the NEW namespace join? This
allowlist is exactly that shape, and it is the kind of thing kustomize
build and dry-run cannot surface, because the manifests are all valid — the
denial happens at ESO's admission of a cross-namespace reference.
Added in sorted position, keeping the list's stated ordering.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
From the fourth review on #504 — no blocking issues, but this one is a real
gap rather than a style point.
The woodpecker baseline was the only one of the repo's namespace policies
without an ICMP allow. That matters here specifically: wp-pg-allow-cnpg-operator
attaches an L4 rule (toPorts :8000) to the wp-pg pods, and Cilium blocks ICMP
to an endpoint once any layer 4 policy selects it unless the traffic is
CT-related. So PMTUD Destination Unreachable and Packet Too Big toward wp-pg
would have been dropped silently, surfacing as stalled transfers rather than
as a policy denial.
Uses the 16-type set the other baselines now carry after #495 trimmed them
from 39, rather than reintroducing the old list.
Granting this namespace-wide does not contradict the caution comment above it.
The :8000 case had to be per-workload because that port is shared between the
woodpecker server and the CNPG instance manager's admin API, so a blanket rule
handed CI step pods a route to the database. ICMP echo and path-MTU messages
expose no API surface, so a namespace-wide grant concedes nothing a scoped one
would withhold — which is why every other baseline does it this way too.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Addresses the third blocking review finding on #504.
The PR claimed the namespace move fixes wp-pg's absent metrics. It does not,
on its own. Prometheus requires the `release: kube-prometheus-stack` label
that CNPG's own enablePodMonitor omits, so every scraped cluster has a
hand-written PodMonitor in apps/monitoring-rules/cnpg-podmonitors.yaml —
ak-pg, im-pg, gf-pg, pprl-pg. There has never been one for wp-pg.
So wp-pg was missing both halves: no scrape target configured, and no
monitoring ingress allow in the apps baseline. Fixing the policy half was
necessary but not sufficient, and the runbook's cnpg_collector_up check
would have returned empty either way — unable to confirm the thing it
existed to confirm. Adds the PodMonitor in the woodpecker namespace.
Also from the review:
- Added a ScheduledBackup for the restored cluster. Every other CNPG
cluster here pairs isWALArchiver with one. Without it the new
wp-pg-woodpecker prefix takes continuous WAL but never a base backup, so
it has no independent recovery point and wp-pg-store's 30d retention has
nothing to prune against — the WAL chain grows unbounded for as long as
stage 1 sits before cutover. Scheduled 02:40, clear of the apps cluster's
02:35 and the other clusters' 02:30.
- Documented that the `woodpecker` KS goes not-Ready while the restore runs.
Its healthCheckExprs matches Cluster by GVK with no name qualifier, so
Flux evaluates it against both the live apps/wp-pg and the restoring
woodpecker/wp-pg, and wait: true makes the KS that owns production
Woodpecker report unhealthy. Expected, does not cascade since nothing
dependsOn it, but it would read as an outage without a note.
- Dropped the app.kubernetes.io/name label from the new Namespace; no other
namespace in that file carries one.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Addresses the second blocking review finding on #504.
bootstrap.recovery set only `source`. The live CRD documents the defaults:
`database` defaults to `app`, and `owner` defaults to whatever `database`
is. Neither name exists in the physically restored data, which carries
woodpecker/woodpecker from the backup.
So CNPG would have restored the real data, then provisioned a separate
empty `app` database and pointed the generated wp-pg-app Secret and the
-app Service at that instead — a cluster reporting healthy, holding the
right bytes, with the credentials aimed at the wrong database. The runbook's
own verification (psql -U woodpecker -d woodpecker using the wp-pg-app
password) would not have authenticated, so the failure would have surfaced
as a confusing auth error rather than as what it was.
kubernetes/RECOVERY.md sets both fields on all three recovery examples it
documents (ak-pg, im-pg, pprl-pg) for exactly this reason; this one did not
follow the pattern.
Added a runbook step to read dbname/username back out of the wp-pg-app
Secret before the row-count checks, so the mismatch is caught directly
rather than inferred from a psql failure.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Addresses the blocking review finding on #504.
The "step pods call the server back on :8000" rule was attached to the
namespace baseline, which uses endpointSelector: {}. A toPorts block
narrows the port, not the destination workload — that is governed solely by
the enclosing policy's endpointSelector. So the rule granted woodpecker-ci
access to :8000 on every pod in the namespace, and the CNPG instance
manager serves its status API on :8000 (server.yaml:94 uses the same port).
Arbitrary repo-supplied build steps would have had a route to wp-pg's
instance manager.
That is the same endpointSelector:{}-means-blanket-grant failure this PR
exists to escape, reproduced inside the fix.
Removed rather than rescoped: there is no woodpecker server in this
namespace until stage 2, so the rule was premature as well as wrong. It
lands there as its own policy selecting the server pod, the way
woodpecker-allow-rotators already scopes its :8000 rule.
The baseline now carries no port-scoped rules at all, so nothing in it can
be mistaken for workload scoping, and a comment warns why that must stay
true.
Also from the review:
- Corrected the KS attribution. The network policies ship with
infrastructure-crds (infrastructure/post-crds includes
../network-policies/), not the top-level infrastructure KS. The ordering
conclusion is unchanged — infrastructure-crds dependsOn infrastructure,
and woodpecker dependsOn both — but the comments named the wrong one.
- Documented why the baseline has no egress section, instead of leaving it
to read as an oversight. Writing that ruleset in stage 2, with the server
and agent present, beats guessing it now and breaking the WAL archiving
this stage exists to verify.
- Dropped the monitoring -> :9187 rule from wp-pg-allow-cnpg-operator. The
baseline already allows the monitoring namespace unrestricted, so it added
no coverage while implying scoping it did not provide.
- Added a cilium-dbg policy trace step to the runbook. dry-run validates
manifest shape, not Cilium semantics, and would not have caught this bug.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Addresses #433. No cutover, no CI impact: this stands up the `woodpecker`
namespace and restores wp-pg into it from barman, running in parallel with
the live apps/wp-pg. Nothing points at it yet.
Why a namespace move rather than a policy fix. wp-pg has been Ready=False
with "Instance Status Extraction Error" for three weeks because the CNPG
operator in `database` cannot reach https://<pod-ip>:8000/pg/status. The
apps default-deny-ingress uses endpointSelector:{}, so the only
namespace-level remedy is to allow the whole database namespace in — which
would expose gitea, gatus, external-dns and ical-proxy along with wp-pg, to
fix one pod on one port. The same gap explains why cnpg_collector_up has
series for authentik, immich, monitoring and paperless but not wp-pg: apps
never allowed the monitoring namespace either.
woodpecker-ci is deliberately not the target. It enforces pod-security:
privileged and exists for ephemeral step pods running repo-supplied code;
the database has no business sharing it.
Two ObjectStores, deliberately. wp-pg-store-source reads the existing
s3://renner-uno-k8s-cnpg/wp-pg prefix; wp-pg-store archives to a new
wp-pg-woodpecker prefix. Two clusters writing one destinationPath would
interleave their WAL streams and corrupt both, and splitting them keeps the
pre-migration backup intact as rollback.
The Namespace is declared in infrastructure/namespaces rather than beside
these workloads: its network policies ship with the `infrastructure` KS,
which the `woodpecker` KS dependsOn, so creating it in the workload KS would
invert that order and deadlock a cold rebuild.
The restored cluster deliberately carries no prune: disabled guard. The
original has one because it holds the only copy of the data; this one is
reconstructable, and a prune guard on a migration artifact is how orphans
outlive an abandoned migration. It gains the guard in stage 3.
Barman creds still resolve from seed/paperless/hetzner-creds. #397 wants a
woodpecker-owned path and this was the natural moment, but creating it needs
OpenBao write access that is not available here (bao kv list seed -> 403).
Pointing at a path that may not exist would leave the restored cluster
unable to archive WAL, which is what this stage exists to prove.
Runbook, including how to verify the restore holds real data rather than
merely coming up healthy: MIGRATION-namespace.md.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Addresses #434.
Ceph's 30% default assumes the mon owns its filesystem, where 30% free is
a real reserve. Our mons sit on dataDirHostPath /var/lib/rook — the node's
/var, shared with ~29 GB of containerd storage — and the mon DB is ~106 MB.
The default therefore measures container churn rather than Ceph state, and
k8s-2 trips MON_DISK_LOW whenever its image cache fills.
Tuning image GC does not help: measured on k8s-2, dropping the GC high
threshold to 60 deleted 28 images and moved free space by 0.02 percentage
points, because most of /var/lib/containerd is overlayfs snapshots held by
running containers, which image GC cannot reclaim.
20% of a 60 GB /var still leaves ~12 GB behind a 106 MB database.
mon_data_avail_crit is untouched at 5%, and CephMonDiskLow watches /var
directly for the real risk — the node filling, not the mon growing.
Set through cephConfig so it is reconciled from git rather than applied as
a live `ceph config set`.
The comment records that this override should be revisited, and dropped,
when mons get their own volume at bare-metal migration. That is not
possible on the current cluster: every StorageClass is Ceph-backed, so a
mon PVC would be circular, and Rook applies mon.volumeClaimTemplate only
to newly created mons.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
The 60/50 change in #500 was based on a wrong premise. Reverting the
values and correcting the reasoning recorded around them.
Applied to k8s-2 alone and measured. GC did fire: 28 images deleted,
content store 9.93 -> 8.62 GB. Free space moved 31.49% -> 31.47% — no
effect.
The premise was that ~19 GB of overlayfs snapshots would be reclaimed
along with their images, at roughly 2.9x the compressed size. That is
wrong. /var/lib/containerd is 20.5 GB of snapshots against 8.6 GB of
content, and those snapshots are unpacked layers shared with running
containers. Image GC cannot touch them, so deleting unused image
references frees only their unshared blobs. Remaining reclaimable
content is now ~0.7 GB; further sweeps free nothing.
70/60 stands on its own merits as a bound on the image cache — it is
simply not a lever on MON_DISK_LOW, and the comment no longer claims it
is. The mon's own data is ~106 MB, so a 30%-free threshold on a 60 GB
filesystem it merely shares says nothing about Ceph.
The CephMonDiskLow runbook text carried the same wrong claim and is
corrected to point at finding the real consumer instead.
k8s-2's live kubelet is already back to 70/60 via talosctl patch; node
Ready, Ceph HEALTH_OK with mon quorum a,b,c intact. k8s-1 and k8s-3 were
never changed.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Addresses #434.
Gap 2 — no admin path. There is still no rook-ceph-tools deployment, so
ceph status, ceph health detail, ceph osd df tree, ceph osd perf and ceph
device ls are all unreachable during an incident: mons, OSDs and mgr
mount no admin keyring and the operator's /etc/ceph is empty. Adds the
upstream toolbox from rook v1.20.4 (matching the running operator image)
with the image pinned to the cluster's own quay.io/ceph/ceph:v19.2.5 and
carrying the same imagepolicy marker as cephcluster.yaml, so the client
cannot drift ahead of the mons. Prerequisites verified present: the
rook-ceph-default ServiceAccount, the rook-ceph-mon-endpoints ConfigMap,
and ceph-secret/ceph-username in the rook-ceph-mon secret.
Image GC. The thresholds were already lowered once from the kubelet
defaults, to 70/60, explicitly to stay under Ceph's mon_data_avail_warn.
But that line is <=30% free and a high of 70 fires at exactly 30% free —
on the threshold, not below it. So the cache fills to precisely the point
where the mon warns, and MON_DISK_LOW trips before kubelet reclaims
anything. Observed on k8s-2: 29.81% free and HEALTH_WARN MON_DISK_LOW,
clearing to 31.64% once GC ran. 60/50 fires at 40% free and reclaims to
50%, leaving 10 points of margin.
There is plenty to reclaim, so the lower threshold will bite: containerd
holds ~29 GB of k8s-2's 60 GB /var and caches 179 images against 60 in
use, some dating to May. (The ~309 GB talosctl attributes to
/var/lib/kubelet is mounted Ceph PVCs counted through mount points, not
local disk.)
Alerting, the follow-up #434 parked until ceph_* series existed — they do
now, via #432. CephHealthWarn deliberately waits 6h: Ceph warns for
self-clearing conditions and paging on those teaches people to ignore it,
so what is worth waking for is a warning that does not clear.
CephHealthError fires at 5m. CephMonDiskLow watches /var free directly
rather than going through Ceph, because this fires for a full image cache
rather than for Ceph growth, and naming that keeps the next person from
debugging the wrong subsystem.
NOTE: cluster.yaml is a bootstrap patch. Merging changes nothing on
running nodes — the live kubelet value needs talosctl patch machineconfig.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
The earlier commits in this branch raised policyMapMax and added an alert.
Neither addresses why the map fills. This does.
Measured on k8s-1 with cilium-dbg: 8 endpoint policy maps sit at exactly
16384 entries — full — and the rest cluster at ~8,500. The split tracks
policy direction, so one direction costs ~8,500 entries. Dumping velero's
endpoint 2328 (8,499 entries, ingress only) shows 217 identities x 39
port/proto values, of which 38 are ICMP types. The map is ~99% ICMP, and
3,744 of its entries (44%) belong to woodpecker-ci-step identities.
Two multiplying causes, both fixed here.
The ICMP rule in every default-deny-ingress allowed 39 enumerated types
from `namespace: Exists` — every identity in the cluster. Cilium expands
that to one entry per (identity, type), per endpoint, per direction, in
19 namespaces. The rule's own comment records it was widened "from the
error class to the full standard type set"; that widening is the cost.
Trimmed to 16: the error types that motivated the rule (v4 3/11/12, v6
1/2/3/4 — PMTUD is the documented need), echo, and the IPv6 ND/RA/MLDv2
types observed in flight via Hubble (133, 135, 143). Dropped types are
deprecated (v4 4 Source Quench, RFC 6633), router-only (v4 5, v6 137),
obsolete (v4 13-18, 30), superseded (v6 130-132 by 143), or never
deployed (v6 138, 141/142, 148/149, 151-153). openbao-ns.yaml already
runs an error-only set of 7 in production, so 16 is the conservative side
of existing practice.
Cilium had no `labels` filter, so identity came from all pod labels.
Woodpecker stamps each step pod with woodpecker-ci.org/task-uuid and
step=<name>, making every build step its own security identity — 96 of
them minted in the last hour, against zero pods in the namespace. The GC
is healthy and simply outpaced. Excluding those two label patterns
collapses 78 current identities to one. No CiliumNetworkPolicy selects on
them, so they bought churn and nothing else.
Together these take a direction from ~8,500 entries to roughly 140
identities x 17 values ~= 2,400 — about 4,800 for both directions, well
inside even the old 16384 ceiling. policyMapMax stays raised as headroom,
not as the fix.
Per Cilium's docs, existing identities are not recomputed by a `labels`
change; new ones use the filter and the old ones age out through normal
GC, so the map drains over roughly an hour rather than instantly.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
renner.uno and www.renner.uno were pinned to 159.69.42.237, the VM that hosted
WordPress before it moved into the cluster. Neither record is live on the UDM
any more (verified: a LAN query for www.renner.uno returns the public answer,
proxy.renner.uno), so the file was drift waiting to be re-applied — a tofu
apply would have pointed the house at a host that is being decommissioned,
while the rest of the world already reaches the cluster.
Their absence is the correct end state, not an omission: www.renner.uno
carries dns.renner.uno/public, so external-dns-unifi writes no UDM record by
design and LAN clients take the public path and hairpin back in. The apex
stays with Cloudflare, which now redirects renner.uno/* to www.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014giKN9N6e5J2DF8QZmVh98
Two optional points from the second review on #495.
The PrometheusRule alerts on cilium_bpf_map_pressure without filtering on
map_name, so it covers conntrack, NAT and LB maps as well as the policy
map. That is intended — running out of any of them degrades the datapath
— but the PR is framed around the policy map, so a reader could take the
narrow reading. Say so in the file.
The cross-references between the three copies of bpf-policy-map-max were
one-directional: helmrelease.yaml and patches/cilium-cni.yaml each named
only one other site, while talos/cilium.yaml named both. Each now names
the other two.
Not taken: the suggestion to quote policyMapMax as "65536". The chart
declares it `type: [null, integer]` in its values schema
(helm show values cilium --version 1.20.0), so an integer is correct and
a quoted string would be the schema violation.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Addresses the blocking review finding on #495.
Raising bpf.policyMapMax only in the HelmRelease left the bootstrap path
on the old value. kubernetes/bootstrap.sh:126 passes
kubernetes/talos/patches/cilium-cni.yaml to talosctl apply-config for
every node, and that manifest hardcoded bpf-policy-map-max: "16384" — so
a cold rebuild would install Cilium with the map size this PR exists to
fix, and stay there until Flux reconciled the HelmRelease.
The sibling bpf-lb-map-max is "65536" in both files, matching the live
agent config, so keeping these in sync is the repo's existing practice.
kubernetes/talos/cilium.yaml carries the same key and is updated too.
Nothing in the repo references that file — only patches/cilium-cni.yaml
is used by bootstrap.sh — so it looks like an orphaned rendered snapshot.
Syncing it rather than deleting it: whether it should exist at all is a
separate question from this fix.
Each of the three sites now names the others, so the next change to this
value has the lockstep written down where someone will hit it.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Closes#451. Closes#454.
Both issues are the same fault: a pod's dedicated cilium_policy_v3_<id>
BPF map ran out of space, endpoint regeneration failed with "no space
left on device", the CNI ADD call never returned, and the pod sat in
sandbox creation. For paperless — replicas: 1 with strategy Recreate —
that was a 41-minute outage on 2026-07-30.
It is not over. Prometheus (metrics landed via #452) puts
cilium_bpf_map_pressure{map_name="cilium_policy_v3_*"} at 0.867 on all
three nodes right now, and max_over_time(...[30d]) at 1.03 — the map has
been driven past capacity. Loki has the overflow recurring as recently
as 2026-08-12 17:46Z, hitting authentik and esphome endpoints.
Three changes:
- bpf.policyMapMax 16384 (chart default) -> 65536, matching the cluster's
existing bpf-lb-map-max. Takes steady-state pressure from ~87% to ~22%.
The map is sized at creation, so this only takes effect as agents
restart.
- The chart range ">=1.19.0" is what let Flux take 1.19.6 -> 1.20.0
unattended 45 minutes before the first outage. Narrowed to
">=1.20.0 <1.21.0" so a CNI minor is a reviewed diff, not a surprise.
- A PrometheusRule on cilium_bpf_map_pressure: warning over 80% for 15m,
critical at 100%. The metric was already scraped but nothing alerted on
it, so the first signal was a downstream replica mismatch.
Not addressed: why the busiest endpoint needs ~14k entries. Only 178
CiliumIdentities exist and no CiliumNetworkPolicy declares more than 9
ports, so the entry count is not obviously explained by policy shape.
Establishing that needs `cilium-dbg bpf policy get` on an agent.
MERGING THIS ROLLS THE CNI. The DaemonSet uses
updateStrategy.rollingUpdate.maxUnavailable: 2, so two of three agents
restart at once.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Verified against the deployed cluster: GET /wp-admin/ correctly 302s to
https://www.renner.uno/outpost.goauthentik.io/start?rd=... — and that URL
returned WordPress's 404 page, because /outpost.goauthentik.io fell through to
the public '/' prefix router.
The effect was that /wp-admin was unreachable rather than protected. An
attacker gains nothing, but neither does the editor the gate exists for.
paperless needs the same route and solves it with a separate Ingress on
auth.renner.uno (apps/authentik/paperless-gate-ingress.yaml). Here the whole
flow stays on www.renner.uno, so the path belongs in this rule set.
These endpoints are the login flow itself and are served by the outpost, so
routing them to it is not an authentication hole.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014giKN9N6e5J2DF8QZmVh98
Closes#461.
Five pipeline steps pinned ci-k8s at two stale tags: e2e-test.yml,
e2e.yml, alert-monitor.yml and issue-fix.yml at 2026-05-01, and
ci-manifests-test.yml at 2026-07-30. 2026-08-10 is the newest dated tag
in gitea_admin/ci-k8s and shares a digest with :latest
(sha256:0019f264...), so all five now run the same image the build
pipeline last produced.
A stale pin eventually points at a garbage-collected tag, and the split
had no deliberate reason behind it.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
The first deploy returned 500 on every request:
PHP Fatal error: Uncaught Error: Call to undefined function add_filter()
in /var/www/html/wp-config.php(128) : eval()'d code
WORDPRESS_CONFIG_EXTRA is eval'd inside wp-config.php, which runs long before
wp-settings.php loads the plugin API. add_filter() does not exist yet at that
point, so the call was fatal on every request.
Removing it exposed the second, worse half: the comment justified the filter by
claiming xmlrpc.php was 'blocked at the ingress'. It was not. The public router
matches '/' as a prefix, so xmlrpc.php was served like any other path — the
claim was written without being checked.
An Ingress has no deny verb, so xmlrpc.php now routes to the Authentik gate
alongside /wp-admin and /wp-login.php: anonymous callers meet Authentik instead
of the XML-RPC endpoint. Nothing on this site uses it — no Jetpack, no mobile
app, no pingbacks worth keeping.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014giKN9N6e5J2DF8QZmVh98
Addresses the automated review on #492.
Blocking: two comments claimed the public/admin split was kept honest by
kubernetes/apps/wordpress/network-policy.yaml. That file does not exist — the
comment was written against the paperless template and never updated once the
namespace policy proved sufficient. The deeper point in the review is the real
one: a paperless-style workload-scoped policy would be a no-op here, because
traefik must reach the pod directly for every public page. The split is
enforced by the Ingress path rules and Authentik, not at the network layer,
and the comment now says so — an auditor looking for a network control would
otherwise have gone hunting for a file that was never there.
Velero: the schedule excluded the database PVC by app.kubernetes.io/instance,
a label the mariadb-operator was only assumed to set. A wrong assumption there
fails silently and sweeps the live database volume into the snapshot. Inverted
to select the content PVC by labels this repo sets on it, the same direction
openbao-daily already uses.
The Service-name assumption flagged as a suggestion is kept: the CRD's separate
primaryService/secondaryService fields indicate the base Service carries the
resource name, and a mismatch fails loudly at pod startup. Written down as a
first-deploy check rather than left implicit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014giKN9N6e5J2DF8QZmVh98
Migrates the Hetzner VM 'wordpress-hochzeit' into the cluster. The site is not
the frozen wedding archive it looks like: it carries an unpublished, fully
composed birth announcement from 2026-05-14, so wp-admin has to keep working
for a human author rather than being replaced by a Git workflow.
Split routing on one host: '/' goes straight to WordPress so the blog stays
publicly readable, while /wp-admin and /wp-login.php route through a dedicated
wordpress-gate outpost. The inverse -- gate the host and skip_path_regex the
public paths, as paperless-gate does -- would need a negative lookahead, and
the outpost's RE2 has none.
Gating all of /wp-admin is only safe because the dead wpgiftregistry plugin is
dropped in the migration: it was the sole emitter of ajaxurl on this site, and
of 121 admin-ajax hits in the current log window not one returned 200.
Core lives in the pinned image, not on the volume, so upgrading is a tag bump
in Git. Only wp-content is persisted.
Backups are split the same way CNPG/Barman and Velero are elsewhere: the
operator takes a logical dump to Hetzner Object Storage at 02:45, Velero takes
the wp-content PVC at 08:45 and skips the database volume -- a crash-consistent
snapshot of a running database is not a backup.
Group membership for the gate is owned by Git, as with the Home Assistant,
Shelly and zone-configurator gates. s.renner is the Authentik account for the
site's WordPress Editor, whose login there is still the pre-2019 s.birzer.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014giKN9N6e5J2DF8QZmVh98
WordPress speaks MySQL natively. The PostgreSQL route would mean pg4wp, an
SQL-rewriting shim whose README lists WordPress 6.5.3 as its newest tested
version while renner.uno runs 7.0.4 — every core update would become a
compatibility gamble.
Charts are OCI artifacts, so this follows the OCIRepository pattern from
spegel rather than the HelmRepository pattern used for CNPG. CRDs ship as a
separate chart that upstream requires first, hence the dependsOn. The version
range is capped at the year boundary: the project uses calendar versioning and
ships an upgrade guide with each minor.
renner-uno-k8s-mariadb carries the same 30d noncurrent-version expiry as the
CNPG and Velero buckets, matched to the Backup resource's maxRetention.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014giKN9N6e5J2DF8QZmVh98
The passkey apps were reachable only with a WebAuthn credential:
passkey-passwordless-authentication denies at its validation stage, and
passkey-stepup-authorization demands a passkey on every authorize. An
account with no passkey had no way in.
Branch both flows on enrollment with an expression policy,
passkey-user-has-webauthn. Enrolled accounts keep the mandatory passkey
at login and at every authorize. Accounts without one skip the WebAuthn
stage and get default-authentication-password followed by
default-authentication-mfa-validation — the same pair, in the same
order, that default-authentication-flow already runs, so the fallback is
not a weaker way in than auth.renner.uno's own login.
The policy runs against pending_user, so the branch follows the
identified account's enrollment and is not selectable by the requester.
Bindings keep evaluate_on_plan false and re_evaluate_policies true;
on-plan evaluation would run before a user is resolved.
The negated bindings carry failure_result: true. UserLoginStage defaults
a missing authentication backend to BACKEND_INBUILT rather than
refusing, so a flow that dropped both branches would log the user in on
a username alone. A PolicyException resolves to failure_result before
negate is applied, so a broken expression degrades to password + MFA
instead of to no factor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every fresh login to a passkey app costs two touches: one in
passkey-passwordless-authentication, a second immediately after in
passkey-stepup-authorization.
last_auth_threshold is enforced with a single browser cookie
(authentik_mfa) whose JWT is signed with, and carries, the validation
stage's PK. A stage only honours a cookie it wrote itself, and both
stages write to the same cookie name — so two stage objects each demand
their own touch and clobber each other's window.
Collapse them into one passkey-webauthn-validation stage bound into both
flows. not_configured_action is deny, since the authentication flow
reaches the stage with the user only identified and no password stage in
front of it. Passkey-less users enroll from user settings after a
password login; default-authenticator-webauthn-setup has a
configure_flow, so that path is live.
The 8h cookie now also covers the login flow, so a re-login in the same
browser inside the window is satisfied by the cookie alone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
:latest never rolls. The tag moves when a build publishes, but nothing
tells the Deployment: the pod template is unchanged, so Kubernetes has
no reason to replace anything.
Nothing else covers it either. This image is deliberately not enrolled in
Flux image automation — kubernetes/infrastructure/image-automation/README.md
lists ghcr.io/konairius/ical-proxy as having "only main, latest, and
main-<sha> commit builds; no release tags", so no ImagePolicy watches it.
The two pods have been running a 21-day-old image while :latest moved
underneath them, and the only way to pick up a new build was a manual
restart, which leaves no record anywhere.
Pinning makes the rollout a git change. Flux applies the new pod
template, Kubernetes rolls it, and what is RUNNING becomes recoverable
from this repo instead of "whatever :latest meant the last time a pod
happened to restart".
main-99a912b is the merge of konairius/ical-proxy#12, which adds the
summary_match and summary_replace parameters. The waste calendar needs
them to strip "| Abfuhrkalender - Landkreis Amberg-Sulzbach" from every
collection — the feed repeats the whole calendar's name on each event,
and no consumer can shorten it after the fact.
Bumping this tag is now a manual, reviewed edit. Format is
main-<short-sha>, from docker/metadata-action type=sha with the branch
as prefix.
k.renner
issue-fix-bot
claude-bot
security-bot