fix(unifi): capture the API response on create only, not on every write

Root cause of the 60 permanently-dirty hairpin NAT rules.

The provider was configured write_returns_object = true, which makes it store
the API response into `data` after POST *and* PUT. The UDM assigns `_id`;
unifi_nat_hairpin.tf's data = jsonencode({...}) never sets it. So every write
put a field into state that the config does not have, and all 60 rules planned
as "will be updated in-place" with `- _id` as the entire diff — re-dirtying
themselves on the next write, indefinitely.

No provider-side ignore can suppress this, because `data` is a configured
attribute and Terraform compares config to state in core, before provider
logic runs. Three were tried and disproven against real plans rather than
argued about:

  ignore_changes_to = ["_id"]                       plan 5661 — 60 remained
  ignore_all_server_changes = true                  plan 5662 — 60 remained
  restapi 3.x + ignore_server_additions = true      plan 5667 — 60 remained,
                                                    plus a new api_data diff

create_returns_object keeps the part the provider genuinely needs — the POST
response is how it learns `_id` for id_attribute = "_id" — while no longer
letting updates rewrite `data` from the response.

This does not clear the existing pollution by itself: state still carries `_id`
in `data` for all 60 rules, so the plan will keep showing them until one apply
rewrites `data` to the configured value. What it changes is that the apply then
sticks, instead of the next write undoing it.

Co-Authored-By: Claude Bot <claude-bot@renner.uno>

.woodpecker/ci-manifests-test.yml

@@ -115,7 +115,7 @@ steps:
   # An earlier revision of this comment also asserted that several deploy-*.yml
   # files emit "Additional property timeout is not allowed" warnings. That was
   # observed under the v3.9.0 CLI this step used to pin, and it is NOT known to
-  # still hold on v3.16.0 — review raised the likelihood that the warning was
+  # still hold on current CLIs — review raised the likelihood that the warning was
   # schema drift in a 7-minor-old CLI rather than real debt in those files, which
   # is one of the reasons this pin moved to latest stable. It cannot be confirmed
   # from a green run either, because the loop below sends lint output to /tmp/out
@@ -123,9 +123,15 @@ steps:
   # The claim is dropped rather than restated: what this step guarantees is the
   # exit-code contract above, which is what the gate actually rests on.
   # Latest stable, per the standing rule that any software uses the latest
-  # stable version. Matching the deployed woodpecker-server (v3.16.0) is a
-  # CONSEQUENCE of that, not a separate requirement — the server is bumped by
-  # image-automation within the 3.x range, so both land on the same release.
+  # stable version. Matching the deployed woodpecker-server is a CONSEQUENCE of
+  # that, not a separate requirement — the server is bumped by image-automation
+  # within the 3.x range, so both should land on the same release.
+  #
+  # They do not land together, though: the server is a Kubernetes manifest and
+  # is bumped automatically, this pipeline file is not and cannot be (see
+  # infrastructure/image-automation/README.md). So the server moves first and
+  # this trails until someone bumps it by hand — which is what happened at
+  # v3.16.0 -> v3.17.0, caught by the update audit (#426) rather than by CI.
   #
   # It matters here because the CLI compiles the config against its own schema:
   # one behind the server can pass a config the server then rejects, or reject
@@ -152,7 +158,7 @@ steps:
   # whenever it trails the deployed server. If you bump the server, bump here.
   - name: woodpecker-lint
     depends_on: [] # see note above `steps:`
-    image: woodpeckerci/woodpecker-cli:v3.16.0-alpine
+    image: woodpeckerci/woodpecker-cli:v3.17.0-alpine
     commands:
       - |
         set -eu

kubernetes/apps/monitoring-rules/cnpg-podmonitors.yaml

@@ -73,3 +73,29 @@ spec:
       interval: 30s
   podTargetLabels:
     - role
+---
+# wp-pg, restored into the `woodpecker` namespace (issue #433). It has never
+# been scraped: `apps` had no PodMonitor for it AND its baseline network policy
+# never allowed the monitoring namespace, so `cnpg_collector_up` has series for
+# the four clusters above but not this one. The namespace move fixes the policy
+# half; without this PodMonitor there is still no scrape target at all.
+#
+# Deliberately not adding one for the cluster still in `apps`: it is retired in
+# stage 3, and a PodMonitor there would only be reconciled to be deleted again.
+apiVersion: monitoring.coreos.com/v1
+kind: PodMonitor
+metadata:
+  name: cnpg-wp-pg
+  namespace: woodpecker
+  labels:
+    app.kubernetes.io/name: cloudnative-pg
+    release: kube-prometheus-stack
+spec:
+  selector:
+    matchLabels:
+      cnpg.io/cluster: wp-pg
+  podMetricsEndpoints:
+    - port: metrics
+      interval: 30s
+  podTargetLabels:
+    - role

kubernetes/apps/monitoring-rules/prometheusrule-ceph.yaml

@@ -37,11 +37,14 @@ spec:
 
         - alert: CephMonDiskLow
           # Named explicitly rather than left to CephHealthWarn, because the
-          # mons sit on the node's /var alongside containerd's image cache:
-          # this fires for a full image cache, not for Ceph growth. Cleared by
-          # kubelet image GC (imageGCHighThresholdPercent in
-          # kubernetes/talos/patches/cluster.yaml), which must stay below
-          # Ceph's mon_data_avail_warn of 30% free.
+          # mons sit on the node's /var alongside container storage: this
+          # fires for a full node, not for Ceph growth. The mon's own data is
+          # ~106 MB, so 30% of a 60 GB shared filesystem says nothing about
+          # Ceph.
+          # Image GC does not clear this — measured on k8s-2, the reclaimable
+          # content store is a small fraction of /var/lib/containerd next to
+          # ~20 GB of overlayfs snapshots held by running containers. Look at
+          # what actually grew before reaching for a threshold.
           expr: |
             100 * node_filesystem_avail_bytes{mountpoint="/var"}
               / node_filesystem_size_bytes{mountpoint="/var"} < 30
@@ -50,4 +53,4 @@ spec:
             severity: warning
           annotations:
             summary: "{{ $labels.instance }} /var below 30% free — Ceph mon will warn"
-            description: "/var on {{ $labels.instance }} is under 30% free, which is Ceph's mon_data_avail_warn threshold, and the mon on this node reports MON_DISK_LOW below it. Usually containerd's image cache; check whether kubelet image GC is reclaiming with `talosctl -n <node> usage /var/lib --depth 1`."
+            description: "/var on {{ $labels.instance }} is under 30% free, which is Ceph's mon_data_avail_warn threshold, so the mon on this node will report MON_DISK_LOW. Find the actual consumer with `talosctl -n <node> usage /var/lib --depth 1` and then `/var/lib/containerd --depth 1` — note that overlayfs snapshots belong to running containers and are not reclaimable by image GC."

kubernetes/apps/monitoring-rules/prometheusrule-woodpecker-rotation.yaml

@@ -12,37 +12,37 @@ spec:
   groups:
     - name: woodpecker-rotation.rules
       # Convergence safety net for the automated WOODPECKER_AGENT_SECRET rotation
-      # (kubernetes/apps/woodpecker/rotate-agent-secret.yaml). That rotation Job is
+      # (kubernetes/infrastructure/openbao/rotation/woodpecker.yaml). That rotation Job is
       # fire-and-forget: it writes OpenBao and exits WITHOUT verifying ESO re-synced
       # or that both consumers restarted healthy. These alerts close that gap. ESO
       # metrics are not scraped in this cluster, so everything here is built on
-      # kube-state-metrics. The CronJob/Job series stay empty until the rotation
-      # feature (branch feat/woodpecker-secret-auto-rotation) is merged — harmless.
+      # kube-state-metrics. The rotation has since landed: the CronJob lives in the
+      # openbao namespace, not apps, which is where the last two rules look.
       rules:
         - alert: WoodpeckerServerNotReady
           # 0 ready replicas well past the 10m ESO refresh + Reloader restart cycle.
-          expr: kube_statefulset_status_replicas_ready{namespace="apps",statefulset="woodpecker-server"} == 0
+          expr: kube_deployment_status_replicas_ready{namespace="woodpecker",deployment="woodpecker-server"} == 0
           for: 20m
           labels:
             severity: critical
           annotations:
             summary: "Woodpecker CI server has no ready replica"
             description: >-
-              StatefulSet woodpecker-server (ns apps) has had 0 ready replicas for 20m;
-              all CI is down (agents cannot reach woodpecker.apps.svc:9000). If this
-              followed a secret rotation, the server pod failed to return after its
-              Reloader-triggered restart — check `kubectl -n apps describe pod
-              woodpecker-server-0` (PVC attach / crashloop) and the woodpecker-secret
-              ExternalSecret sync status.
+              Deployment woodpecker-server (ns woodpecker) has had 0 ready replicas
+              for 20m; all CI is down (agents cannot reach
+              woodpecker.woodpecker.svc:9000). If this followed a secret rotation, the
+              server pod failed to return after its Reloader-triggered restart — check
+              `kubectl -n woodpecker describe deploy woodpecker-server` and the
+              woodpecker-secret ExternalSecret sync status.
         - alert: WoodpeckerAgentNotReady
-          expr: kube_deployment_status_replicas_available{namespace="apps",deployment="woodpecker-agent"} == 0
+          expr: kube_deployment_status_replicas_available{namespace="woodpecker",deployment="woodpecker-agent"} == 0
           for: 20m
           labels:
             severity: warning
           annotations:
             summary: "Woodpecker CI agent has no available replica"
             description: >-
-              Deployment woodpecker-agent (ns apps) has had 0 available replicas for
+              Deployment woodpecker-agent (ns woodpecker) has had 0 available replicas for
               20m; no pipelines will run. If this followed a secret rotation, the agent
               did not return after its Reloader restart, or its WOODPECKER_AGENT_SECRET
               no longer matches the server's — check the agent pod logs for a gRPC auth
@@ -50,29 +50,29 @@ spec:
         - alert: WoodpeckerSecretRotationJobFailed
           # Named, clearer signal than the generic KubeJobFailed for the OpenBao-write
           # step. Fires while a failed rotation Job is retained (ttl 3d).
-          expr: kube_job_status_failed{namespace="apps",job_name=~"woodpecker-secret-rotate.*"} > 0
+          expr: kube_job_status_failed{namespace="openbao",job_name=~"woodpecker-secret-rotate.*"} > 0
           for: 15m
           labels:
             severity: warning
           annotations:
             summary: "Woodpecker agent-secret rotation Job failed"
             description: >-
-              A woodpecker-secret-rotate Job (ns apps) failed at the OpenBao-write step;
+              A woodpecker-secret-rotate Job (ns openbao) failed at the OpenBao-write step;
               the agent secret was NOT rotated. Most common cause: the one-time OpenBao
               `rotator` role/policy bootstrap was never applied (see
               kubernetes/infrastructure/openbao/policies/rotator.hcl). Check
-              `kubectl -n apps logs job/<name>`.
+              `kubectl -n openbao logs job/<name>`.
         - alert: WoodpeckerSecretRotationStale
           # Quarterly cadence (~91d) + grace. The series is ABSENT until the first
           # successful run, so this cannot false-fire before rotation has ever worked.
-          expr: (time() - kube_cronjob_status_last_successful_time{namespace="apps",cronjob="woodpecker-secret-rotate"}) > 100 * 24 * 3600
+          expr: (time() - kube_cronjob_status_last_successful_time{namespace="openbao",cronjob="woodpecker-secret-rotate"}) > 100 * 24 * 3600
           for: 1h
           labels:
             severity: warning
           annotations:
             summary: "Woodpecker agent secret has not rotated in >100 days"
             description: >-
-              The quarterly woodpecker-secret-rotate CronJob (ns apps) has not completed
+              The quarterly woodpecker-secret-rotate CronJob (ns openbao) has not completed
               successfully in over 100 days — rotation has silently stopped (CronJob
               suspended/deleted, or every run failing). Investigate the CronJob and its
               recent Jobs.

kubernetes/apps/woodpecker/MIGRATION-namespace.md

@@ -0,0 +1,388 @@
+# Woodpecker: `apps` → `woodpecker` namespace migration
+
+Moves the Woodpecker service — server, agent and the `wp-pg` CNPG backend — out
+of the shared `apps` namespace into one of its own. Distinct from
+`MIGRATION.md`, which covers the earlier SQLite → CNPG move.
+
+## Why
+
+`apps` holds gitea, gatus, external-dns, ical-proxy and woodpecker together,
+and its `default-deny-ingress` uses `endpointSelector: {}`. Every allow rule it
+carries therefore applies to *every* workload in the namespace.
+
+That is not academic. Issue #433 — `wp-pg` stuck at `Ready=False` with
+`Instance Status Extraction Error` for three weeks — is caused by the CNPG
+operator in `database` being unable to reach `https://<pod-ip>:8000/pg/status`.
+The only namespace-level fix is `fromEndpoints: namespace=database`, which
+would expose gitea, gatus, external-dns and ical-proxy to the whole `database`
+namespace to solve a problem affecting one pod on one port.
+
+`cnpg_collector_up` has series for authentik, immich, monitoring and paperless
+but not for `wp-pg`, for the same reason: `apps` never allowed the monitoring
+namespace either.
+
+A namespace per service keeps each grant on the workload that needs it.
+
+**`woodpecker-ci` is not the target.** It enforces `pod-security: privileged`
+and exists solely for ephemeral pipeline step pods running repo-supplied code.
+Putting the database there would let any build step reach it through the
+intra-namespace allow.
+
+## Staging
+
+Rollback is trivial through stage 2. The `apps` cluster is untouched until
+stage 3, so reverting means pointing the DSN back and scaling up.
+
+| Stage | Contents | CI impact |
+|---|---|---|
+| **1** (this PR) | `woodpecker` namespace, its network policies, ExternalSecret, ObjectStores, and `wp-pg` restored from barman | none — parallel, nothing points at it |
+| **runbook** | verify the restore holds real data | none |
+| **2** | move server + agent, flip DSN, RBAC subject, rotator policies, docs | **downtime** |
+| **3** | delete the `apps` originals, add the prune guard to the survivor | none |
+
+## Stage 1 design notes
+
+**Two ObjectStores, deliberately.** `wp-pg-store-source` points at
+`s3://renner-uno-k8s-cnpg/wp-pg` — the prefix the live cluster archives to —
+and is read-only. `wp-pg-store` points at a **new** prefix,
+`s3://renner-uno-k8s-cnpg/wp-pg-woodpecker`, for ongoing archiving.
+
+Both clusters writing one `destinationPath` would interleave their WAL streams
+and corrupt both. Splitting them also leaves the pre-migration backup intact
+and independently restorable.
+
+**`serverName: wp-pg`** on the `externalCluster` names the cluster that
+*produced* the backup, not the one being created.
+
+**The namespace is declared in `infrastructure/namespaces/namespaces.yaml`,**
+not alongside these workloads. Its CiliumNetworkPolicies ship with the
+`infrastructure-crds` KS (`infrastructure/post-crds` includes
+`../network-policies/`), which `dependsOn` `infrastructure` — so the namespace
+exists before any policy targets it. Creating it in the `woodpecker` workload
+KS would invert that order, since that KS runs after both.
+
+**Ingress from `woodpecker-ci` is deliberately absent in stage 1.** Step pods do
+call the server back on `:8000`, but the CNPG instance manager serves its status
+API on `:8000` too, and the baseline policy uses `endpointSelector: {}`. A
+`toPorts` block narrows the port, not the destination workload — so putting that
+rule on the baseline would hand arbitrary repo-supplied build steps a route to
+wp-pg's instance manager. It lands in stage 2 as its own policy selecting the
+server pod.
+
+**No `prune: disabled` guard on the restored cluster.** The original carries one
+because it holds the only copy of the data. This one is reconstructable by
+re-running the restore, and a prune guard on a migration artifact is how orphans
+survive an abandoned migration. It gains the guard in stage 3.
+
+**Barman credentials still come from `seed/paperless/hetzner-creds`.** Issue
+#397 wants woodpecker to own `seed/woodpecker/hetzner-creds`, and this would
+have been the natural moment — but creating that path needs OpenBao write
+access that was not available (`bao kv list seed` → 403). Repointing at a path
+that may not exist would leave the restored cluster unable to archive WAL,
+which is the thing this stage exists to prove. #397 stays open; the swap is one
+line once the path exists.
+
+## Expect the `woodpecker` Kustomization to go not-Ready during the restore
+
+`kubernetes/clusters/homelab/woodpecker.yaml` sets `wait: true` and a
+`healthCheckExprs` entry matching `postgresql.cnpg.io/v1 Cluster` **with no
+name or namespace qualifier**. Flux evaluates it against every Cluster in that
+Kustomization's inventory — which, after this stage, is both the live
+`apps/wp-pg` and the restoring `woodpecker/wp-pg`.
+
+So while the restore runs, the whole `woodpecker` KS — the one that owns
+production Woodpecker — reports not-Ready, and a reconcile timeout in that
+window will trip the `flux-failures` Alert. **That is expected and is not
+Woodpecker breaking.** Nothing `dependsOn: woodpecker`, so it does not cascade.
+
+If the restore is going to sit for a long time, or the noise is unwelcome, the
+options are to suspend the KS while it completes, or to split the migration
+resources into their own `wait: false` Kustomization. For a ~660 MB database the
+window should be minutes.
+
+## Verifying stage 1
+
+Restore is asynchronous. Watch it reach a healthy state:
+
+```sh
+kubectl -n woodpecker get cluster wp-pg -w
+kubectl -n woodpecker get pods
+```
+
+Expect `Cluster in healthy state`. If it sticks in bootstrap, the recovery job's
+logs carry the reason:
+
+```sh
+kubectl -n woodpecker logs -l cnpg.io/jobRole=full-recovery --tail=100
+```
+
+**First check the generated app credentials target the right database.** CNPG
+defaults `bootstrap.recovery.database` to `app`; the restored data uses
+`woodpecker`. If this reads anything but `woodpecker`, the cluster has
+provisioned an empty `app` database and the real data is unreachable through
+the `-app` Secret and Service:
+
+```sh
+kubectl -n woodpecker get secret wp-pg-app -o jsonpath='{.data.dbname}' | base64 -d; echo
+kubectl -n woodpecker get secret wp-pg-app -o jsonpath='{.data.username}' | base64 -d; echo
+```
+
+**Then confirm the restore actually holds Woodpecker's data** — a cluster that
+comes up healthy but empty is the failure mode worth catching:
+
+```sh
+PGPASS=$(kubectl -n woodpecker get secret wp-pg-app -o jsonpath='{.data.password}' | base64 -d)
+kubectl -n woodpecker exec -it wp-pg-1 -- env PGPASSWORD="$PGPASS" \
+  psql -U woodpecker -d woodpecker -c '\dt'
+kubectl -n woodpecker exec -it wp-pg-1 -- env PGPASSWORD="$PGPASS" \
+  psql -U woodpecker -d woodpecker -c 'select count(*) from repos;'
+```
+
+Row counts should match the live cluster:
+
+```sh
+PGPASS_OLD=$(kubectl -n apps get secret wp-pg-app -o jsonpath='{.data.password}' | base64 -d)
+kubectl -n apps exec -it wp-pg-1 -- env PGPASSWORD="$PGPASS_OLD" \
+  psql -U woodpecker -d woodpecker -c 'select count(*) from repos;'
+```
+
+They will not match exactly if builds ran after the backup the restore used.
+
+**And stage 2 does not close that gap.** An earlier draft of this document
+claimed it quiesced the server first; it does not — Flux simply prunes the
+`apps` workloads and creates the `woodpecker` ones, so anything written to the
+old database between the restore and the cutover stays only in the old database.
+
+Measured on the live cutover: the restore captured through pipeline `9488`,
+four more (`9491`, `9492`, `9494`, `9496`) landed in `apps/wp-pg` before the
+swap, and the new cluster resumed at `9521`. Four pipeline records and their
+log entries exist only in the old cluster. No ID collision, no functional
+breakage — Woodpecker's own manifest calls this database "small and
+non-critical (state is largely reconstructable)" — but it is real history loss
+and should be a decision, not a surprise.
+
+If that gap matters for a future migration, quiesce before cutting over:
+
+```sh
+flux suspend kustomization woodpecker
+kubectl -n <old-ns> scale deploy/woodpecker-server deploy/woodpecker-agent --replicas=0
+# re-run the restore so it captures the final state, then resume
+```
+
+The old cluster keeps its `prune: disabled` guard until stage 3, so the missing
+rows remain recoverable until then.
+
+Confirm the two fixes this move exists for:
+
+```sh
+# #433: operator can now reach the instance
+kubectl -n woodpecker get cluster wp-pg -o jsonpath='{.status.conditions}'
+# metrics now scraped — needs BOTH the PodMonitor (cnpg-podmonitors.yaml) and
+# the monitoring ingress allow; wp-pg had neither in `apps`, which is why it is
+# absent from cnpg_collector_up while ak-pg/im-pg/gf-pg/pprl-pg are present
+kubectl get --raw '/api/v1/namespaces/monitoring/services/kube-prometheus-stack-prometheus:9090/proxy/api/v1/query?query=cnpg_collector_up{namespace="woodpecker"}'
+```
+
+Confirm the scrape target is actually registered, not just that the metric is
+missing for some other reason:
+
+```sh
+kubectl -n woodpecker get podmonitor cnpg-wp-pg
+```
+
+**Check the policy semantics by hand.** `kubectl --dry-run=server` validates
+manifest shape, not Cilium behaviour, so it cannot catch a rule that is
+syntactically fine but scoped wider than intended.
+
+`cilium-dbg policy trace` does **not** exist in Cilium 1.20 — it was removed,
+and `cilium-dbg policy` now offers only `get`, `selectors`, `subject-selectors`
+and `wait`. Read the realized BPF policy map instead, which is better evidence
+anyway: it is the datapath as programmed, not a simulation of it, and it
+carries live byte/packet counters.
+
+Find the agent on wp-pg's node and the endpoint id:
+
+```sh
+kubectl -n woodpecker get pod wp-pg-1 -o jsonpath='{.spec.nodeName}'
+kubectl -n kube-system get pod -l k8s-app=cilium \
+  --field-selector spec.nodeName=<node> --no-headers -o custom-columns=N:.metadata.name
+
+kubectl -n kube-system exec <cilium-pod> -c cilium-agent -- \
+  cilium-dbg endpoint list -o json \
+  | jq -r '.[] | select((.status.labels."security-relevant"[]?|tostring)
+      | test("cnpg.io/cluster=wp-pg")) | "\(.id) \(.status.networking.addressing[0].ipv4)"'
+```
+
+Then, on that endpoint id — who may reach `:8000`:
+
+```sh
+kubectl -n kube-system exec <cilium-pod> -c cilium-agent -- \
+  cilium-dbg bpf policy get <endpoint-id> | awk '$4 ~ /8000/ || NR<=1'
+```
+
+Expect exactly one row, the CNPG operator, with non-zero counters once it has
+polled:
+
+```
+Allow  Ingress  k8s:app.kubernetes.io/instance=cloudnative-pg  8000/TCP  11756  58
+```
+
+And confirm CI step pods reach **ICMP only** — never a TCP port:
+
+```sh
+kubectl -n kube-system exec <cilium-pod> -c cilium-agent -- \
+  cilium-dbg bpf policy get <endpoint-id> | grep woodpecker-ci | awk '{print $4}' | sort -u
+```
+
+Every line must be an ICMP/ICMPv6 type. A bare port number here means the
+namespace baseline has grown a rule it should not have.
+
+Note the operator's allow is also confirmed indirectly and more strongly by
+`Ready=True` on the Cluster: that condition only goes true once the operator has
+successfully polled this exact endpoint and port.
+
+And that archiving goes to the new prefix, leaving the old one alone:
+
+```sh
+kubectl -n woodpecker get cluster wp-pg -o jsonpath='{.status.conditions[?(@.type=="ContinuousArchiving")]}'
+```
+
+## Stage 2 — the cutover
+
+Moves server, agent, service, ingress, pdb, the agent ServiceAccount, the
+`woodpecker-secret` ExternalSecret and the rotator policy into `woodpecker`,
+points the server's DSN at `wp-pg-rw.woodpecker.svc`, repoints the agent's gRPC
+at `woodpecker.woodpecker.svc:9000`, moves the openbao rotators' egress, and
+adds the `woodpecker-ci` → server `:8000` policy that stage 1 deliberately
+deferred — scoped to the server pod, not the namespace baseline.
+
+**This is the downtime window.** Flux prunes the `apps` copies and creates the
+`woodpecker` ones; `ci.renner.uno` is down in between, and in-flight pipelines
+die with their agent.
+
+**The outage outlasts the pods becoming Ready, because of TLS.** Certificates
+are namespace-scoped, so moving the Ingress does not move `woodpecker-tls` —
+cert-manager issues a fresh one in the new namespace. Until it completes,
+Traefik serves a default certificate and clients fail before they get an HTTP
+status at all:
+
+```
+curl: (60) SSL: no alternative certificate subject name matches target hostname 'ci.renner.uno'
+```
+
+That is expected, not a routing or policy fault. Watch it clear with:
+
+```sh
+kubectl -n woodpecker get certificate woodpecker-tls -w
+```
+
+`letsencrypt-cluster` solves **DNS-01** via Cloudflare, so there is no in-cluster
+ACME solver pod and no HTTP challenge to admit through the network policies —
+worth knowing before scoping any ingress rule, since an HTTP-01 issuer would
+need the solver reachable too.
+
+The four CNPG files (`cluster.yaml`, `objectstore.yaml`, `scheduled-backup.yaml`,
+`hetzner-creds-externalsecret.yaml`) still describe the **original** apps-namespace
+cluster and are deliberately untouched — they are the rollback. Stage 3 retires
+them.
+
+### Verifying stage 2
+
+```sh
+kubectl -n woodpecker get pods
+kubectl -n apps get pods -l app.kubernetes.io/name=woodpecker   # expect none
+```
+
+The server must reach the restored database, not the old one:
+
+```sh
+kubectl -n woodpecker logs deploy/woodpecker-server --tail=40 | grep -i "database\|migrat\|error"
+kubectl -n woodpecker exec deploy/woodpecker-server -- env | grep DATASOURCE
+```
+
+Then confirm the UI and a real pipeline, which is the only check that exercises
+server → agent → step pod → server end to end:
+
+```sh
+curl -sS -o /dev/null -w '%{http_code}\n' https://ci.renner.uno/
+woodpecker pipeline create --branch master --var PIPELINE=e2e-test k.renner/homelab
+```
+
+Step pods must still reach the server on `:8000` while still being unable to
+reach wp-pg on any TCP port — re-run the `bpf policy get` checks above against
+both the server's and wp-pg's endpoint ids.
+
+Check the rotators can still reach the server, since their egress moved:
+
+```sh
+kubectl -n openbao get cronjob
+kubectl -n openbao logs -l app.kubernetes.io/component=secret-rotator --tail=20
+```
+
+### Rolling back stage 2
+
+Revert the PR. Flux recreates the `apps` workloads and the DSN points back at
+`wp-pg-rw.apps.svc`, which never stopped running. Data written to the new
+cluster during the window is lost, so roll back promptly or not at all — if
+pipelines have run against `woodpecker/wp-pg`, prefer fixing forward.
+
+## Stage 3 — retiring the old cluster
+
+Deletes the four `apps` manifests, promotes the `migration-*` set onto their
+names, and gives the surviving Cluster the `prune: disabled` data guard it was
+deliberately denied while a rollback still existed.
+
+**Merging stage 3 does not delete anything.** The old resources carry
+`kustomize.toolkit.fluxcd.io/prune: disabled`, so dropping them from git only
+stops Flux managing them — they keep running, and keep occupying their PVC.
+That is the point: the PR is reversible, the teardown is a separate, deliberate
+act.
+
+Before deleting anything, confirm the new cluster is genuinely standing on its
+own — that it has taken at least one base backup to its own prefix, so recovery
+no longer depends on the retired cluster's data:
+
+```sh
+kubectl -n woodpecker get backup.postgresql.cnpg.io
+kubectl -n woodpecker get cluster wp-pg -o jsonpath='{.status.conditions}'
+```
+
+`LastBackupSucceeded` must be `True`, not merely `BackupStarted`.
+
+Then retire the old set:
+
+```sh
+kubectl -n apps delete scheduledbackup.postgresql.cnpg.io wp-pg-daily
+kubectl -n apps delete cluster.postgresql.cnpg.io wp-pg          # deletes its PVCs
+kubectl -n apps delete objectstore.barmancloud.cnpg.io wp-pg-store
+kubectl -n apps delete externalsecret hetzner-s3-creds
+```
+
+Delete the ScheduledBackup first: leaving it while the Cluster goes away
+produces failing backup jobs against a cluster that no longer exists.
+
+**What this destroys and what it does not.** It destroys the old cluster's PVCs
+— including the four pipeline records that only ever existed there (see the
+stage-1 verification section). It does **not** touch
+`s3://renner-uno-k8s-cnpg/wp-pg`: that prefix keeps its backups until their own
+retention expires, and `wp-pg-store-source` is retained precisely so it stays
+restorable. That is the real safety net, not the running cluster.
+
+### After the teardown
+
+The `apps` namespace should have no woodpecker-related objects left:
+
+```sh
+kubectl -n apps get cluster,objectstore,scheduledbackup,externalsecret 2>/dev/null | grep -i "wp-pg\|hetzner" || echo "clean"
+```
+
+## Rolling back stage 1
+
+Delete the four resources and the namespace. Nothing references them, and the
+`apps` cluster is untouched:
+
+```sh
+kubectl delete ns woodpecker
+```
+
+Then revert this PR so Flux stops recreating them.

kubernetes/apps/woodpecker/agent.yaml

@@ -2,7 +2,7 @@ apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: woodpecker-agent
-  namespace: apps
+  namespace: woodpecker
   labels:
     app.kubernetes.io/name: woodpecker
     app.kubernetes.io/component: agent
@@ -12,7 +12,7 @@ metadata:
     secret.reloader.stakater.com/reload: woodpecker-secret
 spec:
   # 1: the in-cluster server owns ci.renner.uno now, so this agent registers with
-  # it (woodpecker.apps.svc:9000) and runs pipelines on the kubernetes backend,
+  # it (woodpecker.woodpecker.svc:9000) and runs pipelines on the kubernetes backend,
   # replacing the retired docker agents (LXC 910/911).
   replicas: 1
   selector:
@@ -37,7 +37,7 @@ spec:
           image: woodpeckerci/woodpecker-agent:v3.17.0 # {"$imagepolicy": "flux-system:woodpecker-agent"}
           env:
             - name: WOODPECKER_SERVER
-              value: woodpecker.apps.svc.cluster.local:9000 # in-cluster gRPC, replaces LXC :9000
+              value: woodpecker.woodpecker.svc.cluster.local:9000 # in-cluster gRPC, replaces LXC :9000
             - name: WOODPECKER_MAX_WORKFLOWS
               value: "2"
             # --- kubernetes backend: pipeline steps run as native pods (retires

kubernetes/apps/woodpecker/cluster.yaml

@@ -1,19 +1,27 @@
+---
+# Woodpecker's Postgres backend. Serves the server at wp-pg-rw.woodpecker.svc.
+#
+# Originally created in `apps` with bootstrap.initdb; moved here by restoring
+# from barman into the new namespace rather than by editing a namespace field,
+# because PVCs are namespace-scoped and a Cluster cannot be relocated in place
+# (issue #433, MIGRATION-namespace.md).
+#
+# The bootstrap.recovery stanza and its externalCluster are retained rather than
+# reverted to initdb. CNPG honours `bootstrap` only when PGDATA is absent, so on
+# this live cluster it is inert — and keeping it means a cold rebuild of an
+# empty cluster restores from backup instead of coming up empty, which is the
+# gap kubernetes/RECOVERY.md exists to close for the other CNPG clusters.
 apiVersion: postgresql.cnpg.io/v1
 kind: Cluster
 metadata:
   name: wp-pg
-  namespace: apps
+  namespace: woodpecker
   annotations:
-    # CRITICAL data-guard: prune-disabled so the seed-woodpecker -> dedicated
-    # woodpecker KS ownership handoff can never prune this Cluster (which would
-    # delete its data PVCs). Landed here first, via seed-woodpecker, so it's
-    # live before the move. Kept permanently as a standing guard.
+    # Data guard, earned at stage 3 now that this is the only copy: prevents a
+    # Flux prune from deleting the Cluster and with it the data PVCs. The
+    # migration deliberately withheld this while a rollback still existed.
     kustomize.toolkit.fluxcd.io/prune: disabled
 spec:
-  # Single instance: the Woodpecker CI DB is small and non-critical (state is
-  # largely reconstructable), so HA isn't warranted — but WAL archiving to
-  # object storage still gives point-in-time backups. Mirrors grafana-db's
-  # single-instance shape, with barman enabled like immich/paperless.
   instances: 1
   imageName: ghcr.io/cloudnative-pg/postgresql:16
   storage:
@@ -25,9 +33,25 @@ spec:
       parameters:
         barmanObjectName: wp-pg-store
   bootstrap:
-    initdb:
+    recovery:
+      source: wp-pg-apps
+      # Must mirror the initdb values of the cluster this restores from
+      # (apps/wp-pg: database woodpecker, owner woodpecker). CNPG defaults
+      # `database` to `app` and `owner` to whatever `database` is, and those
+      # names do not exist in the physically restored data — it would provision
+      # an empty `app` database, point the generated wp-pg-app Secret and the
+      # -app Service at it, and leave the real data unreachable through them.
+      # Same pattern as every recovery example in kubernetes/RECOVERY.md.
       database: woodpecker
       owner: woodpecker
+  externalClusters:
+    - name: wp-pg-apps
+      plugin:
+        name: barman-cloud.cloudnative-pg.io
+        parameters:
+          barmanObjectName: wp-pg-store-source
+          # The cluster that produced the backup, not this one.
+          serverName: wp-pg
   postgresql:
     parameters:
       shared_buffers: "128MB"

kubernetes/apps/woodpecker/externalsecret.yaml

@@ -2,7 +2,7 @@ apiVersion: external-secrets.io/v1
 kind: ExternalSecret
 metadata:
   name: woodpecker-secret
-  namespace: apps
+  namespace: woodpecker
 spec:
   # 10m (was 1h) so an automated rotation (rotate-agent-secret.yaml writes a new
   # value to OpenBao) propagates into the k8s Secret within ~10m without a manual

kubernetes/apps/woodpecker/hetzner-creds-externalsecret.yaml

@@ -1,11 +1,20 @@
+---
+# Barman S3 credentials for wp-pg.
+#
+# Sourced from seed/paperless/hetzner-creds, which is not where they belong.
+# Issue #397 wants woodpecker to own seed/woodpecker/hetzner-creds; creating
+# that path needs OpenBao write access (`bao kv list seed` → 403 for the
+# identities available here), so it stays open. The values are identical, so it
+# is a one-line swap of the `key:` below once the path exists.
+#
+# The namespace allowlist on the openbao ClusterSecretStore
+# (infrastructure/post-crds/openbao-clustersecretstore.yaml) must include this
+# namespace, or ESO silently never syncs — see #505.
 apiVersion: external-secrets.io/v1
 kind: ExternalSecret
 metadata:
   name: hetzner-s3-creds
-  namespace: apps
-  annotations:
-    # Prune-disabled — a standing guard kept from the seed->dedicated-KS handoff (barman S3 creds).
-    kustomize.toolkit.fluxcd.io/prune: disabled
+  namespace: woodpecker
 spec:
   refreshInterval: 1h
   secretStoreRef:
@@ -14,10 +23,5 @@ spec:
   target:
     name: hetzner-s3-creds
   dataFrom:
-    # Reuses the paperless copy of the shared Hetzner S3 credentials (identical
-    # ACCESS_KEY_ID / ACCESS_SECRET_KEY / REGION) to avoid seeding a new OpenBao
-    # path for this migration. FOLLOW-UP: give woodpecker its own
-    # seed/woodpecker/hetzner-creds path once OpenBao can be written, and repoint
-    # this here (the value is the same, so it's a drop-in swap).
     - extract:
         key: seed/paperless/hetzner-creds

kubernetes/apps/woodpecker/ingress.yaml

@@ -2,7 +2,7 @@ apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
   name: woodpecker
-  namespace: apps
+  namespace: woodpecker
   labels:
     app.kubernetes.io/name: woodpecker
   annotations:

kubernetes/apps/woodpecker/kustomization.yaml

@@ -21,11 +21,20 @@ resources:
   # (kubernetes/infrastructure/openbao/rotation/woodpecker.yaml) — intra-ns
   # OpenBao access, no NetworkPolicy needed. server.yaml/agent.yaml keep their
   # Reloader annotations; externalsecret.yaml keeps the 10m refreshInterval.
-  # wp-pg CNPG Postgres backend (moved here from the retired seed-woodpecker KS;
-  # each object carries prune: disabled so ownership transferred without a
-  # teardown). The dedicated `woodpecker` KS depends on infrastructure-cnpg-barman
-  # for the barman plugin behind the ObjectStore/ScheduledBackup.
-  - cluster.yaml
-  - objectstore.yaml
-  - scheduled-backup.yaml
+  # --- namespace migration, stages 1-2 (issue #433) ---
+  # Stage 2 has moved the workloads above (server, agent, service, ingress, pdb,
+  # rbac, secrets, policies) into the `woodpecker` namespace and pointed the
+  # server's DSN at wp-pg-rw.woodpecker.svc. Stage 3 retired the `apps`
+  # originals; these four now describe the surviving cluster. History and the
+  # reasoning behind the three stages: MIGRATION-namespace.md.
+  #
+  # The `woodpecker` Namespace itself is declared in infrastructure/namespaces,
+  # not here: it must exist before the `infrastructure-crds` KS applies the
+  # namespace's CiliumNetworkPolicies, and this KS runs after both of those.
+  #
+  # This KS depends on infrastructure-cnpg-barman for the barman plugin behind
+  # the ObjectStore and ScheduledBackup.
   - hetzner-creds-externalsecret.yaml
+  - objectstore.yaml
+  - cluster.yaml
+  - scheduled-backup.yaml

kubernetes/apps/woodpecker/networkpolicy.yaml

@@ -1,20 +1,23 @@
 # Secret rotation: the review-bot and claude-oauth rotators (openbao ns) reach
 # the Woodpecker SERVER to upsert org-level CI secrets (review_bot_token,
 # claude_code_oauth_token). They use the in-cluster Service
-# (woodpecker.apps.svc:8000) — the public ci.renner.uno name hairpins off the
-# ingress LB from a pod. The apps default-deny-ingress only allows same-ns /
-# host / a few named sources, so without this the rotator's connect to :8000 is
+# (woodpecker.woodpecker.svc:8000) — the public ci.renner.uno name hairpins off the
+# ingress LB from a pod. This namespace's baseline
+# (network-policies/woodpecker-ns.yaml) admits only same-namespace, monitoring,
+# host/remote-node and ICMP, so without this the rotator's connect to :8000 is
 # dropped. Scoped to the woodpecker SERVER pod :8000 and the two rotator labels;
 # the egress halves are rotator-review-bot-egress + rotator-claude-oauth-egress
 # in network-policies/openbao-ns.yaml.
 #
-# This is additive to the apps-ns baseline allows (allow-ingress.yaml selects all
-# apps pods), so the server's existing ingress — Traefik, agents — is unaffected.
+# Cilium unions allow rules, so this composes with the baseline and with the
+# sibling policies below (traefik, CI steps) rather than replacing them. Note
+# allow-ingress.yaml does NOT apply here — it is hard-scoped to `namespace:
+# apps`, which is why traefik needs its own rule in this file now.
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
 metadata:
   name: woodpecker-allow-rotators
-  namespace: apps
+  namespace: woodpecker
 spec:
   endpointSelector:
     matchLabels:
@@ -34,3 +37,25 @@ spec:
         - ports:
             - port: "8000"
               protocol: TCP
+---
+# Pipeline step pods call the server back on :8000 to stream logs and report
+# status. Deferred from stage 1 on purpose, and scoped to the SERVER pod rather
+# than granted on the namespace baseline.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: woodpecker-allow-ci-steps
+  namespace: woodpecker
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: woodpecker
+      app.kubernetes.io/component: server
+  ingress:
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": woodpecker-ci
+      toPorts:
+        - ports:
+            - port: "8000"
+              protocol: TCP

kubernetes/apps/woodpecker/objectstore.yaml

@@ -1,18 +1,20 @@
 ---
+# RECOVERY SOURCE — read-only. The prefix the retired `apps/wp-pg` archived to.
+#
+# Kept after the migration on purpose. Nothing writes to it, and the backups
+# under it age out on their own retention, but while they last this is a second,
+# independent restore point that predates the move — worth having until the new
+# prefix has a meaningful backup history of its own.
+#
+# serverName on the consuming externalCluster must be `wp-pg`, the name of the
+# cluster that produced these backups.
 apiVersion: barmancloud.cnpg.io/v1
 kind: ObjectStore
 metadata:
-  name: wp-pg-store
-  namespace: apps
-  annotations:
-    # Prune-disabled — a standing guard kept from the seed->dedicated-KS handoff (backup target).
-    kustomize.toolkit.fluxcd.io/prune: disabled
+  name: wp-pg-store-source
+  namespace: woodpecker
 spec:
   configuration:
-    # Same offsite Hetzner Object Storage bucket as the other CNPG clusters,
-    # under a per-cluster prefix. See apps/paperless/objectstore.yaml for the
-    # SSE / addressing rationale (Hetzner is SSE-C only; transit is TLS,
-    # bucket private).
     destinationPath: s3://renner-uno-k8s-cnpg/wp-pg
     endpointURL: https://fsn1.your-objectstorage.com
     s3Credentials:
@@ -29,4 +31,36 @@ spec:
       compression: gzip
     data:
       compression: gzip
+  # No retentionPolicy: this store is a read source during the migration, and
+  # the old cluster in `apps` owns the lifecycle of the data underneath it.
+---
+# ARCHIVING TARGET — a DIFFERENT prefix from the source, deliberately.
+#
+# Two clusters archiving WAL to one destinationPath interleave their streams and
+# corrupt both. That mattered during the parallel-run window; it still matters
+# now, because the source prefix above is retained and must stay frozen for the
+# independent restore point it provides.
+apiVersion: barmancloud.cnpg.io/v1
+kind: ObjectStore
+metadata:
+  name: wp-pg-store
+  namespace: woodpecker
+spec:
+  configuration:
+    destinationPath: s3://renner-uno-k8s-cnpg/wp-pg-woodpecker
+    endpointURL: https://fsn1.your-objectstorage.com
+    s3Credentials:
+      accessKeyId:
+        name: hetzner-s3-creds
+        key: ACCESS_KEY_ID
+      secretAccessKey:
+        name: hetzner-s3-creds
+        key: ACCESS_SECRET_KEY
+      region:
+        name: hetzner-s3-creds
+        key: REGION
+    wal:
+      compression: gzip
+    data:
+      compression: gzip
   retentionPolicy: "30d"

kubernetes/apps/woodpecker/pdb.yaml

@@ -15,7 +15,7 @@ apiVersion: policy/v1
 kind: PodDisruptionBudget
 metadata:
   name: woodpecker-server
-  namespace: apps
+  namespace: woodpecker
   labels:
     app.kubernetes.io/name: woodpecker
     app.kubernetes.io/component: server

kubernetes/apps/woodpecker/rbac.yaml

@@ -17,7 +17,7 @@ apiVersion: v1
 kind: ServiceAccount
 metadata:
   name: woodpecker-agent
-  namespace: apps
+  namespace: woodpecker
   labels:
     app.kubernetes.io/name: woodpecker
     app.kubernetes.io/component: agent
@@ -60,4 +60,7 @@ roleRef:
 subjects:
   - kind: ServiceAccount
     name: woodpecker-agent
-    namespace: apps
+    # The agent's SA lives in `woodpecker`; this Role/RoleBinding stays in
+    # `woodpecker-ci`, because that is the namespace it grants pod
+    # create/delete in. Cross-namespace subject, deliberately.
+    namespace: woodpecker

kubernetes/apps/woodpecker/scheduled-backup.yaml

@@ -1,18 +1,19 @@
 ---
-# Daily base backup at 02:35 UTC (offset from the other clusters' 02:30 to
-# avoid a thundering herd against the Hetzner bucket). WAL archiving runs
-# continuously (see cluster.yaml spec.plugins[].isWALArchiver). immediate:true
-# takes the first base backup as soon as this CR applies.
+# Daily base backups onto the wp-pg-woodpecker prefix. Continuous WAL archiving
+# runs separately (cluster.yaml spec.plugins[].isWALArchiver); without periodic
+# base backups the prefix would have no recovery point of its own and
+# wp-pg-store's retentionPolicy: 30d would have nothing to prune against, so the
+# WAL chain would grow without bound.
+#
+# 02:40 UTC, offset from the other CNPG clusters' 02:30 so they do not all hit
+# the Hetzner bucket together.
 apiVersion: postgresql.cnpg.io/v1
 kind: ScheduledBackup
 metadata:
   name: wp-pg-daily
-  namespace: apps
-  annotations:
-    # Prune-disabled — a standing guard kept from the seed->dedicated-KS handoff (backup schedule).
-    kustomize.toolkit.fluxcd.io/prune: disabled
+  namespace: woodpecker
 spec:
-  schedule: "0 35 2 * * *"
+  schedule: "0 40 2 * * *"
   immediate: true
   backupOwnerReference: self
   cluster:

kubernetes/apps/woodpecker/secret.example.yaml

@@ -14,7 +14,7 @@ apiVersion: v1
 kind: Secret
 metadata:
   name: woodpecker-secret
-  namespace: apps
+  namespace: woodpecker
   labels:
     app.kubernetes.io/name: woodpecker
 type: Opaque

kubernetes/apps/woodpecker/server.yaml

@@ -2,7 +2,7 @@ apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: woodpecker-server
-  namespace: apps
+  namespace: woodpecker
   labels:
     app.kubernetes.io/name: woodpecker
     app.kubernetes.io/component: server
@@ -84,7 +84,7 @@ spec:
             - name: WOODPECKER_DATABASE_DRIVER
               value: postgres
             - name: WOODPECKER_DATABASE_DATASOURCE
-              value: "host=wp-pg-rw.apps.svc.cluster.local port=5432 user=$(WP_DB_USER) password=$(WP_DB_PASS) dbname=woodpecker sslmode=require"
+              value: "host=wp-pg-rw.woodpecker.svc.cluster.local port=5432 user=$(WP_DB_USER) password=$(WP_DB_PASS) dbname=woodpecker sslmode=require"
           # AGENT_SECRET + GITEA_CLIENT + GITEA_SECRET come from the SOPS secret.
           envFrom:
             - secretRef:

kubernetes/apps/woodpecker/service.yaml

@@ -2,13 +2,13 @@ apiVersion: v1
 kind: Service
 metadata:
   name: woodpecker
-  namespace: apps
+  namespace: woodpecker
   labels:
     app.kubernetes.io/name: woodpecker
 spec:
   # Flipped ExternalName -> ClusterIP: ci.renner.uno (via the Ingress, by name)
   # and the in-cluster agent both target this Service now. http:8000 serves the
-  # UI/API; grpc:9000 is how the agent (woodpecker.apps.svc:9000) reaches the
+  # UI/API; grpc:9000 is how the agent (woodpecker.woodpecker.svc:9000) reaches the
   # server. Selects the server pod only (not the agent).
   type: ClusterIP
   selector:

kubernetes/infrastructure/namespaces/namespaces.yaml

@@ -160,3 +160,26 @@ apiVersion: v1
 kind: Namespace
 metadata:
   name: wordpress
+---
+# The Woodpecker service itself — server, agent and its wp-pg Postgres backend,
+# moving out of the shared `apps` namespace (issue #433). Distinct from
+# `woodpecker-ci`, which enforces pod-security: privileged and holds only
+# ephemeral pipeline step pods.
+#
+# `apps` uses an endpointSelector:{} default-deny, so every allow it carries
+# applies to gitea, gatus, external-dns and ical-proxy alike — granting the
+# CNPG operator access to wp-pg there would have opened all of `apps` to all of
+# `database`. A namespace per service keeps that grant on the one workload.
+#
+# Declared here rather than alongside the workloads so it is created by the
+# `infrastructure` KS. The namespace's CiliumNetworkPolicies ship with the
+# `infrastructure-crds` KS (infrastructure/post-crds includes
+# ../network-policies/), which itself dependsOn `infrastructure` — so the
+# namespace exists before any policy targets it. Creating it in the `woodpecker`
+# workload KS would invert that order, since that KS runs after both.
+#
+# No pod-security labels: the cluster default applies.
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: woodpecker

kubernetes/infrastructure/network-policies/kustomization.yaml

@@ -21,6 +21,7 @@ resources:
   - openbao-ns.yaml
   - external-secrets-ns.yaml
   - woodpecker-ci-ns.yaml
+  - woodpecker-ns.yaml
   - trmnl-ns.yaml
   - wordpress-ns.yaml
   - allow-cilium-health.yaml

kubernetes/infrastructure/network-policies/openbao-ns.yaml

@@ -141,9 +141,9 @@ spec:
 ---
 # Egress for the review-bot rotator. Its CronJob has two steps: the init step
 # mints review-bot's PAT against gitea (apps :3000), the main step upserts the
-# Woodpecker org secret against the woodpecker server (apps :8000). Both run
+# Woodpecker org secret against the woodpecker server (woodpecker :8000). Both run
 # under the same pod labels, so one egress policy covers both hops. Complements
-# gitea-allow-review-bot-rotator + woodpecker-allow-rotators (apps side).
+# gitea-allow-review-bot-rotator (apps) + woodpecker-allow-rotators (woodpecker).
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
 metadata:
@@ -165,16 +165,16 @@ spec:
               protocol: TCP
     - toEndpoints:
         - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": apps
+            "k8s:io.kubernetes.pod.namespace": woodpecker
             app.kubernetes.io/name: woodpecker
       toPorts:
         - ports:
             - port: "8000"
               protocol: TCP
 ---
-# Egress for the claude-oauth sync rotator: woodpecker server only (apps :8000).
+# Egress for the claude-oauth sync rotator: woodpecker server only (woodpecker :8000).
 # It never touches gitea (the OAuth value is seeded into OpenBao by hand, not
-# minted). Complements woodpecker-allow-rotators (apps side).
+# minted). Complements woodpecker-allow-rotators (woodpecker side).
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
 metadata:
@@ -188,7 +188,7 @@ spec:
   egress:
     - toEndpoints:
         - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": apps
+            "k8s:io.kubernetes.pod.namespace": woodpecker
             app.kubernetes.io/name: woodpecker
       toPorts:
         - ports:

kubernetes/infrastructure/network-policies/woodpecker-ns.yaml

@@ -0,0 +1,156 @@
+---
+# Baseline ingress policy for the `woodpecker` namespace (wp-pg now; server and
+# agent arrive in stage 2).
+#
+# Unlike the `apps` baseline this replaces, every allow here lands on a
+# namespace holding one service, so a namespace-scoped grant is not a blanket
+# grant across unrelated workloads. That is the point of the move: in `apps`,
+# letting the CNPG operator reach wp-pg would also have opened gitea, gatus,
+# external-dns and ical-proxy to the whole database namespace (issue #433).
+#
+# CAUTION when adding rules here: this policy has `endpointSelector: {}`, so
+# every rule below applies to EVERY pod in the namespace. A `toPorts` block
+# narrows the port, NOT the destination workload — that is governed solely by
+# the enclosing policy's endpointSelector. Anything meant for one workload
+# belongs in its own policy with its own selector, as
+# `wp-pg-allow-cnpg-operator` below does.
+#
+# No egress section, so egress is unrestricted. Deliberate for now and revisited
+# in stage 2: wp-pg needs DNS plus `world` for WAL archiving to Hetzner S3, and
+# the server/agent add gitea, the Kubernetes API and outbound plugin pulls.
+# Writing that ruleset once, with the full workload set present, beats guessing
+# it here and silently breaking the WAL archiving this stage exists to verify.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: default-deny-ingress
+  namespace: woodpecker
+spec:
+  endpointSelector: {}
+  ingress:
+    # Intra-namespace: server <-> agent, server -> wp-pg.
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": woodpecker
+    # NOT here: ingress from the `ingress` namespace. Traefik terminates
+    # ci.renner.uno and forwards to the server, but that namespace also holds
+    # netbird-ingress, and this policy's endpointSelector:{} would extend the
+    # grant to wp-pg and the agent as well. Scoped to traefik -> server in the
+    # companion policy at the bottom of this file.
+    # Prometheus scrape. Namespace-wide and port-unrestricted, matching the
+    # other *-ns baselines; this is what covers wp-pg's :9187 metrics.
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": monitoring
+    # Kubelet probes and hostNetwork siblings.
+    - fromEntities:
+        - host
+        - remote-node
+    # ICMP — from any cluster pod. See home-assistant-ns.yaml for the rationale.
+    # INGRESS ONLY: do NOT add an egress section to this CNP.
+    #
+    # Needed here specifically because wp-pg-allow-cnpg-operator below attaches
+    # an L4 rule (toPorts :8000) to the wp-pg pods, and Cilium blocks ICMP to an
+    # endpoint once any layer 4 policy selects it unless the traffic is
+    # CT-related. Without this, PMTUD Destination Unreachable and Packet Too Big
+    # toward wp-pg are dropped silently — the failure shows up as stalled
+    # transfers, not as a policy denial.
+    #
+    # Namespace-wide is right for this one, unlike the :8000 case noted below:
+    # these are diagnostic and path-MTU messages that expose no API surface, so
+    # granting them across the namespace concedes nothing a per-workload rule
+    # would withhold.
+    - fromEndpoints:
+        - matchExpressions:
+            - key: io.kubernetes.pod.namespace
+              operator: Exists
+      icmps:
+        - fields:
+            - { type: 0,   family: IPv4 }
+            - { type: 3,   family: IPv4 }
+            - { type: 8,   family: IPv4 }
+            - { type: 11,  family: IPv4 }
+            - { type: 12,  family: IPv4 }
+            - { type: 1,   family: IPv6 }
+            - { type: 2,   family: IPv6 }
+            - { type: 3,   family: IPv6 }
+            - { type: 4,   family: IPv6 }
+            - { type: 128, family: IPv6 }
+            - { type: 129, family: IPv6 }
+            - { type: 133, family: IPv6 }
+            - { type: 134, family: IPv6 }
+            - { type: 135, family: IPv6 }
+            - { type: 136, family: IPv6 }
+            - { type: 143, family: IPv6 }
+    #
+    # NOT here: ingress from `woodpecker-ci`. Pipeline step pods do call the
+    # server back on :8000 to stream logs, but the CNPG instance manager serves
+    # its status API on :8000 too — so granting that on this endpointSelector:{}
+    # policy would hand arbitrary repo-supplied build steps a route to wp-pg's
+    # instance manager. It lands in stage 2 as its own policy selecting the
+    # server pod, the way woodpecker-allow-rotators already scopes its :8000
+    # rule in apps/woodpecker/networkpolicy.yaml.
+---
+# The CNPG operator polls each instance's status endpoint over HTTPS
+# (https://<pod-ip>:8000/pg/status) to populate Cluster status. Without it the
+# Cluster reports "Instance Status Extraction Error: HTTP communication issue"
+# while Postgres serves normally, and the operator loses failover and
+# switchover decisions — issue #433, where the operator's dial timed out
+# because the apps baseline never allowed the database namespace.
+#
+# Scoped to the wp-pg endpoints and that one port: the operator has no reason to
+# reach the woodpecker server or agent, and the barman and mariadb operators
+# sharing its namespace have no reason to reach anything here at all.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: wp-pg-allow-cnpg-operator
+  namespace: woodpecker
+spec:
+  endpointSelector:
+    matchLabels:
+      cnpg.io/cluster: wp-pg
+  ingress:
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": database
+            app.kubernetes.io/name: cloudnative-pg
+      toPorts:
+        - ports:
+            - port: "8000"
+              protocol: TCP
+---
+# Traefik terminates ci.renner.uno and forwards to the server on :8000.
+#
+# Lives here, beside the baseline, rather than with the workload policies in
+# apps/woodpecker/networkpolicy.yaml — deliberately. The baseline's `ingress`
+# admit was removed in the same commit, and these two ship in the same
+# Kustomization (infrastructure-crds), so the swap lands in one reconcile. Split
+# across Kustomizations it would not: infrastructure-crds reconciles before the
+# `woodpecker` KS, so the admit would be withdrawn minutes before its
+# replacement arrived, and ci.renner.uno would 503 in the gap.
+#
+# Scoped to traefik and to the server pod. The `ingress` namespace also holds
+# netbird-ingress, and the baseline's endpointSelector:{} would have extended a
+# namespace-level admit to wp-pg and the agent. The `apps` policy this replaces
+# (allow-ingress.yaml) matches traefik by label for the same reason, and does
+# NOT apply here — it is hard-scoped to `namespace: apps`.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: woodpecker-allow-traefik
+  namespace: woodpecker
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: woodpecker
+      app.kubernetes.io/component: server
+  ingress:
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": ingress
+            app.kubernetes.io/name: traefik
+      toPorts:
+        - ports:
+            - port: "8000"
+              protocol: TCP

kubernetes/infrastructure/openbao/rotation/claude-oauth.yaml

@@ -14,8 +14,10 @@
 # immediately:
 #   kubectl -n openbao create job --from=cronjob/claude-oauth-secret-sync claude-oauth-sync-now
 #
-# woodpecker is reached in-cluster at woodpecker.apps.svc:8000. Needs the netpol
-# halves woodpecker-allow-rotators (apps/*) + rotator-claude-oauth-egress
+# woodpecker is reached in-cluster at woodpecker.woodpecker.svc:8000 (it moved
+# out of `apps` in the namespace migration — see
+# kubernetes/apps/woodpecker/MIGRATION-namespace.md). Needs the netpol
+# halves woodpecker-allow-rotators (woodpecker ns) + rotator-claude-oauth-egress
 # (openbao-ns.yaml). OpenBao policy/role rotator-claude-oauth + the bootstraps
 # (woodpecker-bot token, the seeded OAuth value) are provisioned by hand — see
 # policies/rotator-claude-oauth.hcl.
@@ -89,7 +91,7 @@ spec:
                 - name: BAO_FIELD
                   value: setup-token
                 - name: WOODPECKER_URL
-                  value: http://woodpecker.apps.svc.cluster.local:8000
+                  value: http://woodpecker.woodpecker.svc.cluster.local:8000
                 - name: WOODPECKER_ORG
                   value: k.renner
                 - name: WOODPECKER_SECRET_NAME

kubernetes/infrastructure/openbao/rotation/review-bot.yaml

@@ -25,9 +25,10 @@
 # but a failed sync should be fixed before the next run rather than left.
 #
 # gitea is reached in-cluster at gitea.apps.svc:3000, woodpecker at
-# woodpecker.apps.svc:8000 (both hairpin off the ingress LB from a pod). Needs
+# woodpecker.woodpecker.svc:8000 (it moved out of `apps` in the namespace
+# migration — see kubernetes/apps/woodpecker/MIGRATION-namespace.md). Needs
 # the netpol halves: gitea-allow-review-bot-rotator + woodpecker-allow-rotators
-# (apps/*) and rotator-review-bot-egress (openbao-ns.yaml).
+# (woodpecker ns) and rotator-review-bot-egress (openbao-ns.yaml).
 #
 # OpenBao policy/role rotator-review-bot + the bootstraps (review-bot password,
 # woodpecker-bot token, seed review-bot#token) are provisioned by hand — see
@@ -147,7 +148,7 @@ spec:
                 - name: BAO_FIELD
                   value: token
                 - name: WOODPECKER_URL
-                  value: http://woodpecker.apps.svc.cluster.local:8000
+                  value: http://woodpecker.woodpecker.svc.cluster.local:8000
                 - name: WOODPECKER_ORG
                   value: k.renner
                 - name: WOODPECKER_SECRET_NAME

kubernetes/infrastructure/post-crds/openbao-clustersecretstore.yaml

@@ -30,6 +30,7 @@ spec:
         - paperless
         - trmnl
         - velero
+        - woodpecker # wp-pg barman creds; woodpecker moving out of `apps` (#433)
         - wordpress
   provider:
     vault: # ESO's provider name; speaks OpenBao fine

kubernetes/talos/patches/cluster.yaml

@@ -26,23 +26,23 @@ machine:
     nodeIP:
       validSubnets:
         - 10.2.40.0/24
-    # Lower the image GC thresholds from the kubelet defaults (85/80) so
-    # containerd's image cache is pruned before /var crosses Ceph's
-    # mon_data_avail_warn line — the mon DB warns when its hostpath has
-    # <=30% free, and mons live on /var/lib/rook (hostPath, no PVC).
-    # High must stay strictly below that line, not on it: a high of 70
-    # means GC first fires at exactly 30% free, which is the warn point
-    # itself, so every cache fill-up trips MON_DISK_LOW before kubelet
-    # reclaims anything. 60/50 fires at 40% free and reclaims to 50%,
-    # keeping 10 points of margin.
-    # containerd is the dominant local consumer of /var (~29 GB of 60 on
-    # k8s-2, which carries 75 pods against 42 on k8s-1) and most of it is
-    # reclaimable — 179 images cached against 60 in use. Spegel P2P
-    # pull-through softens the re-pull cost when GC'd images are needed
-    # back.
+    # Lower the image GC thresholds from the kubelet defaults (85/80) to
+    # bound containerd's image cache, which is the largest single consumer
+    # of /var. Spegel P2P pull-through softens the re-pull cost when GC'd
+    # images are needed back.
+    #
+    # These thresholds do NOT control whether Ceph's mon reports
+    # MON_DISK_LOW, and must not be tuned for that. Measured on k8s-2:
+    # image GC frees only the content store (~8.6 GB), while the bulk of
+    # /var/lib/containerd is ~20 GB of overlayfs snapshots — unpacked
+    # layers shared with running containers, which GC cannot reclaim.
+    # Dropping high to 60 deleted 28 images and moved free space by 0.02
+    # percentage points. The mon's own data is ~106 MB; its 30%-free
+    # warning is measuring a filesystem it merely shares, so the lever for
+    # it is elsewhere (mon volume, /var size, or mon_data_avail_warn).
     extraConfig:
-      imageGCHighThresholdPercent: 60
-      imageGCLowThresholdPercent: 50
+      imageGCHighThresholdPercent: 70
+      imageGCLowThresholdPercent: 60
 cluster:
   controlPlane:
     endpoint: https://k8s.renner.uno:6443

opentofu/lxc_800_rustfs_decommission.tf

@@ -1,4 +1,4 @@
-# TEMPORARY shims v2 — rustfs LXCs 800/801/802, decommission in progress.
+# TEMPORARY shims v3 — rustfs LXCs 800/801/802, decommission in progress.
 #
 # Stage 2 (#270) failed: both destroy attempts died instantly with
 # "context deadline exceeded" on the pre-destroy status GET, while the PVE
@@ -10,34 +10,113 @@
 #
 # Fix: declare the timeouts explicitly, apply ONCE so they land in state
 # (in-place no-API update), then delete this file again and the next apply
-# destroys the containers. No lifecycle ignore_changes this time — the
-# timeout update must actually apply; the other attributes mirror the real
-# containers so the plan stays in-place.
+# destroys the containers.
+#
+# v3: v2 declared ONLY the ids, `started`, `unprivileged` and the timeouts,
+# claiming "the other attributes mirror the real containers so the plan stays
+# in-place". They did not. Everything absent from config planned as a removal,
+# and two of those removals — `mount_point` and `operating_system` — force
+# replacement, so the plan came back `must be replaced` for all three. Applying
+# that would have destroyed and recreated containers tagged `critical`,
+# `state-backend`, `terraform-backend` instead of nudging a timeout into state.
+#
+# So the blocks below are not decoration: each one exists to keep an attribute
+# out of the diff. They mirror the live containers, read from the plan's own
+# state values. `tags` is declared for the same reason — without it the apply
+# strips the tags off the real containers.
 #
 #   1. merge; run PIPELINE=deploy-infrastructure (plan MUST show in-place
 #      update only, no replace), then TOFU_APPLY=true
 #   2. follow-up PR deletes this file; plan shows 3 destroys; apply.
 
 locals {
+  # Per-container values mirrored from live state. hostname and mac_address
+  # differ per container; declaring them wrong would itself force a diff.
   rustfs_decommission = {
-    "800" = "saturn"
-    "801" = "jupiter"
-    "802" = "neptune"
+    "800" = { node = "saturn", hostname = "rustfs-1.srv.renner.uno", mac = "BC:24:11:0B:E5:7B" }
+    "801" = { node = "jupiter", hostname = "rustfs-2.srv.renner.uno", mac = "BC:24:11:EA:1D:38" }
+    "802" = { node = "neptune", hostname = "rustfs-3.srv.renner.uno", mac = "BC:24:11:8F:F4:67" }
   }
 }
 
 resource "proxmox_virtual_environment_container" "rustfs_decommission" {
   for_each = local.rustfs_decommission
 
-  node_name     = each.value
+  node_name     = each.value.node
   vm_id         = tonumber(each.key)
   started       = false
   unprivileged  = true
   start_on_boot = true
 
+  tags = [
+    "critical",
+    "rustfs",
+    "state-backend",
+    "terraform-backend",
+  ]
+
   timeout_create = 1800
   timeout_clone  = 1800
   timeout_update = 1800
   timeout_delete = 1800
   timeout_start  = 300
+
+  cpu {
+    architecture = "amd64"
+    cores        = 1
+    units        = 1024
+  }
+
+  memory {
+    dedicated = 1024
+    swap      = 0
+  }
+
+  disk {
+    datastore_id = "local"
+    size         = 10
+  }
+
+  # forces replacement if absent — this is the block that broke v2.
+  mount_point {
+    path      = "/mnt/data"
+    size      = "50G"
+    volume    = "local:${each.key}/vm-${each.key}-disk-1.raw"
+    backup    = true
+    replicate = true
+  }
+
+  network_interface {
+    name        = "eth0"
+    bridge      = "v40srv"
+    enabled     = true
+    firewall    = false
+    mac_address = each.value.mac
+  }
+
+  initialization {
+    hostname = each.value.hostname
+
+    ip_config {
+      ipv4 {
+        address = "dhcp"
+      }
+    }
+  }
+
+  # operating_system is NOT declared, and must not be. The provider requires
+  # template_file_id inside it, which is a create-time-only value: PVE does not
+  # report the original template for an existing container, so it is absent
+  # from state and cannot be mirrored. Declaring the block without it fails
+  # validation ("Missing required argument"); declaring it with a guessed
+  # template would be a lie that only surfaces if something ever recreates
+  # these.
+  #
+  # Left undeclared it would plan as a removal, and removing operating_system
+  # forces replacement — the exact failure this file exists to avoid. So it is
+  # ignored instead. The timeouts above are deliberately NOT ignored: they are
+  # the one thing this shim needs to actually apply.
+  lifecycle {
+    ignore_changes = [operating_system]
+  }
 }

opentofu/providers.tf

@@ -61,11 +61,27 @@ provider "unifi" {
 # as the unifi provider above; the v2 endpoints are simply not covered by it.
 # UniFi PUT quirk: updates may require _id in the body, which restapi does
 # not send — if an in-place update 400s, taint the object so CI recreates it.
+#
+# create_returns_object, NOT write_returns_object. Both make the provider store
+# the API response into `data`; write_returns_object does it for PUT as well as
+# POST. Since `data` in unifi_nat_hairpin.tf never sets the UDM-assigned `_id`,
+# every write put a field into state that config does not have, so all 60 rules
+# planned as "will be updated in-place" with `- _id` as the whole diff — and
+# re-dirtied themselves on the next write, forever.
+#
+# No provider-side ignore can suppress that: `data` is a configured attribute,
+# compared by Terraform core before provider logic runs. Tried and disproven —
+# ignore_changes_to = ["_id"] (plan 5661), ignore_all_server_changes (5662),
+# and ignore_server_additions on restapi 3.x (5667). All three left the 60.
+#
+# create_returns_object keeps what the provider actually needs — the POST
+# response is how it learns `_id` for id_attribute below — without letting
+# updates rewrite `data`.
 provider "restapi" {
-  uri                  = "${var.unifi_api_url}/proxy/network/v2/api/site/default"
-  insecure             = var.unifi_insecure
-  write_returns_object = true
-  id_attribute         = "_id"
+  uri                   = "${var.unifi_api_url}/proxy/network/v2/api/site/default"
+  insecure              = var.unifi_insecure
+  create_returns_object = true
+  id_attribute          = "_id"
   headers = {
     "X-API-KEY"    = var.unifi_api_key
     "Content-Type" = "application/json"