fix(unifi): capture the API response on create only, not on every write

Root cause of the 60 permanently-dirty hairpin NAT rules.

The provider was configured write_returns_object = true, which makes it store
the API response into `data` after POST *and* PUT. The UDM assigns `_id`;
unifi_nat_hairpin.tf's data = jsonencode({...}) never sets it. So every write
put a field into state that the config does not have, and all 60 rules planned
as "will be updated in-place" with `- _id` as the entire diff — re-dirtying
themselves on the next write, indefinitely.

No provider-side ignore can suppress this, because `data` is a configured
attribute and Terraform compares config to state in core, before provider
logic runs. Three were tried and disproven against real plans rather than
argued about:

  ignore_changes_to = ["_id"]                       plan 5661 — 60 remained
  ignore_all_server_changes = true                  plan 5662 — 60 remained
  restapi 3.x + ignore_server_additions = true      plan 5667 — 60 remained,
                                                    plus a new api_data diff

create_returns_object keeps the part the provider genuinely needs — the POST
response is how it learns `_id` for id_attribute = "_id" — while no longer
letting updates rewrite `data` from the response.

This does not clear the existing pollution by itself: state still carries `_id`
in `data` for all 60 rules, so the plan will keep showing them until one apply
rewrites `data` to the configured value. What it changes is that the apply then
sticks, instead of the next write undoing it.

Co-Authored-By: Claude Bot <claude-bot@renner.uno>

.woodpecker/ci-manifests-test.yml

@@ -115,7 +115,7 @@ steps:
   # An earlier revision of this comment also asserted that several deploy-*.yml
   # files emit "Additional property timeout is not allowed" warnings. That was
   # observed under the v3.9.0 CLI this step used to pin, and it is NOT known to
-  # still hold on v3.16.0 — review raised the likelihood that the warning was
+  # still hold on current CLIs — review raised the likelihood that the warning was
   # schema drift in a 7-minor-old CLI rather than real debt in those files, which
   # is one of the reasons this pin moved to latest stable. It cannot be confirmed
   # from a green run either, because the loop below sends lint output to /tmp/out
@@ -123,9 +123,15 @@ steps:
   # The claim is dropped rather than restated: what this step guarantees is the
   # exit-code contract above, which is what the gate actually rests on.
   # Latest stable, per the standing rule that any software uses the latest
-  # stable version. Matching the deployed woodpecker-server (v3.16.0) is a
-  # CONSEQUENCE of that, not a separate requirement — the server is bumped by
-  # image-automation within the 3.x range, so both land on the same release.
+  # stable version. Matching the deployed woodpecker-server is a CONSEQUENCE of
+  # that, not a separate requirement — the server is bumped by image-automation
+  # within the 3.x range, so both should land on the same release.
+  #
+  # They do not land together, though: the server is a Kubernetes manifest and
+  # is bumped automatically, this pipeline file is not and cannot be (see
+  # infrastructure/image-automation/README.md). So the server moves first and
+  # this trails until someone bumps it by hand — which is what happened at
+  # v3.16.0 -> v3.17.0, caught by the update audit (#426) rather than by CI.
   #
   # It matters here because the CLI compiles the config against its own schema:
   # one behind the server can pass a config the server then rejects, or reject
@@ -152,7 +158,7 @@ steps:
   # whenever it trails the deployed server. If you bump the server, bump here.
   - name: woodpecker-lint
     depends_on: [] # see note above `steps:`
-    image: woodpeckerci/woodpecker-cli:v3.16.0-alpine
+    image: woodpeckerci/woodpecker-cli:v3.17.0-alpine
     commands:
       - |
         set -eu

kubernetes/apps/monitoring-rules/prometheusrule-woodpecker-rotation.yaml

@@ -12,37 +12,37 @@ spec:
   groups:
     - name: woodpecker-rotation.rules
       # Convergence safety net for the automated WOODPECKER_AGENT_SECRET rotation
-      # (kubernetes/apps/woodpecker/rotate-agent-secret.yaml). That rotation Job is
+      # (kubernetes/infrastructure/openbao/rotation/woodpecker.yaml). That rotation Job is
       # fire-and-forget: it writes OpenBao and exits WITHOUT verifying ESO re-synced
       # or that both consumers restarted healthy. These alerts close that gap. ESO
       # metrics are not scraped in this cluster, so everything here is built on
-      # kube-state-metrics. The CronJob/Job series stay empty until the rotation
-      # feature (branch feat/woodpecker-secret-auto-rotation) is merged — harmless.
+      # kube-state-metrics. The rotation has since landed: the CronJob lives in the
+      # openbao namespace, not apps, which is where the last two rules look.
       rules:
         - alert: WoodpeckerServerNotReady
           # 0 ready replicas well past the 10m ESO refresh + Reloader restart cycle.
-          expr: kube_statefulset_status_replicas_ready{namespace="apps",statefulset="woodpecker-server"} == 0
+          expr: kube_deployment_status_replicas_ready{namespace="woodpecker",deployment="woodpecker-server"} == 0
           for: 20m
           labels:
             severity: critical
           annotations:
             summary: "Woodpecker CI server has no ready replica"
             description: >-
-              StatefulSet woodpecker-server (ns apps) has had 0 ready replicas for 20m;
-              all CI is down (agents cannot reach woodpecker.apps.svc:9000). If this
-              followed a secret rotation, the server pod failed to return after its
-              Reloader-triggered restart — check `kubectl -n apps describe pod
-              woodpecker-server-0` (PVC attach / crashloop) and the woodpecker-secret
-              ExternalSecret sync status.
+              Deployment woodpecker-server (ns woodpecker) has had 0 ready replicas
+              for 20m; all CI is down (agents cannot reach
+              woodpecker.woodpecker.svc:9000). If this followed a secret rotation, the
+              server pod failed to return after its Reloader-triggered restart — check
+              `kubectl -n woodpecker describe deploy woodpecker-server` and the
+              woodpecker-secret ExternalSecret sync status.
         - alert: WoodpeckerAgentNotReady
-          expr: kube_deployment_status_replicas_available{namespace="apps",deployment="woodpecker-agent"} == 0
+          expr: kube_deployment_status_replicas_available{namespace="woodpecker",deployment="woodpecker-agent"} == 0
           for: 20m
           labels:
             severity: warning
           annotations:
             summary: "Woodpecker CI agent has no available replica"
             description: >-
-              Deployment woodpecker-agent (ns apps) has had 0 available replicas for
+              Deployment woodpecker-agent (ns woodpecker) has had 0 available replicas for
               20m; no pipelines will run. If this followed a secret rotation, the agent
               did not return after its Reloader restart, or its WOODPECKER_AGENT_SECRET
               no longer matches the server's — check the agent pod logs for a gRPC auth
@@ -50,29 +50,29 @@ spec:
         - alert: WoodpeckerSecretRotationJobFailed
           # Named, clearer signal than the generic KubeJobFailed for the OpenBao-write
           # step. Fires while a failed rotation Job is retained (ttl 3d).
-          expr: kube_job_status_failed{namespace="apps",job_name=~"woodpecker-secret-rotate.*"} > 0
+          expr: kube_job_status_failed{namespace="openbao",job_name=~"woodpecker-secret-rotate.*"} > 0
           for: 15m
           labels:
             severity: warning
           annotations:
             summary: "Woodpecker agent-secret rotation Job failed"
             description: >-
-              A woodpecker-secret-rotate Job (ns apps) failed at the OpenBao-write step;
+              A woodpecker-secret-rotate Job (ns openbao) failed at the OpenBao-write step;
               the agent secret was NOT rotated. Most common cause: the one-time OpenBao
               `rotator` role/policy bootstrap was never applied (see
               kubernetes/infrastructure/openbao/policies/rotator.hcl). Check
-              `kubectl -n apps logs job/<name>`.
+              `kubectl -n openbao logs job/<name>`.
         - alert: WoodpeckerSecretRotationStale
           # Quarterly cadence (~91d) + grace. The series is ABSENT until the first
           # successful run, so this cannot false-fire before rotation has ever worked.
-          expr: (time() - kube_cronjob_status_last_successful_time{namespace="apps",cronjob="woodpecker-secret-rotate"}) > 100 * 24 * 3600
+          expr: (time() - kube_cronjob_status_last_successful_time{namespace="openbao",cronjob="woodpecker-secret-rotate"}) > 100 * 24 * 3600
           for: 1h
           labels:
             severity: warning
           annotations:
             summary: "Woodpecker agent secret has not rotated in >100 days"
             description: >-
-              The quarterly woodpecker-secret-rotate CronJob (ns apps) has not completed
+              The quarterly woodpecker-secret-rotate CronJob (ns openbao) has not completed
               successfully in over 100 days — rotation has silently stopped (CronJob
               suspended/deleted, or every run failing). Investigate the CronJob and its
               recent Jobs.

kubernetes/apps/woodpecker/MIGRATION-namespace.md

@@ -146,8 +146,31 @@ kubectl -n apps exec -it wp-pg-1 -- env PGPASSWORD="$PGPASS_OLD" \
   psql -U woodpecker -d woodpecker -c 'select count(*) from repos;'
 ```
 
-They will not match exactly if builds ran after the backup the restore used —
-that is expected, and stage 2 quiesces the server before the final cutover.
+They will not match exactly if builds ran after the backup the restore used.
+
+**And stage 2 does not close that gap.** An earlier draft of this document
+claimed it quiesced the server first; it does not — Flux simply prunes the
+`apps` workloads and creates the `woodpecker` ones, so anything written to the
+old database between the restore and the cutover stays only in the old database.
+
+Measured on the live cutover: the restore captured through pipeline `9488`,
+four more (`9491`, `9492`, `9494`, `9496`) landed in `apps/wp-pg` before the
+swap, and the new cluster resumed at `9521`. Four pipeline records and their
+log entries exist only in the old cluster. No ID collision, no functional
+breakage — Woodpecker's own manifest calls this database "small and
+non-critical (state is largely reconstructable)" — but it is real history loss
+and should be a decision, not a surprise.
+
+If that gap matters for a future migration, quiesce before cutting over:
+
+```sh
+flux suspend kustomization woodpecker
+kubectl -n <old-ns> scale deploy/woodpecker-server deploy/woodpecker-agent --replicas=0
+# re-run the restore so it captures the final state, then resume
+```
+
+The old cluster keeps its `prune: disabled` guard until stage 3, so the missing
+rows remain recoverable until then.
 
 Confirm the two fixes this move exists for:
 
@@ -169,29 +192,190 @@ kubectl -n woodpecker get podmonitor cnpg-wp-pg
 
 **Check the policy semantics by hand.** `kubectl --dry-run=server` validates
 manifest shape, not Cilium behaviour, so it cannot catch a rule that is
-syntactically fine but scoped wider than intended. Confirm what actually reaches
-wp-pg's instance-manager port — the operator should, a CI step pod should not:
+syntactically fine but scoped wider than intended.
+
+`cilium-dbg policy trace` does **not** exist in Cilium 1.20 — it was removed,
+and `cilium-dbg policy` now offers only `get`, `selectors`, `subject-selectors`
+and `wait`. Read the realized BPF policy map instead, which is better evidence
+anyway: it is the datapath as programmed, not a simulation of it, and it
+carries live byte/packet counters.
+
+Find the agent on wp-pg's node and the endpoint id:
 
 ```sh
-CILIUM=$(kubectl -n kube-system get pod -l k8s-app=cilium \
-  --field-selector spec.nodeName=$(kubectl -n woodpecker get pod wp-pg-1 -o jsonpath='{.spec.nodeName}') \
-  -o name | head -1)
+kubectl -n woodpecker get pod wp-pg-1 -o jsonpath='{.spec.nodeName}'
+kubectl -n kube-system get pod -l k8s-app=cilium \
+  --field-selector spec.nodeName=<node> --no-headers -o custom-columns=N:.metadata.name
 
-# Expect ALLOWED
-kubectl -n kube-system exec ${CILIUM#pod/} -c cilium-agent -- cilium-dbg policy trace \
-  --src-k8s-pod database:cloudnative-pg --dst-k8s-pod woodpecker:wp-pg-1 --dport 8000
-
-# Expect DENIED — arbitrary build steps must not reach the database
-kubectl -n kube-system exec ${CILIUM#pod/} -c cilium-agent -- cilium-dbg policy trace \
-  --src-k8s-pod woodpecker-ci:any --dst-k8s-pod woodpecker:wp-pg-1 --dport 8000
+kubectl -n kube-system exec <cilium-pod> -c cilium-agent -- \
+  cilium-dbg endpoint list -o json \
+  | jq -r '.[] | select((.status.labels."security-relevant"[]?|tostring)
+      | test("cnpg.io/cluster=wp-pg")) | "\(.id) \(.status.networking.addressing[0].ipv4)"'
 ```
 
+Then, on that endpoint id — who may reach `:8000`:
+
+```sh
+kubectl -n kube-system exec <cilium-pod> -c cilium-agent -- \
+  cilium-dbg bpf policy get <endpoint-id> | awk '$4 ~ /8000/ || NR<=1'
+```
+
+Expect exactly one row, the CNPG operator, with non-zero counters once it has
+polled:
+
+```
+Allow  Ingress  k8s:app.kubernetes.io/instance=cloudnative-pg  8000/TCP  11756  58
+```
+
+And confirm CI step pods reach **ICMP only** — never a TCP port:
+
+```sh
+kubectl -n kube-system exec <cilium-pod> -c cilium-agent -- \
+  cilium-dbg bpf policy get <endpoint-id> | grep woodpecker-ci | awk '{print $4}' | sort -u
+```
+
+Every line must be an ICMP/ICMPv6 type. A bare port number here means the
+namespace baseline has grown a rule it should not have.
+
+Note the operator's allow is also confirmed indirectly and more strongly by
+`Ready=True` on the Cluster: that condition only goes true once the operator has
+successfully polled this exact endpoint and port.
+
 And that archiving goes to the new prefix, leaving the old one alone:
 
 ```sh
 kubectl -n woodpecker get cluster wp-pg -o jsonpath='{.status.conditions[?(@.type=="ContinuousArchiving")]}'
 ```
 
+## Stage 2 — the cutover
+
+Moves server, agent, service, ingress, pdb, the agent ServiceAccount, the
+`woodpecker-secret` ExternalSecret and the rotator policy into `woodpecker`,
+points the server's DSN at `wp-pg-rw.woodpecker.svc`, repoints the agent's gRPC
+at `woodpecker.woodpecker.svc:9000`, moves the openbao rotators' egress, and
+adds the `woodpecker-ci` → server `:8000` policy that stage 1 deliberately
+deferred — scoped to the server pod, not the namespace baseline.
+
+**This is the downtime window.** Flux prunes the `apps` copies and creates the
+`woodpecker` ones; `ci.renner.uno` is down in between, and in-flight pipelines
+die with their agent.
+
+**The outage outlasts the pods becoming Ready, because of TLS.** Certificates
+are namespace-scoped, so moving the Ingress does not move `woodpecker-tls` —
+cert-manager issues a fresh one in the new namespace. Until it completes,
+Traefik serves a default certificate and clients fail before they get an HTTP
+status at all:
+
+```
+curl: (60) SSL: no alternative certificate subject name matches target hostname 'ci.renner.uno'
+```
+
+That is expected, not a routing or policy fault. Watch it clear with:
+
+```sh
+kubectl -n woodpecker get certificate woodpecker-tls -w
+```
+
+`letsencrypt-cluster` solves **DNS-01** via Cloudflare, so there is no in-cluster
+ACME solver pod and no HTTP challenge to admit through the network policies —
+worth knowing before scoping any ingress rule, since an HTTP-01 issuer would
+need the solver reachable too.
+
+The four CNPG files (`cluster.yaml`, `objectstore.yaml`, `scheduled-backup.yaml`,
+`hetzner-creds-externalsecret.yaml`) still describe the **original** apps-namespace
+cluster and are deliberately untouched — they are the rollback. Stage 3 retires
+them.
+
+### Verifying stage 2
+
+```sh
+kubectl -n woodpecker get pods
+kubectl -n apps get pods -l app.kubernetes.io/name=woodpecker   # expect none
+```
+
+The server must reach the restored database, not the old one:
+
+```sh
+kubectl -n woodpecker logs deploy/woodpecker-server --tail=40 | grep -i "database\|migrat\|error"
+kubectl -n woodpecker exec deploy/woodpecker-server -- env | grep DATASOURCE
+```
+
+Then confirm the UI and a real pipeline, which is the only check that exercises
+server → agent → step pod → server end to end:
+
+```sh
+curl -sS -o /dev/null -w '%{http_code}\n' https://ci.renner.uno/
+woodpecker pipeline create --branch master --var PIPELINE=e2e-test k.renner/homelab
+```
+
+Step pods must still reach the server on `:8000` while still being unable to
+reach wp-pg on any TCP port — re-run the `bpf policy get` checks above against
+both the server's and wp-pg's endpoint ids.
+
+Check the rotators can still reach the server, since their egress moved:
+
+```sh
+kubectl -n openbao get cronjob
+kubectl -n openbao logs -l app.kubernetes.io/component=secret-rotator --tail=20
+```
+
+### Rolling back stage 2
+
+Revert the PR. Flux recreates the `apps` workloads and the DSN points back at
+`wp-pg-rw.apps.svc`, which never stopped running. Data written to the new
+cluster during the window is lost, so roll back promptly or not at all — if
+pipelines have run against `woodpecker/wp-pg`, prefer fixing forward.
+
+## Stage 3 — retiring the old cluster
+
+Deletes the four `apps` manifests, promotes the `migration-*` set onto their
+names, and gives the surviving Cluster the `prune: disabled` data guard it was
+deliberately denied while a rollback still existed.
+
+**Merging stage 3 does not delete anything.** The old resources carry
+`kustomize.toolkit.fluxcd.io/prune: disabled`, so dropping them from git only
+stops Flux managing them — they keep running, and keep occupying their PVC.
+That is the point: the PR is reversible, the teardown is a separate, deliberate
+act.
+
+Before deleting anything, confirm the new cluster is genuinely standing on its
+own — that it has taken at least one base backup to its own prefix, so recovery
+no longer depends on the retired cluster's data:
+
+```sh
+kubectl -n woodpecker get backup.postgresql.cnpg.io
+kubectl -n woodpecker get cluster wp-pg -o jsonpath='{.status.conditions}'
+```
+
+`LastBackupSucceeded` must be `True`, not merely `BackupStarted`.
+
+Then retire the old set:
+
+```sh
+kubectl -n apps delete scheduledbackup.postgresql.cnpg.io wp-pg-daily
+kubectl -n apps delete cluster.postgresql.cnpg.io wp-pg          # deletes its PVCs
+kubectl -n apps delete objectstore.barmancloud.cnpg.io wp-pg-store
+kubectl -n apps delete externalsecret hetzner-s3-creds
+```
+
+Delete the ScheduledBackup first: leaving it while the Cluster goes away
+produces failing backup jobs against a cluster that no longer exists.
+
+**What this destroys and what it does not.** It destroys the old cluster's PVCs
+— including the four pipeline records that only ever existed there (see the
+stage-1 verification section). It does **not** touch
+`s3://renner-uno-k8s-cnpg/wp-pg`: that prefix keeps its backups until their own
+retention expires, and `wp-pg-store-source` is retained precisely so it stays
+restorable. That is the real safety net, not the running cluster.
+
+### After the teardown
+
+The `apps` namespace should have no woodpecker-related objects left:
+
+```sh
+kubectl -n apps get cluster,objectstore,scheduledbackup,externalsecret 2>/dev/null | grep -i "wp-pg\|hetzner" || echo "clean"
+```
+
 ## Rolling back stage 1
 
 Delete the four resources and the namespace. Nothing references them, and the

kubernetes/apps/woodpecker/agent.yaml

@@ -2,7 +2,7 @@ apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: woodpecker-agent
-  namespace: apps
+  namespace: woodpecker
   labels:
     app.kubernetes.io/name: woodpecker
     app.kubernetes.io/component: agent
@@ -12,7 +12,7 @@ metadata:
     secret.reloader.stakater.com/reload: woodpecker-secret
 spec:
   # 1: the in-cluster server owns ci.renner.uno now, so this agent registers with
-  # it (woodpecker.apps.svc:9000) and runs pipelines on the kubernetes backend,
+  # it (woodpecker.woodpecker.svc:9000) and runs pipelines on the kubernetes backend,
   # replacing the retired docker agents (LXC 910/911).
   replicas: 1
   selector:
@@ -37,7 +37,7 @@ spec:
           image: woodpeckerci/woodpecker-agent:v3.17.0 # {"$imagepolicy": "flux-system:woodpecker-agent"}
           env:
             - name: WOODPECKER_SERVER
-              value: woodpecker.apps.svc.cluster.local:9000 # in-cluster gRPC, replaces LXC :9000
+              value: woodpecker.woodpecker.svc.cluster.local:9000 # in-cluster gRPC, replaces LXC :9000
             - name: WOODPECKER_MAX_WORKFLOWS
               value: "2"
             # --- kubernetes backend: pipeline steps run as native pods (retires

kubernetes/apps/woodpecker/cluster.yaml

@@ -1,19 +1,27 @@
+---
+# Woodpecker's Postgres backend. Serves the server at wp-pg-rw.woodpecker.svc.
+#
+# Originally created in `apps` with bootstrap.initdb; moved here by restoring
+# from barman into the new namespace rather than by editing a namespace field,
+# because PVCs are namespace-scoped and a Cluster cannot be relocated in place
+# (issue #433, MIGRATION-namespace.md).
+#
+# The bootstrap.recovery stanza and its externalCluster are retained rather than
+# reverted to initdb. CNPG honours `bootstrap` only when PGDATA is absent, so on
+# this live cluster it is inert — and keeping it means a cold rebuild of an
+# empty cluster restores from backup instead of coming up empty, which is the
+# gap kubernetes/RECOVERY.md exists to close for the other CNPG clusters.
 apiVersion: postgresql.cnpg.io/v1
 kind: Cluster
 metadata:
   name: wp-pg
-  namespace: apps
+  namespace: woodpecker
   annotations:
-    # CRITICAL data-guard: prune-disabled so the seed-woodpecker -> dedicated
-    # woodpecker KS ownership handoff can never prune this Cluster (which would
-    # delete its data PVCs). Landed here first, via seed-woodpecker, so it's
-    # live before the move. Kept permanently as a standing guard.
+    # Data guard, earned at stage 3 now that this is the only copy: prevents a
+    # Flux prune from deleting the Cluster and with it the data PVCs. The
+    # migration deliberately withheld this while a rollback still existed.
     kustomize.toolkit.fluxcd.io/prune: disabled
 spec:
-  # Single instance: the Woodpecker CI DB is small and non-critical (state is
-  # largely reconstructable), so HA isn't warranted — but WAL archiving to
-  # object storage still gives point-in-time backups. Mirrors grafana-db's
-  # single-instance shape, with barman enabled like immich/paperless.
   instances: 1
   imageName: ghcr.io/cloudnative-pg/postgresql:16
   storage:
@@ -25,9 +33,25 @@ spec:
       parameters:
         barmanObjectName: wp-pg-store
   bootstrap:
-    initdb:
+    recovery:
+      source: wp-pg-apps
+      # Must mirror the initdb values of the cluster this restores from
+      # (apps/wp-pg: database woodpecker, owner woodpecker). CNPG defaults
+      # `database` to `app` and `owner` to whatever `database` is, and those
+      # names do not exist in the physically restored data — it would provision
+      # an empty `app` database, point the generated wp-pg-app Secret and the
+      # -app Service at it, and leave the real data unreachable through them.
+      # Same pattern as every recovery example in kubernetes/RECOVERY.md.
       database: woodpecker
       owner: woodpecker
+  externalClusters:
+    - name: wp-pg-apps
+      plugin:
+        name: barman-cloud.cloudnative-pg.io
+        parameters:
+          barmanObjectName: wp-pg-store-source
+          # The cluster that produced the backup, not this one.
+          serverName: wp-pg
   postgresql:
     parameters:
       shared_buffers: "128MB"

kubernetes/apps/woodpecker/externalsecret.yaml

@@ -2,7 +2,7 @@ apiVersion: external-secrets.io/v1
 kind: ExternalSecret
 metadata:
   name: woodpecker-secret
-  namespace: apps
+  namespace: woodpecker
 spec:
   # 10m (was 1h) so an automated rotation (rotate-agent-secret.yaml writes a new
   # value to OpenBao) propagates into the k8s Secret within ~10m without a manual

kubernetes/apps/woodpecker/hetzner-creds-externalsecret.yaml

@@ -1,11 +1,20 @@
+---
+# Barman S3 credentials for wp-pg.
+#
+# Sourced from seed/paperless/hetzner-creds, which is not where they belong.
+# Issue #397 wants woodpecker to own seed/woodpecker/hetzner-creds; creating
+# that path needs OpenBao write access (`bao kv list seed` → 403 for the
+# identities available here), so it stays open. The values are identical, so it
+# is a one-line swap of the `key:` below once the path exists.
+#
+# The namespace allowlist on the openbao ClusterSecretStore
+# (infrastructure/post-crds/openbao-clustersecretstore.yaml) must include this
+# namespace, or ESO silently never syncs — see #505.
 apiVersion: external-secrets.io/v1
 kind: ExternalSecret
 metadata:
   name: hetzner-s3-creds
-  namespace: apps
-  annotations:
-    # Prune-disabled — a standing guard kept from the seed->dedicated-KS handoff (barman S3 creds).
-    kustomize.toolkit.fluxcd.io/prune: disabled
+  namespace: woodpecker
 spec:
   refreshInterval: 1h
   secretStoreRef:
@@ -14,10 +23,5 @@ spec:
   target:
     name: hetzner-s3-creds
   dataFrom:
-    # Reuses the paperless copy of the shared Hetzner S3 credentials (identical
-    # ACCESS_KEY_ID / ACCESS_SECRET_KEY / REGION) to avoid seeding a new OpenBao
-    # path for this migration. FOLLOW-UP: give woodpecker its own
-    # seed/woodpecker/hetzner-creds path once OpenBao can be written, and repoint
-    # this here (the value is the same, so it's a drop-in swap).
     - extract:
         key: seed/paperless/hetzner-creds

kubernetes/apps/woodpecker/ingress.yaml

@@ -2,7 +2,7 @@ apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
   name: woodpecker
-  namespace: apps
+  namespace: woodpecker
   labels:
     app.kubernetes.io/name: woodpecker
   annotations:

kubernetes/apps/woodpecker/kustomization.yaml

@@ -21,26 +21,20 @@ resources:
   # (kubernetes/infrastructure/openbao/rotation/woodpecker.yaml) — intra-ns
   # OpenBao access, no NetworkPolicy needed. server.yaml/agent.yaml keep their
   # Reloader annotations; externalsecret.yaml keeps the 10m refreshInterval.
-  # wp-pg CNPG Postgres backend (moved here from the retired seed-woodpecker KS;
-  # each object carries prune: disabled so ownership transferred without a
-  # teardown). The dedicated `woodpecker` KS depends on infrastructure-cnpg-barman
-  # for the barman plugin behind the ObjectStore/ScheduledBackup.
-  - cluster.yaml
-  - objectstore.yaml
-  - scheduled-backup.yaml
-  - hetzner-creds-externalsecret.yaml
-  # --- namespace migration, stage 1 (issue #433) ---
-  # Woodpecker is moving out of the shared `apps` namespace into its own.
-  # These resources restore wp-pg into the target namespace from barman,
-  # running in parallel with the live cluster. Nothing points at them yet, so
-  # this stage carries no CI impact and is revertible by deleting them.
-  # Cutover (server/agent/DSN/RBAC) is stage 2; removing the `apps` originals
-  # is stage 3. Runbook: MIGRATION-namespace.md.
+  # --- namespace migration, stages 1-2 (issue #433) ---
+  # Stage 2 has moved the workloads above (server, agent, service, ingress, pdb,
+  # rbac, secrets, policies) into the `woodpecker` namespace and pointed the
+  # server's DSN at wp-pg-rw.woodpecker.svc. Stage 3 retired the `apps`
+  # originals; these four now describe the surviving cluster. History and the
+  # reasoning behind the three stages: MIGRATION-namespace.md.
   #
   # The `woodpecker` Namespace itself is declared in infrastructure/namespaces,
   # not here: it must exist before the `infrastructure-crds` KS applies the
   # namespace's CiliumNetworkPolicies, and this KS runs after both of those.
-  - migration-externalsecret-hetzner.yaml
-  - migration-objectstore.yaml
-  - migration-cluster.yaml
-  - migration-scheduled-backup.yaml
+  #
+  # This KS depends on infrastructure-cnpg-barman for the barman plugin behind
+  # the ObjectStore and ScheduledBackup.
+  - hetzner-creds-externalsecret.yaml
+  - objectstore.yaml
+  - cluster.yaml
+  - scheduled-backup.yaml

kubernetes/apps/woodpecker/migration-cluster.yaml

@@ -1,62 +0,0 @@
----
-# wp-pg restored into the `woodpecker` namespace from the barman backup of the
-# live apps/wp-pg. Runs in parallel with the original until cutover (PR 2);
-# nothing points at it yet, so it carries no traffic.
-#
-# Spec mirrors apps/wp-pg exactly — same instance count, image, storage class
-# and Postgres parameters — so the only variable under test is the restore
-# itself. The differences are deliberate and limited to:
-#   - bootstrap: recovery (not initdb), sourced from the old prefix
-#   - plugins.barmanObjectName: the NEW store, so it archives somewhere else
-#
-# NOT prune-disabled, unlike apps/wp-pg. That guard exists there because the
-# original holds the only copy of the data through a KS ownership handoff. This
-# one is reconstructable by re-running the restore, and a prune guard on a
-# migration artifact is how orphans get left behind if the migration is
-# abandoned. It gains the guard in PR 3, once it is the surviving cluster.
-apiVersion: postgresql.cnpg.io/v1
-kind: Cluster
-metadata:
-  name: wp-pg
-  namespace: woodpecker
-spec:
-  instances: 1
-  imageName: ghcr.io/cloudnative-pg/postgresql:16
-  storage:
-    storageClass: ceph-block
-    size: 5Gi
-  plugins:
-    - name: barman-cloud.cloudnative-pg.io
-      isWALArchiver: true
-      parameters:
-        barmanObjectName: wp-pg-store
-  bootstrap:
-    recovery:
-      source: wp-pg-apps
-      # Must mirror the initdb values of the cluster this restores from
-      # (apps/wp-pg: database woodpecker, owner woodpecker). CNPG defaults
-      # `database` to `app` and `owner` to whatever `database` is, and those
-      # names do not exist in the physically restored data — it would provision
-      # an empty `app` database, point the generated wp-pg-app Secret and the
-      # -app Service at it, and leave the real data unreachable through them.
-      # Same pattern as every recovery example in kubernetes/RECOVERY.md.
-      database: woodpecker
-      owner: woodpecker
-  externalClusters:
-    - name: wp-pg-apps
-      plugin:
-        name: barman-cloud.cloudnative-pg.io
-        parameters:
-          barmanObjectName: wp-pg-store-source
-          # The cluster that produced the backup, not this one.
-          serverName: wp-pg
-  postgresql:
-    parameters:
-      shared_buffers: "128MB"
-      max_connections: "50"
-  resources:
-    requests:
-      cpu: 100m
-      memory: 256Mi
-    limits:
-      memory: 512Mi

kubernetes/apps/woodpecker/migration-externalsecret-hetzner.yaml

@@ -1,28 +0,0 @@
----
-# Barman S3 credentials for the wp-pg cluster in its new `woodpecker` home.
-# Secrets are namespace-scoped, so the copy in `apps` cannot be reused; this is
-# the same material fetched into the new namespace.
-#
-# Still sourced from seed/paperless/hetzner-creds, deliberately. Issue #397
-# wants woodpecker to own seed/woodpecker/hetzner-creds instead, and this
-# migration would have been the natural moment — but creating that path needs
-# OpenBao write access that is not available here (`bao kv list seed` → 403).
-# Repointing at a path that may not exist would leave the migrated cluster
-# unable to archive WAL, which is exactly what the restore is meant to prove
-# works. #397 stays open and is a drop-in swap of the `key:` below once the
-# path exists; the values are identical.
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: hetzner-s3-creds
-  namespace: woodpecker
-spec:
-  refreshInterval: 1h
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: hetzner-s3-creds
-  dataFrom:
-    - extract:
-        key: seed/paperless/hetzner-creds

kubernetes/apps/woodpecker/migration-objectstore.yaml

@@ -1,64 +0,0 @@
----
-# RECOVERY SOURCE — read-only. Points at the prefix the live `apps/wp-pg`
-# cluster has been archiving to since it was created. The restored cluster
-# bootstraps from here and never writes to it.
-#
-# serverName on the consuming externalCluster must be `wp-pg`, the name of the
-# cluster that produced these backups.
-apiVersion: barmancloud.cnpg.io/v1
-kind: ObjectStore
-metadata:
-  name: wp-pg-store-source
-  namespace: woodpecker
-spec:
-  configuration:
-    destinationPath: s3://renner-uno-k8s-cnpg/wp-pg
-    endpointURL: https://fsn1.your-objectstorage.com
-    s3Credentials:
-      accessKeyId:
-        name: hetzner-s3-creds
-        key: ACCESS_KEY_ID
-      secretAccessKey:
-        name: hetzner-s3-creds
-        key: ACCESS_SECRET_KEY
-      region:
-        name: hetzner-s3-creds
-        key: REGION
-    wal:
-      compression: gzip
-    data:
-      compression: gzip
-  # No retentionPolicy: this store is a read source during the migration, and
-  # the old cluster in `apps` owns the lifecycle of the data underneath it.
----
-# ARCHIVING TARGET — a DIFFERENT prefix from the source, deliberately.
-#
-# Two clusters archiving WAL to one destinationPath interleave their streams
-# and corrupt both. During the parallel-run window the old apps/wp-pg is still
-# live and still archiving to s3://renner-uno-k8s-cnpg/wp-pg, so the restored
-# cluster must not share it. Splitting the paths also keeps the pre-migration
-# backup intact and independently restorable as a rollback.
-apiVersion: barmancloud.cnpg.io/v1
-kind: ObjectStore
-metadata:
-  name: wp-pg-store
-  namespace: woodpecker
-spec:
-  configuration:
-    destinationPath: s3://renner-uno-k8s-cnpg/wp-pg-woodpecker
-    endpointURL: https://fsn1.your-objectstorage.com
-    s3Credentials:
-      accessKeyId:
-        name: hetzner-s3-creds
-        key: ACCESS_KEY_ID
-      secretAccessKey:
-        name: hetzner-s3-creds
-        key: ACCESS_SECRET_KEY
-      region:
-        name: hetzner-s3-creds
-        key: REGION
-    wal:
-      compression: gzip
-    data:
-      compression: gzip
-  retentionPolicy: "30d"

kubernetes/apps/woodpecker/migration-scheduled-backup.yaml

@@ -1,32 +0,0 @@
----
-# Base backups for the restored cluster, onto the new wp-pg-woodpecker prefix.
-#
-# Without this the cluster archives continuous WAL to that prefix but never
-# takes a base backup there, which leaves two problems: the prefix has no
-# independent recovery point of its own (recovery would have to chain all the
-# way back through the source prefix), and wp-pg-store's retentionPolicy: 30d
-# has no base backups to prune against, so the WAL chain grows unbounded for as
-# long as stage 1 sits before cutover.
-#
-# 02:40 UTC, offset from the apps cluster's 02:35 and the other clusters' 02:30
-# so the parallel-run window does not put two woodpecker base backups against
-# the Hetzner bucket at once. immediate: true takes the first as soon as the
-# restore finishes, which is also the cheapest confirmation that the new
-# credentials and the new ObjectStore both work end to end.
-#
-# Not prune-disabled, for the same reason as migration-cluster.yaml: this is a
-# migration artifact until stage 3 promotes it.
-apiVersion: postgresql.cnpg.io/v1
-kind: ScheduledBackup
-metadata:
-  name: wp-pg-daily
-  namespace: woodpecker
-spec:
-  schedule: "0 40 2 * * *"
-  immediate: true
-  backupOwnerReference: self
-  cluster:
-    name: wp-pg
-  method: plugin
-  pluginConfiguration:
-    name: barman-cloud.cloudnative-pg.io

kubernetes/apps/woodpecker/networkpolicy.yaml

@@ -1,20 +1,23 @@
 # Secret rotation: the review-bot and claude-oauth rotators (openbao ns) reach
 # the Woodpecker SERVER to upsert org-level CI secrets (review_bot_token,
 # claude_code_oauth_token). They use the in-cluster Service
-# (woodpecker.apps.svc:8000) — the public ci.renner.uno name hairpins off the
-# ingress LB from a pod. The apps default-deny-ingress only allows same-ns /
-# host / a few named sources, so without this the rotator's connect to :8000 is
+# (woodpecker.woodpecker.svc:8000) — the public ci.renner.uno name hairpins off the
+# ingress LB from a pod. This namespace's baseline
+# (network-policies/woodpecker-ns.yaml) admits only same-namespace, monitoring,
+# host/remote-node and ICMP, so without this the rotator's connect to :8000 is
 # dropped. Scoped to the woodpecker SERVER pod :8000 and the two rotator labels;
 # the egress halves are rotator-review-bot-egress + rotator-claude-oauth-egress
 # in network-policies/openbao-ns.yaml.
 #
-# This is additive to the apps-ns baseline allows (allow-ingress.yaml selects all
-# apps pods), so the server's existing ingress — Traefik, agents — is unaffected.
+# Cilium unions allow rules, so this composes with the baseline and with the
+# sibling policies below (traefik, CI steps) rather than replacing them. Note
+# allow-ingress.yaml does NOT apply here — it is hard-scoped to `namespace:
+# apps`, which is why traefik needs its own rule in this file now.
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
 metadata:
   name: woodpecker-allow-rotators
-  namespace: apps
+  namespace: woodpecker
 spec:
   endpointSelector:
     matchLabels:
@@ -34,3 +37,25 @@ spec:
         - ports:
             - port: "8000"
               protocol: TCP
+---
+# Pipeline step pods call the server back on :8000 to stream logs and report
+# status. Deferred from stage 1 on purpose, and scoped to the SERVER pod rather
+# than granted on the namespace baseline.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: woodpecker-allow-ci-steps
+  namespace: woodpecker
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: woodpecker
+      app.kubernetes.io/component: server
+  ingress:
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": woodpecker-ci
+      toPorts:
+        - ports:
+            - port: "8000"
+              protocol: TCP

kubernetes/apps/woodpecker/objectstore.yaml

@@ -1,18 +1,20 @@
 ---
+# RECOVERY SOURCE — read-only. The prefix the retired `apps/wp-pg` archived to.
+#
+# Kept after the migration on purpose. Nothing writes to it, and the backups
+# under it age out on their own retention, but while they last this is a second,
+# independent restore point that predates the move — worth having until the new
+# prefix has a meaningful backup history of its own.
+#
+# serverName on the consuming externalCluster must be `wp-pg`, the name of the
+# cluster that produced these backups.
 apiVersion: barmancloud.cnpg.io/v1
 kind: ObjectStore
 metadata:
-  name: wp-pg-store
-  namespace: apps
-  annotations:
-    # Prune-disabled — a standing guard kept from the seed->dedicated-KS handoff (backup target).
-    kustomize.toolkit.fluxcd.io/prune: disabled
+  name: wp-pg-store-source
+  namespace: woodpecker
 spec:
   configuration:
-    # Same offsite Hetzner Object Storage bucket as the other CNPG clusters,
-    # under a per-cluster prefix. See apps/paperless/objectstore.yaml for the
-    # SSE / addressing rationale (Hetzner is SSE-C only; transit is TLS,
-    # bucket private).
     destinationPath: s3://renner-uno-k8s-cnpg/wp-pg
     endpointURL: https://fsn1.your-objectstorage.com
     s3Credentials:
@@ -29,4 +31,36 @@ spec:
       compression: gzip
     data:
       compression: gzip
+  # No retentionPolicy: this store is a read source during the migration, and
+  # the old cluster in `apps` owns the lifecycle of the data underneath it.
+---
+# ARCHIVING TARGET — a DIFFERENT prefix from the source, deliberately.
+#
+# Two clusters archiving WAL to one destinationPath interleave their streams and
+# corrupt both. That mattered during the parallel-run window; it still matters
+# now, because the source prefix above is retained and must stay frozen for the
+# independent restore point it provides.
+apiVersion: barmancloud.cnpg.io/v1
+kind: ObjectStore
+metadata:
+  name: wp-pg-store
+  namespace: woodpecker
+spec:
+  configuration:
+    destinationPath: s3://renner-uno-k8s-cnpg/wp-pg-woodpecker
+    endpointURL: https://fsn1.your-objectstorage.com
+    s3Credentials:
+      accessKeyId:
+        name: hetzner-s3-creds
+        key: ACCESS_KEY_ID
+      secretAccessKey:
+        name: hetzner-s3-creds
+        key: ACCESS_SECRET_KEY
+      region:
+        name: hetzner-s3-creds
+        key: REGION
+    wal:
+      compression: gzip
+    data:
+      compression: gzip
   retentionPolicy: "30d"

kubernetes/apps/woodpecker/pdb.yaml

@@ -15,7 +15,7 @@ apiVersion: policy/v1
 kind: PodDisruptionBudget
 metadata:
   name: woodpecker-server
-  namespace: apps
+  namespace: woodpecker
   labels:
     app.kubernetes.io/name: woodpecker
     app.kubernetes.io/component: server

kubernetes/apps/woodpecker/rbac.yaml

@@ -17,7 +17,7 @@ apiVersion: v1
 kind: ServiceAccount
 metadata:
   name: woodpecker-agent
-  namespace: apps
+  namespace: woodpecker
   labels:
     app.kubernetes.io/name: woodpecker
     app.kubernetes.io/component: agent
@@ -60,4 +60,7 @@ roleRef:
 subjects:
   - kind: ServiceAccount
     name: woodpecker-agent
-    namespace: apps
+    # The agent's SA lives in `woodpecker`; this Role/RoleBinding stays in
+    # `woodpecker-ci`, because that is the namespace it grants pod
+    # create/delete in. Cross-namespace subject, deliberately.
+    namespace: woodpecker

kubernetes/apps/woodpecker/scheduled-backup.yaml

@@ -1,18 +1,19 @@
 ---
-# Daily base backup at 02:35 UTC (offset from the other clusters' 02:30 to
-# avoid a thundering herd against the Hetzner bucket). WAL archiving runs
-# continuously (see cluster.yaml spec.plugins[].isWALArchiver). immediate:true
-# takes the first base backup as soon as this CR applies.
+# Daily base backups onto the wp-pg-woodpecker prefix. Continuous WAL archiving
+# runs separately (cluster.yaml spec.plugins[].isWALArchiver); without periodic
+# base backups the prefix would have no recovery point of its own and
+# wp-pg-store's retentionPolicy: 30d would have nothing to prune against, so the
+# WAL chain would grow without bound.
+#
+# 02:40 UTC, offset from the other CNPG clusters' 02:30 so they do not all hit
+# the Hetzner bucket together.
 apiVersion: postgresql.cnpg.io/v1
 kind: ScheduledBackup
 metadata:
   name: wp-pg-daily
-  namespace: apps
-  annotations:
-    # Prune-disabled — a standing guard kept from the seed->dedicated-KS handoff (backup schedule).
-    kustomize.toolkit.fluxcd.io/prune: disabled
+  namespace: woodpecker
 spec:
-  schedule: "0 35 2 * * *"
+  schedule: "0 40 2 * * *"
   immediate: true
   backupOwnerReference: self
   cluster:

kubernetes/apps/woodpecker/secret.example.yaml

@@ -14,7 +14,7 @@ apiVersion: v1
 kind: Secret
 metadata:
   name: woodpecker-secret
-  namespace: apps
+  namespace: woodpecker
   labels:
     app.kubernetes.io/name: woodpecker
 type: Opaque

kubernetes/apps/woodpecker/server.yaml

@@ -2,7 +2,7 @@ apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: woodpecker-server
-  namespace: apps
+  namespace: woodpecker
   labels:
     app.kubernetes.io/name: woodpecker
     app.kubernetes.io/component: server
@@ -84,7 +84,7 @@ spec:
             - name: WOODPECKER_DATABASE_DRIVER
               value: postgres
             - name: WOODPECKER_DATABASE_DATASOURCE
-              value: "host=wp-pg-rw.apps.svc.cluster.local port=5432 user=$(WP_DB_USER) password=$(WP_DB_PASS) dbname=woodpecker sslmode=require"
+              value: "host=wp-pg-rw.woodpecker.svc.cluster.local port=5432 user=$(WP_DB_USER) password=$(WP_DB_PASS) dbname=woodpecker sslmode=require"
           # AGENT_SECRET + GITEA_CLIENT + GITEA_SECRET come from the SOPS secret.
           envFrom:
             - secretRef:

kubernetes/apps/woodpecker/service.yaml

@@ -2,13 +2,13 @@ apiVersion: v1
 kind: Service
 metadata:
   name: woodpecker
-  namespace: apps
+  namespace: woodpecker
   labels:
     app.kubernetes.io/name: woodpecker
 spec:
   # Flipped ExternalName -> ClusterIP: ci.renner.uno (via the Ingress, by name)
   # and the in-cluster agent both target this Service now. http:8000 serves the
-  # UI/API; grpc:9000 is how the agent (woodpecker.apps.svc:9000) reaches the
+  # UI/API; grpc:9000 is how the agent (woodpecker.woodpecker.svc:9000) reaches the
   # server. Selects the server pod only (not the agent).
   type: ClusterIP
   selector:

kubernetes/infrastructure/network-policies/openbao-ns.yaml

@@ -141,9 +141,9 @@ spec:
 ---
 # Egress for the review-bot rotator. Its CronJob has two steps: the init step
 # mints review-bot's PAT against gitea (apps :3000), the main step upserts the
-# Woodpecker org secret against the woodpecker server (apps :8000). Both run
+# Woodpecker org secret against the woodpecker server (woodpecker :8000). Both run
 # under the same pod labels, so one egress policy covers both hops. Complements
-# gitea-allow-review-bot-rotator + woodpecker-allow-rotators (apps side).
+# gitea-allow-review-bot-rotator (apps) + woodpecker-allow-rotators (woodpecker).
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
 metadata:
@@ -165,16 +165,16 @@ spec:
               protocol: TCP
     - toEndpoints:
         - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": apps
+            "k8s:io.kubernetes.pod.namespace": woodpecker
             app.kubernetes.io/name: woodpecker
       toPorts:
         - ports:
             - port: "8000"
               protocol: TCP
 ---
-# Egress for the claude-oauth sync rotator: woodpecker server only (apps :8000).
+# Egress for the claude-oauth sync rotator: woodpecker server only (woodpecker :8000).
 # It never touches gitea (the OAuth value is seeded into OpenBao by hand, not
-# minted). Complements woodpecker-allow-rotators (apps side).
+# minted). Complements woodpecker-allow-rotators (woodpecker side).
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
 metadata:
@@ -188,7 +188,7 @@ spec:
   egress:
     - toEndpoints:
         - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": apps
+            "k8s:io.kubernetes.pod.namespace": woodpecker
             app.kubernetes.io/name: woodpecker
       toPorts:
         - ports:

kubernetes/infrastructure/network-policies/woodpecker-ns.yaml

@@ -32,10 +32,11 @@ spec:
     - fromEndpoints:
         - matchLabels:
             "k8s:io.kubernetes.pod.namespace": woodpecker
-    # Traefik terminates ci.renner.uno and forwards to the server (stage 2).
-    - fromEndpoints:
-        - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": ingress
+    # NOT here: ingress from the `ingress` namespace. Traefik terminates
+    # ci.renner.uno and forwards to the server, but that namespace also holds
+    # netbird-ingress, and this policy's endpointSelector:{} would extend the
+    # grant to wp-pg and the agent as well. Scoped to traefik -> server in the
+    # companion policy at the bottom of this file.
     # Prometheus scrape. Namespace-wide and port-unrestricted, matching the
     # other *-ns baselines; this is what covers wp-pg's :9187 metrics.
     - fromEndpoints:
@@ -118,3 +119,38 @@ spec:
         - ports:
             - port: "8000"
               protocol: TCP
+---
+# Traefik terminates ci.renner.uno and forwards to the server on :8000.
+#
+# Lives here, beside the baseline, rather than with the workload policies in
+# apps/woodpecker/networkpolicy.yaml — deliberately. The baseline's `ingress`
+# admit was removed in the same commit, and these two ship in the same
+# Kustomization (infrastructure-crds), so the swap lands in one reconcile. Split
+# across Kustomizations it would not: infrastructure-crds reconciles before the
+# `woodpecker` KS, so the admit would be withdrawn minutes before its
+# replacement arrived, and ci.renner.uno would 503 in the gap.
+#
+# Scoped to traefik and to the server pod. The `ingress` namespace also holds
+# netbird-ingress, and the baseline's endpointSelector:{} would have extended a
+# namespace-level admit to wp-pg and the agent. The `apps` policy this replaces
+# (allow-ingress.yaml) matches traefik by label for the same reason, and does
+# NOT apply here — it is hard-scoped to `namespace: apps`.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: woodpecker-allow-traefik
+  namespace: woodpecker
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: woodpecker
+      app.kubernetes.io/component: server
+  ingress:
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": ingress
+            app.kubernetes.io/name: traefik
+      toPorts:
+        - ports:
+            - port: "8000"
+              protocol: TCP

kubernetes/infrastructure/openbao/rotation/claude-oauth.yaml

@@ -14,8 +14,10 @@
 # immediately:
 #   kubectl -n openbao create job --from=cronjob/claude-oauth-secret-sync claude-oauth-sync-now
 #
-# woodpecker is reached in-cluster at woodpecker.apps.svc:8000. Needs the netpol
-# halves woodpecker-allow-rotators (apps/*) + rotator-claude-oauth-egress
+# woodpecker is reached in-cluster at woodpecker.woodpecker.svc:8000 (it moved
+# out of `apps` in the namespace migration — see
+# kubernetes/apps/woodpecker/MIGRATION-namespace.md). Needs the netpol
+# halves woodpecker-allow-rotators (woodpecker ns) + rotator-claude-oauth-egress
 # (openbao-ns.yaml). OpenBao policy/role rotator-claude-oauth + the bootstraps
 # (woodpecker-bot token, the seeded OAuth value) are provisioned by hand — see
 # policies/rotator-claude-oauth.hcl.
@@ -89,7 +91,7 @@ spec:
                 - name: BAO_FIELD
                   value: setup-token
                 - name: WOODPECKER_URL
-                  value: http://woodpecker.apps.svc.cluster.local:8000
+                  value: http://woodpecker.woodpecker.svc.cluster.local:8000
                 - name: WOODPECKER_ORG
                   value: k.renner
                 - name: WOODPECKER_SECRET_NAME

kubernetes/infrastructure/openbao/rotation/review-bot.yaml

@@ -25,9 +25,10 @@
 # but a failed sync should be fixed before the next run rather than left.
 #
 # gitea is reached in-cluster at gitea.apps.svc:3000, woodpecker at
-# woodpecker.apps.svc:8000 (both hairpin off the ingress LB from a pod). Needs
+# woodpecker.woodpecker.svc:8000 (it moved out of `apps` in the namespace
+# migration — see kubernetes/apps/woodpecker/MIGRATION-namespace.md). Needs
 # the netpol halves: gitea-allow-review-bot-rotator + woodpecker-allow-rotators
-# (apps/*) and rotator-review-bot-egress (openbao-ns.yaml).
+# (woodpecker ns) and rotator-review-bot-egress (openbao-ns.yaml).
 #
 # OpenBao policy/role rotator-review-bot + the bootstraps (review-bot password,
 # woodpecker-bot token, seed review-bot#token) are provisioned by hand — see
@@ -147,7 +148,7 @@ spec:
                 - name: BAO_FIELD
                   value: token
                 - name: WOODPECKER_URL
-                  value: http://woodpecker.apps.svc.cluster.local:8000
+                  value: http://woodpecker.woodpecker.svc.cluster.local:8000
                 - name: WOODPECKER_ORG
                   value: k.renner
                 - name: WOODPECKER_SECRET_NAME

kubernetes/infrastructure/post-crds/openbao-clustersecretstore.yaml

@@ -30,6 +30,7 @@ spec:
         - paperless
         - trmnl
         - velero
+        - woodpecker # wp-pg barman creds; woodpecker moving out of `apps` (#433)
         - wordpress
   provider:
     vault: # ESO's provider name; speaks OpenBao fine

opentofu/lxc_800_rustfs_decommission.tf

@@ -1,4 +1,4 @@
-# TEMPORARY shims v2 — rustfs LXCs 800/801/802, decommission in progress.
+# TEMPORARY shims v3 — rustfs LXCs 800/801/802, decommission in progress.
 #
 # Stage 2 (#270) failed: both destroy attempts died instantly with
 # "context deadline exceeded" on the pre-destroy status GET, while the PVE
@@ -10,34 +10,113 @@
 #
 # Fix: declare the timeouts explicitly, apply ONCE so they land in state
 # (in-place no-API update), then delete this file again and the next apply
-# destroys the containers. No lifecycle ignore_changes this time — the
-# timeout update must actually apply; the other attributes mirror the real
-# containers so the plan stays in-place.
+# destroys the containers.
+#
+# v3: v2 declared ONLY the ids, `started`, `unprivileged` and the timeouts,
+# claiming "the other attributes mirror the real containers so the plan stays
+# in-place". They did not. Everything absent from config planned as a removal,
+# and two of those removals — `mount_point` and `operating_system` — force
+# replacement, so the plan came back `must be replaced` for all three. Applying
+# that would have destroyed and recreated containers tagged `critical`,
+# `state-backend`, `terraform-backend` instead of nudging a timeout into state.
+#
+# So the blocks below are not decoration: each one exists to keep an attribute
+# out of the diff. They mirror the live containers, read from the plan's own
+# state values. `tags` is declared for the same reason — without it the apply
+# strips the tags off the real containers.
 #
 #   1. merge; run PIPELINE=deploy-infrastructure (plan MUST show in-place
 #      update only, no replace), then TOFU_APPLY=true
 #   2. follow-up PR deletes this file; plan shows 3 destroys; apply.
 
 locals {
+  # Per-container values mirrored from live state. hostname and mac_address
+  # differ per container; declaring them wrong would itself force a diff.
   rustfs_decommission = {
-    "800" = "saturn"
-    "801" = "jupiter"
-    "802" = "neptune"
+    "800" = { node = "saturn", hostname = "rustfs-1.srv.renner.uno", mac = "BC:24:11:0B:E5:7B" }
+    "801" = { node = "jupiter", hostname = "rustfs-2.srv.renner.uno", mac = "BC:24:11:EA:1D:38" }
+    "802" = { node = "neptune", hostname = "rustfs-3.srv.renner.uno", mac = "BC:24:11:8F:F4:67" }
   }
 }
 
 resource "proxmox_virtual_environment_container" "rustfs_decommission" {
   for_each = local.rustfs_decommission
 
-  node_name     = each.value
+  node_name     = each.value.node
   vm_id         = tonumber(each.key)
   started       = false
   unprivileged  = true
   start_on_boot = true
 
+  tags = [
+    "critical",
+    "rustfs",
+    "state-backend",
+    "terraform-backend",
+  ]
+
   timeout_create = 1800
   timeout_clone  = 1800
   timeout_update = 1800
   timeout_delete = 1800
   timeout_start  = 300
+
+  cpu {
+    architecture = "amd64"
+    cores        = 1
+    units        = 1024
+  }
+
+  memory {
+    dedicated = 1024
+    swap      = 0
+  }
+
+  disk {
+    datastore_id = "local"
+    size         = 10
+  }
+
+  # forces replacement if absent — this is the block that broke v2.
+  mount_point {
+    path      = "/mnt/data"
+    size      = "50G"
+    volume    = "local:${each.key}/vm-${each.key}-disk-1.raw"
+    backup    = true
+    replicate = true
+  }
+
+  network_interface {
+    name        = "eth0"
+    bridge      = "v40srv"
+    enabled     = true
+    firewall    = false
+    mac_address = each.value.mac
+  }
+
+  initialization {
+    hostname = each.value.hostname
+
+    ip_config {
+      ipv4 {
+        address = "dhcp"
+      }
+    }
+  }
+
+  # operating_system is NOT declared, and must not be. The provider requires
+  # template_file_id inside it, which is a create-time-only value: PVE does not
+  # report the original template for an existing container, so it is absent
+  # from state and cannot be mirrored. Declaring the block without it fails
+  # validation ("Missing required argument"); declaring it with a guessed
+  # template would be a lie that only surfaces if something ever recreates
+  # these.
+  #
+  # Left undeclared it would plan as a removal, and removing operating_system
+  # forces replacement — the exact failure this file exists to avoid. So it is
+  # ignored instead. The timeouts above are deliberately NOT ignored: they are
+  # the one thing this shim needs to actually apply.
+  lifecycle {
+    ignore_changes = [operating_system]
+  }
 }

opentofu/providers.tf

@@ -61,11 +61,27 @@ provider "unifi" {
 # as the unifi provider above; the v2 endpoints are simply not covered by it.
 # UniFi PUT quirk: updates may require _id in the body, which restapi does
 # not send — if an in-place update 400s, taint the object so CI recreates it.
+#
+# create_returns_object, NOT write_returns_object. Both make the provider store
+# the API response into `data`; write_returns_object does it for PUT as well as
+# POST. Since `data` in unifi_nat_hairpin.tf never sets the UDM-assigned `_id`,
+# every write put a field into state that config does not have, so all 60 rules
+# planned as "will be updated in-place" with `- _id` as the whole diff — and
+# re-dirtied themselves on the next write, forever.
+#
+# No provider-side ignore can suppress that: `data` is a configured attribute,
+# compared by Terraform core before provider logic runs. Tried and disproven —
+# ignore_changes_to = ["_id"] (plan 5661), ignore_all_server_changes (5662),
+# and ignore_server_additions on restapi 3.x (5667). All three left the 60.
+#
+# create_returns_object keeps what the provider actually needs — the POST
+# response is how it learns `_id` for id_attribute below — without letting
+# updates rewrite `data`.
 provider "restapi" {
-  uri                  = "${var.unifi_api_url}/proxy/network/v2/api/site/default"
-  insecure             = var.unifi_insecure
-  write_returns_object = true
-  id_attribute         = "_id"
+  uri                   = "${var.unifi_api_url}/proxy/network/v2/api/site/default"
+  insecure              = var.unifi_insecure
+  create_returns_object = true
+  id_attribute          = "_id"
   headers = {
     "X-API-KEY"    = var.unifi_api_key
     "Content-Type" = "application/json"