fix(unifi): capture the API response on create only, not on every write

Root cause of the 60 permanently-dirty hairpin NAT rules.

The provider was configured write_returns_object = true, which makes it store
the API response into `data` after POST *and* PUT. The UDM assigns `_id`;
unifi_nat_hairpin.tf's data = jsonencode({...}) never sets it. So every write
put a field into state that the config does not have, and all 60 rules planned
as "will be updated in-place" with `- _id` as the entire diff — re-dirtying
themselves on the next write, indefinitely.

No provider-side ignore can suppress this, because `data` is a configured
attribute and Terraform compares config to state in core, before provider
logic runs. Three were tried and disproven against real plans rather than
argued about:

  ignore_changes_to = ["_id"]                       plan 5661 — 60 remained
  ignore_all_server_changes = true                  plan 5662 — 60 remained
  restapi 3.x + ignore_server_additions = true      plan 5667 — 60 remained,
                                                    plus a new api_data diff

create_returns_object keeps the part the provider genuinely needs — the POST
response is how it learns `_id` for id_attribute = "_id" — while no longer
letting updates rewrite `data` from the response.

This does not clear the existing pollution by itself: state still carries `_id`
in `data` for all 60 rules, so the plan will keep showing them until one apply
rewrites `data` to the configured value. What it changes is that the apply then
sticks, instead of the next write undoing it.

Co-Authored-By: Claude Bot <claude-bot@renner.uno>

.woodpecker/ci-manifests-test.yml

@@ -115,7 +115,7 @@ steps:
   # An earlier revision of this comment also asserted that several deploy-*.yml
   # files emit "Additional property timeout is not allowed" warnings. That was
   # observed under the v3.9.0 CLI this step used to pin, and it is NOT known to
-  # still hold on v3.16.0 — review raised the likelihood that the warning was
+  # still hold on current CLIs — review raised the likelihood that the warning was
   # schema drift in a 7-minor-old CLI rather than real debt in those files, which
   # is one of the reasons this pin moved to latest stable. It cannot be confirmed
   # from a green run either, because the loop below sends lint output to /tmp/out
@@ -123,9 +123,15 @@ steps:
   # The claim is dropped rather than restated: what this step guarantees is the
   # exit-code contract above, which is what the gate actually rests on.
   # Latest stable, per the standing rule that any software uses the latest
-  # stable version. Matching the deployed woodpecker-server (v3.16.0) is a
-  # CONSEQUENCE of that, not a separate requirement — the server is bumped by
-  # image-automation within the 3.x range, so both land on the same release.
+  # stable version. Matching the deployed woodpecker-server is a CONSEQUENCE of
+  # that, not a separate requirement — the server is bumped by image-automation
+  # within the 3.x range, so both should land on the same release.
+  #
+  # They do not land together, though: the server is a Kubernetes manifest and
+  # is bumped automatically, this pipeline file is not and cannot be (see
+  # infrastructure/image-automation/README.md). So the server moves first and
+  # this trails until someone bumps it by hand — which is what happened at
+  # v3.16.0 -> v3.17.0, caught by the update audit (#426) rather than by CI.
   #
   # It matters here because the CLI compiles the config against its own schema:
   # one behind the server can pass a config the server then rejects, or reject
@@ -152,7 +158,7 @@ steps:
   # whenever it trails the deployed server. If you bump the server, bump here.
   - name: woodpecker-lint
     depends_on: [] # see note above `steps:`
-    image: woodpeckerci/woodpecker-cli:v3.16.0-alpine
+    image: woodpeckerci/woodpecker-cli:v3.17.0-alpine
     commands:
       - |
         set -eu

kubernetes/apps/monitoring-rules/prometheusrule-woodpecker-rotation.yaml

@@ -12,37 +12,37 @@ spec:
   groups:
     - name: woodpecker-rotation.rules
       # Convergence safety net for the automated WOODPECKER_AGENT_SECRET rotation
-      # (kubernetes/apps/woodpecker/rotate-agent-secret.yaml). That rotation Job is
+      # (kubernetes/infrastructure/openbao/rotation/woodpecker.yaml). That rotation Job is
       # fire-and-forget: it writes OpenBao and exits WITHOUT verifying ESO re-synced
       # or that both consumers restarted healthy. These alerts close that gap. ESO
       # metrics are not scraped in this cluster, so everything here is built on
-      # kube-state-metrics. The CronJob/Job series stay empty until the rotation
-      # feature (branch feat/woodpecker-secret-auto-rotation) is merged — harmless.
+      # kube-state-metrics. The rotation has since landed: the CronJob lives in the
+      # openbao namespace, not apps, which is where the last two rules look.
       rules:
         - alert: WoodpeckerServerNotReady
           # 0 ready replicas well past the 10m ESO refresh + Reloader restart cycle.
-          expr: kube_statefulset_status_replicas_ready{namespace="apps",statefulset="woodpecker-server"} == 0
+          expr: kube_deployment_status_replicas_ready{namespace="woodpecker",deployment="woodpecker-server"} == 0
           for: 20m
           labels:
             severity: critical
           annotations:
             summary: "Woodpecker CI server has no ready replica"
             description: >-
-              StatefulSet woodpecker-server (ns apps) has had 0 ready replicas for 20m;
-              all CI is down (agents cannot reach woodpecker.apps.svc:9000). If this
-              followed a secret rotation, the server pod failed to return after its
-              Reloader-triggered restart — check `kubectl -n apps describe pod
-              woodpecker-server-0` (PVC attach / crashloop) and the woodpecker-secret
-              ExternalSecret sync status.
+              Deployment woodpecker-server (ns woodpecker) has had 0 ready replicas
+              for 20m; all CI is down (agents cannot reach
+              woodpecker.woodpecker.svc:9000). If this followed a secret rotation, the
+              server pod failed to return after its Reloader-triggered restart — check
+              `kubectl -n woodpecker describe deploy woodpecker-server` and the
+              woodpecker-secret ExternalSecret sync status.
         - alert: WoodpeckerAgentNotReady
-          expr: kube_deployment_status_replicas_available{namespace="apps",deployment="woodpecker-agent"} == 0
+          expr: kube_deployment_status_replicas_available{namespace="woodpecker",deployment="woodpecker-agent"} == 0
           for: 20m
           labels:
             severity: warning
           annotations:
             summary: "Woodpecker CI agent has no available replica"
             description: >-
-              Deployment woodpecker-agent (ns apps) has had 0 available replicas for
+              Deployment woodpecker-agent (ns woodpecker) has had 0 available replicas for
               20m; no pipelines will run. If this followed a secret rotation, the agent
               did not return after its Reloader restart, or its WOODPECKER_AGENT_SECRET
               no longer matches the server's — check the agent pod logs for a gRPC auth
@@ -50,29 +50,29 @@ spec:
         - alert: WoodpeckerSecretRotationJobFailed
           # Named, clearer signal than the generic KubeJobFailed for the OpenBao-write
           # step. Fires while a failed rotation Job is retained (ttl 3d).
-          expr: kube_job_status_failed{namespace="apps",job_name=~"woodpecker-secret-rotate.*"} > 0
+          expr: kube_job_status_failed{namespace="openbao",job_name=~"woodpecker-secret-rotate.*"} > 0
           for: 15m
           labels:
             severity: warning
           annotations:
             summary: "Woodpecker agent-secret rotation Job failed"
             description: >-
-              A woodpecker-secret-rotate Job (ns apps) failed at the OpenBao-write step;
+              A woodpecker-secret-rotate Job (ns openbao) failed at the OpenBao-write step;
               the agent secret was NOT rotated. Most common cause: the one-time OpenBao
               `rotator` role/policy bootstrap was never applied (see
               kubernetes/infrastructure/openbao/policies/rotator.hcl). Check
-              `kubectl -n apps logs job/<name>`.
+              `kubectl -n openbao logs job/<name>`.
         - alert: WoodpeckerSecretRotationStale
           # Quarterly cadence (~91d) + grace. The series is ABSENT until the first
           # successful run, so this cannot false-fire before rotation has ever worked.
-          expr: (time() - kube_cronjob_status_last_successful_time{namespace="apps",cronjob="woodpecker-secret-rotate"}) > 100 * 24 * 3600
+          expr: (time() - kube_cronjob_status_last_successful_time{namespace="openbao",cronjob="woodpecker-secret-rotate"}) > 100 * 24 * 3600
           for: 1h
           labels:
             severity: warning
           annotations:
             summary: "Woodpecker agent secret has not rotated in >100 days"
             description: >-
-              The quarterly woodpecker-secret-rotate CronJob (ns apps) has not completed
+              The quarterly woodpecker-secret-rotate CronJob (ns openbao) has not completed
               successfully in over 100 days — rotation has silently stopped (CronJob
               suspended/deleted, or every run failing). Investigate the CronJob and its
               recent Jobs.

kubernetes/apps/woodpecker/MIGRATION-namespace.md

@@ -326,6 +326,56 @@ Revert the PR. Flux recreates the `apps` workloads and the DSN points back at
 cluster during the window is lost, so roll back promptly or not at all — if
 pipelines have run against `woodpecker/wp-pg`, prefer fixing forward.
 
+## Stage 3 — retiring the old cluster
+
+Deletes the four `apps` manifests, promotes the `migration-*` set onto their
+names, and gives the surviving Cluster the `prune: disabled` data guard it was
+deliberately denied while a rollback still existed.
+
+**Merging stage 3 does not delete anything.** The old resources carry
+`kustomize.toolkit.fluxcd.io/prune: disabled`, so dropping them from git only
+stops Flux managing them — they keep running, and keep occupying their PVC.
+That is the point: the PR is reversible, the teardown is a separate, deliberate
+act.
+
+Before deleting anything, confirm the new cluster is genuinely standing on its
+own — that it has taken at least one base backup to its own prefix, so recovery
+no longer depends on the retired cluster's data:
+
+```sh
+kubectl -n woodpecker get backup.postgresql.cnpg.io
+kubectl -n woodpecker get cluster wp-pg -o jsonpath='{.status.conditions}'
+```
+
+`LastBackupSucceeded` must be `True`, not merely `BackupStarted`.
+
+Then retire the old set:
+
+```sh
+kubectl -n apps delete scheduledbackup.postgresql.cnpg.io wp-pg-daily
+kubectl -n apps delete cluster.postgresql.cnpg.io wp-pg          # deletes its PVCs
+kubectl -n apps delete objectstore.barmancloud.cnpg.io wp-pg-store
+kubectl -n apps delete externalsecret hetzner-s3-creds
+```
+
+Delete the ScheduledBackup first: leaving it while the Cluster goes away
+produces failing backup jobs against a cluster that no longer exists.
+
+**What this destroys and what it does not.** It destroys the old cluster's PVCs
+— including the four pipeline records that only ever existed there (see the
+stage-1 verification section). It does **not** touch
+`s3://renner-uno-k8s-cnpg/wp-pg`: that prefix keeps its backups until their own
+retention expires, and `wp-pg-store-source` is retained precisely so it stays
+restorable. That is the real safety net, not the running cluster.
+
+### After the teardown
+
+The `apps` namespace should have no woodpecker-related objects left:
+
+```sh
+kubectl -n apps get cluster,objectstore,scheduledbackup,externalsecret 2>/dev/null | grep -i "wp-pg\|hetzner" || echo "clean"
+```
+
 ## Rolling back stage 1
 
 Delete the four resources and the namespace. Nothing references them, and the

kubernetes/apps/woodpecker/cluster.yaml

@@ -1,19 +1,27 @@
+---
+# Woodpecker's Postgres backend. Serves the server at wp-pg-rw.woodpecker.svc.
+#
+# Originally created in `apps` with bootstrap.initdb; moved here by restoring
+# from barman into the new namespace rather than by editing a namespace field,
+# because PVCs are namespace-scoped and a Cluster cannot be relocated in place
+# (issue #433, MIGRATION-namespace.md).
+#
+# The bootstrap.recovery stanza and its externalCluster are retained rather than
+# reverted to initdb. CNPG honours `bootstrap` only when PGDATA is absent, so on
+# this live cluster it is inert — and keeping it means a cold rebuild of an
+# empty cluster restores from backup instead of coming up empty, which is the
+# gap kubernetes/RECOVERY.md exists to close for the other CNPG clusters.
 apiVersion: postgresql.cnpg.io/v1
 kind: Cluster
 metadata:
   name: wp-pg
-  namespace: apps
+  namespace: woodpecker
   annotations:
-    # CRITICAL data-guard: prune-disabled so the seed-woodpecker -> dedicated
-    # woodpecker KS ownership handoff can never prune this Cluster (which would
-    # delete its data PVCs). Landed here first, via seed-woodpecker, so it's
-    # live before the move. Kept permanently as a standing guard.
+    # Data guard, earned at stage 3 now that this is the only copy: prevents a
+    # Flux prune from deleting the Cluster and with it the data PVCs. The
+    # migration deliberately withheld this while a rollback still existed.
     kustomize.toolkit.fluxcd.io/prune: disabled
 spec:
-  # Single instance: the Woodpecker CI DB is small and non-critical (state is
-  # largely reconstructable), so HA isn't warranted — but WAL archiving to
-  # object storage still gives point-in-time backups. Mirrors grafana-db's
-  # single-instance shape, with barman enabled like immich/paperless.
   instances: 1
   imageName: ghcr.io/cloudnative-pg/postgresql:16
   storage:
@@ -25,9 +33,25 @@ spec:
       parameters:
         barmanObjectName: wp-pg-store
   bootstrap:
-    initdb:
+    recovery:
+      source: wp-pg-apps
+      # Must mirror the initdb values of the cluster this restores from
+      # (apps/wp-pg: database woodpecker, owner woodpecker). CNPG defaults
+      # `database` to `app` and `owner` to whatever `database` is, and those
+      # names do not exist in the physically restored data — it would provision
+      # an empty `app` database, point the generated wp-pg-app Secret and the
+      # -app Service at it, and leave the real data unreachable through them.
+      # Same pattern as every recovery example in kubernetes/RECOVERY.md.
       database: woodpecker
       owner: woodpecker
+  externalClusters:
+    - name: wp-pg-apps
+      plugin:
+        name: barman-cloud.cloudnative-pg.io
+        parameters:
+          barmanObjectName: wp-pg-store-source
+          # The cluster that produced the backup, not this one.
+          serverName: wp-pg
   postgresql:
     parameters:
       shared_buffers: "128MB"

kubernetes/apps/woodpecker/hetzner-creds-externalsecret.yaml

@@ -1,11 +1,20 @@
+---
+# Barman S3 credentials for wp-pg.
+#
+# Sourced from seed/paperless/hetzner-creds, which is not where they belong.
+# Issue #397 wants woodpecker to own seed/woodpecker/hetzner-creds; creating
+# that path needs OpenBao write access (`bao kv list seed` → 403 for the
+# identities available here), so it stays open. The values are identical, so it
+# is a one-line swap of the `key:` below once the path exists.
+#
+# The namespace allowlist on the openbao ClusterSecretStore
+# (infrastructure/post-crds/openbao-clustersecretstore.yaml) must include this
+# namespace, or ESO silently never syncs — see #505.
 apiVersion: external-secrets.io/v1
 kind: ExternalSecret
 metadata:
   name: hetzner-s3-creds
-  namespace: apps
-  annotations:
-    # Prune-disabled — a standing guard kept from the seed->dedicated-KS handoff (barman S3 creds).
-    kustomize.toolkit.fluxcd.io/prune: disabled
+  namespace: woodpecker
 spec:
   refreshInterval: 1h
   secretStoreRef:
@@ -14,10 +23,5 @@ spec:
   target:
     name: hetzner-s3-creds
   dataFrom:
-    # Reuses the paperless copy of the shared Hetzner S3 credentials (identical
-    # ACCESS_KEY_ID / ACCESS_SECRET_KEY / REGION) to avoid seeding a new OpenBao
-    # path for this migration. FOLLOW-UP: give woodpecker its own
-    # seed/woodpecker/hetzner-creds path once OpenBao can be written, and repoint
-    # this here (the value is the same, so it's a drop-in swap).
     - extract:
         key: seed/paperless/hetzner-creds

kubernetes/apps/woodpecker/kustomization.yaml

@@ -21,35 +21,20 @@ resources:
   # (kubernetes/infrastructure/openbao/rotation/woodpecker.yaml) — intra-ns
   # OpenBao access, no NetworkPolicy needed. server.yaml/agent.yaml keep their
   # Reloader annotations; externalsecret.yaml keeps the 10m refreshInterval.
-  # wp-pg CNPG Postgres backend (moved here from the retired seed-woodpecker KS;
-  # each object carries prune: disabled so ownership transferred without a
-  # teardown). The dedicated `woodpecker` KS depends on infrastructure-cnpg-barman
-  # for the barman plugin behind the ObjectStore/ScheduledBackup.
-  - cluster.yaml
-  - objectstore.yaml
-  - scheduled-backup.yaml
-  - hetzner-creds-externalsecret.yaml
   # --- namespace migration, stages 1-2 (issue #433) ---
   # Stage 2 has moved the workloads above (server, agent, service, ingress, pdb,
   # rbac, secrets, policies) into the `woodpecker` namespace and pointed the
-  # server's DSN at wp-pg-rw.woodpecker.svc.
-  #
-  # cluster.yaml / objectstore.yaml / scheduled-backup.yaml /
-  # hetzner-creds-externalsecret.yaml still describe the ORIGINAL apps-namespace
-  # cluster and are deliberately left in place: they are the rollback. Their
-  # replacements are the migration-* files below. Stage 3 deletes the old four
-  # and renames the migration-* set onto their names.
-  # Woodpecker is moving out of the shared `apps` namespace into its own.
-  # These resources restore wp-pg into the target namespace from barman,
-  # running in parallel with the live cluster. Nothing points at them yet, so
-  # this stage carries no CI impact and is revertible by deleting them.
-  # Cutover (server/agent/DSN/RBAC) is stage 2; removing the `apps` originals
-  # is stage 3. Runbook: MIGRATION-namespace.md.
+  # server's DSN at wp-pg-rw.woodpecker.svc. Stage 3 retired the `apps`
+  # originals; these four now describe the surviving cluster. History and the
+  # reasoning behind the three stages: MIGRATION-namespace.md.
   #
   # The `woodpecker` Namespace itself is declared in infrastructure/namespaces,
   # not here: it must exist before the `infrastructure-crds` KS applies the
   # namespace's CiliumNetworkPolicies, and this KS runs after both of those.
-  - migration-externalsecret-hetzner.yaml
-  - migration-objectstore.yaml
-  - migration-cluster.yaml
-  - migration-scheduled-backup.yaml
+  #
+  # This KS depends on infrastructure-cnpg-barman for the barman plugin behind
+  # the ObjectStore and ScheduledBackup.
+  - hetzner-creds-externalsecret.yaml
+  - objectstore.yaml
+  - cluster.yaml
+  - scheduled-backup.yaml

kubernetes/apps/woodpecker/migration-cluster.yaml

@@ -1,62 +0,0 @@
----
-# wp-pg restored into the `woodpecker` namespace from the barman backup of the
-# live apps/wp-pg. Runs in parallel with the original until cutover (PR 2);
-# nothing points at it yet, so it carries no traffic.
-#
-# Spec mirrors apps/wp-pg exactly — same instance count, image, storage class
-# and Postgres parameters — so the only variable under test is the restore
-# itself. The differences are deliberate and limited to:
-#   - bootstrap: recovery (not initdb), sourced from the old prefix
-#   - plugins.barmanObjectName: the NEW store, so it archives somewhere else
-#
-# NOT prune-disabled, unlike apps/wp-pg. That guard exists there because the
-# original holds the only copy of the data through a KS ownership handoff. This
-# one is reconstructable by re-running the restore, and a prune guard on a
-# migration artifact is how orphans get left behind if the migration is
-# abandoned. It gains the guard in PR 3, once it is the surviving cluster.
-apiVersion: postgresql.cnpg.io/v1
-kind: Cluster
-metadata:
-  name: wp-pg
-  namespace: woodpecker
-spec:
-  instances: 1
-  imageName: ghcr.io/cloudnative-pg/postgresql:16
-  storage:
-    storageClass: ceph-block
-    size: 5Gi
-  plugins:
-    - name: barman-cloud.cloudnative-pg.io
-      isWALArchiver: true
-      parameters:
-        barmanObjectName: wp-pg-store
-  bootstrap:
-    recovery:
-      source: wp-pg-apps
-      # Must mirror the initdb values of the cluster this restores from
-      # (apps/wp-pg: database woodpecker, owner woodpecker). CNPG defaults
-      # `database` to `app` and `owner` to whatever `database` is, and those
-      # names do not exist in the physically restored data — it would provision
-      # an empty `app` database, point the generated wp-pg-app Secret and the
-      # -app Service at it, and leave the real data unreachable through them.
-      # Same pattern as every recovery example in kubernetes/RECOVERY.md.
-      database: woodpecker
-      owner: woodpecker
-  externalClusters:
-    - name: wp-pg-apps
-      plugin:
-        name: barman-cloud.cloudnative-pg.io
-        parameters:
-          barmanObjectName: wp-pg-store-source
-          # The cluster that produced the backup, not this one.
-          serverName: wp-pg
-  postgresql:
-    parameters:
-      shared_buffers: "128MB"
-      max_connections: "50"
-  resources:
-    requests:
-      cpu: 100m
-      memory: 256Mi
-    limits:
-      memory: 512Mi

kubernetes/apps/woodpecker/migration-externalsecret-hetzner.yaml

@@ -1,28 +0,0 @@
----
-# Barman S3 credentials for the wp-pg cluster in its new `woodpecker` home.
-# Secrets are namespace-scoped, so the copy in `apps` cannot be reused; this is
-# the same material fetched into the new namespace.
-#
-# Still sourced from seed/paperless/hetzner-creds, deliberately. Issue #397
-# wants woodpecker to own seed/woodpecker/hetzner-creds instead, and this
-# migration would have been the natural moment — but creating that path needs
-# OpenBao write access that is not available here (`bao kv list seed` → 403).
-# Repointing at a path that may not exist would leave the migrated cluster
-# unable to archive WAL, which is exactly what the restore is meant to prove
-# works. #397 stays open and is a drop-in swap of the `key:` below once the
-# path exists; the values are identical.
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: hetzner-s3-creds
-  namespace: woodpecker
-spec:
-  refreshInterval: 1h
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: hetzner-s3-creds
-  dataFrom:
-    - extract:
-        key: seed/paperless/hetzner-creds

kubernetes/apps/woodpecker/migration-objectstore.yaml

@@ -1,64 +0,0 @@
----
-# RECOVERY SOURCE — read-only. Points at the prefix the live `apps/wp-pg`
-# cluster has been archiving to since it was created. The restored cluster
-# bootstraps from here and never writes to it.
-#
-# serverName on the consuming externalCluster must be `wp-pg`, the name of the
-# cluster that produced these backups.
-apiVersion: barmancloud.cnpg.io/v1
-kind: ObjectStore
-metadata:
-  name: wp-pg-store-source
-  namespace: woodpecker
-spec:
-  configuration:
-    destinationPath: s3://renner-uno-k8s-cnpg/wp-pg
-    endpointURL: https://fsn1.your-objectstorage.com
-    s3Credentials:
-      accessKeyId:
-        name: hetzner-s3-creds
-        key: ACCESS_KEY_ID
-      secretAccessKey:
-        name: hetzner-s3-creds
-        key: ACCESS_SECRET_KEY
-      region:
-        name: hetzner-s3-creds
-        key: REGION
-    wal:
-      compression: gzip
-    data:
-      compression: gzip
-  # No retentionPolicy: this store is a read source during the migration, and
-  # the old cluster in `apps` owns the lifecycle of the data underneath it.
----
-# ARCHIVING TARGET — a DIFFERENT prefix from the source, deliberately.
-#
-# Two clusters archiving WAL to one destinationPath interleave their streams
-# and corrupt both. During the parallel-run window the old apps/wp-pg is still
-# live and still archiving to s3://renner-uno-k8s-cnpg/wp-pg, so the restored
-# cluster must not share it. Splitting the paths also keeps the pre-migration
-# backup intact and independently restorable as a rollback.
-apiVersion: barmancloud.cnpg.io/v1
-kind: ObjectStore
-metadata:
-  name: wp-pg-store
-  namespace: woodpecker
-spec:
-  configuration:
-    destinationPath: s3://renner-uno-k8s-cnpg/wp-pg-woodpecker
-    endpointURL: https://fsn1.your-objectstorage.com
-    s3Credentials:
-      accessKeyId:
-        name: hetzner-s3-creds
-        key: ACCESS_KEY_ID
-      secretAccessKey:
-        name: hetzner-s3-creds
-        key: ACCESS_SECRET_KEY
-      region:
-        name: hetzner-s3-creds
-        key: REGION
-    wal:
-      compression: gzip
-    data:
-      compression: gzip
-  retentionPolicy: "30d"

kubernetes/apps/woodpecker/migration-scheduled-backup.yaml

@@ -1,32 +0,0 @@
----
-# Base backups for the restored cluster, onto the new wp-pg-woodpecker prefix.
-#
-# Without this the cluster archives continuous WAL to that prefix but never
-# takes a base backup there, which leaves two problems: the prefix has no
-# independent recovery point of its own (recovery would have to chain all the
-# way back through the source prefix), and wp-pg-store's retentionPolicy: 30d
-# has no base backups to prune against, so the WAL chain grows unbounded for as
-# long as stage 1 sits before cutover.
-#
-# 02:40 UTC, offset from the apps cluster's 02:35 and the other clusters' 02:30
-# so the parallel-run window does not put two woodpecker base backups against
-# the Hetzner bucket at once. immediate: true takes the first as soon as the
-# restore finishes, which is also the cheapest confirmation that the new
-# credentials and the new ObjectStore both work end to end.
-#
-# Not prune-disabled, for the same reason as migration-cluster.yaml: this is a
-# migration artifact until stage 3 promotes it.
-apiVersion: postgresql.cnpg.io/v1
-kind: ScheduledBackup
-metadata:
-  name: wp-pg-daily
-  namespace: woodpecker
-spec:
-  schedule: "0 40 2 * * *"
-  immediate: true
-  backupOwnerReference: self
-  cluster:
-    name: wp-pg
-  method: plugin
-  pluginConfiguration:
-    name: barman-cloud.cloudnative-pg.io

kubernetes/apps/woodpecker/objectstore.yaml

@@ -1,18 +1,20 @@
 ---
+# RECOVERY SOURCE — read-only. The prefix the retired `apps/wp-pg` archived to.
+#
+# Kept after the migration on purpose. Nothing writes to it, and the backups
+# under it age out on their own retention, but while they last this is a second,
+# independent restore point that predates the move — worth having until the new
+# prefix has a meaningful backup history of its own.
+#
+# serverName on the consuming externalCluster must be `wp-pg`, the name of the
+# cluster that produced these backups.
 apiVersion: barmancloud.cnpg.io/v1
 kind: ObjectStore
 metadata:
-  name: wp-pg-store
-  namespace: apps
-  annotations:
-    # Prune-disabled — a standing guard kept from the seed->dedicated-KS handoff (backup target).
-    kustomize.toolkit.fluxcd.io/prune: disabled
+  name: wp-pg-store-source
+  namespace: woodpecker
 spec:
   configuration:
-    # Same offsite Hetzner Object Storage bucket as the other CNPG clusters,
-    # under a per-cluster prefix. See apps/paperless/objectstore.yaml for the
-    # SSE / addressing rationale (Hetzner is SSE-C only; transit is TLS,
-    # bucket private).
     destinationPath: s3://renner-uno-k8s-cnpg/wp-pg
     endpointURL: https://fsn1.your-objectstorage.com
     s3Credentials:
@@ -29,4 +31,36 @@ spec:
       compression: gzip
     data:
       compression: gzip
+  # No retentionPolicy: this store is a read source during the migration, and
+  # the old cluster in `apps` owns the lifecycle of the data underneath it.
+---
+# ARCHIVING TARGET — a DIFFERENT prefix from the source, deliberately.
+#
+# Two clusters archiving WAL to one destinationPath interleave their streams and
+# corrupt both. That mattered during the parallel-run window; it still matters
+# now, because the source prefix above is retained and must stay frozen for the
+# independent restore point it provides.
+apiVersion: barmancloud.cnpg.io/v1
+kind: ObjectStore
+metadata:
+  name: wp-pg-store
+  namespace: woodpecker
+spec:
+  configuration:
+    destinationPath: s3://renner-uno-k8s-cnpg/wp-pg-woodpecker
+    endpointURL: https://fsn1.your-objectstorage.com
+    s3Credentials:
+      accessKeyId:
+        name: hetzner-s3-creds
+        key: ACCESS_KEY_ID
+      secretAccessKey:
+        name: hetzner-s3-creds
+        key: ACCESS_SECRET_KEY
+      region:
+        name: hetzner-s3-creds
+        key: REGION
+    wal:
+      compression: gzip
+    data:
+      compression: gzip
   retentionPolicy: "30d"

kubernetes/apps/woodpecker/scheduled-backup.yaml

@@ -1,18 +1,19 @@
 ---
-# Daily base backup at 02:35 UTC (offset from the other clusters' 02:30 to
-# avoid a thundering herd against the Hetzner bucket). WAL archiving runs
-# continuously (see cluster.yaml spec.plugins[].isWALArchiver). immediate:true
-# takes the first base backup as soon as this CR applies.
+# Daily base backups onto the wp-pg-woodpecker prefix. Continuous WAL archiving
+# runs separately (cluster.yaml spec.plugins[].isWALArchiver); without periodic
+# base backups the prefix would have no recovery point of its own and
+# wp-pg-store's retentionPolicy: 30d would have nothing to prune against, so the
+# WAL chain would grow without bound.
+#
+# 02:40 UTC, offset from the other CNPG clusters' 02:30 so they do not all hit
+# the Hetzner bucket together.
 apiVersion: postgresql.cnpg.io/v1
 kind: ScheduledBackup
 metadata:
   name: wp-pg-daily
-  namespace: apps
-  annotations:
-    # Prune-disabled — a standing guard kept from the seed->dedicated-KS handoff (backup schedule).
-    kustomize.toolkit.fluxcd.io/prune: disabled
+  namespace: woodpecker
 spec:
-  schedule: "0 35 2 * * *"
+  schedule: "0 40 2 * * *"
   immediate: true
   backupOwnerReference: self
   cluster:

kubernetes/infrastructure/openbao/rotation/claude-oauth.yaml

@@ -14,8 +14,10 @@
 # immediately:
 #   kubectl -n openbao create job --from=cronjob/claude-oauth-secret-sync claude-oauth-sync-now
 #
-# woodpecker is reached in-cluster at woodpecker.apps.svc:8000. Needs the netpol
-# halves woodpecker-allow-rotators (apps/*) + rotator-claude-oauth-egress
+# woodpecker is reached in-cluster at woodpecker.woodpecker.svc:8000 (it moved
+# out of `apps` in the namespace migration — see
+# kubernetes/apps/woodpecker/MIGRATION-namespace.md). Needs the netpol
+# halves woodpecker-allow-rotators (woodpecker ns) + rotator-claude-oauth-egress
 # (openbao-ns.yaml). OpenBao policy/role rotator-claude-oauth + the bootstraps
 # (woodpecker-bot token, the seeded OAuth value) are provisioned by hand — see
 # policies/rotator-claude-oauth.hcl.
@@ -89,7 +91,7 @@ spec:
                 - name: BAO_FIELD
                   value: setup-token
                 - name: WOODPECKER_URL
-                  value: http://woodpecker.apps.svc.cluster.local:8000
+                  value: http://woodpecker.woodpecker.svc.cluster.local:8000
                 - name: WOODPECKER_ORG
                   value: k.renner
                 - name: WOODPECKER_SECRET_NAME

kubernetes/infrastructure/openbao/rotation/review-bot.yaml

@@ -25,9 +25,10 @@
 # but a failed sync should be fixed before the next run rather than left.
 #
 # gitea is reached in-cluster at gitea.apps.svc:3000, woodpecker at
-# woodpecker.apps.svc:8000 (both hairpin off the ingress LB from a pod). Needs
+# woodpecker.woodpecker.svc:8000 (it moved out of `apps` in the namespace
+# migration — see kubernetes/apps/woodpecker/MIGRATION-namespace.md). Needs
 # the netpol halves: gitea-allow-review-bot-rotator + woodpecker-allow-rotators
-# (apps/*) and rotator-review-bot-egress (openbao-ns.yaml).
+# (woodpecker ns) and rotator-review-bot-egress (openbao-ns.yaml).
 #
 # OpenBao policy/role rotator-review-bot + the bootstraps (review-bot password,
 # woodpecker-bot token, seed review-bot#token) are provisioned by hand — see
@@ -147,7 +148,7 @@ spec:
                 - name: BAO_FIELD
                   value: token
                 - name: WOODPECKER_URL
-                  value: http://woodpecker.apps.svc.cluster.local:8000
+                  value: http://woodpecker.woodpecker.svc.cluster.local:8000
                 - name: WOODPECKER_ORG
                   value: k.renner
                 - name: WOODPECKER_SECRET_NAME

opentofu/lxc_800_rustfs_decommission.tf

@@ -1,4 +1,4 @@
-# TEMPORARY shims v2 — rustfs LXCs 800/801/802, decommission in progress.
+# TEMPORARY shims v3 — rustfs LXCs 800/801/802, decommission in progress.
 #
 # Stage 2 (#270) failed: both destroy attempts died instantly with
 # "context deadline exceeded" on the pre-destroy status GET, while the PVE
@@ -10,34 +10,113 @@
 #
 # Fix: declare the timeouts explicitly, apply ONCE so they land in state
 # (in-place no-API update), then delete this file again and the next apply
-# destroys the containers. No lifecycle ignore_changes this time — the
-# timeout update must actually apply; the other attributes mirror the real
-# containers so the plan stays in-place.
+# destroys the containers.
+#
+# v3: v2 declared ONLY the ids, `started`, `unprivileged` and the timeouts,
+# claiming "the other attributes mirror the real containers so the plan stays
+# in-place". They did not. Everything absent from config planned as a removal,
+# and two of those removals — `mount_point` and `operating_system` — force
+# replacement, so the plan came back `must be replaced` for all three. Applying
+# that would have destroyed and recreated containers tagged `critical`,
+# `state-backend`, `terraform-backend` instead of nudging a timeout into state.
+#
+# So the blocks below are not decoration: each one exists to keep an attribute
+# out of the diff. They mirror the live containers, read from the plan's own
+# state values. `tags` is declared for the same reason — without it the apply
+# strips the tags off the real containers.
 #
 #   1. merge; run PIPELINE=deploy-infrastructure (plan MUST show in-place
 #      update only, no replace), then TOFU_APPLY=true
 #   2. follow-up PR deletes this file; plan shows 3 destroys; apply.
 
 locals {
+  # Per-container values mirrored from live state. hostname and mac_address
+  # differ per container; declaring them wrong would itself force a diff.
   rustfs_decommission = {
-    "800" = "saturn"
-    "801" = "jupiter"
-    "802" = "neptune"
+    "800" = { node = "saturn", hostname = "rustfs-1.srv.renner.uno", mac = "BC:24:11:0B:E5:7B" }
+    "801" = { node = "jupiter", hostname = "rustfs-2.srv.renner.uno", mac = "BC:24:11:EA:1D:38" }
+    "802" = { node = "neptune", hostname = "rustfs-3.srv.renner.uno", mac = "BC:24:11:8F:F4:67" }
   }
 }
 
 resource "proxmox_virtual_environment_container" "rustfs_decommission" {
   for_each = local.rustfs_decommission
 
-  node_name     = each.value
+  node_name     = each.value.node
   vm_id         = tonumber(each.key)
   started       = false
   unprivileged  = true
   start_on_boot = true
 
+  tags = [
+    "critical",
+    "rustfs",
+    "state-backend",
+    "terraform-backend",
+  ]
+
   timeout_create = 1800
   timeout_clone  = 1800
   timeout_update = 1800
   timeout_delete = 1800
   timeout_start  = 300
+
+  cpu {
+    architecture = "amd64"
+    cores        = 1
+    units        = 1024
+  }
+
+  memory {
+    dedicated = 1024
+    swap      = 0
+  }
+
+  disk {
+    datastore_id = "local"
+    size         = 10
+  }
+
+  # forces replacement if absent — this is the block that broke v2.
+  mount_point {
+    path      = "/mnt/data"
+    size      = "50G"
+    volume    = "local:${each.key}/vm-${each.key}-disk-1.raw"
+    backup    = true
+    replicate = true
+  }
+
+  network_interface {
+    name        = "eth0"
+    bridge      = "v40srv"
+    enabled     = true
+    firewall    = false
+    mac_address = each.value.mac
+  }
+
+  initialization {
+    hostname = each.value.hostname
+
+    ip_config {
+      ipv4 {
+        address = "dhcp"
+      }
+    }
+  }
+
+  # operating_system is NOT declared, and must not be. The provider requires
+  # template_file_id inside it, which is a create-time-only value: PVE does not
+  # report the original template for an existing container, so it is absent
+  # from state and cannot be mirrored. Declaring the block without it fails
+  # validation ("Missing required argument"); declaring it with a guessed
+  # template would be a lie that only surfaces if something ever recreates
+  # these.
+  #
+  # Left undeclared it would plan as a removal, and removing operating_system
+  # forces replacement — the exact failure this file exists to avoid. So it is
+  # ignored instead. The timeouts above are deliberately NOT ignored: they are
+  # the one thing this shim needs to actually apply.
+  lifecycle {
+    ignore_changes = [operating_system]
+  }
 }

opentofu/providers.tf

@@ -61,11 +61,27 @@ provider "unifi" {
 # as the unifi provider above; the v2 endpoints are simply not covered by it.
 # UniFi PUT quirk: updates may require _id in the body, which restapi does
 # not send — if an in-place update 400s, taint the object so CI recreates it.
+#
+# create_returns_object, NOT write_returns_object. Both make the provider store
+# the API response into `data`; write_returns_object does it for PUT as well as
+# POST. Since `data` in unifi_nat_hairpin.tf never sets the UDM-assigned `_id`,
+# every write put a field into state that config does not have, so all 60 rules
+# planned as "will be updated in-place" with `- _id` as the whole diff — and
+# re-dirtied themselves on the next write, forever.
+#
+# No provider-side ignore can suppress that: `data` is a configured attribute,
+# compared by Terraform core before provider logic runs. Tried and disproven —
+# ignore_changes_to = ["_id"] (plan 5661), ignore_all_server_changes (5662),
+# and ignore_server_additions on restapi 3.x (5667). All three left the 60.
+#
+# create_returns_object keeps what the provider actually needs — the POST
+# response is how it learns `_id` for id_attribute below — without letting
+# updates rewrite `data`.
 provider "restapi" {
-  uri                  = "${var.unifi_api_url}/proxy/network/v2/api/site/default"
-  insecure             = var.unifi_insecure
-  write_returns_object = true
-  id_attribute         = "_id"
+  uri                   = "${var.unifi_api_url}/proxy/network/v2/api/site/default"
+  insecure              = var.unifi_insecure
+  create_returns_object = true
+  id_attribute          = "_id"
   headers = {
     "X-API-KEY"    = var.unifi_api_key
     "Content-Type" = "application/json"