fix(unifi): capture the API response on create only, not on every write

Root cause of the 60 permanently-dirty hairpin NAT rules.

The provider was configured write_returns_object = true, which makes it store
the API response into `data` after POST *and* PUT. The UDM assigns `_id`;
unifi_nat_hairpin.tf's data = jsonencode({...}) never sets it. So every write
put a field into state that the config does not have, and all 60 rules planned
as "will be updated in-place" with `- _id` as the entire diff — re-dirtying
themselves on the next write, indefinitely.

No provider-side ignore can suppress this, because `data` is a configured
attribute and Terraform compares config to state in core, before provider
logic runs. Three were tried and disproven against real plans rather than
argued about:

  ignore_changes_to = ["_id"]                       plan 5661 — 60 remained
  ignore_all_server_changes = true                  plan 5662 — 60 remained
  restapi 3.x + ignore_server_additions = true      plan 5667 — 60 remained,
                                                    plus a new api_data diff

create_returns_object keeps the part the provider genuinely needs — the POST
response is how it learns `_id` for id_attribute = "_id" — while no longer
letting updates rewrite `data` from the response.

This does not clear the existing pollution by itself: state still carries `_id`
in `data` for all 60 rules, so the plan will keep showing them until one apply
rewrites `data` to the configured value. What it changes is that the apply then
sticks, instead of the next write undoing it.

Co-Authored-By: Claude Bot <claude-bot@renner.uno>

kubernetes/apps/monitoring-rules/prometheusrule-woodpecker-rotation.yaml

@@ -12,37 +12,37 @@ spec:
   groups:
     - name: woodpecker-rotation.rules
       # Convergence safety net for the automated WOODPECKER_AGENT_SECRET rotation
-      # (kubernetes/apps/woodpecker/rotate-agent-secret.yaml). That rotation Job is
+      # (kubernetes/infrastructure/openbao/rotation/woodpecker.yaml). That rotation Job is
       # fire-and-forget: it writes OpenBao and exits WITHOUT verifying ESO re-synced
       # or that both consumers restarted healthy. These alerts close that gap. ESO
       # metrics are not scraped in this cluster, so everything here is built on
-      # kube-state-metrics. The CronJob/Job series stay empty until the rotation
-      # feature (branch feat/woodpecker-secret-auto-rotation) is merged — harmless.
+      # kube-state-metrics. The rotation has since landed: the CronJob lives in the
+      # openbao namespace, not apps, which is where the last two rules look.
       rules:
         - alert: WoodpeckerServerNotReady
           # 0 ready replicas well past the 10m ESO refresh + Reloader restart cycle.
-          expr: kube_statefulset_status_replicas_ready{namespace="apps",statefulset="woodpecker-server"} == 0
+          expr: kube_deployment_status_replicas_ready{namespace="woodpecker",deployment="woodpecker-server"} == 0
           for: 20m
           labels:
             severity: critical
           annotations:
             summary: "Woodpecker CI server has no ready replica"
             description: >-
-              StatefulSet woodpecker-server (ns apps) has had 0 ready replicas for 20m;
-              all CI is down (agents cannot reach woodpecker.apps.svc:9000). If this
-              followed a secret rotation, the server pod failed to return after its
-              Reloader-triggered restart — check `kubectl -n apps describe pod
-              woodpecker-server-0` (PVC attach / crashloop) and the woodpecker-secret
-              ExternalSecret sync status.
+              Deployment woodpecker-server (ns woodpecker) has had 0 ready replicas
+              for 20m; all CI is down (agents cannot reach
+              woodpecker.woodpecker.svc:9000). If this followed a secret rotation, the
+              server pod failed to return after its Reloader-triggered restart — check
+              `kubectl -n woodpecker describe deploy woodpecker-server` and the
+              woodpecker-secret ExternalSecret sync status.
         - alert: WoodpeckerAgentNotReady
-          expr: kube_deployment_status_replicas_available{namespace="apps",deployment="woodpecker-agent"} == 0
+          expr: kube_deployment_status_replicas_available{namespace="woodpecker",deployment="woodpecker-agent"} == 0
           for: 20m
           labels:
             severity: warning
           annotations:
             summary: "Woodpecker CI agent has no available replica"
             description: >-
-              Deployment woodpecker-agent (ns apps) has had 0 available replicas for
+              Deployment woodpecker-agent (ns woodpecker) has had 0 available replicas for
               20m; no pipelines will run. If this followed a secret rotation, the agent
               did not return after its Reloader restart, or its WOODPECKER_AGENT_SECRET
               no longer matches the server's — check the agent pod logs for a gRPC auth
@@ -50,29 +50,29 @@ spec:
         - alert: WoodpeckerSecretRotationJobFailed
           # Named, clearer signal than the generic KubeJobFailed for the OpenBao-write
           # step. Fires while a failed rotation Job is retained (ttl 3d).
-          expr: kube_job_status_failed{namespace="apps",job_name=~"woodpecker-secret-rotate.*"} > 0
+          expr: kube_job_status_failed{namespace="openbao",job_name=~"woodpecker-secret-rotate.*"} > 0
           for: 15m
           labels:
             severity: warning
           annotations:
             summary: "Woodpecker agent-secret rotation Job failed"
             description: >-
-              A woodpecker-secret-rotate Job (ns apps) failed at the OpenBao-write step;
+              A woodpecker-secret-rotate Job (ns openbao) failed at the OpenBao-write step;
               the agent secret was NOT rotated. Most common cause: the one-time OpenBao
               `rotator` role/policy bootstrap was never applied (see
               kubernetes/infrastructure/openbao/policies/rotator.hcl). Check
-              `kubectl -n apps logs job/<name>`.
+              `kubectl -n openbao logs job/<name>`.
         - alert: WoodpeckerSecretRotationStale
           # Quarterly cadence (~91d) + grace. The series is ABSENT until the first
           # successful run, so this cannot false-fire before rotation has ever worked.
-          expr: (time() - kube_cronjob_status_last_successful_time{namespace="apps",cronjob="woodpecker-secret-rotate"}) > 100 * 24 * 3600
+          expr: (time() - kube_cronjob_status_last_successful_time{namespace="openbao",cronjob="woodpecker-secret-rotate"}) > 100 * 24 * 3600
           for: 1h
           labels:
             severity: warning
           annotations:
             summary: "Woodpecker agent secret has not rotated in >100 days"
             description: >-
-              The quarterly woodpecker-secret-rotate CronJob (ns apps) has not completed
+              The quarterly woodpecker-secret-rotate CronJob (ns openbao) has not completed
               successfully in over 100 days — rotation has silently stopped (CronJob
               suspended/deleted, or every run failing). Investigate the CronJob and its
               recent Jobs.

kubernetes/infrastructure/openbao/rotation/claude-oauth.yaml

@@ -14,8 +14,10 @@
 # immediately:
 #   kubectl -n openbao create job --from=cronjob/claude-oauth-secret-sync claude-oauth-sync-now
 #
-# woodpecker is reached in-cluster at woodpecker.apps.svc:8000. Needs the netpol
-# halves woodpecker-allow-rotators (apps/*) + rotator-claude-oauth-egress
+# woodpecker is reached in-cluster at woodpecker.woodpecker.svc:8000 (it moved
+# out of `apps` in the namespace migration — see
+# kubernetes/apps/woodpecker/MIGRATION-namespace.md). Needs the netpol
+# halves woodpecker-allow-rotators (woodpecker ns) + rotator-claude-oauth-egress
 # (openbao-ns.yaml). OpenBao policy/role rotator-claude-oauth + the bootstraps
 # (woodpecker-bot token, the seeded OAuth value) are provisioned by hand — see
 # policies/rotator-claude-oauth.hcl.
@@ -89,7 +91,7 @@ spec:
                 - name: BAO_FIELD
                   value: setup-token
                 - name: WOODPECKER_URL
-                  value: http://woodpecker.apps.svc.cluster.local:8000
+                  value: http://woodpecker.woodpecker.svc.cluster.local:8000
                 - name: WOODPECKER_ORG
                   value: k.renner
                 - name: WOODPECKER_SECRET_NAME

kubernetes/infrastructure/openbao/rotation/review-bot.yaml

@@ -25,9 +25,10 @@
 # but a failed sync should be fixed before the next run rather than left.
 #
 # gitea is reached in-cluster at gitea.apps.svc:3000, woodpecker at
-# woodpecker.apps.svc:8000 (both hairpin off the ingress LB from a pod). Needs
+# woodpecker.woodpecker.svc:8000 (it moved out of `apps` in the namespace
+# migration — see kubernetes/apps/woodpecker/MIGRATION-namespace.md). Needs
 # the netpol halves: gitea-allow-review-bot-rotator + woodpecker-allow-rotators
-# (apps/*) and rotator-review-bot-egress (openbao-ns.yaml).
+# (woodpecker ns) and rotator-review-bot-egress (openbao-ns.yaml).
 #
 # OpenBao policy/role rotator-review-bot + the bootstraps (review-bot password,
 # woodpecker-bot token, seed review-bot#token) are provisioned by hand — see
@@ -147,7 +148,7 @@ spec:
                 - name: BAO_FIELD
                   value: token
                 - name: WOODPECKER_URL
-                  value: http://woodpecker.apps.svc.cluster.local:8000
+                  value: http://woodpecker.woodpecker.svc.cluster.local:8000
                 - name: WOODPECKER_ORG
                   value: k.renner
                 - name: WOODPECKER_SECRET_NAME

opentofu/lxc_800_rustfs_decommission.tf

@@ -1,4 +1,4 @@
-# TEMPORARY shims v2 — rustfs LXCs 800/801/802, decommission in progress.
+# TEMPORARY shims v3 — rustfs LXCs 800/801/802, decommission in progress.
 #
 # Stage 2 (#270) failed: both destroy attempts died instantly with
 # "context deadline exceeded" on the pre-destroy status GET, while the PVE
@@ -10,34 +10,113 @@
 #
 # Fix: declare the timeouts explicitly, apply ONCE so they land in state
 # (in-place no-API update), then delete this file again and the next apply
-# destroys the containers. No lifecycle ignore_changes this time — the
-# timeout update must actually apply; the other attributes mirror the real
-# containers so the plan stays in-place.
+# destroys the containers.
+#
+# v3: v2 declared ONLY the ids, `started`, `unprivileged` and the timeouts,
+# claiming "the other attributes mirror the real containers so the plan stays
+# in-place". They did not. Everything absent from config planned as a removal,
+# and two of those removals — `mount_point` and `operating_system` — force
+# replacement, so the plan came back `must be replaced` for all three. Applying
+# that would have destroyed and recreated containers tagged `critical`,
+# `state-backend`, `terraform-backend` instead of nudging a timeout into state.
+#
+# So the blocks below are not decoration: each one exists to keep an attribute
+# out of the diff. They mirror the live containers, read from the plan's own
+# state values. `tags` is declared for the same reason — without it the apply
+# strips the tags off the real containers.
 #
 #   1. merge; run PIPELINE=deploy-infrastructure (plan MUST show in-place
 #      update only, no replace), then TOFU_APPLY=true
 #   2. follow-up PR deletes this file; plan shows 3 destroys; apply.
 
 locals {
+  # Per-container values mirrored from live state. hostname and mac_address
+  # differ per container; declaring them wrong would itself force a diff.
   rustfs_decommission = {
-    "800" = "saturn"
-    "801" = "jupiter"
-    "802" = "neptune"
+    "800" = { node = "saturn", hostname = "rustfs-1.srv.renner.uno", mac = "BC:24:11:0B:E5:7B" }
+    "801" = { node = "jupiter", hostname = "rustfs-2.srv.renner.uno", mac = "BC:24:11:EA:1D:38" }
+    "802" = { node = "neptune", hostname = "rustfs-3.srv.renner.uno", mac = "BC:24:11:8F:F4:67" }
   }
 }
 
 resource "proxmox_virtual_environment_container" "rustfs_decommission" {
   for_each = local.rustfs_decommission
 
-  node_name     = each.value
+  node_name     = each.value.node
   vm_id         = tonumber(each.key)
   started       = false
   unprivileged  = true
   start_on_boot = true
 
+  tags = [
+    "critical",
+    "rustfs",
+    "state-backend",
+    "terraform-backend",
+  ]
+
   timeout_create = 1800
   timeout_clone  = 1800
   timeout_update = 1800
   timeout_delete = 1800
   timeout_start  = 300
+
+  cpu {
+    architecture = "amd64"
+    cores        = 1
+    units        = 1024
+  }
+
+  memory {
+    dedicated = 1024
+    swap      = 0
+  }
+
+  disk {
+    datastore_id = "local"
+    size         = 10
+  }
+
+  # forces replacement if absent — this is the block that broke v2.
+  mount_point {
+    path      = "/mnt/data"
+    size      = "50G"
+    volume    = "local:${each.key}/vm-${each.key}-disk-1.raw"
+    backup    = true
+    replicate = true
+  }
+
+  network_interface {
+    name        = "eth0"
+    bridge      = "v40srv"
+    enabled     = true
+    firewall    = false
+    mac_address = each.value.mac
+  }
+
+  initialization {
+    hostname = each.value.hostname
+
+    ip_config {
+      ipv4 {
+        address = "dhcp"
+      }
+    }
+  }
+
+  # operating_system is NOT declared, and must not be. The provider requires
+  # template_file_id inside it, which is a create-time-only value: PVE does not
+  # report the original template for an existing container, so it is absent
+  # from state and cannot be mirrored. Declaring the block without it fails
+  # validation ("Missing required argument"); declaring it with a guessed
+  # template would be a lie that only surfaces if something ever recreates
+  # these.
+  #
+  # Left undeclared it would plan as a removal, and removing operating_system
+  # forces replacement — the exact failure this file exists to avoid. So it is
+  # ignored instead. The timeouts above are deliberately NOT ignored: they are
+  # the one thing this shim needs to actually apply.
+  lifecycle {
+    ignore_changes = [operating_system]
+  }
 }

opentofu/providers.tf

@@ -61,11 +61,27 @@ provider "unifi" {
 # as the unifi provider above; the v2 endpoints are simply not covered by it.
 # UniFi PUT quirk: updates may require _id in the body, which restapi does
 # not send — if an in-place update 400s, taint the object so CI recreates it.
+#
+# create_returns_object, NOT write_returns_object. Both make the provider store
+# the API response into `data`; write_returns_object does it for PUT as well as
+# POST. Since `data` in unifi_nat_hairpin.tf never sets the UDM-assigned `_id`,
+# every write put a field into state that config does not have, so all 60 rules
+# planned as "will be updated in-place" with `- _id` as the whole diff — and
+# re-dirtied themselves on the next write, forever.
+#
+# No provider-side ignore can suppress that: `data` is a configured attribute,
+# compared by Terraform core before provider logic runs. Tried and disproven —
+# ignore_changes_to = ["_id"] (plan 5661), ignore_all_server_changes (5662),
+# and ignore_server_additions on restapi 3.x (5667). All three left the 60.
+#
+# create_returns_object keeps what the provider actually needs — the POST
+# response is how it learns `_id` for id_attribute below — without letting
+# updates rewrite `data`.
 provider "restapi" {
-  uri                  = "${var.unifi_api_url}/proxy/network/v2/api/site/default"
-  insecure             = var.unifi_insecure
-  write_returns_object = true
-  id_attribute         = "_id"
+  uri                   = "${var.unifi_api_url}/proxy/network/v2/api/site/default"
+  insecure              = var.unifi_insecure
+  create_returns_object = true
+  id_attribute          = "_id"
   headers = {
     "X-API-KEY"    = var.unifi_api_key
     "Content-Type" = "application/json"