kubernetes/apps/woodpecker/MIGRATION-namespace.md

@@ -326,6 +326,56 @@ Revert the PR. Flux recreates the `apps` workloads and the DSN points back at
 cluster during the window is lost, so roll back promptly or not at all — if
 pipelines have run against `woodpecker/wp-pg`, prefer fixing forward.
 
+## Stage 3 — retiring the old cluster
+
+Deletes the four `apps` manifests, promotes the `migration-*` set onto their
+names, and gives the surviving Cluster the `prune: disabled` data guard it was
+deliberately denied while a rollback still existed.
+
+**Merging stage 3 does not delete anything.** The old resources carry
+`kustomize.toolkit.fluxcd.io/prune: disabled`, so dropping them from git only
+stops Flux managing them — they keep running, and keep occupying their PVC.
+That is the point: the PR is reversible, the teardown is a separate, deliberate
+act.
+
+Before deleting anything, confirm the new cluster is genuinely standing on its
+own — that it has taken at least one base backup to its own prefix, so recovery
+no longer depends on the retired cluster's data:
+
+```sh
+kubectl -n woodpecker get backup.postgresql.cnpg.io
+kubectl -n woodpecker get cluster wp-pg -o jsonpath='{.status.conditions}'
+```
+
+`LastBackupSucceeded` must be `True`, not merely `BackupStarted`.
+
+Then retire the old set:
+
+```sh
+kubectl -n apps delete scheduledbackup.postgresql.cnpg.io wp-pg-daily
+kubectl -n apps delete cluster.postgresql.cnpg.io wp-pg          # deletes its PVCs
+kubectl -n apps delete objectstore.barmancloud.cnpg.io wp-pg-store
+kubectl -n apps delete externalsecret hetzner-s3-creds
+```
+
+Delete the ScheduledBackup first: leaving it while the Cluster goes away
+produces failing backup jobs against a cluster that no longer exists.
+
+**What this destroys and what it does not.** It destroys the old cluster's PVCs
+— including the four pipeline records that only ever existed there (see the
+stage-1 verification section). It does **not** touch
+`s3://renner-uno-k8s-cnpg/wp-pg`: that prefix keeps its backups until their own
+retention expires, and `wp-pg-store-source` is retained precisely so it stays
+restorable. That is the real safety net, not the running cluster.
+
+### After the teardown
+
+The `apps` namespace should have no woodpecker-related objects left:
+
+```sh
+kubectl -n apps get cluster,objectstore,scheduledbackup,externalsecret 2>/dev/null | grep -i "wp-pg\|hetzner" || echo "clean"
+```
+
 ## Rolling back stage 1
 
 Delete the four resources and the namespace. Nothing references them, and the

kubernetes/apps/woodpecker/cluster.yaml

@@ -1,19 +1,27 @@
+---
+# Woodpecker's Postgres backend. Serves the server at wp-pg-rw.woodpecker.svc.
+#
+# Originally created in `apps` with bootstrap.initdb; moved here by restoring
+# from barman into the new namespace rather than by editing a namespace field,
+# because PVCs are namespace-scoped and a Cluster cannot be relocated in place
+# (issue #433, MIGRATION-namespace.md).
+#
+# The bootstrap.recovery stanza and its externalCluster are retained rather than
+# reverted to initdb. CNPG honours `bootstrap` only when PGDATA is absent, so on
+# this live cluster it is inert — and keeping it means a cold rebuild of an
+# empty cluster restores from backup instead of coming up empty, which is the
+# gap kubernetes/RECOVERY.md exists to close for the other CNPG clusters.
 apiVersion: postgresql.cnpg.io/v1
 kind: Cluster
 metadata:
   name: wp-pg
-  namespace: apps
+  namespace: woodpecker
   annotations:
-    # CRITICAL data-guard: prune-disabled so the seed-woodpecker -> dedicated
-    # woodpecker KS ownership handoff can never prune this Cluster (which would
-    # delete its data PVCs). Landed here first, via seed-woodpecker, so it's
-    # live before the move. Kept permanently as a standing guard.
+    # Data guard, earned at stage 3 now that this is the only copy: prevents a
+    # Flux prune from deleting the Cluster and with it the data PVCs. The
+    # migration deliberately withheld this while a rollback still existed.
     kustomize.toolkit.fluxcd.io/prune: disabled
 spec:
-  # Single instance: the Woodpecker CI DB is small and non-critical (state is
-  # largely reconstructable), so HA isn't warranted — but WAL archiving to
-  # object storage still gives point-in-time backups. Mirrors grafana-db's
-  # single-instance shape, with barman enabled like immich/paperless.
   instances: 1
   imageName: ghcr.io/cloudnative-pg/postgresql:16
   storage:
@@ -25,9 +33,25 @@ spec:
       parameters:
         barmanObjectName: wp-pg-store
   bootstrap:
-    initdb:
+    recovery:
+      source: wp-pg-apps
+      # Must mirror the initdb values of the cluster this restores from
+      # (apps/wp-pg: database woodpecker, owner woodpecker). CNPG defaults
+      # `database` to `app` and `owner` to whatever `database` is, and those
+      # names do not exist in the physically restored data — it would provision
+      # an empty `app` database, point the generated wp-pg-app Secret and the
+      # -app Service at it, and leave the real data unreachable through them.
+      # Same pattern as every recovery example in kubernetes/RECOVERY.md.
       database: woodpecker
       owner: woodpecker
+  externalClusters:
+    - name: wp-pg-apps
+      plugin:
+        name: barman-cloud.cloudnative-pg.io
+        parameters:
+          barmanObjectName: wp-pg-store-source
+          # The cluster that produced the backup, not this one.
+          serverName: wp-pg
   postgresql:
     parameters:
       shared_buffers: "128MB"

kubernetes/apps/woodpecker/hetzner-creds-externalsecret.yaml

@@ -1,11 +1,20 @@
+---
+# Barman S3 credentials for wp-pg.
+#
+# Sourced from seed/paperless/hetzner-creds, which is not where they belong.
+# Issue #397 wants woodpecker to own seed/woodpecker/hetzner-creds; creating
+# that path needs OpenBao write access (`bao kv list seed` → 403 for the
+# identities available here), so it stays open. The values are identical, so it
+# is a one-line swap of the `key:` below once the path exists.
+#
+# The namespace allowlist on the openbao ClusterSecretStore
+# (infrastructure/post-crds/openbao-clustersecretstore.yaml) must include this
+# namespace, or ESO silently never syncs — see #505.
 apiVersion: external-secrets.io/v1
 kind: ExternalSecret
 metadata:
   name: hetzner-s3-creds
-  namespace: apps
-  annotations:
-    # Prune-disabled — a standing guard kept from the seed->dedicated-KS handoff (barman S3 creds).
-    kustomize.toolkit.fluxcd.io/prune: disabled
+  namespace: woodpecker
 spec:
   refreshInterval: 1h
   secretStoreRef:
@@ -14,10 +23,5 @@ spec:
   target:
     name: hetzner-s3-creds
   dataFrom:
-    # Reuses the paperless copy of the shared Hetzner S3 credentials (identical
-    # ACCESS_KEY_ID / ACCESS_SECRET_KEY / REGION) to avoid seeding a new OpenBao
-    # path for this migration. FOLLOW-UP: give woodpecker its own
-    # seed/woodpecker/hetzner-creds path once OpenBao can be written, and repoint
-    # this here (the value is the same, so it's a drop-in swap).
     - extract:
         key: seed/paperless/hetzner-creds

kubernetes/apps/woodpecker/kustomization.yaml

@@ -21,35 +21,20 @@ resources:
   # (kubernetes/infrastructure/openbao/rotation/woodpecker.yaml) — intra-ns
   # OpenBao access, no NetworkPolicy needed. server.yaml/agent.yaml keep their
   # Reloader annotations; externalsecret.yaml keeps the 10m refreshInterval.
-  # wp-pg CNPG Postgres backend (moved here from the retired seed-woodpecker KS;
-  # each object carries prune: disabled so ownership transferred without a
-  # teardown). The dedicated `woodpecker` KS depends on infrastructure-cnpg-barman
-  # for the barman plugin behind the ObjectStore/ScheduledBackup.
-  - cluster.yaml
-  - objectstore.yaml
-  - scheduled-backup.yaml
-  - hetzner-creds-externalsecret.yaml
   # --- namespace migration, stages 1-2 (issue #433) ---
   # Stage 2 has moved the workloads above (server, agent, service, ingress, pdb,
   # rbac, secrets, policies) into the `woodpecker` namespace and pointed the
-  # server's DSN at wp-pg-rw.woodpecker.svc.
-  #
-  # cluster.yaml / objectstore.yaml / scheduled-backup.yaml /
-  # hetzner-creds-externalsecret.yaml still describe the ORIGINAL apps-namespace
-  # cluster and are deliberately left in place: they are the rollback. Their
-  # replacements are the migration-* files below. Stage 3 deletes the old four
-  # and renames the migration-* set onto their names.
-  # Woodpecker is moving out of the shared `apps` namespace into its own.
-  # These resources restore wp-pg into the target namespace from barman,
-  # running in parallel with the live cluster. Nothing points at them yet, so
-  # this stage carries no CI impact and is revertible by deleting them.
-  # Cutover (server/agent/DSN/RBAC) is stage 2; removing the `apps` originals
-  # is stage 3. Runbook: MIGRATION-namespace.md.
+  # server's DSN at wp-pg-rw.woodpecker.svc. Stage 3 retired the `apps`
+  # originals; these four now describe the surviving cluster. History and the
+  # reasoning behind the three stages: MIGRATION-namespace.md.
   #
   # The `woodpecker` Namespace itself is declared in infrastructure/namespaces,
   # not here: it must exist before the `infrastructure-crds` KS applies the
   # namespace's CiliumNetworkPolicies, and this KS runs after both of those.
-  - migration-externalsecret-hetzner.yaml
-  - migration-objectstore.yaml
-  - migration-cluster.yaml
-  - migration-scheduled-backup.yaml
+  #
+  # This KS depends on infrastructure-cnpg-barman for the barman plugin behind
+  # the ObjectStore and ScheduledBackup.
+  - hetzner-creds-externalsecret.yaml
+  - objectstore.yaml
+  - cluster.yaml
+  - scheduled-backup.yaml

kubernetes/apps/woodpecker/migration-cluster.yaml

@@ -1,62 +0,0 @@
----
-# wp-pg restored into the `woodpecker` namespace from the barman backup of the
-# live apps/wp-pg. Runs in parallel with the original until cutover (PR 2);
-# nothing points at it yet, so it carries no traffic.
-#
-# Spec mirrors apps/wp-pg exactly — same instance count, image, storage class
-# and Postgres parameters — so the only variable under test is the restore
-# itself. The differences are deliberate and limited to:
-#   - bootstrap: recovery (not initdb), sourced from the old prefix
-#   - plugins.barmanObjectName: the NEW store, so it archives somewhere else
-#
-# NOT prune-disabled, unlike apps/wp-pg. That guard exists there because the
-# original holds the only copy of the data through a KS ownership handoff. This
-# one is reconstructable by re-running the restore, and a prune guard on a
-# migration artifact is how orphans get left behind if the migration is
-# abandoned. It gains the guard in PR 3, once it is the surviving cluster.
-apiVersion: postgresql.cnpg.io/v1
-kind: Cluster
-metadata:
-  name: wp-pg
-  namespace: woodpecker
-spec:
-  instances: 1
-  imageName: ghcr.io/cloudnative-pg/postgresql:16
-  storage:
-    storageClass: ceph-block
-    size: 5Gi
-  plugins:
-    - name: barman-cloud.cloudnative-pg.io
-      isWALArchiver: true
-      parameters:
-        barmanObjectName: wp-pg-store
-  bootstrap:
-    recovery:
-      source: wp-pg-apps
-      # Must mirror the initdb values of the cluster this restores from
-      # (apps/wp-pg: database woodpecker, owner woodpecker). CNPG defaults
-      # `database` to `app` and `owner` to whatever `database` is, and those
-      # names do not exist in the physically restored data — it would provision
-      # an empty `app` database, point the generated wp-pg-app Secret and the
-      # -app Service at it, and leave the real data unreachable through them.
-      # Same pattern as every recovery example in kubernetes/RECOVERY.md.
-      database: woodpecker
-      owner: woodpecker
-  externalClusters:
-    - name: wp-pg-apps
-      plugin:
-        name: barman-cloud.cloudnative-pg.io
-        parameters:
-          barmanObjectName: wp-pg-store-source
-          # The cluster that produced the backup, not this one.
-          serverName: wp-pg
-  postgresql:
-    parameters:
-      shared_buffers: "128MB"
-      max_connections: "50"
-  resources:
-    requests:
-      cpu: 100m
-      memory: 256Mi
-    limits:
-      memory: 512Mi

kubernetes/apps/woodpecker/migration-externalsecret-hetzner.yaml

@@ -1,28 +0,0 @@
----
-# Barman S3 credentials for the wp-pg cluster in its new `woodpecker` home.
-# Secrets are namespace-scoped, so the copy in `apps` cannot be reused; this is
-# the same material fetched into the new namespace.
-#
-# Still sourced from seed/paperless/hetzner-creds, deliberately. Issue #397
-# wants woodpecker to own seed/woodpecker/hetzner-creds instead, and this
-# migration would have been the natural moment — but creating that path needs
-# OpenBao write access that is not available here (`bao kv list seed` → 403).
-# Repointing at a path that may not exist would leave the migrated cluster
-# unable to archive WAL, which is exactly what the restore is meant to prove
-# works. #397 stays open and is a drop-in swap of the `key:` below once the
-# path exists; the values are identical.
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: hetzner-s3-creds
-  namespace: woodpecker
-spec:
-  refreshInterval: 1h
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: hetzner-s3-creds
-  dataFrom:
-    - extract:
-        key: seed/paperless/hetzner-creds

kubernetes/apps/woodpecker/migration-objectstore.yaml

@@ -1,64 +0,0 @@
----
-# RECOVERY SOURCE — read-only. Points at the prefix the live `apps/wp-pg`
-# cluster has been archiving to since it was created. The restored cluster
-# bootstraps from here and never writes to it.
-#
-# serverName on the consuming externalCluster must be `wp-pg`, the name of the
-# cluster that produced these backups.
-apiVersion: barmancloud.cnpg.io/v1
-kind: ObjectStore
-metadata:
-  name: wp-pg-store-source
-  namespace: woodpecker
-spec:
-  configuration:
-    destinationPath: s3://renner-uno-k8s-cnpg/wp-pg
-    endpointURL: https://fsn1.your-objectstorage.com
-    s3Credentials:
-      accessKeyId:
-        name: hetzner-s3-creds
-        key: ACCESS_KEY_ID
-      secretAccessKey:
-        name: hetzner-s3-creds
-        key: ACCESS_SECRET_KEY
-      region:
-        name: hetzner-s3-creds
-        key: REGION
-    wal:
-      compression: gzip
-    data:
-      compression: gzip
-  # No retentionPolicy: this store is a read source during the migration, and
-  # the old cluster in `apps` owns the lifecycle of the data underneath it.
----
-# ARCHIVING TARGET — a DIFFERENT prefix from the source, deliberately.
-#
-# Two clusters archiving WAL to one destinationPath interleave their streams
-# and corrupt both. During the parallel-run window the old apps/wp-pg is still
-# live and still archiving to s3://renner-uno-k8s-cnpg/wp-pg, so the restored
-# cluster must not share it. Splitting the paths also keeps the pre-migration
-# backup intact and independently restorable as a rollback.
-apiVersion: barmancloud.cnpg.io/v1
-kind: ObjectStore
-metadata:
-  name: wp-pg-store
-  namespace: woodpecker
-spec:
-  configuration:
-    destinationPath: s3://renner-uno-k8s-cnpg/wp-pg-woodpecker
-    endpointURL: https://fsn1.your-objectstorage.com
-    s3Credentials:
-      accessKeyId:
-        name: hetzner-s3-creds
-        key: ACCESS_KEY_ID
-      secretAccessKey:
-        name: hetzner-s3-creds
-        key: ACCESS_SECRET_KEY
-      region:
-        name: hetzner-s3-creds
-        key: REGION
-    wal:
-      compression: gzip
-    data:
-      compression: gzip
-  retentionPolicy: "30d"

kubernetes/apps/woodpecker/migration-scheduled-backup.yaml

@@ -1,32 +0,0 @@
----
-# Base backups for the restored cluster, onto the new wp-pg-woodpecker prefix.
-#
-# Without this the cluster archives continuous WAL to that prefix but never
-# takes a base backup there, which leaves two problems: the prefix has no
-# independent recovery point of its own (recovery would have to chain all the
-# way back through the source prefix), and wp-pg-store's retentionPolicy: 30d
-# has no base backups to prune against, so the WAL chain grows unbounded for as
-# long as stage 1 sits before cutover.
-#
-# 02:40 UTC, offset from the apps cluster's 02:35 and the other clusters' 02:30
-# so the parallel-run window does not put two woodpecker base backups against
-# the Hetzner bucket at once. immediate: true takes the first as soon as the
-# restore finishes, which is also the cheapest confirmation that the new
-# credentials and the new ObjectStore both work end to end.
-#
-# Not prune-disabled, for the same reason as migration-cluster.yaml: this is a
-# migration artifact until stage 3 promotes it.
-apiVersion: postgresql.cnpg.io/v1
-kind: ScheduledBackup
-metadata:
-  name: wp-pg-daily
-  namespace: woodpecker
-spec:
-  schedule: "0 40 2 * * *"
-  immediate: true
-  backupOwnerReference: self
-  cluster:
-    name: wp-pg
-  method: plugin
-  pluginConfiguration:
-    name: barman-cloud.cloudnative-pg.io

kubernetes/apps/woodpecker/objectstore.yaml

@@ -1,18 +1,20 @@
 ---
+# RECOVERY SOURCE — read-only. The prefix the retired `apps/wp-pg` archived to.
+#
+# Kept after the migration on purpose. Nothing writes to it, and the backups
+# under it age out on their own retention, but while they last this is a second,
+# independent restore point that predates the move — worth having until the new
+# prefix has a meaningful backup history of its own.
+#
+# serverName on the consuming externalCluster must be `wp-pg`, the name of the
+# cluster that produced these backups.
 apiVersion: barmancloud.cnpg.io/v1
 kind: ObjectStore
 metadata:
-  name: wp-pg-store
-  namespace: apps
-  annotations:
-    # Prune-disabled — a standing guard kept from the seed->dedicated-KS handoff (backup target).
-    kustomize.toolkit.fluxcd.io/prune: disabled
+  name: wp-pg-store-source
+  namespace: woodpecker
 spec:
   configuration:
-    # Same offsite Hetzner Object Storage bucket as the other CNPG clusters,
-    # under a per-cluster prefix. See apps/paperless/objectstore.yaml for the
-    # SSE / addressing rationale (Hetzner is SSE-C only; transit is TLS,
-    # bucket private).
     destinationPath: s3://renner-uno-k8s-cnpg/wp-pg
     endpointURL: https://fsn1.your-objectstorage.com
     s3Credentials:
@@ -29,4 +31,36 @@ spec:
       compression: gzip
     data:
       compression: gzip
+  # No retentionPolicy: this store is a read source during the migration, and
+  # the old cluster in `apps` owns the lifecycle of the data underneath it.
+---
+# ARCHIVING TARGET — a DIFFERENT prefix from the source, deliberately.
+#
+# Two clusters archiving WAL to one destinationPath interleave their streams and
+# corrupt both. That mattered during the parallel-run window; it still matters
+# now, because the source prefix above is retained and must stay frozen for the
+# independent restore point it provides.
+apiVersion: barmancloud.cnpg.io/v1
+kind: ObjectStore
+metadata:
+  name: wp-pg-store
+  namespace: woodpecker
+spec:
+  configuration:
+    destinationPath: s3://renner-uno-k8s-cnpg/wp-pg-woodpecker
+    endpointURL: https://fsn1.your-objectstorage.com
+    s3Credentials:
+      accessKeyId:
+        name: hetzner-s3-creds
+        key: ACCESS_KEY_ID
+      secretAccessKey:
+        name: hetzner-s3-creds
+        key: ACCESS_SECRET_KEY
+      region:
+        name: hetzner-s3-creds
+        key: REGION
+    wal:
+      compression: gzip
+    data:
+      compression: gzip
   retentionPolicy: "30d"

kubernetes/apps/woodpecker/scheduled-backup.yaml

@@ -1,18 +1,19 @@
 ---
-# Daily base backup at 02:35 UTC (offset from the other clusters' 02:30 to
-# avoid a thundering herd against the Hetzner bucket). WAL archiving runs
-# continuously (see cluster.yaml spec.plugins[].isWALArchiver). immediate:true
-# takes the first base backup as soon as this CR applies.
+# Daily base backups onto the wp-pg-woodpecker prefix. Continuous WAL archiving
+# runs separately (cluster.yaml spec.plugins[].isWALArchiver); without periodic
+# base backups the prefix would have no recovery point of its own and
+# wp-pg-store's retentionPolicy: 30d would have nothing to prune against, so the
+# WAL chain would grow without bound.
+#
+# 02:40 UTC, offset from the other CNPG clusters' 02:30 so they do not all hit
+# the Hetzner bucket together.
 apiVersion: postgresql.cnpg.io/v1
 kind: ScheduledBackup
 metadata:
   name: wp-pg-daily
-  namespace: apps
-  annotations:
-    # Prune-disabled — a standing guard kept from the seed->dedicated-KS handoff (backup schedule).
-    kustomize.toolkit.fluxcd.io/prune: disabled
+  namespace: woodpecker
 spec:
-  schedule: "0 35 2 * * *"
+  schedule: "0 40 2 * * *"
   immediate: true
   backupOwnerReference: self
   cluster: