fix(woodpecker): add the missing PodMonitor and base backup for wp-pg

Addresses the third blocking review finding on #504.

The PR claimed the namespace move fixes wp-pg's absent metrics. It does not,
on its own. Prometheus requires the `release: kube-prometheus-stack` label
that CNPG's own enablePodMonitor omits, so every scraped cluster has a
hand-written PodMonitor in apps/monitoring-rules/cnpg-podmonitors.yaml —
ak-pg, im-pg, gf-pg, pprl-pg. There has never been one for wp-pg.

So wp-pg was missing both halves: no scrape target configured, and no
monitoring ingress allow in the apps baseline. Fixing the policy half was
necessary but not sufficient, and the runbook's cnpg_collector_up check
would have returned empty either way — unable to confirm the thing it
existed to confirm. Adds the PodMonitor in the woodpecker namespace.

Also from the review:

- Added a ScheduledBackup for the restored cluster. Every other CNPG
  cluster here pairs isWALArchiver with one. Without it the new
  wp-pg-woodpecker prefix takes continuous WAL but never a base backup, so
  it has no independent recovery point and wp-pg-store's 30d retention has
  nothing to prune against — the WAL chain grows unbounded for as long as
  stage 1 sits before cutover. Scheduled 02:40, clear of the apps cluster's
  02:35 and the other clusters' 02:30.

- Documented that the `woodpecker` KS goes not-Ready while the restore runs.
  Its healthCheckExprs matches Cluster by GVK with no name qualifier, so
  Flux evaluates it against both the live apps/wp-pg and the restoring
  woodpecker/wp-pg, and wait: true makes the KS that owns production
  Woodpecker report unhealthy. Expected, does not cascade since nothing
  dependsOn it, but it would read as an outage without a note.

- Dropped the app.kubernetes.io/name label from the new Namespace; no other
  namespace in that file carries one.

Co-Authored-By: Claude Bot <claude-bot@renner.uno>

kubernetes/apps/monitoring-rules/cnpg-podmonitors.yaml

@@ -73,3 +73,29 @@ spec:
       interval: 30s
   podTargetLabels:
     - role
+---
+# wp-pg, restored into the `woodpecker` namespace (issue #433). It has never
+# been scraped: `apps` had no PodMonitor for it AND its baseline network policy
+# never allowed the monitoring namespace, so `cnpg_collector_up` has series for
+# the four clusters above but not this one. The namespace move fixes the policy
+# half; without this PodMonitor there is still no scrape target at all.
+#
+# Deliberately not adding one for the cluster still in `apps`: it is retired in
+# stage 3, and a PodMonitor there would only be reconciled to be deleted again.
+apiVersion: monitoring.coreos.com/v1
+kind: PodMonitor
+metadata:
+  name: cnpg-wp-pg
+  namespace: woodpecker
+  labels:
+    app.kubernetes.io/name: cloudnative-pg
+    release: kube-prometheus-stack
+spec:
+  selector:
+    matchLabels:
+      cnpg.io/cluster: wp-pg
+  podMetricsEndpoints:
+    - port: metrics
+      interval: 30s
+  podTargetLabels:
+    - role

kubernetes/apps/woodpecker/MIGRATION-namespace.md

@@ -82,6 +82,24 @@ that may not exist would leave the restored cluster unable to archive WAL,
 which is the thing this stage exists to prove. #397 stays open; the swap is one
 line once the path exists.
 
+## Expect the `woodpecker` Kustomization to go not-Ready during the restore
+
+`kubernetes/clusters/homelab/woodpecker.yaml` sets `wait: true` and a
+`healthCheckExprs` entry matching `postgresql.cnpg.io/v1 Cluster` **with no
+name or namespace qualifier**. Flux evaluates it against every Cluster in that
+Kustomization's inventory — which, after this stage, is both the live
+`apps/wp-pg` and the restoring `woodpecker/wp-pg`.
+
+So while the restore runs, the whole `woodpecker` KS — the one that owns
+production Woodpecker — reports not-Ready, and a reconcile timeout in that
+window will trip the `flux-failures` Alert. **That is expected and is not
+Woodpecker breaking.** Nothing `dependsOn: woodpecker`, so it does not cascade.
+
+If the restore is going to sit for a long time, or the noise is unwelcome, the
+options are to suspend the KS while it completes, or to split the migration
+resources into their own `wait: false` Kustomization. For a ~660 MB database the
+window should be minutes.
+
 ## Verifying stage 1
 
 Restore is asynchronous. Watch it reach a healthy state:
@@ -136,10 +154,19 @@ Confirm the two fixes this move exists for:
 ```sh
 # #433: operator can now reach the instance
 kubectl -n woodpecker get cluster wp-pg -o jsonpath='{.status.conditions}'
-# metrics now scraped
+# metrics now scraped — needs BOTH the PodMonitor (cnpg-podmonitors.yaml) and
+# the monitoring ingress allow; wp-pg had neither in `apps`, which is why it is
+# absent from cnpg_collector_up while ak-pg/im-pg/gf-pg/pprl-pg are present
 kubectl get --raw '/api/v1/namespaces/monitoring/services/kube-prometheus-stack-prometheus:9090/proxy/api/v1/query?query=cnpg_collector_up{namespace="woodpecker"}'
 ```
 
+Confirm the scrape target is actually registered, not just that the metric is
+missing for some other reason:
+
+```sh
+kubectl -n woodpecker get podmonitor cnpg-wp-pg
+```
+
 **Check the policy semantics by hand.** `kubectl --dry-run=server` validates
 manifest shape, not Cilium behaviour, so it cannot catch a rule that is
 syntactically fine but scoped wider than intended. Confirm what actually reaches

kubernetes/apps/woodpecker/kustomization.yaml

@@ -43,3 +43,4 @@ resources:
   - migration-externalsecret-hetzner.yaml
   - migration-objectstore.yaml
   - migration-cluster.yaml
+  - migration-scheduled-backup.yaml

kubernetes/apps/woodpecker/migration-scheduled-backup.yaml

@@ -0,0 +1,32 @@
+---
+# Base backups for the restored cluster, onto the new wp-pg-woodpecker prefix.
+#
+# Without this the cluster archives continuous WAL to that prefix but never
+# takes a base backup there, which leaves two problems: the prefix has no
+# independent recovery point of its own (recovery would have to chain all the
+# way back through the source prefix), and wp-pg-store's retentionPolicy: 30d
+# has no base backups to prune against, so the WAL chain grows unbounded for as
+# long as stage 1 sits before cutover.
+#
+# 02:40 UTC, offset from the apps cluster's 02:35 and the other clusters' 02:30
+# so the parallel-run window does not put two woodpecker base backups against
+# the Hetzner bucket at once. immediate: true takes the first as soon as the
+# restore finishes, which is also the cheapest confirmation that the new
+# credentials and the new ObjectStore both work end to end.
+#
+# Not prune-disabled, for the same reason as migration-cluster.yaml: this is a
+# migration artifact until stage 3 promotes it.
+apiVersion: postgresql.cnpg.io/v1
+kind: ScheduledBackup
+metadata:
+  name: wp-pg-daily
+  namespace: woodpecker
+spec:
+  schedule: "0 40 2 * * *"
+  immediate: true
+  backupOwnerReference: self
+  cluster:
+    name: wp-pg
+  method: plugin
+  pluginConfiguration:
+    name: barman-cloud.cloudnative-pg.io

kubernetes/infrastructure/namespaces/namespaces.yaml

@@ -183,5 +183,3 @@ apiVersion: v1
 kind: Namespace
 metadata:
   name: woodpecker
-  labels:
-    app.kubernetes.io/name: woodpecker