feat(woodpecker): stage 1 — restore wp-pg into a dedicated namespace #504

Merged
k.renner merged 5 commits from migrate-woodpecker-namespace into master 2026-08-18 20:56:43 +00:00
10 changed files with 575 additions and 0 deletions
@@ -73,3 +73,29 @@ spec:
interval: 30s
podTargetLabels:
- role
---
# wp-pg, restored into the `woodpecker` namespace (issue #433). It has never
# been scraped: `apps` had no PodMonitor for it AND its baseline network policy
# never allowed the monitoring namespace, so `cnpg_collector_up` has series for
# the four clusters above but not this one. The namespace move fixes the policy
# half; without this PodMonitor there is still no scrape target at all.
#
# Deliberately not adding one for the cluster still in `apps`: it is retired in
# stage 3, and a PodMonitor there would only be reconciled to be deleted again.
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
name: cnpg-wp-pg
namespace: woodpecker
labels:
app.kubernetes.io/name: cloudnative-pg
release: kube-prometheus-stack
spec:
selector:
matchLabels:
cnpg.io/cluster: wp-pg
podMetricsEndpoints:
- port: metrics
interval: 30s
podTargetLabels:
- role
@@ -0,0 +1,204 @@
# Woodpecker: `apps` → `woodpecker` namespace migration
Moves the Woodpecker service — server, agent and the `wp-pg` CNPG backend — out
of the shared `apps` namespace into one of its own. Distinct from
`MIGRATION.md`, which covers the earlier SQLite → CNPG move.
## Why
`apps` holds gitea, gatus, external-dns, ical-proxy and woodpecker together,
and its `default-deny-ingress` uses `endpointSelector: {}`. Every allow rule it
carries therefore applies to *every* workload in the namespace.
That is not academic. Issue #433 — `wp-pg` stuck at `Ready=False` with
`Instance Status Extraction Error` for three weeks — is caused by the CNPG
operator in `database` being unable to reach `https://<pod-ip>:8000/pg/status`.
The only namespace-level fix is `fromEndpoints: namespace=database`, which
would expose gitea, gatus, external-dns and ical-proxy to the whole `database`
namespace to solve a problem affecting one pod on one port.
`cnpg_collector_up` has series for authentik, immich, monitoring and paperless
but not for `wp-pg`, for the same reason: `apps` never allowed the monitoring
namespace either.
A namespace per service keeps each grant on the workload that needs it.
**`woodpecker-ci` is not the target.** It enforces `pod-security: privileged`
and exists solely for ephemeral pipeline step pods running repo-supplied code.
Putting the database there would let any build step reach it through the
intra-namespace allow.
## Staging
Rollback is trivial through stage 2. The `apps` cluster is untouched until
stage 3, so reverting means pointing the DSN back and scaling up.
| Stage | Contents | CI impact |
|---|---|---|
| **1** (this PR) | `woodpecker` namespace, its network policies, ExternalSecret, ObjectStores, and `wp-pg` restored from barman | none — parallel, nothing points at it |
| **runbook** | verify the restore holds real data | none |
| **2** | move server + agent, flip DSN, RBAC subject, rotator policies, docs | **downtime** |
| **3** | delete the `apps` originals, add the prune guard to the survivor | none |
## Stage 1 design notes
**Two ObjectStores, deliberately.** `wp-pg-store-source` points at
`s3://renner-uno-k8s-cnpg/wp-pg` — the prefix the live cluster archives to —
and is read-only. `wp-pg-store` points at a **new** prefix,
`s3://renner-uno-k8s-cnpg/wp-pg-woodpecker`, for ongoing archiving.
Both clusters writing one `destinationPath` would interleave their WAL streams
and corrupt both. Splitting them also leaves the pre-migration backup intact
and independently restorable.
**`serverName: wp-pg`** on the `externalCluster` names the cluster that
*produced* the backup, not the one being created.
**The namespace is declared in `infrastructure/namespaces/namespaces.yaml`,**
not alongside these workloads. Its CiliumNetworkPolicies ship with the
`infrastructure-crds` KS (`infrastructure/post-crds` includes
`../network-policies/`), which `dependsOn` `infrastructure` — so the namespace
exists before any policy targets it. Creating it in the `woodpecker` workload
KS would invert that order, since that KS runs after both.
**Ingress from `woodpecker-ci` is deliberately absent in stage 1.** Step pods do
call the server back on `:8000`, but the CNPG instance manager serves its status
API on `:8000` too, and the baseline policy uses `endpointSelector: {}`. A
`toPorts` block narrows the port, not the destination workload — so putting that
rule on the baseline would hand arbitrary repo-supplied build steps a route to
wp-pg's instance manager. It lands in stage 2 as its own policy selecting the
server pod.
**No `prune: disabled` guard on the restored cluster.** The original carries one
because it holds the only copy of the data. This one is reconstructable by
re-running the restore, and a prune guard on a migration artifact is how orphans
survive an abandoned migration. It gains the guard in stage 3.
**Barman credentials still come from `seed/paperless/hetzner-creds`.** Issue
#397 wants woodpecker to own `seed/woodpecker/hetzner-creds`, and this would
have been the natural moment — but creating that path needs OpenBao write
access that was not available (`bao kv list seed` → 403). Repointing at a path
that may not exist would leave the restored cluster unable to archive WAL,
which is the thing this stage exists to prove. #397 stays open; the swap is one
line once the path exists.
## Expect the `woodpecker` Kustomization to go not-Ready during the restore
`kubernetes/clusters/homelab/woodpecker.yaml` sets `wait: true` and a
`healthCheckExprs` entry matching `postgresql.cnpg.io/v1 Cluster` **with no
name or namespace qualifier**. Flux evaluates it against every Cluster in that
Kustomization's inventory — which, after this stage, is both the live
`apps/wp-pg` and the restoring `woodpecker/wp-pg`.
So while the restore runs, the whole `woodpecker` KS — the one that owns
production Woodpecker — reports not-Ready, and a reconcile timeout in that
window will trip the `flux-failures` Alert. **That is expected and is not
Woodpecker breaking.** Nothing `dependsOn: woodpecker`, so it does not cascade.
If the restore is going to sit for a long time, or the noise is unwelcome, the
options are to suspend the KS while it completes, or to split the migration
resources into their own `wait: false` Kustomization. For a ~660 MB database the
window should be minutes.
## Verifying stage 1
Restore is asynchronous. Watch it reach a healthy state:
```sh
kubectl -n woodpecker get cluster wp-pg -w
kubectl -n woodpecker get pods
```
Expect `Cluster in healthy state`. If it sticks in bootstrap, the recovery job's
logs carry the reason:
```sh
kubectl -n woodpecker logs -l cnpg.io/jobRole=full-recovery --tail=100
```
**First check the generated app credentials target the right database.** CNPG
defaults `bootstrap.recovery.database` to `app`; the restored data uses
`woodpecker`. If this reads anything but `woodpecker`, the cluster has
provisioned an empty `app` database and the real data is unreachable through
the `-app` Secret and Service:
```sh
kubectl -n woodpecker get secret wp-pg-app -o jsonpath='{.data.dbname}' | base64 -d; echo
kubectl -n woodpecker get secret wp-pg-app -o jsonpath='{.data.username}' | base64 -d; echo
```
**Then confirm the restore actually holds Woodpecker's data** — a cluster that
comes up healthy but empty is the failure mode worth catching:
```sh
PGPASS=$(kubectl -n woodpecker get secret wp-pg-app -o jsonpath='{.data.password}' | base64 -d)
kubectl -n woodpecker exec -it wp-pg-1 -- env PGPASSWORD="$PGPASS" \
psql -U woodpecker -d woodpecker -c '\dt'
kubectl -n woodpecker exec -it wp-pg-1 -- env PGPASSWORD="$PGPASS" \
psql -U woodpecker -d woodpecker -c 'select count(*) from repos;'
```
Row counts should match the live cluster:
```sh
PGPASS_OLD=$(kubectl -n apps get secret wp-pg-app -o jsonpath='{.data.password}' | base64 -d)
kubectl -n apps exec -it wp-pg-1 -- env PGPASSWORD="$PGPASS_OLD" \
psql -U woodpecker -d woodpecker -c 'select count(*) from repos;'
```
They will not match exactly if builds ran after the backup the restore used —
that is expected, and stage 2 quiesces the server before the final cutover.
Confirm the two fixes this move exists for:
```sh
# #433: operator can now reach the instance
kubectl -n woodpecker get cluster wp-pg -o jsonpath='{.status.conditions}'
# metrics now scraped — needs BOTH the PodMonitor (cnpg-podmonitors.yaml) and
# the monitoring ingress allow; wp-pg had neither in `apps`, which is why it is
# absent from cnpg_collector_up while ak-pg/im-pg/gf-pg/pprl-pg are present
kubectl get --raw '/api/v1/namespaces/monitoring/services/kube-prometheus-stack-prometheus:9090/proxy/api/v1/query?query=cnpg_collector_up{namespace="woodpecker"}'
```
Confirm the scrape target is actually registered, not just that the metric is
missing for some other reason:
```sh
kubectl -n woodpecker get podmonitor cnpg-wp-pg
```
**Check the policy semantics by hand.** `kubectl --dry-run=server` validates
manifest shape, not Cilium behaviour, so it cannot catch a rule that is
syntactically fine but scoped wider than intended. Confirm what actually reaches
wp-pg's instance-manager port — the operator should, a CI step pod should not:
```sh
CILIUM=$(kubectl -n kube-system get pod -l k8s-app=cilium \
--field-selector spec.nodeName=$(kubectl -n woodpecker get pod wp-pg-1 -o jsonpath='{.spec.nodeName}') \
-o name | head -1)
# Expect ALLOWED
kubectl -n kube-system exec ${CILIUM#pod/} -c cilium-agent -- cilium-dbg policy trace \
--src-k8s-pod database:cloudnative-pg --dst-k8s-pod woodpecker:wp-pg-1 --dport 8000
# Expect DENIED — arbitrary build steps must not reach the database
kubectl -n kube-system exec ${CILIUM#pod/} -c cilium-agent -- cilium-dbg policy trace \
--src-k8s-pod woodpecker-ci:any --dst-k8s-pod woodpecker:wp-pg-1 --dport 8000
```
And that archiving goes to the new prefix, leaving the old one alone:
```sh
kubectl -n woodpecker get cluster wp-pg -o jsonpath='{.status.conditions[?(@.type=="ContinuousArchiving")]}'
```
## Rolling back stage 1
Delete the four resources and the namespace. Nothing references them, and the
`apps` cluster is untouched:
```sh
kubectl delete ns woodpecker
```
Then revert this PR so Flux stops recreating them.
@@ -29,3 +29,18 @@ resources:
- objectstore.yaml
- scheduled-backup.yaml
- hetzner-creds-externalsecret.yaml
# --- namespace migration, stage 1 (issue #433) ---
# Woodpecker is moving out of the shared `apps` namespace into its own.
# These resources restore wp-pg into the target namespace from barman,
# running in parallel with the live cluster. Nothing points at them yet, so
# this stage carries no CI impact and is revertible by deleting them.
# Cutover (server/agent/DSN/RBAC) is stage 2; removing the `apps` originals
# is stage 3. Runbook: MIGRATION-namespace.md.
#
# The `woodpecker` Namespace itself is declared in infrastructure/namespaces,
# not here: it must exist before the `infrastructure-crds` KS applies the
# namespace's CiliumNetworkPolicies, and this KS runs after both of those.
- migration-externalsecret-hetzner.yaml
- migration-objectstore.yaml
- migration-cluster.yaml
- migration-scheduled-backup.yaml
@@ -0,0 +1,62 @@
---
# wp-pg restored into the `woodpecker` namespace from the barman backup of the
# live apps/wp-pg. Runs in parallel with the original until cutover (PR 2);
# nothing points at it yet, so it carries no traffic.
#
# Spec mirrors apps/wp-pg exactly — same instance count, image, storage class
# and Postgres parameters — so the only variable under test is the restore
# itself. The differences are deliberate and limited to:
# - bootstrap: recovery (not initdb), sourced from the old prefix
# - plugins.barmanObjectName: the NEW store, so it archives somewhere else
#
# NOT prune-disabled, unlike apps/wp-pg. That guard exists there because the
# original holds the only copy of the data through a KS ownership handoff. This
# one is reconstructable by re-running the restore, and a prune guard on a
# migration artifact is how orphans get left behind if the migration is
# abandoned. It gains the guard in PR 3, once it is the surviving cluster.
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: wp-pg
namespace: woodpecker
spec:
instances: 1
imageName: ghcr.io/cloudnative-pg/postgresql:16
storage:
storageClass: ceph-block
size: 5Gi
plugins:
- name: barman-cloud.cloudnative-pg.io
isWALArchiver: true
parameters:
barmanObjectName: wp-pg-store
bootstrap:
recovery:
source: wp-pg-apps
# Must mirror the initdb values of the cluster this restores from
# (apps/wp-pg: database woodpecker, owner woodpecker). CNPG defaults
# `database` to `app` and `owner` to whatever `database` is, and those
# names do not exist in the physically restored data — it would provision
# an empty `app` database, point the generated wp-pg-app Secret and the
# -app Service at it, and leave the real data unreachable through them.
# Same pattern as every recovery example in kubernetes/RECOVERY.md.
database: woodpecker
owner: woodpecker
externalClusters:
- name: wp-pg-apps
plugin:
name: barman-cloud.cloudnative-pg.io
parameters:
barmanObjectName: wp-pg-store-source
# The cluster that produced the backup, not this one.
serverName: wp-pg
postgresql:
parameters:
shared_buffers: "128MB"
max_connections: "50"
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
memory: 512Mi
@@ -0,0 +1,28 @@
---
# Barman S3 credentials for the wp-pg cluster in its new `woodpecker` home.
# Secrets are namespace-scoped, so the copy in `apps` cannot be reused; this is
# the same material fetched into the new namespace.
#
# Still sourced from seed/paperless/hetzner-creds, deliberately. Issue #397
# wants woodpecker to own seed/woodpecker/hetzner-creds instead, and this
# migration would have been the natural moment — but creating that path needs
# OpenBao write access that is not available here (`bao kv list seed` → 403).
# Repointing at a path that may not exist would leave the migrated cluster
# unable to archive WAL, which is exactly what the restore is meant to prove
# works. #397 stays open and is a drop-in swap of the `key:` below once the
# path exists; the values are identical.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: hetzner-s3-creds
namespace: woodpecker
spec:
refreshInterval: 1h
secretStoreRef:
kind: ClusterSecretStore
name: openbao
target:
name: hetzner-s3-creds
dataFrom:
- extract:
key: seed/paperless/hetzner-creds
@@ -0,0 +1,64 @@
---
# RECOVERY SOURCE — read-only. Points at the prefix the live `apps/wp-pg`
# cluster has been archiving to since it was created. The restored cluster
# bootstraps from here and never writes to it.
#
# serverName on the consuming externalCluster must be `wp-pg`, the name of the
# cluster that produced these backups.
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: wp-pg-store-source
namespace: woodpecker
spec:
configuration:
destinationPath: s3://renner-uno-k8s-cnpg/wp-pg
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
# No retentionPolicy: this store is a read source during the migration, and
# the old cluster in `apps` owns the lifecycle of the data underneath it.
---
# ARCHIVING TARGET — a DIFFERENT prefix from the source, deliberately.
#
# Two clusters archiving WAL to one destinationPath interleave their streams
# and corrupt both. During the parallel-run window the old apps/wp-pg is still
# live and still archiving to s3://renner-uno-k8s-cnpg/wp-pg, so the restored
# cluster must not share it. Splitting the paths also keeps the pre-migration
# backup intact and independently restorable as a rollback.
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
metadata:
name: wp-pg-store
namespace: woodpecker
spec:
configuration:
destinationPath: s3://renner-uno-k8s-cnpg/wp-pg-woodpecker
endpointURL: https://fsn1.your-objectstorage.com
s3Credentials:
accessKeyId:
name: hetzner-s3-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: hetzner-s3-creds
key: ACCESS_SECRET_KEY
region:
name: hetzner-s3-creds
key: REGION
wal:
compression: gzip
data:
compression: gzip
retentionPolicy: "30d"
@@ -0,0 +1,32 @@
---
# Base backups for the restored cluster, onto the new wp-pg-woodpecker prefix.
#
# Without this the cluster archives continuous WAL to that prefix but never
# takes a base backup there, which leaves two problems: the prefix has no
# independent recovery point of its own (recovery would have to chain all the
# way back through the source prefix), and wp-pg-store's retentionPolicy: 30d
# has no base backups to prune against, so the WAL chain grows unbounded for as
# long as stage 1 sits before cutover.
#
# 02:40 UTC, offset from the apps cluster's 02:35 and the other clusters' 02:30
# so the parallel-run window does not put two woodpecker base backups against
# the Hetzner bucket at once. immediate: true takes the first as soon as the
# restore finishes, which is also the cheapest confirmation that the new
# credentials and the new ObjectStore both work end to end.
#
# Not prune-disabled, for the same reason as migration-cluster.yaml: this is a
# migration artifact until stage 3 promotes it.
apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
name: wp-pg-daily
namespace: woodpecker
spec:
schedule: "0 40 2 * * *"
immediate: true
backupOwnerReference: self
cluster:
name: wp-pg
method: plugin
pluginConfiguration:
name: barman-cloud.cloudnative-pg.io
@@ -160,3 +160,26 @@ apiVersion: v1
kind: Namespace
metadata:
name: wordpress
---
# The Woodpecker service itself — server, agent and its wp-pg Postgres backend,
# moving out of the shared `apps` namespace (issue #433). Distinct from
# `woodpecker-ci`, which enforces pod-security: privileged and holds only
# ephemeral pipeline step pods.
#
# `apps` uses an endpointSelector:{} default-deny, so every allow it carries
# applies to gitea, gatus, external-dns and ical-proxy alike — granting the
# CNPG operator access to wp-pg there would have opened all of `apps` to all of
# `database`. A namespace per service keeps that grant on the one workload.
#
# Declared here rather than alongside the workloads so it is created by the
# `infrastructure` KS. The namespace's CiliumNetworkPolicies ship with the
# `infrastructure-crds` KS (infrastructure/post-crds includes
# ../network-policies/), which itself dependsOn `infrastructure` — so the
# namespace exists before any policy targets it. Creating it in the `woodpecker`
# workload KS would invert that order, since that KS runs after both.
#
# No pod-security labels: the cluster default applies.
apiVersion: v1
kind: Namespace
metadata:
name: woodpecker
@@ -21,6 +21,7 @@ resources:
- openbao-ns.yaml
- external-secrets-ns.yaml
- woodpecker-ci-ns.yaml
- woodpecker-ns.yaml
- trmnl-ns.yaml
- wordpress-ns.yaml
- allow-cilium-health.yaml
@@ -0,0 +1,120 @@
---
# Baseline ingress policy for the `woodpecker` namespace (wp-pg now; server and
# agent arrive in stage 2).
#
# Unlike the `apps` baseline this replaces, every allow here lands on a
# namespace holding one service, so a namespace-scoped grant is not a blanket
# grant across unrelated workloads. That is the point of the move: in `apps`,
# letting the CNPG operator reach wp-pg would also have opened gitea, gatus,
# external-dns and ical-proxy to the whole database namespace (issue #433).
#
# CAUTION when adding rules here: this policy has `endpointSelector: {}`, so
# every rule below applies to EVERY pod in the namespace. A `toPorts` block
# narrows the port, NOT the destination workload — that is governed solely by
# the enclosing policy's endpointSelector. Anything meant for one workload
# belongs in its own policy with its own selector, as
# `wp-pg-allow-cnpg-operator` below does.
#
# No egress section, so egress is unrestricted. Deliberate for now and revisited
# in stage 2: wp-pg needs DNS plus `world` for WAL archiving to Hetzner S3, and
# the server/agent add gitea, the Kubernetes API and outbound plugin pulls.
# Writing that ruleset once, with the full workload set present, beats guessing
# it here and silently breaking the WAL archiving this stage exists to verify.
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: default-deny-ingress
namespace: woodpecker
spec:
endpointSelector: {}
ingress:
# Intra-namespace: server <-> agent, server -> wp-pg.
- fromEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": woodpecker
# Traefik terminates ci.renner.uno and forwards to the server (stage 2).
- fromEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": ingress
# Prometheus scrape. Namespace-wide and port-unrestricted, matching the
# other *-ns baselines; this is what covers wp-pg's :9187 metrics.
- fromEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": monitoring
# Kubelet probes and hostNetwork siblings.
- fromEntities:
- host
- remote-node
# ICMP — from any cluster pod. See home-assistant-ns.yaml for the rationale.
# INGRESS ONLY: do NOT add an egress section to this CNP.
#
# Needed here specifically because wp-pg-allow-cnpg-operator below attaches
# an L4 rule (toPorts :8000) to the wp-pg pods, and Cilium blocks ICMP to an
# endpoint once any layer 4 policy selects it unless the traffic is
# CT-related. Without this, PMTUD Destination Unreachable and Packet Too Big
# toward wp-pg are dropped silently — the failure shows up as stalled
# transfers, not as a policy denial.
#
# Namespace-wide is right for this one, unlike the :8000 case noted below:
# these are diagnostic and path-MTU messages that expose no API surface, so
# granting them across the namespace concedes nothing a per-workload rule
# would withhold.
- fromEndpoints:
- matchExpressions:
- key: io.kubernetes.pod.namespace
operator: Exists
icmps:
- fields:
- { type: 0, family: IPv4 }
- { type: 3, family: IPv4 }
- { type: 8, family: IPv4 }
- { type: 11, family: IPv4 }
- { type: 12, family: IPv4 }
- { type: 1, family: IPv6 }
- { type: 2, family: IPv6 }
- { type: 3, family: IPv6 }
- { type: 4, family: IPv6 }
- { type: 128, family: IPv6 }
- { type: 129, family: IPv6 }
- { type: 133, family: IPv6 }
- { type: 134, family: IPv6 }
- { type: 135, family: IPv6 }
- { type: 136, family: IPv6 }
- { type: 143, family: IPv6 }
#
# NOT here: ingress from `woodpecker-ci`. Pipeline step pods do call the
# server back on :8000 to stream logs, but the CNPG instance manager serves
# its status API on :8000 too — so granting that on this endpointSelector:{}
# policy would hand arbitrary repo-supplied build steps a route to wp-pg's
# instance manager. It lands in stage 2 as its own policy selecting the
# server pod, the way woodpecker-allow-rotators already scopes its :8000
# rule in apps/woodpecker/networkpolicy.yaml.
---
# The CNPG operator polls each instance's status endpoint over HTTPS
# (https://<pod-ip>:8000/pg/status) to populate Cluster status. Without it the
# Cluster reports "Instance Status Extraction Error: HTTP communication issue"
# while Postgres serves normally, and the operator loses failover and
# switchover decisions — issue #433, where the operator's dial timed out
# because the apps baseline never allowed the database namespace.
#
# Scoped to the wp-pg endpoints and that one port: the operator has no reason to
# reach the woodpecker server or agent, and the barman and mariadb operators
# sharing its namespace have no reason to reach anything here at all.
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: wp-pg-allow-cnpg-operator
namespace: woodpecker
spec:
endpointSelector:
matchLabels:
cnpg.io/cluster: wp-pg
ingress:
- fromEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": database
app.kubernetes.io/name: cloudnative-pg
toPorts:
- ports:
- port: "8000"
protocol: TCP