feat(woodpecker): stage 3 — retire the apps cluster, promote the survivor #508

Merged

k.renner merged 1 commits from migrate-woodpecker-stage3 into master 2026-08-18 21:38:53 +00:00

Conversation 1 Commits 1 Files Changed 10

+160 -248

claude-bot commented 2026-08-18 21:33:58 +00:00

Member

Copy Link

Copy Source

Closes #433. Final stage of moving Woodpecker out of the shared apps namespace.

Stacked on #507 — that PR is still open, so this branch contains its two commits as well. Merge #507 first, or merge this and #507 becomes a no-op.

What this does

Deletes the four apps-namespace manifests, promotes the migration-* set onto their names, and gives the surviving Cluster the prune: disabled data guard it was deliberately denied while a rollback still existed.

The kustomization now renders nothing into apps:

woodpecker/{Cluster,ObjectStore x2,ScheduledBackup,ExternalSecret x2,Deployment x2,Service,Ingress,PDB,SA,CNP x2}
woodpecker-ci/{Role,RoleBinding,ResourceQuota,LimitRange}

Merging this deletes nothing

The old resources carry prune: disabled, so dropping them from git only stops Flux managing them — they keep running and keep their PVC. The teardown is a separate, deliberate act, and the runbook carries the commands plus the precondition.

That precondition is already satisfied:

$ kubectl -n woodpecker get backup.postgresql.cnpg.io
wp-pg-daily-20260818205831   wp-pg   plugin   completed

$ kubectl -n woodpecker get cluster wp-pg -o jsonpath='{.status.conditions}'
Initialized=True ConsistentSystemID=True Ready=True
ContinuousArchiving=True LastBackupSucceeded=True

The new cluster has taken its own base backup to the wp-pg-woodpecker prefix, so recovery no longer depends on the retired cluster's data. LastBackupSucceeded=True, not merely BackupStarted.

Two things kept rather than cleaned up

bootstrap.recovery and its externalCluster stay on the Cluster. CNPG honours bootstrap only when PGDATA is absent, so it is inert on a live cluster — and keeping it means a cold rebuild restores from backup instead of coming up empty. That is precisely the gap RECOVERY.md exists to close for the other CNPG clusters, and wp-pg now gets it for free.

wp-pg-store-source stays, read-only, pointing at the old prefix. Nothing writes to it, and it is a second independent restore point predating the move. It is also the actual safety net for the teardown — the old running cluster was never the safety net, the S3 prefix is, and that survives the deletion until its own retention expires.

Also

The hetzner-creds-externalsecret comment now records the openbao ClusterSecretStore allowlist requirement that cost stage 1 a failed restore (#505), so the next namespace move does not rediscover it the hard way.

#397 remains open and is unchanged: the credentials still resolve from seed/paperless/hetzner-creds, because creating seed/woodpecker/hetzner-creds needs OpenBao write access that isn't available. The comment now says so in place.

What the teardown will destroy

The old cluster's PVCs — including the four pipeline records (9491, 9492, 9494, 9496) that only ever existed there, which has been accepted. It does not touch s3://renner-uno-k8s-cnpg/wp-pg.

Closes #433. Final stage of moving Woodpecker out of the shared `apps` namespace. **Stacked on #507** — that PR is still open, so this branch contains its two commits as well. Merge #507 first, or merge this and #507 becomes a no-op. ## What this does Deletes the four `apps`-namespace manifests, promotes the `migration-*` set onto their names, and gives the surviving Cluster the `prune: disabled` data guard it was deliberately denied while a rollback still existed. The kustomization now renders **nothing** into `apps`: ``` woodpecker/{Cluster,ObjectStore x2,ScheduledBackup,ExternalSecret x2,Deployment x2,Service,Ingress,PDB,SA,CNP x2} woodpecker-ci/{Role,RoleBinding,ResourceQuota,LimitRange} ``` ## Merging this deletes nothing The old resources carry `prune: disabled`, so dropping them from git only stops Flux managing them — they keep running and keep their PVC. **The teardown is a separate, deliberate act**, and the runbook carries the commands plus the precondition. That precondition is already satisfied: ``` $ kubectl -n woodpecker get backup.postgresql.cnpg.io wp-pg-daily-20260818205831 wp-pg plugin completed $ kubectl -n woodpecker get cluster wp-pg -o jsonpath='{.status.conditions}' Initialized=True ConsistentSystemID=True Ready=True ContinuousArchiving=True LastBackupSucceeded=True ``` The new cluster has taken its own base backup to the `wp-pg-woodpecker` prefix, so recovery no longer depends on the retired cluster's data. `LastBackupSucceeded=True`, not merely `BackupStarted`. ## Two things kept rather than cleaned up **`bootstrap.recovery` and its `externalCluster` stay on the Cluster.** CNPG honours `bootstrap` only when PGDATA is absent, so it is inert on a live cluster — and keeping it means a cold rebuild restores from backup instead of coming up empty. That is precisely the gap `RECOVERY.md` exists to close for the other CNPG clusters, and wp-pg now gets it for free. **`wp-pg-store-source` stays**, read-only, pointing at the old prefix. Nothing writes to it, and it is a second independent restore point predating the move. It is also the actual safety net for the teardown — the old *running cluster* was never the safety net, the S3 prefix is, and that survives the deletion until its own retention expires. ## Also The `hetzner-creds-externalsecret` comment now records the openbao `ClusterSecretStore` allowlist requirement that cost stage 1 a failed restore (#505), so the next namespace move does not rediscover it the hard way. `#397` remains open and is unchanged: the credentials still resolve from `seed/paperless/hetzner-creds`, because creating `seed/woodpecker/hetzner-creds` needs OpenBao write access that isn't available. The comment now says so in place. ## What the teardown will destroy The old cluster's PVCs — including the four pipeline records (`9491`, `9492`, `9494`, `9496`) that only ever existed there, which has been accepted. It does **not** touch `s3://renner-uno-k8s-cnpg/wp-pg`.

claude-bot added 3 commits 2026-08-18 21:33:59 +00:00

fix(woodpecker): scope traefik ingress to the server, not the namespace ... d697f0f2c2

Follow-up to #506, which merged before this landed. Addresses the review's
stale-comment suggestion, and a real over-grant found while checking it.

The comment above woodpecker-allow-rotators still explained the rule in terms
of the `apps` baseline and allow-ingress.yaml. Chasing that down showed
allow-ingress.yaml is hard-scoped to `namespace: apps` and matches traefik by
label — so it does not apply in the new namespace, and the traefik admit I put
in the woodpecker baseline instead was broader than the thing it replaced:
endpointSelector:{} plus a whole-namespace source, where the `ingress`
namespace also holds netbird-ingress. That let those pods reach wp-pg and the
agent, not just the server.

Replaced with woodpecker-allow-traefik, scoped to traefik and to the server
pod, matching how the rotator and CI-step rules are already written.

It lives in network-policies/woodpecker-ns.yaml rather than beside the other
workload policies, deliberately: the baseline admit is withdrawn in the same
file, so both halves ship in the infrastructure-crds Kustomization and swap in
one reconcile. Split across Kustomizations they would not — infrastructure-crds
reconciles before the `woodpecker` KS, so the admit would be withdrawn minutes
before its replacement arrived and ci.renner.uno would 503 in the gap.

Confirmed safe against certificate renewal: letsencrypt-cluster solves DNS-01
through Cloudflare, so there is no in-cluster ACME solver pod needing ingress.
With an HTTP-01 issuer this scoping would have broken renewals silently, 90
days later.

Runbook: records that the stage-2 outage outlasts pods becoming Ready, because
certificates are namespace-scoped and cert-manager must reissue woodpecker-tls.
Observed during the live cutover as a TLS SNI failure with no HTTP status at
all, which reads like a routing fault and is not one.

Co-Authored-By: Claude Bot <claude-bot@renner.uno>

docs(woodpecker): record the cutover's data gap instead of claiming there is none ... c4be814524

The stage-1 verification section said row counts "will not match exactly if
builds ran after the backup the restore used — that is expected, and stage 2
quiesces the server before the final cutover".

Stage 2 does not quiesce. Flux prunes the apps workloads and creates the
woodpecker ones; nothing stops writes to the old database in between. The
sentence described an intent that was never implemented, which is worse than
saying nothing.

Measured on the live cutover rather than estimated: the restore captured
through pipeline 9488; 9491, 9492, 9494 and 9496 landed in apps/wp-pg before
the swap; the new cluster resumed at 9521. Four pipeline records and their log
entries exist only in the old cluster. No ID collision and no functional
breakage — cluster.yaml itself calls this database small and non-critical with
largely reconstructable state — but it is real history loss, and it should be a
decision rather than something discovered later from a row count.

Documents how to avoid it next time (suspend the KS, scale the old server and
agent to zero, re-run the restore) and notes the rows stay recoverable while
the old cluster keeps its prune guard, which is until stage 3.

Co-Authored-By: Claude Bot <claude-bot@renner.uno>

feat(woodpecker): stage 3 — retire the apps cluster, promote the survivor ... cca475ffe6

Closes #433.

Deletes the four apps-namespace manifests, promotes the migration-* set onto
their names, and gives the surviving Cluster the prune: disabled data guard it
was deliberately denied while a rollback still existed. After this the
kustomization renders nothing into `apps` at all.

Merging this deletes nothing. The old resources carry prune: disabled, so
dropping them from git only stops Flux managing them — they keep running and
keep their PVC. The teardown is a separate, deliberate act, and the runbook
carries the commands plus the precondition to check first.

That precondition is already satisfied: the new cluster took its own base
backup to the wp-pg-woodpecker prefix (wp-pg-daily-20260818205831, completed)
and reports Initialized, ConsistentSystemID, Ready, ContinuousArchiving and
LastBackupSucceeded all True. Recovery no longer depends on the retired
cluster's data.

Two things kept rather than cleaned up, both deliberate:

- bootstrap.recovery and its externalCluster stay on the Cluster. CNPG honours
  bootstrap only when PGDATA is absent, so it is inert on a live cluster — and
  keeping it means a cold rebuild restores from backup instead of coming up
  empty, which is the gap RECOVERY.md exists to close for the other clusters.
- wp-pg-store-source stays, pointing read-only at the old prefix. Nothing
  writes to it, and it is a second independent restore point predating the
  move — worth having until the new prefix builds real backup history. It is
  also the actual safety net for the teardown, rather than the running cluster.

The hetzner-creds comment now records the openbao ClusterSecretStore allowlist
requirement that cost stage 1 a failed restore (#505), so the next namespace
move does not rediscover it.

Co-Authored-By: Claude Bot <claude-bot@renner.uno>

review-bot reviewed 2026-08-18 21:37:37 +00:00

review-bot left a comment

Member

Copy Link

Copy Source

🤖 Automated review — model CLI default (complexity: complex), commit cca475ffe6aa87ada082cf4df7f3c280208be09d.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

---

Summary

Stage 3 of the Woodpecker namespace migration (#433): the four apps-namespace CNPG manifests are deleted, the migration-* files are promoted onto their canonical names (cluster.yaml, objectstore.yaml, scheduled-backup.yaml, hetzner-creds-externalsecret.yaml), the surviving Cluster gains the prune: disabled guard, and a MIGRATION-namespace.md runbook section documents the teardown. It also folds in an unrelated but genuine improvement: narrowing the woodpecker namespace's baseline ingress policy so Traefik access is scoped to the server pod instead of the whole namespace. I traced every renamed/promoted object (Cluster, both ObjectStores, ScheduledBackup, ExternalSecret) against the deleted migration-*.yaml files and the still-existing apps-namespace originals, and the object identities, prune: disabled guards, and cross-references (barmanObjectName, source, serverName) are all internally consistent. No blocking issues found.

🟡 Suggestions

• kubernetes/infrastructure/network-policies/woodpecker-ns.yaml:35-92 and kubernetes/apps/woodpecker/networkpolicy.yaml — the Traefik-ingress narrowing (removing the blanket ingress-namespace admit from the endpointSelector: {} baseline and replacing it with a scoped woodpecker-allow-traefik policy) is a real, worthwhile security tightening, but it isn't mentioned anywhere in the PR title/body, which frames this purely as "retire the apps cluster, promote the survivor." Worth calling out explicitly in the PR description so a reviewer doesn't have to discover a namespace-wide ingress policy change by reading the diff of what looks like an unrelated CNPG migration.
• kubernetes/apps/woodpecker/cluster.yaml:19-23, scheduled-backup.yaml, objectstore.yaml, hetzner-creds-externalsecret.yaml — only the Cluster keeps prune: disabled after promotion; the ObjectStores, ScheduledBackup, and ExternalSecret do not (matching the rationale already documented in the now-deleted migration-*.yaml files: only the object holding the data PVC needs the guard). That's a reasonable call, but it does mean a future accidental removal of objectstore.yaml/scheduled-backup.yaml/hetzner-creds-externalsecret.yaml from kustomization.yaml would silently prune those CRs (backups would then fail against a live cluster, or WAL archiving would break) with no guard to catch it. Not something to fix here, just worth being aware of since it's a behavior change from how the apps originals were guarded (all four had prune: disabled there).

Tests

This is a GitOps manifest-only change; there's no unit-test layer for it, and the repo's convention (.woodpecker/ci-manifests-test.yml) is to validate via kustomize build / manifest checks in CI rather than hand-written tests, which is appropriate here. I manually verified (by reading the checkout, not just the diff) that: the promoted cluster.yaml/objectstore.yaml/scheduled-backup.yaml/hetzner-creds-externalsecret.yaml exactly reproduce the deleted migration-*.yaml content (object names, namespaces, barmanObjectName/source/serverName cross-references all line up); no file anywhere still references the deleted migration-*.yaml names; the openbao ClusterSecretStore's namespace allowlist already includes woodpecker (kubernetes/infrastructure/post-crds/openbao-clustersecretstore.yaml:33), so the comment added to hetzner-creds-externalsecret.yaml about the allowlist requirement is documenting an already-satisfied precondition, not a missing one; and the new woodpecker-allow-traefik policy's label selectors (app.kubernetes.io/name: woodpecker / component: server, and app.kubernetes.io/name: traefik in the ingress namespace) match the actual server.yaml pod labels and the pattern used in allow-ingress.yaml. I could not execute kustomize build or kubeconform in this environment (no kustomize/kubectl on PATH here), so final schema/render validation still rests on CI.

Recommendation

Looks reasonable — no blocking issues; comments above are optional.

🤖 **Automated review** — model `CLI default` (complexity: complex), commit `cca475ffe6aa87ada082cf4df7f3c280208be09d`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary Stage 3 of the Woodpecker namespace migration (#433): the four `apps`-namespace CNPG manifests are deleted, the `migration-*` files are promoted onto their canonical names (`cluster.yaml`, `objectstore.yaml`, `scheduled-backup.yaml`, `hetzner-creds-externalsecret.yaml`), the surviving Cluster gains the `prune: disabled` guard, and a `MIGRATION-namespace.md` runbook section documents the teardown. It also folds in an unrelated but genuine improvement: narrowing the `woodpecker` namespace's baseline ingress policy so Traefik access is scoped to the server pod instead of the whole namespace. I traced every renamed/promoted object (Cluster, both ObjectStores, ScheduledBackup, ExternalSecret) against the deleted `migration-*.yaml` files and the still-existing `apps`-namespace originals, and the object identities, `prune: disabled` guards, and cross-references (`barmanObjectName`, `source`, `serverName`) are all internally consistent. No blocking issues found. ## 🟡 Suggestions - `kubernetes/infrastructure/network-policies/woodpecker-ns.yaml:35-92` and `kubernetes/apps/woodpecker/networkpolicy.yaml` — the Traefik-ingress narrowing (removing the blanket `ingress`-namespace admit from the `endpointSelector: {}` baseline and replacing it with a scoped `woodpecker-allow-traefik` policy) is a real, worthwhile security tightening, but it isn't mentioned anywhere in the PR title/body, which frames this purely as "retire the apps cluster, promote the survivor." Worth calling out explicitly in the PR description so a reviewer doesn't have to discover a namespace-wide ingress policy change by reading the diff of what looks like an unrelated CNPG migration. - `kubernetes/apps/woodpecker/cluster.yaml:19-23`, `scheduled-backup.yaml`, `objectstore.yaml`, `hetzner-creds-externalsecret.yaml` — only the Cluster keeps `prune: disabled` after promotion; the ObjectStores, ScheduledBackup, and ExternalSecret do not (matching the rationale already documented in the now-deleted `migration-*.yaml` files: only the object holding the data PVC needs the guard). That's a reasonable call, but it does mean a future accidental removal of `objectstore.yaml`/`scheduled-backup.yaml`/`hetzner-creds-externalsecret.yaml` from `kustomization.yaml` would silently prune those CRs (backups would then fail against a live cluster, or WAL archiving would break) with no guard to catch it. Not something to fix here, just worth being aware of since it's a behavior change from how the `apps` originals were guarded (all four had `prune: disabled` there). ## Tests This is a GitOps manifest-only change; there's no unit-test layer for it, and the repo's convention (`.woodpecker/ci-manifests-test.yml`) is to validate via kustomize build / manifest checks in CI rather than hand-written tests, which is appropriate here. I manually verified (by reading the checkout, not just the diff) that: the promoted `cluster.yaml`/`objectstore.yaml`/`scheduled-backup.yaml`/`hetzner-creds-externalsecret.yaml` exactly reproduce the deleted `migration-*.yaml` content (object names, namespaces, `barmanObjectName`/`source`/`serverName` cross-references all line up); no file anywhere still references the deleted `migration-*.yaml` names; the `openbao` ClusterSecretStore's namespace allowlist already includes `woodpecker` (`kubernetes/infrastructure/post-crds/openbao-clustersecretstore.yaml:33`), so the comment added to `hetzner-creds-externalsecret.yaml` about the allowlist requirement is documenting an already-satisfied precondition, not a missing one; and the new `woodpecker-allow-traefik` policy's label selectors (`app.kubernetes.io/name: woodpecker` / `component: server`, and `app.kubernetes.io/name: traefik` in the `ingress` namespace) match the actual `server.yaml` pod labels and the pattern used in `allow-ingress.yaml`. I could not execute `kustomize build` or `kubeconform` in this environment (no `kustomize`/`kubectl` on PATH here), so final schema/render validation still rests on CI. ## Recommendation Looks reasonable — no blocking issues; comments above are optional. <!-- pr-review:sha=cca475ffe6aa87ada082cf4df7f3c280208be09d -->

k.renner merged commit 2e93edfda9 into master 2026-08-18 21:38:53 +00:00

k.renner referenced this issue from a commit 2026-08-18 21:38:54 +00:00

Merge pull request 'feat(woodpecker): stage 3 — retire the apps cluster, promote the survivor' (#508) from migrate-woodpecker-stage3 into master

Sign in to join this conversation.

Reviewers

No Reviewers

review-bot

Labels

Clear labels

apollo

Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant

atlas

Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring

clio

Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites

daedalus

Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes

hestia

Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#508