feat(woodpecker): provision wp-pg CNPG cluster to migrate off sqlite (#295, #324) #355

2026-07-22T13:28:59Z

claude-bot commented

2026-07-22 13:28:59 +00:00

Stage 1 of 3 of migrating woodpecker-server off its sqlite-on-ceph-block PVC onto CNPG postgres.

Why

data-woodpecker-server-0's Velero CSI-snapshot backup fails permanently — ext4 structure needs cleaning, because the source filesystem is error-flagged and journal replay (PR #255) can't clear it; only offline e2fsck could, and it'd likely recur. Rather than repeatedly fsck a fragile sqlite volume, move woodpecker's DB onto CNPG postgres like authentik/immich/monitoring/paperless already do. That gives WAL-based barman backups and removes the PVC entirely — resolving #295 and #324 by deletion. (This also answers "why does woodpecker even have a PVC" — it's the server's sqlite control-plane DB; the agents are already stateless.)

This PR — empty backend only, zero woodpecker impact

kubernetes/seed/woodpecker/: wp-pg Cluster (1 instance, initdb db woodpecker), ObjectStore → Hetzner S3, ScheduledBackup, hetzner-s3-creds ExternalSecret.
seed-woodpecker Flux Kustomization (dependsOn infrastructure + cnpg-barman, CNPG-health-gated).
MIGRATION.md: the one-time schema-first / data-only pgloader runbook (excludes the migrations bookkeeping table, casts int→bool, resets sequences) + staged cutover.

Merging this just brings up an empty wp-pg. Nothing touches woodpecker yet.

Remaining stages (separate PRs, per the runbook)

Data migration (offline runbook): quiesce woodpecker → extract sqlite → woodpecker builds the pg schema → pgloader data-only → reset sequences → verify parity.
PR 2 — flip server.yaml WOODPECKER_DATABASE_DRIVER/DATASOURCE to postgres (cutover). Woodpecker stays on v3.13.0.
PR 3 (after functional verify) — drop volumeClaimTemplates + mount, StatefulSet→Deployment, delete the old PVC.

Rollback is trivial until PR 3 — the sqlite PVC is untouched.

Note

The S3 creds ExternalSecret reuses the shared seed/paperless/hetzner-creds OpenBao path (identical Hetzner creds) to avoid seeding a new key (bao isn't writable from here). Inline follow-up to give it its own path.

🤖 Generated with Claude Code

**Stage 1 of 3** of migrating `woodpecker-server` off its sqlite-on-ceph-block PVC onto CNPG postgres. ## Why `data-woodpecker-server-0`'s Velero CSI-snapshot backup fails permanently — ext4 `structure needs cleaning`, because the **source filesystem is error-flagged** and journal replay (PR #255) can't clear it; only offline `e2fsck` could, and it'd likely recur. Rather than repeatedly fsck a fragile sqlite volume, move woodpecker's DB onto **CNPG postgres** like `authentik`/`immich`/`monitoring`/`paperless` already do. That gives WAL-based barman backups **and removes the PVC entirely — resolving #295 and #324 by deletion.** (This also answers "why does woodpecker even have a PVC" — it's the server's sqlite control-plane DB; the agents are already stateless.) ## This PR — empty backend only, zero woodpecker impact - `kubernetes/seed/woodpecker/`: `wp-pg` Cluster (1 instance, `initdb` db `woodpecker`), `ObjectStore` → Hetzner S3, `ScheduledBackup`, `hetzner-s3-creds` ExternalSecret. - `seed-woodpecker` Flux Kustomization (`dependsOn` infrastructure + cnpg-barman, CNPG-health-gated). - `MIGRATION.md`: the one-time **schema-first / data-only pgloader** runbook (excludes the `migrations` bookkeeping table, casts int→bool, resets sequences) + staged cutover. Merging this just brings up an empty `wp-pg`. Nothing touches woodpecker yet. ## Remaining stages (separate PRs, per the runbook) - **Data migration** (offline runbook): quiesce woodpecker → extract sqlite → woodpecker builds the pg schema → pgloader data-only → reset sequences → verify parity. - **PR 2** — flip `server.yaml` `WOODPECKER_DATABASE_DRIVER/DATASOURCE` to postgres (cutover). Woodpecker stays on **v3.13.0**. - **PR 3** (after functional verify) — drop `volumeClaimTemplates` + mount, StatefulSet→Deployment, delete the old PVC. Rollback is trivial until PR 3 — the sqlite PVC is untouched. ## Note The S3 creds ExternalSecret reuses the shared `seed/paperless/hetzner-creds` OpenBao path (identical Hetzner creds) to avoid seeding a new key (bao isn't writable from here). Inline follow-up to give it its own path. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit 2026-07-22 13:28:59 +00:00

feat(woodpecker): provision wp-pg CNPG cluster to migrate off sqlite (#295 , #324 ) 5cb90432e7

woodpecker-server runs sqlite on a ceph-block PVC (data-woodpecker-server-0)
whose Velero CSI-snapshot backup fails permanently (ext4 "structure needs
cleaning" — the source fs is error-flagged and journal replay can't clear it).
Rather than repeatedly fsck a fragile sqlite, move woodpecker's DB onto CNPG
postgres like authentik/immich/monitoring/paperless already do — giving proper
WAL-based barman backups and letting the PVC be removed entirely, which
resolves #295/#324 by deletion.

This PR (stage 1 of 3) provisions only the empty backend — no woodpecker
impact:
- kubernetes/seed/woodpecker/: wp-pg Cluster (1 instance, initdb woodpecker),
  ObjectStore -> Hetzner S3 (s3://renner-uno-k8s-cnpg/wp-pg), ScheduledBackup,
  and a hetzner-s3-creds ExternalSecret (reuses the shared paperless S3 path to
  avoid seeding a new OpenBao key; follow-up noted inline to give it its own).
- seed-woodpecker Flux Kustomization (dependsOn infrastructure + cnpg-barman,
  CNPG health-gated), auto-synced from clusters/homelab/.
- MIGRATION.md: the one-time schema-first / data-only pgloader runbook and the
  staged cutover (PR 2 = server.yaml -> postgres; PR 3 = drop PVC, STS->Deploy).

sqlite PVC is untouched; rollback is reverting env vars until PR 3.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

k.renner merged commit 61b1f7b64d into master

2026-07-22 13:30:07 +00:00

k.renner referenced this issue from a commit

2026-07-22 13:30:08 +00:00

Merge pull request 'feat(woodpecker): provision wp-pg CNPG cluster to migrate off sqlite (#295, #324)' (#355) from claude-bot/woodpecker-cnpg into master

claude-bot referenced this pull request

2026-07-22 13:57:30 +00:00

feat(woodpecker): cut server over to wp-pg postgres (#295, #324) #356

claude-bot referenced this pull request

2026-07-22 14:10:44 +00:00

feat(woodpecker): drop the sqlite PVC, StatefulSet->Deployment (#295, #324) #357

Sign in to join this conversation.