fix(woodpecker): scope traefik ingress to the server, and record the cutover data gap #507

2026-08-18T21:29:37Z

claude-bot commented

2026-08-18 21:29:37 +00:00

Follow-up to #506, which merged while these were still being written. The cutover itself succeeded — ci.renner.uno is back at 200, both pods Ready, and the server is writing to the new database.

1. Traefik ingress was over-granted

The review flagged a stale comment describing the old apps baseline. Chasing it down turned up something functional behind the prose: allow-ingress.yaml is hard-scoped to namespace: apps and matches traefik by label, so it does not apply in the new namespace — and the admit I put in the woodpecker baseline instead was broader than the thing it replaced.

The baseline uses endpointSelector: {} and admitted the whole ingress namespace, which also holds netbird-ingress. Those pods could reach wp-pg and the agent, not just the server.

Replaced with woodpecker-allow-traefik, scoped to traefik and to the server pod — matching how the rotator and CI-step rules are already written. Same lesson as the first review on #504, found one layer down.

It lives in network-policies/woodpecker-ns.yaml, not beside the other workload policies, deliberately. The baseline admit is withdrawn in the same file, so both halves ship in infrastructure-crds and swap in one reconcile. Split across Kustomizations they would not: infrastructure-crds reconciles before the woodpecker KS, so the admit would be withdrawn minutes before its replacement arrived and ci.renner.uno would 503 in the gap.

Checked before scoping: letsencrypt-cluster solves DNS-01 via Cloudflare, so there is no in-cluster ACME solver pod needing ingress. With an HTTP-01 issuer this scoping would have broken certificate renewal silently, 90 days later.

2. The runbook claimed a quiesce that never happened

It said row counts "will not match exactly if builds ran after the backup — that is expected, and stage 2 quiesces the server before the final cutover."

Stage 2 does not quiesce. Flux prunes the apps workloads and creates the woodpecker ones; nothing stops writes to the old database in between. That sentence described an intent that was never implemented.

Measured on the live cutover, not estimated:


restore captured through	pipeline `9488`
landed in `apps/wp-pg` before the swap	`9491`, `9492`, `9494`, `9496`
new cluster resumed at	`9521`

Four pipeline records and their log entries exist only in the old cluster. No ID collision, no functional breakage — cluster.yaml itself calls this database "small and non-critical (state is largely reconstructable)" — but it is real history loss, and it should be a decision rather than something discovered later from a row count.

The runbook now documents how to avoid it (suspend the KS, scale the old server and agent to zero, re-run the restore) and notes the rows stay recoverable while the old cluster keeps its prune guard — which is until stage 3.

Cutover verification (already done, live)

check	result
`apps` woodpecker pods	0
`woodpecker` pods	2, both Ready
`ci.renner.uno`	200
`Certificate/woodpecker-tls`	`READY=True`
Gitea webhooks	arriving at the new server
new DB receiving writes	6125 → 6126, pipeline `9521`

The TLS gap during cutover was expected but undocumented — certificates are namespace-scoped, so cert-manager had to reissue, and until it did, clients failed with an SNI error and no HTTP status at all, which reads like a routing fault. That is now written down too.

Follow-up to #506, which merged while these were still being written. The cutover itself succeeded — `ci.renner.uno` is back at 200, both pods Ready, and the server is writing to the new database. ## 1. Traefik ingress was over-granted The review flagged a stale comment describing the old `apps` baseline. Chasing it down turned up something functional behind the prose: `allow-ingress.yaml` is hard-scoped to `namespace: apps` **and matches traefik by label**, so it does not apply in the new namespace — and the admit I put in the `woodpecker` baseline instead was *broader* than the thing it replaced. The baseline uses `endpointSelector: {}` and admitted the whole `ingress` namespace, which also holds **netbird-ingress**. Those pods could reach wp-pg and the agent, not just the server. Replaced with `woodpecker-allow-traefik`, scoped to traefik and to the server pod — matching how the rotator and CI-step rules are already written. Same lesson as the first review on #504, found one layer down. **It lives in `network-policies/woodpecker-ns.yaml`, not beside the other workload policies, deliberately.** The baseline admit is withdrawn in the same file, so both halves ship in `infrastructure-crds` and swap in one reconcile. Split across Kustomizations they would not: `infrastructure-crds` reconciles *before* the `woodpecker` KS, so the admit would be withdrawn minutes before its replacement arrived and `ci.renner.uno` would 503 in the gap. **Checked before scoping:** `letsencrypt-cluster` solves **DNS-01** via Cloudflare, so there is no in-cluster ACME solver pod needing ingress. With an HTTP-01 issuer this scoping would have broken certificate renewal silently, 90 days later. ## 2. The runbook claimed a quiesce that never happened It said row counts "will not match exactly if builds ran after the backup — that is expected, and stage 2 quiesces the server before the final cutover." **Stage 2 does not quiesce.** Flux prunes the `apps` workloads and creates the `woodpecker` ones; nothing stops writes to the old database in between. That sentence described an intent that was never implemented. Measured on the live cutover, not estimated: | | | |---|---| | restore captured through | pipeline `9488` | | landed in `apps/wp-pg` before the swap | `9491`, `9492`, `9494`, `9496` | | new cluster resumed at | `9521` | **Four pipeline records and their log entries exist only in the old cluster.** No ID collision, no functional breakage — `cluster.yaml` itself calls this database "small and non-critical (state is largely reconstructable)" — but it is real history loss, and it should be a decision rather than something discovered later from a row count. The runbook now documents how to avoid it (suspend the KS, scale the old server and agent to zero, re-run the restore) and notes the rows stay recoverable while the old cluster keeps its prune guard — which is until stage 3. ## Cutover verification (already done, live) | check | result | |---|---| | `apps` woodpecker pods | 0 | | `woodpecker` pods | 2, both Ready | | `ci.renner.uno` | **200** | | `Certificate/woodpecker-tls` | `READY=True` | | Gitea webhooks | arriving at the new server | | new DB receiving writes | 6125 → 6126, pipeline `9521` | The TLS gap during cutover was expected but undocumented — certificates are namespace-scoped, so cert-manager had to reissue, and until it did, clients failed with an SNI error and *no HTTP status at all*, which reads like a routing fault. That is now written down too.

claude-bot added 2 commits 2026-08-18 21:29:38 +00:00

fix(woodpecker): scope traefik ingress to the server, not the namespace d697f0f2c2

Follow-up to #506, which merged before this landed. Addresses the review's
stale-comment suggestion, and a real over-grant found while checking it.

The comment above woodpecker-allow-rotators still explained the rule in terms
of the `apps` baseline and allow-ingress.yaml. Chasing that down showed
allow-ingress.yaml is hard-scoped to `namespace: apps` and matches traefik by
label — so it does not apply in the new namespace, and the traefik admit I put
in the woodpecker baseline instead was broader than the thing it replaced:
endpointSelector:{} plus a whole-namespace source, where the `ingress`
namespace also holds netbird-ingress. That let those pods reach wp-pg and the
agent, not just the server.

Replaced with woodpecker-allow-traefik, scoped to traefik and to the server
pod, matching how the rotator and CI-step rules are already written.

It lives in network-policies/woodpecker-ns.yaml rather than beside the other
workload policies, deliberately: the baseline admit is withdrawn in the same
file, so both halves ship in the infrastructure-crds Kustomization and swap in
one reconcile. Split across Kustomizations they would not — infrastructure-crds
reconciles before the `woodpecker` KS, so the admit would be withdrawn minutes
before its replacement arrived and ci.renner.uno would 503 in the gap.

Confirmed safe against certificate renewal: letsencrypt-cluster solves DNS-01
through Cloudflare, so there is no in-cluster ACME solver pod needing ingress.
With an HTTP-01 issuer this scoping would have broken renewals silently, 90
days later.

Runbook: records that the stage-2 outage outlasts pods becoming Ready, because
certificates are namespace-scoped and cert-manager must reissue woodpecker-tls.
Observed during the live cutover as a TLS SNI failure with no HTTP status at
all, which reads like a routing fault and is not one.

Co-Authored-By: Claude Bot <claude-bot@renner.uno>

docs(woodpecker): record the cutover's data gap instead of claiming there is none

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/pr-review Pipeline was successful

Details

c4be814524

The stage-1 verification section said row counts "will not match exactly if
builds ran after the backup the restore used — that is expected, and stage 2
quiesces the server before the final cutover".

Stage 2 does not quiesce. Flux prunes the apps workloads and creates the
woodpecker ones; nothing stops writes to the old database in between. The
sentence described an intent that was never implemented, which is worse than
saying nothing.

Measured on the live cutover rather than estimated: the restore captured
through pipeline 9488; 9491, 9492, 9494 and 9496 landed in apps/wp-pg before
the swap; the new cluster resumed at 9521. Four pipeline records and their log
entries exist only in the old cluster. No ID collision and no functional
breakage — cluster.yaml itself calls this database small and non-critical with
largely reconstructable state — but it is real history loss, and it should be a
decision rather than something discovered later from a row count.

Documents how to avoid it next time (suspend the KS, scale the old server and
agent to zero, re-run the restore) and notes the rows stay recoverable while
the old cluster keeps its prune guard, which is until stage 3.

Co-Authored-By: Claude Bot <claude-bot@renner.uno>

review-bot reviewed

2026-08-18 21:31:48 +00:00

review-bot left a comment

🤖 Automated review — model CLI default (complexity: complex), commit c4be814524fba0b7af4c4f02a1d9ee7178daa05e.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

Two-part follow-up to #506/#504: (1) replaces an over-broad namespace-wide Traefik ingress admit on the woodpecker baseline with a policy scoped to Traefik → the server pod only, withdrawing the old admit in the same file/Kustomization so the swap is atomic; (2) corrects MIGRATION-namespace.md to stop claiming stage 2 quiesces the server before cutover, and documents the measured data gap (4 pipeline records) plus a TLS-reissue outage detail. I checked every specific claim in the diff and PR description against the checkout (label selectors, Kustomization dependsOn/path wiring, the letsencrypt-cluster DNS-01 issuer, the prune: disabled guard, pod labels on server.yaml) and all of them hold up. No blocking issues.

🟡 Suggestions

kubernetes/infrastructure/network-policies/woodpecker-ns.yaml:167 — the quiesce snippet in the doc uses kubectl -n <old-ns> scale deploy/woodpecker-server deploy/woodpecker-agent --replicas=0, but by the time this runbook step would run in a future migration, "old" is context-dependent (here it'd be apps). Consider making the placeholder explicit (e.g. apps with a note "the namespace being migrated from") since every other command in this doc uses a concrete namespace rather than a placeholder — small inconsistency in an otherwise very concrete runbook.

Tests

No automated tests apply to this change — it's Cilium policy YAML and markdown runbook prose in a GitOps repo, consistent with how the rest of this repo handles infra/network-policy changes (verification is manual, via the cilium-dbg bpf policy get steps already documented). The PR's own description includes live verification evidence (200 on ci.renner.uno, Certificate Ready, DB writes advancing) which is the appropriate substitute for tests here. I independently verified the two structural claims that verification-by-observation can't fully cover on its own: that infrastructure-crds (which bundles ../network-policies/) is a hard dependency of the woodpecker Kustomization (confirmed in kubernetes/clusters/homelab/woodpecker.yaml and infrastructure.yaml), and that the new policy's endpointSelector labels (app.kubernetes.io/name: woodpecker, app.kubernetes.io/component: server) match the actual server pod labels in server.yaml.

Recommendation

Looks reasonable — no blocking issues; comment above is optional.

🤖 **Automated review** — model `CLI default` (complexity: complex), commit `c4be814524fba0b7af4c4f02a1d9ee7178daa05e`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary Two-part follow-up to #506/#504: (1) replaces an over-broad namespace-wide Traefik ingress admit on the `woodpecker` baseline with a policy scoped to Traefik → the server pod only, withdrawing the old admit in the same file/Kustomization so the swap is atomic; (2) corrects `MIGRATION-namespace.md` to stop claiming stage 2 quiesces the server before cutover, and documents the measured data gap (4 pipeline records) plus a TLS-reissue outage detail. I checked every specific claim in the diff and PR description against the checkout (label selectors, Kustomization `dependsOn`/`path` wiring, the `letsencrypt-cluster` DNS-01 issuer, the `prune: disabled` guard, pod labels on `server.yaml`) and all of them hold up. No blocking issues. ## 🟡 Suggestions - `kubernetes/infrastructure/network-policies/woodpecker-ns.yaml:167` — the quiesce snippet in the doc uses `kubectl -n <old-ns> scale deploy/woodpecker-server deploy/woodpecker-agent --replicas=0`, but by the time this runbook step would run in a *future* migration, "old" is context-dependent (here it'd be `apps`). Consider making the placeholder explicit (e.g. `apps` with a note "the namespace being migrated *from*") since every other command in this doc uses a concrete namespace rather than a placeholder — small inconsistency in an otherwise very concrete runbook. ## Tests No automated tests apply to this change — it's Cilium policy YAML and markdown runbook prose in a GitOps repo, consistent with how the rest of this repo handles infra/network-policy changes (verification is manual, via the `cilium-dbg bpf policy get` steps already documented). The PR's own description includes live verification evidence (200 on `ci.renner.uno`, Certificate Ready, DB writes advancing) which is the appropriate substitute for tests here. I independently verified the two structural claims that verification-by-observation can't fully cover on its own: that `infrastructure-crds` (which bundles `../network-policies/`) is a hard dependency of the `woodpecker` Kustomization (confirmed in `kubernetes/clusters/homelab/woodpecker.yaml` and `infrastructure.yaml`), and that the new policy's `endpointSelector` labels (`app.kubernetes.io/name: woodpecker`, `app.kubernetes.io/component: server`) match the actual server pod labels in `server.yaml`. ## Recommendation Looks reasonable — no blocking issues; comment above is optional.

claude-bot referenced this pull request

2026-08-18 21:33:58 +00:00

feat(woodpecker): stage 3 — retire the apps cluster, promote the survivor #508

k.renner merged commit 550f7c223e into master

2026-08-18 21:34:38 +00:00

k.renner referenced this issue from a commit

2026-08-18 21:34:39 +00:00

Merge pull request 'fix(woodpecker): scope traefik ingress to the server, and record the cutover data gap' (#507) from migrate-woodpecker-stage2 into master

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#507