feat(woodpecker): automated WOODPECKER_AGENT_SECRET rotation (OpenBao + ESO + Reloader) #297

2026-07-18T12:46:47Z

claude-bot commented

2026-07-18 12:46:47 +00:00

What

Automatic, cluster-internal rotation of the shared woodpecker server↔agent gRPC secret (WOODPECKER_AGENT_SECRET). Proof-of-concept scope: woodpecker only.

How it works

CronJob woodpecker-secret-rotate (quarterly + on-demand) logs into OpenBao via a new k8s-auth role rotator, generates a random value, and bao kv patches homelab/apps/woodpecker — leaving the two Gitea OAuth fields untouched. OpenBao-only, no k8s RBAC (same pattern as the raft-snapshot job).
ESO re-syncs the value into the woodpecker-secret Secret within its refreshInterval (lowered 1h→10m).
Stakater Reloader (new addon) rolling-restarts woodpecker-server + woodpecker-agent (via reload annotations) so both pick up the new secret.

⚠️ One-time manual bootstrap required before this works

OpenBao roles/policies are not GitOps-managed in this repo (same as eso-read.hcl). Apply the rotator policy + role once (commands documented in kubernetes/infrastructure/openbao/policies/rotator.hcl):

kubectl -n openbao exec -it openbao-0 -- sh
bao policy write rotator kubernetes/infrastructure/openbao/policies/rotator.hcl
bao write auth/kubernetes/role/rotator bound_service_account_names=woodpecker-secret-rotator bound_service_account_namespaces=apps policies=rotator ttl=15m

Until then the CronJob fails at the OpenBao login step (deliberate signal).

Test / on-demand rotation

kubectl -n apps create job --from=cronjob/woodpecker-secret-rotate woodpecker-secret-rotate-now
kubectl -n apps annotate externalsecret woodpecker-secret force-sync=$(date +%s) --overwrite

Notes

Least-privilege: rotator policy grants patch on the single woodpecker path only (honors #213 blast-radius lesson; aligns with #202 per-app segregation).
Reloader is new cluster-wide but only acts on annotated workloads.
Two-sided restart: server+agent restart within seconds, not atomically — an in-flight CI job may briefly fail the handshake and retry (quarterly schedule runs outside CI hours).
Verified: kubectl kustomize builds clean for both the app and infrastructure trees; yamllint clean.
Reloader chart version pin is open-lower-bound (repo convention) — confirm resolved version on first sync.

## What Automatic, cluster-internal rotation of the shared woodpecker server↔agent gRPC secret (`WOODPECKER_AGENT_SECRET`). Proof-of-concept scope: woodpecker only. ## How it works 1. **CronJob** `woodpecker-secret-rotate` (quarterly + on-demand) logs into OpenBao via a new k8s-auth role `rotator`, generates a random value, and `bao kv patch`es `homelab/apps/woodpecker` — leaving the two Gitea OAuth fields untouched. OpenBao-only, no k8s RBAC (same pattern as the raft-snapshot job). 2. **ESO** re-syncs the value into the `woodpecker-secret` Secret within its refreshInterval (lowered 1h→10m). 3. **Stakater Reloader** (new addon) rolling-restarts `woodpecker-server` + `woodpecker-agent` (via reload annotations) so both pick up the new secret. ## ⚠️ One-time manual bootstrap required before this works OpenBao roles/policies are not GitOps-managed in this repo (same as `eso-read.hcl`). Apply the `rotator` policy + role once (commands documented in `kubernetes/infrastructure/openbao/policies/rotator.hcl`): ``` kubectl -n openbao exec -it openbao-0 -- sh bao policy write rotator kubernetes/infrastructure/openbao/policies/rotator.hcl bao write auth/kubernetes/role/rotator bound_service_account_names=woodpecker-secret-rotator bound_service_account_namespaces=apps policies=rotator ttl=15m ``` Until then the CronJob fails at the OpenBao login step (deliberate signal). ## Test / on-demand rotation ``` kubectl -n apps create job --from=cronjob/woodpecker-secret-rotate woodpecker-secret-rotate-now kubectl -n apps annotate externalsecret woodpecker-secret force-sync=$(date +%s) --overwrite ``` ## Notes - Least-privilege: `rotator` policy grants `patch` on the single woodpecker path only (honors #213 blast-radius lesson; aligns with #202 per-app segregation). - Reloader is new cluster-wide but only acts on annotated workloads. - Two-sided restart: server+agent restart within seconds, not atomically — an in-flight CI job may briefly fail the handshake and retry (quarterly schedule runs outside CI hours). - Verified: `kubectl kustomize` builds clean for both the app and infrastructure trees; yamllint clean. - Reloader chart version pin is open-lower-bound (repo convention) — confirm resolved version on first sync.

claude-bot added 1 commit 2026-07-18 12:46:47 +00:00

feat(woodpecker): automated WOODPECKER_AGENT_SECRET rotation (OpenBao + ESO + Reloader) 646bbf41e4

End-to-end cluster-internal rotation of the shared server<->agent gRPC secret, no per-rotation human step:
- rotate-agent-secret.yaml: quarterly CronJob (+ on-demand) logs into OpenBao via k8s-auth role 'rotator', generates a random value, bao kv patch's homelab/apps/woodpecker (Gitea OAuth fields untouched).
- infrastructure/reloader: Stakater Reloader restarts annotated workloads on Secret change.
- server.yaml/agent.yaml: reloader annotation for woodpecker-secret.
- externalsecret.yaml: refreshInterval 1h -> 10m so a rotation propagates within ~10m.
- openbao/policies/rotator.hcl: least-privilege policy (patch on the one path only); NOT Flux-applied, one-time manual bootstrap documented in-file (eso-read.hcl convention).

PoC scope: woodpecker only. Requires a one-time OpenBao bootstrap (rotator policy + role) before the CronJob succeeds.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ

claude-bot added 1 commit 2026-07-18 13:12:37 +00:00

fix(woodpecker-rotation): address adversarial review (pin Reloader, doc gaps) 142ff676bb

- H1: pin Reloader chart >=1.0.0 <2.0.0 (was open upper bound; a cluster-wide restart-authority controller must not float across majors — repo pinning discipline).
- M2: correct the restart-downtime comment (server StatefulSet is unavailable for tens of seconds, not 'within seconds').
- M1: note the on-demand/emergency rotation runs during incidents (the 'outside CI hours' safety doesn't apply there).
- H2: document the no-convergence-check limitation + follow-up (add rollout/health check or alert before scaling the pattern).
- L1: correct rotator.hcl rationale — native 'bao kv patch' is a server-side merge, needs no client read-modify; 'read' is surplus.
- L2: warn against token_no_default_policy=true (breaks the kv-v2 preflight).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ

claude-bot referenced this issue from a commit

2026-07-18 18:32:57 +00:00

feat(monitoring): woodpecker secret-rotation convergence alerts

claude-bot referenced this pull request

2026-07-18 18:33:18 +00:00

feat(monitoring): woodpecker secret-rotation convergence alerts #298

k.renner merged commit 92146ffb5c into master

2026-07-18 18:38:19 +00:00

k.renner referenced this issue from a commit

2026-07-18 18:38:20 +00:00

Merge pull request 'feat(woodpecker): automated WOODPECKER_AGENT_SECRET rotation (OpenBao + ESO + Reloader)' (#297) from feat/woodpecker-secret-auto-rotation into master

claude-bot referenced this issue from a commit