Same fix proven on netbird (#307/#308): alpine+apk -> python:3.13-alpine stdlib so it runs under the non-root restricted PSA. Uses a dedicated no-verify SSL context for the self-signed Proxmox API only (OpenBao stays plain http). Still suspended + untested; the privsep=0 caveat + brief monitoring gap on DELETE+recreate remain documented in the header.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
Applies the same fix proven on netbird (#307/#308): switch alpine+apk to python:3.13-alpine stdlib so the job runs under the non-root restricted PSA. CF's roll is in-place (PUT /tokens/{id}/value) so no key-settings pitfall like netbird had. Still suspended + untested (needs the bootstrap admin-token + token-id).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
Testing #304 end-to-end revealed the minted setup key had the WRONG settings vs the ingress key: ephemeral=false (should be true — peers must be auto-reaped; these DaemonSet peers re-enroll fresh each restart) and allow_extra_dns_labels=false (should be true — the DNS capability). Remediated live and fixed the payload:
- add ephemeral: true, allow_extra_dns_labels: true to the setup-keys POST.
- NB_AUTO_GROUPS = d8umic2oc9cs73d4e1f0 (k8s-ingress-router).
- KEEP the 1y expiry (intentional: an auto-rotated key should expire so a broken rotation surfaces — do NOT match the original's never-expire).
- suspend: false (tested + enabled).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
Testing #304 revealed the runtime 'apk add curl jq' fails ('Unable to lock database: Permission denied') because the job runs non-root (runAsUser 65534) and the openbao ns enforces restricted PSA — apk needs root. Switch to python:3.13-alpine and do the HTTP/JSON via stdlib (urllib/json), no package install, non-root-safe. Same fix must be applied to the still-open cloudflare (#305) and proxmox (#306) Class-B jobs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
Rotates pve-exporter@pve!monitoring via the Proxmox API. Proxmox has no in-place secret roll, so DELETE+re-create (privsep=0), regenerate pve.yml with the new token_value, patch OpenBao. openbao-ns pattern; per-secret rotator-proxmox role reads a bootstrap privileged Proxmox token + patches pve-exporter-config. refreshInterval 1h->10m; Reloader restarts pve-exporter.
SHIPPED SUSPENDED + UNTESTED (riskiest Class-B): needs a bootstrap privileged Proxmox token at homelab/infrastructure/secret-rotators-bootstrap/proxmox, VERIFICATION that the token uses privsep=0 (else re-grant its ACL after re-create), and awareness of the brief monitoring gap during DELETE+create. Test before spec.suspend=false.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
Rolls the cert-manager DNS-01 Cloudflare token in place (PUT /user/tokens/{id}/value) then patches OpenBao. openbao-ns pattern; per-secret rotator-cloudflare role reads a bootstrap admin-token+token-id and patches the api-token path. refreshInterval 1h->10m; no Reloader (cert-manager reads fresh per challenge).
SHIPPED SUSPENDED + UNTESTED: needs the bootstrap admin-token (Edit User API Tokens perm) + the cert-manager token-id provisioned to homelab/infrastructure/secret-rotators-bootstrap/cloudflare, then a manual test (issue/renew a cert) before spec.suspend=false. Risk: a bad roll breaks TLS issuance.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
First Class-B rotation: calls the self-hosted NetBird mgmt API to mint a fresh reusable setup key, then patches OpenBao (ESO syncs, Reloader restarts the netbird-ingress DaemonSet to re-enroll). openbao-ns pattern; per-secret rotator-netbird role/policy that reads a bootstrap NetBird PAT + patches the setup-key path. refreshInterval 1h->10m.
SHIPPED SUSPENDED + UNTESTED: needs (1) the bootstrap PAT provisioned to homelab/infrastructure/secret-rotators-bootstrap/netbird, (2) NB_AUTO_GROUPS filled with real NetBird group IDs, (3) a manual on-demand test, before spec.suspend is set false. Class-B template for the cloudflare + proxmox rotations to follow.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
Rotates crowdsec-bouncer-key:key quarterly + on-demand via the openbao-ns pattern (per-secret rotator-crowdsec-bouncer role/policy, no netpol). Self-sync: the single KV path feeds both the crowdsec + ingress ns Secrets; Reloader restarts crowdsec-lapi. Both ExternalSecrets refreshInterval 1h->10m.
CAVEAT handled: traefik/crowdsec-middleware.yaml has a THIRD hardcoded copy of this key (not Secret-sourced) — that middleware is inert, and a loud guard comment now warns it must be refreshed from OpenBao before re-enabling. Requires the one-time OpenBao bootstrap.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
Rotates authentik-secrets:secret-key (Django signing key) quarterly + on-demand via the openbao-ns pattern (per-secret rotator-authentik role/policy, no netpol). Self-sync 2-sided: server + worker read the same secret; Reloader restarts both via global.podAnnotations. smtp-password in the same KV entry is preserved. externalsecret refreshInterval 1h->10m. BLAST RADIUS: invalidates active SSO sessions + in-flight signed links (re-login); downstream LDAP/proxy outposts unaffected. Requires the one-time OpenBao bootstrap.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
Moves the WOODPECKER_AGENT_SECRET rotation from the apps namespace to the openbao namespace (matching the new paperless prototype), so it reaches OpenBao intra-namespace:
- openbao/rotation/woodpecker.yaml: SA + CronJob now in openbao ns, per-secret role rotator-woodpecker.
- openbao/policies/rotator-woodpecker.hcl: replaces the shared rotator policy (retired); per-secret least-privilege.
- REVERTS the #299 netpol pair (apps egress + openbao ingress) — no longer needed intra-ns.
- deletes apps/woodpecker/rotate-agent-secret.yaml and the old rotator.hcl.
server.yaml/agent.yaml (Reloader annotations) and externalsecret.yaml (10m) are unchanged.
Bootstrap: create rotator-woodpecker role/policy in openbao and delete the old rotator role/policy (documented in rotator-woodpecker.hcl).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
First fan-out of the secret-rotation pattern proven on woodpecker, now using the cleaner openbao-namespace design: the rotation Job runs in the openbao ns so it reaches OpenBao intra-namespace — NO NetworkPolicy edits needed (unlike woodpecker's apps-ns job). Per-secret isolation: its own scoped policy+role (rotator-paperless) so a compromised job can only touch the paperless KV path.
- openbao/rotation/paperless.yaml: SA + quarterly/on-demand CronJob (openbao ns) that bao kv patch's secret-key at homelab/apps/paperless (DB creds preserved).
- openbao/policies/rotator-paperless.hcl: least-privilege policy (manual bootstrap, documented in-file).
- paperless.yaml: reloader annotation; externalsecret.yaml: refreshInterval 1h->10m.
Rotating the Django SECRET_KEY invalidates paperless sessions (re-login); no data loss. Requires the one-time OpenBao bootstrap before first run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
The woodpecker-secret-rotate CronJob (#297) runs in ns apps and could not reach OpenBao (i/o timeout on auth/kubernetes/login) — blocked on BOTH sides: apps is egress-default-deny (no openbao allow) and openbao is ingress-default-deny (rotator not allowlisted). The snapshot job avoids this by living in the openbao ns; ESO is explicitly allowlisted.
- default-deny.yaml (apps egress): namespace-wide allow to openbao:8200, scoped in effect by openbao's ingress (mosquitto pattern).
- openbao-ns.yaml (openbao ingress): scoped allow from the rotator pod labels (apps/woodpecker/secret-rotator) — the real security boundary.
Verified apps has no other blocking egress; kustomize build + yamllint clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
Closes the H2 gap from the rotation PR (#297) adversarial review: the rotation Job is fire-and-forget, so a downstream failure (ESO/Reloader/pod-reschedule) yields a green Job with no signal. ESO metrics aren't scraped here, so these are kube-state-metrics based:
- WoodpeckerServerNotReady (critical): server StatefulSet 0 ready for 20m.
- WoodpeckerAgentNotReady (warning): agent Deployment 0 available for 20m.
- WoodpeckerSecretRotationJobFailed (warning): named signal for a failed rotation Job (e.g. missing OpenBao bootstrap).
- WoodpeckerSecretRotationStale (warning): quarterly CronJob hasn't succeeded in >100d; absent-until-first-success so it can't false-fire early.
Independent of #297: readiness alerts are useful immediately; the Job/CronJob series stay empty until that PR merges.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
- H1: pin Reloader chart >=1.0.0 <2.0.0 (was open upper bound; a cluster-wide restart-authority controller must not float across majors — repo pinning discipline).
- M2: correct the restart-downtime comment (server StatefulSet is unavailable for tens of seconds, not 'within seconds').
- M1: note the on-demand/emergency rotation runs during incidents (the 'outside CI hours' safety doesn't apply there).
- H2: document the no-convergence-check limitation + follow-up (add rollout/health check or alert before scaling the pattern).
- L1: correct rotator.hcl rationale — native 'bao kv patch' is a server-side merge, needs no client read-modify; 'read' is surplus.
- L2: warn against token_no_default_policy=true (breaks the kv-v2 preflight).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
End-to-end cluster-internal rotation of the shared server<->agent gRPC secret, no per-rotation human step:
- rotate-agent-secret.yaml: quarterly CronJob (+ on-demand) logs into OpenBao via k8s-auth role 'rotator', generates a random value, bao kv patch's homelab/apps/woodpecker (Gitea OAuth fields untouched).
- infrastructure/reloader: Stakater Reloader restarts annotated workloads on Secret change.
- server.yaml/agent.yaml: reloader annotation for woodpecker-secret.
- externalsecret.yaml: refreshInterval 1h -> 10m so a rotation propagates within ~10m.
- openbao/policies/rotator.hcl: least-privilege policy (patch on the one path only); NOT Flux-applied, one-time manual bootstrap documented in-file (eso-read.hcl convention).
PoC scope: woodpecker only. Requires a one-time OpenBao bootstrap (rotator policy + role) before the CronJob succeeds.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
No hardcoded IP remains in the pve scrape config: the node job now
targets jupiter/neptune/saturn.renner.uno. instance label reads the
hostname, and a renumbered node follows DNS without a config edit.
Verified: exporter returns node-local metrics when targeted at
jupiter.renner.uno from inside the cluster.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
The pve cluster-collector job hardcoded 10.2.40.201, so cluster-wide
metrics (pve_up, guest info, storage) paused whenever jupiter
specifically was down, even with the cluster healthy. Point it at the
pve Service DNS (pve.apps.svc.cluster.local) whose EndpointSlice
load-balances the PVE API across all three nodes, so the scrape
follows whichever node is up. Node-local job keeps the three explicit
targets — it needs per-node identity.
Verified: exporter returns pve_up for cluster + all nodes when
targeted at the Service DNS.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
claude-bot
k.renner
gitea_admin
security-bot