17 Commits

Include renames

Author	SHA1	Message	Date
hermes-bot	6f1a7b6d96	fix(secret-rotator): never fail the HA run after the store; stage the image rollout ... Folds the good parts of #1412 into this PR and addresses both review-bot reviews (#1412 review 1789, #1414 review 1791): - Everything after the OpenBao store is best-effort. Decode, reconnect, listing and revokes run in reap_superseded(); reap_summary() turns any error into "reap skipped" plus a warning, so rotate_ha_token returns Ok. A failed run would start a second pod (backoffLimit: 1) that keeps the token just stored and its successor and reaps the token consumers still hold until ESO syncs, i.e. the #1081 outage. Two unit tests cover it. - Warn when the replaced token is not named "<client_name> [<6 hex>]": no later run reaps it, so it needs a manual revoke (from #1412). - Staged image rollout (from #1412): only the three HA CronJobs carry the flux-system:secret-rotator marker; the other 17 rotators stay on 2026-07-30 until a manual HA run validates the new binary, then a follow-up adds the marker to them. rotation/kustomization.yaml records the exception. - Merged current main (a6f37e2b, 1691a0ac): the Reloader annotations for zone-configurator and speech-to-phrase are already there via #1409.	2026-10-21 07:59:52 +00:00
hermes-bot	0fd90e07e2	fix(secret-rotator): keep the replaced HA token until the next rotation ... ha-token mode revoked the outgoing LLAT seconds after storing the new one in OpenBao. Consumers read the k8s Secret, which ESO syncs every 1h, so on 2026-10-01 the ServiceMonitor scrape got 401 from 02:00 to ~02:36Z (HomeAssistantDown, TargetDown) and zone-configurator crashlooped (#1081, #1082, #1084). The reap now keeps the token just stored and the one it replaced, and revokes only rotator-minted tokens (exact "<client_name> [6 hex]") other than those two, the same keep-two rule gitea-token mode got in 59e804d9. The replaced token is revoked by the following quarterly run. The binary only reaches the CronJobs through a new image, and every rotator was hand-pinned to secret-rotator:2026-07-30. Enrol the image in Flux image automation (dated-tag policy, as for ci-backup-verify) and mark all 20 rotator image lines, so they move together to the newest built tag. Source between 2026-07-30 and this commit differs only by the Rust 1.91 -> 1.99 build image (68c09786). Closes #1081	2026-10-21 07:41:16 +00:00
renovate-bot	68c0978660	chore(deps): update all non-major dependencies	2026-10-05 02:01:26 +00:00
claude-botandClaude Opus 4.8	16013a4ca0	feat(secret-rotator): sync bot tokens into Woodpecker org secrets ... Add a woodpecker-secret mode to secret-rotator that upserts an OpenBao-held value into a Woodpecker ORG-level CI secret (create if absent, else patch value+events), and wire two rotators that use it — moving review_bot_token and claude_code_oauth_token off hand-loading into the rotation toolchain and onto org-level secrets every repo inherits. secret-rotator: - woodpecker.rs: org-id lookup, existence probe (404=absent), create/patch an org secret via a dedicated woodpecker-bot bearer token. Never logs the value. - main.rs: woodpecker-secret mode reads a value from BAO_PATH#BAO_FIELD (errors if missing/empty, so a blank never overwrites a working secret) and the woodpecker-bot token from the bootstrap, then upserts the org secret with the configured events (must include pull_request for the PR-review pipeline). rotators (kubernetes/infrastructure/openbao/rotation/): - review-bot.yaml: one CronJob, init step mints review-bot's PAT (gitea-token mode, pull-verified against a private repo it collaborates on; push not verified — it's a read collaborator that posts reviews via the API), main step syncs it to org secret review_bot_token (events cron,manual,pull_request). - claude-oauth.yaml: sync-only (the OAuth token is interactive setup-token, not mintable) — daily push of an owner-seeded OpenBao value to org secret claude_code_oauth_token. - policies/rotator-{review-bot,claude-oauth}.hcl: least-privilege OpenBao policies + documented hand-seeded bootstraps (review-bot password, woodpecker-bot token, seeded OAuth/token fields). netpol: rotator-{review-bot,claude-oauth}-egress (openbao-ns) + gitea-allow-review-bot-rotator + a new woodpecker-allow-rotators, so the rotators reach gitea:3000 / woodpecker:8000 through apps default-deny. Deploy prerequisites (can't be done from git): pin the image @sha256 once a build including woodpecker-secret mode lands; apply the two OpenBao policies/ roles; seed the bootstraps. After the org secrets exist, drop homelab's repo-level claude_code_oauth_token / review_bot_token copies. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-23 12:52:49 +02:00
claude-botandClaude Opus 4.8	59e804d9a4	fix(secret-rotator): keep the previous gitea token through the ESO propagation window ... Second review pass (no blocking issues). The important one: the reap deleted EVERY other prefix token right after store, including the previous run's token — which is still the live credential in the k8s Secret until ESO syncs the new value and source-controller re-reads it. Deleting it immediately opens an availability window (ESO refreshInterval + reconcile lag) where Flux's stored token has just been revoked. Since this mode exists specifically to avoid stranding Flux, that window shouldn't exist. Now the reap keeps the TWO newest tokens (the one just stored + the previous) and deletes only older ones. tokens_with_prefix returns (id, name) so the caller can order by recency; the previous token survives until the next quarterly run, long after ESO has synced. Also: - doc-note that PUSH is the load-bearing gate check: for a public verify_repo, git-upload-pack returns 200 for any valid token, so pull barely gates — don't set GITEA_REQUIRE_PUSH=false against a public repo. - fetch tokens with limit=50 (single effective page) and note the count stays tiny given keep-two reaping, so pagination can't silently miss old tokens. - tidy the cleanup error message connector. clippy -D warnings clean; 14 tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-22 12:45:07 +02:00
claude-botandClaude Opus 4.8	82f6bc765f	fix(secret-rotator): clean up orphaned gitea token on all post-mint failures ... Addresses the review on the gitea-token mode. Blocking: the cleanup only ran on the Ok(false) gate branch. If token_allows returned Err (a wrong GITEA_VERIFY_REPO, a 5xx) or the OpenBao store failed after verification passed, the `?` propagated and the just-minted, write-scoped PAT was orphaned with no delete and no log — contradicting the stated guarantee, and unlike ha.rs which logs its orphan. Fixed: verify+store now run in a closure, and ANY error after the mint deletes the token; if the delete also fails, a CRITICAL line names the live token so it can be revoked by hand. Also from the review: - distinguish 401 (minted token rejected — auth, surfaced as an error) from 403 (valid token, insufficient scope — a real gate "no") in token_allows, instead of collapsing both to Ok(false). - reject an empty GITEA_SCOPES up front, matching BAO_FIELDS / NB_AUTO_GROUPS. - add a basic() unit test (gitea.rs had none) pinning the auth-header format; crate now has 14 tests. - tighten the doc comments that overstated the guarantee. clippy -D warnings clean; 14 tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-22 09:25:20 +02:00
claude-botandClaude Opus 4.8	3862e9a806	feat(secret-rotator): gitea-token mode ... Rotates a gitea PAT used as a git credential — for Flux's own flux-system secret. Gitea blocks token-from-token (403), so minting uses BASIC auth with a dedicated service account's password as the bootstrap, like the Cloudflare and Proxmox privileged bootstraps. Make-before-break with a hard safety gate: mint a new token, VERIFY it can both pull (git-upload-pack) AND push (git-receive-pack, since Flux image-automation writes back), and only then store it. A token that fails verification is deleted and the rotation aborts WITHOUT storing — so a bad rotation can never strand Flux the way a revoked token did. Old rotated tokens (same name prefix) are reaped after, non-fatally. The push check is required by default but env-toggleable (GITEA_REQUIRE_PUSH) for read-only consumers. Scopes, verify-repo, name prefix and field names are all env-driven. Validated against the real gitea API on throwaway tokens (zero blast radius, live flux token untouched): - write token -> pull 200 + push 200 -> gate PASS - read-only token -> pull 200 + push 403 -> gate REJECTS it (correctly refuses to store a token that cannot push) - list-by-prefix + delete (reap) works clippy -D warnings clean; 13 tests pass. No rotator uses it yet — the ExternalSecret + CronJob follow once CI publishes the image. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-22 08:46:07 +02:00
claude-botandClaude Opus 4.8	af1737a657	fix(secret-rotator): correct proxmox TLS rationale + log DELETE failures ... Addresses the advisory review on the proxmox-token mode. The code claimed the PVE cert is self-signed. It is not — the in-cluster Service presents a VALID Let's Encrypt cert (verified live: CN=neptune.renner.uno, issuer Let's Encrypt). Verification fails only on HOSTNAME mismatch: the job connects to the pve.apps.svc.cluster.local Service (for 3-node failover) while the cert is for a node FQDN. Corrected the module header and the insecure_agent comment to say this accurately, and spelled out the real tradeoff — ureq 3 has no skip-hostname-only option, so it's disable-verification-with-failover (this, matching the python's prior CERT_NONE) vs pin-to-one-node-FQDN-with-full-verify -but-SPOF. Noted the deferred improvement (a custom rustls verifier that validates the LE chain but skips hostname). Also from the review: - log a warning when the DELETE result is an Err, so a re-run needn't rediscover why a first attempt behaved oddly (still non-fatal — a leftover token makes the POST fail loud). - documented why the raw URL-path interpolation is safe (fixed operator config, never untrusted; percent-encoding @ to %40 would break PVE's userid lookup). No behaviour change to the rotation itself. clippy -D warnings clean; 13 tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-22 06:52:37 +02:00
claude-botandClaude Opus 4.8	1324728721	feat(secret-rotator): proxmox-token mode ... Adds the last mode — the one that will replace the pve-exporter python rotator, the hardest of the fleet. Proxmox has no in-place token roll, so this DELETEs and recreates the token with the same id, then patches OpenBao with a rendered pve.yml document. Two capabilities new to the binary: - no-verify TLS: a dedicated agent with rustls verification disabled, used ONLY for the PVE calls. The API is reached via the in-cluster Service ClusterIP, whose cert never matches *.svc.cluster.local, so webpki verification can't apply. OpenBao still uses the normal agent. - a rendered document, not a bare value: pve.yml is emitted byte-identical to the python template (2-space indent, trailing newline, verify_ssl:false), unit-tested. privsep=0 on recreate makes the token inherit pve-exporter@pve's own permissions, so no separate token ACL is re-applied — verified against the live cluster earlier (the PVEAuditor ACL is on the user, not the token). The auth credential is the least-privilege secret-rotator@pve!rotate identity, not root. DELETE ignores its own outcome (a missing token is a no-op; a real auth/perm problem surfaces at the POST, where it's fatal — a delete that leaves the token in place makes the POST fail loud with "already exists"). If the recreate fails after a successful delete, the store never happens and the error is hard: the consumer is down until re-run. This destructive window is inherent to Proxmox and unavoidable without changing the token id each rotation. Validated against real Proxmox 8.4.18 on a throwaway token (zero blast radius, never touched the real 'monitoring' token): DELETE 200 -> POST privsep=0 form -> 36-char value, no trailing newline, and the new token authenticates (/version 8.4.18). render_pve_yml is byte-exact unit-tested. clippy -D warnings clean; 13 tests pass. No rotator uses it yet — the manifest port follows once CI publishes the image. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-22 00:51:30 +02:00
claude-botandClaude Opus 4.8	2611d7bc81	feat(secret-rotator): netbird-key mode ... Adds the mode that will replace the NetBird python rotator. Mints a fresh reusable setup key via POST /api/setup-keys and patches it into OpenBao. Create-new, not roll-in-place: the old key is left to expire (keys are ephemeral with a bounded expiry), so nothing is deleted — no destructive step, no window. This deliberately does NOT delete old keys; a past manual cleanup that filtered keys for deletion wiped unrelated ones (a jq precedence bug), and that risk has no place in an automated rotator. NetBird auth is `Authorization: Token <pat>` — NOT Bearer. The key/type/ usage_limit semantics (reusable, unlimited) are fixed; name, expiry, ephemeral, allow_extra_dns_labels and auto_groups come from the environment. The returned key is used verbatim (serde_json, no trailing newline). assert_fields_exist guards the target path. An error that echoes the response redacts the `key` field so a value can never leak into logs. Validated against the real NetBird API on a throwaway key (created then deleted, zero blast radius): TLS verifies with the built-in webpki roots (no no-verify context needed), POST with Token auth returns a 36-char key with no trailing newline and the expected settings (reusable/ephemeral/usage_limit 0/ 1 auto_group). clippy -D warnings clean; 12 tests pass; the in-cluster pieces are the same code already proven live by the other modes. No rotator uses it yet — the manifest port follows once CI publishes the image. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-21 17:20:43 +02:00
claude-botandClaude Opus 4.8	d88c4a6cd9	feat(secret-rotator): cloudflare-roll mode ... Adds the mode that will replace the two identical Cloudflare python rotators (cert-manager's token and external-dns's). They differ only in which bootstrap field holds the token id and which OpenBao path receives the value: cloudflare bootstrap#token-id -> infrastructure/cloudflare#api-token external-dns bootstrap#external-dns-token-id -> apps/external-dns#api-token Both read the admin token from the same shared bootstrap, so it lives in one place; only the token-id pin differs, keeping the two tokens rolling independently. Rolls the token value in place via PUT /user/tokens/{id}/value — same id, scope and policy, new value — so there is no make-before-break and no destructive window. assert_fields_exist guards the target path (a typo would strand the real token), and the API value is used verbatim: a stray newline would surface as "invalid header field value for Authorization" and crash the consumer (the bug that hit external-dns once), so the string comes straight from serde_json with nothing appended. Handles result as either a bare string or {value: ...} so a future API shape change fails loud rather than storing "null". Validated against the real Cloudflare API on a throwaway token (zero blast radius): PUT with empty body returns success + a bare-string value, new != old with no trailing newline, new verifies active while the old immediately reads "Invalid API Token" (confirming roll-in-place has no window). clippy -D warnings clean; 12 tests pass. No rotator uses it yet — the manifest port follows once CI publishes the image. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-21 16:58:40 +02:00
claude-botandClaude Opus 4.8	2e18d68a94	fix(secret-rotator): revoke old HA token over a new-token session ... Found by running the ha-token rotation end-to-end against live HA. The rotation was fully correct — new token minted, stored, and the old one revoked — but the job logged a false "OLD token REMAINS VALID" warning. Root cause: HA binds a WebSocket session to the refresh token behind its access token. The rotator authenticated with the OLD token and revoked that same token over that same session, so HA tore the session down as it processed the delete — the delete succeeded server-side but the ack could not be read, surfacing as "connection closed by HA". Verified: after the false-warning run, the old token's refresh id was already gone from HA. Fix: after storing the new token, drop the old session and reconnect with the NEW token to perform the revoke. The session is then not the one being deleted, so the ack reads cleanly and the report is accurate. This matters beyond cosmetics: the false warning fired on every successful rotation, which would train an operator to ignore it — and a genuine revoke failure would then be indistinguishable noise. With the fix, a warning means a real un-revoked token. Verified end-to-end against live HA on a throwaway token: revoke over the new session returns success=True and the old token is confirmed gone (where the old flow got "connection closed"). clippy -D warnings clean; 12 tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-21 12:14:55 +02:00
claude-botandClaude Opus 4.8	30c0e5632a	fix(secret-rotator): mint HA tokens under a unique name ... Found by testing ha-token mode against live HA before wiring up the CronJobs. Home Assistant refuses to mint a second long-lived access token whose client_name already exists — and make-before-break necessarily mints while the outgoing token (same base name) is still present. So every real rotation would fail at the mint step with "Unknown error", exactly as observed: same-name mint -> FAIL {'code':'unknown_error'} unique-name mint -> SUCCESS The earlier ad-hoc test only passed because it minted a NEW name ("Config Sync") rather than rotating an existing one. Fix: mint under "<base> [<6 hex>]". Correctness does not depend on the name — the outgoing token is identified by its JWT `iss`, never by name — so a random tag is free, and it also avoids colliding with an orphan left by a prior failed run. The failure mode was safe throughout: mint fails before store or revoke, so the old token stays valid and stored (a no-op, not an outage). Verified end-to-end against live HA on a throwaway token: unique mint succeeds, the outgoing token is found by iss, store patches OpenBao, and the old token is revoked and stops authenticating. clippy -D warnings clean; 12 tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-21 12:03:23 +02:00
claude-botandClaude Opus 4.8	808cb5397d	fix(secret-rotator): HTTP/WS timeouts, entropy floor, orphan-token logging ... Addresses three review findings, all real bugs rather than style: 1. No timeouts anywhere. The Python this replaces passed timeout=30 on every call; ureq 3's default agent has none and tungstenite has no read timeout, so a hung endpoint would stall the Job silently until activeDeadlineSeconds with no useful error — and the cluster demonstrably has hanging paths (the ingress-LB hairpin). Now: a shared ureq::Agent with timeout_global(30s) threaded through every OpenBao call, and a 30s read timeout set on the TcpStream under tungstenite's socket (handling both plain and rustls streams). 2. RANDOM_BYTES=0 would generate a near-empty secret, store it, and report success — the same silent-misconfig class assert_fields_exist guards. Now floored at 16 bytes (128 bits); every real rotator uses 32 or 50. 3. If mint succeeds but the OpenBao store fails, the freshly minted token is valid for its full lifespan, referenced nowhere, and never auto-revoked (revocation only targets the OLD stored id). The store error path now logs the orphan's refresh-token id — recovered from its own JWT — so it can be revoked by hand, and makes clear the consumer is unaffected because the old token is still stored and valid. clippy -D warnings clean; 10 tests pass; scratch image builds (4.92 MB) and all error paths exit non-zero. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-21 09:07:52 +02:00
claude-botandClaude Opus 4.8	49b2eaf819	feat(secret-rotator): random mode, replacing the five Class-A rotators ... paperless, authentik, woodpecker, crowdsec-bouncer-key and shelly-manager all ran the same shell one-liner: head -c <N> /dev/urandom | base64 | tr -d '\n' so the port has to be byte-compatible, not merely similar: N raw bytes, the STANDARD base64 alphabet (+ and /, not URL-safe), WITH = padding, no newline. Consumers store these verbatim and none filter the character set, so quietly switching to base64url or dropping padding would change the credential shape for no reason. The encoder is verified against the RFC 4648 vectors, which are what coreutils base64 produces, plus a case asserting + and / rather than - and _. BAO_FIELDS is comma-separated and each field gets its own independent value, written in ONE merge-patch. That preserves shelly-manager's behaviour, where admin-password and api-key must land together rather than leaving a window with one rotated and one stale. Adds assert_fields_exist as a pre-flight guard. Merge-patch CREATES a field that isn't present, so a typo in BAO_FIELDS would add `secret-ky`, report success, and leave the real credential un-rotated — a silent no-op that would only surface during an incident. Rotation only ever replaces existing fields, so a missing field is now a hard error that names both what is missing and what is actually there. Byte lengths preserved from the originals: 50 for the Django-style keys (paperless, authentik), 32 for the rest. Module is `secretgen`, not `gen` — the latter is a reserved keyword in the 2024 edition. 11 tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-20 13:47:09 +02:00
claude-botandClaude Opus 4.8	fdf3818ac5	fix(secret-rotator): identify the outgoing token by JWT iss, not client_name ... Revoking by client_name would have destroyed live credentials. Checked the three stored HA tokens against auth/refresh_tokens by decoding each one's JWT iss claim, and the names are not what they look like: apps/home-assistant/prometheus-token -> "ServiceMonitor" apps/home-assistant/config-sync-token -> "laptop local" apps/home-assistant/shelly-manager-llat-> "Shelly Manager" Separate, unrelated "Prometheus" and "Config Reconciler" tokens also exist. A name-matching rotator pointed at prometheus-token would have revoked "Prometheus" — a live credential belonging to something else — and left the actual stored token valid. Same for config-sync. An HA long-lived access token is a JWT whose iss claim IS its own refresh-token id, so the outgoing token identifies itself exactly. The rotator now reads that, confirms with HA that the id really is a long-lived access token, and refuses to proceed if it is not. No name matching anywhere. The JWT signature is deliberately not verified: this is not authentication, only reading which id the token claims, and HA remains the authority via the auth/refresh_tokens confirmation before any revoke. Adds a small base64url decoder (payloads are unpadded) with unit tests covering iss extraction, unpadded decoding, non-JWT input, and a JWT with no iss. 6 tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-20 13:31:57 +02:00
claude-botandClaude Opus 4.8	d78e703575	feat(secret-rotator): Rust rotator binary in a FROM-scratch image ... Replaces the inline-Python approach before it grows a tenth copy. Today the fleet carries ~130 lines of Python embedded in YAML strings across 9 manifests, with auth/kubernetes/login duplicated in all 9 and the call()/merge-patch helpers in 4 — none of it lintable, testable, or reviewable as code. The Home Assistant rotator would have been the worst of them: a hand-rolled WebSocket handshake with frame masking inside a YAML string. HA mints long-lived access tokens only over its WebSocket API (verified against 2026.7.2 — auth/long_lived_access_token and auth/delete_refresh_token both exist; the REST API has no equivalent). Ships as a static musl binary in FROM scratch: 4.85 MB, one file, no libc, no shell, no package manager. Nothing to patch and nothing for a compromised rotator to exec. rustls with webpki-roots compiles the trust store into the binary, so TLS works without a CA bundle on disk — in-cluster traffic is plain HTTP, but that keeps the door open to take over the Cloudflare and Proxmox rotators. ha-token mode is make-before-break, and the ordering is load-bearing: list existing token ids for the client_name BEFORE minting (so the new token can never land in the revocation set), mint, write to OpenBao, and only then revoke. If the OpenBao write fails, the old token is still valid and still stored, so the run is a no-op rather than an outage. A failed revoke is deliberately non-fatal but logged loudly: the new token is already live, and a token meant to be retired that is still valid is a real finding, not noise. Verified: static-pie linked and stripped; image contains only the binary (3.2 MB binary, 4.85 MB image); /bin/sh absent; exits non-zero on usage error; cargo test passes. The pipeline path trigger now includes secret-rotator/**, without which source changes would silently not rebuild the image. No CronJob uses it yet — the Home Assistant rotator needs a network decision first: the home-assistant namespace is default-deny-ingress, so openbao cannot reach it, and the public name hairpins off the ingress LB. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-20 12:14:39 +02:00