feat(flux): auto-rotate the Flux git token (gitea-token mode) #353

2026-07-22T10:57:44Z

claude-bot commented

2026-07-22 10:57:44 +00:00

Final piece of the Flux-token work. Wires the dedicated flux gitea account's PAT into the rotation fleet — so the credential behind the recent GitOps outage can't silently die again. Uses the gitea-token mode from #348.

Flow

mint (basic auth, flux password bootstrap) → verify pull AND push → store in OpenBao → ESO merges into the flux-system secret → source-controller reads fresh. A token that fails verification is deleted and the run aborts without storing; the reap keeps the two newest so the previous token stays valid through the ESO propagation window.

Reachability + wiring

gitea via the in-cluster Service gitea.apps.svc:3000 (the public name hairpins off the ingress LB from a pod). Both netpol halves added.
flux-system added to the openbao ClusterSecretStore's allowed namespaces.
ExternalSecret creationPolicy: Merge — an ESO failure leaves Flux's last-good credential in place.

Tested live end-to-end

job: minted flux-gitops-1b8d20, pull+push verified, stored (0 reaped)
OpenBao: e26e8bf1 -> 9666320e; gitea: new + previous KEPT
ESO synced -> flux-system secret
Flux pull: fetched;  image-automation push: Ready=True

Completes the fleet: the Flux git credential now rotates quarterly with a pull+push safety gate. Policy/role + flux password bootstrap provisioned by hand (rotator-gitea-flux.hcl).

🤖 Generated with Claude Code

Final piece of the Flux-token work. Wires the dedicated `flux` gitea account's PAT into the rotation fleet — so the credential behind the recent GitOps outage can't silently die again. Uses the `gitea-token` mode from #348. ## Flow mint (basic auth, flux password bootstrap) → **verify pull AND push** → store in OpenBao → ESO **merges** into the `flux-system` secret → source-controller reads fresh. A token that fails verification is deleted and the run aborts without storing; the reap **keeps the two newest** so the previous token stays valid through the ESO propagation window. ## Reachability + wiring - gitea via the in-cluster Service `gitea.apps.svc:3000` (the public name hairpins off the ingress LB from a pod). Both netpol halves added. - `flux-system` added to the openbao ClusterSecretStore's allowed namespaces. - ExternalSecret `creationPolicy: Merge` — an ESO failure leaves Flux's last-good credential in place. ## Tested live end-to-end ``` job: minted flux-gitops-1b8d20, pull+push verified, stored (0 reaped) OpenBao: e26e8bf1 -> 9666320e; gitea: new + previous KEPT ESO synced -> flux-system secret Flux pull: fetched; image-automation push: Ready=True ``` Completes the fleet: the Flux git credential now rotates quarterly with a pull+push safety gate. Policy/role + flux password bootstrap provisioned by hand (rotator-gitea-flux.hcl). 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit 2026-07-22 10:57:44 +00:00

feat(flux): auto-rotate the Flux git token (gitea-token mode) 0d6fc81e9b

Wires the Flux git credential — the dedicated `flux` gitea account — into the
rotation fleet, so it can't silently die again (the cause of the recent GitOps
outage). Quarterly, via the secret-rotator binary's gitea-token mode.

Flow: mint a new PAT (basic auth, the flux password bootstrap) -> VERIFY it can
pull AND push (Flux image-automation writes back) -> store in OpenBao -> ESO
merges it into the flux-system secret -> source-controller reads it fresh. A
token that fails verification is deleted and the run aborts without storing, and
the reap keeps the two newest tokens so the previous one stays valid through the
ESO propagation window — Flux is never stranded.

Reachability: gitea via the in-cluster Service gitea.apps.svc:3000 — the public
gitea.renner.uno name hairpins off the ingress LB from a pod (same as
pve.renner.uno). Needs both netpol halves (gitea-allow-flux-rotator in apps +
rotator-gitea-flux-egress in openbao). Added flux-system to the openbao
ClusterSecretStore's allowed namespaces so the ExternalSecret can sync there.
The ExternalSecret uses creationPolicy: Merge, so an ESO failure leaves the
last-good credential in place rather than blanking Flux's secret.

Tested live end-to-end: job minted flux-gitops-1b8d20, verified pull+push,
stored (OpenBao e26e8bf1 -> 9666320e), kept the previous token (2 tokens, 0
reaped), ESO synced, Flux pull fetched, and image-automation pushed
(Ready=True). OpenBao policy/role + the gitea-flux password bootstrap are
provisioned by hand (see rotator-gitea-flux.hcl).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

review-bot reviewed

2026-07-22 11:20:41 +00:00

review-bot left a comment

🤖 Automated review — model claude-opus-4-8 (complexity: complex), commit 0d6fc81e9bf81f6307ccaa6208a8ab80b669f005.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

Adds the final piece of the Flux git-token rotation: a gitea-token-mode
rotator CronJob + ESO ExternalSecret that mints, pull/push-verifies, and
syncs the dedicated flux account's PAT into the flux-system git secret, plus
the two netpol halves and the ClusterSecretStore namespace allow. The wiring is
correct and consistent with the existing rotator fleet and the gitea-token
implementation in secret-rotator/src/main.rs. No blocking issues.

🔴 Blocking

None.

🟡 Suggestions

Pin GITEA_REQUIRE_PUSH: "true" explicitly in
kubernetes/infrastructure/openbao/rotation/flux-git.yaml:200. verify_repo
here (k.renner/homelab) is a public repo (pr.json → repo.private: false), and the rotator's own comment
(secret-rotator/src/main.rs:283-287) warns that for a public repo the
pull check (git-upload-pack) returns 200 for any valid token regardless of
scope — so the push check is the only load-bearing verification. It currently
works because require_push defaults to true
(main.rs:255), but that safety property is invisible in this manifest and
would silently degrade to "is this any valid token" if the default ever
changed. Making it explicit documents the invariant at the call site. Optional.
creationPolicy: Merge won't recreate flux-system if it's deleted. ESO
Merge only patches an existing secret; if the flux-system secret is ever
removed (e.g. namespace rebuild) the rotated credential won't be re-seeded and
Flux git auth stays broken until the bootstrap runs again
(flux-git.yaml:135). This is the intended trade-off (a sync failure leaves
the last-good value in place, per the header comment) and matches how Flux
bootstrap owns creation, so no change needed — just confirming the failure
mode is understood.

🟢 Nits

Schedule 0 4 1 1,4,7,10 * coincides with ~10 other rotators at 04:00
(authentik, cloudflare, pve-exporter, etc.). This is the fleet norm and
they hit independent targets, so it's fine — the flux-git job is the only one
touching gitea's git endpoints. No action.

Tests

This is declarative Kubernetes/OpenBao config; the repo carries no manifest unit
tests for the sibling rotators, so none are expected here. The rotation logic it
drives lives in secret-rotator/src/main.rs::rotate_gitea_token and is exercised
there. The PR body documents a live end-to-end run (mint → pull+push verify →
OpenBao patch → ESO merge → Flux fetch + image-automation push Ready=True),
which is the right validation for an infra change of this shape. I verified the
static wiring by reading the code: bootstrap/target paths and read/patch
capabilities in policies/rotator-gitea-flux.hcl match the env vars and the
assert_fields_exist+patch_field calls; the ESO key infrastructure/flux-git
resolves under the store's homelab v2 path to the policy's
homelab/data/infrastructure/flux-git; the merged username/password keys
match what the flux-system GitRepository secretRef expects
(clusters/homelab/flux-system/gotk-sync.yaml); and both netpol halves' selectors
match the CronJob pod labels and the gitea endpoint selector used by the existing
metrics-scrape policy.

Recommendation

Looks reasonable — no blocking issues; comments above are optional.

🤖 **Automated review** — model `claude-opus-4-8` (complexity: complex), commit `0d6fc81e9bf81f6307ccaa6208a8ab80b669f005`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary Adds the final piece of the Flux git-token rotation: a `gitea-token`-mode rotator CronJob + ESO `ExternalSecret` that mints, pull/push-verifies, and syncs the dedicated `flux` account's PAT into the `flux-system` git secret, plus the two netpol halves and the ClusterSecretStore namespace allow. The wiring is correct and consistent with the existing rotator fleet and the `gitea-token` implementation in `secret-rotator/src/main.rs`. No blocking issues. ## 🔴 Blocking None. ## 🟡 Suggestions - **Pin `GITEA_REQUIRE_PUSH: "true"` explicitly** in `kubernetes/infrastructure/openbao/rotation/flux-git.yaml:200`. `verify_repo` here (`k.renner/homelab`) is a **public** repo (`pr.json` → `repo.private: false`), and the rotator's own comment (`secret-rotator/src/main.rs:283-287`) warns that for a public repo the pull check (`git-upload-pack`) returns 200 for *any* valid token regardless of scope — so the push check is the only load-bearing verification. It currently works because `require_push` defaults to `true` (`main.rs:255`), but that safety property is invisible in this manifest and would silently degrade to "is this any valid token" if the default ever changed. Making it explicit documents the invariant at the call site. Optional. - **`creationPolicy: Merge` won't recreate `flux-system` if it's deleted.** ESO Merge only patches an existing secret; if the `flux-system` secret is ever removed (e.g. namespace rebuild) the rotated credential won't be re-seeded and Flux git auth stays broken until the bootstrap runs again (`flux-git.yaml:135`). This is the intended trade-off (a sync failure leaves the last-good value in place, per the header comment) and matches how Flux bootstrap owns creation, so no change needed — just confirming the failure mode is understood. ## 🟢 Nits - Schedule `0 4 1 1,4,7,10 *` coincides with ~10 other rotators at 04:00 (`authentik`, `cloudflare`, `pve-exporter`, etc.). This is the fleet norm and they hit independent targets, so it's fine — the flux-git job is the only one touching gitea's git endpoints. No action. ## Tests This is declarative Kubernetes/OpenBao config; the repo carries no manifest unit tests for the sibling rotators, so none are expected here. The rotation logic it drives lives in `secret-rotator/src/main.rs::rotate_gitea_token` and is exercised there. The PR body documents a live end-to-end run (mint → pull+push verify → OpenBao patch → ESO merge → Flux fetch + image-automation push `Ready=True`), which is the right validation for an infra change of this shape. I verified the static wiring by reading the code: bootstrap/target paths and `read`/`patch` capabilities in `policies/rotator-gitea-flux.hcl` match the env vars and the `assert_fields_exist`+`patch_field` calls; the ESO key `infrastructure/flux-git` resolves under the store's `homelab` v2 path to the policy's `homelab/data/infrastructure/flux-git`; the merged `username`/`password` keys match what the `flux-system` GitRepository `secretRef` expects (`clusters/homelab/flux-system/gotk-sync.yaml`); and both netpol halves' selectors match the CronJob pod labels and the gitea endpoint selector used by the existing metrics-scrape policy. ## Recommendation Looks reasonable — no blocking issues; comments above are optional.

claude-bot added 1 commit 2026-07-22 11:49:46 +00:00

chore(flux): pin GITEA_REQUIRE_PUSH=true on the flux-git rotator 9f43ad7abb

Per review. k.renner/homelab is public, so the pull verification passes for any
valid token — push is the only load-bearing check. It already defaulted to true
in the binary, but pinning it in the manifest documents the safety invariant at
the call site so it can't silently degrade if the default ever changed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

claude-bot commented

2026-07-22 11:50:03 +00:00

Review addressed — no blockers

Pinned GITEA_REQUIRE_PUSH: "true" in the manifest (commit pushed). You're right that push is the only load-bearing check here — the repo is public, so git-upload-pack passes for any valid token — and it should be visible at the call site, not just a binary default. Added with a comment explaining why.

creationPolicy: Merge won't recreate a deleted flux-system secret — confirmed, that's intended: Flux bootstrap owns creation, ESO owns the values, and Merge means a sync failure leaves the last-good credential rather than blanking Flux's git auth. On a namespace rebuild the bootstrap re-seeds it (documented in the cold-rebuild runbook path). No change.

Schedule at 04:00 with the other rotators — fleet norm, independent targets; flux-git is the only one touching gitea. No action.

Thanks for the static-wiring verification — good to have the paths/capabilities/netpol-selectors independently confirmed.

## Review addressed — no blockers **Pinned `GITEA_REQUIRE_PUSH: "true"`** in the manifest (commit pushed). You're right that push is the only load-bearing check here — the repo is public, so `git-upload-pack` passes for any valid token — and it should be visible at the call site, not just a binary default. Added with a comment explaining why. **`creationPolicy: Merge` won't recreate a deleted `flux-system` secret** — confirmed, that's intended: Flux bootstrap owns creation, ESO owns the values, and Merge means a sync failure leaves the last-good credential rather than blanking Flux's git auth. On a namespace rebuild the bootstrap re-seeds it (documented in the cold-rebuild runbook path). No change. **Schedule at 04:00 with the other rotators** — fleet norm, independent targets; flux-git is the only one touching gitea. No action. Thanks for the static-wiring verification — good to have the paths/capabilities/netpol-selectors independently confirmed.

k.renner merged commit ab66e4c67e into master

2026-07-22 11:54:40 +00:00

k.renner referenced this issue from a commit

2026-07-22 11:54:41 +00:00

Merge pull request 'feat(flux): auto-rotate the Flux git token (gitea-token mode)' (#353) from feat/flux-token-rotation into master

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#353