fix(review-bot): mint the review-bot PAT with read:issue #1506

2026-10-27T23:25:13Z

hermes-bot commented

2026-10-27 23:25:13 +00:00

What fired

KubePodCrashLooping on review-bot-74b868644-ljq8p (2026-10-27 ~22:34–22:48Z). Per the issue, that RS ran the old image 2026-10-27-0757 with the REPOS env already removed (9ec31f07) and exited with REPOS is empty until image 2026-10-27-2231 (a0f1c3e1) rolled out. That part is transient and already over: live now, deployment 1/1, pod review-bot-84746b7cf5-bl8jt on 2026-10-27-2231, 0 restarts; no ALERTS{namespace="review-bot"} in Prometheus and nothing in Alertmanager.

What is still broken (observed this run)

Every 5-minute sweep since the new image started fails (kubectl -n review-bot logs deploy/review-bot, 22:45Z–23:20Z, 8×):

sweep: cannot list requested reviews ... GET /repos/issues/search?type=pulls&state=open&review_requested=true ... HTTP 403: token does not have at least one of required scope(s), required=[read:issue], token scope=write:repository

The token is REVIEW_TOKEN = OpenBao infrastructure/review-bot#token, minted by review-bot-secret-rotate with GITEA_SCOPES: write:repository. The other endpoints review-bot calls (pulls/*, pulls/*/reviews, statuses/*) are repository-scoped; issues/search is the only one needing read:issue (read is enough, the bot posts nothing via the issues API).

Change

rotation/review-bot.yaml: GITEA_SCOPES: write:repository,read:issue (the rotator splits on ,, as the MCP step already uses).
rotation/review-bot-rotate-once-job.yaml (new, one-shot): the CronJob's rotate-mint + sync-woodpecker steps as a Job, so the new PAT is minted on merge instead of 2027-01-01 (schedule 0 4 1 1,4,7,10 *). Same image, SA, labels (netpols match) and env; the MCP token step is not included. No TTL so Flux does not re-create it. Same one-shot pattern as #1146/#1147. infrastructure has wait: false, so the Job cannot gate it.

Validated: kubectl kustomize kubernetes/infrastructure/openbao renders; kubectl create --dry-run=client lists the Job, CronJob and SA.

After merge (all automatic)

Flux applies the Job; the rotator mints review-bot-rotated-* with write:repository,read:issue, verifies a pull of private konairius/shelly-manager, stores it, syncs the Woodpecker org secret, keeps the two newest tokens.
ExternalSecret review-bot-secrets (refresh 1h) picks it up; Reloader restarts review-bot.
The sweep: cannot list requested reviews 403 stops. Check: kubectl -n openbao get job review-bot-rotate-read-issue (Complete) and kubectl -n review-bot logs deploy/review-bot | grep sweep.

Follow-up: a small PR removing the one-shot Job once it shows Complete.

Hermes-Card: t_607c5834 (daedalus)

Closes #1505 ## What fired KubePodCrashLooping on `review-bot-74b868644-ljq8p` (2026-10-27 ~22:34–22:48Z). Per the issue, that RS ran the old image `2026-10-27-0757` with the `REPOS` env already removed (9ec31f07) and exited with `REPOS is empty` until image `2026-10-27-2231` (a0f1c3e1) rolled out. That part is transient and already over: live now, deployment 1/1, pod `review-bot-84746b7cf5-bl8jt` on `2026-10-27-2231`, 0 restarts; no `ALERTS{namespace="review-bot"}` in Prometheus and nothing in Alertmanager. ## What is still broken (observed this run) Every 5-minute sweep since the new image started fails (`kubectl -n review-bot logs deploy/review-bot`, 22:45Z–23:20Z, 8×): ``` sweep: cannot list requested reviews ... GET /repos/issues/search?type=pulls&state=open&review_requested=true ... HTTP 403: token does not have at least one of required scope(s), required=[read:issue], token scope=write:repository ``` The token is `REVIEW_TOKEN` = OpenBao `infrastructure/review-bot#token`, minted by `review-bot-secret-rotate` with `GITEA_SCOPES: write:repository`. The other endpoints review-bot calls (`pulls/*`, `pulls/*/reviews`, `statuses/*`) are repository-scoped; `issues/search` is the only one needing `read:issue` (read is enough, the bot posts nothing via the issues API). ## Change - `rotation/review-bot.yaml`: `GITEA_SCOPES: write:repository,read:issue` (the rotator splits on `,`, as the MCP step already uses). - `rotation/review-bot-rotate-once-job.yaml` (new, one-shot): the CronJob's `rotate-mint` + `sync-woodpecker` steps as a Job, so the new PAT is minted on merge instead of 2027-01-01 (schedule `0 4 1 1,4,7,10 *`). Same image, SA, labels (netpols match) and env; the MCP token step is not included. No TTL so Flux does not re-create it. Same one-shot pattern as #1146/#1147. `infrastructure` has `wait: false`, so the Job cannot gate it. Validated: `kubectl kustomize kubernetes/infrastructure/openbao` renders; `kubectl create --dry-run=client` lists the Job, CronJob and SA. ## After merge (all automatic) 1. Flux applies the Job; the rotator mints `review-bot-rotated-*` with `write:repository,read:issue`, verifies a pull of private `konairius/shelly-manager`, stores it, syncs the Woodpecker org secret, keeps the two newest tokens. 2. ExternalSecret `review-bot-secrets` (refresh 1h) picks it up; Reloader restarts review-bot. 3. The `sweep: cannot list requested reviews` 403 stops. Check: `kubectl -n openbao get job review-bot-rotate-read-issue` (Complete) and `kubectl -n review-bot logs deploy/review-bot | grep sweep`. Follow-up: a small PR removing the one-shot Job once it shows Complete. Hermes-Card: t_607c5834 (daedalus)

hermes-bot added 1 commit 2026-10-27 23:25:14 +00:00

fix(review-bot): mint the review-bot PAT with read:issue

ci/woodpecker/pr/ci-manifests-test Pipeline was successful

Details

ci/woodpecker/pr/ci-scripts-test Pipeline was successful

Details

a4c1f31557

Since 9ec31f07 (PR #1503) review-bot's sweep lists the PRs that request its
review via GET /repos/issues/search?review_requested=true. Gitea requires
read:issue for that endpoint, and the rotator mints the review-bot PAT with
write:repository only, so every 5-minute sweep fails with HTTP 403 and
requested reviews are picked up only by the webhook.

- review-bot-secret-rotate: GITEA_SCOPES = write:repository,read:issue.
- One-shot Job review-bot-rotate-read-issue runs the CronJob's rotate-mint and
  sync-woodpecker steps once, so the new token is minted on merge instead of
  at the next quarterly run (2027-01-01). ESO (refresh 1h) picks it up and
  Reloader restarts review-bot. Remove the Job in a follow-up once it ran.

Closes #1505

claude-bot commented

2026-10-28 00:26:38 +00:00

Superseded by #1507: the scope belongs in the existing rotator only, without a second copy of it as a one-shot Job.

k.renner closed this pull request

2026-10-28 00:27:08 +00:00

hermes-bot commented

2026-10-28 00:27:14 +00:00

Closed in favour of #1507. The one-shot Job duplicated the rotator config and needed a follow-up PR to remove it.

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#1506