konairius/homelab
10
0
Fork 0
Code Issues 9 Pull Requests 2 Packages Projects Releases 1 Wiki Activity

[alert] KubePodCrashLooping (review-bot) #1505

New Issue
Closed
opened 2026-10-27 23:21:26 +00:00 by hermes-bot · 1 comment
hermes-bot commented 2026-10-27 23:21:26 +00:00
Member
Copy Link
Copy Source

What is firing: KubePodCrashLooping (warning), review-bot/review-bot, pod review-bot-74b868644-ljq8p, pending ~22:34–22:48 UTC 2026-10-27 (blip, resolved by the next rollout).

Evidence

  • Loki {namespace="review-bot",pod="review-bot-74b868644-ljq8p"}: "msg":"invalid configuration","err":"REPOS is empty" (7× 22:33–22:39Z).
  • ReplicaSet diff 548c44c857 → 74b868644: REPOS env removed, image still review-bot:2026-10-27-0757. Commit 9ec31f0 "review every repository that requests a review" landed before image 2026-10-27-2231 (bumped in a0f1c3e); the old binary still required REPOS → crashloop until the bump.
  • The current pod (84746b7cf5, image 2026-10-27-2231) runs, but every 5 min sweep fails (kubectl -n review-bot logs deploy/review-bot):
    sweep: cannot list requested reviews ... GET /repos/issues/search?type=pulls&state=open&review_requested=true ... HTTP 403: token does not have at least one of required scope(s), required=[read:issue], token scope=write:repository

Root cause

  • Crashloop: verified; deployment config shipped ahead of the image that understands it (transient, recovered).
  • Still present: the new request-driven sweep needs read:issue, but the review-bot Gitea token is minted with only write:repository by the rotation CronJob (kubernetes/infrastructure/openbao/rotation/review-bot.yaml line ~115). Review requests are not picked up by the sweep.

Where to fix it

  • kubernetes/infrastructure/openbao/rotation/review-bot.yaml: add read:issue (or write:issue if it comments on PRs via the issues API) to the token scopes, then re-run the rotation so a new token is minted; deployment restart picks it up.
  • Optionally make config changes that depend on a new binary land together with/after the image bump.

Filed automatically by the Hermes alert-monitor job (2026-10-27) for the daedalus expert, whose kanban card takes it to a fix PR.

**What is firing**: KubePodCrashLooping (warning), review-bot/review-bot, pod `review-bot-74b868644-ljq8p`, pending ~22:34–22:48 UTC 2026-10-27 (blip, resolved by the next rollout). **Evidence** - Loki `{namespace="review-bot",pod="review-bot-74b868644-ljq8p"}`: `"msg":"invalid configuration","err":"REPOS is empty"` (7× 22:33–22:39Z). - ReplicaSet diff 548c44c857 → 74b868644: `REPOS` env removed, image still `review-bot:2026-10-27-0757`. Commit 9ec31f0 "review every repository that requests a review" landed before image `2026-10-27-2231` (bumped in a0f1c3e); the old binary still required REPOS → crashloop until the bump. - The current pod (`84746b7cf5`, image 2026-10-27-2231) runs, but every 5 min sweep fails (`kubectl -n review-bot logs deploy/review-bot`): `sweep: cannot list requested reviews ... GET /repos/issues/search?type=pulls&state=open&review_requested=true ... HTTP 403: token does not have at least one of required scope(s), required=[read:issue], token scope=write:repository` **Root cause** - Crashloop: verified; deployment config shipped ahead of the image that understands it (transient, recovered). - Still present: the new request-driven sweep needs `read:issue`, but the review-bot Gitea token is minted with only `write:repository` by the rotation CronJob (`kubernetes/infrastructure/openbao/rotation/review-bot.yaml` line ~115). Review requests are not picked up by the sweep. **Where to fix it** - `kubernetes/infrastructure/openbao/rotation/review-bot.yaml`: add `read:issue` (or `write:issue` if it comments on PRs via the issues API) to the token scopes, then re-run the rotation so a new token is minted; deployment restart picks it up. - Optionally make config changes that depend on a new binary land together with/after the image bump. --- <!-- alert-monitor:fingerprint=2736f5cbaa995735 --> _Filed automatically by the Hermes alert-monitor job (2026-10-27) for the `daedalus` expert, whose kanban card takes it to a fix PR._
hermes-bot added the daedalus label 2026-10-27 23:21:26 +00:00
hermes-bot commented 2026-10-27 23:21:31 +00:00
Author
Member
Copy Link
Copy Source

Hermes kanban card t_607c5834 for this issue is assigned to daedalus.

Hermes kanban card `t_607c5834` for this issue is assigned to daedalus. <!-- hermes-kanban-card:t_607c5834 -->
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels daedalus
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#1505