konairius/homelab
10
0
Fork 0
Code Issues 11 Pull Requests 1 Packages Projects Releases 1 Wiki Activity

Secret rotation is broken: 11 of 14 rotators pin image digests that no longer exist #458

New Issue
Closed
opened 2026-07-30 16:42:19 +00:00 by claude-bot · 1 comment
claude-bot commented 2026-07-30 16:42:19 +00:00
Member
Copy Link
Copy Source

Found while reviewing a rotator digest pin in #455. Most of the secret-rotation fleet is pinned to images that have been garbage-collected from the registry, and the failure will surface all at once on 2026-10-01.

Evidence

Verified by actually pulling each digest in-cluster (kubectl run), not by inspecting metadata — the registry API returns 401 for a PAT, so metadata checks are misleading here.

digest pull result rotators pinning it
sha256:159e5888… ImagePullBackOff / NotFound authentik, crowdsec-bouncer-key, paperless, shelly-manager, woodpecker, home-assistant
sha256:978fe53c… ImagePullBackOff / NotFound cloudflare, external-dns
sha256:49bfddd8… ImagePullBackOff / NotFound netbird-ingress
sha256:97c22bcf… ImagePullBackOff / NotFound pve-exporter
sha256:b41cc364… pulled + ran flux-git
sha256:828bf486… pulled + ran claude-oauth, review-bot

secret-rotator:latest currently resolves to sha256:dcf4267a…, which pulls fine and carries all modes.

The pattern is diagnostic

The two surviving digests belong to the rotators that run often. claude-oauth-secret-sync is daily and ran 12h ago. Every gone digest belongs to a quarterly rotator.

That fits registry garbage collection of untagged manifests: the build pushes :$TAG and :latest, and once a couple of rebuilds move those tags on, the older manifests become unreferenced and are collected. A digest pinned today stops resolving after a few rebuilds — so digest-pinning, adopted deliberately for immutability, is decaying into unavailability.

Impact, and why it has not bitten yet

Every quarterly CronJob shows lastScheduleTime: <none>. They were created between 2026-07-18 and 2026-07-23, i.e. after the last quarterly fire on 2026-07-01 — so none of them has ever executed.

Their first fire is 2026-10-01 04:00 Europe/Berlin, when roughly ten will fail simultaneously with ImagePullBackOff.

The failure mode is the dangerous one: a quarterly rotation that never starts looks identical to nothing happening. No alert fires for a Job that was never created successfully, and the credential it should have rotated simply does not. Every affected secret silently stops rotating while the manifests continue to assert that it does.

Not suspended, not otherwise misconfigured — spec.suspend: false across the board.

Options

  1. Repin all 11 to sha256:dcf4267a… (current :latest). Fastest; restores correctness today but re-decays after the next few rebuilds — a patch, not a fix.
  2. Stop GC-ing untagged manifests, or retain the dated tags the builds already push, so digest pins stay resolvable. Keeps immutability, which is why digests were chosen.
  3. Pin by dated tag instead of digest, matching the ci-ansible:2026-04-29 / alpine-jq:2026-07-30 convention already used elsewhere in the repo. Tags are not immutable, but they survive GC and a bump stays a reviewable diff.
  4. Repin as part of the build pipeline, so consumers are updated whenever the image is rather than drifting.

My inclination is 2 or 3 rather than 1, since 1 leaves the same trap armed for whoever pins the next rotator.

Related

  • #455 pins the new mosquitto-health rotator to sha256:dcf4267a… rather than copying the sibling pin, specifically to avoid shipping a CronJob that would fail on first run. That PR works around this issue; it does not fix it.
  • Worth checking whether anything else in the repo digest-pins images from this registry and is quietly affected the same way — secret-rotator is unlikely to be the only one.
Found while reviewing a rotator digest pin in #455. **Most of the secret-rotation fleet is pinned to images that have been garbage-collected from the registry**, and the failure will surface all at once on **2026-10-01**. ## Evidence Verified by actually pulling each digest in-cluster (`kubectl run`), not by inspecting metadata — the registry API returns 401 for a PAT, so metadata checks are misleading here. | digest | pull result | rotators pinning it | |---|---|---| | `sha256:159e5888…` | **ImagePullBackOff / NotFound** | authentik, crowdsec-bouncer-key, paperless, shelly-manager, woodpecker, home-assistant | | `sha256:978fe53c…` | **ImagePullBackOff / NotFound** | cloudflare, external-dns | | `sha256:49bfddd8…` | **ImagePullBackOff / NotFound** | netbird-ingress | | `sha256:97c22bcf…` | **ImagePullBackOff / NotFound** | pve-exporter | | `sha256:b41cc364…` | pulled + ran | flux-git | | `sha256:828bf486…` | pulled + ran | claude-oauth, review-bot | `secret-rotator:latest` currently resolves to `sha256:dcf4267a…`, which pulls fine and carries all modes. ## The pattern is diagnostic **The two surviving digests belong to the rotators that run often.** `claude-oauth-secret-sync` is daily and ran 12h ago. Every *gone* digest belongs to a quarterly rotator. That fits registry garbage collection of untagged manifests: the build pushes `:$TAG` and `:latest`, and once a couple of rebuilds move those tags on, the older manifests become unreferenced and are collected. **A digest pinned today stops resolving after a few rebuilds** — so digest-pinning, adopted deliberately for immutability, is decaying into unavailability. ## Impact, and why it has not bitten yet Every quarterly CronJob shows `lastScheduleTime: <none>`. They were created between **2026-07-18 and 2026-07-23**, i.e. after the last quarterly fire on 2026-07-01 — so **none of them has ever executed**. Their first fire is **2026-10-01 04:00 Europe/Berlin**, when roughly ten will fail simultaneously with `ImagePullBackOff`. The failure mode is the dangerous one: a quarterly rotation that never starts looks identical to nothing happening. No alert fires for a Job that was never created successfully, and the credential it should have rotated simply does not. Every affected secret silently stops rotating while the manifests continue to assert that it does. Not suspended, not otherwise misconfigured — `spec.suspend: false` across the board. ## Options 1. **Repin all 11 to `sha256:dcf4267a…`** (current `:latest`). Fastest; restores correctness today but re-decays after the next few rebuilds — a patch, not a fix. 2. **Stop GC-ing untagged manifests**, or retain the dated tags the builds already push, so digest pins stay resolvable. Keeps immutability, which is why digests were chosen. 3. **Pin by dated tag instead of digest**, matching the `ci-ansible:2026-04-29` / `alpine-jq:2026-07-30` convention already used elsewhere in the repo. Tags are not immutable, but they survive GC and a bump stays a reviewable diff. 4. **Repin as part of the build pipeline**, so consumers are updated whenever the image is rather than drifting. My inclination is 2 or 3 rather than 1, since 1 leaves the same trap armed for whoever pins the next rotator. ## Related - #455 pins the *new* mosquitto-health rotator to `sha256:dcf4267a…` rather than copying the sibling pin, specifically to avoid shipping a CronJob that would fail on first run. That PR works around this issue; it does not fix it. - Worth checking whether anything else in the repo digest-pins images from this registry and is quietly affected the same way — `secret-rotator` is unlikely to be the only one.
claude-bot commented 2026-07-30 16:48:28 +00:00
Author
Member
Copy Link
Copy Source

Owner directive settles the approach: our own images do not need digest pinning, and :latest is never to be used. So this is option 3 — dated tags — not a menu.

That also matches the convention already in the repo (ci-ansible:2026-04-29) and the images added today (alpine-jq:2026-07-30, cni-plugins:2026-07-30, alpine-openssl:2026-07-30).

Concrete remediation

Repin all 11 to a dated tag. Verified these exist and pull:

secret-rotator:2026-07-30  -> sha256:dcf4267a…  (current; == :latest)
secret-rotator:2026-07-23  -> sha256:828bf486…
secret-rotator:2026-07-21  -> pulls

2026-07-30 is the one to use. Note it is today, not 2026-07-23 (the last commit touching secret-rotator/) — because the build step also triggers on .woodpecker/build-ci-images.yml, which was edited several times today while adding the alpine-jq / cni-plugins / alpine-openssl images. So secret-rotator was rebuilt and republished from unchanged source.

That incidental rebuild is itself part of the mechanism: every unrelated pipeline edit moves :latest and orphans the previous manifest, which is then garbage-collected. Digest pins therefore decay on a cadence driven by edits to a shared file that has nothing to do with the pinned image. That is why option 1 (repin to the current digest) would have been a trap — it would decay again the next time someone adds an unrelated image.

Affected files

rotation/authentik.yaml             rotation/crowdsec-bouncer-key.yaml
rotation/paperless.yaml             rotation/shelly-manager.yaml
rotation/woodpecker.yaml            rotation/home-assistant.yaml   (3 CronJobs)
rotation/cloudflare.yaml            rotation/external-dns.yaml
rotation/netbird-ingress.yaml       rotation/pve-exporter.yaml

rotation/flux-git.yaml (b41cc364) and rotation/claude-oauth.yaml + rotation/review-bot.yaml (828bf486) currently still resolve, but they are pinned the same way and will decay identically — worth converting in the same pass rather than waiting for them to break too.

Already done

#455 now pins secret-rotator:2026-07-30, verified to pull, to list random among its modes, and to carry the RANDOM_BYTES>=16 floor. It is out of this trap rather than joining it.

**Owner directive settles the approach**: our own images do not need digest pinning, and `:latest` is never to be used. So this is option 3 — **dated tags** — not a menu. That also matches the convention already in the repo (`ci-ansible:2026-04-29`) and the images added today (`alpine-jq:2026-07-30`, `cni-plugins:2026-07-30`, `alpine-openssl:2026-07-30`). ### Concrete remediation Repin all 11 to a dated tag. Verified these exist and pull: ``` secret-rotator:2026-07-30 -> sha256:dcf4267a… (current; == :latest) secret-rotator:2026-07-23 -> sha256:828bf486… secret-rotator:2026-07-21 -> pulls ``` `2026-07-30` is the one to use. Note it is **today**, not 2026-07-23 (the last commit touching `secret-rotator/`) — because the build step also triggers on `.woodpecker/build-ci-images.yml`, which was edited several times today while adding the alpine-jq / cni-plugins / alpine-openssl images. So secret-rotator was rebuilt and republished from unchanged source. That incidental rebuild is itself part of the mechanism: **every unrelated pipeline edit moves `:latest` and orphans the previous manifest**, which is then garbage-collected. Digest pins therefore decay on a cadence driven by edits to a shared file that has nothing to do with the pinned image. That is why option 1 (repin to the current digest) would have been a trap — it would decay again the next time someone adds an unrelated image. ### Affected files ``` rotation/authentik.yaml rotation/crowdsec-bouncer-key.yaml rotation/paperless.yaml rotation/shelly-manager.yaml rotation/woodpecker.yaml rotation/home-assistant.yaml (3 CronJobs) rotation/cloudflare.yaml rotation/external-dns.yaml rotation/netbird-ingress.yaml rotation/pve-exporter.yaml ``` `rotation/flux-git.yaml` (`b41cc364`) and `rotation/claude-oauth.yaml` + `rotation/review-bot.yaml` (`828bf486`) currently still resolve, but they are pinned the same way and will decay identically — worth converting in the same pass rather than waiting for them to break too. ### Already done #455 now pins `secret-rotator:2026-07-30`, verified to pull, to list `random` among its modes, and to carry the `RANDOM_BYTES>=16` floor. It is out of this trap rather than joining it.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#458