fix(ci): prune old dated tags from the Gitea registry to bound its size (#336) #349

2026-07-22T07:06:10Z

claude-bot commented

2026-07-22 07:06:10 +00:00

Addresses #336 — data-gitea-0 is ~96% full and the container registry is 18.6 G of the 18.8 G used (repos are only 117 M).

Root cause

build-ci-images pushes :YYYY-MM-DD + :latest for 5 images (ci-ansible, ci-k8s, ci-tofu, ci-claude, secret-rotator) on every run and never deletes old dated tags. Gitea's nightly [cron.cleanup_packages] already GCs blobs, but frees nothing while every blob is still referenced by an un-deleted tag.

Fix (GitOps-native)

Gitea has no cleanup-rule API (rules are DB/UI-only), so the in-repo mechanism is to prune via the package REST API:

scripts/ci/prune-ci-registry.sh — for each image keep the newest KEEP=10 dated tags + :latest (and any non-date tag), DELETE the rest. Idempotent; only ever deletes YYYY-MM-DD tags beyond the newest N.
build-ci-images post-build step prune-old-tags runs it after every build, so growth is bounded at the source. The nightly cleanup_packages cron then reclaims the now-unreferenced blobs.
Reuses ci_images_token (already package: read+write, includes delete). No new secret.

Verified: sh -n clean, pipeline YAML valid, retention filter unit-tested (keeps newest N dates + :latest, deletes older dates only).

One-off backlog reclaim (separate, needs doing once)

This bounds future growth but doesn't retroactively delete the existing ~18 G backlog — and a normal build can't do it safely (it pushes GBs first, and there's only ~740 MB free). The same script run once against the live registry (delete-only, safe) reclaims it; then the nightly cron (or an admin POST /api/v1/admin/cron/cleanup_packages with OLDER_THAN lowered) frees the blobs. See the follow-up comment on #336.

🤖 Generated with Claude Code

Addresses #336 — `data-gitea-0` is ~96% full and the container registry is **18.6 G of the 18.8 G used** (repos are only 117 M). ## Root cause `build-ci-images` pushes `:YYYY-MM-DD` + `:latest` for 5 images (`ci-ansible`, `ci-k8s`, `ci-tofu`, `ci-claude`, `secret-rotator`) on every run and **never deletes old dated tags**. Gitea's nightly `[cron.cleanup_packages]` already GCs blobs, but frees nothing while every blob is still referenced by an un-deleted tag. ## Fix (GitOps-native) Gitea has **no cleanup-rule API** (rules are DB/UI-only), so the in-repo mechanism is to prune via the package REST API: - **`scripts/ci/prune-ci-registry.sh`** — for each image keep the newest `KEEP=10` dated tags + `:latest` (and any non-date tag), `DELETE` the rest. Idempotent; only ever deletes `YYYY-MM-DD` tags beyond the newest N. - **`build-ci-images` post-build step** `prune-old-tags` runs it after every build, so growth is bounded at the source. The nightly `cleanup_packages` cron then reclaims the now-unreferenced blobs. - Reuses `ci_images_token` (already `package: read+write`, includes delete). No new secret. Verified: `sh -n` clean, pipeline YAML valid, retention filter unit-tested (keeps newest N dates + `:latest`, deletes older dates only). ## One-off backlog reclaim (separate, needs doing once) This bounds *future* growth but doesn't retroactively delete the existing ~18 G backlog — and a normal build can't do it safely (it pushes GBs first, and there's only ~740 MB free). The same script run **once** against the live registry (delete-only, safe) reclaims it; then the nightly cron (or an admin `POST /api/v1/admin/cron/cleanup_packages` with `OLDER_THAN` lowered) frees the blobs. See the follow-up comment on #336. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit 2026-07-22 07:06:10 +00:00

fix(ci): prune old dated tags from the Gitea registry to bound its size (#336 )

ci/woodpecker/push/build-ci-images Pipeline is running

Details

f2c308fa5e

data-gitea-0 is ~96% full (18.6G of 18.8G is the container registry).
build-ci-images pushes a :YYYY-MM-DD tag plus :latest for five images on
every run and never deletes the old dated tags, so the registry grew
unbounded. Gitea's nightly [cron.cleanup_packages] already GCs blobs, but
it frees nothing while every blob is still referenced by an un-deleted tag.

Gitea has no cleanup-rule API (rules are DB/UI-only, not GitOps-able), so
the GitOps-native retention mechanism is to prune via the package REST API.
Add scripts/ci/prune-ci-registry.sh (keep newest KEEP=10 dated tags + :latest
per image, DELETE the rest) and run it as a post-build step in
build-ci-images. Pruning on every build bounds growth at the source; the
nightly cron then reclaims the now-unreferenced blobs. The script is
idempotent and doubles as the one-off backlog reclaim. Reuses ci_images_token
(package read+write, includes delete).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

review-bot reviewed

2026-07-22 07:25:10 +00:00

review-bot left a comment

🤖 Automated review — model claude-sonnet-5 (complexity: normal), commit f2c308fa5e19b6e170f67293b8a0c44e98211c43.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.

Summary

Adds a prune-old-tags post-build step to build-ci-images.yml and a new scripts/ci/prune-ci-registry.sh that keeps the newest KEEP=10 dated (YYYY-MM-DD) tags per CI image and deletes the rest via the Gitea package REST API, letting the nightly cleanup_packages cron reclaim the blobs. The script's core logic (regex-filter dated tags, sort -r + tail -n +$((KEEP+1)) to isolate stale entries, DELETE /packages/{owner}/container/{name}/{version}) is sound and matches the destinations/owner already used by the build steps it follows. No blocking issues found; a few robustness points below are worth a look before the one-off backlog run mentioned in the PR description.

🟡 Suggestions

scripts/ci/prune-ci-registry.sh:78-84 — the pagination loop terminates when the post-filter batch (versions matching select(.name == $n)) is empty, not when the raw API page itself is empty or a total-count is reached. If a page happens to contain zero entries for a given image while a later page still has some (e.g. depends on how q=${img} sorts/interleaves entries across the owner's five image names), the loop would stop early and under-prune that image. In practice the five image names are pushed roughly once per build so entries are usually well-interleaved, but this is exactly the kind of edge case that matters most for the one-off run against the existing ~18.6G/hundreds-of-tags backlog described in the PR body. Worth either checking Gitea's actual pagination behavior for GET /packages/{owner}?q=, or making termination robust by checking the raw (unfiltered) page length / a returned total-count instead of the filtered result.
scripts/ci/prune-ci-registry.sh:79 — the list call (api "${API}/packages/...") is not wrapped in an if, so a single transient HTTP error (network blip, auth hiccup) trips set -e and aborts the whole script immediately — including images not yet processed in the for img in $IMAGES loop. The DELETE call a few lines down (line 97) already degrades gracefully with a WARN instead of aborting; consider the same pattern for the list call so one image's listing failure doesn't also skip pruning for the remaining four.

🟢 Nits

PR description says the retention filter is "unit-tested," but the diff contains no test file, and no other script under scripts/ci/ has one either — consistent with repo convention, just noting the claim isn't verifiable from the diff alone.

Tests

No automated test is included, consistent with the rest of scripts/ci/ (no existing script there has one). The retention logic (sort + tail -n +$((KEEP+1)), dated-tag regex, :latest/non-dated tags always preserved) is simple enough that this is a reasonable trade-off, but given this script also does one-off destructive deletes against production data, a small smoke test (feed a synthetic list of dates through the same grep|sort|tail pipeline and assert the survivors) would be cheap insurance against a future edit to the filter logic.

Recommendation

Looks reasonable — no blocking issues; comments above are optional but worth checking before the one-off backlog-reclaim run.

🤖 **Automated review** — model `claude-sonnet-5` (complexity: normal), commit `f2c308fa5e19b6e170f67293b8a0c44e98211c43`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary Adds a `prune-old-tags` post-build step to `build-ci-images.yml` and a new `scripts/ci/prune-ci-registry.sh` that keeps the newest `KEEP=10` dated (`YYYY-MM-DD`) tags per CI image and deletes the rest via the Gitea package REST API, letting the nightly `cleanup_packages` cron reclaim the blobs. The script's core logic (regex-filter dated tags, `sort -r` + `tail -n +$((KEEP+1))` to isolate stale entries, `DELETE /packages/{owner}/container/{name}/{version}`) is sound and matches the destinations/owner already used by the build steps it follows. No blocking issues found; a few robustness points below are worth a look before the one-off backlog run mentioned in the PR description. ## 🟡 Suggestions - **`scripts/ci/prune-ci-registry.sh:78-84`** — the pagination loop terminates when the *post-filter* `batch` (versions matching `select(.name == $n)`) is empty, not when the raw API page itself is empty or a total-count is reached. If a page happens to contain zero entries for a given image while a later page still has some (e.g. depends on how `q=${img}` sorts/interleaves entries across the owner's five image names), the loop would stop early and under-prune that image. In practice the five image names are pushed roughly once per build so entries are usually well-interleaved, but this is exactly the kind of edge case that matters most for the one-off run against the existing ~18.6G/hundreds-of-tags backlog described in the PR body. Worth either checking Gitea's actual pagination behavior for `GET /packages/{owner}?q=`, or making termination robust by checking the raw (unfiltered) page length / a returned total-count instead of the filtered result. - **`scripts/ci/prune-ci-registry.sh:79`** — the list call (`api "${API}/packages/..."`) is not wrapped in an `if`, so a single transient HTTP error (network blip, auth hiccup) trips `set -e` and aborts the whole script immediately — including images not yet processed in the `for img in $IMAGES` loop. The `DELETE` call a few lines down (line 97) already degrades gracefully with a `WARN` instead of aborting; consider the same pattern for the list call so one image's listing failure doesn't also skip pruning for the remaining four. ## 🟢 Nits - PR description says the retention filter is "unit-tested," but the diff contains no test file, and no other script under `scripts/ci/` has one either — consistent with repo convention, just noting the claim isn't verifiable from the diff alone. ## Tests No automated test is included, consistent with the rest of `scripts/ci/` (no existing script there has one). The retention logic (sort + `tail -n +$((KEEP+1))`, dated-tag regex, `:latest`/non-dated tags always preserved) is simple enough that this is a reasonable trade-off, but given this script also does one-off destructive deletes against production data, a small smoke test (feed a synthetic list of dates through the same `grep|sort|tail` pipeline and assert the survivors) would be cheap insurance against a future edit to the filter logic. ## Recommendation Looks reasonable — no blocking issues; comments above are optional but worth checking before the one-off backlog-reclaim run.

claude-bot commented

2026-07-22 08:19:42 +00:00

Closing: this targeted tag accumulation, but the real cause of #336 is orphaned-blob GC cadence, not tag count. The registry is 8.7G of live images + 10.4G of orphaned blobs (from image rebuilds overwriting tags) that Gitea's default @midnight / OLDER_THAN=24h GC let pile up. Superseded by the cleanup_packages cadence fix (OLDER_THAN=4h, @every 6h).

claude-bot closed this pull request

2026-07-22 08:19:43 +00:00

claude-bot referenced this issue from a commit

2026-07-22 08:20:39 +00:00

fix(gitea): run package-registry GC every 6h to stop orphan blobs filling the volume (#336)

claude-bot referenced this pull request

2026-07-22 08:26:07 +00:00

fix(#336): stop the registry filling — frequent GC + stop needless CI rebuilds #350