chore(ci): make ci-claude reproducible and keep it on latest (#351)

The ci-claude image installed a fixed, pinned Claude Code version and every
consumer pipeline pinned a stale dated image tag. Two problems:

- The npm install layer changed on every build (install-time mtimes + the
  npm cache), so it never deduped in the Gitea registry — the single largest
  orphan pattern behind #336 (~260 MB orphaned per no-op rebuild).
- A frozen version silently rots: old Claude Code releases eventually stop
  working against the API.

Containerfile: default to @latest (npm auto-selects the musl native binary on
this alpine base) and make the install layer byte-stable — clean the npm cache
and normalize install-time mtimes on the installed tree and the directories the
RUN touches. Two independent --no-cache builds now produce a byte-identical npm
layer, so a rebuild of the same version dedups instead of orphaning a copy.

New ci-claude-refresh.yml owns the image lifecycle: weekly (and on-demand) it
resolves the latest published version, and only when it differs from what the
consumers pin does it build ci-claude:<version> (+:latest) and open a PR
rewiring the 5 consumer pipelines. Version tags make "did it change?" a
race-free string compare and keep the cron from opening no-op PRs. Nothing
auto-merges. build-ci-images.yml no longer builds ci-claude so the two never
fight over tags.

Requires owner setup before it does anything (documented in the pipeline
header): a dedicated ci-claude-bump-bot identity (write:repository only) as the
ci_claude_bump_bot_token secret, and a cron named "ci-claude-refresh".

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

.woodpecker/build-ci-images.yml

@@ -100,29 +100,10 @@ steps:
           --destination gitea.renner.uno/gitea_admin/ci-tofu:$TAG \
           --destination gitea.renner.uno/gitea_admin/ci-tofu:latest
 
-  - name: build-claude
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
-    directory: ci/images/claude
-    depends_on: []
-    when:
-      - event: manual
-      - event: push
-        path: ["ci/images/claude/**", ".woodpecker/build-ci-images.yml"]
-    environment:
-      GITEA_USER: gitea_admin
-      GITEA_TOKEN:
-        from_secret: ci_images_token
-    commands:
-      - mkdir -p /kaniko/.docker
-      - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
-      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
-      - TAG=$(date +%Y-%m-%d)
-      - |
-        /kaniko/executor \
-          --context . \
-          --dockerfile Containerfile \
-          --destination gitea.renner.uno/gitea_admin/ci-claude:$TAG \
-          --destination gitea.renner.uno/gitea_admin/ci-claude:latest
+  # NOTE: ci-claude is deliberately NOT built here. It tracks upstream Claude
+  # Code releases on its own cadence and is built + version-tagged by
+  # ci-claude-refresh.yml, which also opens the PR that rolls a new version out
+  # to the consumer pipelines. Building it here too would fight over tags.
 
   # Runtime image, not CI tooling: a static musl binary in a FROM-scratch image,
   # used as the base for secret-rotation CronJobs. Replaces per-job inline Python.

.woodpecker/ci-claude-refresh.yml

@@ -0,0 +1,203 @@
+# ci-claude refresh — keep the CI agent image on the LATEST Claude Code and
+# roll that version out to every pipeline that runs it, automatically.
+#
+# Why this exists (#351, follow-up from #336): the ci-claude image used to pin
+# a fixed Claude Code version in its Containerfile, and every consumer pipeline
+# pinned a stale dated image tag. Old Claude Code releases eventually stop
+# working against the API, so both pins silently rot the agent pipelines. This
+# pipeline owns the whole lifecycle instead:
+#
+#   1. decide   — resolve the latest published Claude Code version and compare
+#                 it to the tag the consumer pipelines currently reference.
+#   2. build    — (only when a rebuild is warranted) kaniko-build the image
+#                 pinned to that exact version and push  ci-claude:<version>
+#                 and  ci-claude:latest.  The Containerfile makes this layer
+#                 byte-reproducible, so rebuilding an unchanged version dedups
+#                 in the registry instead of orphaning a fresh ~260 MB copy.
+#   3. bump-pr  — (only when the version actually changed) rewrite the 5
+#                 consumer pins to  ci-claude:<version>  and open a PR. A human
+#                 reviews and merges it; nothing auto-merges.
+#
+# TAG SCHEME — deliberately version tags (ci-claude:2.1.218), NOT the dated
+# tags the other ci-* images use. The tag encodes the thing that changes here
+# (the Claude version), which makes "did it change?" a trivial, race-free
+# string compare and keeps the weekly cron from opening no-op PRs. The other
+# images change only when their Containerfile changes, so dated tags fit them;
+# this one tracks an upstream release cadence.
+#
+# This pipeline OWNS building ci-claude: the image's build step was removed
+# from build-ci-images.yml so the two never fight over tags.
+#
+# TOKENS — two dedicated, least-privilege identities, per the #330 rule that
+# every pipeline gets its own scoped bot and never the admin gitea_token:
+#   - ci_images_token      : package read+write ONLY (the kaniko push). Shared
+#                            with build-ci-images.yml — same job, same scope.
+#   - ci_claude_bump_bot_token : write:repository ONLY, owned by a dedicated
+#                            ci-claude-bump-bot identity. It pushes a branch and
+#                            opens a PR; it does NOT comment on issues, so it
+#                            needs no write:issue. It must be a write
+#                            collaborator on k.renner/homelab.
+#
+# OWNER SETUP (one-time — this pipeline is inert until all three are done):
+#   1. Create the ci-claude-bump-bot Gitea user, add it as a write
+#      collaborator on k.renner/homelab, mint a PAT scoped write:repository
+#      only, and store it as the Woodpecker secret  ci_claude_bump_bot_token.
+#   2. Confirm the existing  ci_images_token  secret is available to this repo.
+#   3. Register the cron (weekly is plenty; Claude Code ships often but the
+#      pipelines tolerate being a few days behind):
+#        woodpecker repo cron add --repo k.renner/homelab \
+#          --name ci-claude-refresh --branch master --schedule '@weekly'
+#
+# Triggering:
+#   - cron   : the cron NAMED exactly "ci-claude-refresh" (see above).
+#   - manual : woodpecker pipeline create --branch master \
+#                --var PIPELINE=ci-claude-refresh k.renner/homelab
+#   - push   : a change to the Containerfile (or this file) forces a rebuild of
+#              the current latest version so Containerfile fixes ship promptly.
+when:
+  - event: cron
+    cron: ci-claude-refresh
+  - event: manual
+    evaluate: 'PIPELINE == "ci-claude-refresh"'
+  - event: push
+    path:
+      - "ci/images/claude/**"
+      - ".woodpecker/ci-claude-refresh.yml"
+
+steps:
+  # 1. Resolve latest vs. currently-pinned, and decide what to do. Writes
+  #    sentinels to $CI_WORKSPACE/.ci-refresh/ (the workspace volume is the
+  #    only path shared across steps; /tmp is per-container). Uses the ci-claude
+  #    image purely as a toolbox (it ships npm, jq, git, curl).
+  - name: decide
+    image: gitea.renner.uno/gitea_admin/ci-claude:latest
+    commands:
+      - |
+        set -eu
+        mkdir -p "$$CI_WORKSPACE/.ci-refresh"
+        # Files that pin the image. Keep this list in sync with the consumers.
+        CONSUMERS="
+          .woodpecker/security-scan.yml
+          .woodpecker/alert-monitor.yml
+          .woodpecker/update-audit.yml
+          .woodpecker/pr-review.yml
+          .woodpecker/issue-fix.yml
+        "
+        NEW=$$(npm view @anthropic-ai/claude-code version)
+        case "$$NEW" in
+          [0-9]*.[0-9]*.[0-9]*) : ;;
+          *) echo "npm returned an implausible version '$$NEW' — aborting."; exit 1 ;;
+        esac
+        # The tag every consumer currently references. They are kept identical,
+        # so this must resolve to EXACTLY ONE non-empty tag; anything else
+        # (no match, or a partial/disagreeing set) aborts rather than risk a
+        # corrupting sed later. Counting with `grep -c .` (non-empty lines)
+        # instead of `wc -l` is deliberate: an empty result yields 0 here but
+        # 1 under wc -l, which would slip past the guard.
+        CUR=$$(grep -hoE 'ci-claude:[0-9A-Za-z._-]+' $$CONSUMERS | sed 's/^ci-claude://' | sort -u)
+        NTAGS=$$(printf '%s' "$$CUR" | grep -c . || true)
+        if [ -z "$$CUR" ] || [ "$$NTAGS" -ne 1 ]; then
+          echo "Expected exactly one ci-claude tag across the consumers, got [$$CUR]."
+          echo "Refusing to guess — fix the pins by hand."; exit 1
+        fi
+        echo "latest=$$NEW  current=$$CUR  event=$$CI_PIPELINE_EVENT"
+        printf '%s' "$$NEW" > "$$CI_WORKSPACE/.ci-refresh/version"
+        printf '%s' "$$CUR" > "$$CI_WORKSPACE/.ci-refresh/current"
+        # Build when the version changed, or on a push (Containerfile edit must
+        # ship even if the version is unchanged). Bump the pins only when the
+        # version the consumers reference is not the latest.
+        if [ "$$CUR" != "$$NEW" ] || [ "$$CI_PIPELINE_EVENT" = "push" ]; then
+          touch "$$CI_WORKSPACE/.ci-refresh/do-build"; echo "will build $$NEW"
+        else
+          echo "already on $$NEW — nothing to build"
+        fi
+        if [ "$$CUR" != "$$NEW" ]; then
+          touch "$$CI_WORKSPACE/.ci-refresh/do-bump"; echo "will open bump PR $$CUR -> $$NEW"
+        else
+          echo "consumers already pin $$NEW — no PR"
+        fi
+
+  # 2. Build the exact resolved version and push version + latest tags. Pinning
+  #    --build-arg to the resolved version (not "latest") guarantees the tag and
+  #    the image content agree even if a new release lands mid-run.
+  - name: build
+    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    directory: ci/images/claude
+    depends_on: [decide]
+    environment:
+      GITEA_USER: gitea_admin
+      GITEA_TOKEN:
+        from_secret: ci_images_token
+    commands:
+      - |
+        [ -f "$$CI_WORKSPACE/.ci-refresh/do-build" ] || { echo "no build needed"; exit 0; }
+        set -eu
+        VERSION=$$(cat "$$CI_WORKSPACE/.ci-refresh/version")
+        mkdir -p /kaniko/.docker
+        AUTH=$$(printf '%s:%s' "$$GITEA_USER" "$$GITEA_TOKEN" | base64 | tr -d '\n')
+        printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$$AUTH" > /kaniko/.docker/config.json
+        /kaniko/executor \
+          --context . \
+          --dockerfile Containerfile \
+          --build-arg CLAUDE_CODE_VERSION="$$VERSION" \
+          --destination gitea.renner.uno/gitea_admin/ci-claude:"$$VERSION" \
+          --destination gitea.renner.uno/gitea_admin/ci-claude:latest
+
+  # 3. Rewrite the consumer pins and open a PR. Never auto-merges. Skips itself
+  #    when the version was unchanged, and when a bump branch already exists (a
+  #    prior run's PR is still open / being merged).
+  - name: bump-pr
+    image: gitea.renner.uno/gitea_admin/ci-claude:latest
+    depends_on: [build]
+    environment:
+      PUSH_TOKEN:
+        from_secret: ci_claude_bump_bot_token
+    commands:
+      - |
+        [ -f "$$CI_WORKSPACE/.ci-refresh/do-bump" ] || { echo "no version change — no PR"; exit 0; }
+        set -eu
+        NEW=$$(cat "$$CI_WORKSPACE/.ci-refresh/version")
+        CUR=$$(cat "$$CI_WORKSPACE/.ci-refresh/current")
+        # Both were validated non-empty by the decide step; re-assert before a
+        # sed that would corrupt every pin if either were blank.
+        [ -n "$$NEW" ] && [ -n "$$CUR" ] || { echo "empty version/current sentinel — aborting"; exit 1; }
+        API="https://gitea.renner.uno/api/v1/repos/k.renner/homelab"
+        BRANCH="ci-image/claude-$$NEW"
+        if git ls-remote --heads "https://gitea.renner.uno/k.renner/homelab.git" "$$BRANCH" | grep -q .; then
+          echo "Branch $$BRANCH already exists — a PR for $$NEW is already open. Nothing to do."
+          exit 0
+        fi
+        CONSUMERS="
+          .woodpecker/security-scan.yml
+          .woodpecker/alert-monitor.yml
+          .woodpecker/update-audit.yml
+          .woodpecker/pr-review.yml
+          .woodpecker/issue-fix.yml
+        "
+        # Exact-string swap of the current tag for the new one, consumers only.
+        for f in $$CONSUMERS; do
+          sed -i "s#ci-claude:$$CUR#ci-claude:$$NEW#g" "$$f"
+        done
+        git config user.email "ci-claude-bump-bot@renner.uno"
+        git config user.name "CI Claude Bump"
+        git checkout -b "$$BRANCH"
+        git add $$CONSUMERS
+        git commit -m "chore(ci): bump ci-claude image to $$NEW"
+        git push "https://$$PUSH_TOKEN@gitea.renner.uno/k.renner/homelab.git" HEAD:"$$BRANCH"
+        {
+          printf 'Automated bump of the ci-claude agent image from `%s` to `%s`.\n\n' "$$CUR" "$$NEW"
+          printf 'The refresh pipeline built and pushed `ci-claude:%s` (and `:latest`) from the\n' "$$NEW"
+          printf 'latest published `@anthropic-ai/claude-code`. This rewires the 5 pipelines that\n'
+          printf 'run the image so CI stops running a stale Claude Code.\n\n'
+          printf 'Merging is a human decision — review, then merge to roll it out.\n'
+        } > /tmp/pr-body.md
+        jq -n --arg title "chore(ci): bump ci-claude image to $$NEW" \
+              --rawfile body /tmp/pr-body.md \
+              --arg head "$$BRANCH" \
+          '{title: $$title, body: $$body, head: $$head, base: "master"}' > /tmp/pr.json
+        code=$$(curl -s -o /tmp/resp.json -w '%{http_code}' \
+          -X POST -H "Authorization: token $$PUSH_TOKEN" \
+          -H "Content-Type: application/json" --data @/tmp/pr.json "$$API/pulls")
+        echo "POST pull -> HTTP $$code"
+        [ "$$code" -ge 200 ] && [ "$$code" -lt 300 ] || { cat /tmp/resp.json; exit 1; }
+        jq '{number, html_url, title}' /tmp/resp.json

ci/images/claude/Containerfile

@@ -1,6 +1,10 @@
 FROM node:22-alpine@sha256:16e22a550f3863206a3f701448c45f7912c6896a62de43add43bb9c86130c3e2
 
-ARG CLAUDE_CODE_VERSION=2.1.209
+# Default to the latest Claude Code. Pinning a fixed version is the wrong
+# default here: old releases eventually stop working against the API, so a
+# stale pin silently rots the agent pipelines. Override only to dodge a known
+# broken release, e.g.  --build-arg CLAUDE_CODE_VERSION=2.1.217
+ARG CLAUDE_CODE_VERSION=latest
 
 RUN apk add --no-cache \
       bash \
@@ -11,5 +15,32 @@ RUN apk add --no-cache \
       openssh-client \
       openssl
 
+# Install Claude Code, then make the resulting layer byte-stable so that a
+# rebuild of the SAME version DEDUPS in the Gitea registry instead of orphaning
+# a fresh ~260 MB copy on every build (#351, follow-up from #336). npm picks the
+# musl-linked native binary automatically on this alpine base.
+#
+# Two sources of per-build churn are eliminated inside this one layer:
+#   1. the npm cache under /root/.npm — its index/log files carry build-time
+#      content and timestamps, and it never belongs in the image anyway.
+#   2. install-time mtimes on every installed file and on the directories this
+#      RUN modified — normalized to a fixed date. Without this the tar bytes
+#      (and thus the layer digest) differ each build even when the version is
+#      unchanged, defeating registry dedup.
+# Content itself is deterministic per version: the registry tarballs are
+# immutable, so same version in -> same bytes out once timestamps are pinned.
 RUN npm install -g @anthropic-ai/claude-code@${CLAUDE_CODE_VERSION} \
- && claude --version
+ && claude --version \
+ && npm cache clean --force \
+ && rm -rf /root/.npm /tmp/* \
+ && find /usr/local/lib/node_modules/@anthropic-ai -exec touch -h -t 202001010000.00 {} + \
+ && touch -h -t 202001010000.00 \
+      /usr/local/bin/claude \
+      /usr/local/lib/node_modules/@anthropic-ai \
+      /usr/local/lib/node_modules \
+      /usr/local/lib \
+      /usr/local/bin \
+      /usr/local \
+      /usr \
+      /root \
+      /tmp