The ci-claude image installed a fixed, pinned Claude Code version and every
consumer pipeline pinned a stale dated image tag. Two problems:
- The npm install layer changed on every build (install-time mtimes + the
npm cache), so it never deduped in the Gitea registry — the single largest
orphan pattern behind #336 (~260 MB orphaned per no-op rebuild).
- A frozen version silently rots: old Claude Code releases eventually stop
working against the API.
Containerfile: default to @latest (npm auto-selects the musl native binary on
this alpine base) and make the install layer byte-stable — clean the npm cache
and normalize install-time mtimes on the installed tree and the directories the
RUN touches. Two independent --no-cache builds now produce a byte-identical npm
layer, so a rebuild of the same version dedups instead of orphaning a copy.
New ci-claude-refresh.yml owns the image lifecycle: weekly (and on-demand) it
resolves the latest published version, and only when it differs from what the
consumers pin does it build ci-claude:<version> (+:latest) and open a PR
rewiring the 5 consumer pipelines. Version tags make "did it change?" a
race-free string compare and keep the cron from opening no-op PRs. Nothing
auto-merges. build-ci-images.yml no longer builds ci-claude so the two never
fight over tags.
Requires owner setup before it does anything (documented in the pipeline
header): a dedicated ci-claude-bump-bot identity (write:repository only) as the
ci_claude_bump_bot_token secret, and a cron named "ci-claude-refresh".
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The triage + review agents run with REVIEW_TOKEN and CLAUDE_CODE_OAUTH_TOKEN in
their environment, process attacker-controlled PR title/body/diff, and their
output is posted verbatim as a public PR comment. Without a Read sandbox a
prompt-injected diff could read /proc/self/environ (the tokens) or .git/config
(Woodpecker's clone credential) and get it published. Every other agent
pipeline already denies this via --settings; pr-review had none.
Add ci/agents/pr-review/settings.json (mirrors issue-fix/update-audit, gitea
#197) denying Read on /proc,/root,/home,/etc,.git, and pass --settings
$PROMPTS/settings.json to both claude invocations (resolves alongside the
prompts, so it works for the repos that fetch prompts from the homelab clone).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On a pull_request event Woodpecker's built-in clone already checks the repo out
at the PR head in the workspace, so the agents read that directly for context.
Restores full-context review (the prior commit over-corrected to diff-only)
while keeping the manual, token-bearing clone gone. Prompts point at the
working-directory checkout instead of ./work/target.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The clone existed only to hand review_bot_token to git, which persisted it in
work/target/.git/config — a file the review agent (Read/Glob/Grep, output
posted publicly) could read. Nothing needed the checkout that the API-fetched
diff + metadata don't already provide, so drop the clone entirely: no token on
disk, no work/target, no cleanup. The PR + diff are still read over the API
with ephemeral curl Authorization headers (never written to disk).
Prompts updated to match: the agents review the diff alone (no ./work/target),
and are told to flag, not guess, when a judgement needs code they can't see.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Both agent steps hold Gitea write credentials in their environment
(issue-fix holds a push token too), but neither passed --settings, so
their Read tool was unrestricted — unlike update-audit, which got a
Read sandbox for exactly this reason (gitea issue #197).
That left a complete exfil path: Read /proc/self/environ yields the
token, Write puts it in the agent's output file, and the pipeline posts
that output verbatim as a Gitea issue body / PR body on a PUBLIC repo.
Add the same deny-list used by update-audit (/proc, /root, /home, /etc,
.git) to both agents and wire --settings into each invocation. The repo
checkout, .ci-alert-monitor/ and /tmp/alert-fix/ stay readable, which is
all either agent needs.
This is defense-in-depth only. The root cause — CI pipelines holding an
admin-owned token at all — is tracked separately.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New Woodpecker cron pipeline that mirrors the alert-monitor / issue-fix
fleet pattern (cron -> headless `claude` -> Gitea API), but reviews open
pull requests instead of triaging alerts:
- .woodpecker/pr-review.yml: hourly `pr-review` cron. Discovers open PRs
across k.renner/{homelab,shelly-manager,authentik-samba-bridge,
homeassistant}, skips drafts/WIP and PRs already reviewed at their
current head sha (dedup marker), caps at 5/run. For each: clones the PR
head, has a cheap Haiku triage agent pick the model by complexity
(haiku/sonnet/opus; security-sensitive paths force opus), runs the
review with that model, and posts a detailed report as a Gitea PR
review (COMMENT, or REQUEST_CHANGES on a real defect — never APPROVE).
- ci/agents/pr-review/triage-prompt.md: model-selection rubric.
- ci/agents/pr-review/prompt.md: the reviewer's task + report format.
Runs under a dedicated `review-bot` Gitea account (secret
review_bot_token) so reviews are visibly bot-authored and independently
revocable. Static review only — no cluster access; agent tools are
Read/Glob/Grep/Write with no Bash.
Not yet active: needs the review-bot account + token, the review_bot_token
Woodpecker secret, and the `pr-review` cron registered (see file header).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The live 15-min snapshot structurally misses anything that fires and
resolves between ticks: on 2026-07-16 a 17-minute KubeHpaMaxedOut
(authentik) and last evening's four-alert home-assistant incident were
invisible to the monitor (owner noticed the blips; monitor did not).
The fetch step now also queries Prometheus's ALERTS history over a
35-minute window (~2 ticks + jitter). The dedup step synthesizes
pseudo-alerts for series not currently firing at all — checked against
the PRE-filter snapshot so live-but-null-routed alerts don't reappear
as blips — with a stable sha256-derived fingerprint so one noise
verdict suppresses a flappy alert permanently. Blips then flow through
the same title/noise dedup and triage as live alerts; the prompt
explains their semantics (self-resolved; judge the underlying cause;
null-routed per the repo route config = noise).
Verified against live data: currently-firing KubeCPUOvercommit
excluded, injected KubeHpaMaxedOut blip synthesized correctly;
sha256sum confirmed present in ci-claude; lookback query tested via
the service proxy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
The #259 lesson, both directions: the agent rightly refused
unauthenticated instructions embedded in the issue body — and then had
no channel for the owner's authentic reply 18 minutes later, so a
human ended up doing the sweep the agent should have owned.
- Pick step: an issue with an alert-fix:assessed marker becomes
eligible again when a TRUSTED_AUTHORS comment is NEWER than the
latest marker (timestamps compared in jq — busybox ash test has no
string-<). Owner replies on the issue now drive the loop.
- Trusted comments are appended to the agent's issue.md as 'Verified
owner guidance'. Trust anchor is the hardcoded author list in this
file — deliberately not the Gitea permission API, because the
pipeline itself comments as gitea_admin and agent-authored text must
never be able to masquerade as owner guidance.
- Prompt: verified guidance is authoritative for intent and scope (may
lift the surgical ceiling when it calls for a sweep), does not lift
the edit-path allowlist, and factual mismatches with live state are
surfaced, not silently resolved in either direction.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
Both wrong assessments (#249 openbao 'sealed', #250 three untested
hypotheses while one 'kubectl logs' showed the 401) share one root
cause: the agents diagnose blind, so they narrate plausible fiction.
Labeling unverified claims (previous commit) treats the symptom; this
gives them the means to verify:
- kubernetes/apps/monitoring/ci-diagnostics-rbac.yaml: ServiceAccount
ci-diagnostics with a read-only ClusterRole (workloads, events,
pods/log, Flux/ESO/cert-manager/monitoring/ceph CRDs — NO secrets)
plus a namespaced services/proxy Role for the Prometheus and
Alertmanager HTTP APIs.
- scripts/ci/make-diag-kubeconfig.sh: mints a 1h token per run from
the admin kubeconfig (no new long-lived secret), builds
.ci-diag/kubeconfig, copies the static kubectl binary for the
jq-only ci-claude image, smoke-tests with the restricted identity,
and degrades to snapshot-only mode when the SA isn't reconciled.
- alert-monitor.yml / issue-fix.yml: agents get exactly one extra tool,
Bash(./.ci-diag/kubectl:*) — prefix match forbids pipes and any other
binary; RBAC forbids mutation and secret reads.
- prompts: verification is now mandatory, not aspirational — 'a
hypothesis you could have settled with one read-only command but
didn't is a defect in your output'; fix agent must test the issue's
claims before building on them; minimal-quote rule against dumping
logs into public issues.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
The first issue-fix run on #249 asserted the openbao pod was '0/1 and
sealed' — it was Ready and unsealed the whole time; the alert fired
because the chart's updateStrategy: OnDelete holds the 2.5.5→2.6.0
image bump until an operator deletes the pod. The agent has no cluster
access and invented runtime state from the alert's generic annotation
text.
- issue-fix prompt: runtime claims must be attributed or framed as
hypothesis, with read-only verification commands in the summary.
- alert-monitor prompt: when context.txt does not confirm the alert's
symptom, say so explicitly — the mismatch is often the real clue.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
Two new cron pipelines closing the loop from firing alert to fix PR:
- alert-monitor (*/15 min): reads active alerts from Alertmanager via the
apiserver service proxy (woodpecker-ci is netpol-blocked from monitoring
ClusterIPs; the Authentik gate blocks the public hostname), dedups
against open issues and a persistent noise-log issue, then a headless
Claude agent triages the new ones against the repo plus a cluster
snapshot. Confirmed problems become Gitea issues; noise is recorded in
the noise log so it is never re-triaged.
- issue-fix (hourly): picks the oldest open '[alert] ...' issue with no
fix branch and no assessed-marker comment, hands it to a fix agent
(same tool sandbox as security-fix), pushes alert-fix/issue-<n> and
opens a PR with 'Closes #<n>'. Not-repo-fixable verdicts are posted as
an issue comment instead.
- e2e.yml: scope the bare 'event: cron' to the three existing cron names
— otherwise the 15-minute cron would piggyback a full e2e run every 15
minutes.
The ci-k8s image has no jq, so the alert fetch step only talks to the
cluster; all JSON filtering/dedup/gating lives in the ci-claude step.
Cron registration (after this lands on master):
woodpecker repo cron add --repo gitea_admin/homelab \
--name alert-monitor --branch master --schedule '*/15 * * * *'
woodpecker repo cron add --repo gitea_admin/homelab \
--name issue-fix --branch master --schedule '41 * * * *'
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
Closes gitea issue #197. The update-audit agent fetches untrusted
third-party release pages via WebFetch/WebSearch and its Write output
is posted verbatim to a public Gitea issue, so injected content in a
fetched page could try to redirect the agent's behavior. Harden the
tool surface (verified empirically against the pinned Claude Code CLI):
- Drop WebSearch: it cannot be domain-scoped like WebFetch, and the
prompt already gives explicit source URLs or a well-known registry
for everything it needs to look up.
- Add ci/agents/update-audit/settings.json (loaded via --settings) that
denies Read on /proc, /root, /home, /etc, and the repo's own .git/ —
closing the credential-file and env-var-dump read paths a
prompt-injected instruction could otherwise try. Verified this deny
mechanism actually blocks reads, unlike Write path-scoping, which
was tested and found NOT enforced under --permission-mode acceptEdits
(documented in the pipeline comment; a full WebFetch domain allowlist
was also considered and rejected, since Helm chart lookups legitimately
hit whichever domain each HelmRelease's own trusted source URL points
at).
- prompt.md now tells the agent fetched content is untrusted data, not
instructions, and to never echo it verbatim into the report.
Not fixed here: GITEA_TOKEN is still the broad registry/release-writing
token (no write:issue-only token exists yet) — that requires an
operator to mint one; tracked via a comment on #197 and a code comment
at the secret binding.
Addresses #209. All four ci/images/*/Containerfile FROM lines were
tag-pinned only (alpine:3.21, node:22-alpine), letting the base silently
change between builds. Pin each to its current manifest-list digest.
ci/images/k8s/Containerfile downloaded talosctl, kubectl, and velero via
curl with no integrity check, and installed flux via `curl | bash`
(itself an unpinned, unverified fetch-and-execute). Replace the flux
install with a pinned-version tarball download verified against its
published sha256 checksum, same as talosctl/kubectl/velero now are.
kubectl's version was also resolved dynamically from dl.k8s.io's
stable.txt at build time instead of being pinned; add KUBECTL_VERSION.
ci/images/tofu/Containerfile's tofu.zip download gets the same
sha256 verification.
ci/images/claude/Containerfile's CLAUDE_CODE_VERSION defaulted to
`latest` and was never overridden by build-ci-images.yml, so the image
holding repo/issue-write tokens floated to whatever npm resolved at
build time. Pin it to the current release (2.1.209).
Checksums and digests verified against the respective registries/release
APIs at fix time (2026-07-14).
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
rebuild-k8s.yml was the one-shot PVE->K8s migration/rebuild pipeline; it
is no longer needed and still pointed at the retired Garage S3 on the
now-off nas-1. Remove it (DR restore is via Velero + flux bootstrap).
The live cluster-access path is kept and migrated off Garage onto Hetzner
Object Storage (renner-uno-k8s-secrets, fsn1.your-objectstorage.com,
region fsn1, hetzner_s3_* keypair):
- scripts/ci/fetch-kubeconfig.sh: Garage -> Hetzner; env vars renamed
GARAGE_SECRETS_* -> S3_ACCESS_KEY_ID/S3_SECRET_ACCESS_KEY.
- .woodpecker/e2e.yml, e2e-test.yml: pass hetzner_s3_access_key/secret_key
(the live cron e2e pipeline needs this — it was broken against dead nas-1).
- scripts/import-cluster-access.sh: Garage -> Hetzner endpoint/bucket/region.
NOTE: the workstation aws-cli "secrets" profile must now hold the Hetzner
Object Storage keypair.
Also: delete the orphaned test-garage-smoke play + pipeline, and fix the
two dangling rebuild-k8s.yml references (zfs-tuning play comment;
update-audit prompt Talos-pin file list).
Assumes cluster-access objects (kubeconfig.sops/talosconfig.sops) were
migrated into renner-uno-k8s-secrets; with rebuild-k8s gone, nothing
auto-refreshes them, which is fine for a stable cluster.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The netbird-ingress DaemonSet, netbird reverse-proxy, and Traefik have
fully superseded the old ingress tiers, which have been stopped (not
deleted) since the 2026-05-09 PVE cutover. Remove the dead code:
- Caddy HA: LXCs 232/233/234 (opentofu/ingress.tf), caddy-ha role,
install-caddy-ha + install-crowdsec-caddy plays, caddy cred vars,
proxmox_caddy_ha group_vars, build-caddy/deploy-caddy pipelines,
ci-caddy-builder image.
- cloudflared: LXC 117, cloudflared-lxc role, install-cloudflared play,
cloudflare_tunnel/variables_cloudflare disabled tofu stubs.
- netbird-gateway: LXC 260 (replaced by the in-cluster netbird-ingress
DaemonSet), netbird-gateway role + play.
- backup_jobs.tf: drop VMIDs 117/232/233/234/260 from the nightly vzdump
include list (those LXCs are being destroyed).
- netbird-server: point the setup-key hint at netbird-ingress/netbird-proxy.
Kept deliberately:
- crowdsec role (reusable; basis for the pending Traefik bouncer).
- netbird-client role + play (loki/prometheus LXCs are tagged
netbird_client, so it remains live architecture).
Note: applying the opentofu change will plan a DESTROY of stopped LXCs
117/232/233/234/260 — review the plan before apply.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a Woodpecker cron pipeline that runs a read-only Claude Code agent to
audit the components NOT covered by existing automation, and posts the report
to a single persistent Gitea issue.
Scope (complements, does not duplicate, existing automation):
- Flux image-automation already bumps enrolled container tags.
- The daily security-scan agent already owns CVEs.
- This agent audits manually-pinned currency instead: Talos / Kubernetes /
Flux versions, Helm chart minor/major bumps held back by ~MAJOR.MINOR.0
constraints, images excluded from automation, and Ansible/OpenTofu drift.
The agent is read-only (allowedTools: Read,Glob,Grep,Write,WebFetch,WebSearch;
no Edit/Bash) so it can never mutate the repo. It writes a report to
/tmp/update-audit.md; the pipeline find-or-creates the "Infrastructure update
audit" Gitea issue and refreshes its body.
Cron uses Woodpecker's documented `cron:` name filter (rather than the injected
PIPELINE var the other pipelines use) so the schedule is CLI-creatable:
woodpecker repo cron add --repo gitea_admin/homelab \
--name update-audit --branch master --schedule '@weekly'
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Container image tag updates are Flux's job — the homelab cluster
already runs image-reflector-controller and image-automation-
controller (in flux-system/gotk-components.yaml). Setting up
ImageRepository / ImagePolicy / ImageUpdateAutomation resources
is the right home for "bump postgres:15.17 → 15.18 when a new
image is published"; doing it from a CVE-driven scan agent
duplicates that pipeline and competes with it.
Reverting:
- Agent prompt: drop the `## Image tag bumps` section; image
findings now go straight to the Skipped list with a "Flux
image-automation owns this" reason.
- Woodpecker yml: drop Bash(crane *) from the agent's tool
allowlist (no longer used) and the `pull: true` workaround
that existed only to pick up crane's mid-day rebuild.
- ci-claude Containerfile: remove the crane install. The image
is back to Node + Claude Code + git/jq/curl basics.
The agent is now scoped exclusively to IaC misconfigurations
(Trivy fs scan results). Image vulns surface in the log + summary
for visibility but no PR action.
Note: there are currently zero ImageRepository / ImagePolicy /
ImageUpdateAutomation resources in kubernetes/ — the controllers
are running idle. Wiring them up for the actual images is a
separate workstream.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The seed KS owns influxdb-seed-ssh Secret in the influxdb namespace, but
the namespace was previously created by apps/influxdb/namespace.yaml —
which dependsOn seed → seed never reconciles → circular.
Hoist the namespace into infrastructure/namespaces/namespaces.yaml
(consistent with all the other app namespaces) and drop it from the
apps/influxdb Kustomization.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pipeline 858's agent run found 277 image vulns (CRITICAL/HIGH) and
correctly skipped all of them — but the right action for many is a
patch-level image-tag bump (postgres:15.17 → 15.18,
distribution:3.0.0 → 3.0.1, etc.). The agent had no way to verify
a candidate tag exists in the registry, so it conservatively
skipped.
crane (google/go-containerregistry) handles:
crane ls <repository> — list available tags
crane manifest <image>:<tag> — verify tag resolves
Single static binary, works against Docker Hub, GHCR, gitea, quay,
any OCI registry. Pinned to v0.21.5.
The agent prompt update (only-patch bumps, lockstep cross-file edits,
compatibility-caveat surfacing) and the woodpecker allowlist update
(Bash(crane ls *), Bash(crane manifest *)) ride in a follow-up
commit so this image change can build first.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pipeline 856's agent run found the Trivy JSON files via the
pipeline's own jq summary (IaC=0, Image=645 findings) but the
agent itself reported "No Trivy scan files (/tmp/trivy-iac.json
...) were present" — it hallucinated the /tmp prefix and didn't
look in the workspace root.
Prompt now states explicitly that the JSONs are in the current
working directory (project root, not /tmp), and instructs the
agent's first action to be `Glob trivy-*.json` to enumerate what
exists. Adds a stop-condition for the genuinely-empty case so
the agent can't get into a "I'll assume they're somewhere else"
loop.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The cron security-scan now ends in an autonomous fix loop:
iac-scan → trivy-iac.json
image-scans → trivy-image-N.json (per scanned image)
security-fix-agent (master only):
1. Gates on >=1 CRITICAL/HIGH finding across all JSON outputs.
2. Creates branch security-fix/<date>-<short-sha>.
3. Hands the workspace to Claude Code in headless --bare mode
(subscription-backed via CLAUDE_CODE_OAUTH_TOKEN, no API key)
with a narrow tool allowlist (Read/Edit/Write/Glob/Grep —
no Bash). The agent reads the JSON, applies surgical fixes
under ansible/, opentofu/, kubernetes/, ci/images/, and writes
a Markdown summary to /tmp/security-fix-summary.md.
4. If the agent produced a diff, the step commits as
"Security Fix Agent <security-bot@renner.uno>" and pushes to
Gitea, then opens a PR via the pulls API using a dedicated
security_bot_token (NOT gitea_admin) so the PR is visibly
machine-authored.
The step is gated to branch=master so it doesn't fire on its own
PR branches — those still trigger the scan workflow as a re-validation
gate, but skip the agent (no fork-bombing).
Trivy now writes JSON output (--format json -o), and findings appear
both in the JSON and as one-line summaries in the step log via jq.
The legacy --exit-code 1 was kept off so the workflow always reaches
the agent step on master; on PR branches the scans are informational
and a human reviews the agent's PR.
Required Woodpecker secrets (added by user, not by this commit):
- claude_code_oauth_token (claude setup-token output)
- security_bot_token (Gitea PAT for the security-bot user)
Required Gitea setup (also out of band):
- User "security-bot" with email security-bot@renner.uno
- Write collaborator on gitea_admin/homelab
Agent constraints live in ci/agents/security-fix/prompt.md so they're
diffable + reviewable separately from the pipeline plumbing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fifth image in the ci-* family. Bakes Node.js 22 + the Claude Code
CLI (@anthropic-ai/claude-code) plus git, openssh-client, curl, jq.
Used by the upcoming security-fix-agent step in security-scan.yml,
which runs Claude headlessly inside CI to propose fixes for Trivy
findings and open a PR.
Auth pattern (deferred to the consumer commit): subscription-backed
CLAUDE_CODE_OAUTH_TOKEN secret, injected as an env var, no API key.
build-ci-images.yml gains a build-claude step that mirrors the
existing four (kaniko, parallel, registry-mirror cache, date + latest
tags pushed to gitea.renner.uno/gitea_admin/).
Pinning: Containerfile uses CLAUDE_CODE_VERSION=latest at image-build
time. The resulting date-tagged image freezes whatever was current
when build-ci-images ran. Bumping = trigger build-ci-images, get a
new date tag, bump the consumer pin.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fourth image in the ci-* family (alongside ci-ansible, ci-k8s,
ci-tofu). Bakes the Go toolchain (golang:1.26 base) plus xcaddy
pinned to v0.4.5 so the build-caddy pipeline doesn't have to
`go install ...@latest` on every run.
build-ci-images.yml gains a build-caddy-builder step that mirrors
the existing three (kaniko, parallel, pull-through-cached registry,
date + latest tags pushed to gitea.renner.uno/gitea_admin/).
The build-caddy.yml consumer swap is a separate commit so it can
land after the first build-ci-images run produces the image.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two fixes after pipeline 829's wipe-backups step blew up:
- aws-cli is not in ci-k8s:2026-04-29 — wipe ran "aws s3 rm ..." and
got "/bin/sh: aws: not found", then exited 0 because set -eu doesn't
catch failures inside pipes / command substitutions.
ci/images/k8s/Containerfile gains aws-cli; pushing this triggers the
build-ci-images pipeline, which builds gitea.renner.uno/gitea_admin/
ci-k8s:2026-05-01 (date tag).
The wipe-backups step is bumped to that tag and switches to
set -euo pipefail with an explicit "remaining objects must be 0"
check after each rm — so a failed wipe halts the rebuild before the
new cluster's seed Jobs run against stale state.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ghcr.io/opentofu/opentofu:1.11 has an ONBUILD that prints a deprecation
warning and runs `exit 1` — the maintainers explicitly disabled using
that image as a base in 1.10+ (opentofu/opentofu#1931). Build pipeline
810 hit it with `error building stage: ... exit status 1`.
Switch the base to alpine:3.21 and install the tofu binary directly
from the GitHub release zip (TOFU_VERSION=1.11.0 ARG, matches what the
pipeline previously pinned).
deploy-infrastructure was the one pipeline still installing
ansible/proxmoxer/galaxy collections at job time, because tofu apply
invokes ansible playbooks via module ansible_playbook provisioners —
both tools have to be in the same image.
ci-tofu inherits FROM ghcr.io/opentofu/opentofu:1.11 and bakes ansible
+ community.{proxmox,docker} collections + python deps + utility CLIs
on top. ENTRYPOINT/CMD reset so Woodpecker can run shell commands.
Both plan and apply steps swap to ci-tofu:2026-04-29. Drops apk add
git from plan and apk/pip/galaxy from apply.
Drops the per-job apk/pip/galaxy/curl-kubectl/curl-velero/curl-flux
preamble across 14 pipelines. Setup phase eliminated; only the
credential-staging steps remain. Pilot on update-all (commit b990749)
proved the swap mechanics; this completes the rollout.
ci-k8s gains velero (v1.16.0 ARG) so the migrate-* seed-pvcs steps
don't have to download it on every run.
ci-ansible swaps:
- update-all (already landed in b990749 — listed for completeness)
- install-{garage-synology,woodpecker-server,registry-mirror}
- deploy-caddy
- test-garage-smoke
- migrate-{immich,paperless,authentik-db}.backup-legacy-db
- rebuild-k8s.tune-cp-zvols
ci-k8s swaps:
- migrate-{influxdb,immich,paperless,music-assistant,audiobookshelf}.seed-pvcs
- migrate-authentik-db.wipe-and-restore
- e2e, e2e-test
- rebuild-k8s: talos-bootstrap, wipe-osd-disks, export-access,
flux-bootstrap, wait-apps, e2e-test
Left alone:
- deploy-infrastructure (special tofu+ansible mix on tofu image)
- migrate-tfstate (alpine + aws-cli/jq is fine)
- build-caddy (golang + alpine for release)
- security-scan (trivy)
- migrate-*.seed-barman steps (postgres-version-pinned images)
Image refs pin :2026-04-29; pipelines also use shell `base64 -d` in
place of the prior `python3 -c "import base64..."` (matches the
deploy-infrastructure.yml pattern).
apk add + pip install + ansible-galaxy ran on every job across ~12
pipelines and dominated wall time. Bake the toolsets into two images
pushed to Gitea's container registry; consuming pipelines pull instead.
ci-ansible: ansible + community.{proxmox,docker} + proxmoxer/requests
+ utility CLIs (sops, aws-cli, jq, sshpass, ...).
ci-k8s: talosctl (pinned to v1.12.6), kubectl (stable at build
time), flux + sops/openssl/bind-tools.
Tags YYYY-MM-DD plus :latest. Build pipeline triggers on changes under
ci/images/** or manual PIPELINE=build-ci-images.
claude-botandClaude Opus 4.8
k.renner