From cd12bc16c9b2bf78b02b95ba97870a1184ab371a Mon Sep 17 00:00:00 2001
From: Claude Bot <claude-bot@renner.uno>
Date: Thu, 30 Jul 2026 19:56:06 +0200
Subject: [PATCH 01/14] ci: validate the GitOps tree and pipeline configs on
 every PR
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Nothing covered kubernetes/** before this. deploy-infrastructure.yml gates
only opentofu/**, and ci-scripts-test.yml runs only the scripts/ci/ unit
tests — so a broken manifest reached the cluster and was caught by Flux
failing to reconcile, or by someone noticing. Reviewers flagged the gap on
three separate PRs today.

Four static checks, all offline (no cluster, no registry, no secrets):

  kustomize-build   every kustomization under kubernetes/ must build
  woodpecker-lint   every .woodpecker/*.yml must compile
  shellcheck        standalone .sh under kubernetes/, severity=error
  no-digest-pins    rotators must not reintroduce @sha256 image pins

Each was verified GREEN against the tree as it stands before being added,
so this gates new breakage rather than failing on pre-existing debt: all 60
kustomizations build, all pipeline configs lint, all 6 shell scripts are
clean at error severity, and no digest pins remain after #459.

The woodpecker-lint step is the one that matters most: it exits 1 on the
exact defect that took every pipeline in this repo down earlier today
(#443) — an unquoted YAML scalar containing ": ", which `yq` accepts
because it IS valid YAML, just a map rather than a string. Verified both
directions in the image the step actually uses: rc=1 on that construct,
rc=0 on the pre-existing "Additional property timeout is not allowed"
warnings in four deploy-*.yml files, so it fails on real breakage without
blocking on legacy noise.

Image choices, all version-pinned with no floating tags:

  ci-k8s:2026-07-30                       kubectl for kustomize; also the
                                          shell for the grep-based guard
  woodpecker-cli:v3.9.0-alpine            the non-alpine image is distroless
                                          and has no /bin/sh, so Woodpecker
                                          `commands:` cannot run in it; the
                                          binary is /bin/woodpecker-cli, not
                                          on PATH as `woodpecker`
  shellcheck-alpine:v0.10.0               pinned rather than :stable, which
                                          floats like :latest

Deliberately NOT included: kubeconform. It needs CRD schemas for
ExternalSecret, Certificate, HelmRelease and friends, so it either needs a
schema-fetching step (network, and a new failure mode) or
-ignore-missing-schemas, which would skip precisely the CRDs most of this
tree is made of. Worth doing properly as its own change.

Known gap, called out in the file rather than implied away: shellcheck
cannot see shell embedded in manifests (init containers, probe exec
blocks), which is where several real bugs lived this year.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 .woodpecker/ci-manifests-test.yml | 117 ++++++++++++++++++++++++++++++
 1 file changed, 117 insertions(+)
 create mode 100644 .woodpecker/ci-manifests-test.yml

diff --git a/.woodpecker/ci-manifests-test.yml b/.woodpecker/ci-manifests-test.yml
new file mode 100644
index 00000000..7f137d84
--- /dev/null
+++ b/.woodpecker/ci-manifests-test.yml
@@ -0,0 +1,117 @@
+# Validate the GitOps tree and the pipeline definitions on every pull_request.
+#
+# Nothing covered kubernetes/** before this. deploy-infrastructure.yml only
+# gates opentofu/**, and ci-scripts-test.yml only runs scripts/ci/ unit tests —
+# so a broken manifest reached the cluster and was caught by Flux failing to
+# reconcile, or by a human noticing. Three separate defects on 2026-07-30 would
+# have been caught here instead:
+#
+#   * an unquoted YAML scalar containing ": " in .woodpecker/build-ci-images.yml,
+#     which made EVERY pipeline in the repo fail to compile (#443). `yq` parsed
+#     it happily — it is valid YAML, just a map instead of a string — so
+#     "it parses" was mistaken for "it is valid". Only woodpecker lint catches it.
+#   * a rotator pinned to an image digest that no longer existed (#458).
+#   * manifests whose kustomize wiring was only ever checked by hand.
+#
+# Deliberately CHEAP and offline: no cluster, no registry, no secrets. Everything
+# here is a static check that either passes or fails on the checkout alone, so it
+# can gate every PR without becoming a bottleneck or a flake source.
+when:
+  - event: pull_request
+
+steps:
+  # Every kustomization must build. This is the check that catches a resource
+  # entry pointing at a file that does not exist, a bad patch target, or YAML
+  # that is fine in isolation but broken once overlaid — the class of error that
+  # otherwise surfaces as a Flux reconcile failure after merge.
+  #
+  # Verified green across all 60 kustomizations in the tree at the time of
+  # writing, so this gates new breakage rather than pre-existing debt.
+  - name: kustomize-build
+    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-07-30
+    commands:
+      - |
+        set -eu
+        fail=0
+        for d in $(find kubernetes -name kustomization.yaml -printf '%h\n' | sort); do
+          if kubectl kustomize "$d" >/dev/null 2>/tmp/err; then
+            echo "ok    $d"
+          else
+            echo "FAIL  $d"
+            sed 's/^/        /' /tmp/err
+            fail=1
+          fi
+        done
+        exit $fail
+
+  # Lint the pipeline definitions themselves. This is the one that would have
+  # prevented the repo-wide CI outage: `woodpecker lint` exits 1 on a config that
+  # fails to compile, while exiting 0 on the "Additional property timeout is not
+  # allowed" warnings that several deploy-*.yml files already carry — so it fails
+  # on real breakage without blocking on pre-existing noise. Both behaviours were
+  # verified against this repo before adding the step.
+  - name: woodpecker-lint
+    image: woodpeckerci/woodpecker-cli:v3.9.0-alpine
+    commands:
+      - |
+        set -eu
+        fail=0
+        for f in .woodpecker/*.yml; do
+          if /bin/woodpecker-cli lint "$f" >/tmp/out 2>&1; then
+            echo "ok    $f"
+          else
+            echo "FAIL  $f"
+            sed 's/^/        /' /tmp/out
+            fail=1
+          fi
+        done
+        exit $fail
+
+  # Standalone shell scripts under kubernetes/ (sidecar and bootstrap scripts
+  # mounted via configMapGenerator). Severity is pinned to `error` on purpose:
+  # style and info findings would fail the build on existing scripts for no
+  # safety gain. All scripts in the tree are clean at this level today.
+  #
+  # NOTE this does NOT cover shell embedded in manifests (init containers, probe
+  # exec blocks) — shellcheck cannot see inside YAML. That embedded shell is
+  # where several real bugs lived this year, so it remains a gap worth closing
+  # separately rather than one this step quietly implies is covered.
+  - name: shellcheck
+    image: koalaman/shellcheck-alpine:v0.10.0
+    commands:
+      - |
+        set -eu
+        fail=0
+        for f in $(find kubernetes -name '*.sh' | sort); do
+          if shellcheck -S error "$f"; then
+            echo "ok    $f"
+          else
+            echo "FAIL  $f"
+            fail=1
+          fi
+        done
+        exit $fail
+
+  # Guard the convention that cost a day to diagnose: registry-local images are
+  # pinned by DATED TAG, never by digest. Digests here are garbage-collected once
+  # :latest and the dated tags move past them — and that happens on edits to
+  # .woodpecker/build-ci-images.yml, a shared file, so an unrelated image addition
+  # silently orphans the manifest a rotator was pinned to. 11 rotators were dead
+  # this way and would have failed on their first scheduled run months later.
+  #
+  # Scoped to the rotation directory, which is where the convention is documented
+  # and where the failure actually bit. `image:` anchors the match so the
+  # kustomization comment documenting this rule does not flag itself.
+  - name: no-digest-pins
+    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-07-30
+    commands:
+      - |
+        set -eu
+        hits=$(grep -l 'image:.*@sha256' kubernetes/infrastructure/openbao/rotation/*.yaml 2>/dev/null \
+                 | grep -v kustomization.yaml || true)
+        if [ -n "$hits" ]; then
+          echo "FAIL: digest-pinned images found; use a dated tag (see rotation/kustomization.yaml):"
+          echo "$hits" | sed 's/^/        /'
+          exit 1
+        fi
+        echo "ok    no digest pins under openbao/rotation/"
-- 
2.54.0


From 790fd1933a4c45ecd49cdf31a91c8e4da2072a4b Mon Sep 17 00:00:00 2001
From: Claude Bot <claude-bot@renner.uno>
Date: Thu, 30 Jul 2026 20:04:48 +0200
Subject: [PATCH 02/14] fix(ci): kustomize-build validated nothing and reported
 green
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Blocking review point on #460, and it was exactly right.

`find -printf` is a GNU findutils extension. ci-k8s is alpine-based and
never installs findutils, so find is BusyBox — confirmed by running it in
the image:

  find: unrecognized: -printf
  BusyBox v1.37.0

Because the result fed a command substitution used as a word list, the
failure was SILENT: empty output, zero loop iterations, fail stayed 0, and
the step exited green having run kubectl kustomize on nothing at all.

The proof is in this PR's own pipeline run (3158). The step is recorded as
"success", and its log contains ZERO `ok` lines — sixty were expected. So
the gate I added to catch silent no-ops was itself a silent no-op, and CI
reported it as passing.

Worth stating why neither of my checks caught it: I dry-ran the loop on my
workstation, which has GNU find, and the CI run cannot distinguish a
zero-iteration loop from a working one when every kustomization already
builds — both exit 0. The review's suggested test (paste the per-directory
log lines, not the exit code) is the only one that separates them.

Fixed by replacing -printf with a portable pipe through sed. Verified in
the actual image, with a synthetic tree, that the loop now iterates.

Also added a discovery assertion to both find-based steps: fail if the
count is implausible (kustomize-build expects >=50 of ~60; shellcheck
expects >=1) rather than proceeding with an empty list. That is the general
fix — without it, ANY future breakage of discovery degrades back into a
green no-op instead of a failure, which is the trap this whole commit is
about rather than the -printf detail specifically.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 .woodpecker/ci-manifests-test.yml | 35 +++++++++++++++++++++++++++++--
 1 file changed, 33 insertions(+), 2 deletions(-)

diff --git a/.woodpecker/ci-manifests-test.yml b/.woodpecker/ci-manifests-test.yml
index 7f137d84..9e625c29 100644
--- a/.woodpecker/ci-manifests-test.yml
+++ b/.woodpecker/ci-manifests-test.yml
@@ -32,8 +32,29 @@ steps:
     commands:
       - |
         set -eu
+        # NOT `find -printf '%h\n'`: that is a GNU findutils extension and this
+        # image ships BusyBox find (v1.37.0), which rejects it outright with
+        # "find: unrecognized: -printf". Because the result feeds a command
+        # substitution used as a word list, the failure is SILENT — empty output,
+        # zero loop iterations, fail=0, step exits green having validated
+        # nothing. That is exactly what this pipeline shipped in its first
+        # revision, and CI reported it as passing.
+        dirs=$(find kubernetes -name kustomization.yaml | sed 's|/kustomization.yaml$||' | sort)
+
+        # Assert the discovery actually found something. Without this, any future
+        # breakage of the find/sed above degrades back into a green no-op rather
+        # than a failure — the specific trap that made the first revision of this
+        # step useless while looking healthy.
+        count=$(printf '%s\n' "$dirs" | grep -c . || true)
+        if [ "$count" -lt 50 ]; then
+          echo "FAIL: found only $count kustomizations; expected ~60."
+          echo "      Discovery is broken — refusing to report success."
+          exit 1
+        fi
+        echo "discovered $count kustomizations"
+
         fail=0
-        for d in $(find kubernetes -name kustomization.yaml -printf '%h\n' | sort); do
+        for d in $dirs; do
           if kubectl kustomize "$d" >/dev/null 2>/tmp/err; then
             echo "ok    $d"
           else
@@ -81,8 +102,18 @@ steps:
     commands:
       - |
         set -eu
+        files=$(find kubernetes -name '*.sh' | sort)
+        # Same no-op guard as kustomize-build: if discovery ever returns nothing,
+        # fail loudly instead of passing having checked no files.
+        count=$(printf '%s\n' "$files" | grep -c . || true)
+        if [ "$count" -lt 1 ]; then
+          echo "FAIL: no shell scripts found under kubernetes/ — discovery is broken."
+          exit 1
+        fi
+        echo "discovered $count shell scripts"
+
         fail=0
-        for f in $(find kubernetes -name '*.sh' | sort); do
+        for f in $files; do
           if shellcheck -S error "$f"; then
             echo "ok    $f"
           else
-- 
2.54.0


From ddf5f5007bad978c797f1e3cf75fc0468daa0142 Mon Sep 17 00:00:00 2001
From: Claude Bot <claude-bot@renner.uno>
Date: Thu, 30 Jul 2026 20:11:26 +0200
Subject: [PATCH 03/14] docs(ci): correct how the digest guard avoids
 self-matching
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Review point on #460. The comment claimed "`image:` anchors the match so
the kustomization comment documenting this rule does not flag itself".
That is wrong about the mechanism, and the reviewer proved it by running
the bare grep — kustomization.yaml DOES match, because the comment
documenting the rule contains the literal string `image:.*@sha256`.

What actually excludes it is the `grep -v kustomization.yaml` filter. The
step's behaviour was always correct; only the explanation was wrong.

Worth fixing rather than shrugging at: this is the third time today the
same self-matching trap has caught me. Twice while writing the guard in
rotation/kustomization.yaml (where the final version documents it
accurately), and now once more in the comment describing it here. A
comment that misidentifies which line is load-bearing invites someone to
delete the filter as redundant tidying, at which point the step fails
permanently on its own documentation.

The comment now says the filter is load-bearing and why.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 .woodpecker/ci-manifests-test.yml | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/.woodpecker/ci-manifests-test.yml b/.woodpecker/ci-manifests-test.yml
index 9e625c29..32847750 100644
--- a/.woodpecker/ci-manifests-test.yml
+++ b/.woodpecker/ci-manifests-test.yml
@@ -131,8 +131,14 @@ steps:
   # this way and would have failed on their first scheduled run months later.
   #
   # Scoped to the rotation directory, which is where the convention is documented
-  # and where the failure actually bit. `image:` anchors the match so the
-  # kustomization comment documenting this rule does not flag itself.
+  # and where the failure actually bit.
+  #
+  # The `grep -v kustomization.yaml` is load-bearing, not tidying: that file
+  # documents this very rule, so its comment contains the literal string
+  # `image:.*@sha256` and therefore MATCHES the pattern. Anchoring on `image:`
+  # does not prevent that — verified by running the bare grep, which returns
+  # kustomization.yaml. Only the filename filter excludes it. Drop the filter and
+  # this step fails permanently on its own documentation.
   - name: no-digest-pins
     image: gitea.renner.uno/gitea_admin/ci-k8s:2026-07-30
     commands:
-- 
2.54.0


From 1911a20db529a730fca297860fd2d5f46b15048d Mon Sep 17 00:00:00 2001
From: Claude Bot <claude-bot@renner.uno>
Date: Thu, 30 Jul 2026 20:17:27 +0200
Subject: [PATCH 04/14] ci: path-filter the manifest gate and harden its loops
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Review points on #460.

Path-filter the pipeline to kubernetes/** and .woodpecker/**. It was
unfiltered, matching ci-scripts-test.yml, but that comparison does not
hold: ci-scripts-test is one step on one already-cached image, whereas this
pulls three images and builds 60 kustomizations — real cost to impose on a
docs-only PR. The filter still covers everything the steps read, and
including .woodpecker/** means an edit to this file re-runs its own checks.

Pin IFS to newline in both find-driven loops so a path containing spaces is
passed as one argument rather than split into fragments. Verified in the
actual image: a directory named "has space" now iterates once, intact.

Deliberately NOT rewritten as `printf | while read`, which is the usual
advice: that puts the loop in a subshell, so `fail` would not propagate and
a failing kustomize build would exit 0. That is the same silent-pass shape
this step already shipped once with -printf, and fixing a latent
whitespace bug by reintroducing a live correctness bug would be a poor
trade.

Also corrected the file header, which still claimed the gate runs on every
pull_request.

On the ci-k8s:2026-07-30 tag the review asked about again: confirmed by
execution rather than inspection. Pipeline 3161 ran this pipeline against a
real checkout with all five steps green, which requires all three images to
have pulled in the runner, and its kustomize-build step logged "discovered
60 kustomizations" with 60 ok lines — so the image both resolved and did
the work.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 .woodpecker/ci-manifests-test.yml | 22 ++++++++++++++++++++--
 1 file changed, 20 insertions(+), 2 deletions(-)

diff --git a/.woodpecker/ci-manifests-test.yml b/.woodpecker/ci-manifests-test.yml
index 32847750..492300d5 100644
--- a/.woodpecker/ci-manifests-test.yml
+++ b/.woodpecker/ci-manifests-test.yml
@@ -1,4 +1,4 @@
-# Validate the GitOps tree and the pipeline definitions on every pull_request.
+# Validate the GitOps tree and the pipeline definitions on PRs that touch them.
 #
 # Nothing covered kubernetes/** before this. deploy-infrastructure.yml only
 # gates opentofu/**, and ci-scripts-test.yml only runs scripts/ci/ unit tests —
@@ -15,9 +15,18 @@
 #
 # Deliberately CHEAP and offline: no cluster, no registry, no secrets. Everything
 # here is a static check that either passes or fails on the checkout alone, so it
-# can gate every PR without becoming a bottleneck or a flake source.
+# can gate PRs without becoming a bottleneck or a flake source.
 when:
+  # Path-filtered on purpose, unlike ci-scripts-test.yml. That pipeline is one
+  # step on one already-cached image, so running it everywhere is free; this one
+  # pulls three images and builds 60 kustomizations, which is real cost to put on
+  # a docs-only PR. The filter covers everything the steps actually read:
+  # kubernetes/** for the manifests and scripts, .woodpecker/** for the lint (and
+  # so an edit to this file re-runs its own checks).
   - event: pull_request
+    path:
+      - "kubernetes/**"
+      - ".woodpecker/**"
 
 steps:
   # Every kustomization must build. This is the check that catches a resource
@@ -53,6 +62,12 @@ steps:
         fi
         echo "discovered $count kustomizations"
 
+        # IFS pinned to newline so a path containing spaces is not split into
+        # fragments. Deliberately NOT `printf | while read`: that puts the loop
+        # in a subshell, so `fail` would not propagate and a failing build would
+        # exit 0 — reintroducing the silent-pass bug this step already had once.
+        IFS="
+"
         fail=0
         for d in $dirs; do
           if kubectl kustomize "$d" >/dev/null 2>/tmp/err; then
@@ -112,6 +127,9 @@ steps:
         fi
         echo "discovered $count shell scripts"
 
+        # Same IFS reasoning as kustomize-build above.
+        IFS="
+"
         fail=0
         for f in $files; do
           if shellcheck -S error "$f"; then
-- 
2.54.0


From 9d3dc152c1b03afe6d810673664b64ea5886b764 Mon Sep 17 00:00:00 2001
From: Claude Bot <claude-bot@renner.uno>
Date: Thu, 30 Jul 2026 20:18:39 +0200
Subject: [PATCH 05/14] fix(ci): repair the block scalar I broke in the
 previous commit
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The previous commit set IFS to a literal newline by writing

        IFS="
"

inside a `- |` block scalar. The closing quote sits at column 0, which
terminates the block — so the config no longer parsed:

  go-yaml load error in scanner (while scanning a simple key) at
  L70.C1-L89.C3: could not find expected ':'

Same class as #443, in the very pipeline written to catch #443, and I
pushed it: `woodpecker lint` printed the error in the same command as the
push, and I read the push output rather than the lint output.

Replaced with `while IFS= read -r` fed from a FILE rather than a pipe.
That handles paths with spaces without needing a literal newline anywhere,
and the file redirection keeps the loop in the current shell so `fail`
still propagates — a pipe would have run it in a subshell and made a
failing kustomize build exit 0, which is the silent-pass shape this step
already shipped once with -printf.

Verified both properties in the actual image rather than reasoning about
them: a directory named "has space" iterates once intact, and a `fail=1`
set inside the loop is still 1 after it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 .woodpecker/ci-manifests-test.yml | 27 ++++++++++++++-------------
 1 file changed, 14 insertions(+), 13 deletions(-)

diff --git a/.woodpecker/ci-manifests-test.yml b/.woodpecker/ci-manifests-test.yml
index 492300d5..b56436ef 100644
--- a/.woodpecker/ci-manifests-test.yml
+++ b/.woodpecker/ci-manifests-test.yml
@@ -62,14 +62,15 @@ steps:
         fi
         echo "discovered $count kustomizations"
 
-        # IFS pinned to newline so a path containing spaces is not split into
-        # fragments. Deliberately NOT `printf | while read`: that puts the loop
-        # in a subshell, so `fail` would not propagate and a failing build would
-        # exit 0 — reintroducing the silent-pass bug this step already had once.
-        IFS="
-"
+        # `while read` fed from a FILE, not a pipe: a pipe would run the loop in
+        # a subshell where `fail` cannot propagate, so a failing build would exit
+        # 0 — the same silent-pass shape this step already shipped once. Reading
+        # a file keeps the loop in the current shell, and read -r handles paths
+        # containing spaces without IFS games.
+        printf '%s\n' "$dirs" > /tmp/dirs
         fail=0
-        for d in $dirs; do
+        while IFS= read -r d; do
+          [ -n "$d" ] || continue
           if kubectl kustomize "$d" >/dev/null 2>/tmp/err; then
             echo "ok    $d"
           else
@@ -77,7 +78,7 @@ steps:
             sed 's/^/        /' /tmp/err
             fail=1
           fi
-        done
+        done < /tmp/dirs
         exit $fail
 
   # Lint the pipeline definitions themselves. This is the one that would have
@@ -127,18 +128,18 @@ steps:
         fi
         echo "discovered $count shell scripts"
 
-        # Same IFS reasoning as kustomize-build above.
-        IFS="
-"
+        # Same file-fed while-read reasoning as kustomize-build above.
+        printf '%s\n' "$files" > /tmp/files
         fail=0
-        for f in $files; do
+        while IFS= read -r f; do
+          [ -n "$f" ] || continue
           if shellcheck -S error "$f"; then
             echo "ok    $f"
           else
             echo "FAIL  $f"
             fail=1
           fi
-        done
+        done < /tmp/files
         exit $fail
 
   # Guard the convention that cost a day to diagnose: registry-local images are
-- 
2.54.0


From 888aa60fe048bf8eb5f7a3cc6c1b099296906a54 Mon Sep 17 00:00:00 2001
From: Claude Bot <claude-bot@renner.uno>
Date: Thu, 30 Jul 2026 20:25:23 +0200
Subject: [PATCH 06/14] ci: run the four checks in parallel so every failure
 shows at once
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Review points on #460.

Woodpecker runs steps sequentially by default, so a failing kustomize-build
hid woodpecker-lint, shellcheck and no-digest-pins entirely — a contributor
would fix one thing, repush, and only then discover the next. All four are
independent, cheap and offline, with no ordering between them, so
depends_on: [] fans them out and one run reports everything.

Verified the structure rather than the diff: all four steps now render with
depends_on=[].

Also made the discovery-floor failure actionable. The message said only
"Discovery is broken", which is wrong advice if the tree legitimately
shrank below 50 kustomizations — it now tells the reader to lower the floor
in that case, and explicitly not to delete it, since removing the guard
restores the silent-green no-op it exists to prevent.

On the ci-k8s:2026-07-30 tag, raised for the third time: confirmed by
execution, not inspection. It resolved in pipeline 3161, whose
kustomize-build and no-digest-pins steps both ran that image and logged
"discovered 60 kustomizations" with 60 ok lines. An image that failed to
pull could not have produced that output. The review is right that nothing
in THIS PR rebuilds the tag — it was published earlier today by
build-ci-images runs triggered by edits to .woodpecker/build-ci-images.yml,
which is also exactly the drift that left other pipelines pinned to
ci-k8s:2026-05-01.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 .woodpecker/ci-manifests-test.yml | 24 +++++++++++++++++++++++-
 1 file changed, 23 insertions(+), 1 deletion(-)

diff --git a/.woodpecker/ci-manifests-test.yml b/.woodpecker/ci-manifests-test.yml
index b56436ef..6ba9bdcc 100644
--- a/.woodpecker/ci-manifests-test.yml
+++ b/.woodpecker/ci-manifests-test.yml
@@ -37,6 +37,11 @@ steps:
   # Verified green across all 60 kustomizations in the tree at the time of
   # writing, so this gates new breakage rather than pre-existing debt.
   - name: kustomize-build
+    # depends_on: [] so all four run in parallel and every failure shows in one
+    # run. Sequential default would stop at the first, making a contributor
+    # fix-and-repush to discover the next — and these are independent, cheap,
+    # offline checks with no ordering between them.
+    depends_on: []
     image: gitea.renner.uno/gitea_admin/ci-k8s:2026-07-30
     commands:
       - |
@@ -57,7 +62,9 @@ steps:
         count=$(printf '%s\n' "$dirs" | grep -c . || true)
         if [ "$count" -lt 50 ]; then
           echo "FAIL: found only $count kustomizations; expected ~60."
-          echo "      Discovery is broken — refusing to report success."
+          echo "      If the tree legitimately shrank, lower this floor in"
+          echo "      .woodpecker/ci-manifests-test.yml — do not delete it."
+          echo "      Otherwise discovery is broken — refusing to report success."
           exit 1
         fi
         echo "discovered $count kustomizations"
@@ -88,6 +95,11 @@ steps:
   # on real breakage without blocking on pre-existing noise. Both behaviours were
   # verified against this repo before adding the step.
   - name: woodpecker-lint
+    # depends_on: [] so all four run in parallel and every failure shows in one
+    # run. Sequential default would stop at the first, making a contributor
+    # fix-and-repush to discover the next — and these are independent, cheap,
+    # offline checks with no ordering between them.
+    depends_on: []
     image: woodpeckerci/woodpecker-cli:v3.9.0-alpine
     commands:
       - |
@@ -114,6 +126,11 @@ steps:
   # where several real bugs lived this year, so it remains a gap worth closing
   # separately rather than one this step quietly implies is covered.
   - name: shellcheck
+    # depends_on: [] so all four run in parallel and every failure shows in one
+    # run. Sequential default would stop at the first, making a contributor
+    # fix-and-repush to discover the next — and these are independent, cheap,
+    # offline checks with no ordering between them.
+    depends_on: []
     image: koalaman/shellcheck-alpine:v0.10.0
     commands:
       - |
@@ -159,6 +176,11 @@ steps:
   # kustomization.yaml. Only the filename filter excludes it. Drop the filter and
   # this step fails permanently on its own documentation.
   - name: no-digest-pins
+    # depends_on: [] so all four run in parallel and every failure shows in one
+    # run. Sequential default would stop at the first, making a contributor
+    # fix-and-repush to discover the next — and these are independent, cheap,
+    # offline checks with no ordering between them.
+    depends_on: []
     image: gitea.renner.uno/gitea_admin/ci-k8s:2026-07-30
     commands:
       - |
-- 
2.54.0


From 7b5f244b7f44e7d1749d7a8168007fbc3e57c5b9 Mon Sep 17 00:00:00 2001
From: Claude Bot <claude-bot@renner.uno>
Date: Thu, 30 Jul 2026 20:56:37 +0200
Subject: [PATCH 07/14] ci: use latest stable versions, and match the
 woodpecker CLI to the server
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Violated the standing rule: "use the latest stable version. Old versions
must not be hardcoded without a specific, documented reason."

woodpecker-cli was pinned at v3.9.0 while the deployed server and agent run
v3.16.0 — seven minor versions behind. That version was never chosen; I
picked it while probing for any tag that had a shell and then never
revisited it.

For this step the skew is not cosmetic. The CLI compiles the config against
its own schema, so one behind the server can pass a config the server later
rejects, or reject one the server accepts. Either way the gate stops
predicting what the server will do, which is the only reason it exists —
and the failure it was added to catch (#443) is precisely a
server-side compile error.

Now v3.16.0-alpine, matched to the running deployment (checked against the
live Deployment, not assumed), with a comment saying it must be bumped
alongside the server. Verified it still behaves correctly: rc=0 on a valid
config, rc=1 on the #443 construct.

shellcheck was pinned at v0.10.0 for the same non-reason — it was simply
the newer of the two tags I happened to probe. v0.11.0 exists, so bumped.
Verified first that the existing scripts still pass at -S error under the
newer version, since a shellcheck bump can add checks and I did not want to
ship a gate that fails on the current tree.

ci-k8s stays on the dated-tag convention for registry-local images.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 .woodpecker/ci-manifests-test.yml | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/.woodpecker/ci-manifests-test.yml b/.woodpecker/ci-manifests-test.yml
index 6ba9bdcc..1777937c 100644
--- a/.woodpecker/ci-manifests-test.yml
+++ b/.woodpecker/ci-manifests-test.yml
@@ -94,13 +94,19 @@ steps:
   # allowed" warnings that several deploy-*.yml files already carry — so it fails
   # on real breakage without blocking on pre-existing noise. Both behaviours were
   # verified against this repo before adding the step.
+  # CLI version MUST track the deployed woodpecker-server (v3.16.0 — checked
+  # against the running deployment, not assumed). A CLI behind the server
+  # compiles against an older schema, so it can pass a config the server then
+  # rejects, or reject one the server accepts — either way the gate stops
+  # reflecting what will actually happen, which is its whole purpose. Bump this
+  # with the server.
   - name: woodpecker-lint
     # depends_on: [] so all four run in parallel and every failure shows in one
     # run. Sequential default would stop at the first, making a contributor
     # fix-and-repush to discover the next — and these are independent, cheap,
     # offline checks with no ordering between them.
     depends_on: []
-    image: woodpeckerci/woodpecker-cli:v3.9.0-alpine
+    image: woodpeckerci/woodpecker-cli:v3.16.0-alpine
     commands:
       - |
         set -eu
@@ -131,7 +137,7 @@ steps:
     # fix-and-repush to discover the next — and these are independent, cheap,
     # offline checks with no ordering between them.
     depends_on: []
-    image: koalaman/shellcheck-alpine:v0.10.0
+    image: koalaman/shellcheck-alpine:v0.11.0
     commands:
       - |
         set -eu
-- 
2.54.0


From fbfaf90923251aeed6b2b2ae400daac6210472da Mon Sep 17 00:00:00 2001
From: Claude Bot <claude-bot@renner.uno>
Date: Thu, 30 Jul 2026 21:02:23 +0200
Subject: [PATCH 08/14] docs(ci): frame the CLI pin as the latest-stable rule,
 not a separate one
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

I had written that the CLI "MUST track the deployed woodpecker-server",
which invents a rule that does not exist and misstates the one that does.

The standing rule already covers it: any software uses the latest stable
version. CLI tools are software. If the CLI and the server both follow it,
they match by construction — matching is a CONSEQUENCE, not a separate
requirement. I had read "any software" as meaning installs and images but
not CLIs, and was about to propose adding a redundant rule to the wiki off
the back of that misreading.

The comment now says that, and keeps the reason the skew mattered here: the
CLI compiles against its own schema, so one behind the server can pass a
config the server rejects, and the gate stops predicting server behaviour.

Also recorded a real limitation found while checking this. The server and
agent are bumped automatically by image-automation within the 3.x range,
and their policy comment notes they must stay version-matched. This pin is
manual, because ImageUpdateAutomation is scoped to
`update.path: ./kubernetes` and cannot maintain a tag under .woodpecker/.
So the CLI will lag a server upgrade until someone bumps it. Widening that
path would let a policy own this line the same way it owns the server and
agent — noted rather than done, since it changes automation scope repo-wide.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 .woodpecker/ci-manifests-test.yml | 21 +++++++++++++++------
 1 file changed, 15 insertions(+), 6 deletions(-)

diff --git a/.woodpecker/ci-manifests-test.yml b/.woodpecker/ci-manifests-test.yml
index 1777937c..2bf913fe 100644
--- a/.woodpecker/ci-manifests-test.yml
+++ b/.woodpecker/ci-manifests-test.yml
@@ -94,12 +94,21 @@ steps:
   # allowed" warnings that several deploy-*.yml files already carry — so it fails
   # on real breakage without blocking on pre-existing noise. Both behaviours were
   # verified against this repo before adding the step.
-  # CLI version MUST track the deployed woodpecker-server (v3.16.0 — checked
-  # against the running deployment, not assumed). A CLI behind the server
-  # compiles against an older schema, so it can pass a config the server then
-  # rejects, or reject one the server accepts — either way the gate stops
-  # reflecting what will actually happen, which is its whole purpose. Bump this
-  # with the server.
+  # Latest stable, per the standing rule that any software uses the latest
+  # stable version. Matching the deployed woodpecker-server (v3.16.0) is a
+  # CONSEQUENCE of that, not a separate requirement — the server is bumped by
+  # image-automation within the 3.x range, so both land on the same release.
+  #
+  # It matters here because the CLI compiles the config against its own schema:
+  # one behind the server can pass a config the server then rejects, or reject
+  # one it accepts, and the gate stops predicting what the server will do —
+  # which is its only purpose.
+  #
+  # DRIFT RISK: the server auto-bumps (image-automation), this pin does not.
+  # ImageUpdateAutomation is scoped to `update.path: ./kubernetes`, so it cannot
+  # maintain a tag in .woodpecker/. Widening that path would let a policy own
+  # this line the way it owns the server and agent; until then this is a manual
+  # bump and will lag a server upgrade.
   - name: woodpecker-lint
     # depends_on: [] so all four run in parallel and every failure shows in one
     # run. Sequential default would stop at the first, making a contributor
-- 
2.54.0


From 8a580e341d61b194858584a484792cd5a5cc02af Mon Sep 17 00:00:00 2001
From: Claude Bot <claude-bot@renner.uno>
Date: Thu, 30 Jul 2026 21:19:02 +0200
Subject: [PATCH 09/14] docs(ci): record why woodpecker-cli cannot be enrolled
 in image-automation
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

I proposed widening ImageUpdateAutomation's `update.path` to `./` and adding
a woodpecker-cli ImagePolicy so automation would own the lint pin the way it
owns the server and agent. Checking that before building it killed the plan,
and corrected two things I had asserted without evidence.

WRONG #1 — my reason for hesitating. I claimed the 7 ansible-vault encrypted
`.yml` files would break the walk and take all 33 automated images down with
them. That was a guess about kyaml I never verified. Objects that are not
Kubernetes manifests are SKIPPED, not fatal.

WRONG #2 — and this is the one that matters: the plan cannot work at all.
Image-automation only patches YAML that is a Kubernetes manifest. Maintainer,
fluxcd/flux2#2826: "Flux can only patch Kubernetes manifests. This means the
YAML manifest must have `apiVersion` and `kind` fields." The docs agree —
markers go "in any Kubernetes custom resource stored in Git". A Woodpecker
pipeline file is plain YAML with no apiVersion/kind, so a marker in it is
silently ignored.

That is a property of the FILE, not of `update.path`. Widening the path buys
exactly nothing and would drag 197 unrelated ansible/esphome YAMLs into the
walk. So no policy, no second automation, no path change — all three would
have been motion that looked like a fix and changed nothing. The failure mode
would have been the quiet kind this repo keeps getting bitten by: a marker
sitting in the file looking maintained, never once updated.

What actually closes the gap, given the pin must stay manual:

  * README gains a "Not enrolled (structurally impossible)" section holding
    the quote and the explicit "do not try widening the path" warning, so the
    next person does not re-derive this.
  * update-audit item 5 now covers `.woodpecker/*.yml` step images and is told
    to flag woodpecker-cli whenever it trails the DEPLOYED SERVER, not merely
    the newest CLI release — the server auto-bumps and the CLI does not, so
    that skew is the thing worth reporting. Also told to compare only the
    vX.Y.Z part (the tag carries `-alpine`; the plain tag is scratch with no
    shell and would break the step) and to ignore the `next-*`/`pull_*` dev
    tags, which are the newest by date on that repo and are not releases.
  * the step comment states the real constraint instead of my earlier wrong
    one, which told the reader that widening the path was the fix.

Verified: v3.16.0-alpine is the newest 3.x alpine release tag, and matches the
deployed server v3.16.0. Config still parses; step image unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 .woodpecker/ci-manifests-test.yml             | 23 ++++++++++++++----
 ci/agents/update-audit/prompt.md              | 16 +++++++++++++
 .../infrastructure/image-automation/README.md | 24 +++++++++++++++++++
 3 files changed, 58 insertions(+), 5 deletions(-)

diff --git a/.woodpecker/ci-manifests-test.yml b/.woodpecker/ci-manifests-test.yml
index 2bf913fe..93aae0d9 100644
--- a/.woodpecker/ci-manifests-test.yml
+++ b/.woodpecker/ci-manifests-test.yml
@@ -104,11 +104,24 @@ steps:
   # one it accepts, and the gate stops predicting what the server will do —
   # which is its only purpose.
   #
-  # DRIFT RISK: the server auto-bumps (image-automation), this pin does not.
-  # ImageUpdateAutomation is scoped to `update.path: ./kubernetes`, so it cannot
-  # maintain a tag in .woodpecker/. Widening that path would let a policy own
-  # this line the way it owns the server and agent; until then this is a manual
-  # bump and will lag a server upgrade.
+  # `-alpine` is required, not cosmetic: the plain vX.Y.Z CLI image is
+  # scratch-based (19 MB, 2 layers, no shell) and the loop below needs `for`,
+  # `sed` and `echo` around the binary.
+  #
+  # DRIFT RISK — accepted, and NOT fixable here. The server auto-bumps via
+  # image-automation; this pin cannot, so the two will drift apart.
+  # Image-automation can only patch YAML that is a Kubernetes manifest —
+  # fluxcd/flux2#2826: "Flux can only patch Kubernetes manifests. This means the
+  # YAML manifest must have `apiVersion` and `kind` fields." A Woodpecker
+  # pipeline file has neither, so an `$imagepolicy` marker here is silently
+  # ignored. That is a property of the FILE, not of `update.path`: widening the
+  # automation's path to `./` would buy nothing and would drag 197 unrelated
+  # ansible/esphome YAMLs into the walk. Do not try it — it looks like the fix
+  # and is not.
+  #
+  # So this bump is manual, and drift is caught by the update-audit agent
+  # (ci/agents/update-audit/prompt.md, item 5) which is told to flag this image
+  # whenever it trails the deployed server. If you bump the server, bump here.
   - name: woodpecker-lint
     # depends_on: [] so all four run in parallel and every failure shows in one
     # run. Sequential default would stop at the first, making a contributor
diff --git a/ci/agents/update-audit/prompt.md b/ci/agents/update-audit/prompt.md
index 3fe15f17..571eaa2e 100644
--- a/ci/agents/update-audit/prompt.md
+++ b/ci/agents/update-audit/prompt.md
@@ -59,6 +59,22 @@ For each, find the **current pin in the repo** (Read/Glob/Grep), then look up th
 5. **Images deliberately excluded from automation** — listed in the
    image-automation `README.md` (e.g. vsftpd, otbr, ical-proxy) and any image with
    no `# {"$imagepolicy": ...}` marker. Check whether a newer usable tag exists.
+   This includes **CI step images in `.woodpecker/*.yml`**, which are all manual:
+   image-automation cannot patch those files at all (they are not Kubernetes
+   manifests — no `apiVersion`/`kind` — so markers there are ignored), and no
+   amount of `update.path` widening changes that. Grep `.woodpecker/` for
+   `image:` and check each pin.
+
+   Give `woodpeckerci/woodpecker-cli` in `.woodpecker/ci-manifests-test.yml`
+   particular attention: the woodpecker **server** IS auto-bumped (`3.x`), the
+   CLI is not, so they silently drift apart. The CLI lints pipeline configs
+   against its own schema, so one behind the server can pass a config the server
+   then rejects — the gate stops predicting server behaviour. Report the CLI as
+   needing an update whenever it trails the server version, not only when it
+   trails the newest CLI release. Its tag carries an `-alpine` suffix (the plain
+   tag is scratch-based with no shell and would break the step) — compare the
+   `vX.Y.Z` part, and ignore the `next-*` / `pull_*` development tags, which are
+   the newest by date on that repo and are not releases.
 6. **Ansible roles & OpenTofu providers** — `ansible/requirements.yml` (Galaxy
    roles/collections) and `opentofu/providers.tf` + `opentofu/.terraform.lock.hcl`.
    Report any with a newer published version.
diff --git a/kubernetes/infrastructure/image-automation/README.md b/kubernetes/infrastructure/image-automation/README.md
index 643a9d79..323b1331 100644
--- a/kubernetes/infrastructure/image-automation/README.md
+++ b/kubernetes/infrastructure/image-automation/README.md
@@ -55,6 +55,30 @@ existing `flux-system` GitRepository whose Secret was created by
 | mmwave-configurator | `2.x` | Plain 3-component filter (rejects `addon-`/`-armv7`/`-arm64`/`-amd64` variants) |
 | cnpg-barman-plugin | `0.x` | v-prefix filter; marker lives in upstream-imported manifest (see below) |
 
+## Not enrolled (structurally impossible)
+
+- `woodpeckerci/woodpecker-cli` — the lint image in
+  `.woodpecker/ci-manifests-test.yml`. It publishes a perfectly good
+  version stream (`v3.16.0-alpine`), so unlike the three below this is
+  **not** a tag-format problem. It cannot be enrolled because
+  **image-automation can only patch YAML that is a Kubernetes manifest** —
+  a maintainer states it plainly in fluxcd/flux2#2826: *"Flux can only
+  patch Kubernetes manifests. This means the YAML manifest must have
+  `apiVersion` and `kind` fields."* A Woodpecker pipeline file is plain
+  YAML with neither, so the marker would be silently ignored.
+
+  Note this is **not** fixable by widening `update.path`. The path only
+  controls which directory is walked; objects without `apiVersion`/`kind`
+  are skipped regardless. Widening the path to `./` buys nothing here and
+  puts 197 unrelated ansible/esphome YAML files in the walk, so don't.
+
+  Consequence: the CLI is a **manual** pin while the server and agent
+  auto-bump, so it *will* drift. That is a known, accepted gap, covered
+  by the update-audit agent (item 5) rather than by automation. When
+  bumping the server major/minor, bump this too — it must stay on the
+  latest stable like everything else, and a CLI behind the server can
+  pass a config the server then rejects.
+
 ## Not enrolled (deliberate)
 
 These three images publish no usable version stream — there is nothing
-- 
2.54.0


From 51153788358ed862125c75521709622a61b90660 Mon Sep 17 00:00:00 2001
From: Claude Bot <claude-bot@renner.uno>
Date: Thu, 30 Jul 2026 21:36:49 +0200
Subject: [PATCH 10/14] =?UTF-8?q?ci:=20address=20review=20on=20#460=20?=
 =?UTF-8?q?=E2=80=94=20drop=20an=20unverifiable=20claim,=20dedupe=20ration?=
 =?UTF-8?q?ale?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Review raised two 🟡 points and one nit. Both 🟡 points turned out to be
already answered by this PR's own CI run, but one of them exposed a real
accuracy problem in a comment, which is fixed here.

1. ci-k8s:2026-07-30 might never have been built. Good reasoning — build-k8s
   only mints a dated tag when ci/images/k8s/** or build-ci-images.yml changes
   on a push, and this PR touches neither; secret-rotator:2026-07-30 does not
   transfer, being a separately-gated step. Checked rather than assumed: the
   registry API is closed to this token (no read:package), so the evidence is
   pipeline 3178 on 8a580e34, where kustomize-build and no-digest-pins both ran
   green on that exact tag, with 60 real `ok` lines — a pull failure could not
   produce that. Tag exists. No change needed.

2. v3.16.0-alpine unverified for /bin/sh and the lint invocation. Same run:
   woodpecker-lint emitted 18 `ok` lines, one per .woodpecker/*.yml, which is
   only reachable if the image has a working shell and /bin/woodpecker-cli.

   But checking this found a claim I could NOT verify, which the review had
   independently guessed at: the comment asserted that several deploy-*.yml
   files emit "Additional property timeout is not allowed" warnings. That was
   observed under the v3.9.0 CLI this step used to pin. Whether v3.16.0 still
   emits it is unknown — and a green run cannot settle it, because the loop
   redirects lint output to /tmp/out and prints it only on failure, so warnings
   are invisible exactly when the step passes. Review's hypothesis (schema
   drift in a 7-minor-old CLI, not real debt in those files) is plausible and
   untested. Dropped the claim instead of restating it; the comment now
   documents only the exit-code contract the gate actually rests on.

3. Nit: the depends_on rationale was duplicated verbatim across all four
   steps. Hoisted to one note above `steps:`, with a warning that the key
   itself must stay on every step — it is per-step, so removing it from one
   silently re-serialises that step.

Also updated the PR description, which both reviews flagged as stale: it still
quoted v3.9.0-alpine / v0.10.0 as shipped, so the probing evidence in the body
referred to different tags than the diff. Corrected, with a note recording the
bump rather than quietly rewriting history.

Verified: all four steps still parse with depends_on: [] and unchanged images.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 .woodpecker/ci-manifests-test.yml | 48 +++++++++++++++----------------
 1 file changed, 24 insertions(+), 24 deletions(-)

diff --git a/.woodpecker/ci-manifests-test.yml b/.woodpecker/ci-manifests-test.yml
index 93aae0d9..42a6d593 100644
--- a/.woodpecker/ci-manifests-test.yml
+++ b/.woodpecker/ci-manifests-test.yml
@@ -28,6 +28,12 @@ when:
       - "kubernetes/**"
       - ".woodpecker/**"
 
+# Every step carries `depends_on: []` so all four run in PARALLEL and every
+# failure shows in one run. Woodpecker's default is sequential, which would stop
+# at the first failure and make a contributor fix-and-repush to discover the
+# next — and these four are independent, cheap, offline checks with no ordering
+# between them. Keep it on every step: `depends_on` is per-step, so dropping it
+# from one silently re-serialises that step behind the others.
 steps:
   # Every kustomization must build. This is the check that catches a resource
   # entry pointing at a file that does not exist, a bad patch target, or YAML
@@ -37,11 +43,7 @@ steps:
   # Verified green across all 60 kustomizations in the tree at the time of
   # writing, so this gates new breakage rather than pre-existing debt.
   - name: kustomize-build
-    # depends_on: [] so all four run in parallel and every failure shows in one
-    # run. Sequential default would stop at the first, making a contributor
-    # fix-and-repush to discover the next — and these are independent, cheap,
-    # offline checks with no ordering between them.
-    depends_on: []
+    depends_on: [] # see note above `steps:`
     image: gitea.renner.uno/gitea_admin/ci-k8s:2026-07-30
     commands:
       - |
@@ -90,10 +92,20 @@ steps:
 
   # Lint the pipeline definitions themselves. This is the one that would have
   # prevented the repo-wide CI outage: `woodpecker lint` exits 1 on a config that
-  # fails to compile, while exiting 0 on the "Additional property timeout is not
-  # allowed" warnings that several deploy-*.yml files already carry — so it fails
-  # on real breakage without blocking on pre-existing noise. Both behaviours were
-  # verified against this repo before adding the step.
+  # fails to compile, and exits 0 on mere warnings — so it fails on real breakage
+  # without blocking on pre-existing noise. Both behaviours were verified against
+  # this repo (rc=1 reproducing #443, rc=0 on the tree as it stands).
+  #
+  # An earlier revision of this comment also asserted that several deploy-*.yml
+  # files emit "Additional property timeout is not allowed" warnings. That was
+  # observed under the v3.9.0 CLI this step used to pin, and it is NOT known to
+  # still hold on v3.16.0 — review raised the likelihood that the warning was
+  # schema drift in a 7-minor-old CLI rather than real debt in those files, which
+  # is one of the reasons this pin moved to latest stable. It cannot be confirmed
+  # from a green run either, because the loop below sends lint output to /tmp/out
+  # and prints it only on failure, so warnings are invisible when the step passes.
+  # The claim is dropped rather than restated: what this step guarantees is the
+  # exit-code contract above, which is what the gate actually rests on.
   # Latest stable, per the standing rule that any software uses the latest
   # stable version. Matching the deployed woodpecker-server (v3.16.0) is a
   # CONSEQUENCE of that, not a separate requirement — the server is bumped by
@@ -123,11 +135,7 @@ steps:
   # (ci/agents/update-audit/prompt.md, item 5) which is told to flag this image
   # whenever it trails the deployed server. If you bump the server, bump here.
   - name: woodpecker-lint
-    # depends_on: [] so all four run in parallel and every failure shows in one
-    # run. Sequential default would stop at the first, making a contributor
-    # fix-and-repush to discover the next — and these are independent, cheap,
-    # offline checks with no ordering between them.
-    depends_on: []
+    depends_on: [] # see note above `steps:`
     image: woodpeckerci/woodpecker-cli:v3.16.0-alpine
     commands:
       - |
@@ -154,11 +162,7 @@ steps:
   # where several real bugs lived this year, so it remains a gap worth closing
   # separately rather than one this step quietly implies is covered.
   - name: shellcheck
-    # depends_on: [] so all four run in parallel and every failure shows in one
-    # run. Sequential default would stop at the first, making a contributor
-    # fix-and-repush to discover the next — and these are independent, cheap,
-    # offline checks with no ordering between them.
-    depends_on: []
+    depends_on: [] # see note above `steps:`
     image: koalaman/shellcheck-alpine:v0.11.0
     commands:
       - |
@@ -204,11 +208,7 @@ steps:
   # kustomization.yaml. Only the filename filter excludes it. Drop the filter and
   # this step fails permanently on its own documentation.
   - name: no-digest-pins
-    # depends_on: [] so all four run in parallel and every failure shows in one
-    # run. Sequential default would stop at the first, making a contributor
-    # fix-and-repush to discover the next — and these are independent, cheap,
-    # offline checks with no ordering between them.
-    depends_on: []
+    depends_on: [] # see note above `steps:`
     image: gitea.renner.uno/gitea_admin/ci-k8s:2026-07-30
     commands:
       - |
-- 
2.54.0


From 13798134596471a34b314f538cc8c1f8bb6a74ea Mon Sep 17 00:00:00 2001
From: Claude Bot <claude-bot@renner.uno>
Date: Thu, 30 Jul 2026 21:42:42 +0200
Subject: [PATCH 11/14] docs(ci): correct the file count in the update.path
 argument (197 -> 180)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Review could not reproduce "197 unrelated ansible/esphome YAML files" and
counted 180. The reviewer is right, and the number was wrong twice over.

Recounted in this checkout: ansible/ has 179, esphome/ has 1 -> 180, which
matches the review exactly. My 197 came from also folding in .woodpecker/,
which is wrong on both counts: those files are neither ansible nor esphome,
and they are precisely the directory you WOULD want in the walk, so counting
them as "unrelated" inverts the argument the sentence is making. The composite
is also 198 rather than 197 today — the original count was taken against
master, where .woodpecker/ still had 17 files because ci-manifests-test.yml
did not exist yet.

The conclusion is unchanged (widening update.path is pointless because Flux
cannot patch non-Kubernetes YAML at all, and it would pull in a large number
of unrelated files), but the supporting figure now reproduces, and both places
that cite it are fixed with the derivation shown so it can be re-checked.

Also recorded the review's rule-of-three note where it will actually be read:
kustomize-build and shellcheck share a discovery shape, two copies is
deliberate because the two traps commented around are different, and a third
instance is the signal to factor it into scripts/ci/ — which would pick up
unit tests via ci-scripts-test.yml on the way.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 .woodpecker/ci-manifests-test.yml                 | 15 ++++++++++++---
 .../infrastructure/image-automation/README.md     |  3 ++-
 2 files changed, 14 insertions(+), 4 deletions(-)

diff --git a/.woodpecker/ci-manifests-test.yml b/.woodpecker/ci-manifests-test.yml
index 42a6d593..4bac31cb 100644
--- a/.woodpecker/ci-manifests-test.yml
+++ b/.woodpecker/ci-manifests-test.yml
@@ -28,6 +28,15 @@ when:
       - "kubernetes/**"
       - ".woodpecker/**"
 
+# ADDING A THIRD DISCOVERY CHECK: kustomize-build and shellcheck both hand-roll
+# the same shape — `find`, a floor guard against empty discovery, then a
+# file-fed `while read`. Two copies is deliberate: the two traps being commented
+# around (BusyBox find lacking `-printf`, and a piped loop running in a subshell
+# where `fail` cannot propagate) are different, and both are worth spelling out
+# where they bite. A third instance is the point to stop copying and factor it
+# into a shared script under `scripts/ci/` — which is already covered by
+# ci-scripts-test.yml, so it would gain unit tests in the process.
+#
 # Every step carries `depends_on: []` so all four run in PARALLEL and every
 # failure shows in one run. Woodpecker's default is sequential, which would stop
 # at the first failure and make a contributor fix-and-repush to discover the
@@ -127,9 +136,9 @@ steps:
   # YAML manifest must have `apiVersion` and `kind` fields." A Woodpecker
   # pipeline file has neither, so an `$imagepolicy` marker here is silently
   # ignored. That is a property of the FILE, not of `update.path`: widening the
-  # automation's path to `./` would buy nothing and would drag 197 unrelated
-  # ansible/esphome YAMLs into the walk. Do not try it — it looks like the fix
-  # and is not.
+  # automation's path to `./` would buy nothing and would drag 180 unrelated
+  # ansible/esphome YAMLs into the walk (179 + 1). Do not try it — it looks like
+  # the fix and is not.
   #
   # So this bump is manual, and drift is caught by the update-audit agent
   # (ci/agents/update-audit/prompt.md, item 5) which is told to flag this image
diff --git a/kubernetes/infrastructure/image-automation/README.md b/kubernetes/infrastructure/image-automation/README.md
index 323b1331..440cb519 100644
--- a/kubernetes/infrastructure/image-automation/README.md
+++ b/kubernetes/infrastructure/image-automation/README.md
@@ -70,7 +70,8 @@ existing `flux-system` GitRepository whose Secret was created by
   Note this is **not** fixable by widening `update.path`. The path only
   controls which directory is walked; objects without `apiVersion`/`kind`
   are skipped regardless. Widening the path to `./` buys nothing here and
-  puts 197 unrelated ansible/esphome YAML files in the walk, so don't.
+  puts 180 unrelated ansible/esphome YAML files in the walk, so don't.
+  (179 under `ansible/` + 1 under `esphome/`.)
 
   Consequence: the CLI is a **manual** pin while the server and agent
   auto-bump, so it *will* drift. That is a known, accepted gap, covered
-- 
2.54.0


From ba052332bf8e47462a4ec489c7c145f9a2333050 Mon Sep 17 00:00:00 2001
From: Claude Bot <claude-bot@renner.uno>
Date: Thu, 30 Jul 2026 21:47:11 +0200
Subject: [PATCH 12/14] =?UTF-8?q?docs(ci):=20drop=20snapshot=20counts=20fr?=
 =?UTF-8?q?om=20comments=20=E2=80=94=20they=20decay=20and=20prove=20nothin?=
 =?UTF-8?q?g?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The file count was meaningless and would be wrong tomorrow. Correcting it from
197 to 180 last commit was the wrong fix: a number that has to be re-derived
every time the tree changes does not belong in a comment at all. It cost two
review rounds — one to flag that 197 did not reproduce, one to correct it —
and neither round was about anything that mattered, because the count was
never the argument. Widening update.path is pointless because Flux cannot
patch non-Kubernetes YAML at all; the size of the walk was only ever a
secondary aside.

Swept every snapshot figure I had introduced, not just the one flagged:

  * "180 unrelated ansible/esphome YAMLs (179 + 1)" -> "the whole ansible/ and
    esphome/ trees", in both the pipeline comment and the README.
  * "builds 60 kustomizations" -> "builds every kustomization in the tree".
  * "verified green across all 60 kustomizations" -> "across every
    kustomization", keeping the as-of-writing hedge that made it a historical
    claim rather than a live one.
  * "expected ~60" in the discovery-floor failure message -> "floor is 50",
    which cites the actual guard on the line above instead of a tree size that
    drifts away from it. The message already prints the real $count.
  * "scratch-based (19 MB, 2 layers, no shell)" -> "ships no shell", plus what
    the reader actually needs: a step on the plain tag fails to START rather
    than failing a check, which is the confusing symptom. The byte size would
    have gone stale on the next upstream release.

The floor of 50 itself stays — it is a deliberately loose guard against
discovery breaking, not a description of the tree, and it is asserted in code
where a wrong value fails loudly instead of merely misinforming.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 .woodpecker/ci-manifests-test.yml             | 22 +++++++++----------
 .../infrastructure/image-automation/README.md |  4 ++--
 2 files changed, 13 insertions(+), 13 deletions(-)

diff --git a/.woodpecker/ci-manifests-test.yml b/.woodpecker/ci-manifests-test.yml
index 4bac31cb..990df49f 100644
--- a/.woodpecker/ci-manifests-test.yml
+++ b/.woodpecker/ci-manifests-test.yml
@@ -19,8 +19,8 @@
 when:
   # Path-filtered on purpose, unlike ci-scripts-test.yml. That pipeline is one
   # step on one already-cached image, so running it everywhere is free; this one
-  # pulls three images and builds 60 kustomizations, which is real cost to put on
-  # a docs-only PR. The filter covers everything the steps actually read:
+  # pulls three images and builds every kustomization in the tree, which is real
+  # cost to put on a docs-only PR. The filter covers everything the steps actually read:
   # kubernetes/** for the manifests and scripts, .woodpecker/** for the lint (and
   # so an edit to this file re-runs its own checks).
   - event: pull_request
@@ -49,8 +49,8 @@ steps:
   # that is fine in isolation but broken once overlaid — the class of error that
   # otherwise surfaces as a Flux reconcile failure after merge.
   #
-  # Verified green across all 60 kustomizations in the tree at the time of
-  # writing, so this gates new breakage rather than pre-existing debt.
+  # Verified green across every kustomization in the tree when this was added,
+  # so this gates new breakage rather than pre-existing debt.
   - name: kustomize-build
     depends_on: [] # see note above `steps:`
     image: gitea.renner.uno/gitea_admin/ci-k8s:2026-07-30
@@ -72,7 +72,7 @@ steps:
         # step useless while looking healthy.
         count=$(printf '%s\n' "$dirs" | grep -c . || true)
         if [ "$count" -lt 50 ]; then
-          echo "FAIL: found only $count kustomizations; expected ~60."
+          echo "FAIL: found only $count kustomizations; floor is 50."
           echo "      If the tree legitimately shrank, lower this floor in"
           echo "      .woodpecker/ci-manifests-test.yml — do not delete it."
           echo "      Otherwise discovery is broken — refusing to report success."
@@ -125,9 +125,9 @@ steps:
   # one it accepts, and the gate stops predicting what the server will do —
   # which is its only purpose.
   #
-  # `-alpine` is required, not cosmetic: the plain vX.Y.Z CLI image is
-  # scratch-based (19 MB, 2 layers, no shell) and the loop below needs `for`,
-  # `sed` and `echo` around the binary.
+  # `-alpine` is required, not cosmetic: the plain vX.Y.Z CLI image ships no
+  # shell, and the loop below needs `for`, `sed` and `echo` around the binary.
+  # A step using the plain tag fails to start rather than failing a check.
   #
   # DRIFT RISK — accepted, and NOT fixable here. The server auto-bumps via
   # image-automation; this pin cannot, so the two will drift apart.
@@ -136,9 +136,9 @@ steps:
   # YAML manifest must have `apiVersion` and `kind` fields." A Woodpecker
   # pipeline file has neither, so an `$imagepolicy` marker here is silently
   # ignored. That is a property of the FILE, not of `update.path`: widening the
-  # automation's path to `./` would buy nothing and would drag 180 unrelated
-  # ansible/esphome YAMLs into the walk (179 + 1). Do not try it — it looks like
-  # the fix and is not.
+  # automation's path to `./` would buy nothing, and would additionally drag the
+  # whole ansible/ and esphome/ trees into the walk. Do not try it — it looks
+  # like the fix and is not.
   #
   # So this bump is manual, and drift is caught by the update-audit agent
   # (ci/agents/update-audit/prompt.md, item 5) which is told to flag this image
diff --git a/kubernetes/infrastructure/image-automation/README.md b/kubernetes/infrastructure/image-automation/README.md
index 440cb519..dc52819b 100644
--- a/kubernetes/infrastructure/image-automation/README.md
+++ b/kubernetes/infrastructure/image-automation/README.md
@@ -70,8 +70,8 @@ existing `flux-system` GitRepository whose Secret was created by
   Note this is **not** fixable by widening `update.path`. The path only
   controls which directory is walked; objects without `apiVersion`/`kind`
   are skipped regardless. Widening the path to `./` buys nothing here and
-  puts 180 unrelated ansible/esphome YAML files in the walk, so don't.
-  (179 under `ansible/` + 1 under `esphome/`.)
+  additionally pulls the whole `ansible/` and `esphome/` trees into the
+  walk, so don't.
 
   Consequence: the CLI is a **manual** pin while the server and agent
   auto-bump, so it *will* drift. That is a known, accepted gap, covered
-- 
2.54.0


From b1f5d6dcbc5fa42e33457904edcbf87ee18b91ea Mon Sep 17 00:00:00 2001
From: Claude Bot <claude-bot@renner.uno>
Date: Thu, 30 Jul 2026 21:49:42 +0200
Subject: [PATCH 13/14] ci: explain the ci-k8s tag split and track the stale
 pins (#461)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Review noted this file pins ci-k8s at a newer dated tag than e2e.yml,
e2e-test.yml, alert-monitor.yml and issue-fix.yml, and asked for either a
stated reason or a follow-up so it does not read as an oversight.

There is no reason — the other four are stale. Per the latest-stable rule all
five should sit on the newest dated tag.

Not bumped here on purpose. Those four pipelines exercise different tooling in
the image and three are cron- or event-triggered, so none of them run on this
PR. Bumping them in this branch would ship four untested changes under cover of
a green run that never touched them — the same shape as the digest decay in
#458, where breakage sat unnoticed precisely because the affected jobs did not
run for months. Each wants its own verification, so it is tracked as #461
rather than smuggled in here.

Comment added above kustomize-build so the split is explained where a reader
meets it, pointing at the issue rather than restating the whole argument.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 .woodpecker/ci-manifests-test.yml | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/.woodpecker/ci-manifests-test.yml b/.woodpecker/ci-manifests-test.yml
index 990df49f..8734c0e5 100644
--- a/.woodpecker/ci-manifests-test.yml
+++ b/.woodpecker/ci-manifests-test.yml
@@ -51,6 +51,13 @@ steps:
   #
   # Verified green across every kustomization in the tree when this was added,
   # so this gates new breakage rather than pre-existing debt.
+  #
+  # ci-k8s TAG SPLIT: this file pins a newer dated tag than e2e.yml, e2e-test.yml,
+  # alert-monitor.yml and issue-fix.yml, which still sit on an older one. That is
+  # staleness in those four, not a deliberate split here — everything should be on
+  # the newest dated tag. They are not bumped here because none of them are
+  # triggered by this PR, so it would ship untested changes under cover of a green
+  # run that never touched them. Tracked in #461.
   - name: kustomize-build
     depends_on: [] # see note above `steps:`
     image: gitea.renner.uno/gitea_admin/ci-k8s:2026-07-30
-- 
2.54.0


From 39dc23a9993cf97de799c4bc13a7ba85b0011e7a Mon Sep 17 00:00:00 2001
From: Claude Bot <claude-bot@renner.uno>
Date: Thu, 30 Jul 2026 21:55:33 +0200
Subject: [PATCH 14/14] ci: document why the shellcheck discovery floor stays
 at 1
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Review flagged that shellcheck's floor (1) is far weaker than kustomize-build's
(50), so it catches total discovery failure but not partial loss.

Correct, and kept as-is with the reasoning written down. The kustomize floor is
loose relative to the tree it guards, which is what lets it catch gross partial
loss without tripping on churn. The shell-script set is small enough that any
floor tight enough to catch partial loss would sit inside normal churn:
deleting one script would fail CI spuriously, and the fix a hurried person
reaches for is lowering the floor — which is how a guard becomes decoration. A
floor that must be edited whenever the tree changes is not a guard, it is a
snapshot with an exit code, and this file just had all its snapshots removed
for the same reason.

So it catches the failure that actually occurred (discovery silently returning
nothing, step green having checked nothing) and not partial loss. Noted as
worth revisiting if kubernetes/ grows a lot more shell, when a loose
proportionate floor becomes possible.

The review's other point — that the image pins and the /bin/sh and
/bin/woodpecker-cli claims are not checkable from a static checkout, and that a
green run is the real confirmation — is now settled by pipeline 3185 on
b1f5d6dc: all four steps green, with 60 / 18 / 6 ok lines from kustomize-build,
woodpecker-lint and shellcheck respectively. Those counts are cited here as
run-time evidence, not written into the file.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 .woodpecker/ci-manifests-test.yml | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/.woodpecker/ci-manifests-test.yml b/.woodpecker/ci-manifests-test.yml
index 8734c0e5..76760de8 100644
--- a/.woodpecker/ci-manifests-test.yml
+++ b/.woodpecker/ci-manifests-test.yml
@@ -186,6 +186,21 @@ steps:
         files=$(find kubernetes -name '*.sh' | sort)
         # Same no-op guard as kustomize-build: if discovery ever returns nothing,
         # fail loudly instead of passing having checked no files.
+        #
+        # This floor is 1 while kustomize-build's is 50, and the asymmetry is
+        # deliberate rather than an oversight. That floor is loose relative to the
+        # tree it guards, so it catches both total failure and gross partial loss
+        # while tolerating normal churn. The shell-script set is small enough that
+        # any floor tight enough to catch PARTIAL loss would sit inside normal
+        # churn — deleting one script would fail CI spuriously, and the fix a
+        # hurried person reaches for is lowering the floor, which is how a guard
+        # quietly becomes decoration. A floor that has to be edited whenever the
+        # tree changes is not a guard, it is a snapshot with an exit code.
+        #
+        # So this catches the failure that actually happened here (discovery
+        # silently returning nothing, step green having checked no files) and not
+        # partial loss. Worth revisiting only if kubernetes/ grows a lot more
+        # shell, at which point a loose proportionate floor becomes possible.
         count=$(printf '%s\n' "$files" | grep -c . || true)
         if [ "$count" -lt 1 ]; then
           echo "FAIL: no shell scripts found under kubernetes/ — discovery is broken."
-- 
2.54.0