diff --git a/.woodpecker/ci-manifests-test.yml b/.woodpecker/ci-manifests-test.yml new file mode 100644 index 00000000..76760de8 --- /dev/null +++ b/.woodpecker/ci-manifests-test.yml @@ -0,0 +1,254 @@ +# Validate the GitOps tree and the pipeline definitions on PRs that touch them. +# +# Nothing covered kubernetes/** before this. deploy-infrastructure.yml only +# gates opentofu/**, and ci-scripts-test.yml only runs scripts/ci/ unit tests — +# so a broken manifest reached the cluster and was caught by Flux failing to +# reconcile, or by a human noticing. Three separate defects on 2026-07-30 would +# have been caught here instead: +# +# * an unquoted YAML scalar containing ": " in .woodpecker/build-ci-images.yml, +# which made EVERY pipeline in the repo fail to compile (#443). `yq` parsed +# it happily — it is valid YAML, just a map instead of a string — so +# "it parses" was mistaken for "it is valid". Only woodpecker lint catches it. +# * a rotator pinned to an image digest that no longer existed (#458). +# * manifests whose kustomize wiring was only ever checked by hand. +# +# Deliberately CHEAP and offline: no cluster, no registry, no secrets. Everything +# here is a static check that either passes or fails on the checkout alone, so it +# can gate PRs without becoming a bottleneck or a flake source. +when: + # Path-filtered on purpose, unlike ci-scripts-test.yml. That pipeline is one + # step on one already-cached image, so running it everywhere is free; this one + # pulls three images and builds every kustomization in the tree, which is real + # cost to put on a docs-only PR. The filter covers everything the steps actually read: + # kubernetes/** for the manifests and scripts, .woodpecker/** for the lint (and + # so an edit to this file re-runs its own checks). + - event: pull_request + path: + - "kubernetes/**" + - ".woodpecker/**" + +# ADDING A THIRD DISCOVERY CHECK: kustomize-build and shellcheck both hand-roll +# the same shape — `find`, a floor guard against empty discovery, then a +# file-fed `while read`. Two copies is deliberate: the two traps being commented +# around (BusyBox find lacking `-printf`, and a piped loop running in a subshell +# where `fail` cannot propagate) are different, and both are worth spelling out +# where they bite. A third instance is the point to stop copying and factor it +# into a shared script under `scripts/ci/` — which is already covered by +# ci-scripts-test.yml, so it would gain unit tests in the process. +# +# Every step carries `depends_on: []` so all four run in PARALLEL and every +# failure shows in one run. Woodpecker's default is sequential, which would stop +# at the first failure and make a contributor fix-and-repush to discover the +# next — and these four are independent, cheap, offline checks with no ordering +# between them. Keep it on every step: `depends_on` is per-step, so dropping it +# from one silently re-serialises that step behind the others. +steps: + # Every kustomization must build. This is the check that catches a resource + # entry pointing at a file that does not exist, a bad patch target, or YAML + # that is fine in isolation but broken once overlaid — the class of error that + # otherwise surfaces as a Flux reconcile failure after merge. + # + # Verified green across every kustomization in the tree when this was added, + # so this gates new breakage rather than pre-existing debt. + # + # ci-k8s TAG SPLIT: this file pins a newer dated tag than e2e.yml, e2e-test.yml, + # alert-monitor.yml and issue-fix.yml, which still sit on an older one. That is + # staleness in those four, not a deliberate split here — everything should be on + # the newest dated tag. They are not bumped here because none of them are + # triggered by this PR, so it would ship untested changes under cover of a green + # run that never touched them. Tracked in #461. + - name: kustomize-build + depends_on: [] # see note above `steps:` + image: gitea.renner.uno/gitea_admin/ci-k8s:2026-07-30 + commands: + - | + set -eu + # NOT `find -printf '%h\n'`: that is a GNU findutils extension and this + # image ships BusyBox find (v1.37.0), which rejects it outright with + # "find: unrecognized: -printf". Because the result feeds a command + # substitution used as a word list, the failure is SILENT — empty output, + # zero loop iterations, fail=0, step exits green having validated + # nothing. That is exactly what this pipeline shipped in its first + # revision, and CI reported it as passing. + dirs=$(find kubernetes -name kustomization.yaml | sed 's|/kustomization.yaml$||' | sort) + + # Assert the discovery actually found something. Without this, any future + # breakage of the find/sed above degrades back into a green no-op rather + # than a failure — the specific trap that made the first revision of this + # step useless while looking healthy. + count=$(printf '%s\n' "$dirs" | grep -c . || true) + if [ "$count" -lt 50 ]; then + echo "FAIL: found only $count kustomizations; floor is 50." + echo " If the tree legitimately shrank, lower this floor in" + echo " .woodpecker/ci-manifests-test.yml — do not delete it." + echo " Otherwise discovery is broken — refusing to report success." + exit 1 + fi + echo "discovered $count kustomizations" + + # `while read` fed from a FILE, not a pipe: a pipe would run the loop in + # a subshell where `fail` cannot propagate, so a failing build would exit + # 0 — the same silent-pass shape this step already shipped once. Reading + # a file keeps the loop in the current shell, and read -r handles paths + # containing spaces without IFS games. + printf '%s\n' "$dirs" > /tmp/dirs + fail=0 + while IFS= read -r d; do + [ -n "$d" ] || continue + if kubectl kustomize "$d" >/dev/null 2>/tmp/err; then + echo "ok $d" + else + echo "FAIL $d" + sed 's/^/ /' /tmp/err + fail=1 + fi + done < /tmp/dirs + exit $fail + + # Lint the pipeline definitions themselves. This is the one that would have + # prevented the repo-wide CI outage: `woodpecker lint` exits 1 on a config that + # fails to compile, and exits 0 on mere warnings — so it fails on real breakage + # without blocking on pre-existing noise. Both behaviours were verified against + # this repo (rc=1 reproducing #443, rc=0 on the tree as it stands). + # + # An earlier revision of this comment also asserted that several deploy-*.yml + # files emit "Additional property timeout is not allowed" warnings. That was + # observed under the v3.9.0 CLI this step used to pin, and it is NOT known to + # still hold on v3.16.0 — review raised the likelihood that the warning was + # schema drift in a 7-minor-old CLI rather than real debt in those files, which + # is one of the reasons this pin moved to latest stable. It cannot be confirmed + # from a green run either, because the loop below sends lint output to /tmp/out + # and prints it only on failure, so warnings are invisible when the step passes. + # The claim is dropped rather than restated: what this step guarantees is the + # exit-code contract above, which is what the gate actually rests on. + # Latest stable, per the standing rule that any software uses the latest + # stable version. Matching the deployed woodpecker-server (v3.16.0) is a + # CONSEQUENCE of that, not a separate requirement — the server is bumped by + # image-automation within the 3.x range, so both land on the same release. + # + # It matters here because the CLI compiles the config against its own schema: + # one behind the server can pass a config the server then rejects, or reject + # one it accepts, and the gate stops predicting what the server will do — + # which is its only purpose. + # + # `-alpine` is required, not cosmetic: the plain vX.Y.Z CLI image ships no + # shell, and the loop below needs `for`, `sed` and `echo` around the binary. + # A step using the plain tag fails to start rather than failing a check. + # + # DRIFT RISK — accepted, and NOT fixable here. The server auto-bumps via + # image-automation; this pin cannot, so the two will drift apart. + # Image-automation can only patch YAML that is a Kubernetes manifest — + # fluxcd/flux2#2826: "Flux can only patch Kubernetes manifests. This means the + # YAML manifest must have `apiVersion` and `kind` fields." A Woodpecker + # pipeline file has neither, so an `$imagepolicy` marker here is silently + # ignored. That is a property of the FILE, not of `update.path`: widening the + # automation's path to `./` would buy nothing, and would additionally drag the + # whole ansible/ and esphome/ trees into the walk. Do not try it — it looks + # like the fix and is not. + # + # So this bump is manual, and drift is caught by the update-audit agent + # (ci/agents/update-audit/prompt.md, item 5) which is told to flag this image + # whenever it trails the deployed server. If you bump the server, bump here. + - name: woodpecker-lint + depends_on: [] # see note above `steps:` + image: woodpeckerci/woodpecker-cli:v3.16.0-alpine + commands: + - | + set -eu + fail=0 + for f in .woodpecker/*.yml; do + if /bin/woodpecker-cli lint "$f" >/tmp/out 2>&1; then + echo "ok $f" + else + echo "FAIL $f" + sed 's/^/ /' /tmp/out + fail=1 + fi + done + exit $fail + + # Standalone shell scripts under kubernetes/ (sidecar and bootstrap scripts + # mounted via configMapGenerator). Severity is pinned to `error` on purpose: + # style and info findings would fail the build on existing scripts for no + # safety gain. All scripts in the tree are clean at this level today. + # + # NOTE this does NOT cover shell embedded in manifests (init containers, probe + # exec blocks) — shellcheck cannot see inside YAML. That embedded shell is + # where several real bugs lived this year, so it remains a gap worth closing + # separately rather than one this step quietly implies is covered. + - name: shellcheck + depends_on: [] # see note above `steps:` + image: koalaman/shellcheck-alpine:v0.11.0 + commands: + - | + set -eu + files=$(find kubernetes -name '*.sh' | sort) + # Same no-op guard as kustomize-build: if discovery ever returns nothing, + # fail loudly instead of passing having checked no files. + # + # This floor is 1 while kustomize-build's is 50, and the asymmetry is + # deliberate rather than an oversight. That floor is loose relative to the + # tree it guards, so it catches both total failure and gross partial loss + # while tolerating normal churn. The shell-script set is small enough that + # any floor tight enough to catch PARTIAL loss would sit inside normal + # churn — deleting one script would fail CI spuriously, and the fix a + # hurried person reaches for is lowering the floor, which is how a guard + # quietly becomes decoration. A floor that has to be edited whenever the + # tree changes is not a guard, it is a snapshot with an exit code. + # + # So this catches the failure that actually happened here (discovery + # silently returning nothing, step green having checked no files) and not + # partial loss. Worth revisiting only if kubernetes/ grows a lot more + # shell, at which point a loose proportionate floor becomes possible. + count=$(printf '%s\n' "$files" | grep -c . || true) + if [ "$count" -lt 1 ]; then + echo "FAIL: no shell scripts found under kubernetes/ — discovery is broken." + exit 1 + fi + echo "discovered $count shell scripts" + + # Same file-fed while-read reasoning as kustomize-build above. + printf '%s\n' "$files" > /tmp/files + fail=0 + while IFS= read -r f; do + [ -n "$f" ] || continue + if shellcheck -S error "$f"; then + echo "ok $f" + else + echo "FAIL $f" + fail=1 + fi + done < /tmp/files + exit $fail + + # Guard the convention that cost a day to diagnose: registry-local images are + # pinned by DATED TAG, never by digest. Digests here are garbage-collected once + # :latest and the dated tags move past them — and that happens on edits to + # .woodpecker/build-ci-images.yml, a shared file, so an unrelated image addition + # silently orphans the manifest a rotator was pinned to. 11 rotators were dead + # this way and would have failed on their first scheduled run months later. + # + # Scoped to the rotation directory, which is where the convention is documented + # and where the failure actually bit. + # + # The `grep -v kustomization.yaml` is load-bearing, not tidying: that file + # documents this very rule, so its comment contains the literal string + # `image:.*@sha256` and therefore MATCHES the pattern. Anchoring on `image:` + # does not prevent that — verified by running the bare grep, which returns + # kustomization.yaml. Only the filename filter excludes it. Drop the filter and + # this step fails permanently on its own documentation. + - name: no-digest-pins + depends_on: [] # see note above `steps:` + image: gitea.renner.uno/gitea_admin/ci-k8s:2026-07-30 + commands: + - | + set -eu + hits=$(grep -l 'image:.*@sha256' kubernetes/infrastructure/openbao/rotation/*.yaml 2>/dev/null \ + | grep -v kustomization.yaml || true) + if [ -n "$hits" ]; then + echo "FAIL: digest-pinned images found; use a dated tag (see rotation/kustomization.yaml):" + echo "$hits" | sed 's/^/ /' + exit 1 + fi + echo "ok no digest pins under openbao/rotation/" diff --git a/ci/agents/update-audit/prompt.md b/ci/agents/update-audit/prompt.md index 3fe15f17..571eaa2e 100644 --- a/ci/agents/update-audit/prompt.md +++ b/ci/agents/update-audit/prompt.md @@ -59,6 +59,22 @@ For each, find the **current pin in the repo** (Read/Glob/Grep), then look up th 5. **Images deliberately excluded from automation** — listed in the image-automation `README.md` (e.g. vsftpd, otbr, ical-proxy) and any image with no `# {"$imagepolicy": ...}` marker. Check whether a newer usable tag exists. + This includes **CI step images in `.woodpecker/*.yml`**, which are all manual: + image-automation cannot patch those files at all (they are not Kubernetes + manifests — no `apiVersion`/`kind` — so markers there are ignored), and no + amount of `update.path` widening changes that. Grep `.woodpecker/` for + `image:` and check each pin. + + Give `woodpeckerci/woodpecker-cli` in `.woodpecker/ci-manifests-test.yml` + particular attention: the woodpecker **server** IS auto-bumped (`3.x`), the + CLI is not, so they silently drift apart. The CLI lints pipeline configs + against its own schema, so one behind the server can pass a config the server + then rejects — the gate stops predicting server behaviour. Report the CLI as + needing an update whenever it trails the server version, not only when it + trails the newest CLI release. Its tag carries an `-alpine` suffix (the plain + tag is scratch-based with no shell and would break the step) — compare the + `vX.Y.Z` part, and ignore the `next-*` / `pull_*` development tags, which are + the newest by date on that repo and are not releases. 6. **Ansible roles & OpenTofu providers** — `ansible/requirements.yml` (Galaxy roles/collections) and `opentofu/providers.tf` + `opentofu/.terraform.lock.hcl`. Report any with a newer published version. diff --git a/kubernetes/infrastructure/image-automation/README.md b/kubernetes/infrastructure/image-automation/README.md index 643a9d79..dc52819b 100644 --- a/kubernetes/infrastructure/image-automation/README.md +++ b/kubernetes/infrastructure/image-automation/README.md @@ -55,6 +55,31 @@ existing `flux-system` GitRepository whose Secret was created by | mmwave-configurator | `2.x` | Plain 3-component filter (rejects `addon-`/`-armv7`/`-arm64`/`-amd64` variants) | | cnpg-barman-plugin | `0.x` | v-prefix filter; marker lives in upstream-imported manifest (see below) | +## Not enrolled (structurally impossible) + +- `woodpeckerci/woodpecker-cli` — the lint image in + `.woodpecker/ci-manifests-test.yml`. It publishes a perfectly good + version stream (`v3.16.0-alpine`), so unlike the three below this is + **not** a tag-format problem. It cannot be enrolled because + **image-automation can only patch YAML that is a Kubernetes manifest** — + a maintainer states it plainly in fluxcd/flux2#2826: *"Flux can only + patch Kubernetes manifests. This means the YAML manifest must have + `apiVersion` and `kind` fields."* A Woodpecker pipeline file is plain + YAML with neither, so the marker would be silently ignored. + + Note this is **not** fixable by widening `update.path`. The path only + controls which directory is walked; objects without `apiVersion`/`kind` + are skipped regardless. Widening the path to `./` buys nothing here and + additionally pulls the whole `ansible/` and `esphome/` trees into the + walk, so don't. + + Consequence: the CLI is a **manual** pin while the server and agent + auto-bump, so it *will* drift. That is a known, accepted gap, covered + by the update-audit agent (item 5) rather than by automation. When + bumping the server major/minor, bump this too — it must stay on the + latest stable like everything else, and a CLI behind the server can + pass a config the server then rejects. + ## Not enrolled (deliberate) These three images publish no usable version stream — there is nothing