|
|
|
@@ -0,0 +1,254 @@
|
|
|
|
|
# Validate the GitOps tree and the pipeline definitions on PRs that touch them.
|
|
|
|
|
#
|
|
|
|
|
# Nothing covered kubernetes/** before this. deploy-infrastructure.yml only
|
|
|
|
|
# gates opentofu/**, and ci-scripts-test.yml only runs scripts/ci/ unit tests —
|
|
|
|
|
# so a broken manifest reached the cluster and was caught by Flux failing to
|
|
|
|
|
# reconcile, or by a human noticing. Three separate defects on 2026-07-30 would
|
|
|
|
|
# have been caught here instead:
|
|
|
|
|
#
|
|
|
|
|
# * an unquoted YAML scalar containing ": " in .woodpecker/build-ci-images.yml,
|
|
|
|
|
# which made EVERY pipeline in the repo fail to compile (#443). `yq` parsed
|
|
|
|
|
# it happily — it is valid YAML, just a map instead of a string — so
|
|
|
|
|
# "it parses" was mistaken for "it is valid". Only woodpecker lint catches it.
|
|
|
|
|
# * a rotator pinned to an image digest that no longer existed (#458).
|
|
|
|
|
# * manifests whose kustomize wiring was only ever checked by hand.
|
|
|
|
|
#
|
|
|
|
|
# Deliberately CHEAP and offline: no cluster, no registry, no secrets. Everything
|
|
|
|
|
# here is a static check that either passes or fails on the checkout alone, so it
|
|
|
|
|
# can gate PRs without becoming a bottleneck or a flake source.
|
|
|
|
|
when:
|
|
|
|
|
# Path-filtered on purpose, unlike ci-scripts-test.yml. That pipeline is one
|
|
|
|
|
# step on one already-cached image, so running it everywhere is free; this one
|
|
|
|
|
# pulls three images and builds every kustomization in the tree, which is real
|
|
|
|
|
# cost to put on a docs-only PR. The filter covers everything the steps actually read:
|
|
|
|
|
# kubernetes/** for the manifests and scripts, .woodpecker/** for the lint (and
|
|
|
|
|
# so an edit to this file re-runs its own checks).
|
|
|
|
|
- event: pull_request
|
|
|
|
|
path:
|
|
|
|
|
- "kubernetes/**"
|
|
|
|
|
- ".woodpecker/**"
|
|
|
|
|
|
|
|
|
|
# ADDING A THIRD DISCOVERY CHECK: kustomize-build and shellcheck both hand-roll
|
|
|
|
|
# the same shape — `find`, a floor guard against empty discovery, then a
|
|
|
|
|
# file-fed `while read`. Two copies is deliberate: the two traps being commented
|
|
|
|
|
# around (BusyBox find lacking `-printf`, and a piped loop running in a subshell
|
|
|
|
|
# where `fail` cannot propagate) are different, and both are worth spelling out
|
|
|
|
|
# where they bite. A third instance is the point to stop copying and factor it
|
|
|
|
|
# into a shared script under `scripts/ci/` — which is already covered by
|
|
|
|
|
# ci-scripts-test.yml, so it would gain unit tests in the process.
|
|
|
|
|
#
|
|
|
|
|
# Every step carries `depends_on: []` so all four run in PARALLEL and every
|
|
|
|
|
# failure shows in one run. Woodpecker's default is sequential, which would stop
|
|
|
|
|
# at the first failure and make a contributor fix-and-repush to discover the
|
|
|
|
|
# next — and these four are independent, cheap, offline checks with no ordering
|
|
|
|
|
# between them. Keep it on every step: `depends_on` is per-step, so dropping it
|
|
|
|
|
# from one silently re-serialises that step behind the others.
|
|
|
|
|
steps:
|
|
|
|
|
# Every kustomization must build. This is the check that catches a resource
|
|
|
|
|
# entry pointing at a file that does not exist, a bad patch target, or YAML
|
|
|
|
|
# that is fine in isolation but broken once overlaid — the class of error that
|
|
|
|
|
# otherwise surfaces as a Flux reconcile failure after merge.
|
|
|
|
|
#
|
|
|
|
|
# Verified green across every kustomization in the tree when this was added,
|
|
|
|
|
# so this gates new breakage rather than pre-existing debt.
|
|
|
|
|
#
|
|
|
|
|
# ci-k8s TAG SPLIT: this file pins a newer dated tag than e2e.yml, e2e-test.yml,
|
|
|
|
|
# alert-monitor.yml and issue-fix.yml, which still sit on an older one. That is
|
|
|
|
|
# staleness in those four, not a deliberate split here — everything should be on
|
|
|
|
|
# the newest dated tag. They are not bumped here because none of them are
|
|
|
|
|
# triggered by this PR, so it would ship untested changes under cover of a green
|
|
|
|
|
# run that never touched them. Tracked in #461.
|
|
|
|
|
- name: kustomize-build
|
|
|
|
|
depends_on: [] # see note above `steps:`
|
|
|
|
|
image: gitea.renner.uno/gitea_admin/ci-k8s:2026-07-30
|
|
|
|
|
commands:
|
|
|
|
|
- |
|
|
|
|
|
set -eu
|
|
|
|
|
# NOT `find -printf '%h\n'`: that is a GNU findutils extension and this
|
|
|
|
|
# image ships BusyBox find (v1.37.0), which rejects it outright with
|
|
|
|
|
# "find: unrecognized: -printf". Because the result feeds a command
|
|
|
|
|
# substitution used as a word list, the failure is SILENT — empty output,
|
|
|
|
|
# zero loop iterations, fail=0, step exits green having validated
|
|
|
|
|
# nothing. That is exactly what this pipeline shipped in its first
|
|
|
|
|
# revision, and CI reported it as passing.
|
|
|
|
|
dirs=$(find kubernetes -name kustomization.yaml | sed 's|/kustomization.yaml$||' | sort)
|
|
|
|
|
|
|
|
|
|
# Assert the discovery actually found something. Without this, any future
|
|
|
|
|
# breakage of the find/sed above degrades back into a green no-op rather
|
|
|
|
|
# than a failure — the specific trap that made the first revision of this
|
|
|
|
|
# step useless while looking healthy.
|
|
|
|
|
count=$(printf '%s\n' "$dirs" | grep -c . || true)
|
|
|
|
|
if [ "$count" -lt 50 ]; then
|
|
|
|
|
echo "FAIL: found only $count kustomizations; floor is 50."
|
|
|
|
|
echo " If the tree legitimately shrank, lower this floor in"
|
|
|
|
|
echo " .woodpecker/ci-manifests-test.yml — do not delete it."
|
|
|
|
|
echo " Otherwise discovery is broken — refusing to report success."
|
|
|
|
|
exit 1
|
|
|
|
|
fi
|
|
|
|
|
echo "discovered $count kustomizations"
|
|
|
|
|
|
|
|
|
|
# `while read` fed from a FILE, not a pipe: a pipe would run the loop in
|
|
|
|
|
# a subshell where `fail` cannot propagate, so a failing build would exit
|
|
|
|
|
# 0 — the same silent-pass shape this step already shipped once. Reading
|
|
|
|
|
# a file keeps the loop in the current shell, and read -r handles paths
|
|
|
|
|
# containing spaces without IFS games.
|
|
|
|
|
printf '%s\n' "$dirs" > /tmp/dirs
|
|
|
|
|
fail=0
|
|
|
|
|
while IFS= read -r d; do
|
|
|
|
|
[ -n "$d" ] || continue
|
|
|
|
|
if kubectl kustomize "$d" >/dev/null 2>/tmp/err; then
|
|
|
|
|
echo "ok $d"
|
|
|
|
|
else
|
|
|
|
|
echo "FAIL $d"
|
|
|
|
|
sed 's/^/ /' /tmp/err
|
|
|
|
|
fail=1
|
|
|
|
|
fi
|
|
|
|
|
done < /tmp/dirs
|
|
|
|
|
exit $fail
|
|
|
|
|
|
|
|
|
|
# Lint the pipeline definitions themselves. This is the one that would have
|
|
|
|
|
# prevented the repo-wide CI outage: `woodpecker lint` exits 1 on a config that
|
|
|
|
|
# fails to compile, and exits 0 on mere warnings — so it fails on real breakage
|
|
|
|
|
# without blocking on pre-existing noise. Both behaviours were verified against
|
|
|
|
|
# this repo (rc=1 reproducing #443, rc=0 on the tree as it stands).
|
|
|
|
|
#
|
|
|
|
|
# An earlier revision of this comment also asserted that several deploy-*.yml
|
|
|
|
|
# files emit "Additional property timeout is not allowed" warnings. That was
|
|
|
|
|
# observed under the v3.9.0 CLI this step used to pin, and it is NOT known to
|
|
|
|
|
# still hold on v3.16.0 — review raised the likelihood that the warning was
|
|
|
|
|
# schema drift in a 7-minor-old CLI rather than real debt in those files, which
|
|
|
|
|
# is one of the reasons this pin moved to latest stable. It cannot be confirmed
|
|
|
|
|
# from a green run either, because the loop below sends lint output to /tmp/out
|
|
|
|
|
# and prints it only on failure, so warnings are invisible when the step passes.
|
|
|
|
|
# The claim is dropped rather than restated: what this step guarantees is the
|
|
|
|
|
# exit-code contract above, which is what the gate actually rests on.
|
|
|
|
|
# Latest stable, per the standing rule that any software uses the latest
|
|
|
|
|
# stable version. Matching the deployed woodpecker-server (v3.16.0) is a
|
|
|
|
|
# CONSEQUENCE of that, not a separate requirement — the server is bumped by
|
|
|
|
|
# image-automation within the 3.x range, so both land on the same release.
|
|
|
|
|
#
|
|
|
|
|
# It matters here because the CLI compiles the config against its own schema:
|
|
|
|
|
# one behind the server can pass a config the server then rejects, or reject
|
|
|
|
|
# one it accepts, and the gate stops predicting what the server will do —
|
|
|
|
|
# which is its only purpose.
|
|
|
|
|
#
|
|
|
|
|
# `-alpine` is required, not cosmetic: the plain vX.Y.Z CLI image ships no
|
|
|
|
|
# shell, and the loop below needs `for`, `sed` and `echo` around the binary.
|
|
|
|
|
# A step using the plain tag fails to start rather than failing a check.
|
|
|
|
|
#
|
|
|
|
|
# DRIFT RISK — accepted, and NOT fixable here. The server auto-bumps via
|
|
|
|
|
# image-automation; this pin cannot, so the two will drift apart.
|
|
|
|
|
# Image-automation can only patch YAML that is a Kubernetes manifest —
|
|
|
|
|
# fluxcd/flux2#2826: "Flux can only patch Kubernetes manifests. This means the
|
|
|
|
|
# YAML manifest must have `apiVersion` and `kind` fields." A Woodpecker
|
|
|
|
|
# pipeline file has neither, so an `$imagepolicy` marker here is silently
|
|
|
|
|
# ignored. That is a property of the FILE, not of `update.path`: widening the
|
|
|
|
|
# automation's path to `./` would buy nothing, and would additionally drag the
|
|
|
|
|
# whole ansible/ and esphome/ trees into the walk. Do not try it — it looks
|
|
|
|
|
# like the fix and is not.
|
|
|
|
|
#
|
|
|
|
|
# So this bump is manual, and drift is caught by the update-audit agent
|
|
|
|
|
# (ci/agents/update-audit/prompt.md, item 5) which is told to flag this image
|
|
|
|
|
# whenever it trails the deployed server. If you bump the server, bump here.
|
|
|
|
|
- name: woodpecker-lint
|
|
|
|
|
depends_on: [] # see note above `steps:`
|
|
|
|
|
image: woodpeckerci/woodpecker-cli:v3.16.0-alpine
|
|
|
|
|
commands:
|
|
|
|
|
- |
|
|
|
|
|
set -eu
|
|
|
|
|
fail=0
|
|
|
|
|
for f in .woodpecker/*.yml; do
|
|
|
|
|
if /bin/woodpecker-cli lint "$f" >/tmp/out 2>&1; then
|
|
|
|
|
echo "ok $f"
|
|
|
|
|
else
|
|
|
|
|
echo "FAIL $f"
|
|
|
|
|
sed 's/^/ /' /tmp/out
|
|
|
|
|
fail=1
|
|
|
|
|
fi
|
|
|
|
|
done
|
|
|
|
|
exit $fail
|
|
|
|
|
|
|
|
|
|
# Standalone shell scripts under kubernetes/ (sidecar and bootstrap scripts
|
|
|
|
|
# mounted via configMapGenerator). Severity is pinned to `error` on purpose:
|
|
|
|
|
# style and info findings would fail the build on existing scripts for no
|
|
|
|
|
# safety gain. All scripts in the tree are clean at this level today.
|
|
|
|
|
#
|
|
|
|
|
# NOTE this does NOT cover shell embedded in manifests (init containers, probe
|
|
|
|
|
# exec blocks) — shellcheck cannot see inside YAML. That embedded shell is
|
|
|
|
|
# where several real bugs lived this year, so it remains a gap worth closing
|
|
|
|
|
# separately rather than one this step quietly implies is covered.
|
|
|
|
|
- name: shellcheck
|
|
|
|
|
depends_on: [] # see note above `steps:`
|
|
|
|
|
image: koalaman/shellcheck-alpine:v0.11.0
|
|
|
|
|
commands:
|
|
|
|
|
- |
|
|
|
|
|
set -eu
|
|
|
|
|
files=$(find kubernetes -name '*.sh' | sort)
|
|
|
|
|
# Same no-op guard as kustomize-build: if discovery ever returns nothing,
|
|
|
|
|
# fail loudly instead of passing having checked no files.
|
|
|
|
|
#
|
|
|
|
|
# This floor is 1 while kustomize-build's is 50, and the asymmetry is
|
|
|
|
|
# deliberate rather than an oversight. That floor is loose relative to the
|
|
|
|
|
# tree it guards, so it catches both total failure and gross partial loss
|
|
|
|
|
# while tolerating normal churn. The shell-script set is small enough that
|
|
|
|
|
# any floor tight enough to catch PARTIAL loss would sit inside normal
|
|
|
|
|
# churn — deleting one script would fail CI spuriously, and the fix a
|
|
|
|
|
# hurried person reaches for is lowering the floor, which is how a guard
|
|
|
|
|
# quietly becomes decoration. A floor that has to be edited whenever the
|
|
|
|
|
# tree changes is not a guard, it is a snapshot with an exit code.
|
|
|
|
|
#
|
|
|
|
|
# So this catches the failure that actually happened here (discovery
|
|
|
|
|
# silently returning nothing, step green having checked no files) and not
|
|
|
|
|
# partial loss. Worth revisiting only if kubernetes/ grows a lot more
|
|
|
|
|
# shell, at which point a loose proportionate floor becomes possible.
|
|
|
|
|
count=$(printf '%s\n' "$files" | grep -c . || true)
|
|
|
|
|
if [ "$count" -lt 1 ]; then
|
|
|
|
|
echo "FAIL: no shell scripts found under kubernetes/ — discovery is broken."
|
|
|
|
|
exit 1
|
|
|
|
|
fi
|
|
|
|
|
echo "discovered $count shell scripts"
|
|
|
|
|
|
|
|
|
|
# Same file-fed while-read reasoning as kustomize-build above.
|
|
|
|
|
printf '%s\n' "$files" > /tmp/files
|
|
|
|
|
fail=0
|
|
|
|
|
while IFS= read -r f; do
|
|
|
|
|
[ -n "$f" ] || continue
|
|
|
|
|
if shellcheck -S error "$f"; then
|
|
|
|
|
echo "ok $f"
|
|
|
|
|
else
|
|
|
|
|
echo "FAIL $f"
|
|
|
|
|
fail=1
|
|
|
|
|
fi
|
|
|
|
|
done < /tmp/files
|
|
|
|
|
exit $fail
|
|
|
|
|
|
|
|
|
|
# Guard the convention that cost a day to diagnose: registry-local images are
|
|
|
|
|
# pinned by DATED TAG, never by digest. Digests here are garbage-collected once
|
|
|
|
|
# :latest and the dated tags move past them — and that happens on edits to
|
|
|
|
|
# .woodpecker/build-ci-images.yml, a shared file, so an unrelated image addition
|
|
|
|
|
# silently orphans the manifest a rotator was pinned to. 11 rotators were dead
|
|
|
|
|
# this way and would have failed on their first scheduled run months later.
|
|
|
|
|
#
|
|
|
|
|
# Scoped to the rotation directory, which is where the convention is documented
|
|
|
|
|
# and where the failure actually bit.
|
|
|
|
|
#
|
|
|
|
|
# The `grep -v kustomization.yaml` is load-bearing, not tidying: that file
|
|
|
|
|
# documents this very rule, so its comment contains the literal string
|
|
|
|
|
# `image:.*@sha256` and therefore MATCHES the pattern. Anchoring on `image:`
|
|
|
|
|
# does not prevent that — verified by running the bare grep, which returns
|
|
|
|
|
# kustomization.yaml. Only the filename filter excludes it. Drop the filter and
|
|
|
|
|
# this step fails permanently on its own documentation.
|
|
|
|
|
- name: no-digest-pins
|
|
|
|
|
depends_on: [] # see note above `steps:`
|
|
|
|
|
image: gitea.renner.uno/gitea_admin/ci-k8s:2026-07-30
|
|
|
|
|
commands:
|
|
|
|
|
- |
|
|
|
|
|
set -eu
|
|
|
|
|
hits=$(grep -l 'image:.*@sha256' kubernetes/infrastructure/openbao/rotation/*.yaml 2>/dev/null \
|
|
|
|
|
| grep -v kustomization.yaml || true)
|
|
|
|
|
if [ -n "$hits" ]; then
|
|
|
|
|
echo "FAIL: digest-pinned images found; use a dated tag (see rotation/kustomization.yaml):"
|
|
|
|
|
echo "$hits" | sed 's/^/ /'
|
|
|
|
|
exit 1
|
|
|
|
|
fi
|
|
|
|
|
echo "ok no digest pins under openbao/rotation/"
|