homelab/.woodpecker/ci-manifests-test.yml

# Validate the GitOps tree and the pipeline definitions on PRs that touch them.
#
# Nothing covered kubernetes/** before this. deploy-infrastructure.yml only
# gates opentofu/**, and ci-scripts-test.yml only runs scripts/ci/ unit tests —
# so a broken manifest reached the cluster and was caught by Flux failing to
# reconcile, or by a human noticing. Three separate defects on 2026-07-30 would
# have been caught here instead:
#
#   * an unquoted YAML scalar containing ": " in .woodpecker/build-ci-images.yml,
#     which made EVERY pipeline in the repo fail to compile (#443). `yq` parsed
#     it happily — it is valid YAML, just a map instead of a string — so
#     "it parses" was mistaken for "it is valid". Only woodpecker lint catches it.
#   * a rotator pinned to an image digest that no longer existed (#458).
#   * manifests whose kustomize wiring was only ever checked by hand.
#
# Deliberately CHEAP and offline: no cluster, no registry, no secrets. Everything
# here is a static check that either passes or fails on the checkout alone, so it
# can gate PRs without becoming a bottleneck or a flake source.
when:
  # Path-filtered on purpose, unlike ci-scripts-test.yml. That pipeline is two
  # steps on already-cached images, so running it everywhere is free; this one
  # pulls four images and builds every kustomization in the tree, which is real
  # cost to put on a docs-only PR. The filter covers everything the steps actually read:
  # kubernetes/** for the manifests, scripts and promtool suites, .woodpecker/**
  # for the lint (and so an edit to this file re-runs its own checks), and the
  # promtool driver script by name.
  #
  # That last entry is NOT covered by kubernetes/**: promtool-rules.sh lives
  # under scripts/, so without it a PR that only edits the script would never run
  # the check it drives. It is listed as one file rather than scripts/ci/**
  # because the rest of that directory is CI-agent helpers with nothing to do
  # with this pipeline, and pulling four images for them is the cost this filter
  # exists to avoid. Their own tests run in ci-scripts-test.yml on every PR.
  - event: pull_request
    path:
      - "kubernetes/**"
      - ".woodpecker/**"
      - "scripts/ci/promtool-rules.sh"
      - "scripts/ci/yaml-citations.sh"

# THE THIRD DISCOVERY CHECK WENT TO A SCRIPT. kustomize-build and shellcheck
# both hand-roll the same shape — `find`, a floor guard against empty discovery,
# then a file-fed `while read`. Two copies was deliberate: the two traps being
# commented around (BusyBox find lacking `-printf`, and a piped loop running in
# a subshell where `fail` cannot propagate) are different, and both are worth
# spelling out where they bite. An earlier revision of this comment said a third
# instance would be the point to stop copying and factor it into `scripts/ci/`,
# where it would gain unit tests — and that is what `promtool` below does. Its
# discovery, floors and guards live in scripts/ci/promtool-rules.sh and are
# tested by scripts/ci/test-promtool-rules.sh on every PR via
# ci-scripts-test.yml. A FOURTH inline copy should go the same way; do not add
# one here.
#
# Every step carries `depends_on: []` so all six run in PARALLEL and every
# failure shows in one run. Woodpecker's default is sequential, which would stop
# at the first failure and make a contributor fix-and-repush to discover the
# next — and these six are independent, cheap, offline checks with no ordering
# between them. Keep it on every step: `depends_on` is per-step, so dropping it
# from one silently re-serialises that step behind the others.
steps:
  # Every kustomization must build. This is the check that catches a resource
  # entry pointing at a file that does not exist, a bad patch target, or YAML
  # that is fine in isolation but broken once overlaid — the class of error that
  # otherwise surfaces as a Flux reconcile failure after merge.
  #
  # Verified green across every kustomization in the tree when this was added,
  # so this gates new breakage rather than pre-existing debt.
  #
  # ci-k8s TAG SPLIT: this file pins a newer dated tag than e2e.yml and
  # e2e-test.yml, which still sit on an older one. That is
  # staleness in those two, not a deliberate split here — everything should be on
  # the newest dated tag. They are not bumped here because none of them are
  # triggered by this PR, so it would ship untested changes under cover of a green
  # run that never touched them. Tracked in #461.
  - name: kustomize-build
    depends_on: [] # see note above `steps:`
    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-09-08
    commands:
      - |
        set -eu
        # NOT `find -printf '%h\n'`: that is a GNU findutils extension and this
        # image ships BusyBox find (v1.37.0), which rejects it outright with
        # "find: unrecognized: -printf". Because the result feeds a command
        # substitution used as a word list, the failure is SILENT — empty output,
        # zero loop iterations, fail=0, step exits green having validated
        # nothing. That is exactly what this pipeline shipped in its first
        # revision, and CI reported it as passing.
        dirs=$(find kubernetes -name kustomization.yaml | sed 's|/kustomization.yaml$||' | sort)

        # Assert the discovery actually found something. Without this, any future
        # breakage of the find/sed above degrades back into a green no-op rather
        # than a failure — the specific trap that made the first revision of this
        # step useless while looking healthy.
        count=$(printf '%s\n' "$dirs" | grep -c . || true)
        if [ "$count" -lt 50 ]; then
          echo "FAIL: found only $count kustomizations; floor is 50."
          echo "      If the tree legitimately shrank, lower this floor in"
          echo "      .woodpecker/ci-manifests-test.yml — do not delete it."
          echo "      Otherwise discovery is broken — refusing to report success."
          exit 1
        fi
        echo "discovered $count kustomizations"

        # `while read` fed from a FILE, not a pipe: a pipe would run the loop in
        # a subshell where `fail` cannot propagate, so a failing build would exit
        # 0 — the same silent-pass shape this step already shipped once. Reading
        # a file keeps the loop in the current shell, and read -r handles paths
        # containing spaces without IFS games.
        printf '%s\n' "$dirs" > /tmp/dirs
        fail=0
        while IFS= read -r d; do
          [ -n "$d" ] || continue
          if kubectl kustomize "$d" >/dev/null 2>/tmp/err; then
            echo "ok    $d"
          else
            echo "FAIL  $d"
            sed 's/^/        /' /tmp/err
            fail=1
          fi
        done < /tmp/dirs
        exit $fail

  # Lint the pipeline definitions themselves. This is the one that would have
  # prevented the repo-wide CI outage: `woodpecker lint` exits 1 on a config that
  # fails to compile, and exits 0 on mere warnings — so it fails on real breakage
  # without blocking on pre-existing noise. Both behaviours were verified against
  # this repo (rc=1 reproducing #443, rc=0 on the tree as it stands).
  #
  # An earlier revision of this comment also asserted that several deploy-*.yml
  # files emit "Additional property timeout is not allowed" warnings. That was
  # observed under the v3.9.0 CLI this step used to pin, and it is NOT known to
  # still hold on current CLIs — review raised the likelihood that the warning was
  # schema drift in a 7-minor-old CLI rather than real debt in those files, which
  # is one of the reasons this pin moved to latest stable. It cannot be confirmed
  # from a green run either, because the loop below sends lint output to /tmp/out
  # and prints it only on failure, so warnings are invisible when the step passes.
  # The claim is dropped rather than restated: what this step guarantees is the
  # exit-code contract above, which is what the gate actually rests on.
  # Latest stable, per the standing rule that any software uses the latest
  # stable version. Matching the deployed woodpecker-server is a CONSEQUENCE of
  # that, not a separate requirement — the server is bumped by image-automation
  # within the 3.x range, so both should land on the same release.
  #
  # They do not land together, though: the server is a Kubernetes manifest and
  # is bumped automatically, this pipeline file is not and cannot be (see
  # infrastructure/image-automation/README.md). So the server moves first and
  # this trails until someone bumps it by hand — which is what happened at
  # v3.16.0 -> v3.17.0, caught by the update audit (#426) rather than by CI.
  #
  # It matters here because the CLI compiles the config against its own schema:
  # one behind the server can pass a config the server then rejects, or reject
  # one it accepts, and the gate stops predicting what the server will do —
  # which is its only purpose.
  #
  # `-alpine` is required, not cosmetic: the plain vX.Y.Z CLI image ships no
  # shell, and the loop below needs `for`, `sed` and `echo` around the binary.
  # A step using the plain tag fails to start rather than failing a check.
  #
  # DRIFT RISK — accepted, and NOT fixable here. The server auto-bumps via
  # image-automation; this pin cannot, so the two will drift apart.
  # Image-automation can only patch YAML that is a Kubernetes manifest —
  # fluxcd/flux2#2826: "Flux can only patch Kubernetes manifests. This means the
  # YAML manifest must have `apiVersion` and `kind` fields." A Woodpecker
  # pipeline file has neither, so an `$imagepolicy` marker here is silently
  # ignored. That is a property of the FILE, not of `update.path`: widening the
  # automation's path to `./` would buy nothing, and would additionally drag the
  # whole ansible/ and esphome/ trees into the walk. Do not try it — it looks
  # like the fix and is not.
  #
  # So this bump is manual, and drift is caught by the update-audit agent
  # (ci/agents/update-audit/prompt.md, item 5) which is told to flag this image
  # whenever it trails the deployed server. If you bump the server, bump here.
  - name: woodpecker-lint
    depends_on: [] # see note above `steps:`
    image: woodpeckerci/woodpecker-cli:v3.19.0-alpine@sha256:27a77117e38d7bbdc380f0e90ff3ee4871ab1eefa78b5d9136a2c969fbecd9f4
    commands:
      - |
        set -eu
        fail=0
        for f in .woodpecker/*.yml; do
          if /bin/woodpecker-cli lint "$f" >/tmp/out 2>&1; then
            echo "ok    $f"
          else
            echo "FAIL  $f"
            sed 's/^/        /' /tmp/out
            fail=1
          fi
        done
        exit $fail

  # Standalone shell scripts under kubernetes/ (sidecar and bootstrap scripts
  # mounted via configMapGenerator). Severity is pinned to `error` on purpose:
  # style and info findings would fail the build on existing scripts for no
  # safety gain. All scripts in the tree are clean at this level today.
  #
  # NOTE this does NOT cover shell embedded in manifests (init containers, probe
  # exec blocks) — shellcheck cannot see inside YAML. That embedded shell is
  # where several real bugs lived this year, so it remains a gap worth closing
  # separately rather than one this step quietly implies is covered.
  - name: shellcheck
    depends_on: [] # see note above `steps:`
    image: koalaman/shellcheck-alpine:v0.11.0@sha256:9955be09ea7f0dbf7ae942ac1f2094355bb30d96fffba0ec09f5432207544002
    commands:
      - |
        set -eu
        files=$(find kubernetes -name '*.sh' | sort)
        # Same no-op guard as kustomize-build: if discovery ever returns nothing,
        # fail loudly instead of passing having checked no files.
        #
        # This floor is 1 while kustomize-build's is 50, and the asymmetry is
        # deliberate rather than an oversight. That floor is loose relative to the
        # tree it guards, so it catches both total failure and gross partial loss
        # while tolerating normal churn. The shell-script set is small enough that
        # any floor tight enough to catch PARTIAL loss would sit inside normal
        # churn — deleting one script would fail CI spuriously, and the fix a
        # hurried person reaches for is lowering the floor, which is how a guard
        # quietly becomes decoration. A floor that has to be edited whenever the
        # tree changes is not a guard, it is a snapshot with an exit code.
        #
        # So this catches the failure that actually happened here (discovery
        # silently returning nothing, step green having checked no files) and not
        # partial loss. Worth revisiting only if kubernetes/ grows a lot more
        # shell, at which point a loose proportionate floor becomes possible.
        count=$(printf '%s\n' "$files" | grep -c . || true)
        if [ "$count" -lt 1 ]; then
          echo "FAIL: no shell scripts found under kubernetes/ — discovery is broken."
          exit 1
        fi
        echo "discovered $count shell scripts"

        # Same file-fed while-read reasoning as kustomize-build above.
        printf '%s\n' "$files" > /tmp/files
        fail=0
        while IFS= read -r f; do
          [ -n "$f" ] || continue
          if shellcheck -S error "$f"; then
            echo "ok    $f"
          else
            echo "FAIL  $f"
            fail=1
          fi
        done < /tmp/files
        exit $fail

  # Guard the convention that cost a day to diagnose: registry-local images are
  # pinned by DATED TAG, never by digest. Digests here are garbage-collected once
  # :latest and the dated tags move past them — and that happens on edits to
  # .woodpecker/build-ci-images.yml, a shared file, so an unrelated image addition
  # silently orphans the manifest a rotator was pinned to. 11 rotators were dead
  # this way and would have failed on their first scheduled run months later.
  #
  # Scoped to the rotation directory, which is where the convention is documented
  # and where the failure actually bit.
  #
  # The `grep -v kustomization.yaml` is load-bearing, not tidying: that file
  # documents this very rule, so its comment contains the literal string
  # `image:.*@sha256` and therefore MATCHES the pattern. Anchoring on `image:`
  # does not prevent that — verified by running the bare grep, which returns
  # kustomization.yaml. Only the filename filter excludes it. Drop the filter and
  # this step fails permanently on its own documentation.
  - name: no-digest-pins
    depends_on: [] # see note above `steps:`
    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-09-08
    commands:
      - |
        set -eu
        hits=$(grep -l 'image:.*@sha256' kubernetes/infrastructure/openbao/rotation/*.yaml 2>/dev/null \
                 | grep -v kustomization.yaml || true)
        if [ -n "$hits" ]; then
          echo "FAIL: digest-pinned images found; use a dated tag (see rotation/kustomization.yaml):"
          echo "$hits" | sed 's/^/        /'
          exit 1
        fi
        echo "ok    no digest pins under openbao/rotation/"

  # Machine-check the alert rules: parse every PrometheusRule, run the committed
  # promtool unit suites, then damage the rules on purpose and require the suites
  # to notice. Everything it does lives in scripts/ci/promtool-rules.sh; read
  # that file and kubernetes/apps/monitoring-rules/tests/README.md for the why.
  #
  # The gap it closes: the other four steps prove a PrometheusRule assembles into
  # valid YAML and nothing more — not that an expression parses, not that a label
  # join matches anything, not that a `for` window does what its comment claims.
  # On REN-129 the CNPG surge rule was written `for: 2m`, which fires on six of
  # seven measured nights and would have mailed the owner twice a night at 04:16
  # and 04:22 local for a backlog that clears itself. Nothing here would have
  # caught it, and the arithmetic that did catch it was hand-written from scratch
  # twice, in two runs, and thrown away both times because there was nowhere in
  # the repo to commit it. REN-152 retired that rule; its `for` margin is now
  # guarded by mutations.txt:backlog-for-12m.
  #
  # Offline, like the rest of this pipeline: no cluster, no registry, no secrets,
  # no network. promtool reads files and exits.
  #
  # IMAGE — prom/prometheus, not a registry-local ci-* image, and pinned to an
  # exact upstream version rather than a dated tag. The dated-tag convention
  # guarded by no-digest-pins above is about OUR images, whose digests are
  # garbage-collected out from under a pin; this is upstream and immutable.
  #
  # The version must be 3.x, and that is not a preference. The cluster runs
  # whatever kube-prometheus-stack >= 72.0.0 ships, which is Prometheus 3, and
  # alert `for` evaluation is version-sensitive — a 2.x promtool would be
  # checking a different evaluator than production, so the margins the suites
  # assert would be claims about the wrong program. promtool-rules.sh asserts the
  # major version it was handed and fails on 2.x, so a careless bump here is a
  # red step rather than a quietly wrong check. Keep this at the latest stable
  # 3.x, per the standing rule that any software uses the latest stable version.
  #
  # This image is NOT covered by image-automation (same reason as
  # woodpecker-lint above: a Woodpecker pipeline file has no apiVersion/kind, so
  # an $imagepolicy marker here is silently ignored — fluxcd/flux2#2826). The
  # bump is manual.
  #
  # `prom/prometheus` is busybox-based and ships /bin/sh, so it can run a
  # `commands:` block — unlike the plain woodpecker-cli tag, which cannot. Its
  # find/grep/awk/sed are BusyBox, which is why promtool-rules.sh avoids
  # `find -printf` and `grep -r --include`; it was verified against BusyBox
  # 1.35.0 as well as GNU. The image also declares `USER nobody`, so the script
  # writes its stripped mirror under mktemp and never into the workspace.
  - name: promtool
    depends_on: [] # see note above `steps:`
    image: prom/prometheus:v3.15.0@sha256:efd719c99d83b060d9daefdcf00360461adf279f45ef5391f8d111892118753e
    commands:
      - sh scripts/ci/promtool-rules.sh

  # Comments cite other YAML blocks by anchor name, never by line number
  # (REN-183; the exact syntax is in CONTRIBUTING.md, not repeated here because
  # an example written in this comment would itself be checked). Line numbers drift on any edit
  # above them and nothing noticed: three drift fixes were merged by hand
  # before this step existed. This fails on a new line-number citation and on
  # a citation whose file or anchor marker does not exist.
  # Existence only — it never judges whether the anchored block still says
  # what the citing comment claims, so ordinary edits cannot turn it red.
  #
  # It scans every YAML file in the checkout, but this pipeline only runs on
  # the paths in `when:` above, so an edit confined to ansible/ or esphome/ is
  # checked on the next PR that does touch kubernetes/ or .woodpecker/.
  # Self-test: scripts/ci/test-yaml-citations.sh, in ci-scripts-test.yml.
  - name: yaml-citations
    depends_on: [] # see note above `steps:`
    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-09-08
    commands:
      - sh scripts/ci/yaml-citations.sh