docs(chrony): correct the root cause recorded for the makestep fix

#465 landed the makestep directive with the wrong root cause in its
comment. It claimed edge-1 "rebooted with an RTC ~16 days in the past".
That is false, and the real cause matters for what to do next.

edge-1's CMOS battery is dead. After a power loss the RTC came up reading
2018-08-20 -- roughly eight years stale, not 16 days:

  PM: RTC time: 09:18:56, date: 2018-08-20
  rtc_cmos 00:01: setting system clock to 2018-08-20T09:18:58 UTC

The clock was then reconstructed entirely in software. systemd floored it
forward to its own build date:

  systemd[1]: System time advanced to built-in epoch: Fri 2026-06-26 11:27:23 UTC

and it settled at ~2026-07-19, about 16 days behind. So the 16 days is
incidental -- it is just where the flooring landed. There is no 16-day
mechanism, and nothing "drifted". The two real faults are a dead RTC
battery and a chrony config that never steps. #465 fixed the second.

Also collapses the duplicated rationale in chrony.conf.j2 to a pointer at
defaults/main.yml, as suggested in review on #465, so the explanation
lives in exactly one place.

No behaviour change: the makestep directive and its defaults are
untouched. Comments only.

The commit message on 7fcaa4b0 still carries the wrong explanation and is
left alone rather than rewriting merged history.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

.woodpecker/ci-manifests-test.yml

@@ -0,0 +1,254 @@
+# Validate the GitOps tree and the pipeline definitions on PRs that touch them.
+#
+# Nothing covered kubernetes/** before this. deploy-infrastructure.yml only
+# gates opentofu/**, and ci-scripts-test.yml only runs scripts/ci/ unit tests —
+# so a broken manifest reached the cluster and was caught by Flux failing to
+# reconcile, or by a human noticing. Three separate defects on 2026-07-30 would
+# have been caught here instead:
+#
+#   * an unquoted YAML scalar containing ": " in .woodpecker/build-ci-images.yml,
+#     which made EVERY pipeline in the repo fail to compile (#443). `yq` parsed
+#     it happily — it is valid YAML, just a map instead of a string — so
+#     "it parses" was mistaken for "it is valid". Only woodpecker lint catches it.
+#   * a rotator pinned to an image digest that no longer existed (#458).
+#   * manifests whose kustomize wiring was only ever checked by hand.
+#
+# Deliberately CHEAP and offline: no cluster, no registry, no secrets. Everything
+# here is a static check that either passes or fails on the checkout alone, so it
+# can gate PRs without becoming a bottleneck or a flake source.
+when:
+  # Path-filtered on purpose, unlike ci-scripts-test.yml. That pipeline is one
+  # step on one already-cached image, so running it everywhere is free; this one
+  # pulls three images and builds every kustomization in the tree, which is real
+  # cost to put on a docs-only PR. The filter covers everything the steps actually read:
+  # kubernetes/** for the manifests and scripts, .woodpecker/** for the lint (and
+  # so an edit to this file re-runs its own checks).
+  - event: pull_request
+    path:
+      - "kubernetes/**"
+      - ".woodpecker/**"
+
+# ADDING A THIRD DISCOVERY CHECK: kustomize-build and shellcheck both hand-roll
+# the same shape — `find`, a floor guard against empty discovery, then a
+# file-fed `while read`. Two copies is deliberate: the two traps being commented
+# around (BusyBox find lacking `-printf`, and a piped loop running in a subshell
+# where `fail` cannot propagate) are different, and both are worth spelling out
+# where they bite. A third instance is the point to stop copying and factor it
+# into a shared script under `scripts/ci/` — which is already covered by
+# ci-scripts-test.yml, so it would gain unit tests in the process.
+#
+# Every step carries `depends_on: []` so all four run in PARALLEL and every
+# failure shows in one run. Woodpecker's default is sequential, which would stop
+# at the first failure and make a contributor fix-and-repush to discover the
+# next — and these four are independent, cheap, offline checks with no ordering
+# between them. Keep it on every step: `depends_on` is per-step, so dropping it
+# from one silently re-serialises that step behind the others.
+steps:
+  # Every kustomization must build. This is the check that catches a resource
+  # entry pointing at a file that does not exist, a bad patch target, or YAML
+  # that is fine in isolation but broken once overlaid — the class of error that
+  # otherwise surfaces as a Flux reconcile failure after merge.
+  #
+  # Verified green across every kustomization in the tree when this was added,
+  # so this gates new breakage rather than pre-existing debt.
+  #
+  # ci-k8s TAG SPLIT: this file pins a newer dated tag than e2e.yml, e2e-test.yml,
+  # alert-monitor.yml and issue-fix.yml, which still sit on an older one. That is
+  # staleness in those four, not a deliberate split here — everything should be on
+  # the newest dated tag. They are not bumped here because none of them are
+  # triggered by this PR, so it would ship untested changes under cover of a green
+  # run that never touched them. Tracked in #461.
+  - name: kustomize-build
+    depends_on: [] # see note above `steps:`
+    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-07-30
+    commands:
+      - |
+        set -eu
+        # NOT `find -printf '%h\n'`: that is a GNU findutils extension and this
+        # image ships BusyBox find (v1.37.0), which rejects it outright with
+        # "find: unrecognized: -printf". Because the result feeds a command
+        # substitution used as a word list, the failure is SILENT — empty output,
+        # zero loop iterations, fail=0, step exits green having validated
+        # nothing. That is exactly what this pipeline shipped in its first
+        # revision, and CI reported it as passing.
+        dirs=$(find kubernetes -name kustomization.yaml | sed 's|/kustomization.yaml$||' | sort)
+
+        # Assert the discovery actually found something. Without this, any future
+        # breakage of the find/sed above degrades back into a green no-op rather
+        # than a failure — the specific trap that made the first revision of this
+        # step useless while looking healthy.
+        count=$(printf '%s\n' "$dirs" | grep -c . || true)
+        if [ "$count" -lt 50 ]; then
+          echo "FAIL: found only $count kustomizations; floor is 50."
+          echo "      If the tree legitimately shrank, lower this floor in"
+          echo "      .woodpecker/ci-manifests-test.yml — do not delete it."
+          echo "      Otherwise discovery is broken — refusing to report success."
+          exit 1
+        fi
+        echo "discovered $count kustomizations"
+
+        # `while read` fed from a FILE, not a pipe: a pipe would run the loop in
+        # a subshell where `fail` cannot propagate, so a failing build would exit
+        # 0 — the same silent-pass shape this step already shipped once. Reading
+        # a file keeps the loop in the current shell, and read -r handles paths
+        # containing spaces without IFS games.
+        printf '%s\n' "$dirs" > /tmp/dirs
+        fail=0
+        while IFS= read -r d; do
+          [ -n "$d" ] || continue
+          if kubectl kustomize "$d" >/dev/null 2>/tmp/err; then
+            echo "ok    $d"
+          else
+            echo "FAIL  $d"
+            sed 's/^/        /' /tmp/err
+            fail=1
+          fi
+        done < /tmp/dirs
+        exit $fail
+
+  # Lint the pipeline definitions themselves. This is the one that would have
+  # prevented the repo-wide CI outage: `woodpecker lint` exits 1 on a config that
+  # fails to compile, and exits 0 on mere warnings — so it fails on real breakage
+  # without blocking on pre-existing noise. Both behaviours were verified against
+  # this repo (rc=1 reproducing #443, rc=0 on the tree as it stands).
+  #
+  # An earlier revision of this comment also asserted that several deploy-*.yml
+  # files emit "Additional property timeout is not allowed" warnings. That was
+  # observed under the v3.9.0 CLI this step used to pin, and it is NOT known to
+  # still hold on v3.16.0 — review raised the likelihood that the warning was
+  # schema drift in a 7-minor-old CLI rather than real debt in those files, which
+  # is one of the reasons this pin moved to latest stable. It cannot be confirmed
+  # from a green run either, because the loop below sends lint output to /tmp/out
+  # and prints it only on failure, so warnings are invisible when the step passes.
+  # The claim is dropped rather than restated: what this step guarantees is the
+  # exit-code contract above, which is what the gate actually rests on.
+  # Latest stable, per the standing rule that any software uses the latest
+  # stable version. Matching the deployed woodpecker-server (v3.16.0) is a
+  # CONSEQUENCE of that, not a separate requirement — the server is bumped by
+  # image-automation within the 3.x range, so both land on the same release.
+  #
+  # It matters here because the CLI compiles the config against its own schema:
+  # one behind the server can pass a config the server then rejects, or reject
+  # one it accepts, and the gate stops predicting what the server will do —
+  # which is its only purpose.
+  #
+  # `-alpine` is required, not cosmetic: the plain vX.Y.Z CLI image ships no
+  # shell, and the loop below needs `for`, `sed` and `echo` around the binary.
+  # A step using the plain tag fails to start rather than failing a check.
+  #
+  # DRIFT RISK — accepted, and NOT fixable here. The server auto-bumps via
+  # image-automation; this pin cannot, so the two will drift apart.
+  # Image-automation can only patch YAML that is a Kubernetes manifest —
+  # fluxcd/flux2#2826: "Flux can only patch Kubernetes manifests. This means the
+  # YAML manifest must have `apiVersion` and `kind` fields." A Woodpecker
+  # pipeline file has neither, so an `$imagepolicy` marker here is silently
+  # ignored. That is a property of the FILE, not of `update.path`: widening the
+  # automation's path to `./` would buy nothing, and would additionally drag the
+  # whole ansible/ and esphome/ trees into the walk. Do not try it — it looks
+  # like the fix and is not.
+  #
+  # So this bump is manual, and drift is caught by the update-audit agent
+  # (ci/agents/update-audit/prompt.md, item 5) which is told to flag this image
+  # whenever it trails the deployed server. If you bump the server, bump here.
+  - name: woodpecker-lint
+    depends_on: [] # see note above `steps:`
+    image: woodpeckerci/woodpecker-cli:v3.16.0-alpine
+    commands:
+      - |
+        set -eu
+        fail=0
+        for f in .woodpecker/*.yml; do
+          if /bin/woodpecker-cli lint "$f" >/tmp/out 2>&1; then
+            echo "ok    $f"
+          else
+            echo "FAIL  $f"
+            sed 's/^/        /' /tmp/out
+            fail=1
+          fi
+        done
+        exit $fail
+
+  # Standalone shell scripts under kubernetes/ (sidecar and bootstrap scripts
+  # mounted via configMapGenerator). Severity is pinned to `error` on purpose:
+  # style and info findings would fail the build on existing scripts for no
+  # safety gain. All scripts in the tree are clean at this level today.
+  #
+  # NOTE this does NOT cover shell embedded in manifests (init containers, probe
+  # exec blocks) — shellcheck cannot see inside YAML. That embedded shell is
+  # where several real bugs lived this year, so it remains a gap worth closing
+  # separately rather than one this step quietly implies is covered.
+  - name: shellcheck
+    depends_on: [] # see note above `steps:`
+    image: koalaman/shellcheck-alpine:v0.11.0
+    commands:
+      - |
+        set -eu
+        files=$(find kubernetes -name '*.sh' | sort)
+        # Same no-op guard as kustomize-build: if discovery ever returns nothing,
+        # fail loudly instead of passing having checked no files.
+        #
+        # This floor is 1 while kustomize-build's is 50, and the asymmetry is
+        # deliberate rather than an oversight. That floor is loose relative to the
+        # tree it guards, so it catches both total failure and gross partial loss
+        # while tolerating normal churn. The shell-script set is small enough that
+        # any floor tight enough to catch PARTIAL loss would sit inside normal
+        # churn — deleting one script would fail CI spuriously, and the fix a
+        # hurried person reaches for is lowering the floor, which is how a guard
+        # quietly becomes decoration. A floor that has to be edited whenever the
+        # tree changes is not a guard, it is a snapshot with an exit code.
+        #
+        # So this catches the failure that actually happened here (discovery
+        # silently returning nothing, step green having checked no files) and not
+        # partial loss. Worth revisiting only if kubernetes/ grows a lot more
+        # shell, at which point a loose proportionate floor becomes possible.
+        count=$(printf '%s\n' "$files" | grep -c . || true)
+        if [ "$count" -lt 1 ]; then
+          echo "FAIL: no shell scripts found under kubernetes/ — discovery is broken."
+          exit 1
+        fi
+        echo "discovered $count shell scripts"
+
+        # Same file-fed while-read reasoning as kustomize-build above.
+        printf '%s\n' "$files" > /tmp/files
+        fail=0
+        while IFS= read -r f; do
+          [ -n "$f" ] || continue
+          if shellcheck -S error "$f"; then
+            echo "ok    $f"
+          else
+            echo "FAIL  $f"
+            fail=1
+          fi
+        done < /tmp/files
+        exit $fail
+
+  # Guard the convention that cost a day to diagnose: registry-local images are
+  # pinned by DATED TAG, never by digest. Digests here are garbage-collected once
+  # :latest and the dated tags move past them — and that happens on edits to
+  # .woodpecker/build-ci-images.yml, a shared file, so an unrelated image addition
+  # silently orphans the manifest a rotator was pinned to. 11 rotators were dead
+  # this way and would have failed on their first scheduled run months later.
+  #
+  # Scoped to the rotation directory, which is where the convention is documented
+  # and where the failure actually bit.
+  #
+  # The `grep -v kustomization.yaml` is load-bearing, not tidying: that file
+  # documents this very rule, so its comment contains the literal string
+  # `image:.*@sha256` and therefore MATCHES the pattern. Anchoring on `image:`
+  # does not prevent that — verified by running the bare grep, which returns
+  # kustomization.yaml. Only the filename filter excludes it. Drop the filter and
+  # this step fails permanently on its own documentation.
+  - name: no-digest-pins
+    depends_on: [] # see note above `steps:`
+    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-07-30
+    commands:
+      - |
+        set -eu
+        hits=$(grep -l 'image:.*@sha256' kubernetes/infrastructure/openbao/rotation/*.yaml 2>/dev/null \
+                 | grep -v kustomization.yaml || true)
+        if [ -n "$hits" ]; then
+          echo "FAIL: digest-pinned images found; use a dated tag (see rotation/kustomization.yaml):"
+          echo "$hits" | sed 's/^/        /'
+          exit 1
+        fi
+        echo "ok    no digest pins under openbao/rotation/"

ansible/roles/chrony-ntp-server/defaults/main.yml

@@ -18,6 +18,26 @@ chrony_upstream_pools:
 # Local stratum if disconnected from upstream
 chrony_local_stratum: 10
 
+# Step (rather than slew) the clock when the offset exceeds the threshold, but
+# only for the first N updates after chronyd starts. Without a makestep
+# directive chronyd only ever slews, so a large offset is never closed in
+# practice.
+#
+# edge-1, 2026-08-04: the CMOS battery is dead, so after a power loss the RTC
+# came up reading 2018-08-20. The kernel set the clock from it, systemd floored
+# that forward to its own built-in epoch (2026-06-26), and it ended up at
+# ~2026-07-19 -- about 16 days behind. chronyd then synced to ptbtime1, reported
+# "System time: 1376235.375 seconds slow of NTP time", and corrected none of it.
+# With host time before the cluster certificate's NotBefore, every outbound TLS
+# handshake failed x509 validation, taking out the MQTT bridge (heating
+# telemetry) and the Loki log shipper at once. The 16 days is incidental -- it
+# is just where the software flooring landed; the fault is that nothing steps.
+#
+# Limiting to the first few updates keeps steps confined to boot, so a
+# long-running server is never jumped out from under its workloads.
+chrony_makestep_threshold: 1.0
+chrony_makestep_limit: 3
+
 # Hardware timestamping. Fine on the PVE nodes' Intel NICs; the Wyse 3040
 # (edge-1) has a Realtek NIC that does not support it, and chronyd logs an
 # error per interface when asked for it. Overridden to false in

ansible/roles/chrony-ntp-server/templates/chrony.conf.j2

@@ -14,6 +14,11 @@ pool {{ pool }}
 # Record the rate at which the system clock gains/losses time
 driftfile /var/lib/chrony/chrony.drift
 
+# Step the clock instead of slewing it when the offset is large, limited to the
+# first few updates after start so only boot-time corrections jump. Rationale
+# and the incident behind it: chrony_makestep_* in the role's defaults/main.yml.
+makestep {{ chrony_makestep_threshold }} {{ chrony_makestep_limit }}
+
 # Allow NTP client access from local networks
 {% for network in chrony_allowed_networks %}
 allow {{ network }}

ci/agents/update-audit/prompt.md

@@ -59,6 +59,22 @@ For each, find the **current pin in the repo** (Read/Glob/Grep), then look up th
 5. **Images deliberately excluded from automation** — listed in the
    image-automation `README.md` (e.g. vsftpd, otbr, ical-proxy) and any image with
    no `# {"$imagepolicy": ...}` marker. Check whether a newer usable tag exists.
+   This includes **CI step images in `.woodpecker/*.yml`**, which are all manual:
+   image-automation cannot patch those files at all (they are not Kubernetes
+   manifests — no `apiVersion`/`kind` — so markers there are ignored), and no
+   amount of `update.path` widening changes that. Grep `.woodpecker/` for
+   `image:` and check each pin.
+
+   Give `woodpeckerci/woodpecker-cli` in `.woodpecker/ci-manifests-test.yml`
+   particular attention: the woodpecker **server** IS auto-bumped (`3.x`), the
+   CLI is not, so they silently drift apart. The CLI lints pipeline configs
+   against its own schema, so one behind the server can pass a config the server
+   then rejects — the gate stops predicting server behaviour. Report the CLI as
+   needing an update whenever it trails the server version, not only when it
+   trails the newest CLI release. Its tag carries an `-alpine` suffix (the plain
+   tag is scratch-based with no shell and would break the step) — compare the
+   `vX.Y.Z` part, and ignore the `next-*` / `pull_*` development tags, which are
+   the newest by date on that repo and are not releases.
 6. **Ansible roles & OpenTofu providers** — `ansible/requirements.yml` (Galaxy
    roles/collections) and `opentofu/providers.tf` + `opentofu/.terraform.lock.hcl`.
    Report any with a newer published version.

kubernetes/apps/paperless/redis.yaml

@@ -25,7 +25,7 @@ spec:
           type: RuntimeDefault
       containers:
         - name: redis
-          image: redis:8.8.1-alpine # {"$imagepolicy": "flux-system:redis-paperless"}
+          image: redis:8.10.0-alpine # {"$imagepolicy": "flux-system:redis-paperless"}
           args:
             - redis-server
             - --save

kubernetes/apps/woodpecker/agent.yaml

@@ -34,7 +34,7 @@ spec:
           # Image automation, range 3.x (see image-automation/woodpecker.yaml).
           # MUST stay version-matched with woodpecker-server (gRPC parity) — both
           # track the same range/filter so they bump together on each release.
-          image: woodpeckerci/woodpecker-agent:v3.16.0 # {"$imagepolicy": "flux-system:woodpecker-agent"}
+          image: woodpeckerci/woodpecker-agent:v3.17.0 # {"$imagepolicy": "flux-system:woodpecker-agent"}
           env:
             - name: WOODPECKER_SERVER
               value: woodpecker.apps.svc.cluster.local:9000 # in-cluster gRPC, replaces LXC :9000

kubernetes/apps/woodpecker/server.yaml

@@ -40,7 +40,7 @@ spec:
           # minor + patch auto-update within 3.x; only the 4.0 major is held for
           # a reviewed bump. Safe now the DB is postgres (the old sqlite-surprise
           # -bump hazard is gone). Stays in lockstep with the agent.
-          image: woodpeckerci/woodpecker-server:v3.16.0 # {"$imagepolicy": "flux-system:woodpecker-server"}
+          image: woodpeckerci/woodpecker-server:v3.17.0 # {"$imagepolicy": "flux-system:woodpecker-server"}
           env:
             - name: WOODPECKER_HOST
               value: https://ci.renner.uno

kubernetes/infrastructure/image-automation/README.md

@@ -55,6 +55,31 @@ existing `flux-system` GitRepository whose Secret was created by
 | mmwave-configurator | `2.x` | Plain 3-component filter (rejects `addon-`/`-armv7`/`-arm64`/`-amd64` variants) |
 | cnpg-barman-plugin | `0.x` | v-prefix filter; marker lives in upstream-imported manifest (see below) |
 
+## Not enrolled (structurally impossible)
+
+- `woodpeckerci/woodpecker-cli` — the lint image in
+  `.woodpecker/ci-manifests-test.yml`. It publishes a perfectly good
+  version stream (`v3.16.0-alpine`), so unlike the three below this is
+  **not** a tag-format problem. It cannot be enrolled because
+  **image-automation can only patch YAML that is a Kubernetes manifest** —
+  a maintainer states it plainly in fluxcd/flux2#2826: *"Flux can only
+  patch Kubernetes manifests. This means the YAML manifest must have
+  `apiVersion` and `kind` fields."* A Woodpecker pipeline file is plain
+  YAML with neither, so the marker would be silently ignored.
+
+  Note this is **not** fixable by widening `update.path`. The path only
+  controls which directory is walked; objects without `apiVersion`/`kind`
+  are skipped regardless. Widening the path to `./` buys nothing here and
+  additionally pulls the whole `ansible/` and `esphome/` trees into the
+  walk, so don't.
+
+  Consequence: the CLI is a **manual** pin while the server and agent
+  auto-bump, so it *will* drift. That is a known, accepted gap, covered
+  by the update-audit agent (item 5) rather than by automation. When
+  bumping the server major/minor, bump this too — it must stay on the
+  latest stable like everything else, and a CLI behind the server can
+  pass a config the server then rejects.
+
 ## Not enrolled (deliberate)
 
 These three images publish no usable version stream — there is nothing