fix(ci): tag images with a dated literal, not the commit SHA

The SHA tag was a workaround for not being able to compute a date in a shell-less build step. That problem is gone — the build timestamp now lives in the image at /etc/image-build-date — so the tag no longer has to carry it. A SHA reads as noise at the point of use and breaks the existing convention every consumer pin follows (ci-k8s:2026-08-10, ci-ansible:2026-04-29, alpine-jq:2026-07-30). Dated literals keep the pins uniform and are bumped in the commit that changes an image, which is the same trigger the `when:` path filters already use.
build(ci): record the build timestamp in ci-claude too
2026-08-23 21:14:51 +02:00 · 2026-08-23 20:59:29 +02:00 · 2026-08-23 20:57:15 +02:00 · 2026-08-23 20:52:06 +02:00
20 changed files with 146 additions and 239 deletions
@@ -14,6 +14,12 @@
 # Flux-deployed CronJobs, so it is tracked separately in #330 rather than
 # bundled here. Note this pipeline runs no LLM agent, so the exfil path that
 # motivated the other cutovers does not apply to it.
+#
+# The build steps run the plain kaniko executor, which has no shell: they pass
+# flags via `entrypoint` and read credentials from DOCKER_CONFIG. Registry auth
+# is written once by the registry-auth step below. Dated tags are literals: bump
+# them in the commit that changes an image. The build timestamp itself lives in
+# the image at /etc/image-build-date, written by a RUN in the Containerfile.
 when:
  - event: push
    path:
@@ -24,10 +30,22 @@ when:
    evaluate: 'PIPELINE == "build-ci-images"'

 steps:
-  - name: build-ansible
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
-    directory: ci/images/ansible
+  - name: registry-auth
+    image: alpine:3.24.1
    depends_on: []
+    environment:
+      GITEA_USER: gitea_admin
+      GITEA_TOKEN:
+        from_secret: ci_images_token
+    commands:
+      - mkdir -p .docker
+      - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
+      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > .docker/config.json
+
+  - name: build-ansible
+    image: gcr.io/kaniko-project/executor:v1.24.0
+    directory: ci/images/ansible
+    depends_on: [registry-auth]
    # Only rebuild when THIS image's source (or the pipeline file) changed, or
    # on a manual run. Without this every step ran on any matching push, so a
    # secret-rotator commit needlessly rebuilt all four ~1.3G ci-* images and
@@ -37,68 +55,47 @@ steps:
      - event: push
        path: ["ci/images/ansible/**", ".woodpecker/build-ci-images.yml"]
    environment:
-      GITEA_USER: gitea_admin
-      GITEA_TOKEN:
-        from_secret: ci_images_token
-    commands:
-      - mkdir -p /kaniko/.docker
-      - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
-      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
-      - TAG=$(date +%Y-%m-%d)
-      - |
-        /kaniko/executor \
-          --context . \
-          --dockerfile Containerfile \
-          --destination gitea.renner.uno/gitea_admin/ci-ansible:$TAG \
-          --destination gitea.renner.uno/gitea_admin/ci-ansible:latest
+      DOCKER_CONFIG: "${CI_WORKSPACE}/.docker"
+    entrypoint:
+      - /kaniko/executor
+      - --context=.
+      - --dockerfile=Containerfile
+      - --destination=gitea.renner.uno/gitea_admin/ci-ansible:2026-08-23
+      - --destination=gitea.renner.uno/gitea_admin/ci-ansible:latest

  - name: build-k8s
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: gcr.io/kaniko-project/executor:v1.24.0
    directory: ci/images/k8s
-    depends_on: []
+    depends_on: [registry-auth]
    when:
      - event: manual
      - event: push
        path: ["ci/images/k8s/**", ".woodpecker/build-ci-images.yml"]
    environment:
-      GITEA_USER: gitea_admin
-      GITEA_TOKEN:
-        from_secret: ci_images_token
-    commands:
-      - mkdir -p /kaniko/.docker
-      - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
-      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
-      - TAG=$(date +%Y-%m-%d)
-      - |
-        /kaniko/executor \
-          --context . \
-          --dockerfile Containerfile \
-          --destination gitea.renner.uno/gitea_admin/ci-k8s:$TAG \
-          --destination gitea.renner.uno/gitea_admin/ci-k8s:latest
+      DOCKER_CONFIG: "${CI_WORKSPACE}/.docker"
+    entrypoint:
+      - /kaniko/executor
+      - --context=.
+      - --dockerfile=Containerfile
+      - --destination=gitea.renner.uno/gitea_admin/ci-k8s:2026-08-23
+      - --destination=gitea.renner.uno/gitea_admin/ci-k8s:latest

  - name: build-tofu
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: gcr.io/kaniko-project/executor:v1.24.0
    directory: ci/images/tofu
-    depends_on: []
+    depends_on: [registry-auth]
    when:
      - event: manual
      - event: push
        path: ["ci/images/tofu/**", ".woodpecker/build-ci-images.yml"]
    environment:
-      GITEA_USER: gitea_admin
-      GITEA_TOKEN:
-        from_secret: ci_images_token
-    commands:
-      - mkdir -p /kaniko/.docker
-      - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
-      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
-      - TAG=$(date +%Y-%m-%d)
-      - |
-        /kaniko/executor \
-          --context . \
-          --dockerfile Containerfile \
-          --destination gitea.renner.uno/gitea_admin/ci-tofu:$TAG \
-          --destination gitea.renner.uno/gitea_admin/ci-tofu:latest
+      DOCKER_CONFIG: "${CI_WORKSPACE}/.docker"
+    entrypoint:
+      - /kaniko/executor
+      - --context=.
+      - --dockerfile=Containerfile
+      - --destination=gitea.renner.uno/gitea_admin/ci-tofu:2026-08-23
+      - --destination=gitea.renner.uno/gitea_admin/ci-tofu:latest

  # NOTE: ci-claude is deliberately NOT built here. It tracks upstream Claude
  # Code releases on its own cadence and is built + version-tagged by
@@ -108,28 +105,21 @@ steps:
  # Runtime image, not CI tooling: a static musl binary in a FROM-scratch image,
  # used as the base for secret-rotation CronJobs. Replaces per-job inline Python.
  - name: build-secret-rotator
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: gcr.io/kaniko-project/executor:v1.24.0
    directory: secret-rotator
-    depends_on: []
+    depends_on: [registry-auth]
    when:
      - event: manual
      - event: push
        path: ["secret-rotator/**", ".woodpecker/build-ci-images.yml"]
    environment:
-      GITEA_USER: gitea_admin
-      GITEA_TOKEN:
-        from_secret: ci_images_token
-    commands:
-      - mkdir -p /kaniko/.docker
-      - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
-      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
-      - TAG=$(date +%Y-%m-%d)
-      - |
-        /kaniko/executor \
-          --context . \
-          --dockerfile Containerfile \
-          --destination gitea.renner.uno/gitea_admin/secret-rotator:$TAG \
-          --destination gitea.renner.uno/gitea_admin/secret-rotator:latest
+      DOCKER_CONFIG: "${CI_WORKSPACE}/.docker"
+    entrypoint:
+      - /kaniko/executor
+      - --context=.
+      - --dockerfile=Containerfile
+      - --destination=gitea.renner.uno/gitea_admin/secret-rotator:2026-08-23
+      - --destination=gitea.renner.uno/gitea_admin/secret-rotator:latest

  # Runtime image, not CI tooling: a general-purpose alpine + jq base for
  # workloads that need to read JSON from a shell script. Named for what it IS,
@@ -137,28 +127,21 @@ steps:
  # container, but nothing about it is zigbee-specific. Exists so tools come from
  # the image rather than an `apk add` on every container start.
  - name: build-alpine-jq
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: gcr.io/kaniko-project/executor:v1.24.0
    directory: ci/images/alpine-jq
-    depends_on: []
+    depends_on: [registry-auth]
    when:
      - event: manual
      - event: push
        path: ["ci/images/alpine-jq/**", ".woodpecker/build-ci-images.yml"]
    environment:
-      GITEA_USER: gitea_admin
-      GITEA_TOKEN:
-        from_secret: ci_images_token
-    commands:
-      - mkdir -p /kaniko/.docker
-      - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
-      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
-      - TAG=$(date +%Y-%m-%d)
-      - |
-        /kaniko/executor \
-          --context . \
-          --dockerfile Containerfile \
-          --destination gitea.renner.uno/gitea_admin/alpine-jq:$TAG \
-          --destination gitea.renner.uno/gitea_admin/alpine-jq:latest
+      DOCKER_CONFIG: "${CI_WORKSPACE}/.docker"
+    entrypoint:
+      - /kaniko/executor
+      - --context=.
+      - --dockerfile=Containerfile
+      - --destination=gitea.renner.uno/gitea_admin/alpine-jq:2026-08-23
+      - --destination=gitea.renner.uno/gitea_admin/alpine-jq:latest

  # Runtime image, not CI tooling: upstream containernetworking/plugins binaries
  # baked in, consumed by the multus cni-plugins-installer DaemonSet. Replaces a
@@ -166,28 +149,21 @@ steps:
  # critical path of every node rebuild. The download now happens here, once,
  # where the sha256 is checked — see the Containerfile for the full rationale.
  - name: build-cni-plugins
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: gcr.io/kaniko-project/executor:v1.24.0
    directory: ci/images/cni-plugins
-    depends_on: []
+    depends_on: [registry-auth]
    when:
      - event: manual
      - event: push
        path: ["ci/images/cni-plugins/**", ".woodpecker/build-ci-images.yml"]
    environment:
-      GITEA_USER: gitea_admin
-      GITEA_TOKEN:
-        from_secret: ci_images_token
-    commands:
-      - mkdir -p /kaniko/.docker
-      - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
-      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
-      - TAG=$(date +%Y-%m-%d)
-      - |
-        /kaniko/executor \
-          --context . \
-          --dockerfile Containerfile \
-          --destination gitea.renner.uno/gitea_admin/cni-plugins:$TAG \
-          --destination gitea.renner.uno/gitea_admin/cni-plugins:latest
+      DOCKER_CONFIG: "${CI_WORKSPACE}/.docker"
+    entrypoint:
+      - /kaniko/executor
+      - --context=.
+      - --dockerfile=Containerfile
+      - --destination=gitea.renner.uno/gitea_admin/cni-plugins:2026-08-23
+      - --destination=gitea.renner.uno/gitea_admin/cni-plugins:latest

  # Runtime image, not CI tooling: alpine + the openssl CLI, for shell scripts
  # needing s_client/x509. Named for what it is; first consumer is the paperless
@@ -195,28 +171,21 @@ steps:
  # every paperless pod start. The openssl CLI is genuinely absent from the alpine
  # base, so this one needs an image rather than a busybox substitute.
  - name: build-alpine-openssl
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: gcr.io/kaniko-project/executor:v1.24.0
    directory: ci/images/alpine-openssl
-    depends_on: []
+    depends_on: [registry-auth]
    when:
      - event: manual
      - event: push
        path: ["ci/images/alpine-openssl/**", ".woodpecker/build-ci-images.yml"]
    environment:
-      GITEA_USER: gitea_admin
-      GITEA_TOKEN:
-        from_secret: ci_images_token
-    commands:
-      - mkdir -p /kaniko/.docker
-      - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
-      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
-      - TAG=$(date +%Y-%m-%d)
-      - |
-        /kaniko/executor \
-          --context . \
-          --dockerfile Containerfile \
-          --destination gitea.renner.uno/gitea_admin/alpine-openssl:$TAG \
-          --destination gitea.renner.uno/gitea_admin/alpine-openssl:latest
+      DOCKER_CONFIG: "${CI_WORKSPACE}/.docker"
+    entrypoint:
+      - /kaniko/executor
+      - --context=.
+      - --dockerfile=Containerfile
+      - --destination=gitea.renner.uno/gitea_admin/alpine-openssl:2026-08-23
+      - --destination=gitea.renner.uno/gitea_admin/alpine-openssl:latest

  # Heating CAN readers: open3e (Vitocal/VX3) + E3onCAN (E380 meter), one image
  # run as two services on edge-1. Replaces pip-into-venv on the host, where an
@@ -224,25 +193,18 @@ steps:
  # Ansible converge. Not a ci-* image -- it is a runtime artifact -- but it
  # builds the same way and lands in the same registry so the audit sees it.
  - name: build-heating-can
-    image: gcr.io/kaniko-project/executor:v1.23.2-debug
+    image: gcr.io/kaniko-project/executor:v1.24.0
    directory: ci/images/heating-can
-    depends_on: []
+    depends_on: [registry-auth]
    when:
      - event: manual
      - event: push
        path: ["ci/images/heating-can/**", ".woodpecker/build-ci-images.yml"]
    environment:
-      GITEA_USER: gitea_admin
-      GITEA_TOKEN:
-        from_secret: ci_images_token
-    commands:
-      - mkdir -p /kaniko/.docker
-      - AUTH=$(printf '%s:%s' "$GITEA_USER" "$GITEA_TOKEN" | base64 | tr -d '\n')
-      - printf '{"auths":{"gitea.renner.uno":{"auth":"%s"}}}' "$AUTH" > /kaniko/.docker/config.json
-      - TAG=$(date +%Y-%m-%d)
-      - |
-        /kaniko/executor \
-          --context . \
-          --dockerfile Containerfile \
-          --destination gitea.renner.uno/gitea_admin/heating-can:$TAG \
-          --destination gitea.renner.uno/gitea_admin/heating-can:latest
+      DOCKER_CONFIG: "${CI_WORKSPACE}/.docker"
+    entrypoint:
+      - /kaniko/executor
+      - --context=.
+      - --dockerfile=Containerfile
+      - --destination=gitea.renner.uno/gitea_admin/heating-can:2026-08-23
+      - --destination=gitea.renner.uno/gitea_admin/heating-can:latest
@@ -13,7 +13,7 @@ when:

 steps:
  - name: validate-secrets
-    image: alpine:3.24.1
+    image: alpine:3.21
    environment:
      TF_VAR_proxmox_api_url:
        from_secret: proxmox_api_url
@@ -22,3 +22,8 @@ FROM alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d650
 # unanchored extraction therefore yields a device link key, and herdsman then
 # forms a NEW Zigbee network and drops every device.
 RUN apk add --no-cache jq
+
+# Build timestamp, recorded in the image rather than the tag: the tag says which
+# source revision was built, this says when. Last layer, so it only invalidates
+# itself.
+RUN date -u +%Y-%m-%dT%H:%M:%SZ > /etc/image-build-date
@@ -19,3 +19,8 @@ FROM alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d650
 # for paperless to trust. busybox has no s_client equivalent, so openssl is not
 # substitutable there.
 RUN apk add --no-cache openssl
+
+# Build timestamp, recorded in the image rather than the tag: the tag says which
+# source revision was built, this says when. Last layer, so it only invalidates
+# itself.
+RUN date -u +%Y-%m-%dT%H:%M:%SZ > /etc/image-build-date
@@ -24,3 +24,8 @@ RUN pip3 install --break-system-packages --no-cache-dir \
 RUN ansible-galaxy collection install \
      community.proxmox \
      community.docker
+
+# Build timestamp, recorded in the image rather than the tag: the tag says which
+# source revision was built, this says when. Last layer, so it only invalidates
+# itself.
+RUN date -u +%Y-%m-%dT%H:%M:%SZ > /etc/image-build-date
@@ -44,3 +44,9 @@ RUN npm install -g @anthropic-ai/claude-code@${CLAUDE_CODE_VERSION} \
      /usr \
      /root \
      /tmp
+
+# Build timestamp, recorded in the image rather than the tag: the tag says which
+# version was built, this says when. Deliberately the LAST layer and its own
+# layer, so the byte-stable npm layer above still dedups in the registry — that
+# dedup is what #351 was for, and a few dozen bytes here does not affect it.
+RUN date -u +%Y-%m-%dT%H:%M:%SZ > /etc/image-build-date
@@ -66,3 +66,8 @@ FROM alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d650
 # Alpine rather than scratch: the installer needs a shell to copy the binaries
 # onto the host. busybox covers cp/mv/chown/ls, so nothing is added.
 COPY --from=fetch /plugins/ /opt/cni-plugins/
+
+# Build timestamp, recorded in the image rather than the tag: the tag says which
+# source revision was built, this says when. Last layer, so it only invalidates
+# itself.
+RUN date -u +%Y-%m-%dT%H:%M:%SZ > /etc/image-build-date
@@ -120,3 +120,8 @@ USER open3e
 WORKDIR /var/lib/open3e

 ENTRYPOINT ["supervisord", "-c", "/etc/supervisord.conf"]
+
+# Build timestamp, recorded in the image rather than the tag: the tag says which
+# source revision was built, this says when. Last layer, so it only invalidates
+# itself.
+RUN date -u +%Y-%m-%dT%H:%M:%SZ > /etc/image-build-date
@@ -42,3 +42,8 @@ RUN curl -sfL -o /tmp/velero.tar.gz \
 && tar -xz -C /tmp -f /tmp/velero.tar.gz \
 && install "/tmp/velero-${VELERO_VERSION}-linux-amd64/velero" /usr/local/bin/velero \
 && rm -rf "/tmp/velero-${VELERO_VERSION}-linux-amd64" /tmp/velero.tar.gz
+
+# Build timestamp, recorded in the image rather than the tag: the tag says which
+# source revision was built, this says when. Last layer, so it only invalidates
+# itself.
+RUN date -u +%Y-%m-%dT%H:%M:%SZ > /etc/image-build-date
@@ -31,3 +31,8 @@ RUN pip3 install --break-system-packages --no-cache-dir \
 RUN ansible-galaxy collection install \
      community.proxmox \
      community.docker
+
+# Build timestamp, recorded in the image rather than the tag: the tag says which
+# source revision was built, this says when. Last layer, so it only invalidates
+# itself.
+RUN date -u +%Y-%m-%dT%H:%M:%SZ > /etc/image-build-date
@@ -84,7 +84,7 @@ spec:
          env:
            - name: DB_URI
              valueFrom:
-                secretKeyRef: {name: ha-pg-app, key: fqdn-uri}
+                secretKeyRef: { name: ha-pg-app, key: fqdn-uri }
          volumeMounts:
            - name: config
              mountPath: /config
@@ -96,7 +96,7 @@ spec:
              memory: 32Mi
      containers:
        - name: home-assistant
-          image: ghcr.io/home-assistant/home-assistant:2026.8.3 # {"$imagepolicy": "flux-system:home-assistant"}
+          image: ghcr.io/home-assistant/home-assistant:2026.8.2 # {"$imagepolicy": "flux-system:home-assistant"}
          env:
            - name: TZ
              value: Europe/Berlin
@@ -176,22 +176,3 @@ data:
        friendly_name: Treppenhaus Haustür Schalter 1
      '0x001788010b00d011':
        friendly_name: Treppenhaus Haustür Schalter 2
-      # Ecowitt WS90 weather station. Joined after the ZHA-mirrored block above
-      # was written, hence listed apart from it.
-      #
-      # The station transmits every 8.8s, as a burst of seven per-cluster ZCL
-      # frames; Z2M publishes its full cumulative state once per frame. debounce
-      # merges a burst into one publish carrying the last value per attribute,
-      # and must stay below the transmit interval — a window that never sees
-      # silence never flushes.
-      #
-      # The firmware drops all readings to zero for a few seconds every few
-      # days. Merged, that surfaces as a single all-zero frame; unmerged, the
-      # zeros spread across the seven frames and none of them is individually
-      # distinguishable from a valid reading.
-      #
-      # Z2M reads this only at startup (requiresRestart), and this file is
-      # SEED-ONLY per the header — the running instance takes it from its PVC.
-      '0xc02cedfffe958f40':
-        friendly_name: Wetterstation
-        debounce: 2
@@ -158,7 +158,7 @@ spec:
          # Pinned tag. No Flux image-automation marker yet (unlike mosquitto) —
          # add an infrastructure/image-automation/zigbee2mqtt.yaml policy as a
          # follow-up if desired; pinned-and-manual is fine for the initial cutover.
-          image: ghcr.io/koenkk/zigbee2mqtt:2.13.0
+          image: ghcr.io/koenkk/zigbee2mqtt:2.12.1
          imagePullPolicy: IfNotPresent
          env:
            - name: TZ
@@ -8,7 +8,6 @@ resources:
  - prometheusrule-cilium.yaml
  - prometheusrule-hetzner-backup.yaml
  - prometheusrule-loki.yaml
-  - prometheusrule-mariadb-backup.yaml
  - prometheusrule-netbird-availability.yaml
  - prometheusrule-netbird-backup.yaml
  - prometheusrule-netbird-ingress.yaml
@@ -25,36 +25,13 @@ spec:
          #
          # 6h: Ceph warns for self-clearing conditions. Only a warning that
          # does not clear is worth paging on.
-          #
-          # BLUESTORE_SLOW_OP_ALERT breaks that premise and is still NOT
-          # excluded (#557). Ceph holds the check for
-          # bluestore_slow_ops_warn_lifetime (24h) after the LAST slow op, so
-          # one multi-second stall outlives the 6h `for` and always pages,
-          # with latency back to normal long before anyone looks. Measured:
-          # three episodes in the 30d to 2026-08-24 (Sun Jul 26, Aug 2 and
-          # Aug 23), each ran ~24-26h and cleared untouched.
-          #
-          # All began inside the weekly Velero restore-test window (Sun 08:00Z
-          # plus up to 6h, see velero/restore-test-cronjob.yaml) — a sustained
-          # restore that outruns the SLC cache on these QLC NVMe. Expected for
-          # the hardware, but not excludable: this is also the earliest signal
-          # of a genuinely failing drive, and all three are at 80-87%
-          # endurance used (see cephcluster.yaml).
-          #
-          # No metric names the stalling OSD: ceph_health_detail carries no osd
-          # label, and ceph_daemon_health_metrics{type="SLOW_OPS"} reads 0 at
-          # every scrape (a stall is shorter than the scrape interval). Only
-          # the OSD's own log has it, hence the Loki query in the description.
-          # Jul 26 and Aug 2 hit osd.0+1+2 together, which is load-driven;
-          # Aug 23 hit osd.1 alone in one burst. If osd.1 alone repeats,
-          # suspect that drive rather than the restore load.
          expr: ceph_health_detail{name!~"AUTH_INSECURE_.+"} == 1
          for: 6h
          labels:
            severity: warning
          annotations:
            summary: "Ceph health check {{ $labels.name }} active for 6h"
-            description: "Ceph health check {{ $labels.name }} has been active for six hours. Only AUTH_INSECURE_* is excluded (kernel-blocked, see #527). BLUESTORE_SLOW_OP_ALERT is sticky for 24h after the last stall and clears itself: check whether the stall fell inside the Sunday 08:00-14:00Z Velero restore-test window (08:00 start, 6h activeDeadlineSeconds cap), which is expected on these QLC drives — one outside that window, or a repeat on the same OSD, means check that drive (#557). No metric names the OSD — get it from Loki, which also works after the check clears: `{namespace=\"rook-ceph\",pod=~\"rook-ceph-osd-.*\"} |= \"slow operation observed\"`. While still active, `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail` adds the op detail."
+            description: "Ceph health check {{ $labels.name }} has been active for six hours, so it is not a self-clearing blip. Only AUTH_INSECURE_* is excluded (kernel-blocked, see #527), so this is a new condition. Detail: `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail`."

        - alert: CephPoolLostRedundancy
          # Names the pool, which POOL_NO_REDUNDANCY does not.
@@ -1,51 +0,0 @@
-apiVersion: monitoring.coreos.com/v1
-kind: PrometheusRule
-metadata:
-  name: mariadb-backup
-  namespace: monitoring
-  labels:
-    # kube-prometheus-stack's Prometheus selects on this label.
-    release: kube-prometheus-stack
-spec:
-  groups:
-    - name: mariadb-backup.rules
-      rules:
-        - alert: WordpressMariadbBackupStale
-          # The nightly 02:45 logical dump (kubernetes/apps/wordpress/mariadb.yaml)
-          # is the only transaction-consistent copy of www.renner.uno's database
-          # — Velero snapshots the PVC, which for a running DB is not a restore
-          # point you would trust.
-          #
-          # KubeJobFailed already covers a failed Job, but only while the Job
-          # object survives. On #550 it cleared when the failed Job was
-          # garbage-collected, not when a backup succeeded — so a failure that
-          # is GC'd before anyone looks leaves no signal at all.
-          # lastSuccessfulTime lives on the CronJob and outlives Job GC.
-          #
-          # 26h = one missed cycle plus 2h slack, matching NetbirdVpsBackupStale.
-          # It counts any Job owned by the CronJob, so a manual
-          # `kubectl create job --from=cronjob/...` re-run clears it too.
-          expr: |
-            time() - kube_cronjob_status_last_successful_time{namespace="wordpress", cronjob="wordpress-mariadb-backup"} > 26 * 3600
-          for: 30m
-          labels:
-            severity: warning
-          annotations:
-            summary: "WordPress MariaDB backup has not succeeded in 26h+"
-            description: "The wordpress-mariadb-backup CronJob's last successful run is over 26 hours old. Check the newest Job's pod logs in the wordpress namespace: `kubectl -n wordpress get jobs`. The 2026-08-19 failure was `The specified bucket does not exist.` against renner-uno-k8s-mariadb while the endpoint itself probed healthy, so a green HetznerBackupEndpointDown does not clear this."
-
-        - alert: WordpressMariadbBackupNeverRan
-          # The staleness expression above goes blind when the metric is
-          # absent, which is what happens once the operator recreates the
-          # CronJob. Without this guard "never backed up" is quieter than
-          # "backed up late".
-          expr: |
-            kube_cronjob_created{namespace="wordpress", cronjob="wordpress-mariadb-backup"}
-            unless
-            kube_cronjob_status_last_successful_time{namespace="wordpress", cronjob="wordpress-mariadb-backup"}
-          for: 6h
-          labels:
-            severity: warning
-          annotations:
-            summary: "WordPress MariaDB backup has no recorded successful run"
-            description: "The wordpress-mariadb-backup CronJob exists but has no lastSuccessfulTime: either it has not completed once since the Backup CR was recreated, or every run since has failed. Check `kubectl -n wordpress get jobs` and that the hetzner-s3-creds ExternalSecret is synced."
@@ -34,7 +34,7 @@ spec:
          # Image automation, range 3.x (see image-automation/woodpecker.yaml).
          # MUST stay version-matched with woodpecker-server (gRPC parity) — both
          # track the same range/filter so they bump together on each release.
-          image: woodpeckerci/woodpecker-agent:v3.18.0 # {"$imagepolicy": "flux-system:woodpecker-agent"}
+          image: woodpeckerci/woodpecker-agent:v3.17.0 # {"$imagepolicy": "flux-system:woodpecker-agent"}
          env:
            - name: WOODPECKER_SERVER
              value: woodpecker.woodpecker.svc.cluster.local:9000 # in-cluster gRPC, replaces LXC :9000
@@ -40,7 +40,7 @@ spec:
          # minor + patch auto-update within 3.x; only the 4.0 major is held for
          # a reviewed bump. Safe now the DB is postgres (the old sqlite-surprise
          # -bump hazard is gone). Stays in lockstep with the agent.
-          image: woodpeckerci/woodpecker-server:v3.18.0 # {"$imagepolicy": "flux-system:woodpecker-server"}
+          image: woodpeckerci/woodpecker-server:v3.17.0 # {"$imagepolicy": "flux-system:woodpecker-server"}
          env:
            - name: WOODPECKER_HOST
              value: https://ci.renner.uno
@@ -51,15 +51,7 @@ spec:
  startingDeadlineSeconds: 3600
  jobTemplate:
    spec:
-      # 4 (not the sibling rotators' 1-2) because this is the only DAILY
-      # rotator — every other CronJob here is quarterly, so it takes ~90x as
-      # many runs and is correspondingly more exposed to per-run transients.
-      # backoffLimit: 1 gave two attempts ~11 s apart, and on 2026-08-19 a
-      # sub-minute resolver transient took both out (Job
-      # claude-oauth-secret-sync-29785100, BackoffLimitExceeded), costing a
-      # day's sync. 4 spreads the attempts over ~150 s (10/20/40/80 s backoff),
-      # still well inside activeDeadlineSeconds.
-      backoffLimit: 4
+      backoffLimit: 1
      activeDeadlineSeconds: 300
      ttlSecondsAfterFinished: 259200 # 3 d
      template:
@@ -21,8 +21,14 @@ COPY src ./src
 RUN cargo build --release --target x86_64-unknown-linux-musl \
 && strip target/x86_64-unknown-linux-musl/release/secret-rotator

+# Build timestamp, recorded in the image rather than the tag: the tag says which
+# source revision was built, this says when. Written here because the final
+# image is scratch and has no shell to run `date` in.
+RUN date -u +%Y-%m-%dT%H:%M:%SZ > /build/image-build-date
+
 FROM scratch
 COPY --from=build /build/target/x86_64-unknown-linux-musl/release/secret-rotator /secret-rotator
+COPY --from=build /build/image-build-date /etc/image-build-date

 # Matches runAsUser on the rotation Jobs. There is no /etc/passwd in a scratch
 # image, so this is a bare uid by necessity — which is also all Kubernetes needs.
Author	SHA1	Message	Date
claude-bot	803c741362	fix(ci): tag images with a dated literal, not the commit SHA ci/woodpecker/push/build-ci-images Pipeline failed Details ci/woodpecker/pr/ci-manifests-test Pipeline was successful Details ci/woodpecker/pr/ci-scripts-test Pipeline was successful Details ci/woodpecker/pr/pr-review Pipeline was successful Details The SHA tag was a workaround for not being able to compute a date in a shell-less build step. That problem is gone — the build timestamp now lives in the image at /etc/image-build-date — so the tag no longer has to carry it. A SHA reads as noise at the point of use and breaks the existing convention every consumer pin follows (ci-k8s:2026-08-10, ci-ansible:2026-04-29, alpine-jq:2026-07-30). Dated literals keep the pins uniform and are bumped in the commit that changes an image, which is the same trigger the `when:` path filters already use.	2026-08-23 21:14:51 +02:00
claude-bot	aff01d5a32	build(ci): record the build timestamp in ci-claude too ci/woodpecker/push/build-ci-images Pipeline was successful Details ci/woodpecker/pr/ci-manifests-test Pipeline was successful Details ci/woodpecker/pr/ci-scripts-test Pipeline was successful Details ci/woodpecker/pr/pr-review Pipeline was successful Details Completes the pattern across every Containerfile in the repo. ci-claude was the last one without it. It was skipped initially over the reproducibility work in `a675bf6c`, on a misreading. That commit did not make the whole image reproducible — it made the npm install layer byte-stable so a rebuild of the same version dedups in the registry instead of orphaning ~260 MB (#336, #351). A timestamp added as its own final layer leaves the npm layer untouched, so that dedup still holds. The cost is a few dozen bytes per build, not 260 MB.	2026-08-23 20:59:29 +02:00
claude-bot	80c3051dc7	build(ci): record the build timestamp in the image instead of the tag ci/woodpecker/pr/ci-manifests-test Pipeline was successful Details ci/woodpecker/push/build-ci-images Pipeline failed Details ci/woodpecker/pr/ci-scripts-test Pipeline was successful Details ci/woodpecker/pr/pr-review Pipeline was canceled Details The date was previously computed by `date` in the pipeline step and used as the image tag, which is why the build step needed a shell and hence the kaniko debug image. It does not need to cross between CI steps at all. The base images have a shell, so a RUN in the Containerfile writes the timestamp during the build itself — kaniko executes the layer and the value lands in the image. Nothing has to be propagated, and the build step stays shell-less. This also separates two things the tag was conflating: the tag says which source revision was built, /etc/image-build-date says when. Placed last so it invalidates only its own layer. secret-rotator writes it in the rust build stage and copies it across, since its final image is FROM scratch and has no shell. Verified against ci/images/alpine-jq: $ docker run --rm localtest:jq cat /etc/image-build-date 2026-08-23T18:56:51Z Note this makes builds non-reproducible byte-for-byte, which `a675bf6c` deliberately arranged for ci-claude. ci-claude is built by ci-claude-refresh.yml and is not touched here.	2026-08-23 20:57:15 +02:00
claude-bot	bbbc42c79f	fix(ci): build with the plain kaniko executor instead of the debug image ci/woodpecker/pr/ci-manifests-test Pipeline was successful Details ci/woodpecker/push/build-ci-images Pipeline failed Details ci/woodpecker/pr/ci-scripts-test Pipeline was successful Details ci/woodpecker/pr/pr-review Pipeline was successful Details The `-debug` image ships a busybox shell so a failing build can be inspected. It was being used as the build image because the steps run shell to assemble registry auth and compute a tag, which is not what that variant is for. The plain executor has no shell, so the eight static builds now pass flags via `entrypoint` and read credentials from DOCKER_CONFIG. A single registry-auth step writes `.docker/config.json` into the workspace once, in alpine, where a shell is the intended interface. Images are tagged `${CI_COMMIT_SHA:0:8}` instead of a build date. Woodpecker exposes no formatted-date variable — every timestamp built in is UNIX — and kaniko takes `--destination` only as a literal argument, with all its file-based flags being outputs. The commit SHA is available through normal substitution and identifies the source revision the image was built from, which a build date does not. Existing dated tags stay in the registry, so the pins that reference them keep resolving; only newly created tags change format. Also moves off v1.23.2, picking up the go-git fix for CVE-2025-21613 and containerd 1.7.18 -> 1.7.27. ci-claude-refresh.yml keeps its shell step: its destination tag is a runtime value read from a file written by an earlier step, and the build is conditionally skipped, neither of which survives a static entrypoint. Left for #552. Needs a pipeline run to confirm that substitution applies to `entrypoint` and `environment` values, and that the auth file written by registry-auth is visible to the build steps. Refs #426, #552	2026-08-23 20:52:06 +02:00