perf(grafana-db): raise archive_timeout to 30min on gf-pg

A WAL segment is a fixed 16 MiB file. archive_timeout forces a segment switch on a timer so the archive never trails reality by more than that interval, and the switched-out segment is written and archived at its full 16 MiB no matter how few bytes of real WAL it holds. CNPG defaults this to 5min, which permits up to 288 segments/day = 4.6 GB/day per cluster. Measured on gf-pg: the database is 21 MB and generates 4.5 MB/day of actual WAL, while its RBD volume takes 211 MB/hour -- against 192 MB/hour predicted from 12 switches of 16 MiB. Essentially every byte written to this volume is padding, roughly 1100x amplification. This resolves an anomaly that has been open since the start of the endurance work: gf-pg appeared to write gigabytes per day for a 21 MB database, and neither WAL volume nor data changes could account for it. Segment padding does, exactly. Applied to gf-pg alone as a staged rollout. It is the lowest-risk cluster and gives the cleanest signal -- its real WAL is small enough that any change in volume is unambiguously attributable to this parameter. The same default affects all seven clusters and is worth an estimated ~24 GB/day of logical writes fleet-wide, but that only goes ahead once this one is verified. The cost is RPO. On total loss of the cluster, up to 30min of Grafana state becomes unrecoverable instead of 5min. That state is dashboards, annotations and preferences, changing at ~4.5 MB/day, so the window is nearly always empty. Clusters holding credentials or job state get evaluated separately rather than inheriting this value by default. archive_timeout is in CNPG's GlobalDefaultSettings, not its FixedConfigurationParameters, so it is user-overridable -- verified against pkg/postgres/configuration.go rather than assumed. Context: drive-endurance work on three QLC NVMe OSDs at 81-87% wear.
2026-08-20 14:21:03 +02:00
32 changed files with 154 additions and 720 deletions
@@ -13,7 +13,7 @@ when:

 steps:
  - name: validate-secrets
-    image: alpine:3.24.1
+    image: alpine:3.21
    environment:
      TF_VAR_proxmox_api_url:
        from_secret: proxmox_api_url
@@ -129,20 +129,13 @@ steps:
      - echo "$VAULT_PASS" > /tmp/.vault_pass && chmod 600 /tmp/.vault_pass
      - export ANSIBLE_VAULT_PASSWORD_FILE=/tmp/.vault_pass
      - |
-        # Ensure Talos image exists on all Proxmox nodes via API.
-        #
-        # jupiter is NOT in this list: it runs Talos on bare metal and is no
-        # longer a Proxmox node, so the storage API returns nothing for it and
-        # the download-wait loop below would burn its full 60x10s before
-        # giving up. saturn and neptune still host k8s-2/k8s-3 as VMs and
-        # still need the nocloud image; bare-metal nodes boot the metal-amd64
-        # image from their own installed disk instead.
+        # Ensure Talos image exists on all Proxmox nodes via API
        TALOS_VERSION="v1.12.6"
        TALOS_SCHEMATIC="88e3752db032a08082972bd0adf16a80849e3870c3f95cf5b669936c5699e5ad"
        TALOS_URL="https://factory.talos.dev/image/$TALOS_SCHEMATIC/$TALOS_VERSION/nocloud-amd64.raw"
        TALOS_FILENAME="talos-$TALOS_VERSION-$(echo "$TALOS_SCHEMATIC" | cut -c1-12)-nocloud-amd64.raw"
        API_TOKEN="$TF_VAR_proxmox_user!$TF_VAR_proxmox_token_id=$TF_VAR_proxmox_token_secret"
-        for node in saturn neptune; do
+        for node in jupiter saturn neptune; do
          EXISTS=$(curl -sk "$TF_VAR_proxmox_api_url/api2/json/nodes/$node/storage/local/content" \
            -H "Authorization: PVEAPIToken=$API_TOKEN" 2>/dev/null | grep -c "$TALOS_FILENAME" || true)
          if [ "$EXISTS" -gt 0 ]; then
@@ -167,7 +167,7 @@ permanent recovery.

 ## CNPG restore-test (verify backups are restorable — non-destructive)

-The monthly Velero restore-test (`kubernetes/infrastructure/velero/`) exercises
+The weekly Velero restore-test (`kubernetes/infrastructure/velero/`) exercises
 PVC restores but **explicitly does not touch CNPG** (barman-cloud is a separate
 code path). Run this drill periodically to prove the CNPG backups are actually
 restorable, **without touching the live clusters**. It restores into a throwaway
@@ -79,30 +79,6 @@ which also pre-seeds the Talos image onto every Proxmox node. Wait until the
 node FQDNs (`k8s-{1,2,3}.srv.renner.uno`) resolve and the VMs are up before
 Step 2 — `bootstrap.sh` polls DNS but will not create the records.

-**k8s-1 is not a VM and OpenTofu does not create it.** jupiter runs Talos on
-bare metal, so this step only provisions k8s-2 (saturn) and k8s-3 (neptune).
-Before Step 2, boot jupiter from the bare-metal image below so it comes up in
-maintenance mode and answers on `k8s-1.srv.renner.uno` — `bootstrap.sh` treats
-all three nodes identically from there, since `apply-config --insecure` does
-not care whether the target is a VM or bare metal.
-
-```
-https://factory.talos.dev/image/a21cec8f35ae32ab5e9a452e7b13500ca8776053930b8e182b9cb23eed186b64/v1.12.6/metal-amd64.iso
-```
-
-That schematic is deliberately not the one `.woodpecker/deploy-infrastructure.yml`
-seeds onto the Proxmox nodes. It keeps the same `net.ifnames=1` kernel argument
-— without it the predictable interface names that `talos/patches/k8s-1.yaml`
-matches its VLAN parents against do not exist — and differs in three ways:
-
- drops `siderolabs/qemu-guest-agent`, which has nothing to talk to on bare metal
- adds `siderolabs/intel-ucode`, since no hypervisor is supplying microcode now
- adds `siderolabs/thunderbolt`, unused so far and inert until
-  `machine.kernel.modules` requests `thunderbolt`/`thunderbolt_net`. It is
-  present only so enabling the MS-01 Thunderbolt mesh later does not require a
-  new image and a reinstall. Note its udev rule auto-authorises any Thunderbolt
-  device attached to the machine once those modules are loaded.
-
 ## Step 2 — talos: form the cluster + get kubeconfig

 ```sh
@@ -67,20 +67,6 @@ spec:
      # and slower than the lz4/zstd this build also accepts). Compresses the
      # full-page images that dominate WAL volume after each checkpoint.
      wal_compression: "zstd"
-      # A WAL segment is a fixed 16 MiB file, and archive_timeout forces a switch
-      # on a timer -- the segment is then written and archived at its full size
-      # no matter how little real WAL it holds. CNPG defaults to 5min, i.e. up
-      # to 288 segments/day = 3.2 GB/day here, most of it padding.
-      #
-      # 15min, NOT the 30min used elsewhere in this rollout. This cluster holds
-      # credentials, sessions and enrolment state, so its RPO is worth more than
-      # the difference: 30min would save a further 0.3 GB/day, which does not
-      # justify doubling the worst-case loss window on this data.
-      #
-      # It does carry a synchronous standby (instances: 2), so the realistic
-      # exposure is losing both instances, not one -- but the conservative value
-      # is cheap here and the aggressive one buys almost nothing.
-      archive_timeout: "15min"
    # Async replication makes CNPG switchover slow because the standby has
    # to chase missing WAL via barman-cloud and replay queued records
    # before it can promote. With one synchronous standby, the standby is
@@ -84,7 +84,7 @@ spec:
          env:
            - name: DB_URI
              valueFrom:
-                secretKeyRef: {name: ha-pg-app, key: fqdn-uri}
+                secretKeyRef: { name: ha-pg-app, key: fqdn-uri }
          volumeMounts:
            - name: config
              mountPath: /config
@@ -96,7 +96,7 @@ spec:
              memory: 32Mi
      containers:
        - name: home-assistant
-          image: ghcr.io/home-assistant/home-assistant:2026.8.3 # {"$imagepolicy": "flux-system:home-assistant"}
+          image: ghcr.io/home-assistant/home-assistant:2026.8.2 # {"$imagepolicy": "flux-system:home-assistant"}
          env:
            - name: TZ
              value: Europe/Berlin
@@ -86,26 +86,8 @@ spec:
      # inserts scattered across pages, so full-page writes after each
      # checkpoint dominate WAL volume; compression cuts that directly.
      #
-      # Full-page images really are ~62% of WAL here: 61.9% measured as a
-      # steady-state delta over one checkpoint cycle, against 62.4% lifetime.
-      #
-      # But switching pglz -> zstd changed NOTHING. Measured by
-      # pg_current_wal_lsn advance over 20min windows: 4911 and 4976 MB/day
-      # before, 5002 MB/day after.
-      #
-      # The reason is that `wal_compression: "on"` was never "no compression" --
-      # it is pglz, which was already compressing these page images. zstd only
-      # changes the ratio, and on recorder pages that margin is too small to
-      # measure. The original framing of this change (that the clusters "were
-      # not getting compression") was wrong; they were.
-      #
-      # Note the 62% figure is an upper bound: it is computed as
-      # wal_fpi * 8192 / wal_bytes, which assumes uncompressed 8 KiB pages. With
-      # compression active the true byte share is lower, which is consistent
-      # with there being little headroom left for a better algorithm.
-      #
-      # Kept because zstd is not worse than pglz and costs nothing. It is simply
-      # not a lever. The real one on this cluster was archive_timeout, below.
+      # Measured on this cluster: full-page images are 62.4% of WAL bytes, so
+      # this is the single largest lever on recorder write volume.
      #
      # zstd, not "on": the boolean spelling resolves to pglz, the original
      # 2016-era algorithm, which is both slower and weaker than the two
@@ -113,35 +95,9 @@ spec:
      # best ratio on 8 KiB page images, and the CPU cost lands on a cluster
      # that is IO-bound on QLC NVMe, not CPU-bound.
      wal_compression: "zstd"
-      # A WAL segment is a fixed 16 MiB file, and archive_timeout forces a switch
-      # on a timer -- the segment is then written and archived at its full size
-      # no matter how little real WAL it holds. CNPG defaults to 5min, i.e. up
-      # to 288 segments/day = 4.6 GB/day that is largely padding.
-      #
-      # Verified on gf-pg first (the staged rollout): 211 -> 41 MB/hour, an 81%
-      # cut, with archived segments falling from ~12/hour to ~1.5/hour in step.
-      # Two independent signals moving together is what confirms the volume was
-      # padding rather than data.
-      #
-      # The cost is RPO: up to 30min of WAL is unrecoverable if this cluster and
-      # its volume are lost outright. This is a single-instance cluster, so the
-      # archive is the only off-node copy.
-      archive_timeout: "30min"
      # Fewer checkpoints => fewer full-page writes, since a page is only
-      # fully-logged on its first modification after a checkpoint.
-      #
-      # That reasoning did not show up in WAL volume: 15min -> 30min moved the
-      # WAL rate not at all (4911/4976 -> 5002 MB/day). The likely reason is
-      # that this workload is append-mostly -- a page is dirtied while it fills
-      # and then not touched again, so its full-page image is paid once
-      # regardless of how often checkpoints run. Interval only helps pages that
-      # are re-dirtied across many cycles, which is not what a recorder does.
-      #
-      # Kept, but NOT because WAL improved. checkpoint_timeout also governs
-      # data-file writes at checkpoint, and those are unmeasured here: this
-      # volume takes ~11 GB/day against 4.9 GB/day of WAL, so most of its
-      # traffic is outside what the WAL measurement covers. Revisit with a
-      # data-file measurement before treating this value as justified.
+      # fully-logged on its first modification after a checkpoint. Doubling the
+      # interval halves how often each hot page pays that 8 KiB toll.
      #
      # Recovery cost is bounded by how much WAL a single interval produces,
      # not by the interval itself: at the measured rate 30min is well under
@@ -176,38 +176,3 @@ data:
        friendly_name: Treppenhaus Haustür Schalter 1
      '0x001788010b00d011':
        friendly_name: Treppenhaus Haustür Schalter 2
-      # Ecowitt WS90 weather station. Joined after the ZHA-mirrored block above
-      # was written, hence listed apart from it.
-      #
-      # This device also carries a non-default ZCL reporting configuration
-      # (per-attribute min interval and reportable-change deadbands). That is
-      # device-side state, not expressible here — Z2M has no settings field for
-      # it — so it is documented, with restore commands, in
-      # zigbee2mqtt-ws90-reporting.md. It is lost whenever Z2M re-runs
-      # `configure` on the device, and a fresh PVC will not have it.
-      #
-      # The station transmits every 8.8s, as a burst of seven per-cluster ZCL
-      # frames; Z2M publishes its full cumulative state once per frame. debounce
-      # merges a burst into one publish carrying the last value per attribute,
-      # and must stay below the transmit interval — a window that never sees
-      # silence never flushes.
-      #
-      # Every few days the firmware zeroes its readings for ~8s, then skips a
-      # transmit cycle and recovers on the next. The zeros propagate attribute
-      # by attribute through a burst rather than arriving together, and they
-      # start inside an otherwise normal cycle — the last valid reading and the
-      # first zero are ~20ms apart.
-      #
-      # Merging cuts an episode from six published bad frames to one. It does
-      # NOT make that frame recognisable: it carries whatever subset had zeroed
-      # when the burst ended, which may be every field or only temperature, and
-      # a frame reading {"temperature":0,"humidity":92,"pressure":96.9} is not
-      # separable from a valid winter reading. Since the merge is last-wins and
-      # the zeros arrive at the tail, it also discards the valid reading that
-      # led the burst. This bounds the corruption; it does not filter it.
-      #
-      # Z2M reads this only at startup (requiresRestart), and this file is
-      # SEED-ONLY per the header — the running instance takes it from its PVC.
-      '0xc02cedfffe958f40':
-        friendly_name: Wetterstation
-        debounce: 2
@@ -158,7 +158,7 @@ spec:
          # Pinned tag. No Flux image-automation marker yet (unlike mosquitto) —
          # add an infrastructure/image-automation/zigbee2mqtt.yaml policy as a
          # follow-up if desired; pinned-and-manual is fine for the initial cutover.
-          image: ghcr.io/koenkk/zigbee2mqtt:2.13.0
+          image: ghcr.io/koenkk/zigbee2mqtt:2.12.1
          imagePullPolicy: IfNotPresent
          env:
            - name: TZ
@@ -1,149 +0,0 @@
-# WS90 Zigbee reporting configuration
-
-Records the ZCL attribute-reporting configuration applied to the Wetterstation
-(Ecowitt WS90 powered by Shelly, IEEE `0xc02cedfffe958f40`) and how to restore
-it, because **this configuration lives on the device and in Z2M's
-`database.db`, not in git**. Zigbee2MQTT has no settings-schema field for
-reporting configuration, so there is nothing declarative to reconcile — Flux
-cannot restore it and a fresh PVC will not have it.
-
-## Applied configuration
-
-Endpoint 1, all intervals in seconds. `change` is in the attribute's raw ZCL
-units; the physical deadband is `change ÷ scale`.
-
-| attribute | cluster | min | max | change | deadband | converter default |
-| --- | --- | --- | --- | --- | --- | --- |
-| temperature | `msTemperatureMeasurement` (1026) | 5 | 3600 | 10 | 0.1 °C | 10 / 3600 / 100 |
-| humidity | `msRelativeHumidity` (1029) | 5 | 3600 | 50 | 0.5 % | 10 / 3600 / 100 |
-| pressure | `msPressureMeasurement` (1027) | 5 | 3600 | 1 | 1 hPa | 10 / 3600 / 50 |
-| illuminance | `msIlluminanceMeasurement` (1024) | 5 | 3600 | 5 | 5 lx | 10 / 3600 / 5 |
-| windSpeed | `shellyWS90Wind` (0xfc01/0) | 5 | 3600 | 1 | — | 10 / 3600 / 1 |
-| windDirection | `shellyWS90Wind` (0xfc01/4) | 5 | 3600 | 1 | — | 10 / 3600 / 1 |
-| gustSpeed | `shellyWS90Wind` (0xfc01/7) | 5 | 3600 | 1 | — | 10 / 3600 / 1 |
-| uvIndex | `shellyWS90UV` (0xfc02/0) | 5 | 3600 | 1 | — | 10 / 3600 / 1 |
-| rainStatus | `shellyWS90Rain` (0xfc03/0) | 5 | 3600 | 1 | — | 10 / 3600 / 1 |
-| precipitation | `shellyWS90Rain` (0xfc03/1) | 5 | 3600 | 1 | — | 10 / 3600 / 1 |
-
-`genPowerCfg.batteryVoltage` (cluster 1, attribute 33) is deliberately left at
-its hourly default.
-
-The three `shellyWS90*` clusters are manufacturer-specific and require
-`"options": {"manufacturerCode": 5264}` on every reconfiguration request.
-
-## Why min is 5 and not 10
-
-The station transmits every 8.8 s. `min` is a hold-off: a change-triggered
-report is held until at least `min` seconds after the previous one. With
-`min: 10` every report was therefore held past a transmit cycle and landed on
-the next, so attributes reported at best every 17.6 s and drifted out of phase
-with each other — the published stream showed no 8.8 s periodicity at all,
-just a smear across 2–8 s.
-
-At `min: 5` each attribute reports on its own transmit. The MQTT stream then
-carries one publish per transmit cycle, measured at a mean gap of 8.69 s.
-Total publish volume *fell* (~631/h to ~445/h) because each publish now
-carries a whole cycle instead of a fragment of one.
-
-Any value below 8.8 s achieves this. 5 leaves margin without approaching the
-cycle time.
-
-## Why the deadbands differ from the converter defaults
-
-The defaults come from the shared `modernExtend` helpers in
-zigbee-herdsman-converters (`m.temperature()`, `m.pressure()`, `m.humidity()`,
-`m.illuminance()`), which the WS90 definition in `shelly.ts` composes without
-overrides. They are anti-chatter values sized for battery-powered indoor
-sensors, where report volume costs coin-cell life and mesh airtime. This
-station is mains-and-solar and exists to resolve small changes, so the
-trade-off runs the other way.
-
-Each value above is matched to the resolution the device actually reports:
-temperature and pressure arrive in 0.1-unit steps, humidity in whole percent.
-
-Pressure is worth calling out. The converter default of `change: 50` with
-`scale: 10` is a 5 kPa (50 hPa) deadband, which real barometric pressure never
-traverses — so pressure only ever updated on the hourly `max` heartbeat, and
-`pressure_trend` and `weather_condition` were derived from that. The same
-library's older `reporting.ts` helper configures the identical ZCL attribute
-with `change: 5`, a 5 hPa deadband, so the two code paths disagree by 10× and
-the modernExtend value is the outlier.
-
-## When this is lost
-
-Anything that makes Z2M re-run `configure` on the device overwrites all of it
-with the converter defaults:
-
- re-pairing or re-joining the device
- an explicit `zigbee2mqtt/bridge/request/device/configure`
- a converter update that changes the definition and triggers reconfiguration
-
-It also will not exist on a fresh PVC. Re-apply with the commands below.
-
-## Re-applying
-
-Published from the `discovery-guard` sidecar, which already holds the
-`zigbee2mqtt` broker credential (ACL `readwrite zigbee2mqtt/#`, which covers
-`bridge/request/#`) so no credential has to leave the cluster:
-
-Ten separate requests, one per attribute — written out literally rather than
-generated by a shell helper, because the nested quoting through `kubectl exec
-- sh -c` is where this goes wrong. Each line below is in the exact form that
-was executed and verified.
-
-```sh
-POD=$(kubectl get pod -n home-assistant -l app.kubernetes.io/name=zigbee2mqtt -o jsonpath='{.items[0].metadata.name}')
-```
-
-```sh
-kubectl exec -n home-assistant "$POD" -c discovery-guard -- sh -c 'mosquitto_pub -h "$MQTT_HOST" -p 1883 -u "$MQTT_USER" -P "$MQTT_PASSWORD" -t "$BASE_TOPIC/bridge/request/device/configure_reporting" -m "{\"id\":\"0xc02cedfffe958f40\",\"endpoint\":1,\"cluster\":\"msTemperatureMeasurement\",\"attribute\":\"measuredValue\",\"minimum_report_interval\":5,\"maximum_report_interval\":3600,\"reportable_change\":10}"'
-
-kubectl exec -n home-assistant "$POD" -c discovery-guard -- sh -c 'mosquitto_pub -h "$MQTT_HOST" -p 1883 -u "$MQTT_USER" -P "$MQTT_PASSWORD" -t "$BASE_TOPIC/bridge/request/device/configure_reporting" -m "{\"id\":\"0xc02cedfffe958f40\",\"endpoint\":1,\"cluster\":\"msRelativeHumidity\",\"attribute\":\"measuredValue\",\"minimum_report_interval\":5,\"maximum_report_interval\":3600,\"reportable_change\":50}"'
-
-kubectl exec -n home-assistant "$POD" -c discovery-guard -- sh -c 'mosquitto_pub -h "$MQTT_HOST" -p 1883 -u "$MQTT_USER" -P "$MQTT_PASSWORD" -t "$BASE_TOPIC/bridge/request/device/configure_reporting" -m "{\"id\":\"0xc02cedfffe958f40\",\"endpoint\":1,\"cluster\":\"msPressureMeasurement\",\"attribute\":\"measuredValue\",\"minimum_report_interval\":5,\"maximum_report_interval\":3600,\"reportable_change\":1}"'
-
-kubectl exec -n home-assistant "$POD" -c discovery-guard -- sh -c 'mosquitto_pub -h "$MQTT_HOST" -p 1883 -u "$MQTT_USER" -P "$MQTT_PASSWORD" -t "$BASE_TOPIC/bridge/request/device/configure_reporting" -m "{\"id\":\"0xc02cedfffe958f40\",\"endpoint\":1,\"cluster\":\"msIlluminanceMeasurement\",\"attribute\":\"measuredValue\",\"minimum_report_interval\":5,\"maximum_report_interval\":3600,\"reportable_change\":5}"'
-
-kubectl exec -n home-assistant "$POD" -c discovery-guard -- sh -c 'mosquitto_pub -h "$MQTT_HOST" -p 1883 -u "$MQTT_USER" -P "$MQTT_PASSWORD" -t "$BASE_TOPIC/bridge/request/device/configure_reporting" -m "{\"id\":\"0xc02cedfffe958f40\",\"endpoint\":1,\"cluster\":\"shellyWS90Wind\",\"attribute\":\"windSpeed\",\"minimum_report_interval\":5,\"maximum_report_interval\":3600,\"reportable_change\":1,\"options\":{\"manufacturerCode\":5264}}"'
-
-kubectl exec -n home-assistant "$POD" -c discovery-guard -- sh -c 'mosquitto_pub -h "$MQTT_HOST" -p 1883 -u "$MQTT_USER" -P "$MQTT_PASSWORD" -t "$BASE_TOPIC/bridge/request/device/configure_reporting" -m "{\"id\":\"0xc02cedfffe958f40\",\"endpoint\":1,\"cluster\":\"shellyWS90Wind\",\"attribute\":\"windDirection\",\"minimum_report_interval\":5,\"maximum_report_interval\":3600,\"reportable_change\":1,\"options\":{\"manufacturerCode\":5264}}"'
-
-kubectl exec -n home-assistant "$POD" -c discovery-guard -- sh -c 'mosquitto_pub -h "$MQTT_HOST" -p 1883 -u "$MQTT_USER" -P "$MQTT_PASSWORD" -t "$BASE_TOPIC/bridge/request/device/configure_reporting" -m "{\"id\":\"0xc02cedfffe958f40\",\"endpoint\":1,\"cluster\":\"shellyWS90Wind\",\"attribute\":\"gustSpeed\",\"minimum_report_interval\":5,\"maximum_report_interval\":3600,\"reportable_change\":1,\"options\":{\"manufacturerCode\":5264}}"'
-
-kubectl exec -n home-assistant "$POD" -c discovery-guard -- sh -c 'mosquitto_pub -h "$MQTT_HOST" -p 1883 -u "$MQTT_USER" -P "$MQTT_PASSWORD" -t "$BASE_TOPIC/bridge/request/device/configure_reporting" -m "{\"id\":\"0xc02cedfffe958f40\",\"endpoint\":1,\"cluster\":\"shellyWS90UV\",\"attribute\":\"uvIndex\",\"minimum_report_interval\":5,\"maximum_report_interval\":3600,\"reportable_change\":1,\"options\":{\"manufacturerCode\":5264}}"'
-
-kubectl exec -n home-assistant "$POD" -c discovery-guard -- sh -c 'mosquitto_pub -h "$MQTT_HOST" -p 1883 -u "$MQTT_USER" -P "$MQTT_PASSWORD" -t "$BASE_TOPIC/bridge/request/device/configure_reporting" -m "{\"id\":\"0xc02cedfffe958f40\",\"endpoint\":1,\"cluster\":\"shellyWS90Rain\",\"attribute\":\"rainStatus\",\"minimum_report_interval\":5,\"maximum_report_interval\":3600,\"reportable_change\":1,\"options\":{\"manufacturerCode\":5264}}"'
-
-kubectl exec -n home-assistant "$POD" -c discovery-guard -- sh -c 'mosquitto_pub -h "$MQTT_HOST" -p 1883 -u "$MQTT_USER" -P "$MQTT_PASSWORD" -t "$BASE_TOPIC/bridge/request/device/configure_reporting" -m "{\"id\":\"0xc02cedfffe958f40\",\"endpoint\":1,\"cluster\":\"shellyWS90Rain\",\"attribute\":\"precipitation\",\"minimum_report_interval\":5,\"maximum_report_interval\":3600,\"reportable_change\":1,\"options\":{\"manufacturerCode\":5264}}"'
-```
-
-These are idempotent — re-sending a request that matches the current
-configuration is a no-op on the device.
-
-The device must be awake and reachable; each request performs a real
-`ConfigureReporting` exchange and fails if it is not.
-
-## Verifying
-
-Read back what the device actually accepted — Z2M records the negotiated
-result, so this reflects the device, not the request:
-
-```sh
-kubectl exec -n home-assistant "$POD" -c zigbee2mqtt -- \
-  grep 0xc02cedfffe958f40 /app/data/database.db |
-  jq -r '.endpoints | to_entries[] | .value.configuredReportings[]?
-         | "cluster=\(.cluster) attr=\(.attrId) min=\(.minRepIntval) max=\(.maxRepIntval) change=\(.repChange)"'
-```
-
-All ten rows above should read `min=5 max=3600`. On the wire, the publish
-cadence on `zigbee2mqtt/Wetterstation` should sit at ~8.7 s and temperature
-should move in 0.1 °C steps; 1 °C steps mean the defaults are back.
-
-## Making it durable upstream
-
-The lasting fix is for the WS90 definition to pass explicit `reporting`
-overrides rather than inherit indoor-sensor defaults, which would fix every
-WS90 rather than this one. `m.temperature()` and the other helpers already
-accept a `reporting` argument. The pressure divergence between
-`modernExtend.ts` and `reporting.ts` is a separate upstream bug worth
-reporting on its own.
@@ -46,20 +46,6 @@ spec:
      # and slower than the lz4/zstd this build also accepts). Compresses the
      # full-page images that dominate WAL volume after each checkpoint.
      wal_compression: "zstd"
-      # A WAL segment is a fixed 16 MiB file, and archive_timeout forces a switch
-      # on a timer -- the segment is then written and archived at its full size
-      # no matter how little real WAL it holds. CNPG defaults to 5min, i.e. up
-      # to 288 segments/day = 4.6 GB/day that is largely padding.
-      #
-      # Verified on gf-pg first (the staged rollout): 211 -> 41 MB/hour, an 81%
-      # cut, with archived segments falling from ~12/hour to ~1.5/hour in step.
-      # Two independent signals moving together is what confirms the volume was
-      # padding rather than data.
-      #
-      # The cost is RPO: up to 30min of WAL is unrecoverable if this cluster and
-      # its volume are lost outright. This is a single-instance cluster, so the
-      # archive is the only off-node copy.
-      archive_timeout: "30min"
      # Write-reduction tuning. These databases are largely idle, and the
      # Postgres defaults favour throughput over storage that wears out.
      #
@@ -27,7 +27,7 @@ spec:
        # Pinned to 7.4-alpine to match the LXC 225 compose exactly.
        # Any redis major-version bump is a separate migration.
        - name: redis
-          image: redis:7.4.11-alpine # {"$imagepolicy": "flux-system:redis-immich"}
+          image: redis:7.4.10-alpine # {"$imagepolicy": "flux-system:redis-immich"}
          args:
            - redis-server
            - --save
@@ -8,7 +8,6 @@ resources:
  - prometheusrule-cilium.yaml
  - prometheusrule-hetzner-backup.yaml
  - prometheusrule-loki.yaml
-  - prometheusrule-mariadb-backup.yaml
  - prometheusrule-netbird-availability.yaml
  - prometheusrule-netbird-backup.yaml
  - prometheusrule-netbird-ingress.yaml
@@ -12,59 +12,18 @@ spec:
    - name: ceph.rules
      rules:
        - alert: CephHealthWarn
-          # Per-check rather than ceph_health_status, so one permanently-active
-          # check cannot mask every other Ceph warning.
-          #
-          # AUTH_INSECURE_* is excluded: it cannot clear until the nodes run a
-          # kernel supporting aes256k over the CSI mount path (#527).
-          #
-          # POOL_NO_REDUNDANCY is in scope. An unreplicated pool makes
-          # `ceph osd ok-to-stop` refuse for every OSD, so it reports blocked
-          # maintenance and not just data risk — do not exclude it for a pool
-          # that is unreplicated deliberately.
-          #
-          # 6h: Ceph warns for self-clearing conditions. Only a warning that
-          # does not clear is worth paging on.
-          #
-          # BLUESTORE_SLOW_OP_ALERT breaks that premise and is still NOT
-          # excluded (#557). Ceph holds the check for
-          # bluestore_slow_ops_warn_lifetime (24h) after the LAST slow op, so
-          # one multi-second stall outlives the 6h `for` and always pages,
-          # with latency back to normal long before anyone looks. Measured:
-          # both episodes in the 30d to 2026-08-23 ran ~26h and cleared
-          # untouched, the first ~24h after the last stall.
-          #
-          # Both began inside the Velero restore-test sweep — a ~5h sustained
-          # restore that outruns the SLC cache on these QLC NVMe. The sweep
-          # runs monthly, 08:00-14:00Z on the 1st.
-          #
-          # Expected for the hardware, but not excludable:
-          # this is also the earliest signal of a genuinely failing drive, and
-          # all three are at 80-87% endurance used (see cephcluster.yaml).
-          expr: ceph_health_detail{name!~"AUTH_INSECURE_.+"} == 1
+          # ceph_health_status: 0 = HEALTH_OK, 1 = HEALTH_WARN, 2 = HEALTH_ERR.
+          # 6h is deliberately long. Ceph warns for self-clearing conditions —
+          # a slow BlueStore op during the Sunday restore sweep, a mon briefly
+          # below its disk threshold — and paging on those trains people to
+          # ignore the alert. What matters is a warning that does not clear.
+          expr: ceph_health_status == 1
          for: 6h
          labels:
            severity: warning
          annotations:
-            summary: "Ceph health check {{ $labels.name }} active for 6h"
-            description: "Ceph health check {{ $labels.name }} has been active for six hours. Only AUTH_INSECURE_* is excluded (kernel-blocked, see #527). BLUESTORE_SLOW_OP_ALERT is sticky for 24h after the last stall and clears itself: check whether the stall fell inside the monthly Velero restore-test window (08:00-14:00Z on the 1st), which is expected on these QLC drives — one outside that window, or a repeat on the same OSD, means check that drive (#557). Detail: `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail`."
-
-        - alert: CephPoolLostRedundancy
-          # Names the pool, which POOL_NO_REDUNDANCY does not.
-          #
-          # There is no pool-size metric, but bytes_used/stored IS the
-          # replication factor. The >1e6 floor drops pools too small to measure
-          # meaningfully, empty ones included.
-          #
-          # No pool is exempt.
-          expr: |
-            ceph_pool_bytes_used / ceph_pool_stored < 1.5 and ceph_pool_stored > 1e6
-          for: 30m
-          labels:
-            severity: critical
-          annotations:
-            summary: "Ceph pool {{ $labels.pool_id }} is running unreplicated"
-            description: "Pool {{ $labels.pool_id }} has a replication factor below 1.5, meaning a single OSD loss destroys data. No pool is meant to be unreplicated — an unreplicated pool also blocks ALL OSD maintenance cluster-wide, since `ceph osd ok-to-stop` refuses while its PGs would go offline. Check with `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd pool ls detail`."
+            summary: "Ceph has been HEALTH_WARN for 6h"
+            description: "The Ceph cluster has held HEALTH_WARN for six hours, so it is not a self-clearing blip. Get the detail with `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail`."

        - alert: CephHealthError
          # HEALTH_ERR means data availability or redundancy is at risk.
@@ -1,51 +0,0 @@
-apiVersion: monitoring.coreos.com/v1
-kind: PrometheusRule
-metadata:
-  name: mariadb-backup
-  namespace: monitoring
-  labels:
-    # kube-prometheus-stack's Prometheus selects on this label.
-    release: kube-prometheus-stack
-spec:
-  groups:
-    - name: mariadb-backup.rules
-      rules:
-        - alert: WordpressMariadbBackupStale
-          # The nightly 02:45 logical dump (kubernetes/apps/wordpress/mariadb.yaml)
-          # is the only transaction-consistent copy of www.renner.uno's database
-          # — Velero snapshots the PVC, which for a running DB is not a restore
-          # point you would trust.
-          #
-          # KubeJobFailed already covers a failed Job, but only while the Job
-          # object survives. On #550 it cleared when the failed Job was
-          # garbage-collected, not when a backup succeeded — so a failure that
-          # is GC'd before anyone looks leaves no signal at all.
-          # lastSuccessfulTime lives on the CronJob and outlives Job GC.
-          #
-          # 26h = one missed cycle plus 2h slack, matching NetbirdVpsBackupStale.
-          # It counts any Job owned by the CronJob, so a manual
-          # `kubectl create job --from=cronjob/...` re-run clears it too.
-          expr: |
-            time() - kube_cronjob_status_last_successful_time{namespace="wordpress", cronjob="wordpress-mariadb-backup"} > 26 * 3600
-          for: 30m
-          labels:
-            severity: warning
-          annotations:
-            summary: "WordPress MariaDB backup has not succeeded in 26h+"
-            description: "The wordpress-mariadb-backup CronJob's last successful run is over 26 hours old. Check the newest Job's pod logs in the wordpress namespace: `kubectl -n wordpress get jobs`. The 2026-08-19 failure was `The specified bucket does not exist.` against renner-uno-k8s-mariadb while the endpoint itself probed healthy, so a green HetznerBackupEndpointDown does not clear this."
-
-        - alert: WordpressMariadbBackupNeverRan
-          # The staleness expression above goes blind when the metric is
-          # absent, which is what happens once the operator recreates the
-          # CronJob. Without this guard "never backed up" is quieter than
-          # "backed up late".
-          expr: |
-            kube_cronjob_created{namespace="wordpress", cronjob="wordpress-mariadb-backup"}
-            unless
-            kube_cronjob_status_last_successful_time{namespace="wordpress", cronjob="wordpress-mariadb-backup"}
-          for: 6h
-          labels:
-            severity: warning
-          annotations:
-            summary: "WordPress MariaDB backup has no recorded successful run"
-            description: "The wordpress-mariadb-backup CronJob exists but has no lastSuccessfulTime: either it has not completed once since the Backup CR was recreated, or every run since has failed. Check `kubectl -n wordpress get jobs` and that the hetzner-s3-creds ExternalSecret is synced."
@@ -58,18 +58,6 @@ spec:
      # and slower than the lz4/zstd this build also accepts). Compresses the
      # full-page images that dominate WAL volume after each checkpoint.
      wal_compression: "zstd"
-      # A WAL segment is a fixed 16 MiB file, and archive_timeout forces a switch
-      # on a timer -- the segment is then written and archived at its full size
-      # no matter how little real WAL it holds. CNPG defaults to 5min, i.e. up
-      # to 288 segments/day = 4.1 GB/day that is largely padding.
-      #
-      # Verified on gf-pg first (the staged rollout): 211 -> 41 MB/hour, an 81%
-      # cut, with archived segments falling from ~12/hour to ~1.5/hour in step.
-      #
-      # The cost is RPO: up to 30min of WAL is unrecoverable if both instances
-      # are lost. The documents themselves live on the media PVC, not in
-      # Postgres -- this database holds their metadata.
-      archive_timeout: "30min"
    # See ak-pg cluster.yaml for the rationale on synchronous replication.
    synchronous:
      method: any
@@ -25,7 +25,7 @@ spec:
          type: RuntimeDefault
      containers:
        - name: redis
-          image: redis:8.10.1-alpine # {"$imagepolicy": "flux-system:redis-paperless"}
+          image: redis:8.10.0-alpine # {"$imagepolicy": "flux-system:redis-paperless"}
          args:
            - redis-server
            - --save
@@ -36,20 +36,6 @@ spec:
      # and slower than the lz4/zstd this build also accepts). Compresses the
      # full-page images that dominate WAL volume after each checkpoint.
      wal_compression: "zstd"
-      # A WAL segment is a fixed 16 MiB file, and archive_timeout forces a switch
-      # on a timer -- the segment is then written and archived at its full size
-      # no matter how little real WAL it holds. CNPG defaults to 5min, i.e. up
-      # to 288 segments/day = 4.6 GB/day that is largely padding.
-      #
-      # Verified on gf-pg first (the staged rollout): 211 -> 41 MB/hour, an 81%
-      # cut, with archived segments falling from ~12/hour to ~1.5/hour in step.
-      # Two independent signals moving together is what confirms the volume was
-      # padding rather than data.
-      #
-      # The cost is RPO: up to 30min of WAL is unrecoverable if this cluster and
-      # its volume are lost outright. This is a single-instance cluster, so the
-      # archive is the only off-node copy.
-      archive_timeout: "30min"
      # Write-reduction tuning. These databases are largely idle, and the
      # Postgres defaults favour throughput over storage that wears out.
      #
@@ -34,7 +34,7 @@ spec:
          # Image automation, range 3.x (see image-automation/woodpecker.yaml).
          # MUST stay version-matched with woodpecker-server (gRPC parity) — both
          # track the same range/filter so they bump together on each release.
-          image: woodpeckerci/woodpecker-agent:v3.18.0 # {"$imagepolicy": "flux-system:woodpecker-agent"}
+          image: woodpeckerci/woodpecker-agent:v3.17.0 # {"$imagepolicy": "flux-system:woodpecker-agent"}
          env:
            - name: WOODPECKER_SERVER
              value: woodpecker.woodpecker.svc.cluster.local:9000 # in-cluster gRPC, replaces LXC :9000
@@ -50,39 +50,12 @@ spec:
            # clone + subsequent steps share the workspace PVC and the k8s backend
            # can schedule them on different nodes -> an RWO volume then fails to
            # mount on the second node. CephFS (RWX) lets the workspace follow
-            # the pipeline across nodes. Reclaim MUST be Delete: workspaces are
-            # throwaway, and a Retain class leaked one Released PV per finished
-            # pipeline (issue #262 follow-up).
-            #
-            # ceph-filesystem-ci, not ceph-filesystem-ephemeral: a dedicated
-            # CephFS whose pools are BOTH size 1 (infrastructure/
-            # rook-ceph-cluster/cephfs-ci.yaml). The shared filesystem
-            # replicates data0 2x and metadata 3x, which is the correct trade
-            # for real data and pure waste for a git clone that is deleted
-            # minutes later.
-            #
-            # Measured 2026-08-21, polling both pools every 15s (a 5-minute
-            # interval misses an entire pipeline -- PVC created, used and
-            # deleted inside ~2 minutes):
-            #   CI active  1.5min: data0 143.2 MB, metadata 17.2 MB
-            #   CI idle   16.0min: data0   0.0 MB, metadata 11.0 MB
-            # data0 write volume is 100% CI. Per pipeline this moves the drive
-            # cost from 71.6x2 + 8.2x3 = ~168 MB to ~80 MB.
-            #
-            # ceph-filesystem-ephemeral is NOT repointed, and the reason is a
-            # technical constraint rather than a safety one. Its other consumer
-            # is velero/restore-test.sh, whose probe PVCs are snapshot CLONES:
-            # `ceph fs subvolume snapshot clone` operates within one filesystem,
-            # so a snapshot taken in `cephfs` cannot be cloned into `cephfs-ci`.
-            #
-            # Those probes are just as throwaway as a CI workspace -- the
-            # authoritative copy is the backup on Hetzner, and a lost probe just
-            # means re-running the test -- so they are equally undeserving of
-            # replication. Landing them unreplicated would mean a second size-1
-            # data pool inside `cephfs`, and restore-test runs MONTHLY against
-            # CI's ~96 pipelines/day, so it is not worth the complexity.
+            # the pipeline across nodes. The -ephemeral variant (reclaim Delete,
+            # rook-ceph-cluster/storageclass.yaml) is REQUIRED here: workspaces
+            # are throwaway, and the Retain class leaked one Released PV per
+            # finished pipeline (issue #262 follow-up).
            - name: WOODPECKER_BACKEND_K8S_STORAGE_CLASS
-              value: ceph-filesystem-ci
+              value: ceph-filesystem-ephemeral
            - name: WOODPECKER_BACKEND_K8S_STORAGE_RWX
              value: "true"
            # MUST be a JSON object literal — the backend json.Unmarshal's this
@@ -80,20 +80,6 @@ spec:
      # and slower than the lz4/zstd this build also accepts). Compresses the
      # full-page images that dominate WAL volume after each checkpoint.
      wal_compression: "zstd"
-      # A WAL segment is a fixed 16 MiB file, and archive_timeout forces a switch
-      # on a timer -- the segment is then written and archived at its full size
-      # no matter how little real WAL it holds. CNPG defaults to 5min, i.e. up
-      # to 288 segments/day = 4.6 GB/day that is largely padding.
-      #
-      # Verified on gf-pg first (the staged rollout): 211 -> 41 MB/hour, an 81%
-      # cut, with archived segments falling from ~12/hour to ~1.5/hour in step.
-      # Two independent signals moving together is what confirms the volume was
-      # padding rather than data.
-      #
-      # The cost is RPO: up to 30min of WAL is unrecoverable if this cluster and
-      # its volume are lost outright. This is a single-instance cluster, so the
-      # archive is the only off-node copy.
-      archive_timeout: "30min"
  resources:
    requests:
      cpu: 100m
@@ -40,7 +40,7 @@ spec:
          # minor + patch auto-update within 3.x; only the 4.0 major is held for
          # a reviewed bump. Safe now the DB is postgres (the old sqlite-surprise
          # -bump hazard is gone). Stays in lockstep with the agent.
-          image: woodpeckerci/woodpecker-server:v3.18.0 # {"$imagepolicy": "flux-system:woodpecker-server"}
+          image: woodpeckerci/woodpecker-server:v3.17.0 # {"$imagepolicy": "flux-system:woodpecker-server"}
          env:
            - name: WOODPECKER_HOST
              value: https://ci.renner.uno
@@ -268,7 +268,7 @@ EOF

 # 120x15s = 30 min, not the old 60x5s = 5 min. node-agent has no
 # loadConcurrency override, so Velero's default of one data path per node
-# applies: while the restore-test sweep holds the slot, even this 1 GiB
+# applies: while the Sunday restore-test sweep holds the slot, even this 1 GiB
 # DataUpload sits in phase Prepared for well over 5 minutes. The old 300 s wait
 # then let the teardown below delete the source namespace and the Backup
 # mid-upload ("Cancel du under phase Prepared because it is being deleted"),
@@ -87,16 +87,6 @@ spec:

    lapi:
      replicas: 1
-      # The chart defaults request 500m. Measured steady-state draw is 2m, and
-      # the request is what the scheduler reserves, so the default parks half a
-      # core of unusable capacity per LAPI. That is affordable on three nodes
-      # and is not on two: with one node out for maintenance the remaining pair
-      # sit at 100% of requested CPU while actually using ~22%, and nothing new
-      # can be placed. The limit stays at the chart default so a log flood can
-      # still burst; only the reservation shrinks.
-      resources:
-        requests:
-          cpu: 50m
      podAnnotations:
        # Reloader restarts the LAPI when crowdsec-bouncer-key changes (it's an
        # env-var secret, so a restart is required to pick up a rotated key). See
@@ -132,12 +122,6 @@ spec:
              key: key

    agent:
-      # Same reasoning as lapi.resources above, multiplied by three: this is a
-      # DaemonSet, so the chart's 500m default reserves half a core on every
-      # node to run a process that measures 1-2m.
-      resources:
-        requests:
-          cpu: 50m
      # The agent parses the ingress controller's access logs (read from
      # the pod's stdout) to detect attacks. Repointed from the retired
      # ingress-nginx (decommissioned 2026-05-17) to Traefik. The
@@ -51,15 +51,7 @@ spec:
  startingDeadlineSeconds: 3600
  jobTemplate:
    spec:
-      # 4 (not the sibling rotators' 1-2) because this is the only DAILY
-      # rotator — every other CronJob here is quarterly, so it takes ~90x as
-      # many runs and is correspondingly more exposed to per-run transients.
-      # backoffLimit: 1 gave two attempts ~11 s apart, and on 2026-08-19 a
-      # sub-minute resolver transient took both out (Job
-      # claude-oauth-secret-sync-29785100, BackoffLimitExceeded), costing a
-      # day's sync. 4 spreads the attempts over ~150 s (10/20/40/80 s backoff),
-      # still well inside activeDeadlineSeconds.
-      backoffLimit: 4
+      backoffLimit: 1
      activeDeadlineSeconds: 300
      ttlSecondsAfterFinished: 259200 # 3 d
      template:
@@ -17,17 +17,6 @@ spec:
    modules:
      - name: pg_autoscaler
        enabled: true
-  # Rotates daemon keys to aes256k for CVE-2025-30156 (#527). Opt-in; disabled
-  # by default. Increment keyGeneration to rotate again.
-  #
-  # Daemon keys only — aes256k over the CSI mount path needs kernel 7.0+ and
-  # these nodes run 6.18, so CSI stays on aes and the client-key checks stay
-  # raised. keyType is unset so Rook picks it.
-  security:
-    cephx:
-      daemon:
-        keyRotationPolicy: KeyGeneration
-        keyGeneration: 2
  # Bump CephFS subvolume cloner pool from 4 → 16. Velero CSI Snapshot
  # Data Movement provisions a backupPVC per source PVC by cloning the
  # VolumeSnapshot, and a 4-thread default stalls migrate-immich's
@@ -164,15 +153,7 @@ spec:
    useAllNodes: true
    useAllDevices: false
    devices:
-      # On k8s-2/k8s-3 this is the NVMe drive PCI-passed-through from saturn
-      # and neptune, and it is the only block device the guest can see. k8s-1
-      # is bare metal on jupiter and sees two: the OSD drive in MS-01 slot 1
-      # (PCI 01:00.0) and the Talos system disk in slot 2 (58:00.0). Slot 1
-      # enumerates first, so nvme0n1 is still the OSD drive there — but that
-      # is enumeration order, not a guarantee. The system disk is protected
-      # from the other direction instead: talos/patches/k8s-1.yaml pins the
-      # installer to it by serial, and Rook skips a disk carrying partitions.
-      - name: nvme0n1
+      - name: nvme0n1 # NVMe drive PCI-passed-through from each PVE node
    config:
      osdsPerDevice: "1"
  # Storage layer must never be evicted by kubelet under node pressure
@@ -1,64 +0,0 @@
---
-# A SEPARATE CephFS whose only purpose is Woodpecker CI workspaces. Each
-# pipeline gets a PVC that is created, filled and deleted within ~2 minutes;
-# keeping that churn off `cephfs` spares its size-3 metadataPool.
-#
-# size 2, not 1: an unreplicated pool makes `ceph osd ok-to-stop` refuse for
-# every OSD, blocking all OSD maintenance cluster-wide. size 1 is viable only
-# once these pools have OSDs of their own.
-apiVersion: ceph.rook.io/v1
-kind: CephFilesystem
-metadata:
-  name: cephfs-ci
-  namespace: rook-ceph
-spec:
-  metadataPool:
-    replicated:
-      size: 2
-  dataPools:
-    - name: data0
-      replicated:
-        size: 2
-  metadataServer:
-    activeCount: 1
-    # No standby-replay, unlike `cephfs`. A standby exists there so a failed
-    # MDS does not hang every CephFS mount cluster-wide; here the only casualty
-    # is a running pipeline, and a second daemon to protect throwaway work is
-    # not worth its memory.
-    activeStandby: false
-    resources:
-      # Deliberately NOT the 4Gi that `cephfs` needs. That figure is driven by
-      # immich's millions of cached inodes; a CI workspace is a few thousand
-      # short-lived files that vanish with the pipeline.
-      requests:
-        cpu: 100m
-        memory: 512Mi
-      limits:
-        memory: 1Gi
---
-# CI-only class. `ceph-filesystem-ephemeral` is deliberately left pointing at
-# the main filesystem: velero/restore-test.sh uses it to restore REAL data for
-# verification, which must not land on unreplicated pools.
-#
-# A new class rather than repointing the old one — StorageClass.parameters are
-# immutable, so `fsName` cannot be edited in place.
-apiVersion: storage.k8s.io/v1
-kind: StorageClass
-metadata:
-  name: ceph-filesystem-ci
-provisioner: rook-ceph.cephfs.csi.ceph.com
-parameters:
-  clusterID: rook-ceph
-  fsName: cephfs-ci
-  pool: cephfs-ci-data0
-  csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-provisioner
-  csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
-  csi.storage.k8s.io/controller-expand-secret-name: rook-csi-cephfs-provisioner
-  csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
-  csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-node
-  csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
-# Delete, for the same reason ceph-filesystem-ephemeral uses it: with Retain,
-# every finished pipeline leaks a Released PV pinning a subvolume (191 of them
-# by 2026-07-16, issue #262 follow-up).
-reclaimPolicy: Delete
-allowVolumeExpansion: true
@@ -4,6 +4,5 @@ resources:
  - cephcluster.yaml
  - toolbox.yaml
  - storageclass.yaml
-  - cephfs-ci.yaml
  - volumesnapshotclass.yaml
  - objectstore.yaml
@@ -87,27 +87,15 @@ metadata:
  name: cephfs
  namespace: rook-ceph
 spec:
-  # TEMPORARY — REVERT TO size 3 / no min_size OVERRIDE ONCE k8s-1 IS BACK.
-  #
-  # The standing trade for this pool is size 3: it holds ~253 MiB against the
-  # data pool's 280 GiB, so it contributes nothing measurable to drive wear —
-  # the entire reason data0 dropped to 2 — while losing it would destroy the
-  # whole filesystem, not just the objects on the lost OSD. Paying 3x on a
-  # quarter of a gigabyte to keep that intact is the right trade in steady
-  # state.
-  #
-  # It does not hold while one of three hosts is out for the jupiter bare-metal
-  # conversion. size 3 on two remaining hosts leaves every PG undersized at two
-  # copies with min_size 2, so a single further OSD loss drops the pool below
-  # min_size and blocks IO on every CephFS mount cluster-wide. At 2/1 the same
-  # two copies are active+clean instead, and a further loss degrades rather
-  # than blocks — availability bought at the cost of accepting writes onto a
-  # single copy while degraded.
+  # metadataPool STAYS at size 3 while the data pool drops to 2. It holds
+  # 467 MiB against the data pool's 275 GiB, so it contributes nothing
+  # measurable to drive wear — the entire reason for the size reduction — while
+  # losing it would destroy the whole filesystem, not just the objects on the
+  # lost OSD. Paying 3x on half a gigabyte to keep that intact is the right
+  # trade; the wear saving comes from data0.
  metadataPool:
    replicated:
-      size: 2
-    parameters:
-      min_size: "1"
+      size: 3
  dataPools:
    - name: data0
      # size 2 / min_size 1 for the same reasons as replicapool above — see
@@ -1,30 +1,19 @@
 ---
-# Monthly Velero restore-test CronJob — 08:00 UTC on the 1st.
+# Weekly Velero restore-test CronJob.
 #
-# The sweep restores every discovered schedule, immich's ~180 GiB library
-# included. At `size: 2` that is ~360 GiB raw, ~120 GiB per OSD, against a
-# ~65 GB/day weekday baseline on three QLC NVMe at 81-87% endurance used —
-# ~47% of each drive's weekly write budget. Monthly keeps every restore path
-# exercised and cuts that share ~4x.
-#
-# 08:00 — *after* that day's own daily backups have finished (paperless
-# 03:15 through apps 07:15, each normally ~1 min). The previous 02:00 slot
-# was chosen as "well before the next day's backups", which missed that the
-# daily schedules fire on the same day: a sweep starting at 02:00 and
-# running ~5 h sat on top of the entire 03:15-07:15 backup window and
+# Sunday 08:00 UTC — *after* Sunday's own daily backups have finished
+# (paperless 03:15 through apps 07:15, each normally ~1 min). The previous
+# 02:00 slot was chosen as "well before Monday's backups", which missed
+# that the daily schedules also fire on Sunday: a sweep starting at 02:00
+# and running ~5 h sat on top of the entire 03:15-07:15 backup window and
 # starved it of node-agent DataUpload concurrency. Observed effect on
 # 2026-07-19 and 2026-07-26 — audiobookshelf-daily took 2 h 18 m and
 # 2 h 33 m respectively against a ~1 min baseline, with the books
-# DataUpload not starting until 06:18. Both are Sundays, the only day the
-# sweep ran then; no other day shows it.
+# DataUpload not starting until 06:18. Both are Sundays; no other day
+# shows it.
 #
-# 08:00 + the 6 h activeDeadlineSeconds cap still lands well clear of the
-# next day's 03:15.
-#
-# Day-of-month rather than a fixed weekday: Kubernetes parses cron with
-# robfig/cron's standard parser, which has no `#` nth-weekday operator, and
-# restricting both day fields (`0 8 1-7 * 0`) ORs them — every Sunday plus
-# the 1st through 7th.
+# 08:00 + the 6 h activeDeadlineSeconds cap still lands well clear of
+# Monday's 03:15.
 #
 # Concurrency policy Forbid: the immich restore (~180 GiB, millions of
 # small files) alone can run multiple hours, so the full sweep is sized
@@ -40,7 +29,7 @@ metadata:
  name: restore-test
  namespace: velero
 spec:
-  schedule: "0 8 1 * *"
+  schedule: "0 8 * * 0"
  timeZone: Etc/UTC
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 2
@@ -50,7 +39,7 @@ spec:
    spec:
      backoffLimit: 0
      activeDeadlineSeconds: 21600  # 6 h hard cap (immich alone can take ~3 h)
-      ttlSecondsAfterFinished: 3456000  # 40 d — enough to inspect after the next monthly run
+      ttlSecondsAfterFinished: 1209600  # 14 d — enough to inspect after the next weekly run
      template:
        spec:
          serviceAccountName: restore-test
@@ -1,5 +1,5 @@
 ---
-# RBAC for the monthly restore-test CronJob.
+# RBAC for the weekly restore-test CronJob.
 #
 # Permissions required by restore-test.sh:
 #   - velero.io: read schedules/backups/BSL, create+delete restores
@@ -1,5 +1,5 @@
 #!/usr/bin/env bash
-# Monthly verification that Velero PVC backups are actually restorable.
+# Weekly verification that Velero PVC backups are actually restorable.
 #
 # What this exercises (per service):
 #   1. Velero Restore reaches phase=Completed against yesterday's
@@ -53,7 +53,7 @@ check() {
 # and ceph-filesystem StorageClasses ship with reclaimPolicy: Retain so that
 # production PVC deletions can be undone manually. The restore-test has the
 # opposite need: every PVC it creates is throwaway and MUST not survive the
-# iteration. Without this, each run leaks O(100 GiB) of CephFS
+# iteration. Without this, each weekly run leaks O(100 GiB) of CephFS
 # subvolumes and the cluster eventually goes OSD_FULL (incident 2026-05-17).
 #
 # Why this still exists after ceph-*-ephemeral landed. Three categories of
@@ -1,101 +1,62 @@
-# k8s-1 runs Talos directly on jupiter's MS-01 hardware; k8s-2 and k8s-3 are
-# still Proxmox VMs. Every machine-specific selector below differs from theirs
-# for that reason: there are no virtio NICs, no PCI-passthrough layer, and both
-# NVMe drives are visible to Talos at once.
-#
-# Networking uses the v1.12 multi-document config rather than the deprecated
-# machine.network.interfaces. That is deliberate: the docs do not define what
-# happens when the same link is configured by both schemas, so this file keeps
-# every link in one schema instead of straddling them.
 machine:
  network:
-    # Shared control-plane VIP 10.2.40.60 is configured as a Layer2VIPConfig
-    # below rather than machine.network.interfaces[].vip. Exposed via UniFi DNS
-    # as k8s.renner.uno.
    hostname: k8s-1
+    # Shared control-plane VIP — etcd-elected leader hosts 10.2.40.60 as a
+    # secondary IP. Exposed via UniFi DNS as k8s.renner.uno.
+    #
+    # Two virtio NICs (mgmt + v30iot). `driver: virtio_net` would match
+    # both, so each is pinned by its PCI bus path:
+    #   - 0000:06:12.0 — original mgmt NIC (the only virtio before)
+    #   - 0000:06:13.0 — extra_network_bridges = ["v30iot"] in tofu
+    # Bus paths are consistent across k8s-{1,2,3}; verified live 2026-05-13.
+    # The X710 selector stays driver-based (i40e is unambiguous) and
+    # tolerates kernel renaming.
+    interfaces:
+      - deviceSelector:
+          busPath: "0000:06:12.0"
+        dhcp: true
+        vip:
+          ip: 10.2.40.60
+      # v30iot: LINK ONLY — no host addressing. The NIC is the macvlan
+      # parent for the v30iot pod attachments (matter/otbr/shelly-manager),
+      # whose children do their own DHCP on the L2; host addresses were
+      # never needed for that. Host DHCP (formerly dhcp + routeMetric 4096)
+      # installed a connected 10.2.30.0/24 route, so node replies to
+      # IoT-subnet clients bypassed the UDM and broke every UDM-DNATed
+      # flow (hairpin NAT: un-translated SYN-ACK -> client kernel RST,
+      # verified live 2026-07-23). The purpose the old comment claimed —
+      # Cilium L2 announce for an IoT LB pool — does not exist (no
+      # L2AnnouncementPolicy, no such pool). v6 RA addressing on this NIC
+      # is disabled via machine.sysctls for the same reason.
+      - deviceSelector:
+          busPath: "0000:06:13.0"
+        dhcp: false
+      # Dedicated Ceph cluster network on the X710 SFP+ port (PCI passthrough,
+      # mapping x710-port1 in opentofu/vm_150_k8s_1.tf). Separate /29 on VLAN 49,
+      # MTU 9000, no default route — used by Rook for OSD↔OSD replication via
+      # CephCluster.spec.network.addressRanges.cluster.
+      - deviceSelector:
+          driver: i40e
+        addresses:
+          - 10.99.0.1/29
+        mtu: 9000
  sysctls:
-    # ens19 (VLAN 30, v30iot) is LINK ONLY — no host addressing. It exists as
-    # the macvlan parent for the v30iot pod attachments
-    # (matter/otbr/shelly-manager/music-assistant), whose children do their own
-    # DHCP on that L2. Host addresses were never needed and are actively
-    # harmful: a connected 10.2.30.0/24 route makes node replies to IoT-subnet
-    # clients bypass the UDM, which breaks every UDM-DNATed flow (hairpin NAT:
-    # un-translated SYN-ACK -> client kernel RST, verified live 2026-07-23).
-    # This disables v6 entirely on that link, including link-local.
+    # v30iot NIC carries no host addressing (see interfaces above); this
+    # disables v6 entirely (incl. link-local) so the host is unreachable on that L2.
+    #
+    # Keyed on the STABLE predictable name `ens19` (busPath 0000:06:13.0 =
+    # v30iot). Predictable naming is re-enabled via the Image Factory
+    # schematic (net.ifnames=1; see .woodpecker/deploy-infrastructure.yml),
+    # so `ens19` is topology-derived and does NOT shuffle across reboots —
+    # unlike the former `eth0/1/2` (driver-probe order) which renamed on a
+    # k8s-3 reboot and silently mis-targeted this sysctl onto mgmt. The four
+    # v30iot macvlan NADs key on the same `ens19`.
    net.ipv6.conf.ens19.disable_ipv6: "1"
  install:
-    # Both NVMe drives are now ordinary block devices to Talos: the Kingston
-    # OM8PGP41024Q-A0 system disk and the Samsung PM983 that Rook claims as the
-    # Ceph OSD. Selecting the installer target by serial means it cannot land
-    # on the OSD drive regardless of which one the kernel enumerates first.
-    diskSelector:
-      serial: 50026B7382B48B86
+    disk: /dev/vda
  certSANs:
    - k8s-1.srv.renner.uno
  nodeLabels:
    topology.kubernetes.io/zone: jupiter
 cluster:
  allowSchedulingOnControlPlanes: true
---
-# Physical NICs are aliased by permanent MAC. Both X710 ports bind i40e, so the
-# driver selector k8s-2/k8s-3 use for their single passed-through port would
-# match two links here. MAC matching is unambiguous and survives udev renaming.
-apiVersion: v1alpha1
-kind: LinkAliasConfig
-name: lan0
-selector:
-  match: glob("58:47:ca:74:70:27", mac(link.permanent_addr))
---
-apiVersion: v1alpha1
-kind: LinkAliasConfig
-name: ceph0
-selector:
-  match: glob("58:47:ca:74:70:28", mac(link.permanent_addr))
---
-# lan0 is the switch trunk (X710 port 0). It carries no addressing of its own;
-# VLAN 40 and VLAN 30 ride on top. Proxmox's vlan-aware vmbr0 did this tagging
-# before, which is why the VM saw two already-untagged virtio NICs instead.
-apiVersion: v1alpha1
-kind: LinkConfig
-name: lan0
-up: true
---
-apiVersion: v1alpha1
-kind: VLANConfig
-name: mgmt0
-vlanID: 40
-parent: lan0
-up: true
---
-apiVersion: v1alpha1
-kind: DHCPv4Config
-name: mgmt0
---
-apiVersion: v1alpha1
-kind: Layer2VIPConfig
-name: 10.2.40.60
-link: mgmt0
---
-# Named ens19 on purpose. The four v30iot macvlan NADs hard-code
-# `master: ens19`, and k8s-2/k8s-3 still present the VLAN as a virtio NIC of
-# that name. VLANConfig.name is free-form, so keeping it lets one set of NADs
-# serve both VM and bare-metal nodes for as long as the cluster is mixed.
-# Nothing else on this machine is called ens19: the physical NICs are
-# enp2s0f0np0, enp2s0f1np1, enp87s0 and enp89s0.
-apiVersion: v1alpha1
-kind: VLANConfig
-name: ens19
-vlanID: 30
-parent: lan0
-up: true
---
-# Dedicated Ceph cluster network, VLAN 49 / 10.99.0.0/29, untagged on the wire
-# on X710 port 1. Used by Rook for OSD<->OSD replication via
-# CephCluster.spec.network.addressRanges.cluster. No default route.
-apiVersion: v1alpha1
-kind: LinkConfig
-name: ceph0
-up: true
-mtu: 9000
-addresses:
-  - address: 10.99.0.1/29
@@ -0,0 +1,51 @@
+module "k8s_1" {
+  source = "./modules/proxmox_vm"
+
+  name        = "k8s-1.srv.renner.uno"
+  target_node = "jupiter"
+  vmid        = 150
+  memory      = 49152
+  cores       = 8
+
+  template_name = null
+  cloud_init    = false
+  full_clone    = false
+  bios          = "ovmf"
+  machine       = "q35"
+  boot          = "order=virtio0"
+  agent_enabled = true
+
+  # Second NIC on v30iot so Cilium L2 announce can win ARP on the IoT
+  # VLAN from this node (previously only k8s-edge had v30iot, so any
+  # IoT-pool LB lease landing here was unreachable from devices).
+  extra_network_bridges = ["v30iot"]
+
+  disks = [
+    {
+      storage = "local-zfs"
+      size    = "64G"
+      format  = "raw"
+      slot    = "virtio0"
+      file_id = local.talos_image
+    }
+  ]
+
+  hostpci = [
+    {
+      mapping = "osd-ssd"
+    },
+    {
+      mapping = "x710-port1"
+    }
+  ]
+
+  tags            = "kubernetes;talos;controlplane"
+  ssh_public_keys = local.common_ssh_keys
+
+  vm_state = "running"
+
+  # Rebuild pipeline issues `-replace`; skip graceful shutdown so a
+  # non-responsive Talos guest can't hang the destroy for 30 minutes.
+  stop_on_destroy = true
+}
+