fix(ceph): stop deferring small writes on SSD (halves write amplification)

bluestore_prefer_deferred_size_ssd was set to 65536 and
bluestore_deferred_batch_ops_ssd to 64 as a QLC throughput tune. Ceph's SSD
defaults are 0 and 16. Reverted to the defaults, set explicitly so the stored
`ceph config` value is overwritten rather than merely un-declared.

prefer_deferred_size defers every write SMALLER than the threshold: it lands in
RocksDB's WAL first and is applied to the object store afterwards. That is a
double write, and at 64 KiB it caught essentially every write this cluster
makes -- SQLite pages are 4 KiB, Postgres pages 8 KiB. Ceph disables deferral on
SSD by default because solid state does not need the random-write batching that
helps spinning media.

Why the original tune was the wrong trade:

  - The benchmark justifying it cannot have exercised it. The 2026-05-05
    baseline used 4 MiB writes, 64x above the 64 KiB cutoff, so none of those
    writes were ever deferred. The setting was validated by a workload it does
    not affect.

  - It doubles writes onto QLC, the lowest-endurance NAND type. Measured
    2026-08-19 from the drives' own SMART counters: all three OSD devices at
    80/82/87% endurance used, advancing ~0.13 points/day, i.e. rated life
    exhausted between Nov 2026 and Jan 2027. Because size=3 on 3 OSDs, every
    drive absorbs 100% of cluster writes, so they wear in lockstep and expire
    together (see #434).

Measured amplification before this change: 640 GB/day written to the three NVMe
devices against 117 GB/day at the RBD client layer -- 5.5x, of which 3x is
replication and ~1.8x BlueStore. Removing the double write should take the
BlueStore factor toward ~1.1x.

The cluster is write-light, so throughput was never the binding constraint;
endurance is. Both parameters are runtime-updatable, so no OSD restart.

Also corrects the drive model in the comment: the OSD devices are
CT1000P3PSSD8 (Crucial P3 Plus), not the P2 named previously.

bdev_enable_discard / bdev_async_discard are unrelated and kept as-is.

Refs #434

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

.woodpecker/alert-monitor.yml

@@ -135,8 +135,38 @@ steps:
         TITLES=$$(jq -c '[.[].title]' .ci-alert-monitor/open-issues.json)
         NOISE_BODY=$$(jq -r 'map(select(.title == "Alert monitor: noise log")) | .[0].body // ""' \
           .ci-alert-monitor/open-issues.json)
-        FPS=$$(printf '%s' "$$NOISE_BODY" | grep -oE '\b[0-9a-f]{16}\b' \
+        # Suppression entries are exactly the structured lines the write-back
+        # step emits: "- <fp> <ALERTNAME> — <YYYY-MM-DD>: <reason>". Parse THAT
+        # shape rather than grepping the whole body for 16-hex tokens. A reason
+        # routinely cites another alert's fingerprint in prose ("see the issue
+        # filed for fingerprint <fp>"), and the body-wide grep turned every such
+        # citation into a silent, permanent suppression of an alert that was
+        # never classified as noise — verified against the live noise log, which
+        # yielded 22 fingerprints for 21 real entries, the extra one being
+        # 2ddc41edb689fc33, an alert that HAD been filed as a genuine issue.
+        #
+        # Entries also expire after NOISE_TTL_DAYS. A noise verdict says "this
+        # occurrence was benign", not "this alert can never be interesting
+        # again" — but the fingerprint is sha256(alertname|scope) and therefore
+        # stable across recurrences, so without an expiry a single blip mutes
+        # that alertname+scope pair forever. Re-triaging a recurrence weeks
+        # later is cheap; missing a real outage because a July blip muted
+        # openbao's KubePodNotReady is not.
+        #
+        # Lines that fail to parse are simply dropped, i.e. they stop
+        # suppressing — this fails toward MORE alerting, never less.
+        #
+        # date is BusyBox here, not GNU: `date -d '30 days ago'` errors out, so
+        # the cutoff goes through epoch arithmetic and `date -d @<epoch>`, which
+        # this image does support.
+        NOISE_TTL_DAYS=$${NOISE_TTL_DAYS:-30}
+        CUTOFF=$$(date -u -d @$$(( $$(date -u +%s) - NOISE_TTL_DAYS * 86400 )) +%F)
+        FPS=$$(printf '%s\n' "$$NOISE_BODY" \
+          | sed -nE 's/^- ([0-9a-f]{16}) [^ ]+ — ([0-9]{4}-[0-9]{2}-[0-9]{2}):.*/\2 \1/p' \
+          | awk -v c="$$CUTOFF" 'BEGIN{gsub(/-/,"",c)} { d=$$1; gsub(/-/,"",d); if (d+0 >= c+0) print $$2 }' \
           | jq -R -s -c 'split("\n") | map(select(length > 0))')
+        echo "Noise suppressions in effect: $$(printf '%s' "$${FPS:-[]}" | jq length)" \
+             "(TTL $${NOISE_TTL_DAYS}d, entries dated >= $$CUTOFF)"
         # Synthesize pseudo-alerts for BLIPS: lookback series whose label set
         # is not currently firing at all (checked against the PRE-filter raw
         # snapshot, so live-but-null-routed alerts don't reappear as blips).
@@ -260,7 +290,7 @@ steps:
               '{body: ($$old + "\n" + $$add)}' > /tmp/gitea-patch.json
             gitea_write PATCH "$$API/issues/$$NOISE_NUM" /tmp/gitea-patch.json
           else
-            printf '%s\n\n' "Fingerprints the alert-monitor triage agent classified as noise. The dedup step skips any alert whose fingerprint appears below — delete a line to make that alert eligible for triage again. Keep this issue open; it is the suppression state." > /tmp/noise-header.txt
+            printf '%s\n\n' "Fingerprints the alert-monitor triage agent classified as noise. The dedup step skips any alert whose fingerprint appears below — delete a line to make that alert eligible for triage again. Keep this issue open; it is the suppression state. Entries EXPIRE after NOISE_TTL_DAYS (default 30) counted from the date in the line, so a one-off blip stops suppressing its alertname+scope pair once the window passes; re-add happens automatically if it is triaged as noise again. Only lines in the exact '- <fingerprint> <ALERTNAME> — <YYYY-MM-DD>: <reason>' form suppress anything — a fingerprint mentioned in prose does not." > /tmp/noise-header.txt
             jq -n --rawfile hdr /tmp/noise-header.txt --rawfile add /tmp/noise-lines.txt \
               '{title: "Alert monitor: noise log", body: ($$hdr + $$add)}' > /tmp/gitea-create.json
             gitea_write POST "$$API/issues" /tmp/gitea-create.json

kubernetes/apps/home-assistant/kustomization.yaml

@@ -79,5 +79,13 @@ configMapGenerator:
     namespace: home-assistant
     files:
       - watch.sh=zigbee2mqtt-interview-watch.sh
+  # Z2M discovery guard sidecar script (gitea issue #381). Hash suffix disabled
+  # (name stays stable), so editing zigbee2mqtt-discovery-guard.sh does not
+  # auto-roll the pod — restart manually with
+  # `kubectl rollout restart deploy/zigbee2mqtt -n home-assistant`.
+  - name: zigbee2mqtt-discovery-guard-script
+    namespace: home-assistant
+    files:
+      - guard.sh=zigbee2mqtt-discovery-guard.sh
 generatorOptions:
   disableNameSuffixHash: true

kubernetes/apps/home-assistant/mosquitto-bootstrap.yaml

@@ -373,8 +373,9 @@ spec:
 # Broker user `zigbee2mqtt` — the Zigbee2MQTT bridge (ZHA replacement). Scoped
 # to its own `zigbee2mqtt/#` subtree plus `write homeassistant/#` (so it can
 # publish HA MQTT-discovery configs, exactly like the `shellies` user) and
-# `read homeassistant/status` (Z2M re-publishes discovery when HA restarts —
-# ACL in mosquitto-deployment.yaml). Own credential so its lifecycle is
+# `read homeassistant/status` (HA's birth/will topic; on `online` Z2M
+# republishes cached device states, NOT discovery configs — ACL and the full
+# reasoning in mosquitto-deployment.yaml). Own credential so its lifecycle is
 # independent of the others. The Z2M init container reads this password and
 # renders it into /app/data/secret.yaml.
 apiVersion: batch/v1

kubernetes/apps/home-assistant/mosquitto-config.yaml

@@ -5,13 +5,31 @@ metadata:
   namespace: home-assistant
 data:
   mosquitto.conf: |
-    # No on-disk persistence. Retained messages + sessions are re-populated by
-    # clients on reconnect (evcc republishes state, Shellies re-send their
-    # announce/online, HA re-publishes discovery), so the broker keeps no
-    # durable state. This drops the Ceph PVC dependency (the broker starts even
-    # if Ceph is degraded) and lets it reschedule anywhere in seconds — no RWO
-    # volume to fence/detach/reattach. Trade-off: a broker restart briefly
-    # empties retained state until clients reconnect (self-healing).
+    # No on-disk persistence, and no PVC of any kind for this broker. The
+    # broker keeps no durable state, so it starts even if Ceph is degraded and
+    # reschedules anywhere in seconds — no RWO volume to fence/detach/reattach.
+    #
+    # Most clients re-populate their retained state on reconnect: evcc
+    # republishes state, Shellies re-send their announce/online, HA re-publishes
+    # its own discovery.
+    #
+    # Zigbee2MQTT is the EXCEPTION and does not self-heal. Its HA discovery
+    # configs are published retained, and retention is the only mechanism it
+    # relies on to survive a broker restart — verified against Z2M 2.12.1
+    # source: discover() runs only from start(), the MQTT reconnect path
+    # (lib/mqtt.ts onConnect) republishes bridge/state and nothing else, and the
+    # homeassistant/status "online" handler (lib/extension/homeassistant.ts
+    # onMQTTMessage) republishes bridge/state + cached device states but never
+    # the discovery configs. There is no bridge/request endpoint that forces a
+    # rediscovery. Upstream has declined to add one (Koenkk/zigbee2mqtt #27458,
+    # #22977, #16950, all closed NOT_PLANNED).
+    #
+    # Consequence: a mosquitto restart that is not followed by a Zigbee2MQTT
+    # restart leaves every Z2M entity unavailable in HA until Z2M is restarted
+    # (`kubectl rollout restart deploy/zigbee2mqtt -n home-assistant`). This is
+    # a live operational hazard, tracked in gitea issue #381 — it caused a ~14 h
+    # outage of every Z2M entity on 2026-07-23. Enabling persistence is NOT the
+    # answer: this broker deliberately has no PVC (see above).
     persistence false
     log_dest stdout
     log_type error

kubernetes/apps/home-assistant/mosquitto-deployment.yaml

@@ -25,7 +25,13 @@ spec:
     # a broker serving throughout an image bump / reschedule (clients on the old
     # pod just reconnect to the new one); maxSurge:1 permits the brief two-pod
     # overlap — two independent stateless brokers behind the LB, fine since there
-    # is no shared state and discovery self-heals.
+    # is no shared state.
+    #
+    # NOTE: connectivity survives the roll, but RETAINED STATE DOES NOT — the
+    # new pod starts with an empty retained store, so a roll is equivalent to a
+    # broker restart for every client that depends on retention. Zigbee2MQTT
+    # does not recover from that on its own; see mosquitto-config.yaml and
+    # gitea issue #381.
     type: RollingUpdate
     rollingUpdate:
       maxUnavailable: 0
@@ -64,7 +70,9 @@ spec:
           # Pinned tag, auto-bumped within the 2.x line by Flux image-automation
           # (infrastructure/image-automation/mosquitto.yaml). The marker tells
           # Flux where to write; patch/minor updates land as git commits and
-          # roll the broker (brief restart; discovery self-heals via HA).
+          # roll the broker. HA re-publishes its own discovery afterwards, but
+          # Zigbee2MQTT does not — an image bump therefore needs the same
+          # follow-up as any other broker restart (see mosquitto-config.yaml).
           image: eclipse-mosquitto:2.0.22 # {"$imagepolicy": "flux-system:mosquitto"}
           # The shim runs as root (image sets no USER) and prepares the
           # auth files for the broker, which drops to user mosquitto
@@ -148,8 +156,16 @@ spec:
               # Owns its own `zigbee2mqtt/#` subtree; may WRITE `homeassistant/#`
               # to publish HA MQTT-discovery configs (write-only there, like the
               # shellies user, to deny discovery-tree recon from a compromised
-              # bridge); and READs `homeassistant/status` so it re-publishes
-              # discovery when HA restarts (birth/will).
+              # bridge); and READs `homeassistant/status`, HA's birth/will topic.
+              #
+              # That read grant does NOT buy a discovery re-publish: on
+              # `online`, Z2M 2.12.1 republishes bridge/state and cached device
+              # states only (lib/extension/homeassistant.ts onMQTTMessage) —
+              # discover() runs solely from start(). The grant is still correct,
+              # because the cached-state republish is what makes HA show current
+              # values right after an HA restart. Restoring discovery after a
+              # BROKER restart is handled by the discovery-guard sidecar in
+              # zigbee2mqtt-deployment.yaml.
               user zigbee2mqtt
               topic readwrite zigbee2mqtt/#
               topic write homeassistant/#

kubernetes/apps/home-assistant/zigbee2mqtt-deployment.yaml

@@ -272,6 +272,64 @@ spec:
               memory: 16Mi
             limits:
               memory: 64Mi
+        # Re-runs Zigbee2MQTT's HA discovery after the broker loses its retained
+        # store. mosquitto runs `persistence false` with no PVC, so every broker
+        # restart wipes the retained discovery configs, and nothing on the Z2M
+        # side restores them — not reconnect, not HA's birth message, and there
+        # is no rediscovery endpoint. Without this, a broker restart leaves every
+        # Z2M entity unavailable in HA until someone restarts Z2M by hand (gitea
+        # issue #381; ~14 h outage on 2026-07-23). The full reasoning, including
+        # why bridge/groups is the sentinel and why the guard is tuned for
+        # sensitivity, is in the script header.
+        - name: discovery-guard
+          # Same image and rationale as interview-watch above: reused purely for
+          # its bundled mosquitto_sub/mosquitto_pub, nothing installed at
+          # runtime. Verified present in this exact image: sh, mosquitto_sub
+          # (incl. -C and -W), mosquitto_pub, date -u +%s, mkdir, cat.
+          # Deliberately NOT carrying a Flux $imagepolicy marker, so it does not
+          # drag the broker's image-automation policy along with it.
+          image: eclipse-mosquitto:2.0.22
+          imagePullPolicy: IfNotPresent
+          command: [sh, /script/guard.sh]
+          env:
+            # Reuses the Z2M broker credential: its ACL is `readwrite
+            # zigbee2mqtt/#`, which covers reading bridge/state/bridge/groups and
+            # writing bridge/request/restart. No new broker user needed.
+            - name: MQTT_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: mosquitto-zigbee2mqtt-credentials
+                  key: password
+            # Must track zigbee2mqtt-config.yaml, same as interview-watch.
+            - name: MQTT_HOST
+              value: mosquitto
+            - name: MQTT_USER
+              value: zigbee2mqtt
+            - name: BASE_TOPIC
+              value: zigbee2mqtt
+            - name: STATEDIR
+              value: /state
+          securityContext:
+            runAsNonRoot: true
+            runAsUser: 1883
+            runAsGroup: 1883
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop: [ALL]
+          volumeMounts:
+            - name: discovery-guard-script
+              mountPath: /script
+              readOnly: true
+            # Restart cooldown stamp. Its own emptyDir rather than the image's
+            # /tmp, so it stays writable as non-root.
+            - name: discovery-guard-state
+              mountPath: /state
+          resources:
+            requests:
+              cpu: 10m
+              memory: 16Mi
+            limits:
+              memory: 64Mi
       volumes:
         - name: data
           persistentVolumeClaim:
@@ -287,3 +345,8 @@ spec:
             name: zigbee2mqtt-interview-watch-script
         - name: interview-watch-state
           emptyDir: {}
+        - name: discovery-guard-script
+          configMap:
+            name: zigbee2mqtt-discovery-guard-script
+        - name: discovery-guard-state
+          emptyDir: {}

kubernetes/apps/home-assistant/zigbee2mqtt-discovery-guard.sh

@@ -0,0 +1,162 @@
+#!/bin/sh
+# Restores Home Assistant MQTT discovery after the broker loses its retained
+# store, which mosquitto does on every restart because it runs `persistence
+# false` and has no PVC (a deliberate, settled decision — see
+# mosquitto-config.yaml).
+#
+# WHY THIS EXISTS
+# Zigbee2MQTT publishes its HA discovery configs retained and relies on broker
+# retention alone to survive a broker restart. Verified against Z2M 2.12.1
+# source, nothing on the Z2M side restores them:
+#   - discover() runs only from HomeAssistant.start().
+#   - lib/mqtt.ts onConnect() republishes bridge/state and resubscribes; the
+#     republishRetainedTimer is armed only inside connect() and never re-armed
+#     on reconnect.
+#   - The homeassistant/status "online" handler republishes bridge/state plus
+#     cached device states, never the discovery configs.
+#   - There is no bridge/request endpoint that forces a rediscovery; upstream
+#     declined to add one (Koenkk/zigbee2mqtt #27458, #22977, #16950).
+# So a broker restart not followed by a Z2M restart leaves every Z2M entity
+# unavailable in HA indefinitely. That cost ~14 h of dead entities on
+# 2026-07-23 (gitea issue #381).
+#
+# DETECTION — why bridge/groups
+# bridge/state and the four bridge/{info,devices,groups,definitions} topics are
+# all retained, but only bridge/state is republished on reconnect (mqtt.ts
+# onConnect). So "bridge/state present + one of the others absent" means: Z2M is
+# talking to this broker right now, but nothing has re-run start() since the
+# retained store was wiped. That is exactly the broken state.
+#
+# The absent-half partner must not be re-publishable by ordinary runtime
+# events, or it self-heals and the guard goes blind while HA is still broken:
+#   bridge/devices     — republished on announce/join/leave/interview. Worst.
+#   bridge/definitions — republished on devicesChanged and OTA.
+#   bridge/info        — republished on devicesChanged, which reaches inbound
+#                        radio traffic via the deviceExposesChanged callback
+#                        handed to every fromZigbee converter (receive.ts).
+#   bridge/health      — self-heals on its own 10 min timer. Useless here.
+#   bridge/groups      — only re-published by deliberate admin actions: group
+#                        membership edits, group add/remove/rename, and scene
+#                        changes (which come solely from an explicit
+#                        <device>/set scene command, never an inbound report).
+# Hence bridge/groups. It is also safe with no groups defined: publishGroups()
+# always publishes, emitting `[]`, which is still a non-empty retained payload.
+#
+# DELIBERATELY NOT AND-ed with bridge/info being absent. That would cut false
+# positives but introduce false negatives, and the costs are wildly asymmetric:
+# a false positive is one ~60 s Z2M restart, a false negative is every Zigbee
+# entity dead until a human notices. This guard is tuned for sensitivity.
+#
+# Two known ways to get a spurious positive, both handled by CONFIRM_DELAY:
+#   - Startup race. bridge/state is published at mqtt.connect() but
+#     bridge/groups only at the end of BridgeExtension.start(), so the signature
+#     is legitimately true for a few seconds during every normal Z2M boot.
+#   - QoS 0. The bridge/* topics are published with no qos option, so they go
+#     out at-most-once and a publish can in principle be dropped.
+#
+# ACTION
+# Publishes bridge/request/restart, which is a full in-process restart: index.js
+# restart() -> stop(true) tears down every extension, then start() builds a new
+# Controller and re-runs HomeAssistant.start(), hence discover(). It is NOT a
+# herdsman-only restart. discover() republishes only configs that are missing or
+# whose payload changed, which after a wipe is all of them.
+#
+# Credentials: reuses the existing zigbee2mqtt broker user, whose ACL is
+# `readwrite zigbee2mqtt/#` — enough to read bridge/state and bridge/groups and
+# to write bridge/request/restart. No new broker user, and no read grant on
+# homeassistant/# (checking for the discovery configs directly would have needed
+# one, which the ACL denies on purpose).
+set -eu
+
+MQTT_HOST="${MQTT_HOST:-mosquitto}"
+MQTT_PORT="${MQTT_PORT:-1883}"
+MQTT_USER="${MQTT_USER:-zigbee2mqtt}"
+BASE_TOPIC="${BASE_TOPIC:-zigbee2mqtt}"
+# Poll cadence. Worst-case time-to-repair is CHECK_INTERVAL + CONFIRM_DELAY.
+CHECK_INTERVAL="${CHECK_INTERVAL:-300}"
+# Re-check gap before acting, to ride out a normal Z2M boot and a dropped QoS 0
+# publish. Must comfortably exceed a Z2M cold start.
+CONFIRM_DELAY="${CONFIRM_DELAY:-90}"
+# Floor between two restarts, so a Z2M that will not come back healthy is not
+# restarted in a loop.
+COOLDOWN="${COOLDOWN:-1800}"
+# How long to wait for a retained message to land before calling it absent.
+READ_TIMEOUT="${READ_TIMEOUT:-5}"
+# Hard ceiling on a single client invocation, as a backstop to the protocol-level
+# waits below. Every MQTT call here is bounded with `timeout`, because neither
+# client bounds a TCP connect that HANGS rather than being refused — and a stalled
+# connect is most likely precisely when this guard runs, around a broker restart.
+# Without it the loop would block forever and stop logging, which is exactly the
+# silent-failure mode the guard exists to prevent. Same reasoning and the same
+# PUB_TIMEOUT default as zigbee2mqtt-interview-watch.sh.
+#
+# NOT `mosquitto_pub -W`: that flag does not exist for mosquitto_pub (it is a
+# mosquitto_sub option) and makes every publish fail on a usage error.
+# mosquitto_sub keeps its -W, which bounds the wait for a message once CONNECTED;
+# the `timeout` wrapper is what covers the connect phase, so it is set above -W.
+PUB_TIMEOUT="${PUB_TIMEOUT:-10}"
+STATEDIR="${STATEDIR:-/state}"
+STAMP="$STATEDIR/last-restart"
+
+log() { echo "[$(date -u '+%Y-%m-%dT%H:%M:%SZ')] discovery-guard: $*"; }
+
+# Echoes the retained payload of $1, or nothing if the topic holds none.
+# A retained message is delivered immediately on subscribe, so the wait is only
+# ever paid when the topic is genuinely empty.
+retained() {
+  timeout "$((READ_TIMEOUT + 5))" mosquitto_sub -h "$MQTT_HOST" -p "$MQTT_PORT" \
+    -u "$MQTT_USER" -P "$MQTT_PASSWORD" \
+    -t "$1" -C 1 -W "$READ_TIMEOUT" 2>/dev/null || true
+}
+
+# True when Z2M is connected to this broker but has not re-run start() since the
+# retained store was last wiped.
+discovery_lost() {
+  case "$(retained "$BASE_TOPIC/bridge/state")" in
+    *online*) ;;
+    *) return 1 ;;
+  esac
+  [ -z "$(retained "$BASE_TOPIC/bridge/groups")" ]
+}
+
+in_cooldown() {
+  [ -f "$STAMP" ] || return 1
+  last=$(cat "$STAMP" 2>/dev/null || echo 0)
+  now=$(date -u +%s)
+  [ $((now - last)) -lt "$COOLDOWN" ]
+}
+
+mkdir -p "$STATEDIR"
+log "watching $BASE_TOPIC/bridge/groups on $MQTT_HOST:$MQTT_PORT" \
+    "(interval ${CHECK_INTERVAL}s, confirm ${CONFIRM_DELAY}s, cooldown ${COOLDOWN}s)"
+
+while true; do
+  sleep "$CHECK_INTERVAL"
+
+  discovery_lost || continue
+
+  log "bridge/groups missing while Z2M is online — retained store looks wiped;" \
+      "re-checking in ${CONFIRM_DELAY}s before acting"
+  sleep "$CONFIRM_DELAY"
+
+  if ! discovery_lost; then
+    log "recovered on its own, no action taken"
+    continue
+  fi
+
+  if in_cooldown; then
+    log "confirmed, but a restart was requested less than ${COOLDOWN}s ago —" \
+        "skipping. Z2M may be failing to come up; check its logs."
+    continue
+  fi
+
+  log "confirmed — requesting a Zigbee2MQTT restart to re-run discovery"
+  if timeout "$PUB_TIMEOUT" mosquitto_pub -h "$MQTT_HOST" -p "$MQTT_PORT" \
+       -u "$MQTT_USER" -P "$MQTT_PASSWORD" \
+       -t "$BASE_TOPIC/bridge/request/restart" -m '' 2>/dev/null; then
+    date -u +%s > "$STAMP"
+    log "restart requested"
+  else
+    log "failed to publish the restart request; will retry next cycle"
+  fi
+done

kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml

@@ -47,16 +47,47 @@ spec:
       mon_data_avail_warn: "20"
     mgr:
       mgr/volumes/max_concurrent_clones: "16"
-    # QLC tune for the Crucial P2 NVMe per host. Baseline 2026-05-05:
-    # 71.6 MB/s avg / 0.89 s avg latency on a 5-min replicated 4 MiB
-    # write bench (post-SLC-cache). bdev_*_discard need an OSD restart
-    # to take effect (init-time); the bluestore_* params are runtime.
-    # See ceph-speed-baseline-2026-05-05 in the wiki.
+    # Discard/TRIM for the QLC NVMe per host (CT1000P3PSSD8, Crucial P3 Plus —
+    # earlier comments here said "P2", which is a different drive). These need
+    # an OSD restart to take effect (init-time). Baseline 2026-05-05:
+    # 71.6 MB/s avg / 0.89 s avg latency on a 5-min replicated 4 MiB write
+    # bench (post-SLC-cache). See ceph-speed-baseline-2026-05-05 in the wiki.
+    #
+    # bluestore_prefer_deferred_size_ssd / bluestore_deferred_batch_ops_ssd are
+    # DELIBERATELY LEFT AT THE CEPH DEFAULTS (0 and 16) and must not be raised
+    # again without re-reading this.
+    #
+    # They were previously set to the HDD-style 65536 / 64 as a throughput tune.
+    # prefer_deferred_size defers every write SMALLER than the threshold: the
+    # write goes first into RocksDB's WAL and is applied to the object store
+    # afterwards. That is a double write, and at 65536 it caught essentially
+    # every write this cluster makes — SQLite pages are 4 KiB, Postgres pages
+    # 8 KiB. Ceph disables deferral on SSD (default 0) precisely because solid
+    # state does not need the random-write batching that helps spinning rust.
+    #
+    # Two reasons it was the wrong trade here:
+    #   - The benchmark that justified it cannot have exercised it. It used
+    #     4 MiB writes, 64x ABOVE the 64 KiB cutoff, so not one of those writes
+    #     was ever deferred. The setting was validated by a workload it does
+    #     not affect.
+    #   - It doubles writes onto QLC, which has the lowest endurance of any
+    #     NAND type. Measured 2026-08-19: all three OSD drives at 80/82/87%
+    #     endurance used, advancing ~0.13 points/day, i.e. rated life exhausted
+    #     between Nov 2026 and Jan 2027 — and simultaneously, since size=3 on
+    #     3 OSDs means every drive absorbs 100% of cluster writes (see #434).
+    #
+    # This cluster is write-light (~117 GB/day at the RBD client layer), so
+    # throughput was never the binding constraint. Endurance is. If small-write
+    # latency ever becomes the real problem, fix it with more OSDs rather than
+    # by paying a 2x write tax on every 4 KiB page.
     osd:
       bdev_enable_discard: "true"
       bdev_async_discard: "true"
-      bluestore_prefer_deferred_size_ssd: "65536"
-      bluestore_deferred_batch_ops_ssd: "64"
+      # Ceph defaults, set explicitly so the previous override is actively
+      # overwritten rather than merely removed — `ceph config` retains a value
+      # that is only deleted from this file.
+      bluestore_prefer_deferred_size_ssd: "0"
+      bluestore_deferred_batch_ops_ssd: "16"
   dashboard:
     enabled: true
     ssl: false

opentofu/lxc_800_rustfs_decommission.tf

@@ -1,122 +0,0 @@
-# TEMPORARY shims v3 — rustfs LXCs 800/801/802, decommission in progress.
-#
-# Stage 2 (#270) failed: both destroy attempts died instantly with
-# "context deadline exceeded" on the pre-destroy status GET, while the PVE
-# API answers the same call in <1s. Cause: the bpg provider reads
-# `timeout_delete` from STATE at destroy time, and importing a resource
-# whose config declares no timeouts stored 0 — a zero-second context.
-# (The import/plan read path uses its own timeout, which is why stage 1
-# planned clean.)
-#
-# Fix: declare the timeouts explicitly, apply ONCE so they land in state
-# (in-place no-API update), then delete this file again and the next apply
-# destroys the containers.
-#
-# v3: v2 declared ONLY the ids, `started`, `unprivileged` and the timeouts,
-# claiming "the other attributes mirror the real containers so the plan stays
-# in-place". They did not. Everything absent from config planned as a removal,
-# and two of those removals — `mount_point` and `operating_system` — force
-# replacement, so the plan came back `must be replaced` for all three. Applying
-# that would have destroyed and recreated containers tagged `critical`,
-# `state-backend`, `terraform-backend` instead of nudging a timeout into state.
-#
-# So the blocks below are not decoration: each one exists to keep an attribute
-# out of the diff. They mirror the live containers, read from the plan's own
-# state values. `tags` is declared for the same reason — without it the apply
-# strips the tags off the real containers.
-#
-#   1. merge; run PIPELINE=deploy-infrastructure (plan MUST show in-place
-#      update only, no replace), then TOFU_APPLY=true
-#   2. follow-up PR deletes this file; plan shows 3 destroys; apply.
-
-locals {
-  # Per-container values mirrored from live state. hostname and mac_address
-  # differ per container; declaring them wrong would itself force a diff.
-  rustfs_decommission = {
-    "800" = { node = "saturn", hostname = "rustfs-1.srv.renner.uno", mac = "BC:24:11:0B:E5:7B" }
-    "801" = { node = "jupiter", hostname = "rustfs-2.srv.renner.uno", mac = "BC:24:11:EA:1D:38" }
-    "802" = { node = "neptune", hostname = "rustfs-3.srv.renner.uno", mac = "BC:24:11:8F:F4:67" }
-  }
-}
-
-resource "proxmox_virtual_environment_container" "rustfs_decommission" {
-  for_each = local.rustfs_decommission
-
-  node_name     = each.value.node
-  vm_id         = tonumber(each.key)
-  started       = false
-  unprivileged  = true
-  start_on_boot = true
-
-  tags = [
-    "critical",
-    "rustfs",
-    "state-backend",
-    "terraform-backend",
-  ]
-
-  timeout_create = 1800
-  timeout_clone  = 1800
-  timeout_update = 1800
-  timeout_delete = 1800
-  timeout_start  = 300
-
-  cpu {
-    architecture = "amd64"
-    cores        = 1
-    units        = 1024
-  }
-
-  memory {
-    dedicated = 1024
-    swap      = 0
-  }
-
-  disk {
-    datastore_id = "local"
-    size         = 10
-  }
-
-  # forces replacement if absent — this is the block that broke v2.
-  mount_point {
-    path      = "/mnt/data"
-    size      = "50G"
-    volume    = "local:${each.key}/vm-${each.key}-disk-1.raw"
-    backup    = true
-    replicate = true
-  }
-
-  network_interface {
-    name        = "eth0"
-    bridge      = "v40srv"
-    enabled     = true
-    firewall    = false
-    mac_address = each.value.mac
-  }
-
-  initialization {
-    hostname = each.value.hostname
-
-    ip_config {
-      ipv4 {
-        address = "dhcp"
-      }
-    }
-  }
-
-  # operating_system is NOT declared, and must not be. The provider requires
-  # template_file_id inside it, which is a create-time-only value: PVE does not
-  # report the original template for an existing container, so it is absent
-  # from state and cannot be mirrored. Declaring the block without it fails
-  # validation ("Missing required argument"); declaring it with a guessed
-  # template would be a lie that only surfaces if something ever recreates
-  # these.
-  #
-  # Left undeclared it would plan as a removal, and removing operating_system
-  # forces replacement — the exact failure this file exists to avoid. So it is
-  # ignored instead. The timeouts above are deliberately NOT ignored: they are
-  # the one thing this shim needs to actually apply.
-  lifecycle {
-    ignore_changes = [operating_system]
-  }
-}