fix(ceph): stop mon compacting on every paxos trim (240 GB/day of writes)

The three Ceph monitors were writing 240 GB/day between them for a ~106 MB database. That was ~62% of all OS-disk writes on the nodes and roughly a quarter of everything the cluster writes -- more than every client workload combined. Measured 2026-08-19: paxos transactions ~99,000/day (1.15/s, sampled over 60 s) paxos logical data 811 MB/day (21.9 GB over 27 d mon uptime) ACTUAL mon disk writes 80 GB/day per mon, 240 GB/day for three (cgroup io.stat: 2.15 TB / 27 d) => ~98x write amplification Cause: mon_compact_on_trim, upstream default true, "compact a certain prefix (including paxos) when we trim its old states". With paxos_trim_min at 250 and ~99k transactions/day, that is hundreds of forced range compactions per day, each rewriting the store. The churn driving it is not cluster activity. 99% of paxos versions are cluster log and mgr stats -- logm 6.7M, mgrstat 4.4M, against osdmap's 9.6k. Every Rook operator poll (mgr stat / versions / status / report / osd dump; 23 audit entries per minute per mon) becomes a cluster-log entry, hence a paxos transaction, replicated to all three mons. Reducing that polling is the complementary fix and is tracked separately; this change removes the amplification regardless of the rate. This does NOT disable compaction. RocksDB's own levelled compaction still runs and still reclaims trimmed keys; what stops is the extra forced compact_range on every trim. Two backstops if that assumption is wrong: - mon_data_size_warn (15 GB default) raises HEALTH_WARN, which CephHealthWarn in monitoring-rules already alerts on. - mon_compact_on_start is set to true here (upstream default false), so every mon restart -- upgrade, node reboot, reschedule -- does a full compaction. Cheap on a ~106 MB store. Baseline for verification, mon.a cgroup io.stat at 2026-08-19T12:56:29Z: wbytes=2154257645056, wios=22509683, uptime since 2026-07-23T13:20:44Z. Note on methodology: cAdvisor's container_fs_writes_bytes_total roughly double-counts here -- it reported 425 GB/day for OSDs where nvme0n1 shows 212, and 155 GB/day for mons where cgroup io.stat shows 80. It was used only to rank writers; every figure above comes from cgroup io.stat or kernel diskstats, which reconcile with each other and with the drives' SMART counters. Refs #434 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Merge pull request 'fix(ceph): replicapool and cephfs data to size 2, min_size 1' (#519 ) from fix-ceph-replica-size-2 into master
2026-08-19 14:56:47 +02:00 · 2026-08-19 12:40:01 +00:00 · 2026-08-19 14:35:58 +02:00 · 2026-08-19 12:30:21 +00:00 · 2026-08-19 14:27:01 +02:00 · 2026-08-19 10:54:27 +00:00
12 changed files with 492 additions and 158 deletions
@@ -135,8 +135,38 @@ steps:
        TITLES=$$(jq -c '[.[].title]' .ci-alert-monitor/open-issues.json)
        NOISE_BODY=$$(jq -r 'map(select(.title == "Alert monitor: noise log")) | .[0].body // ""' \
          .ci-alert-monitor/open-issues.json)
-        FPS=$$(printf '%s' "$$NOISE_BODY" | grep -oE '\b[0-9a-f]{16}\b' \
+        # Suppression entries are exactly the structured lines the write-back
+        # step emits: "- <fp> <ALERTNAME> — <YYYY-MM-DD>: <reason>". Parse THAT
+        # shape rather than grepping the whole body for 16-hex tokens. A reason
+        # routinely cites another alert's fingerprint in prose ("see the issue
+        # filed for fingerprint <fp>"), and the body-wide grep turned every such
+        # citation into a silent, permanent suppression of an alert that was
+        # never classified as noise — verified against the live noise log, which
+        # yielded 22 fingerprints for 21 real entries, the extra one being
+        # 2ddc41edb689fc33, an alert that HAD been filed as a genuine issue.
+        #
+        # Entries also expire after NOISE_TTL_DAYS. A noise verdict says "this
+        # occurrence was benign", not "this alert can never be interesting
+        # again" — but the fingerprint is sha256(alertname|scope) and therefore
+        # stable across recurrences, so without an expiry a single blip mutes
+        # that alertname+scope pair forever. Re-triaging a recurrence weeks
+        # later is cheap; missing a real outage because a July blip muted
+        # openbao's KubePodNotReady is not.
+        #
+        # Lines that fail to parse are simply dropped, i.e. they stop
+        # suppressing — this fails toward MORE alerting, never less.
+        #
+        # date is BusyBox here, not GNU: `date -d '30 days ago'` errors out, so
+        # the cutoff goes through epoch arithmetic and `date -d @<epoch>`, which
+        # this image does support.
+        NOISE_TTL_DAYS=$${NOISE_TTL_DAYS:-30}
+        CUTOFF=$$(date -u -d @$$(( $$(date -u +%s) - NOISE_TTL_DAYS * 86400 )) +%F)
+        FPS=$$(printf '%s\n' "$$NOISE_BODY" \
+          | sed -nE 's/^- ([0-9a-f]{16}) [^ ]+ — ([0-9]{4}-[0-9]{2}-[0-9]{2}):.*/\2 \1/p' \
+          | awk -v c="$$CUTOFF" 'BEGIN{gsub(/-/,"",c)} { d=$$1; gsub(/-/,"",d); if (d+0 >= c+0) print $$2 }' \
          | jq -R -s -c 'split("\n") | map(select(length > 0))')
+        echo "Noise suppressions in effect: $$(printf '%s' "$${FPS:-[]}" | jq length)" \
+             "(TTL $${NOISE_TTL_DAYS}d, entries dated >= $$CUTOFF)"
        # Synthesize pseudo-alerts for BLIPS: lookback series whose label set
        # is not currently firing at all (checked against the PRE-filter raw
        # snapshot, so live-but-null-routed alerts don't reappear as blips).
@@ -260,7 +290,7 @@ steps:
              '{body: ($$old + "\n" + $$add)}' > /tmp/gitea-patch.json
            gitea_write PATCH "$$API/issues/$$NOISE_NUM" /tmp/gitea-patch.json
          else
-            printf '%s\n\n' "Fingerprints the alert-monitor triage agent classified as noise. The dedup step skips any alert whose fingerprint appears below — delete a line to make that alert eligible for triage again. Keep this issue open; it is the suppression state." > /tmp/noise-header.txt
+            printf '%s\n\n' "Fingerprints the alert-monitor triage agent classified as noise. The dedup step skips any alert whose fingerprint appears below — delete a line to make that alert eligible for triage again. Keep this issue open; it is the suppression state. Entries EXPIRE after NOISE_TTL_DAYS (default 30) counted from the date in the line, so a one-off blip stops suppressing its alertname+scope pair once the window passes; re-add happens automatically if it is triaged as noise again. Only lines in the exact '- <fingerprint> <ALERTNAME> — <YYYY-MM-DD>: <reason>' form suppress anything — a fingerprint mentioned in prose does not." > /tmp/noise-header.txt
            jq -n --rawfile hdr /tmp/noise-header.txt --rawfile add /tmp/noise-lines.txt \
              '{title: "Alert monitor: noise log", body: ($$hdr + $$add)}' > /tmp/gitea-create.json
            gitea_write POST "$$API/issues" /tmp/gitea-create.json
@@ -79,5 +79,13 @@ configMapGenerator:
    namespace: home-assistant
    files:
      - watch.sh=zigbee2mqtt-interview-watch.sh
+  # Z2M discovery guard sidecar script (gitea issue #381). Hash suffix disabled
+  # (name stays stable), so editing zigbee2mqtt-discovery-guard.sh does not
+  # auto-roll the pod — restart manually with
+  # `kubectl rollout restart deploy/zigbee2mqtt -n home-assistant`.
+  - name: zigbee2mqtt-discovery-guard-script
+    namespace: home-assistant
+    files:
+      - guard.sh=zigbee2mqtt-discovery-guard.sh
 generatorOptions:
  disableNameSuffixHash: true
@@ -373,8 +373,9 @@ spec:
 # Broker user `zigbee2mqtt` — the Zigbee2MQTT bridge (ZHA replacement). Scoped
 # to its own `zigbee2mqtt/#` subtree plus `write homeassistant/#` (so it can
 # publish HA MQTT-discovery configs, exactly like the `shellies` user) and
-# `read homeassistant/status` (Z2M re-publishes discovery when HA restarts —
-# ACL in mosquitto-deployment.yaml). Own credential so its lifecycle is
+# `read homeassistant/status` (HA's birth/will topic; on `online` Z2M
+# republishes cached device states, NOT discovery configs — ACL and the full
+# reasoning in mosquitto-deployment.yaml). Own credential so its lifecycle is
 # independent of the others. The Z2M init container reads this password and
 # renders it into /app/data/secret.yaml.
 apiVersion: batch/v1
@@ -5,13 +5,31 @@ metadata:
  namespace: home-assistant
 data:
  mosquitto.conf: |
-    # No on-disk persistence. Retained messages + sessions are re-populated by
-    # clients on reconnect (evcc republishes state, Shellies re-send their
-    # announce/online, HA re-publishes discovery), so the broker keeps no
-    # durable state. This drops the Ceph PVC dependency (the broker starts even
-    # if Ceph is degraded) and lets it reschedule anywhere in seconds — no RWO
-    # volume to fence/detach/reattach. Trade-off: a broker restart briefly
-    # empties retained state until clients reconnect (self-healing).
+    # No on-disk persistence, and no PVC of any kind for this broker. The
+    # broker keeps no durable state, so it starts even if Ceph is degraded and
+    # reschedules anywhere in seconds — no RWO volume to fence/detach/reattach.
+    #
+    # Most clients re-populate their retained state on reconnect: evcc
+    # republishes state, Shellies re-send their announce/online, HA re-publishes
+    # its own discovery.
+    #
+    # Zigbee2MQTT is the EXCEPTION and does not self-heal. Its HA discovery
+    # configs are published retained, and retention is the only mechanism it
+    # relies on to survive a broker restart — verified against Z2M 2.12.1
+    # source: discover() runs only from start(), the MQTT reconnect path
+    # (lib/mqtt.ts onConnect) republishes bridge/state and nothing else, and the
+    # homeassistant/status "online" handler (lib/extension/homeassistant.ts
+    # onMQTTMessage) republishes bridge/state + cached device states but never
+    # the discovery configs. There is no bridge/request endpoint that forces a
+    # rediscovery. Upstream has declined to add one (Koenkk/zigbee2mqtt #27458,
+    # #22977, #16950, all closed NOT_PLANNED).
+    #
+    # Consequence: a mosquitto restart that is not followed by a Zigbee2MQTT
+    # restart leaves every Z2M entity unavailable in HA until Z2M is restarted
+    # (`kubectl rollout restart deploy/zigbee2mqtt -n home-assistant`). This is
+    # a live operational hazard, tracked in gitea issue #381 — it caused a ~14 h
+    # outage of every Z2M entity on 2026-07-23. Enabling persistence is NOT the
+    # answer: this broker deliberately has no PVC (see above).
    persistence false
    log_dest stdout
    log_type error
@@ -25,7 +25,13 @@ spec:
    # a broker serving throughout an image bump / reschedule (clients on the old
    # pod just reconnect to the new one); maxSurge:1 permits the brief two-pod
    # overlap — two independent stateless brokers behind the LB, fine since there
-    # is no shared state and discovery self-heals.
+    # is no shared state.
+    #
+    # NOTE: connectivity survives the roll, but RETAINED STATE DOES NOT — the
+    # new pod starts with an empty retained store, so a roll is equivalent to a
+    # broker restart for every client that depends on retention. Zigbee2MQTT
+    # does not recover from that on its own; see mosquitto-config.yaml and
+    # gitea issue #381.
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
@@ -64,7 +70,9 @@ spec:
          # Pinned tag, auto-bumped within the 2.x line by Flux image-automation
          # (infrastructure/image-automation/mosquitto.yaml). The marker tells
          # Flux where to write; patch/minor updates land as git commits and
-          # roll the broker (brief restart; discovery self-heals via HA).
+          # roll the broker. HA re-publishes its own discovery afterwards, but
+          # Zigbee2MQTT does not — an image bump therefore needs the same
+          # follow-up as any other broker restart (see mosquitto-config.yaml).
          image: eclipse-mosquitto:2.0.22 # {"$imagepolicy": "flux-system:mosquitto"}
          # The shim runs as root (image sets no USER) and prepares the
          # auth files for the broker, which drops to user mosquitto
@@ -148,8 +156,16 @@ spec:
              # Owns its own `zigbee2mqtt/#` subtree; may WRITE `homeassistant/#`
              # to publish HA MQTT-discovery configs (write-only there, like the
              # shellies user, to deny discovery-tree recon from a compromised
-              # bridge); and READs `homeassistant/status` so it re-publishes
-              # discovery when HA restarts (birth/will).
+              # bridge); and READs `homeassistant/status`, HA's birth/will topic.
+              #
+              # That read grant does NOT buy a discovery re-publish: on
+              # `online`, Z2M 2.12.1 republishes bridge/state and cached device
+              # states only (lib/extension/homeassistant.ts onMQTTMessage) —
+              # discover() runs solely from start(). The grant is still correct,
+              # because the cached-state republish is what makes HA show current
+              # values right after an HA restart. Restoring discovery after a
+              # BROKER restart is handled by the discovery-guard sidecar in
+              # zigbee2mqtt-deployment.yaml.
              user zigbee2mqtt
              topic readwrite zigbee2mqtt/#
              topic write homeassistant/#
@@ -272,6 +272,64 @@ spec:
              memory: 16Mi
            limits:
              memory: 64Mi
+        # Re-runs Zigbee2MQTT's HA discovery after the broker loses its retained
+        # store. mosquitto runs `persistence false` with no PVC, so every broker
+        # restart wipes the retained discovery configs, and nothing on the Z2M
+        # side restores them — not reconnect, not HA's birth message, and there
+        # is no rediscovery endpoint. Without this, a broker restart leaves every
+        # Z2M entity unavailable in HA until someone restarts Z2M by hand (gitea
+        # issue #381; ~14 h outage on 2026-07-23). The full reasoning, including
+        # why bridge/groups is the sentinel and why the guard is tuned for
+        # sensitivity, is in the script header.
+        - name: discovery-guard
+          # Same image and rationale as interview-watch above: reused purely for
+          # its bundled mosquitto_sub/mosquitto_pub, nothing installed at
+          # runtime. Verified present in this exact image: sh, mosquitto_sub
+          # (incl. -C and -W), mosquitto_pub, date -u +%s, mkdir, cat.
+          # Deliberately NOT carrying a Flux $imagepolicy marker, so it does not
+          # drag the broker's image-automation policy along with it.
+          image: eclipse-mosquitto:2.0.22
+          imagePullPolicy: IfNotPresent
+          command: [sh, /script/guard.sh]
+          env:
+            # Reuses the Z2M broker credential: its ACL is `readwrite
+            # zigbee2mqtt/#`, which covers reading bridge/state/bridge/groups and
+            # writing bridge/request/restart. No new broker user needed.
+            - name: MQTT_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: mosquitto-zigbee2mqtt-credentials
+                  key: password
+            # Must track zigbee2mqtt-config.yaml, same as interview-watch.
+            - name: MQTT_HOST
+              value: mosquitto
+            - name: MQTT_USER
+              value: zigbee2mqtt
+            - name: BASE_TOPIC
+              value: zigbee2mqtt
+            - name: STATEDIR
+              value: /state
+          securityContext:
+            runAsNonRoot: true
+            runAsUser: 1883
+            runAsGroup: 1883
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop: [ALL]
+          volumeMounts:
+            - name: discovery-guard-script
+              mountPath: /script
+              readOnly: true
+            # Restart cooldown stamp. Its own emptyDir rather than the image's
+            # /tmp, so it stays writable as non-root.
+            - name: discovery-guard-state
+              mountPath: /state
+          resources:
+            requests:
+              cpu: 10m
+              memory: 16Mi
+            limits:
+              memory: 64Mi
      volumes:
        - name: data
          persistentVolumeClaim:
@@ -287,3 +345,8 @@ spec:
            name: zigbee2mqtt-interview-watch-script
        - name: interview-watch-state
          emptyDir: {}
+        - name: discovery-guard-script
+          configMap:
+            name: zigbee2mqtt-discovery-guard-script
+        - name: discovery-guard-state
+          emptyDir: {}
@@ -0,0 +1,162 @@
+#!/bin/sh
+# Restores Home Assistant MQTT discovery after the broker loses its retained
+# store, which mosquitto does on every restart because it runs `persistence
+# false` and has no PVC (a deliberate, settled decision — see
+# mosquitto-config.yaml).
+#
+# WHY THIS EXISTS
+# Zigbee2MQTT publishes its HA discovery configs retained and relies on broker
+# retention alone to survive a broker restart. Verified against Z2M 2.12.1
+# source, nothing on the Z2M side restores them:
+#   - discover() runs only from HomeAssistant.start().
+#   - lib/mqtt.ts onConnect() republishes bridge/state and resubscribes; the
+#     republishRetainedTimer is armed only inside connect() and never re-armed
+#     on reconnect.
+#   - The homeassistant/status "online" handler republishes bridge/state plus
+#     cached device states, never the discovery configs.
+#   - There is no bridge/request endpoint that forces a rediscovery; upstream
+#     declined to add one (Koenkk/zigbee2mqtt #27458, #22977, #16950).
+# So a broker restart not followed by a Z2M restart leaves every Z2M entity
+# unavailable in HA indefinitely. That cost ~14 h of dead entities on
+# 2026-07-23 (gitea issue #381).
+#
+# DETECTION — why bridge/groups
+# bridge/state and the four bridge/{info,devices,groups,definitions} topics are
+# all retained, but only bridge/state is republished on reconnect (mqtt.ts
+# onConnect). So "bridge/state present + one of the others absent" means: Z2M is
+# talking to this broker right now, but nothing has re-run start() since the
+# retained store was wiped. That is exactly the broken state.
+#
+# The absent-half partner must not be re-publishable by ordinary runtime
+# events, or it self-heals and the guard goes blind while HA is still broken:
+#   bridge/devices     — republished on announce/join/leave/interview. Worst.
+#   bridge/definitions — republished on devicesChanged and OTA.
+#   bridge/info        — republished on devicesChanged, which reaches inbound
+#                        radio traffic via the deviceExposesChanged callback
+#                        handed to every fromZigbee converter (receive.ts).
+#   bridge/health      — self-heals on its own 10 min timer. Useless here.
+#   bridge/groups      — only re-published by deliberate admin actions: group
+#                        membership edits, group add/remove/rename, and scene
+#                        changes (which come solely from an explicit
+#                        <device>/set scene command, never an inbound report).
+# Hence bridge/groups. It is also safe with no groups defined: publishGroups()
+# always publishes, emitting `[]`, which is still a non-empty retained payload.
+#
+# DELIBERATELY NOT AND-ed with bridge/info being absent. That would cut false
+# positives but introduce false negatives, and the costs are wildly asymmetric:
+# a false positive is one ~60 s Z2M restart, a false negative is every Zigbee
+# entity dead until a human notices. This guard is tuned for sensitivity.
+#
+# Two known ways to get a spurious positive, both handled by CONFIRM_DELAY:
+#   - Startup race. bridge/state is published at mqtt.connect() but
+#     bridge/groups only at the end of BridgeExtension.start(), so the signature
+#     is legitimately true for a few seconds during every normal Z2M boot.
+#   - QoS 0. The bridge/* topics are published with no qos option, so they go
+#     out at-most-once and a publish can in principle be dropped.
+#
+# ACTION
+# Publishes bridge/request/restart, which is a full in-process restart: index.js
+# restart() -> stop(true) tears down every extension, then start() builds a new
+# Controller and re-runs HomeAssistant.start(), hence discover(). It is NOT a
+# herdsman-only restart. discover() republishes only configs that are missing or
+# whose payload changed, which after a wipe is all of them.
+#
+# Credentials: reuses the existing zigbee2mqtt broker user, whose ACL is
+# `readwrite zigbee2mqtt/#` — enough to read bridge/state and bridge/groups and
+# to write bridge/request/restart. No new broker user, and no read grant on
+# homeassistant/# (checking for the discovery configs directly would have needed
+# one, which the ACL denies on purpose).
+set -eu
+
+MQTT_HOST="${MQTT_HOST:-mosquitto}"
+MQTT_PORT="${MQTT_PORT:-1883}"
+MQTT_USER="${MQTT_USER:-zigbee2mqtt}"
+BASE_TOPIC="${BASE_TOPIC:-zigbee2mqtt}"
+# Poll cadence. Worst-case time-to-repair is CHECK_INTERVAL + CONFIRM_DELAY.
+CHECK_INTERVAL="${CHECK_INTERVAL:-300}"
+# Re-check gap before acting, to ride out a normal Z2M boot and a dropped QoS 0
+# publish. Must comfortably exceed a Z2M cold start.
+CONFIRM_DELAY="${CONFIRM_DELAY:-90}"
+# Floor between two restarts, so a Z2M that will not come back healthy is not
+# restarted in a loop.
+COOLDOWN="${COOLDOWN:-1800}"
+# How long to wait for a retained message to land before calling it absent.
+READ_TIMEOUT="${READ_TIMEOUT:-5}"
+# Hard ceiling on a single client invocation, as a backstop to the protocol-level
+# waits below. Every MQTT call here is bounded with `timeout`, because neither
+# client bounds a TCP connect that HANGS rather than being refused — and a stalled
+# connect is most likely precisely when this guard runs, around a broker restart.
+# Without it the loop would block forever and stop logging, which is exactly the
+# silent-failure mode the guard exists to prevent. Same reasoning and the same
+# PUB_TIMEOUT default as zigbee2mqtt-interview-watch.sh.
+#
+# NOT `mosquitto_pub -W`: that flag does not exist for mosquitto_pub (it is a
+# mosquitto_sub option) and makes every publish fail on a usage error.
+# mosquitto_sub keeps its -W, which bounds the wait for a message once CONNECTED;
+# the `timeout` wrapper is what covers the connect phase, so it is set above -W.
+PUB_TIMEOUT="${PUB_TIMEOUT:-10}"
+STATEDIR="${STATEDIR:-/state}"
+STAMP="$STATEDIR/last-restart"
+
+log() { echo "[$(date -u '+%Y-%m-%dT%H:%M:%SZ')] discovery-guard: $*"; }
+
+# Echoes the retained payload of $1, or nothing if the topic holds none.
+# A retained message is delivered immediately on subscribe, so the wait is only
+# ever paid when the topic is genuinely empty.
+retained() {
+  timeout "$((READ_TIMEOUT + 5))" mosquitto_sub -h "$MQTT_HOST" -p "$MQTT_PORT" \
+    -u "$MQTT_USER" -P "$MQTT_PASSWORD" \
+    -t "$1" -C 1 -W "$READ_TIMEOUT" 2>/dev/null || true
+}
+
+# True when Z2M is connected to this broker but has not re-run start() since the
+# retained store was last wiped.
+discovery_lost() {
+  case "$(retained "$BASE_TOPIC/bridge/state")" in
+    *online*) ;;
+    *) return 1 ;;
+  esac
+  [ -z "$(retained "$BASE_TOPIC/bridge/groups")" ]
+}
+
+in_cooldown() {
+  [ -f "$STAMP" ] || return 1
+  last=$(cat "$STAMP" 2>/dev/null || echo 0)
+  now=$(date -u +%s)
+  [ $((now - last)) -lt "$COOLDOWN" ]
+}
+
+mkdir -p "$STATEDIR"
+log "watching $BASE_TOPIC/bridge/groups on $MQTT_HOST:$MQTT_PORT" \
+    "(interval ${CHECK_INTERVAL}s, confirm ${CONFIRM_DELAY}s, cooldown ${COOLDOWN}s)"
+
+while true; do
+  sleep "$CHECK_INTERVAL"
+
+  discovery_lost || continue
+
+  log "bridge/groups missing while Z2M is online — retained store looks wiped;" \
+      "re-checking in ${CONFIRM_DELAY}s before acting"
+  sleep "$CONFIRM_DELAY"
+
+  if ! discovery_lost; then
+    log "recovered on its own, no action taken"
+    continue
+  fi
+
+  if in_cooldown; then
+    log "confirmed, but a restart was requested less than ${COOLDOWN}s ago —" \
+        "skipping. Z2M may be failing to come up; check its logs."
+    continue
+  fi
+
+  log "confirmed — requesting a Zigbee2MQTT restart to re-run discovery"
+  if timeout "$PUB_TIMEOUT" mosquitto_pub -h "$MQTT_HOST" -p "$MQTT_PORT" \
+       -u "$MQTT_USER" -P "$MQTT_PASSWORD" \
+       -t "$BASE_TOPIC/bridge/request/restart" -m '' 2>/dev/null; then
+    date -u +%s > "$STAMP"
+    log "restart requested"
+  else
+    log "failed to publish the restart request; will retry next cycle"
+  fi
+done
@@ -45,18 +45,92 @@ spec:
      # would be circular, and Rook only applies mon.volumeClaimTemplate to
      # newly created mons, never migrating existing ones.
      mon_data_avail_warn: "20"
+      # Stop compacting the whole RocksDB prefix on every paxos trim.
+      #
+      # Upstream default is true — "Compact a certain prefix (including paxos)
+      # when we trim its old states" — which is fine at low paxos rates and
+      # pathological here. Measured 2026-08-19:
+      #
+      #   paxos transactions      ~99,000/day  (1.15/s, sampled over 60 s)
+      #   paxos logical data       811 MB/day  (21.9 GB over 27 d uptime)
+      #   ACTUAL mon disk writes    80 GB/day  per mon, 240 GB/day for three
+      #                                        (cgroup io.stat: 2.15 TB/27 d)
+      #   => ~98x write amplification, for a ~106 MB database
+      #
+      # That was ~62% of all OS-disk writes on these nodes and roughly a
+      # quarter of everything the cluster writes — more than every client
+      # workload combined. With paxos_trim_min at 250, ~99k transactions/day
+      # means hundreds of forced range compactions per day, each rewriting the
+      # store.
+      #
+      # The churn itself is 99% cluster log and mgr stats, not real cluster
+      # activity: logm holds 6.7M paxos versions and mgrstat 4.4M, against
+      # osdmap's 9.6k. Every Rook-operator poll (mgr stat / versions / status /
+      # report / osd dump — 23 audit entries per minute per mon) becomes a
+      # cluster-log entry, hence a paxos transaction, replicated to all three
+      # mons. Reducing that polling is the complementary fix and is tracked
+      # separately; this setting removes the amplification regardless of rate.
+      #
+      # This does NOT disable compaction. RocksDB's own levelled compaction
+      # still runs and still reclaims trimmed keys; what stops is the extra
+      # forced compact_range on every trim. Store growth is therefore bounded
+      # by normal compaction, and there are two backstops if that assumption is
+      # wrong: mon_data_size_warn (15 GB default) raises HEALTH_WARN, which
+      # CephHealthWarn in monitoring-rules already alerts on, and
+      # mon_compact_on_start below compacts fully on every mon restart.
+      #
+      # Watch after applying: `du -sh /var/lib/rook/mon-*/store.db` on a node,
+      # or the mon store size in `ceph status`. If it grows past ~1 GB and does
+      # not come back down, revisit rather than leaving it.
+      mon_compact_on_trim: "false"
+      # Compensating control for the above: a full compaction on mon start.
+      # Upstream default is false. Mons restart on upgrades, node reboots and
+      # reschedules, so this gives a periodic full reclaim without paying for
+      # one on every trim. Cheap here — the store is ~106 MB.
+      mon_compact_on_start: "true"
    mgr:
      mgr/volumes/max_concurrent_clones: "16"
-    # QLC tune for the Crucial P2 NVMe per host. Baseline 2026-05-05:
-    # 71.6 MB/s avg / 0.89 s avg latency on a 5-min replicated 4 MiB
-    # write bench (post-SLC-cache). bdev_*_discard need an OSD restart
-    # to take effect (init-time); the bluestore_* params are runtime.
-    # See ceph-speed-baseline-2026-05-05 in the wiki.
+    # Discard/TRIM for the QLC NVMe per host (CT1000P3PSSD8, Crucial P3 Plus —
+    # earlier comments here said "P2", which is a different drive). These need
+    # an OSD restart to take effect (init-time). Baseline 2026-05-05:
+    # 71.6 MB/s avg / 0.89 s avg latency on a 5-min replicated 4 MiB write
+    # bench (post-SLC-cache). See ceph-speed-baseline-2026-05-05 in the wiki.
+    #
+    # bluestore_prefer_deferred_size_ssd / bluestore_deferred_batch_ops_ssd are
+    # DELIBERATELY LEFT AT THE CEPH DEFAULTS (0 and 16) and must not be raised
+    # again without re-reading this.
+    #
+    # They were previously set to the HDD-style 65536 / 64 as a throughput tune.
+    # prefer_deferred_size defers every write SMALLER than the threshold: the
+    # write goes first into RocksDB's WAL and is applied to the object store
+    # afterwards. That is a double write, and at 65536 it caught essentially
+    # every write this cluster makes — SQLite pages are 4 KiB, Postgres pages
+    # 8 KiB. Ceph disables deferral on SSD (default 0) precisely because solid
+    # state does not need the random-write batching that helps spinning rust.
+    #
+    # Two reasons it was the wrong trade here:
+    #   - The benchmark that justified it cannot have exercised it. It used
+    #     4 MiB writes, 64x ABOVE the 64 KiB cutoff, so not one of those writes
+    #     was ever deferred. The setting was validated by a workload it does
+    #     not affect.
+    #   - It doubles writes onto QLC, which has the lowest endurance of any
+    #     NAND type. Measured 2026-08-19: all three OSD drives at 80/82/87%
+    #     endurance used, advancing ~0.13 points/day, i.e. rated life exhausted
+    #     between Nov 2026 and Jan 2027 — and simultaneously, since size=3 on
+    #     3 OSDs means every drive absorbs 100% of cluster writes (see #434).
+    #
+    # This cluster is write-light (~117 GB/day at the RBD client layer), so
+    # throughput was never the binding constraint. Endurance is. If small-write
+    # latency ever becomes the real problem, fix it with more OSDs rather than
+    # by paying a 2x write tax on every 4 KiB page.
    osd:
      bdev_enable_discard: "true"
      bdev_async_discard: "true"
-      bluestore_prefer_deferred_size_ssd: "65536"
-      bluestore_deferred_batch_ops_ssd: "64"
+      # Ceph defaults, set explicitly so the previous override is actively
+      # overwritten rather than merely removed — `ceph config` retains a value
+      # that is only deleted from this file.
+      bluestore_prefer_deferred_size_ssd: "0"
+      bluestore_deferred_batch_ops_ssd: "16"
  dashboard:
    enabled: true
    ssl: false
@@ -5,8 +5,31 @@ metadata:
  namespace: rook-ceph
 spec:
  failureDomain: host
+  # size 2, NOT 3. With 3 OSDs on 3 hosts, size=3 meant every write landed on
+  # every drive in the cluster: per-drive write volume equalled total cluster
+  # write volume, and all three drives wore at an identical rate and would have
+  # reached end-of-rated-life together (measured 2026-08-19: 80/82/87% endurance
+  # used, ~0.13 points/day, all three exhausted between Nov 2026 and Jan 2027 —
+  # see #434). Replication does not protect against that failure mode, because
+  # wear-out here is correlated by construction rather than independent.
+  #
+  # size 2 cuts per-drive write volume by a third and desynchronises the wear.
+  #
+  # min_size 1 is REQUIRED and deliberate. Ceph does not lower min_size when
+  # size is reduced — it was 2 here, and leaving it would make the pool 2/2,
+  # where any single OSD down (every rolling node reboot, every OSD restart)
+  # blocks IO for its PGs and hangs every RBD volume in the cluster.
+  #
+  # The trade this accepts: while the pool is degraded (one OSD down), a second
+  # drive failing loses data, and with all three drives at 80%+ endurance that
+  # window is not hypothetical. Mitigations that already exist: CNPG barman and
+  # Velero both ship offsite to Hetzner (opentofu/object_storage.tf), so the
+  # exposure is the RPO of those backups, not total loss. Keep node reboots
+  # serialised and wait for HEALTH_OK between them.
  replicated:
-    size: 3
+    size: 2
+  parameters:
+    min_size: "1"
 ---
 apiVersion: storage.k8s.io/v1
 kind: StorageClass
@@ -64,13 +87,23 @@ metadata:
  name: cephfs
  namespace: rook-ceph
 spec:
+  # metadataPool STAYS at size 3 while the data pool drops to 2. It holds
+  # 467 MiB against the data pool's 275 GiB, so it contributes nothing
+  # measurable to drive wear — the entire reason for the size reduction — while
+  # losing it would destroy the whole filesystem, not just the objects on the
+  # lost OSD. Paying 3x on half a gigabyte to keep that intact is the right
+  # trade; the wear saving comes from data0.
  metadataPool:
    replicated:
      size: 3
  dataPools:
    - name: data0
+      # size 2 / min_size 1 for the same reasons as replicapool above — see
+      # the comment there for the full rationale and the accepted trade.
      replicated:
-        size: 3
+        size: 2
+      parameters:
+        min_size: "1"
  metadataServer:
    activeCount: 1
    activeStandby: true
@@ -1,122 +0,0 @@
-# TEMPORARY shims v3 — rustfs LXCs 800/801/802, decommission in progress.
-#
-# Stage 2 (#270) failed: both destroy attempts died instantly with
-# "context deadline exceeded" on the pre-destroy status GET, while the PVE
-# API answers the same call in <1s. Cause: the bpg provider reads
-# `timeout_delete` from STATE at destroy time, and importing a resource
-# whose config declares no timeouts stored 0 — a zero-second context.
-# (The import/plan read path uses its own timeout, which is why stage 1
-# planned clean.)
-#
-# Fix: declare the timeouts explicitly, apply ONCE so they land in state
-# (in-place no-API update), then delete this file again and the next apply
-# destroys the containers.
-#
-# v3: v2 declared ONLY the ids, `started`, `unprivileged` and the timeouts,
-# claiming "the other attributes mirror the real containers so the plan stays
-# in-place". They did not. Everything absent from config planned as a removal,
-# and two of those removals — `mount_point` and `operating_system` — force
-# replacement, so the plan came back `must be replaced` for all three. Applying
-# that would have destroyed and recreated containers tagged `critical`,
-# `state-backend`, `terraform-backend` instead of nudging a timeout into state.
-#
-# So the blocks below are not decoration: each one exists to keep an attribute
-# out of the diff. They mirror the live containers, read from the plan's own
-# state values. `tags` is declared for the same reason — without it the apply
-# strips the tags off the real containers.
-#
-#   1. merge; run PIPELINE=deploy-infrastructure (plan MUST show in-place
-#      update only, no replace), then TOFU_APPLY=true
-#   2. follow-up PR deletes this file; plan shows 3 destroys; apply.
-
-locals {
-  # Per-container values mirrored from live state. hostname and mac_address
-  # differ per container; declaring them wrong would itself force a diff.
-  rustfs_decommission = {
-    "800" = { node = "saturn", hostname = "rustfs-1.srv.renner.uno", mac = "BC:24:11:0B:E5:7B" }
-    "801" = { node = "jupiter", hostname = "rustfs-2.srv.renner.uno", mac = "BC:24:11:EA:1D:38" }
-    "802" = { node = "neptune", hostname = "rustfs-3.srv.renner.uno", mac = "BC:24:11:8F:F4:67" }
-  }
-}
-
-resource "proxmox_virtual_environment_container" "rustfs_decommission" {
-  for_each = local.rustfs_decommission
-
-  node_name     = each.value.node
-  vm_id         = tonumber(each.key)
-  started       = false
-  unprivileged  = true
-  start_on_boot = true
-
-  tags = [
-    "critical",
-    "rustfs",
-    "state-backend",
-    "terraform-backend",
-  ]
-
-  timeout_create = 1800
-  timeout_clone  = 1800
-  timeout_update = 1800
-  timeout_delete = 1800
-  timeout_start  = 300
-
-  cpu {
-    architecture = "amd64"
-    cores        = 1
-    units        = 1024
-  }
-
-  memory {
-    dedicated = 1024
-    swap      = 0
-  }
-
-  disk {
-    datastore_id = "local"
-    size         = 10
-  }
-
-  # forces replacement if absent — this is the block that broke v2.
-  mount_point {
-    path      = "/mnt/data"
-    size      = "50G"
-    volume    = "local:${each.key}/vm-${each.key}-disk-1.raw"
-    backup    = true
-    replicate = true
-  }
-
-  network_interface {
-    name        = "eth0"
-    bridge      = "v40srv"
-    enabled     = true
-    firewall    = false
-    mac_address = each.value.mac
-  }
-
-  initialization {
-    hostname = each.value.hostname
-
-    ip_config {
-      ipv4 {
-        address = "dhcp"
-      }
-    }
-  }
-
-  # operating_system is NOT declared, and must not be. The provider requires
-  # template_file_id inside it, which is a create-time-only value: PVE does not
-  # report the original template for an existing container, so it is absent
-  # from state and cannot be mirrored. Declaring the block without it fails
-  # validation ("Missing required argument"); declaring it with a guessed
-  # template would be a lie that only surfaces if something ever recreates
-  # these.
-  #
-  # Left undeclared it would plan as a removal, and removing operating_system
-  # forces replacement — the exact failure this file exists to avoid. So it is
-  # ignored instead. The timeouts above are deliberately NOT ignored: they are
-  # the one thing this shim needs to actually apply.
-  lifecycle {
-    ignore_changes = [operating_system]
-  }
-}
@@ -59,8 +59,25 @@ provider "unifi" {

 # UniFi Network v2 API (policy-table NAT rules). Same controller and API key
 # as the unifi provider above; the v2 endpoints are simply not covered by it.
-# UniFi PUT quirk: updates may require _id in the body, which restapi does
-# not send — if an in-place update 400s, taint the object so CI recreates it.
+# UniFi PUT quirk — DISPROVEN 2026-08-19, left here because it was wrong for
+# long enough to do damage. The claim was that a UniFi PUT may require _id in
+# the body, which restapi does not send, and that a 400 should be answered by
+# tainting the object so CI recreates it. Tested directly against a live rule
+# with exactly the body tofu sends:
+#
+#   PUT /proxy/network/v2/api/site/default/nat/6a61c7d7bbd7b2ca90981f51
+#   body: the object, _id removed          -> HTTP 200, object returned intact
+#   rule re-read and diffed before/after   -> byte-identical
+#   rule count                             -> 60 before, 60 after
+#
+# Confirmed again at scale by apply 5675: all 60 rules "Modifications complete
+# after 0s", no 400s, ci.renner.uno and immich.renner.uno both still 200.
+#
+# This mattered. A warning that applying might break firewall rules is a
+# warning nobody argues with, so the apply did not get run — and the
+# renner-uno-k8s-mariadb bucket added in b976a67b sat uncreated for three days
+# while the WordPress backup failed nightly with "The specified bucket does not
+# exist". Applying these NAT rules is safe. Do not taint them.
 #
 # create_returns_object, NOT write_returns_object. Both make the provider store
 # the API response into `data`; write_returns_object does it for PUT as well as
@@ -107,17 +107,51 @@ resource "restapi_object" "hairpin" {
  for_each = local.hairpin_rules

  path = "/nat"
+
+  # KNOWN PERMANENT DIFF — cosmetic. Do not try to fix it with a provider
+  # setting; four have been tried and none work. Recorded here so the next
+  # person does not spend a day rediscovering that.
+  #
+  # Every plan shows all 60 of these as "will be updated in-place" with the
+  # entire diff being one line:
+  #
+  #   ~ data = jsonencode( ~ { - _id = "6a61c7d7bbd7b2ca90981f51"
+  #                              # (15 unchanged attributes hidden) } )
+  #
+  # The UDM assigns _id and returns it; the provider puts the response into
+  # `data`; the `data` below never sets it. So state carries a field the config
+  # does not. It survives an apply: 5675 rewrote all 60, and the very next plan
+  # (5677) showed the same 60 again.
+  #
+  # Tried, with the plan that disproved each — do not repeat these:
+  #
+  #   ignore_changes_to = ["_id"]                    5661  60 remained
+  #   ignore_all_server_changes = true               5662  60 remained
+  #   restapi 3.x + ignore_server_additions = true   5667  60 remained, and
+  #                                                        added an api_data diff
+  #   write_returns_object -> create_returns_object  5677  60 remained
+  #
+  # The first three cannot work in principle: `data` is a configured attribute,
+  # so Terraform core diffs config against state before any provider-side ignore
+  # logic runs. The fourth was the better theory — that PUT responses were
+  # re-polluting `data` — and it is wrong too, which means the response is
+  # written into `data` on the READ path. Settling it properly needs the
+  # provider source, not more guesses at the schema.
+  #
+  # It is safe to apply through. Verified twice: a single PUT without _id
+  # returns 200 and leaves the rule byte-identical (see providers.tf), and
+  # apply 5675 modified all 60 with no errors and no service impact.
  data = jsonencode({
-    description  = "Hairpin ${each.key}"
-    type         = "DNAT"
-    enabled      = true
-    exclude      = false
-    in_interface = each.value.network_id
-    ip_address   = each.value.endpoint.target
-    ip_version   = each.value.endpoint.ip_version
-    protocol     = "tcp_udp"
-    rule_index   = each.value.rule_index
-    logging      = false
+    description              = "Hairpin ${each.key}"
+    type                     = "DNAT"
+    enabled                  = true
+    exclude                  = false
+    in_interface             = each.value.network_id
+    ip_address               = each.value.endpoint.target
+    ip_version               = each.value.endpoint.ip_version
+    protocol                 = "tcp_udp"
+    rule_index               = each.value.rule_index
+    logging                  = false
    is_predefined            = false
    pppoe_use_base_interface = false
    setting_preference       = "manual"