Compare commits

..
Author SHA1 Message Date
issue-fix-bot b1901047a0 fix: [alert] CephHealthWarn (rook-ceph) (#562)
ci/woodpecker/pr/ci-manifests-test Pipeline is pending
ci/woodpecker/pr/ci-scripts-test Pipeline is pending
ci/woodpecker/pr/pr-review Pipeline is pending
2026-08-24 12:47:12 +00:00
3 changed files with 21 additions and 177 deletions
@@ -179,32 +179,16 @@ data:
# Ecowitt WS90 weather station. Joined after the ZHA-mirrored block above
# was written, hence listed apart from it.
#
# This device also carries a non-default ZCL reporting configuration
# (per-attribute min interval and reportable-change deadbands). That is
# device-side state, not expressible here — Z2M has no settings field for
# it — so it is documented, with restore commands, in
# zigbee2mqtt-ws90-reporting.md. It is lost whenever Z2M re-runs
# `configure` on the device, and a fresh PVC will not have it.
#
# The station transmits every 8.8s, as a burst of seven per-cluster ZCL
# frames; Z2M publishes its full cumulative state once per frame. debounce
# merges a burst into one publish carrying the last value per attribute,
# and must stay below the transmit interval — a window that never sees
# silence never flushes.
#
# Every few days the firmware zeroes its readings for ~8s, then skips a
# transmit cycle and recovers on the next. The zeros propagate attribute
# by attribute through a burst rather than arriving together, and they
# start inside an otherwise normal cycle — the last valid reading and the
# first zero are ~20ms apart.
#
# Merging cuts an episode from six published bad frames to one. It does
# NOT make that frame recognisable: it carries whatever subset had zeroed
# when the burst ended, which may be every field or only temperature, and
# a frame reading {"temperature":0,"humidity":92,"pressure":96.9} is not
# separable from a valid winter reading. Since the merge is last-wins and
# the zeros arrive at the tail, it also discards the valid reading that
# led the burst. This bounds the corruption; it does not filter it.
# The firmware drops all readings to zero for a few seconds every few
# days. Merged, that surfaces as a single all-zero frame; unmerged, the
# zeros spread across the seven frames and none of them is individually
# distinguishable from a valid reading.
#
# Z2M reads this only at startup (requiresRestart), and this file is
# SEED-ONLY per the header — the running instance takes it from its PVC.
@@ -1,149 +0,0 @@
# WS90 Zigbee reporting configuration
Records the ZCL attribute-reporting configuration applied to the Wetterstation
(Ecowitt WS90 powered by Shelly, IEEE `0xc02cedfffe958f40`) and how to restore
it, because **this configuration lives on the device and in Z2M's
`database.db`, not in git**. Zigbee2MQTT has no settings-schema field for
reporting configuration, so there is nothing declarative to reconcile — Flux
cannot restore it and a fresh PVC will not have it.
## Applied configuration
Endpoint 1, all intervals in seconds. `change` is in the attribute's raw ZCL
units; the physical deadband is `change ÷ scale`.
| attribute | cluster | min | max | change | deadband | converter default |
| --- | --- | --- | --- | --- | --- | --- |
| temperature | `msTemperatureMeasurement` (1026) | 5 | 3600 | 10 | 0.1 °C | 10 / 3600 / 100 |
| humidity | `msRelativeHumidity` (1029) | 5 | 3600 | 50 | 0.5 % | 10 / 3600 / 100 |
| pressure | `msPressureMeasurement` (1027) | 5 | 3600 | 1 | 1 hPa | 10 / 3600 / 50 |
| illuminance | `msIlluminanceMeasurement` (1024) | 5 | 3600 | 5 | 5 lx | 10 / 3600 / 5 |
| windSpeed | `shellyWS90Wind` (0xfc01/0) | 5 | 3600 | 1 | — | 10 / 3600 / 1 |
| windDirection | `shellyWS90Wind` (0xfc01/4) | 5 | 3600 | 1 | — | 10 / 3600 / 1 |
| gustSpeed | `shellyWS90Wind` (0xfc01/7) | 5 | 3600 | 1 | — | 10 / 3600 / 1 |
| uvIndex | `shellyWS90UV` (0xfc02/0) | 5 | 3600 | 1 | — | 10 / 3600 / 1 |
| rainStatus | `shellyWS90Rain` (0xfc03/0) | 5 | 3600 | 1 | — | 10 / 3600 / 1 |
| precipitation | `shellyWS90Rain` (0xfc03/1) | 5 | 3600 | 1 | — | 10 / 3600 / 1 |
`genPowerCfg.batteryVoltage` (cluster 1, attribute 33) is deliberately left at
its hourly default.
The three `shellyWS90*` clusters are manufacturer-specific and require
`"options": {"manufacturerCode": 5264}` on every reconfiguration request.
## Why min is 5 and not 10
The station transmits every 8.8 s. `min` is a hold-off: a change-triggered
report is held until at least `min` seconds after the previous one. With
`min: 10` every report was therefore held past a transmit cycle and landed on
the next, so attributes reported at best every 17.6 s and drifted out of phase
with each other — the published stream showed no 8.8 s periodicity at all,
just a smear across 2–8 s.
At `min: 5` each attribute reports on its own transmit. The MQTT stream then
carries one publish per transmit cycle, measured at a mean gap of 8.69 s.
Total publish volume *fell* (~631/h to ~445/h) because each publish now
carries a whole cycle instead of a fragment of one.
Any value below 8.8 s achieves this. 5 leaves margin without approaching the
cycle time.
## Why the deadbands differ from the converter defaults
The defaults come from the shared `modernExtend` helpers in
zigbee-herdsman-converters (`m.temperature()`, `m.pressure()`, `m.humidity()`,
`m.illuminance()`), which the WS90 definition in `shelly.ts` composes without
overrides. They are anti-chatter values sized for battery-powered indoor
sensors, where report volume costs coin-cell life and mesh airtime. This
station is mains-and-solar and exists to resolve small changes, so the
trade-off runs the other way.
Each value above is matched to the resolution the device actually reports:
temperature and pressure arrive in 0.1-unit steps, humidity in whole percent.
Pressure is worth calling out. The converter default of `change: 50` with
`scale: 10` is a 5 kPa (50 hPa) deadband, which real barometric pressure never
traverses — so pressure only ever updated on the hourly `max` heartbeat, and
`pressure_trend` and `weather_condition` were derived from that. The same
library's older `reporting.ts` helper configures the identical ZCL attribute
with `change: 5`, a 5 hPa deadband, so the two code paths disagree by 10× and
the modernExtend value is the outlier.
## When this is lost
Anything that makes Z2M re-run `configure` on the device overwrites all of it
with the converter defaults:
- re-pairing or re-joining the device
- an explicit `zigbee2mqtt/bridge/request/device/configure`
- a converter update that changes the definition and triggers reconfiguration
It also will not exist on a fresh PVC. Re-apply with the commands below.
## Re-applying
Published from the `discovery-guard` sidecar, which already holds the
`zigbee2mqtt` broker credential (ACL `readwrite zigbee2mqtt/#`, which covers
`bridge/request/#`) so no credential has to leave the cluster:
Ten separate requests, one per attribute — written out literally rather than
generated by a shell helper, because the nested quoting through `kubectl exec
-- sh -c` is where this goes wrong. Each line below is in the exact form that
was executed and verified.
```sh
POD=$(kubectl get pod -n home-assistant -l app.kubernetes.io/name=zigbee2mqtt -o jsonpath='{.items[0].metadata.name}')
```
```sh
kubectl exec -n home-assistant "$POD" -c discovery-guard -- sh -c 'mosquitto_pub -h "$MQTT_HOST" -p 1883 -u "$MQTT_USER" -P "$MQTT_PASSWORD" -t "$BASE_TOPIC/bridge/request/device/configure_reporting" -m "{\"id\":\"0xc02cedfffe958f40\",\"endpoint\":1,\"cluster\":\"msTemperatureMeasurement\",\"attribute\":\"measuredValue\",\"minimum_report_interval\":5,\"maximum_report_interval\":3600,\"reportable_change\":10}"'
kubectl exec -n home-assistant "$POD" -c discovery-guard -- sh -c 'mosquitto_pub -h "$MQTT_HOST" -p 1883 -u "$MQTT_USER" -P "$MQTT_PASSWORD" -t "$BASE_TOPIC/bridge/request/device/configure_reporting" -m "{\"id\":\"0xc02cedfffe958f40\",\"endpoint\":1,\"cluster\":\"msRelativeHumidity\",\"attribute\":\"measuredValue\",\"minimum_report_interval\":5,\"maximum_report_interval\":3600,\"reportable_change\":50}"'
kubectl exec -n home-assistant "$POD" -c discovery-guard -- sh -c 'mosquitto_pub -h "$MQTT_HOST" -p 1883 -u "$MQTT_USER" -P "$MQTT_PASSWORD" -t "$BASE_TOPIC/bridge/request/device/configure_reporting" -m "{\"id\":\"0xc02cedfffe958f40\",\"endpoint\":1,\"cluster\":\"msPressureMeasurement\",\"attribute\":\"measuredValue\",\"minimum_report_interval\":5,\"maximum_report_interval\":3600,\"reportable_change\":1}"'
kubectl exec -n home-assistant "$POD" -c discovery-guard -- sh -c 'mosquitto_pub -h "$MQTT_HOST" -p 1883 -u "$MQTT_USER" -P "$MQTT_PASSWORD" -t "$BASE_TOPIC/bridge/request/device/configure_reporting" -m "{\"id\":\"0xc02cedfffe958f40\",\"endpoint\":1,\"cluster\":\"msIlluminanceMeasurement\",\"attribute\":\"measuredValue\",\"minimum_report_interval\":5,\"maximum_report_interval\":3600,\"reportable_change\":5}"'
kubectl exec -n home-assistant "$POD" -c discovery-guard -- sh -c 'mosquitto_pub -h "$MQTT_HOST" -p 1883 -u "$MQTT_USER" -P "$MQTT_PASSWORD" -t "$BASE_TOPIC/bridge/request/device/configure_reporting" -m "{\"id\":\"0xc02cedfffe958f40\",\"endpoint\":1,\"cluster\":\"shellyWS90Wind\",\"attribute\":\"windSpeed\",\"minimum_report_interval\":5,\"maximum_report_interval\":3600,\"reportable_change\":1,\"options\":{\"manufacturerCode\":5264}}"'
kubectl exec -n home-assistant "$POD" -c discovery-guard -- sh -c 'mosquitto_pub -h "$MQTT_HOST" -p 1883 -u "$MQTT_USER" -P "$MQTT_PASSWORD" -t "$BASE_TOPIC/bridge/request/device/configure_reporting" -m "{\"id\":\"0xc02cedfffe958f40\",\"endpoint\":1,\"cluster\":\"shellyWS90Wind\",\"attribute\":\"windDirection\",\"minimum_report_interval\":5,\"maximum_report_interval\":3600,\"reportable_change\":1,\"options\":{\"manufacturerCode\":5264}}"'
kubectl exec -n home-assistant "$POD" -c discovery-guard -- sh -c 'mosquitto_pub -h "$MQTT_HOST" -p 1883 -u "$MQTT_USER" -P "$MQTT_PASSWORD" -t "$BASE_TOPIC/bridge/request/device/configure_reporting" -m "{\"id\":\"0xc02cedfffe958f40\",\"endpoint\":1,\"cluster\":\"shellyWS90Wind\",\"attribute\":\"gustSpeed\",\"minimum_report_interval\":5,\"maximum_report_interval\":3600,\"reportable_change\":1,\"options\":{\"manufacturerCode\":5264}}"'
kubectl exec -n home-assistant "$POD" -c discovery-guard -- sh -c 'mosquitto_pub -h "$MQTT_HOST" -p 1883 -u "$MQTT_USER" -P "$MQTT_PASSWORD" -t "$BASE_TOPIC/bridge/request/device/configure_reporting" -m "{\"id\":\"0xc02cedfffe958f40\",\"endpoint\":1,\"cluster\":\"shellyWS90UV\",\"attribute\":\"uvIndex\",\"minimum_report_interval\":5,\"maximum_report_interval\":3600,\"reportable_change\":1,\"options\":{\"manufacturerCode\":5264}}"'
kubectl exec -n home-assistant "$POD" -c discovery-guard -- sh -c 'mosquitto_pub -h "$MQTT_HOST" -p 1883 -u "$MQTT_USER" -P "$MQTT_PASSWORD" -t "$BASE_TOPIC/bridge/request/device/configure_reporting" -m "{\"id\":\"0xc02cedfffe958f40\",\"endpoint\":1,\"cluster\":\"shellyWS90Rain\",\"attribute\":\"rainStatus\",\"minimum_report_interval\":5,\"maximum_report_interval\":3600,\"reportable_change\":1,\"options\":{\"manufacturerCode\":5264}}"'
kubectl exec -n home-assistant "$POD" -c discovery-guard -- sh -c 'mosquitto_pub -h "$MQTT_HOST" -p 1883 -u "$MQTT_USER" -P "$MQTT_PASSWORD" -t "$BASE_TOPIC/bridge/request/device/configure_reporting" -m "{\"id\":\"0xc02cedfffe958f40\",\"endpoint\":1,\"cluster\":\"shellyWS90Rain\",\"attribute\":\"precipitation\",\"minimum_report_interval\":5,\"maximum_report_interval\":3600,\"reportable_change\":1,\"options\":{\"manufacturerCode\":5264}}"'
```
These are idempotent — re-sending a request that matches the current
configuration is a no-op on the device.
The device must be awake and reachable; each request performs a real
`ConfigureReporting` exchange and fails if it is not.
## Verifying
Read back what the device actually accepted — Z2M records the negotiated
result, so this reflects the device, not the request:
```sh
kubectl exec -n home-assistant "$POD" -c zigbee2mqtt -- \
grep 0xc02cedfffe958f40 /app/data/database.db |
jq -r '.endpoints | to_entries[] | .value.configuredReportings[]?
| "cluster=\(.cluster) attr=\(.attrId) min=\(.minRepIntval) max=\(.maxRepIntval) change=\(.repChange)"'
```
All ten rows above should read `min=5 max=3600`. On the wire, the publish
cadence on `zigbee2mqtt/Wetterstation` should sit at ~8.7 s and temperature
should move in 0.1 °C steps; 1 °C steps mean the defaults are back.
## Making it durable upstream
The lasting fix is for the WS90 definition to pass explicit `reporting`
overrides rather than inherit indoor-sensor defaults, which would fix every
WS90 rather than this one. `m.temperature()` and the other helpers already
accept a `reporting` argument. The pressure divergence between
`modernExtend.ts` and `reporting.ts` is a separate upstream bug worth
reporting on its own.
@@ -31,21 +31,30 @@ spec:
# bluestore_slow_ops_warn_lifetime (24h) after the LAST slow op, so
# one multi-second stall outlives the 6h `for` and always pages,
# with latency back to normal long before anyone looks. Measured:
# both episodes in the 30d to 2026-08-23 ran ~26h and cleared
# untouched, the first ~24h after the last stall.
# three episodes in the 30d to 2026-08-24 (Sun Jul 26, Aug 2 and
# Aug 23), each ran ~24-26h and cleared untouched.
#
# Both began inside the weekly Velero restore-test window (Sun
# 08:00-13:00Z) — a 5h sustained restore that outruns the SLC cache
# on these QLC NVMe. Expected for the hardware, but not excludable:
# this is also the earliest signal of a genuinely failing drive, and
# all three are at 80-87% endurance used (see cephcluster.yaml).
# All began inside the weekly Velero restore-test window (Sun 08:00Z
# plus up to 6h, see velero/restore-test-cronjob.yaml) — a sustained
# restore that outruns the SLC cache on these QLC NVMe. Expected for
# the hardware, but not excludable: this is also the earliest signal
# of a genuinely failing drive, and all three are at 80-87%
# endurance used (see cephcluster.yaml).
#
# No metric names the stalling OSD: ceph_health_detail carries no osd
# label, and ceph_daemon_health_metrics{type="SLOW_OPS"} reads 0 at
# every scrape (a stall is shorter than the scrape interval). Only
# the OSD's own log has it, hence the Loki query in the description.
# Jul 26 and Aug 2 hit osd.0+1+2 together, which is load-driven;
# Aug 23 hit osd.1 alone in one burst. If osd.1 alone repeats,
# suspect that drive rather than the restore load.
expr: ceph_health_detail{name!~"AUTH_INSECURE_.+"} == 1
for: 6h
labels:
severity: warning
annotations:
summary: "Ceph health check {{ $labels.name }} active for 6h"
description: "Ceph health check {{ $labels.name }} has been active for six hours. Only AUTH_INSECURE_* is excluded (kernel-blocked, see #527). BLUESTORE_SLOW_OP_ALERT is sticky for 24h after the last stall and clears itself: check whether the stall fell inside the Sunday 08:00-13:00Z Velero restore-test window, which is expected on these QLC drives — one outside that window, or a repeat on the same OSD, means check that drive (#557). Detail: `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail`."
description: "Ceph health check {{ $labels.name }} has been active for six hours. Only AUTH_INSECURE_* is excluded (kernel-blocked, see #527). BLUESTORE_SLOW_OP_ALERT is sticky for 24h after the last stall and clears itself: check whether the stall fell inside the Sunday 08:00-14:00Z Velero restore-test window (08:00 start, 6h activeDeadlineSeconds cap), which is expected on these QLC drives — one outside that window, or a repeat on the same OSD, means check that drive (#557). No metric names the OSD — get it from Loki, which also works after the check clears: `{namespace=\"rook-ceph\",pod=~\"rook-ceph-osd-.*\"} |= \"slow operation observed\"`. While still active, `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail` adds the op detail."
- alert: CephPoolLostRedundancy
# Names the pool, which POOL_NO_REDUNDANCY does not.