edge-host: batch zwave-js fsync via libeatmydata (+ lazytime) to spare the eMMC #120

2026-07-07T09:27:29Z

k.renner commented

2026-07-07 09:27:29 +00:00

Cuts the dominant residual eMMC writer on edge-1. With logs already off flash (#114), zwave-js-ui is what's left: its value DB is fsync'd on every value change (jsonl-db; zwave-js-ui doesn't expose the driver's throttle). Measured: ~2.6 GiB/day of eMMC writes for ~0.1 KiB/s of actual new data — near-pure write amplification (each fsync forces a block write + an ext4 journal commit).

Fix

Preload libeatmydata into the zwave-js container so fsync/fdatasync become no-ops → the kernel batches writes via normal writeback (~30 s) instead of forcing each one.

The value stream to HA (WS/MQTT) and the on-disk cache are unchanged — full data resolution, no re-interview on restart. (This is why libeatmydata, not tmpfs or a mount option: fsync is a durability contract the fs must honor; commit=/data=writeback/nobarrier don't batch it, and tmpfs would make the whole store volatile.)
Verified before building: jsonl-db fsyncs per change (source); the container is Alpine 3.22/musl; LD_PRELOAD alone neutralizes fsync (strace: 1 → 0); the musl .so is extractable via --network=host on edge-1.
The .so is musl, extracted from edge_zwave_alpine_base (must match the image's Alpine version; refresh by deleting the .so on a major bump).

Also

noatime → noatime,lazytime on root — batches inode-timestamp writeback (e.g. the zwave-js lockfile heartbeat) with no data-durability impact.
edge-zwave-settings-backup.timer — hourly validated rolling backup of settings.json (Z-Wave network keys), since neutralized fsync means a power-loss mid settings-write could truncate it. Value-cache loss on crash is harmless (devices re-report; jsonl-db tolerates a partial trailing line).

Rollout

Single target (edge-1); applying recreates the zwave-js container (brief bounce). Reversible: drop the LD_PRELOAD env + .so mount from the compose file and re-up. The running eMMC-write monitor will quantify the drop (I over-claimed a specific multiple earlier — will report the measured number after apply).

ansible-playbook --syntax-check clean; backup script bash -n clean.

Cuts the dominant residual eMMC writer on edge-1. With logs already off flash (#114), zwave-js-ui is what's left: its value DB is `fsync`'d on **every** value change (jsonl-db; zwave-js-ui doesn't expose the driver's throttle). Measured: **~2.6 GiB/day of eMMC writes for ~0.1 KiB/s of actual new data** — near-pure write amplification (each fsync forces a block write + an ext4 journal commit). ## Fix Preload **libeatmydata** into the `zwave-js` container so `fsync`/`fdatasync` become no-ops → the kernel batches writes via normal writeback (~30 s) instead of forcing each one. - The value stream to HA (WS/MQTT) and the on-disk cache are **unchanged** — full data resolution, **no re-interview on restart**. (This is why libeatmydata, not tmpfs or a mount option: `fsync` is a durability contract the fs must honor; `commit=`/`data=writeback`/`nobarrier` don't batch it, and tmpfs would make the whole store volatile.) - Verified before building: jsonl-db fsyncs per change (source); the container is Alpine 3.22/musl; `LD_PRELOAD` alone neutralizes fsync (strace: 1 → 0); the musl `.so` is extractable via `--network=host` on edge-1. - The `.so` is musl, extracted from `edge_zwave_alpine_base` (must match the image's Alpine version; refresh by deleting the `.so` on a major bump). ## Also - **`noatime` → `noatime,lazytime`** on root — batches inode-timestamp writeback (e.g. the zwave-js lockfile heartbeat) with no data-durability impact. - **`edge-zwave-settings-backup.timer`** — hourly validated rolling backup of `settings.json` (Z-Wave **network keys**), since neutralized fsync means a power-loss mid settings-write could truncate it. Value-cache loss on crash is harmless (devices re-report; jsonl-db tolerates a partial trailing line). ## Rollout Single target (edge-1); applying recreates the `zwave-js` container (brief bounce). **Reversible**: drop the `LD_PRELOAD` env + `.so` mount from the compose file and re-up. The running eMMC-write monitor will quantify the drop (I over-claimed a specific multiple earlier — will report the measured number after apply). `ansible-playbook --syntax-check` clean; backup script `bash -n` clean.

k.renner added 1 commit 2026-07-07 09:27:29 +00:00

edge-host: batch zwave-js fsync via libeatmydata (+ lazytime) to spare eMMC 9add6f1a56

With logs off flash, the dominant remaining eMMC writer is zwave-js-ui: its
value DB is fsync'd on every value change (jsonl-db; zwave-js-ui doesn't
expose the driver throttle). On a busy network that fsync storm churned
~2.6 GiB/day for ~0.1 KiB/s of real data — near-pure write amplification
(each fsync forces a block write + ext4 journal commit).

Preload libeatmydata into the zwave-js container (musl .so extracted from the
image's Alpine base, mounted ro, LD_PRELOAD) so fsync/fdatasync become no-ops
and the kernel batches writes via normal writeback. The WS/MQTT value stream
to HA and the on-disk cache are unchanged — full data resolution, no
re-interview on restart. No ext4 mount option can do this (fsync is a
durability contract; commit=/data=writeback/nobarrier don't batch it, and the
only mount-level alternative — tmpfs — makes the whole store volatile).

Also:
- noatime -> noatime,lazytime on root: batches the inode-timestamp writeback
  (e.g. the zwave-js lockfile heartbeat) without weakening data durability.
- edge-zwave-settings-backup.timer: hourly validated rolling backup of
  settings.json (Z-Wave network keys), since neutralized fsync means a
  power-loss mid settings-write could truncate it. Value-cache loss on crash
  is harmless (devices re-report; jsonl-db tolerates a partial trailing line).

Verify the drop on the eMMC-write monitor; rollback = drop the LD_PRELOAD env
+ .so mount from the compose file.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

k.renner merged commit c2f1aad8f7 into master

2026-07-07 09:30:37 +00:00

k.renner referenced this issue from a commit

2026-07-07 09:30:37 +00:00

Merge pull request 'edge-host: batch zwave-js fsync via libeatmydata (+ lazytime) to spare the eMMC' (#120) from feat/edge-zwave-fsync-batching into master

k.renner referenced this pull request

2026-07-07 09:39:33 +00:00

edge-host: fix settings-backup terminator check (trailing newline) #121

k.renner referenced this issue from a commit

2026-07-07 09:39:33 +00:00

edge-host: fix settings-backup terminator check (trailing newline)

Sign in to join this conversation.