edge-1: keep logs off the eMMC (volatile journal + Alloy→Loki) + flash tuning #114

2026-07-07T06:51:41Z

k.renner commented

2026-07-07 06:51:41 +00:00

What & why

edge-1 turned out to be a Dell Wyse 3040 running on a 7.3 GB internal eMMC (root mmcblk0p2, ext4) — not the N100/NVMe box the role README assumed. Two things were churning that flash:

Docker json logs, unrotated — a single zwave-js log had grown to 214 MB.
Persistent journald (/var/log/journal), writing to eMMC by default.

Rather than cap those on disk, this PR keeps logs off local flash entirely and ships them to the aggregator:

journald → Storage=volatile (RAM /run/log/journal), size-capped (RuntimeMaxUse=128M); the on-disk store is purged on first apply.
Docker → journald log driver (/etc/docker/daemon.json) — no more container json files. Both compose services inherit it; docker logs still works.
Grafana Alloy (Arch grafana-alloy pkg) tails the RAM journal and pushes to Loki. Its WAL/positions live on /run/alloy (tmpfs), so the agent writes nothing to eMMC either.

Flash longevity: noatime on root, weekly fstrim.timer, and a weekly edge-emmc-health check of the JEDEC life-time / pre-EOL estimators (currently 0x01 0x01 = <10 % consumed).

Loki endpoint & the new credential

The unauthenticated srv-network Loki (loki.srv.renner.uno:3100) is firewalled off / not resolvable from the IoT VLAN (verified from edge-1: TCP blocked, no DNS). The only reachable endpoint is the authenticated cluster ingress https://loki.renner.uno (verified: returns 401). So Alloy pushes there with a dedicated edge-1 basic-auth user:

htpasswd line appended to kubernetes/apps/monitoring/loki-basic-auth.sops.yaml (existing krenner user untouched)
plaintext in the edge_host ansible vault (vault_edge_loki_password)

⚠️ Apply sequence (order matters)

Merge this PR → Flux reconciles the loki-basic-auth secret so the edge-1 htpasswd line reaches the Traefik middleware. Until this happens Alloy gets 401 and buffers in RAM — no delivery.
Run the play against edge-1:
```
ansible-playbook -i inventory/pve_hosts.yml plays/install-edge-host.yml
```
This restarts dockerd (brief bounce of zwave-js + sml2mqtt), restarts journald, remounts / with noatime, and starts edge-alloy.
Confirm logs land in Loki ({host="edge-1.iot.renner.uno"}) and journalctl shows the journal is volatile.

Design note

The Alloy config is purpose-built inside the edge-host role rather than reusing the shared monitoring-agent role, because that role is a poor fit for this host: it's Debian-oriented (apt + GitHub-binary download), force-enables a docker log source that would double-ship against the journald driver, and always configures a Prometheus remote-write whose endpoint (prometheus.srv.renner.uno) is unreachable from the IoT VLAN. Adapting it would have meant 4+ guarded edits to shared fleet infra. Happy to switch to the shared role if you'd prefer — this is the one call worth a second opinion.

Verification done

ansible-playbook --syntax-check clean; ansible.posix 2.2.0 present (mount module); bash -n clean on the wear-check script.
sops + vault round-trip verified (both krenner and edge-1 present; new vault key decrypts).
Endpoint reachability probed live from edge-1.
Not applied to edge-1 (delivery is gated on the merge above, and applying restarts heat-pump-adjacent services — left for the deliberate step 2).

## What & why `edge-1` turned out to be a **Dell Wyse 3040 running on a 7.3 GB internal eMMC** (root `mmcblk0p2`, ext4) — not the N100/NVMe box the role README assumed. Two things were churning that flash: - **Docker json logs**, unrotated — a single `zwave-js` log had grown to **214 MB**. - **Persistent journald** (`/var/log/journal`), writing to eMMC by default. Rather than *cap* those on disk, this PR keeps logs **off local flash entirely** and ships them to the aggregator: - **journald → `Storage=volatile`** (RAM `/run/log/journal`), size-capped (`RuntimeMaxUse=128M`); the on-disk store is purged on first apply. - **Docker → `journald` log driver** (`/etc/docker/daemon.json`) — no more container json files. Both compose services inherit it; `docker logs` still works. - **Grafana Alloy** (Arch `grafana-alloy` pkg) tails the RAM journal and pushes to Loki. Its WAL/positions live on `/run/alloy` (tmpfs), so the agent writes nothing to eMMC either. Flash longevity: **`noatime`** on root, weekly **`fstrim.timer`**, and a weekly **`edge-emmc-health`** check of the JEDEC life-time / pre-EOL estimators (currently `0x01 0x01` = <10 % consumed). ## Loki endpoint & the new credential The unauthenticated srv-network Loki (`loki.srv.renner.uno:3100`) is **firewalled off / not resolvable from the IoT VLAN** (verified from edge-1: TCP blocked, no DNS). The only reachable endpoint is the **authenticated cluster ingress** `https://loki.renner.uno` (verified: returns 401). So Alloy pushes there with a **dedicated `edge-1` basic-auth user**: - htpasswd line appended to `kubernetes/apps/monitoring/loki-basic-auth.sops.yaml` (existing `krenner` user untouched) - plaintext in the `edge_host` ansible vault (`vault_edge_loki_password`) ## ⚠️ Apply sequence (order matters) 1. **Merge this PR** → Flux reconciles the `loki-basic-auth` secret so the `edge-1` htpasswd line reaches the Traefik middleware. *Until this happens Alloy gets 401 and buffers in RAM — no delivery.* 2. Run the play against edge-1: ``` ansible-playbook -i inventory/pve_hosts.yml plays/install-edge-host.yml ``` This **restarts dockerd** (brief bounce of `zwave-js` + `sml2mqtt`), restarts journald, remounts `/` with noatime, and starts `edge-alloy`. 3. Confirm logs land in Loki (`{host="edge-1.iot.renner.uno"}`) and `journalctl` shows the journal is volatile. ## Design note The Alloy config is **purpose-built inside the edge-host role** rather than reusing the shared `monitoring-agent` role, because that role is a poor fit for this host: it's Debian-oriented (apt + GitHub-binary download), force-enables a docker log source that would **double-ship** against the journald driver, and always configures a Prometheus remote-write whose endpoint (`prometheus.srv.renner.uno`) is **unreachable from the IoT VLAN**. Adapting it would have meant 4+ guarded edits to shared fleet infra. Happy to switch to the shared role if you'd prefer — this is the one call worth a second opinion. ## Verification done - `ansible-playbook --syntax-check` clean; `ansible.posix` 2.2.0 present (mount module); `bash -n` clean on the wear-check script. - sops + vault round-trip verified (both `krenner` and `edge-1` present; new vault key decrypts). - Endpoint reachability probed live from edge-1. - **Not** applied to edge-1 (delivery is gated on the merge above, and applying restarts heat-pump-adjacent services — left for the deliberate step 2).

k.renner added 1 commit 2026-07-07 06:51:41 +00:00

edge-1: keep logs off the eMMC (volatile journal + Alloy->Loki) + flash tuning 3ae01e9f6d

edge-1 is a Dell Wyse 3040 on a 7.3 GB internal eMMC (not the N100/NVMe the
README assumed). Unbounded docker json logs (a single zwave-js log had grown
to 214 MB) plus a persistent journald were churning the flash. Move all
logging off local flash and ship it to Loki instead:

- journald Storage=volatile (RAM), size-capped; purge the on-disk store
- docker json-file -> journald log driver (no container json logs on disk)
- Grafana Alloy (Arch pkg) tails the RAM journal -> Loki, WAL on /run tmpfs

Loki endpoint: the unauthenticated srv-network Loki is firewalled off the IoT
VLAN, so Alloy pushes to the authenticated cluster ingress
(https://loki.renner.uno) with a dedicated 'edge-1' basic-auth user — its
htpasswd line is added to the loki-basic-auth cluster secret, plaintext in the
edge_host ansible vault. Delivery activates once Flux reconciles that secret;
until then Alloy 401s and buffers in RAM.

Flash longevity: noatime on root, weekly fstrim.timer, and a weekly
edge-emmc-health check of the JEDEC life-time / pre-EOL estimators.

The Alloy config is purpose-built in the edge-host role rather than reusing
the shared monitoring-agent role: that role is Debian-oriented (apt + GitHub
binary), force-enables a docker log source that would double-ship against the
journald driver, and always configures a Prometheus remote-write whose
endpoint is unreachable from the IoT VLAN.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

k.renner merged commit 04460bb97d into master

2026-07-07 06:55:16 +00:00

k.renner referenced this issue from a commit

2026-07-07 06:55:16 +00:00

Merge pull request 'edge-1: keep logs off the eMMC (volatile journal + Alloy→Loki) + flash tuning' (#114) from feat/edge-1-emmc-wear-mitigation into master

k.renner referenced this issue from a commit

2026-07-07 07:08:28 +00:00

edge-host: fix edge-alloy ExecStart — Arch binary is grafana-alloy

k.renner referenced this pull request

2026-07-07 07:08:37 +00:00

edge-host: fix edge-alloy ExecStart (Arch binary is grafana-alloy) #116

k.renner referenced this pull request