homelab/ansible/docs/edge-host.md

edge-host — bare-metal Arch thin client (replaces the k8s-edge Talos VM)

Runs the hardware-bound services that used to be pinned to the k8s-edge Talos node (opentofu vm_153_k8s_edge.tf):

Service	How it runs	Hardware	Talks to
zwave-js-ui	docker compose	ZWA-2 Z-Wave USB radio	HA zwave_js via ws://:3000, UI :8091
sml2mqtt	docker compose	FTDI FT230X meter reader	Mosquitto (sml2mqtt user)

The Viessmann E3 CAN bus is not read by an edge host: the ESP32 gateway vitocal-can (open3e-esp32) at the Vitocal publishes open3e/# directly to the cluster broker. plays/install-edge-host.yml removes the former heating-can stack and the can0 configuration from edge hosts.

Everything else that once lived on k8s-edge (HA core, matter-server, OTBR, music-assistant) already moved to the general node pool and stays in the cluster — they need VLAN L2 presence, not USB, so a single LAN NIC is sufficient.

The deployed unit is a Dell Wyse 3040 (Atom x5-Z8350, 2 GB RAM) on a ~7.3 GB internal eMMC (/dev/mmcblk0) — see "Logging & eMMC" for the flash-wear handling that hardware implies. The installer also supports an N100/NVMe box; it autodetects the target disk, so either works.

Base install (once, from the edge-host installer ISO)

Build and boot the dedicated installer image — see images/edge-host-iso/README.md. Short version:

1. sudo pacman -S archiso && sudo images/edge-host-iso/build.sh
2. Write ISO to a USB stick, boot the target box from it, run edge-install <hostname> (partitions the internal disk — NVMe if present, else the eMMC; the installer autodetects — installs the ansible-ready base system with SSH keys, DHCP, docker — asks for confirmation before wiping).
3. Confirm the box resolves and is reachable by name — ssh root@<host> must work before anything else. The UDM registers DHCP leases in DNS, so the record appears on its own; the host is identified by name everywhere (inventory, plays, this file) and its address is never recorded. Do not pin it to a reservation.
4. Add the host to [edge_host] in inventory/pve_hosts.yml, by name. That group is the base: every edge thin client belongs to it, and it assumes no peripherals.
5. From the workstation, in ansible/:

   ansible-playbook site.yml
   

   One unrestricted run, no --limit. It converges every host, which is the point: a run that touches only the new box tells you nothing about the others.

Peripherals

Each attached device is its own inventory group and its own playbook. Group membership states which hardware a host physically has -- there are no flags and no when: on the roles, so moving a stick between boxes is a one-line inventory change.

device	group	playbook
Nabu Casa ZWA-2 Z-Wave radio	zwave_gateway	plays/install-zwave-gateway.yml

Add the host to the matching group, then run that playbook. It depends on the base play having run: it needs docker.

A host with nothing attached is simply absent from these groups and needs none of these commands.

Secrets (before first play run)

Create inventory/group_vars/edge_host/vault.yml with ansible-vault. The zwave-js value comes out of the cluster:

# zwave-js session secret
kubectl -n home-assistant get secret zwave-js-session -o jsonpath='{.data.session-secret}' | base64 -d

# vault.yml
vault_edge_zwavejs_session_secret: "..."
vault_edge_loki_password: "..."   # dedicated edge-1 Loki push cred (see "Logging & eMMC")

vault_edge_loki_password is a freshly-minted secret, not copied from the cluster: a dedicated edge-1 htpasswd user whose hash lives in the cluster secret kubernetes/apps/monitoring/loki-basic-auth.sops.yaml. To rotate, regenerate openssl rand -hex 24, update both the htpasswd line in that sops secret and this vault key, then re-run the play.

The Z-Wave network keys do NOT need separate handling — they live in settings.json inside the store that gets migrated below.

Cutover checklist (service downtime: minutes, do per-service)

Preparation:

• vault.yml populated (above), play runs green against edge-1

zwave-js (order matters — never two servers on one radio):

• Copy the store out of the PVC while the pod still runs: kubectl -n home-assistant exec deploy/zwave-js -- tar cf - -C /usr/src/app store | tar xf - -C /opt/edge-host/zwave-js/
• Scale down in-cluster: kubectl -n home-assistant scale deploy/zwave-js --replicas 0
• Move the ZWA-2 stick to edge-1, docker compose up -d zwave-js
• Point HA zwave_js integration at ws://edge-1.iot.renner.uno:3000
• Nodes alive in the UI on :8091 → remove the flux kustomization home-assistant-zwavejs + PVC in a follow-up PR

sml2mqtt:

• Scale down in-cluster: kubectl -n home-assistant scale deploy/sml2mqtt --replicas 0
• Move the FTDI stick, docker compose up -d sml2mqtt
• HA meter entities update again (same entity_ids — config is a 1:1 port, Wh stays Wh) → remove sml2mqtt from the home-assistant kustomization in the follow-up PR

Afterwards:

• Retire VM 153: remove vm_153_k8s_edge.tf, talos/patches/k8s-edge.yaml, node from the cluster (kubectl delete node k8s-edge, tofu apply)
• Update wiki (k8s-edge-node-pattern note is then historical)

Day-2 notes

• Image bumps: Renovate opens a PR for edge_*_image in roles/*/defaults/main.yml (tag + digest, customManagers in renovate.json). No pipeline deploys to edge-1, so after the merge run ansible-playbook plays/install-zwave-gateway.yml from ansible/ on a mesh-connected workstation. The play reads the image's Alpine release itself and rebuilds the preloaded .so when it changes (see "Logging & eMMC" → zwave-js).

Logging & eMMC

edge-1 boots off a small internal eMMC (Dell Wyse 3040, ~7.3 GB), so the role keeps logs off local flash and monitors wear:

• journald is volatile (Storage=volatile, journald-edge.conf) — the journal lives in /run/log/journal (RAM), capped at edge_journald_runtime_max. The old persistent store /var/log/journal is purged on first apply.
• Docker logs to journald (docker-daemon.json, log-driver: journald) instead of unbounded json files under /var/lib/docker/containers. Both compose services inherit it; docker logs still works.
• Grafana Alloy ships the journal to Loki (edge-alloy.service, which runs the upstream grafana/alloy image with /run/log/journal mounted read-only; Arch's grafana-alloy package has no loki.source.journal). Its WAL/positions sit on /run/alloy (tmpfs), so the agent writes nothing to the eMMC either. Net effect: no log writes hit local flash — the trade-off is that logs are RAM-only and lost on reboot unless already shipped.
• Loki endpoint: the old srv-network Loki (loki.srv.renner.uno:3100) no longer exists — Loki moved into the cluster, so Alloy pushes to the authenticated cluster ingress https://loki.renner.uno/loki/api/v1/push with basic-auth (edge_loki_username / vault_edge_loki_password). Delivery only works once the edge-1 htpasswd line has reconciled into the cluster (Flux) — before that Alloy gets 401 and buffers in RAM.
• Flash longevity: noatime + lazytime on root, weekly fstrim.timer, and a weekly edge-emmc-health.timer that logs the JEDEC life-time / pre-EOL estimators and warns (journal priority) once the flash leaves the healthy band. Check by hand any time with /usr/local/bin/edge-emmc-health or cat /sys/block/mmcblk0/device/life_time (0x01 0x01 = <10 % consumed). mmc extcsd read /dev/mmcblk0 (from mmc-utils) gives the full detail.

zwave-js value-DB writes (the dominant residual writer)

With logs off flash, the biggest remaining eMMC writer is zwave-js-ui: its value database (store/*.jsonl) is fsync'd on every value change (jsonl-db, and zwave-js-ui doesn't expose the driver's throttle). On a busy network that fsync storm churned ~2.6 GiB/day for ~0.1 KiB/s of actual new data — near-pure write amplification (each fsync forces a block write + ext4 journal commit).

• Fix: the zwave-js container preloads libeatmydata (LD_PRELOAD=/usr/lib/libeatmydata.so, mounted read-only from zwave-js/libeatmydata-alpine<X.Y>.so). This makes fsync/fdatasync no-ops for that one process, so the kernel batches the writes via normal writeback (~30 s) instead of forcing each one. The WS/MQTT value stream to Home Assistant and the on-disk cache are unchanged — full data resolution, no re-interview on restart. No mount option can do this: fsync is a durability contract the filesystem must honor (commit=/data=writeback/nobarrier don't batch it); the only mount-level alternative is putting the cache on tmpfs, which makes the whole store volatile → re-interview on every restart.
• The .so is musl (the image is Alpine-based), extracted from alpine:<X.Y>, where X.Y is read from the image's /etc/alpine-release on every run. A new Alpine release gives a new file name, so the play extracts it, recreates the container and deletes the old file.
• Durability guard: because fsync is neutralized, a hard power-loss during the (rare) settings write could truncate settings.json, which holds the Z-Wave network keys. edge-zwave-settings-backup.timer keeps a validated hourly rolling backup (settings.json.bak) on the eMMC so the keys stay recoverable. Value-cache loss on crash is harmless (devices re-report; jsonl-db tolerates a partial trailing line).
• Verify the drop with the eMMC-write sampling described above; roll back by removing the LD_PRELOAD env + .so mount from the compose file.