edge-host: TLS for edge MQTT via a local bridging mosquitto

Security review #sec-1 (edge-1 leg). open3e/e3oncan/sml2mqtt publish to the cluster broker in plaintext over the untrusted IoT VLAN, and open3e/e3oncan have no native MQTTS (their MQTT code does a bare connect()), so "just use 8883" isn't possible for them. Run a local mosquitto on edge-1 that the clients reach over loopback (127.0.0.1:1883), bridged to the cluster broker over TLS (mqtt.renner.uno:8883, public LE cert validated via the system CA bundle). Plaintext stays on loopback; the wire hop is encrypted. Two bridge connections preserve the existing cluster identities (open3e user carries open3e/# out + open3e/cmnd back; sml2mqtt user carries sml2mqtt/# out), so NO cluster-side change is needed and the existing ACLs already suffice. Bonus (availability, #avail-4): the local broker buffers telemetry in RAM and flushes on reconnect, so edge-1 no longer hard-depends on cluster/network uptime for every publish. eMMC: the local broker writes NOTHING to flash — persistence false (queue in RAM) + log_dest stdout (volatile journal -> Loki). Verified before building: edge-1 -> :8883 TLS validates (return code 0); mosquitto 2.1.2 in Arch extra; existing open3e/sml2mqtt cluster ACLs cover the bridge; #125 netpol already allows :8883. Rollback: point edge_mqtt_host back to mqtt.renner.uno and re-run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-08 09:01:24 +02:00
parent b1c380c825
commit a12116486d
7 changed files with 135 additions and 7 deletions
@@ -180,3 +180,35 @@ journal commit).
  jsonl-db tolerates a partial trailing line).
 - Verify the drop with the eMMC-write sampling described above; roll back by
  removing the `LD_PRELOAD` env + `.so` mount from the compose file.
+
+## MQTT bridge (TLS to the cluster broker)
+
+open3e, e3oncan and sml2mqtt do **not** publish to the cluster broker directly.
+They connect to a **local mosquitto on loopback** (`127.0.0.1:1883`,
+`edge_mqtt_host`), which **bridges to the cluster broker over TLS**
+(`mqtt.renner.uno:8883`). Two reasons:
+
+- **Security:** open3e and e3oncan have no native MQTTS support (their MQTT
+  code does a plain `connect()`), so they can't use the broker's 8883 TLS
+  listener. Terminating TLS in the local broker keeps their plaintext MQTT on
+  loopback only — off the untrusted IoT VLAN — while the actual network hop is
+  encrypted. The public Let's Encrypt cert is validated against the system CA
+  bundle (`edge_mqtt_bridge_cafile`); no cert distribution needed.
+- **Availability:** the local broker buffers telemetry (in RAM) if the cluster
+  broker is briefly unreachable, and flushes on reconnect — edge-1 no longer
+  hard-depends on cluster/network uptime for every publish.
+
+Two bridge `connection`s preserve the existing cluster identities, so **no
+cluster-side change was needed**: one authenticates as the `open3e` user
+(carries `open3e/#` out + `open3e/cmnd` back), one as `sml2mqtt` (`sml2mqtt/#`
+out). The local broker is `allow_anonymous` on loopback — the trust boundary
+is the host; nothing off-host can reach it.
+
+**eMMC:** the local broker writes **nothing** to flash — `persistence false`
+(retained/queued messages in RAM) and `log_dest stdout` (to the volatile
+journal → Loki). Do not add persistence or log-file settings to
+`edge-mqtt-bridge.conf.j2`.
+
+Rollback: point `edge_mqtt_host` back to `mqtt.renner.uno` and re-run (clients
+reconnect directly on 1883). Config: `edge-mqtt-bridge.conf.j2` →
+`/etc/mosquitto/mosquitto.conf`.
@@ -23,10 +23,19 @@ edge_alloy_labels:
  environment: production
  datacenter: homelab

-# --- MQTT: all services publish to the central in-cluster Mosquitto via
-# its LoadBalancer hostname (the edge host is outside the cluster).
-edge_mqtt_host: mqtt.renner.uno
+# --- MQTT: edge clients (open3e, e3oncan, sml2mqtt) publish to a LOCAL
+# mosquitto on loopback, which bridges to the central cluster broker over TLS
+# (open3e/e3oncan have no native MQTTS). This keeps plaintext on loopback only
+# — off the untrusted IoT VLAN — and buffers telemetry in RAM during cluster
+# blips. The local broker writes nothing to the eMMC (persistence off, logs to
+# the volatile journal). See edge-mqtt-bridge.conf.j2 and README "MQTT bridge".
+edge_mqtt_host: 127.0.0.1
 edge_mqtt_port: 1883
+# Upstream cluster broker the local bridge connects to (TLS; the public
+# Let's Encrypt cert is validated against the system CA bundle).
+edge_mqtt_cluster_host: mqtt.renner.uno
+edge_mqtt_cluster_tls_port: 8883
+edge_mqtt_bridge_cafile: /etc/ssl/certs/ca-certificates.crt

 # --- USB devices (stable by-id symlinks). After physically moving the
 # sticks to edge-1, verify with `ls -l /dev/serial/by-id/` — the by-id
@@ -31,6 +31,13 @@
    state: restarted
    daemon_reload: true

+# Ordered before the MQTT clients (compose/open3e/e3oncan) so they reconnect
+# to a freshly-configured local broker.
+- name: Restart mosquitto
+  ansible.builtin.service:
+    name: mosquitto
+    state: restarted
+
 - name: Restart edge compose services
  ansible.builtin.command:
    cmd: docker compose up -d --force-recreate
@@ -12,8 +12,29 @@
      - git
      - mmc-utils      # eMMC lifetime/EOL introspection (mmc extcsd read)
      - grafana-alloy  # log shipper -> Loki (see Alloy section below)
+      - mosquitto      # local MQTT bridge broker -> cluster over TLS
    state: present

+# --- Local MQTT bridge broker. Edge clients publish to this over loopback;
+# it bridges to the cluster broker over TLS (open3e/e3oncan can't do MQTTS).
+# Configured with persistence off + stdout logging so it writes nothing to the
+# eMMC. Deployed before the clients so they have somewhere to connect.
+# See edge-mqtt-bridge.conf.j2 / README "MQTT bridge".
+- name: Deploy local mosquitto bridge config
+  ansible.builtin.template:
+    src: edge-mqtt-bridge.conf.j2
+    dest: /etc/mosquitto/mosquitto.conf
+    owner: root
+    group: mosquitto
+    mode: "0640"   # carries the cluster bridge credentials
+  notify: Restart mosquitto
+
+- name: Enable local mosquitto bridge
+  ansible.builtin.systemd:
+    name: mosquitto
+    enabled: true
+    state: started
+
 # ======================================================================
 # eMMC wear mitigation + log shipping. edge-1 (Dell Wyse 3040) runs on a
 # small internal eMMC, so logs must not churn local flash. Strategy:
@@ -6,8 +6,9 @@
 [Unit]
 Description=E3onCAN E380 energy-meter CAN listener (read-only)
 BindsTo=sys-subsystem-net-devices-{{ edge_can_interface }}.device
-After=sys-subsystem-net-devices-{{ edge_can_interface }}.device network-online.target
-Wants=network-online.target
+# mosquitto.service = the local MQTT bridge broker e3oncan publishes to (loopback).
+After=sys-subsystem-net-devices-{{ edge_can_interface }}.device network-online.target mosquitto.service
+Wants=network-online.target mosquitto.service

 [Service]
 User=open3e
@@ -0,0 +1,57 @@
+# Managed by roles/edge-host. Local mosquitto that the edge clients (open3e,
+# e3oncan, sml2mqtt) publish to over loopback; it bridges to the central
+# cluster broker over TLS. open3e/e3oncan have no native MQTTS, so this is how
+# edge-1's MQTT gets off plaintext on the untrusted IoT VLAN, and it buffers
+# telemetry in RAM if the cluster broker is briefly unreachable.
+#
+# eMMC: this host runs on a small internal eMMC, so the local broker writes
+# NOTHING to flash — `persistence false` keeps retained/queued messages in RAM
+# only, and `log_dest stdout` goes to the (volatile, RAM) journal that Alloy
+# ships to Loki. Do not add persistence/log-file settings here.
+persistence false
+log_dest stdout
+log_type error
+log_type warning
+log_type notice
+
+# Loopback only — the trust boundary is the host itself. Any process on edge-1
+# may publish here (same trust as the bridged services); nothing off-host can
+# reach it. Real authentication to the cluster happens on the bridge below.
+# Single v4-loopback listener: all edge clients dial 127.0.0.1 (edge_mqtt_host),
+# and a second same-port listener can make some builds refuse to start.
+listener 1883 127.0.0.1
+allow_anonymous true
+
+# --- Bridge 1: open3e datapoints + command topic, as the cluster `open3e`
+# user (its ACL is `readwrite open3e/#`, which covers publishing datapoints
+# and subscribing to open3e/cmnd). e3oncan also publishes under open3e/#.
+connection edge1-open3e
+address {{ edge_mqtt_cluster_host }}:{{ edge_mqtt_cluster_tls_port }}
+bridge_cafile {{ edge_mqtt_bridge_cafile }}
+bridge_insecure false
+remote_username {{ edge_open3e_mqtt_user }}
+remote_password {{ edge_open3e_mqtt_password }}
+remote_clientid edge1-bridge-open3e
+cleansession false
+try_private true
+notifications false
+restart_timeout 5 30
+queue_qos0_messages true
+topic open3e/# out 0
+topic open3e/cmnd in 0
+
+# --- Bridge 2: power-meter topics, as the cluster `sml2mqtt` user (ACL
+# `write sml2mqtt/#`). Publish-only, so no `in` topics.
+connection edge1-sml2mqtt
+address {{ edge_mqtt_cluster_host }}:{{ edge_mqtt_cluster_tls_port }}
+bridge_cafile {{ edge_mqtt_bridge_cafile }}
+bridge_insecure false
+remote_username sml2mqtt
+remote_password {{ edge_sml2mqtt_mqtt_password }}
+remote_clientid edge1-bridge-sml2mqtt
+cleansession false
+try_private true
+notifications false
+restart_timeout 5 30
+queue_qos0_messages true
+topic sml2mqtt/# out 0
@@ -13,8 +13,9 @@ Description=open3e Viessmann E3 CAN-to-MQTT bridge (read-only)
 # Tie the service to the CAN device: stops on unplug, starts on plug via
 # the device unit rather than polling.
 BindsTo=sys-subsystem-net-devices-{{ edge_can_interface }}.device
-After=sys-subsystem-net-devices-{{ edge_can_interface }}.device network-online.target
-Wants=network-online.target
+# mosquitto.service = the local MQTT bridge broker open3e publishes to (loopback).
+After=sys-subsystem-net-devices-{{ edge_can_interface }}.device network-online.target mosquitto.service
+Wants=network-online.target mosquitto.service

 [Service]
 User=open3e