Compare commits

..
Author SHA1 Message Date
claude-bot c90406cca7 docs(apps): trim otbr, trmnl, zigbee2mqtt — and correct a copy-pasted falsehood
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Comments only across otbr, trmnl/renderer, zigbee2mqtt-deployment and the
smartctl ServiceMonitor. All four still parse; monitoring-rules still renders.

Each keeps the instruction and loses the investigation. otbr stays disabled with
the condition for re-enabling it — verified partition merge, one OMR prefix, the
Nest routers visible — instead of the paragraph reconstructing how the split
partition was diagnosed. Its socat bridge keeps both dead ends, because they are
what stops someone simplifying it: forkpty needs a socat the image does not ship,
and a sidecar cannot supply the PTY because devpts is per-container. zigbee2mqtt
keeps "never run alongside ZHA, and never start without the restored coordinator
backup or you form a new network and drop every device", and loses the cutover
diary around it.

The ServiceMonitor correction is the useful part. Four monitors carry the claim
that kube-prometheus-stack's Prometheus "selects ServiceMonitors by this label;
without it the resource exists but is never discovered". The live resource has
serviceMonitorSelector {} and podMonitorSelector {} — empty, so every monitor is
picked up regardless of labels. Only ruleSelector matches on
release=kube-prometheus-stack, which is why the same sentence on the disk-wear
PrometheusRule is correct and this one is not.

Corrected here for the monitor I added yesterday, which is where I introduced it
by copying a neighbour. The other three carry the same wrong sentence and are
left for a pass that can check each rather than sed across them.
2026-08-28 06:59:04 +02:00
claude-bot 14d62f3aa1 docs(home-assistant): trim the mdns-relay and mosquitto comments
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
mdns-relay 122 -> 64 comment lines, mosquitto 131 -> 118. Comments only; the
diff filtered for non-comment lines is empty and apps/home-assistant still
renders 83 objects.

The mdns-relay header was mine, written yesterday, and was the clearest case
of the problem: a seventy-line essay with sections. What survives is what stops
someone undoing it — that the relay must stay unicast because Cilium runs in
tunnel mode without enable-multicast and drops re-emitted multicast, so an
avahi-style reflector cannot work; that python-zeroconf accepts unicast at all,
which is the non-obvious fact the design rests on; that it restores
announcements but not HA-initiated browsing; and that so-bindtodevice=net1 is
what keeps any sibling pod from injecting records into HA's zeroconf cache.

Cut: the problem statement, the Errno 113 symptom, the exposure analysis
weighing what the IoT VLAN could already forge, and a paragraph explaining that
the relay is not what makes HA's listener reachable in-cluster.

mosquitto keeps every warning and loses the reasoning around them. The one
worth naming is the retained-state note: a roll preserves connectivity but not
the retained store, and Zigbee2MQTT does not recover on its own. That is an
operational consequence someone needs before touching the rollout strategy, so
it stays — the paragraph explaining why maxSurge:1 is safe does not.
2026-08-28 06:55:34 +02:00
k.renner 85b8f0eba9 Merge pull request 'docs(infra): trim the cilium and traefik helm release comments' (#685) from trim-comments-helm into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#685
2026-08-28 04:53:37 +00:00
6 changed files with 64 additions and 166 deletions
@@ -1,80 +1,22 @@
---
# mDNS relay: gives Home Assistant sight of the IoT VLAN's mDNS announcements
# without putting a macvlan on the HA pod itself.
# mDNS relay: holds the multicast membership on the IoT VLAN through its own
# macvlan and forwards each datagram to Home Assistant as ordinary UNICAST UDP.
#
# THE PROBLEM
# It must stay unicast. An avahi-style reflector cannot work here — Cilium runs
# in tunnel mode without enable-multicast, so re-emitted multicast is dropped by
# the CNI. Unicast is carried, and python-zeroconf accepts it: its datagram
# handler checks size and duplicates but never the destination or source
# address. RFC 6762's QU bit makes this protocol-legal rather than a trick.
#
# HA's zeroconf listener joins both mDNS groups on eth0 — 224.0.0.251 and
# ff02::fb — and never hears anything on either.
# Multicast is link-local, so IoT-VLAN announcements do not reach the pod
# CIDR; and Cilium runs in tunnel mode (VXLAN) without `enable-multicast`, so
# it does not replicate multicast between pods either. Both halves of the path
# are dead, which is why an avahi-style reflector — receive on one interface,
# re-emit as multicast on the other — cannot work here: the re-emitted packets
# would be dropped by the CNI.
# This restores passive reception only — announcements, including the
# re-announce a device sends when its address changes. HA-initiated browsing
# still dies on eth0; that would need a querier on the VLAN side, which this
# deliberately is not.
#
# Consequence for HA: the ESPHome integration dials devices at the address
# cached in its config entry. When a lease rotates, every attempt fails with
# `[Errno 113] Connect call failed` and the device stays unreachable, because
# the mDNS announcement that would carry the new address never arrives. New
# devices are never discovered at all.
#
# THE MECHANISM
#
# This relay holds the multicast membership on the IoT VLAN through its own
# macvlan (mdns-relay-network-attachment.yaml) and forwards each datagram it
# receives to HA as ordinary UNICAST UDP. That is the one direction of travel
# the CNI will carry.
#
# It works because python-zeroconf does not filter on how a packet arrived:
# its datagram handler validates size, deduplicates, and parses, but performs
# no check on destination address, source address, or source port. A unicast
# datagram delivered to :5353 is processed exactly like a multicast one. (mDNS
# unicast responses are part of RFC 6762 — the QU bit — so this is within the
# protocol, not a trick.)
#
# WHAT THIS DOES AND DOES NOT RESTORE
#
# Restored: passive reception. RFC 6762 requires a responder to re-announce
# when its address changes, so a device that moves to a new lease announces
# the move, HA sees it, and the ESPHome integration updates the config entry.
# That is the failure this exists to fix. New devices announcing themselves
# are likewise seen.
#
# NOT restored: HA-initiated browsing. Queries HA sends leave on eth0 as
# multicast and die there, so HA cannot actively solicit a response. Anything
# depending on a query rather than an announcement stays broken. Making that
# work needs a querier on the VLAN side, which this relay deliberately is not.
#
# EXPOSURE
#
# The receive sockets carry so-bindtodevice=net1, so only datagrams that
# actually arrived on the IoT VLAN interface are delivered. Binding to net1's
# address instead is not possible — it is DHCP-leased and unknown at exec time
# — but binding to the device achieves the same confinement without needing to
# know the address, and needs no capability: SO_BINDTODEVICE succeeds as an
# unprivileged uid here.
#
# Without it socat would bind the wildcard, and since the namespace CNP grants
# every pod in home-assistant unrestricted access to every other pod on every
# port, any sibling pod could reach :5353 over the cluster network and have
# records forwarded into HA's zeroconf cache.
#
# What remains, and cannot be closed here: any host on the IoT VLAN can send
# this port a datagram. That is the same trust already extended to genuine
# mDNS from that segment, which any device there can equally forge — the relay
# widens who can reach HA's listener, not what can be claimed. macvlan traffic
# bypasses Cilium policy entirely, so the namespace CNP does not constrain
# this interface either way.
#
# Note that the relay is not what makes HA's zeroconf reachable from inside
# the cluster: HA binds :5353 on the pod address (v4) and the wildcard (v6),
# and the namespace CNP already permits any sibling pod to reach it directly.
# Confining the relay removes a second route to that listener; it does not
# remove the first.
#
# Nothing else is exposed: the containers run socat and only socat, with no
# second listener to reach.
# so-bindtodevice=net1 on both receive sockets is load-bearing: without it
# socat binds the wildcard, and the namespace CNP lets any sibling pod reach
# :5353 and inject records into HA's zeroconf cache. Binding net1's address
# instead is impossible — it is DHCP-leased and unknown at exec time.
apiVersion: apps/v1
kind: Deployment
metadata:
@@ -6,32 +6,23 @@ metadata:
labels:
app.kubernetes.io/name: mosquitto
annotations:
# REQUIRED for credential rotation to take effect at all. The startup shim
# rebuilds /mosquitto/auth/passwd from these Secrets on every pod start, so
# a rotated Secret is inert until the pod restarts — without this the new
# password would sit unused until some unrelated restart, while the old one
# stayed valid indefinitely.
# REQUIRED for credential rotation to take effect. The startup shim rebuilds
# the password file only on pod start, so without this a rotated Secret sits
# inert while the old password stays valid indefinitely.
#
# Listed per-Secret rather than using the blanket `reloader.stakater.com/
# auto: "true"`, so that enrolling each remaining broker user in rotation is
# a visible, reviewable edit to this line rather than something that starts
# happening implicitly.
# Listed per-Secret rather than with the blanket auto annotation, so that
# enrolling another broker user is a visible edit to this line.
secret.reloader.stakater.com/reload: mosquitto-health-credentials
spec:
replicas: 1
strategy:
# No PVC anymore, so nothing to fence: the new broker can come up and pass
# its readiness probe before the old one is torn down. maxUnavailable:0 keeps
# a broker serving throughout an image bump / reschedule (clients on the old
# pod just reconnect to the new one); maxSurge:1 permits the brief two-pod
# overlap — two independent stateless brokers behind the LB, fine since there
# is no shared state.
# No PVC, so nothing to fence — the new broker can serve before the old one
# is torn down. Two stateless brokers overlapping behind the LB is fine.
#
# NOTE: connectivity survives the roll, but RETAINED STATE DOES NOT — the
# new pod starts with an empty retained store, so a roll is equivalent to a
# broker restart for every client that depends on retention. Zigbee2MQTT
# does not recover from that on its own; see mosquitto-config.yaml and
# gitea issue #381.
# Connectivity survives a roll but RETAINED STATE DOES NOT: the new pod
# starts with an empty retained store, so every roll is a broker restart
# for clients that depend on retention. Zigbee2MQTT does not recover from
# that on its own (#381).
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0
@@ -44,14 +35,10 @@ spec:
labels:
app.kubernetes.io/name: mosquitto
spec:
# Runs on any node (no nodeSelector); the mosquitto-iot LB forwards
# IoT-VLAN traffic to the broker wherever it lands.
# Availability: home-critical priority (never preempted, reschedules
# first) + short fault tolerations so a node failure reschedules the
# broker in ~30s instead of the default ~5min. With persistence off there
# is no PVC to fence/detach/reattach, so the broker can (re)start anywhere
# instantly, no longer depends on Ceph being healthy, and rolls without a
# tear-down-first gap (see the RollingUpdate strategy above).
# Runs anywhere; the mosquitto-iot LB finds it. Short fault tolerations
# reschedule the broker in ~30s rather than the default ~5min, which is
# only possible because there is no PVC to fence — so it also does not
# depend on Ceph being healthy.
priorityClassName: home-critical
terminationGracePeriodSeconds: 10
tolerations:
@@ -6,12 +6,9 @@ metadata:
labels:
app.kubernetes.io/name: zigbee2mqtt
spec:
# CUTOVER DONE (2026-07-23): ZHA disabled, the coordinator-backup Secret is
# populated in OpenBao, and Z2M is now the live Zigbee stack — so replicas is
# 1. It shipped at 0 as a safety interlock (Z2M contends with ZHA for the
# coordinator's single TCP session, and starting without the restored backup
# would form a NEW network and drop every device); that interlock is spent
# once the migration is committed. To roll back, set 0 and re-enable ZHA.
# Z2M is the live Zigbee stack. Never run it alongside ZHA: they contend for
# the coordinator's single TCP session, and a Z2M start without the restored
# coordinator backup forms a NEW network and drops every paired device.
replicas: 1
# RWO PVC → single writer; never run two pods against the same /app/data.
strategy:
@@ -39,14 +36,10 @@ spec:
# network_key, the latter extracted from the OpenBao coordinator backup
# as a natural-byte-order int array.
- name: seed-data
# alpine + jq, built by .woodpecker/build-ci-images.yml from
# ci/images/alpine-jq/. Replaces `apk add --no-cache jq` on every pod
# start: tools belong in the image, not installed at runtime.
# Referenced by tag rather than digest — these are our own images.
# DATED tag, not :latest, matching the ci-ansible:2026-04-29 precedent
# in kubernetes/infrastructure/netbird-backup/cronjob.yaml. With
# :latest, rebuilding this base for some unrelated consumer would
# silently change what this init container runs on its next pod
# Our own alpine + jq image, built by build-ci-images.yml. DATED tag,
# never :latest — rebuilding this base for an unrelated consumer
# would otherwise silently change what this init container runs on
# its next pod
# restart, with no diff here to review — and this is the container that
# renders the Zigbee network key. Bumping the tag is a deliberate,
# reviewable act; the cost is remembering to bump it.
@@ -5,9 +5,8 @@ metadata:
namespace: monitoring
labels:
app.kubernetes.io/name: smartctl-exporter
# kube-prometheus-stack's Prometheus resource selects ServiceMonitors
# by this label; without it the resource exists but is never
# discovered.
# Not required for discovery — serviceMonitorSelector is empty — but kept
# for consistency with the other monitors.
release: kube-prometheus-stack
spec:
selector:
+17 -35
View File
@@ -5,25 +5,15 @@ metadata:
labels:
app.kubernetes.io/name: otbr
spec:
# Disabled 2026-07-11 (replicas: 0) after otbr broke Matter.
# DISABLED (replicas: 0). Running otbr broke Matter: its radio is RF-isolated
# enough from the Nest mesh that it forms a rival partition of the same
# network, and the two bridge only over Ethernet — fast enough for a ping,
# too slow for the fragmented CASE handshake, so devices stay discoverable
# while every commissioning attempt times out.
#
# otbr is seeded with the Nest network's dataset so it should join the
# existing partition as an additive co-BR. In practice it does not merge
# cleanly: its SLZB-06 radio is RF-isolated enough from the Nest mesh that
# otbr elects itself leader of a rival partition (or the network re-addresses
# under a new OMR prefix, observed fd5f -> fdd5). The two same-network
# partitions then bridge only over the Ethernet backbone, which carries a
# ping but is too slow for the large fragmented CASE handshake -- every
# Matter device stays mDNS-discoverable while CASE times out. Removing otbr
# re-merges the mesh onto the Nest hubs and Matter recovers.
#
# This is NOT a diagnosis of a specific config bug: the DHCP/sbr routing
# changes (19a4482, df61407) were investigated and matter-server's IPv6 path
# to the mesh works (reaches devices, ping ok); sbr's v6 table 100 is an
# inert orphan and otbr's eth0 default is outranked by net1's pref-high RA
# default. The instability is at the Thread partition layer. Re-enable only
# after the co-BR integration is verified to MERGE (one partition, one OMR,
# otbr's router table shows the Nest routers) instead of split.
# The instability is at the Thread partition layer, not in the config below.
# Re-enable only once co-BR integration is verified to MERGE: one partition,
# one OMR prefix, and the Nest routers visible in otbr's router table.
replicas: 0
strategy:
type: Recreate
@@ -46,25 +36,17 @@ spec:
# (slzb-thread.renner.uno). Dropping the workload-class=pinned
# toleration lets the edge taint keep it off the tiny edge box.
containers:
# The Thread RCP lives on a SLZB-06 over Ethernet at
# slzb-thread.renner.uno:6638 (Serial-over-IP). otbr-agent's
# spinel transport supports `spinel+hdlc+forkpty://<exec>?<args>`
# which forks an exec whose stdio becomes the spinel-HDLC
# transport, but the upstream openthread/otbr image does NOT
# ship socat (or any TCP-stream tool), so that URL form ends in
# silent ENOENT and otbr-agent dies right after logging the
# radio URL.
# The Thread RCP is a network-attached radio (Serial-over-IP), so the
# spinel transport needs a TCP-to-PTY bridge. Two obvious approaches
# do not work and should not be retried:
#
# A sidecar socat won't work either: each container has its own
# devpts namespace, so a PTY (e.g. /dev/pts/0) created in a
# sibling container is invisible from this container — a shared
# emptyDir symlink to /dev/pts/0 is dangling.
# - `spinel+hdlc+forkpty://` needs socat, which the upstream image
# does not ship; the URL fails with a silent ENOENT and otbr-agent
# dies right after logging it.
# - A sidecar cannot provide it either: devpts is per-container, so
# a PTY created next door is invisible here and the symlink dangles.
#
# The bridge has to live in the otbr container's own namespace.
# Install socat at startup via apt (Ubuntu 18.04 base, no egress
# restrictions on this namespace), run it in the background as
# the spinel-HDLC TCP↔PTY bridge, then exec the upstream
# entrypoint with RADIO_URL pointing at the PTY symlink.
# Hence socat is installed at startup and run inside this container.
- name: otbr
image: openthread/otbr:latest
command:
+8 -13
View File
@@ -1,20 +1,15 @@
# Headless renderer: screenshots the Home Assistant e-ink dashboard,
# dithers it to 16 greys and hands it to Terminus.
#
# Upstream ships this as a Home Assistant add-on. Our HA is a plain
# container with no Supervisor, so it runs standalone here instead —
# viable because the app prefers HOME_ASSISTANT_URL/ACCESS_TOKEN env
# vars over /data/options.json and authenticates with an ordinary
# long-lived token. SUPERVISOR_TOKEN is deliberately NOT set: it only
# flips an internal isAddOn flag, and setting it would switch the app
# to add-on behaviour (fixed port, /data assumptions).
# Upstream ships this as a Home Assistant add-on; it runs standalone here
# because the app prefers the env vars over /data/options.json. Do NOT set
# SUPERVISOR_TOKEN — it flips an internal isAddOn flag and switches the app to
# add-on behaviour (fixed port, /data assumptions).
#
# DIRECTION OF TRAVEL, which is the non-obvious part: with the
# byos-hanami webhook in `uri` delivery mode the renderer does NOT
# push image bytes. It POSTs a URL pointing at its own /output/<file>
# endpoint, and TERMINUS FETCHES THE IMAGE BACK from it. That is why
# both live in this namespace — the pull is intra-namespace and is
# already permitted by the default-deny-ingress rule.
# DIRECTION OF TRAVEL, the non-obvious part: in `uri` delivery mode the
# renderer does not push image bytes. It POSTs a URL to its own /output/<file>
# and Terminus fetches the image back. Both therefore live in this namespace,
# where that pull is already permitted.
apiVersion: v1
kind: ConfigMap
metadata: