The broker's liveness+readiness were tcpSocket:1883 — a bare TCP accept
proves a listener exists, not that the broker routes. Even a QoS-1 PUBACK
proves only that the broker acknowledged the publish at the protocol level;
a broker wedged so it ACKs writes but drops them (a write blackhole) would
still pass.
So each probe publishes a fresh RANDOM value retained and reads it back,
succeeding only if the exact bytes round-trip. That proves the broker
authenticated, stored the message, and delivered it to a subscriber:
auth + write + storage + read + delivery. Retained (not a live
subscribe-then-publish) so the read never races the connect and can't
false-fail; liveness and readiness use separate topics (probe/live vs
probe/ready) so their rotating retained values never collide.
Runs as a dedicated least-privilege `health` user, not admin. The probes
are exec probes running INSIDE the mosquitto container; admin's password is
only reachable there because the startup shim hashes every user into the
passwd file, but admin is readwrite # and meant for an external workstation
client, so it's the wrong credential for a probe.
- new mosquitto-health-credentials-bootstrap Job -> Secret
mosquitto-health-credentials (same per-user pattern as the other six).
- ACL: readwrite health/# and NOTHING else (read needed for the
read-back). A leak can only touch the throwaway probe subtree that
nothing subscribes to.
- HEALTH_PASSWORD env (the shim needs it to build passwd); the
in-container exec probes read it directly.
startupProbe tcpSocket :1883, 5s x 12 = 60s (passwd/acl build + bind)
readinessProbe round-trip, 15s x 3, timeout 10s
livenessProbe round-trip, 30s x 6 = 3 min sustained before restart, timeout 10s
Verified live before merge (with the admin cred, which already exists —
identical command shape):
- exact retained round-trip script -> exit 0, value read back matches.
- negative control (subscribe to an empty topic) -> times out, mismatch,
non-zero exit. Confirms the probe fails on a blackhole, not just on TCP.
- round-trip latency ~0 ms on loopback, far under the 10s timeout.
SPOF safety (single replica; HA, Shelly, sml2mqtt, open3e, evcc):
- liveness fires slowly (3 min sustained) so a transient hiccup never
bounces the broker.
- Deployment is maxUnavailable:0 / maxSurge:1: the new pod must pass
readiness before the old is torn down, so even if the health wiring were
wrong the old broker keeps serving — a stalled rollout, not an outage.
kubectl kustomize builds. Post-merge: bootstrap Job creates the secret, pod
rolls (old serves until new Ready), health user round-trips green, 0
restarts, clients stay connected; negative-control a wrong cred/blackhole.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
matter-server had NO probes (2026-07-10 audit, flagged HIGH — it's the
otbr sibling and had a real incident). A hung or crashed server kept the
pod Running while HA's WebSocket to it was dead.
`GET /` on :5580 returns 200 from the aiohttp control server, so it's a
real "server up and answering HTTP" check, stronger than a bare tcpSocket.
httpGet targets the pod IP, which is exactly where the server binds
(--listen-address=$(POD_IP), PR #144) — verified GET / -> 200 on the pod
IP, and the /ws upgrade returns 101.
startupProbe GET / :5580, 10s x 30 = 5 min. Suspends the other two
while the SDK loads the fabric and binds the socket.
readinessProbe GET / :5580, 15s x 3. Gates HA's ClusterIP path.
livenessProbe GET / :5580, 30s x 5 = 2.5 min sustained failure before a
restart. Deliberately conservative: a restart churns the
whole Matter fabric (~13 nodes re-subscribe over minutes,
observed in #144), so a transient blip must never trigger
it. The check is a loopback-reliable aiohttp handler, very
low false-positive.
SCOPE, stated honestly: this catches server-down / hung / not-listening —
the current gap. It does NOT catch "fabric lost / 0 nodes": that failure
leaves the :5580 server (on eth0) answering while net1/SDK is broken, so
no :5580 probe can see it. Closing that would need an app-level node-count
signal matter-server doesn't expose over HTTP; out of scope.
Verified: kubectl kustomize builds; rendered probes as intended. Post-merge:
pod rolls, startupProbe passes once :5580 answers, Ready, and the 13-node
fabric re-subscribes (expected churn on any matter-server restart).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up to the Alertmanager (#146) and Prometheus (#147) gates. Both
alerts.renner.uno and prometheus.renner.uno are now Authentik-gated, and
the existing blackbox http_2xx probes target those external hostnames. A
gated endpoint returns 302; blackbox follows it to the Authentik login
(200), so the probe stays GREEN — but it now measures the gate path, not
the app. A dead Prometheus/Alertmanager behind a live gate would show up.
Verified empirically before this change: probe_success{instance=
"https://alerts.renner.uno"}=1 with final_status=200 (3 redirect hops
landing on auth.renner.uno). No probe goes red, but the signal is a false
green for app health.
Fix per monitoring-vantage-point-rule: an in-cluster service must be
probed from an in-cluster prober at its in-cluster address. Add two
INTERNAL targets to the blackbox-http Probe:
http://kube-prometheus-stack-alertmanager.monitoring.svc:9093/-/healthyhttp://kube-prometheus-stack-prometheus.monitoring.svc:9090/-/healthy
The gate is only on the ingress, so these ClusterIP /-/healthy endpoints
are reachable and unauthenticated in-cluster (verified: alertmanager ->
200 "OK", prometheus -> 200 "Prometheus Server is Healthy"). blackbox-
exporter runs in the monitoring namespace, so intra-namespace reach to
:9093/:9090 is allowed. These probes go red on actual app death,
independent of the gate — closing the blind spot.
The external hostname probes are KEPT: they still catch DNS/TLS/Traefik/
gate-outpost failures that an internal probe cannot. So both signals now
exist — app health (internal) and gated-path reachability (external) —
instead of one conflated one.
Scope note: the legacy srv-host Prometheus on VLAN 40
(ansible/roles/prometheus-server, ansible/roles/blackbox-exporter) also
probes https://prometheus.renner.uno, but VLAN 40 has no route to cluster
ClusterIPs, so it structurally cannot adopt the internal-health probe.
Per the vantage-point rule the in-cluster prober is the correct owner of
app health; the srv-host probe stays a pure external-reachability check
and is left untouched (its cleanup belongs with the eventual retirement of
the legacy srv-host monitoring). No alert depends on these probe instances
(only the Hetzner object-storage probe is alerted), so no alert breaks.
Verified: kubectl kustomize builds; rendered Probe carries both internal
targets. Post-merge: probe_success for the two /-/healthy instances == 1,
and killing the app path (or confirming via a scaled test) drops it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Prometheus (prometheus.renner.uno, monitoring ns, :9090) had no auth of its
own. Its admin API is disabled (read-only), but the full 90d metrics DB is
an infra-recon oracle — every hostname, IP and service in the cluster
(2026-07-10 audit). Gate it the same way as alertmanager-gate (#146).
- Blueprint: prometheus-gate proxy provider (internal_host
http://kube-prometheus-stack-prometheus.monitoring.svc:9090, external_host
https://prometheus.renner.uno), application, outpost. REUSES the existing
git-owned "Monitoring Access" group — created generically in #146 for
exactly this, so no new group.
- implicit-consent flow (same rationale as alertmanager/paperless: a
frequently-checked monitoring UI; an authenticated session already closes
the exposure).
- No skip_path_regex: the Prometheus UI is a plain React SPA — verified its
index.html has no serviceWorker/manifest/webmanifest.
- Helm-generated Prometheus Ingress disabled (prometheus.ingress.enabled=
false); standalone prometheus-gate-ingress.yaml backends the outpost
ExternalName. Mirrors the alertmanager gate.
- Scoped CNPs: monitoring ingress rule allowing only the outpost -> :9090;
matching egress in the authentik namespace.
Grafana is UNAFFECTED: its Prometheus datasource targets the ClusterIP
Service (kube-prometheus-stack-prometheus.monitoring:9090), verified in the
live datasource provisioning, not the ingress hostname.
Known side effect, documented not silently shipped: three blackbox
`http_2xx` uptime probes target https://prometheus.renner.uno
(kubernetes/apps/monitoring-rules/probes-blackbox.yaml, the srv-host
ansible prometheus.yml.j2, and the ansible blackbox-exporter role). After
gating they follow the 302 to the Authentik login (200), so they stay GREEN
— but now measure Authentik reachability, not Prometheus. This is already
true for alerts.renner.uno since #146 (verified: probe_success=1,
final_status=200 via redirect-follow). No probe goes red; the monitoring
just becomes a gate-liveness check. Repointing those probes to an internal
endpoint is a separate follow-up, not done here.
Verified: kubectl kustomize builds for authentik, monitoring,
network-policies; embedded outpost survived the blueprint insert; rendered
ingress backends ak-outpost-prometheus-gate:9000; both prometheus and
alertmanager helm ingresses now disabled; egress + CNP scoped to
prometheus:9090.
Post-merge checks: only the gate Ingress serves prometheus.renner.uno (helm
one gone); unauthenticated GET -> 302 to auth.renner.uno; unauthenticated
GET /api/v1/query -> 302; outpost forwards to :9090 after login; Grafana
dashboards still render (ClusterIP datasource intact).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Alertmanager (alerts.renner.uno, monitoring ns, :9093) had no auth of its
own, and its v2 API is writable: an unauthenticated LAN client could POST
/api/v2/silences and suppress ALL alerting — a tamper capability, not just
disclosure (2026-07-10 audit, highest-teeth of the deferred unauth
findings).
Gate it with the same Authentik proxy-outpost pattern as
paperless/vscode/shelly-manager:
- New proxy provider + application + dedicated outpost
(ak-outpost-alertmanager-gate), mode proxy, forwarding to the ClusterIP
Service (kube-prometheus-stack-alertmanager.monitoring.svc:9093).
- New git-owned "Monitoring Access" group (k.renner) bound to the app.
Named generically so it can also gate the still-deferred Prometheus UI.
- The Helm-chart-generated Alertmanager Ingress is disabled
(alertmanager.ingress.enabled=false); a standalone gate Ingress
(alertmanager-gate-ingress.yaml) backends the outpost ExternalName —
two Ingresses for the same host would conflict. Mirrors loki-ingress.
- Scoped CNPs: monitoring ingress rule allowing only the outpost -> :9093;
matching egress rule in the authentik namespace.
implicit-consent flow, NOT passkey step-up: Alertmanager is a
frequently-checked monitoring UI, and requiring an authenticated Authentik
session already closes the unauthenticated-LAN exposure. Passkey re-auth on
every visit is disproportionate friction for a dashboard (paperless-gate
uses implicit-consent for the same reason). Bump to
passkey-stepup-authorization if the write API warrants it.
No skip_path_regex: the Alertmanager UI is a plain Elm SPA with no service
worker and no manifest (verified 2026-07-10 — index.html has no
serviceWorker/manifest/webmanifest), so none of the credentials:'omit' PWA
fetches that forced skip carve-outs elsewhere.
Prometheus's alerting path is UNAFFECTED: it posts to the ClusterIP Service
(verified: spec.alerting.alertmanagers targets
monitoring/kube-prometheus-stack-alertmanager:http-web), not this ingress.
Prometheus's own ingress is untouched (still enabled) — its unauth exposure
is a separate deferred finding, and it is read-only.
Verified: kubectl kustomize builds for authentik, monitoring and
network-policies; the embedded outpost survived the blueprint insert; the
rendered ingress backends ak-outpost-alertmanager-gate:9000, egress is
scoped to alertmanager:9093, and alertmanager.ingress.enabled=false while
prometheus.ingress.enabled=true.
Post-merge checks to run: only the gate Ingress exists for alerts.renner.uno
(the Helm one is gone); unauthenticated GET -> 302 to auth.renner.uno;
POST /api/v2/silences unauthenticated is refused; the outpost forwards to
:9093 after login; Prometheus is still delivering alerts to Alertmanager.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Last of the three IoT-VLAN gate/bind bypasses (shelly-manager #142,
matter-server #144). otbr's net1 is a macvlan on the IoT VLAN (backbone
interface), macvlan bypasses Cilium policy, and REST_HOST=0.0.0.0 bound
otbr-agent's REST API on :8081 to all interfaces — so the unauthenticated
OTBR REST API was reachable from the whole IoT VLAN.
Verified 2026-07-10 from edge-1 (on v30iot): GET http://<net1 ip>:8081/
answered (404 at /, but it is the live otbr-agent REST listener). That API
can read and modify Thread network configuration, including the network
key — i.e. enough to join or disrupt the Thread mesh.
Note the reality vs. the old comment: otbr-agent's REST listens on :8081
(governed by REST_HOST / --rest-listen-address); otbr-web sits on
127.0.0.1:80 (loopback, not exposed); the container's declared rest:8080
port has no listener. So REST_HOST is the whole exposure.
Set REST_HOST to the pod eth0 address via the downward API
(POD_IP=status.podIP; REST_HOST=$(POD_IP)). POD_IP is declared BEFORE
REST_HOST because k8s $(VAR) substitution only sees earlier env entries.
HA's OTBR integration keeps working via the IPv4 ClusterIP Service
(otbr.home-assistant.svc:8081 -> podIP:8081). net1 no longer carries REST.
The border router itself is unaffected: it uses net1 through
BACKBONE_INTERFACE for mDNS/RA/backbone, independent of where REST listens.
Verified: kubectl kustomize builds; rendered env has POD_IP (downward API)
before REST_HOST=$(POD_IP); BACKBONE_INTERFACE=net1 unchanged. Baseline
before change: state=leader, br=running, mesh 2 routers/8 children/8
neighbors.
CAVEAT: this restarts the Thread border router. Expect ~60s of
self-healing CASE-session churn on Thread/Matter end devices as the mesh
re-forms (per the #141 verification). The dataset-restore race and the
protocol-aware probes from #139/#140/#141 mean the pod should come back to
leader/running on its own and only report Ready once Thread is up.
Post-merge checks to run: GET net1:8081 from the IoT VLAN now fails; REST
listens on the pod IP only; HA reaches otbr:8081 via ClusterIP; otbr
returns to leader/br running and the mesh reconverges; matter-server's
Thread nodes re-subscribe.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Same class as the shelly-manager bypass (#142). matter-server is
multi-homed — net1 is a macvlan on the IoT VLAN for Matter commissioning —
and it binds the :5580 control WebSocket to 0.0.0.0. macvlan traffic
bypasses Cilium policy, so the unauthenticated Matter control API was
reachable from the whole IoT VLAN.
Verified 2026-07-10 from edge-1 (on v30iot):
GET http://10.2.30.117:5580/ -> 200
An unauthenticated IoT client could drive the Matter fabric (commission /
remove / control devices, read fabric state) over that socket.
python-matter-server has a dedicated `--listen-address` flag, independent
of `--primary-interface`. Bind the WebSocket to the pod eth0 address
(--listen-address=$(POD_IP), POD_IP=status.podIP) so :5580 is served only
on the cluster pod CIDR. `--primary-interface=net1` is unchanged, so Matter
commissioning/discovery still uses the IoT VLAN. HA reaches the WebSocket
via the IPv4 ClusterIP Service (ws://matter-server...svc:5580/ws), which
targets the pod IP.
The Service is single-stack IPv4, so binding the IPv4 pod IP loses no
consumer. matter-server has no probes, so there is no probe path to
reconcile (that probe gap is a separate, deferred audit finding).
Verified: kubectl kustomize builds; rendered args carry
--listen-address=$(POD_IP) and the POD_IP downward-API env. Post-merge
checks to run: GET net1:5580 from the IoT VLAN now fails; :5580 listens on
the pod IP only; HA's Matter integration reconnects and all 13 nodes
re-subscribe; commissioning still works (net1 unchanged).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The pod was OOMKilled (exit 137) on 2026-07-09 20:48:46 against the 8Gi
limit. The 8Gi ceiling dates from when HA ran on the 12Gi k8s-edge VM;
it now schedules on the general pool, where the nodes have ~47Gi, so
there is no reason to keep it below the 16Gi the HAOS VM used to give
this workload.
The restart that OOMKill caused also broke roborock, dreame_vacuum and
dwd_weather: a fresh container re-runs HA's uv-based requirement install,
which upgraded cffi 2.0.0 -> 2.1.0 on disk after _cffi_backend 2.0.0 had
already been imported, and every pycryptodome-backed integration then
died on the resulting FFI version mismatch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Authentik gate added in #138 only guards the Traefik ingress path.
shelly-manager also has a Multus macvlan (net1) on the IoT VLAN for mDNS
discovery, it binds 0.0.0.0 (image Dockerfile default
ASPNETCORE_URLS=http://0.0.0.0:8080), and macvlan traffic bypasses Cilium
policy entirely. So the unauthenticated admin UI was reachable directly on
the IoT VLAN, past the gate.
Verified 2026-07-10 from edge-1 (on v30iot):
GET http://10.2.30.122:8080/ -> 200, full Shelly Manager UI
An untrusted IoT client could rewrite Shelly device config, push scripts,
and read stored device admin passwords — exactly what the gate was meant
to prevent. The macvlan predates the gate; #138 closed the front door and
left this side door open.
Pin Kestrel to the pod's eth0 address via the downward API
(ASPNETCORE_URLS=http://$(POD_IP):8080, POD_IP=status.podIP) so :8080 is
served only on the cluster pod CIDR, not on net1. Everything that should
reach the app still does — the IPv4 ClusterIP Service, the Authentik
outpost through it, and the kubelet httpGet probes all target the pod IP.
The Service is single-stack IPv4, so no IPv6 consumer is lost.
mDNS discovery is unaffected: the resolver is a separate ZeroconfResolver
socket set that binds its own multicast sockets on net1, independent of
Kestrel (src/ShellyManager.Web/Shelly/MdnsResolver.cs). net1 keeps L2
presence for `_shelly._tcp.local.` browsing; it just no longer carries the
HTTP server.
Also corrects the stale "static IP" NAD comment — net1 is a DHCP lease
since #135.
Verified: kubectl kustomize builds; rendered env has POD_IP (downward API)
and ASPNETCORE_URLS=http://$(POD_IP):8080; probes still httpGet the named
port, which resolves to the pod IP Kestrel now binds. Post-merge checks to
run: discovery still lists all devices, GET to net1:8080 from the IoT VLAN
now fails, gate still serves via Traefik, pod stays Ready.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The only probe on this container was `pgrep -x otbr-agent`. It caught the
upstream entrypoint's habit of keeping PID 1 alive via `tail -f` after the
agent dies. It could not catch what actually happened on 2026-06-27:
otbr-agent ran for twelve days with `ot-ctl state` = disabled and the
border router stopped, while the pod reported Running 1/1 the whole time.
A live process is not a live Thread mesh.
Add a startupProbe and a readinessProbe, and teach liveness the one
protocol state a restart repairs.
startupProbe ot-ctl's control socket answers. 10s x 60 = 10 minutes,
covering the apt install, the socat TCP<->PTY bridge to
the SLZB radio, and the spinel handshake. While it runs
the other probes are suspended, so the entrypoint gets to
finish activating Thread before liveness may act.
readinessProbe state is leader|router|child|detached. `detached` counts
as ready: the stack is up and attaching, REST still
serves, and treating normal mesh churn as NotReady would
flap HA's OTBR integration for nothing. `disabled` does
not count -- it is the twelve-day failure and the one
state that never resolves on its own.
livenessProbe pgrep, plus: restart when state is `disabled` AND a
dataset is persisted. That combination means activation
never ran, and since #139 a restart re-runs a path that
polls for the dataset and starts Thread. Self-healing.
The dataset guard on liveness is load-bearing. On a clean install there is
no dataset and `disabled` is correct until otbr-dataset-bootstrap seeds one
over `kubectl exec`. Restarting there would fight the Job, which has
already been seen to die with BackoffLimitExceeded against a
CrashLoopBackOff. Likewise, if ot-ctl does not answer, the state check
falls through and liveness rests on pgrep alone -- a wedged control socket
must not trigger a restart storm.
All three probes strip CR. ot-ctl speaks a serial-console protocol and
terminates lines with CRLF, so `ot-ctl state` yields `leader\r`, which
matches no shell glob (#140). curl and wget are absent from this image, so
the probes go through ot-ctl rather than REST.
Removing endpoints affects only HA's OTBR REST integration on :8081.
matter-server does not consume this Service -- it learns the OMR prefix by
RA on its own macvlan interface.
Verified. The three scripts were run against the live pod (state=leader):
startup/readiness/liveness all exit 0. The states that cannot be produced
on a healthy border router were driven with stubbed ot-ctl/pgrep:
state dataset agent | startup readiness liveness
leader yes alive | 0 0 0
router yes alive | 0 0 0
detached yes alive | 0 0 0
disabled yes alive | 0 1 1 <- the bug
disabled no alive | 0 1 0 <- clean install
<wedged> yes alive | 1 1 0
leader yes dead | 0 0 1
`sh -n` clean on both probe scripts as rendered by kustomize; kustomize
builds. Entrypoint untouched.
Caveat: an operator running `ot-ctl thread stop` by hand on a provisioned
node gets the container restarted ~3 minutes later. Deliberate --
`disabled` with a dataset is a fault on this deployment.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#139 landed and did its job: the pod now logs
otbr: persisted dataset found after 18 attempt(s); activating Thread
and `ot-ctl state` reports `leader`, `ot-ctl br state` reports `running`,
with the OMR prefix fd5f:5689:e6dc:1::/64 published again. The dataset
appeared ~36s after ot-ctl began answering, so the old single-shot check
never stood a chance.
But the attach-confirmation block in that same PR then printed:
otbr: WARNING Thread not attached after 60s (state=leader)
ot-ctl speaks a serial-console protocol and terminates lines with CRLF, so
`ot-ctl state | head -1` yields `leader\r` -- 7 bytes. That matches no shell
glob, so the `case` fell through to the warning, and the stray CR is
invisible in the log, making the message read as a flat contradiction.
Verified with `od -c` in the live pod (`l e a d e r \r \n`) and by
`case "leader\r" in leader) ...` failing to match while the tr-stripped
value matches. `bash -n` clean, kustomize builds.
The warning was cosmetic -- Thread attached correctly either way -- but a
health line that says the opposite of the truth is worse than no line.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
otbr has had Thread `disabled` since 2026-06-27. `ot-ctl dataset active`
returns NEST-PAN-339C, the seed Job completed 61 days ago, the RCP answers
(OPENTHREAD/1.3.0.1; CC13XX_CC26XX) -- and the stack was simply never
brought up. The pod had 0 restarts, so it never got a second chance.
The entrypoint waits for `ot-ctl state` to answer, then checks for a
persisted dataset exactly once. otbr-agent answers `state` as soon as its
control socket exists, which is before it has restored the dataset from
/var/lib/thread. The check races the restore and loses:
otbr: no persisted dataset; awaiting bootstrap Job
...after which nothing ever calls `ifconfig up` / `thread start`, because
the seed Job only activates Thread on a clean install. Reproduced exactly
on a restart today, before this fix.
Poll the dataset check for up to 60s instead. Then poll `ot-ctl state`
after `thread start` and log the attach result: `thread start` returns
before MLE has attached, so its exit code proves nothing, and a silent
`disabled` is precisely the failure this block exists to prevent.
Note the `|| true` and the `case` rather than a `[ ... ] && [ ... ] && echo`
chain: this script runs under `set -euo pipefail`, so a bare failing test as
the last statement would kill the container instead of logging a warning.
Verified: `bash -n` clean on the extracted script, `kubectl kustomize`
builds. Merging restarts the pod, which is the real test -- the entrypoint
should now log "persisted dataset found after N attempt(s); activating
Thread" followed by "Thread attached as <state>".
Unrelated but confirmed in passing on today's restart: otbr took its DHCP
lease cleanly under #135/#137 (net1=10.2.30.129, single v4 default via eth0,
sbr rule `from 10.2.30.129 lookup 100`).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
shelly-manager has no authentication of its own. Anything that can reach
:8080 can rewrite Shelly device config, push scripts, and read the stored
per-device AdminPassword values. Until now the only control on it was
network reachability of shelly-manager.home.renner.uno.
Put it behind Authentik using the same proxy-outpost pattern as the
vscode gate (760e187) and paperless before it:
- New proxy provider + application + dedicated outpost
(ak-outpost-shelly-manager-gate) in mode: proxy. authorization_flow is
passkey-stepup-authorization, the same flow vscode-gate uses, so every
authorize requires a passkey.
- New git-owned "Shelly Managers" group (membership in the blueprint,
k.renner only) bound to the application as the explicit allowlist.
- shelly-manager Ingress now backends the gate outpost via an ExternalName
mirror instead of the app Service directly.
- home-assistant CNP gains a scoped ingress rule: only the
shelly-manager-gate outpost pod may reach :8080. Matching scoped egress
CNP added in the authentik namespace.
No skip_path_regex, deliberately. The music-assistant gate died on PWA
assets fetched with credentials:'omit' (manifest, service worker), and
paperless/vscode carve those out. shelly-manager ships neither -- wwwroot
is app.css, favicon.png and lib/ only -- so every asset is a same-origin
credentialed fetch. It is a Blazor Web app whose SignalR circuit at
/_blazor upgrades to a websocket carrying the session cookie, which the
proxy outpost forwards.
This also restores the app. It has been returning 504 through Traefik
since 760e187 merged at 01:20 today: `default-deny-ingress` has
endpointSelector {}, so its traefik -> :8080 grant, written for the
code-server sidecar, was namespace-wide and was in fact the only rule
letting Traefik reach shelly-manager, which also listens on 8080. Removing
that grant to gate vscode took shelly-manager's ingress down as collateral.
Gating it properly is the correct fix rather than re-widening the rule --
the app should not have been reachable unauthenticated in the first place.
Verified: kubectl kustomize builds clean for authentik, home-assistant and
network-policies; the rendered Ingress backends ak-outpost-shelly-manager-
gate:9000, the egress CNP is scoped to shelly-manager:8080, and the two
gate ingress rules cannot reach each other's target.
The Authentik proxy flow, the passkey login, and the SignalR circuit
through the outpost all require a browser test after apply.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PR #135 moved the three v30iot macvlan attachments from static IPAM to
DHCP. That was right about ownership and wrong about routing: a leased
address arrives with a default gateway (DHCP option 3, 10.2.30.1), and
the dhcp IPAM plugin installs it as a *second* default route on net1 at
the same metric as eth0's Cilium-managed one.
Cluster-internal replies then egress the IoT VLAN and are dropped. The
kubelet readiness probe comes from the node address and keeps passing for
a few seconds, which makes the pod look healthy; anything from the pod
CIDR -- Traefik -- times out. shelly-manager went NotReady and its
ingress returned 504.
The old static IPAM sidestepped this by simply never setting a gateway,
as its comment said: "No gateway is set on this attachment: default route
stays on the pod's primary (Cilium-managed) interface, so non-IoT egress
doesn't leak through eth1." A leased address comes with a gateway, so it
has to be confined rather than omitted.
Chain the `sbr` plugin (source-based routing, containernetworking/plugins
v1.5.1, already on every node from cni-plugins-installer) after macvlan.
It moves net1's routes into a routing table selected by net1's source
address, leaving eth0's default alone in the main table.
matter-server's chain becomes macvlan -> tuning -> sbr; otbr and
shelly-manager convert from a single-plugin config to a plugins array.
Only shelly-manager currently runs a post-#135 sandbox, so it is the only
pod actually broken right now. matter-server and otbr still hold their
pre-#135 static addresses and single eth0 default route -- but their NADs
already say dhcp, so they would break the same way on any restart. This
change is a strict improvement for them either way.
UNVERIFIED, called out in the matter-server NAD: sbr's effect on the IPv6
OMR route. Matter CASE sessions to Thread devices need the fd5f::/64 RIO
that OTBR publishes on v30iot. Source-based routing should keep it
reachable for traffic sourced from net1's address, but this has not been
demonstrated. Check `ip -6 route` inside matter-server after its first
restart under this config.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Investigating an unexplained 21-minute full-power heating run on
2026-07-09 09:25 showed the HPMU event log is the one diagnostic
surface open3e never published: vitocal.259 (InfoDtcList) and
vitocal.260 (InfoDtcHistory) were absent from the poll list, so only
the VX3's counterpart (vx3.266) was ever visible.
A one-shot `{"mode":"read","data":[259,260],"addr":"0x680"}` over
open3e/cmnd confirmed both decode cleanly through the vcal codec and
are capped lists — 2 and 9 entries respectively — so the multi-frame
ISO-TP read costs little next to the 87 DIDs already polled at 30 s.
Scope note: this log records faults and info events only
(NormalOperation, LagDeviceDtcReported,
LegionellaProtectionTemperatureReached). It does not record setpoint
or operating-program changes, so it would not have attributed the
09:25 run. It is still the right place to catch recurrences of the
legionella heating-rod cycle and lag-device dropouts.
Read-only: no -w flag, no new write path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The three macvlan NADs on the IoT VLAN (matter-server, otbr,
shelly-manager) used static IPAM with hand-picked addresses .226/.227/
.228, chosen because a ping sweep on 2026-05-13 found them idle. All
three sit inside the UniFi DHCP pool (10.2.30.100-.254), so the UDM
leased them out from under the pods.
The controller has been logging CLIENT_IP_CONFLICT continuously since
2026-05-13:
.226 vs Miele oven (00:1d:63:5a:95:4e) 16 alerts
.227 vs Miele fridge (00:1d:63:87:9d:b1) 8 alerts
.228 vs itself 4 alerts
Nobody noticed because these attachments exist only for mDNS/multicast
L2 presence, and multicast reception survives losing the ARP race, so no
pod ever logged an error. The .228 case involves no real device at all:
Multus randomizes the macvlan MAC per sandbox, so a rolling update had
the terminating and starting pods claiming the same static address under
different MACs.
A ping sweep shows which hosts answer at that instant. It says nothing
about what DHCP will lease later. Relocating the addresses to a quieter
part of the subnet would preserve the actual defect -- an address that no
allocator owns. So remove the addresses instead: let the UDM allocate,
as it does for every other device on this VLAN.
Nothing depends on the values. Verified across the repo, every live
cluster object including decoded Secrets, and Home Assistant's stored
config entries: no consumer references .226/.227/.228, and no UniFi
firewall rule, port-forward, or firewall group mentions them. All three
pods are reached via their ClusterIP Services; matter-server binds by
interface name (--primary-interface=net1) and otbr by
BACKBONE_INTERFACE=net1.
Adds cni-dhcp-daemon to infrastructure/multus, which the dhcp IPAM
plugin requires -- the plugin is only a client of a per-node daemon on
/run/cni/dhcp.sock. The dhcp binary is already on the nodes; the sibling
cni-plugins-installer extracts the whole v1.5.1 tarball. Flux's apps
Kustomization already dependsOn infrastructure, so the daemon lands
before the NADs reconcile, and running pods keep their addresses until
restarted.
No stable MAC is pinned: under DHCP the random per-sandbox MAC is what
lets a rolling update take two independent leases rather than collide on
one reservation.
Also corrects two stale comments: matter-server/deployment.yaml still
described the NAD as pinning .226, and bgp-service.yaml claimed the
home-assistant-iot Service at 10.2.30.80 was retained for hard-coded
clients -- it was retired 2026-05-20 and no LoadBalancer Service holds a
10.2.30.x address any more.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
captive.renner.uno has the identical problem just fixed for udm: UniFi
auto-serves only its A record (captive -> 192.168.8.1, the UDM console)
and forwards AAAA upstream, where the *.renner.uno wildcard returns
proxy.renner.uno. Pin the AAAA to the UDM management ULA fd00::1 so
v6-first clients reach the UDM instead of the public reverse proxy.
Same value as udm because both names resolve to the same UDM console
(192.168.8.1 over v4; fd00::1 is its v6, verified serving the controller).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The UniFi Network integration in Home Assistant intermittently drops to
setup_error with:
Cannot connect to host udm.renner.uno:443 ... TLSV1_ALERT_INTERNAL_ERROR
Root cause is IPv6, not TLS or the UDM. UniFi auto-generates only the A
record for the UDM's own hostname (udm -> 192.168.8.1) and forwards AAAA
queries to its upstream, where the *.renner.uno wildcard answers with
proxy.renner.uno (the public NetBird reverse proxy). HA is IPv6-first, so
it resolves udm.renner.uno to the proxy over v6 and gets its handshake
rejected (unknown SNI -> tlsv1 alert internal error). The integration
recovers only when resolution happens to land on the working A path.
Pin an explicit AAAA to the UDM's own management ULA fd00::1:
- verified serving the controller (cert CN=udm.renner.uno),
- a ULA, so it doesn't rotate like the ISP 2003:d6:* GUAs,
- reachable from the cluster via the node v6 default route (UDM ll gw).
With a local AAAA present, UniFi answers it itself instead of forwarding
upstream, so v6-first clients reach the controller. Because the record
matches the cert CN, TLS verification also passes.
Applied via the unifi_dns_record resource (UniFi local DNS) on tofu apply.
Watch the apply: if the UniFi provider/UDM rejects AAAA record types, this
is the point it surfaces.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
code-server (the HA /config editor sidecar) runs with --auth=none, as
root, mounting the entire /config incl. plaintext secrets.yaml, .storage,
and the git PAT in the PVC remote URL. Until now the only control on it
was network reachability of vscode.home.renner.uno — any LAN client could
read every HA secret and edit config.
Put it behind Authentik using the existing paperless-gate proxy-outpost
pattern:
- New Authentik proxy provider + application + dedicated outpost
(ak-outpost-vscode-gate) in mode: proxy. authorization_flow is the
passkey-stepup-authorization flow (same one Homeassistant/Immich/
Audiobookshelf/Netbird use), so every authorize requires a passkey.
- New "Home Assistant Config Editors" group (git-owned membership,
k.renner only) bound to the app — explicit allowlist for who can read
HA secrets via the editor.
- vscode Ingress now backends the gate outpost (ExternalName mirror)
instead of code-server directly.
- home-assistant CNP tightened: traefik no longer allowed to :8080; only
the vscode-gate outpost pod may reach code-server. Matching scoped
egress CNP added in the authentik namespace.
- code-server is a service-worker app; skip_path_regex carves out its
static/manifest/SW tree to avoid the credentials:'omit' redirect loop
that forced retirement of the music-assistant gate. Exact skip set to
be confirmed post-deploy (see PR checklist).
Verified: kubectl kustomize builds clean for home-assistant, authentik,
and network-policies. The Authentik proxy flow + code-server PWA
behaviour require a browser passkey login test after apply.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Music Assistant persists publish_ip (the literal IP advertised to players)
and bind_ip in /data/settings.json. A prior hotfix pinned both to
10.2.30.137 because MA's auto-detect picks the mgmt default-route IP (eth1
carries routeMetric 4096, so v30iot is never the default route), which IoT
players can't reach.
After MA moved to k8s-2, whose DHCP eth1 lease is 10.2.30.193, the stored
.137 is no longer local. The stream server falls back to all-interfaces and
survives, but the Sendspin server (aiosendspin :8927) has no fallback: it
fails to bind and dies, and its proxy then targets the dead .137 instead of
loopback -> Sendspin/Voice PE players never register.
eth1 is DHCP and MA is not node-pinned, so any stored IP goes stale on the
next reschedule or lease change. Instead of pinning an IP, an initContainer
recomputes the current v30iot (10.2.30.0/24) address from the live interface
on every boot and writes publish_ip=<that IP> and bind_ip=0.0.0.0 into
settings.json before MA reads it. hostNetwork means it sees eth1 directly;
it reuses the MA image (stdlib only) and fails closed if no v30iot address
is present, so MA never comes up advertising a wrong IP.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
security-bot
k.renner
