OpenBao was initialized and configured today (kv-v2 at homelab/, k8s
auth, eso-read policy + role bound to the external-secrets SA); the
store can therefore turn Ready and no longer endangers the post-crds
wait gate. Audit device moves into the server HCL because OpenBao 2.5
rejects API-managed audit devices; log_raw stays off so Loki only ever
sees HMAC-redacted values.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CRgBFn9m8a2ncFMLusikH6
The IoT VLAN advertises a global IPv6 prefix (RA/SLAAC), but the Atom Echo
ran v4-only because ESPHome defaults IPv6 off. Enable it so the device is a
proper dual-stack citizen of the segment and has a real global IPv6.
This is the prerequisite for the DNS-side fix: once the device has a real
AAAA, a `unifi_dns_record` AAAA for shelly-ble-proxy-1.iot.renner.uno can be
pinned to it (same pattern as the existing udm/captive records in
opentofu/dns_renner_uno.tf), so the name resolves to the device instead of
falling through to the *.renner.uno proxy wildcard.
Takes effect on the next USB re-flash (no OTA on this device by design).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Single-node raft OpenBao (5Gi ceph-block, UI at openbao.k.renner.uno,
internal-only ingress) and ESO 2.x. No ClusterSecretStore yet — it can
only turn Ready after manual init/unseal, and committing it now would
freeze the infrastructure-crds wait gate (2026-07-13 incident shape).
Full phased plan incl. per-file checklist and DR runbook in
kubernetes/MIGRATION-sops-openbao.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CRgBFn9m8a2ncFMLusikH6
shelly-manager's native ESPHome-API client (PR k.renner/shelly-manager#12)
does the real Noise_NNpsk0 handshake to the Atom Echo bluetooth_proxy when
it has the api.encryption.key. Provide it:
* shelly-ble-proxy-key.sops.yaml — SOPS Secret holding api-key, the same
value as the ESPHome dashboard's shelly_ble_proxy_api_key.
* shelly-manager-deployment: inject it as Ble__Proxies__0__Key via
secretKeyRef.
Effect: /ble-radios upgrades from a bare TCP :6053 check to a real
handshake + DeviceInfo (device name/version), which also validates the
Noise/frame stack on live hardware ahead of the BLE scan/GATT work.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
After the LXC->in-cluster cutover the gitea scrape target became an in-cluster
pod, which the apps default-deny-ingress CiliumNetworkPolicy drops (it allows
only same-ns / home-assistant / host). Prometheus scrapes time out (context
deadline exceeded) and TargetDown keeps firing. Add a scoped CNP allowing the
monitoring namespace to reach the gitea pod on :3000. Monitoring egress is
unrestricted (its only policy is ingress-only), so this ingress allow is
sufficient.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HQhvu1mrQ7EgwLhrzdNyY7
Follow-up to the LXC retirement sweep:
- remove orphan inventory/group_vars/woodpecker_agents (group already deleted)
- drop the retired gitea-runner (root@10.2.40.214 act_runner) journalctl entry
from .vscode/settings.json
Kept (live): the ci.renner.uno gatus check and the woodpecker-ci@homelab
authorized key, which belong to the in-cluster Woodpecker.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HQhvu1mrQ7EgwLhrzdNyY7
Gitea and Woodpecker now run in-cluster (k8s); their Proxmox LXCs are retired.
Remove the provisioning that rebuilt them and every reference to the dead hosts:
- delete roles gitea-lxc, gitea-runner, woodpecker-agent, woodpecker-server
- delete plays install-gitea, install-gitea-runner, configure-gitea-runner-logging,
install-woodpecker-server, install-woodpecker-agent
- drop [woodpecker_server]/[woodpecker_agents] inventory groups + group_vars
- monitoring: remove the gitea Prometheus scrape job + gitea:22 blackbox target;
repoint the gatus / blackbox-dns / chaos-test / proxmox-dns dig probes and the
dns-overview dashboard panel from gitea.srv.renner.uno to k8s-1.srv.renner.uno
- drop the retired gitea-runner journalctl entries from .vscode/settings.json
Live services keep their public hostnames (gitea.renner.uno, ci.renner.uno),
untouched. Other *.srv.renner.uno hosts untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HQhvu1mrQ7EgwLhrzdNyY7
Gitea moved in-cluster (apps ns StatefulSet); LXC 114 is stopped, so the static
`gitea` scrape target gitea.srv.renner.uno:3000 is down -> firing TargetDown.
Point it at the ClusterIP Service (gitea.apps.svc.cluster.local:3000), which
serves /metrics anonymously (verified HTTP 200 with gitea_* metrics).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HQhvu1mrQ7EgwLhrzdNyY7
Add Ble:Proxies[0] = { atom-echo-1, shelly-ble-proxy-1.iot.renner.uno, 6053 } to
the shelly-manager deployment so the flashed M5 Atom Echo esphome bluetooth_proxy
(now up at 10.2.30.109, ESPHome API :6053 open) shows on the /ble-radios page and
is liveness-probed. Reached over the same net1 macvlan route as the Shellys.
Provisioning THROUGH it is the native ESPHome-API client, still to come.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per how it's actually run: drop OTA and the fallback AP/captive_portal — this is
a fixed, low-value proxy, so updates and any WiFi recovery are done by re-flashing
over USB. Secrets shrink to iot_wifi_ssid / iot_wifi_password /
shelly_ble_proxy_api_key (the api encryption key stays; it's the ble-agent
handoff). README secrets list + flashing note updated to match.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
#159 set SSH_USER=gitea, but SSH_USER is cosmetic (clone-URL display only) and
does not change which login the built-in SSH server accepts — so gitea@ still got
Permission denied. BUILTIN_SSH_SERVER_USER is the setting the built-in server
checks the login name against (defaults to RUN_USER=git); pin it to gitea so the
existing gitea@ remotes (the LXC's non-default SSH user) authenticate with zero
client changes. git@ stops authenticating (single built-in login) — unused here.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HQhvu1mrQ7EgwLhrzdNyY7
The retired LXC ran RUN_USER=gitea, so all clone URLs and git remotes are
gitea@gitea.renner.uno. The rootless image runs as `git`, defaulting SSH_USER to
`git` and rejecting `gitea@` logins (Permission denied). Pinning SSH_USER=gitea
restores the pre-migration convention with no per-repo remote changes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HQhvu1mrQ7EgwLhrzdNyY7
Source-of-truth YAML (+ README) for the M5 Atom Echo dedicated Bluetooth proxy
that lends its BLE radio to shelly-manager's ble-agent for provisioning WiFi-less
Shellys over mOS RPC-over-BLE. Built/flashed via the in-cluster ESPHome Device
Builder (esphome.home.renner.uno).
Lean, dedicated config (no voice — bluetooth_proxy + voice boot-loops the
ESP32-PICO), esp-idf framework, bluetooth_proxy active. The api encryption key is
the handoff to the ble-agent (its BLE_PROXY_NOISE_PSK). The dashboard reads from
its PVC, not git, so these files are the reviewed/backed-up copy kept in sync by
hand.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Moves ci.renner.uno off LXC 231 onto the in-cluster StatefulSet (DB loaded from
the LXC beforehand) and brings the kubernetes-backend agent online, retiring the
docker agents (LXC 910/911). server replicas 0->1; Service ExternalName->ClusterIP
with http:8000 (UI) + grpc:9000 (agent<->server); agent replicas 0->1.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HQhvu1mrQ7EgwLhrzdNyY7
Moves gitea.renner.uno and git-over-ssh onto the in-cluster StatefulSet, which
has been loaded from LXC 114 and verified (data intact, SSH host-key identity
preserved, master at the revision Flux expects). gitea Service ExternalName ->
ClusterIP; adds the gitea-ssh IngressRouteTCP binding the Traefik entrypoint
(added in phase 1) to :2222. Woodpecker stays on the LXC (server replicas 0,
Service ExternalName) — its cutover is a separate migration.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HQhvu1mrQ7EgwLhrzdNyY7
woodpecker-server envFrom's woodpecker-secret; without it the pod crashloops and
holds the apps Kustomization (wait:true) red. Encrypted with the repo age
recipient (kubernetes/.sops.yaml); Flux decrypts it via the apps Kustomization's
sops provider. Values (AGENT_SECRET/GITEA_CLIENT/GITEA_SECRET) lifted from the
running CT 231 woodpecker-server.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HQhvu1mrQ7EgwLhrzdNyY7
Stand up Gitea + Woodpecker as in-cluster workloads beside the existing
Proxmox LXCs WITHOUT flipping any traffic. Both Services remain ExternalName
-> the LXCs, the Woodpecker agent ships replicas:0, and the gitea SSH
IngressRouteTCP is absent. Merging this changes nothing user-facing — Flux
brings up empty-PVC pods that get loaded and verified out-of-band before a
phase-2 PR flips the Services to ClusterIP.
Gitea: gitea/gitea:1.21.4-rootless (its entrypoint runs environment-to-ini,
so do NOT override it), built-in SSH server on 2222, forced GITEA__log__ROOT_PATH,
host-key reuse. Traefik gains an additive gitea-ssh TCP entrypoint (:22 on the
shared VIP). Woodpecker: kubernetes backend (build pods in ns woodpecker-ci on
ceph-filesystem RWX, privileged-plugin allow-list + PodSecurity label), agent
replicas:0; SOPS secret is created+committed at pre-flight, never kubectl-applied.
Cutover is fully GitOps: flux suspend + kubectl exec/scale data load, verify
via port-forward, phase-2 flip with a repo-resync; break-glass = repoint the
Flux source to the GitHub mirror (known-good tag before-k8s-migration). See
kubernetes/apps/MIGRATION-gitea-woodpecker.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HQhvu1mrQ7EgwLhrzdNyY7
Adds the ESPHome Device Builder (ghcr.io/esphome/esphome:2026.6.2, :6052) to the
home-assistant namespace so ESPHome devices — starting with an M5 Atom Echo BLE
proxy for shelly-manager's BLE provisioning — can be built, browser/USB-flashed
(ESP Web Tools / Web Serial), and config-managed in-cluster.
v1 deliberately runs with NO macvlan. Unlike shelly-manager it needs no L2
presence on the IoT VLAN: initial flashing is browser-side over USB and builds
are offline. That also sidesteps the macvlan-on-0.0.0.0 hole (a gated pod with a
macvlan bypasses Cilium policy and exposes its unauthenticated UI on the IoT VLAN
— confirmed against shelly-manager/matter-server/otbr). Live status / logs / OTA
would need cluster→v30iot reachability (a separate, still-open item); a macvlan
can be added later if wanted.
Mirrors the shelly-manager gate pattern exactly:
- home-assistant: esphome deployment (Recreate, RWO 10Gi ceph-block PVC for the
PlatformIO build cache), service, gate ExternalName, and a Traefik ingress that
backends the Authentik esphome-gate outpost (not esphome directly).
- authentik: an esphome-gate proxy provider + application + dedicated outpost
(passkey step-up, "ESPHome Managers" group, git-owned membership) in
blueprints-configmap.yaml, plus esphome-gate-egress CNP (outpost → :6052).
- network-policies: the home-assistant default-deny-ingress grants ONLY the
esphome-gate outpost → :6052. :6052 is unique (no collision with the shared
:8080 the vscode/shelly gates ride on).
Validated: kubectl kustomize builds cleanly for the home-assistant, authentik,
and network-policies kustomizations, all rendering the new resources.
Post-deploy checks: confirm the dashboard has no credentials:'omit' service
worker/manifest that 302-loops at the gate (add skip_path_regex if so); confirm
the image serves the dashboard on :6052 with the default entrypoint.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Both install-netbird-server.yml (VPS-1) and install-netbird-proxy.yml (proxy-1)
apply the netbird-proxy-watchdog role, but neither deploy pipeline's path filter
watched ansible/roles/netbird-proxy-watchdog/**. So the flood-watchdog (#153)
did not auto-deploy on merge and had to be triggered manually. Add the role to
both filters so future watchdog changes redeploy both hosts on merge.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The reverse-proxy wedges when an ephemeral NetBird routing peer (a k8s ingress
pod) is DELETED: its route manager pins the now-gone peer and never re-selects
a healthy HA-group peer, serving errors for every domain behind that route
until restarted. Upstream bug, unfixed as of netbird v0.74.4 (netbirdio/netbird
#3976; dnsinterceptor disables HA for an already-routed IP + recalculateRoutes
aborts on the dead peer's teardown error). A restart is the only reliable clear.
The existing URL-probe watchdog does eventually catch it, but far too slowly:
its 127.0.0.1 probe stays warm for ~3 min after the wedge starts, then needs 3
consecutive 60s rounds -- so it took 5-9 min on 2026-07-11, and both proxy
members wedged so the public outage lasted until the slower one recovered.
The wedge floods "couldn't fetch peer state" in the proxy's own log the instant
it happens, minutes before the probe notices. That flood is an unambiguous
fingerprint: 14 days of logs show it ONLY in bursts of tens per minute, never a
benign trickle. So add a persistent follower that restarts the proxy on the
first matching line.
Trade, chosen deliberately: ~1/6 of these bursts self-recover (the re-select
race sometimes wins) and get an unnecessary restart -- but a proxy restart is
near-instant, so a rare spurious restart beats a multi-minute outage.
- new netbird-proxy-flood-watchdog.sh (docker logs -f | grep, restart on match)
- new netbird-proxy-flood-watchdog.service (Type=simple, Restart=always)
- shares LAST_RESTART_FILE + cooldown with the probe watchdog (no loops, no
double-restart); URL-probe watchdog kept unchanged as the backstop.
Deploying only adds a systemd service; it does not touch the proxy or ingress,
so it can't itself cause an outage.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Every netbird image bump (image-automation, ~weekly) rolls this DaemonSet and
takes public ingress (home.renner.uno, mesh-only via the HA route group) down
for ~1 min — observed 2026-07-11 08:58 on the uptime monitor, correlated exactly
with the 3 peers rolling at 06:56:18/39/58 UTC. It is NOT the IPS/firewall
(the UDM was idle; the "intrusions" were the admin laptop hitting its own
ingress) — it's the roll itself failing over slowly.
Root cause: the redundant roll (maxUnavailable:1, 2/3 peers always up) should be
seamless, but two gaps break the failover:
- No preStop drain: a rolled peer is only SIGTERM'd, so the mesh keeps the
dead peer in the route group until a keepalive timeout (~25-30s); traffic
riding through it black-holes until then. Compounded by ephemeral
enrollment (emptyDir) — the restarted pod enrolls a NEW peer and the old one
lingers until reaped.
- No minReadySeconds: the roll advances to the next node as soon as
`netbird status --check ready` passes (seconds), faster than the mesh
re-converges onto the freshly-enrolled peer — so the HA route is degraded
for the whole ~1-min roll.
Fix:
- preStop `netbird down` — deregister the peer from management before the
process dies, so the HA route fails over to the other per-node peers
immediately instead of after a timeout.
- minReadySeconds: 45 — hold each new peer Ready until the mesh has converged
onto it before rolling the next, so >=2 peers are always actually serving.
Keeps auto-updates on; turns the surprise ~1-min outage into a seamless roll.
Validation: watch the uptime monitor across the next roll (natural image bump
or a deliberate `kubectl rollout restart ds/netbird-ingress`).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
otbr is meant to be an additive co-border-router on the existing Nest Thread
network (it's seeded with the Nest dataset to join the same partition). In
practice it does not merge cleanly: its SLZB-06 radio is RF-isolated enough
that otbr elects itself leader of a rival partition, and the network churns
(OMR prefix observed changing fd5f -> fdd5, leader role flapping). The two
same-network partitions then bridge only over the Ethernet backbone, which
carries a ping but is too slow for the large fragmented Matter CASE handshake:
every device stays mDNS-discoverable while CASE times out. Scaling otbr to 0
re-merges the mesh onto the Nest hubs and Matter recovers.
Explicitly NOT a root-cause of a config bug. The DHCP/sbr routing changes
(19a4482, df61407) were suspected and investigated: matter-server's IPv6 path
to the mesh works (reaches devices, ping ok), sbr's v6 table 100 is an inert
orphan (no v6 rule references it), and otbr's eth0 IPv6 default is outranked by
net1's pref-high RA default. The instability sits at the Thread partition
layer, not in the pod's routing table. Re-enable only after the co-BR
integration is verified to MERGE (one partition, one OMR, otbr's router table
shows the Nest routers) rather than split.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
k.renner
claude-botandClaude Fable 5
security-bot