User confirmed the legacy token has operator scope (re-probed
/api/v2/backup/kv: 200 with bolt content). The earlier 401 from
influx backup must have been transient or my earlier probe was
malformed.
Drop the lossy streaming-via-pivot design entirely. CSV→LP conversion
in `influx write --format csv` doesn't escape commas/newlines/embedded
quotes; HA's data routinely contains all three; filtering string-valued
points to dodge the parser bugs is data loss, which the user correctly
rejected for a migration.
Restore the simple shape:
1. influx backup --host=legacy --bucket=suedstr28 /staging/
2. influx bucket delete --name suedstr28 (chart placeholder)
3. influx restore --host=local --bucket=suedstr28 /staging/
Lossless because backup ships the bolt+TSM bytes directly; restore
replays them into a freshly-onboarded instance. The bootstrap-secret
admin token stays the live token (no `--full`).
Sentinel/resume logic from d72fb21 is gone — single-shot backup
either succeeds or fails atomically; nothing to checkpoint.
Per-day-chunk loop also gone.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- opentofu: attach Proxmox `x710-port1` mapping as hostpci1 on the
three K8s controlplane VMs (jupiter/saturn/neptune).
- talos patches: add a second Device per node selecting the i40e
driver, with static /29 addresses (10.99.0.1/2/3), MTU 9000, no
routes — cluster-internal only.
- CephCluster: switch to `provider: host` and declare `addressRanges`
for both public (10.2.40.0/24) and cluster (10.99.0.0/29). Rook
v1.18 docs are explicit that the default CNI provider ignores
addressRanges, so host networking is required to consume the new
link for OSD↔OSD replication.
Activates on the next rebuild-k8s pipeline run.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Third serialization edge case in the CSV pipeline at day 2024-10-21:
Error: ... unable to parse 'state,domain=sensor,
entity_id=sarahs_ipad_geocoded_location,...,
Location_str="[49.48876953125, 11.762847067710858]",
... state="Villenstraße 3': unbalanced quotes
`influx write --format csv` doesn't escape commas/newlines/embedded
quotes inside string values when generating line protocol. HA's
string-valued points routinely contain all three: markdown release
notes, JSON-style geocoded locations, multi-line attribute text.
Filtering by specific entity / domain / field name is whack-a-mole —
every new data shape exposes a new parser break.
Replace the per-incident filters (`_field == "source"` for the
pivot collision is kept; `domain == "update"` for newline release
notes is removed) with a typeOf-based whitelist:
filter(fn: (r) => r._field != "source"
and typeOf(v: r._value) != "string")
Numeric metrics — temperature, humidity, counters, boolean-as-int —
pass through unchanged. String-valued points (HA `state` text mirrors,
entity-attribute strings, geocoded locations) are dropped wholesale.
Grafana's time-series queries are exclusively numeric, so the loss
doesn't affect the dashboards. State-text history is lost; it was
already awkward to query and is mirrored elsewhere (HA recorder).
Sentinel resumption from d72fb21 stays in place — restart picks up
where the previous attempt left off.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Job has been running 2h+ on per-day chunking with one new failure
mode: HA's `update` domain entities carry `release_summary_str` fields
containing multi-line markdown release notes. `influx write --format
csv` rejects the line:
Error: failed to write data: 400 Bad Request: unable to parse
'state,domain=update,entity_id=studio_code_server_update,...,
release_summary_str="## What's changed\n...
(InfluxDB's annotated-CSV nominally handles RFC4180 quoted multi-line
strings; the v2.7 `influx write --format csv` parser doesn't.) Per-day
chunking surfaces this as a one-day failure rather than corrupting a
larger window.
The script then dies via pipefail; kubelet OnFailure restarts the
container; loop replays from 2018 (no checkpointing); same bad day
fails again; backoffLimit gets eaten by retracing empty days that
already-succeeded.
Two fixes:
1. Filter out `domain == "update"` before pivot — these entries are
metadata toggles (update-available y/n, version strings, release
notes) not real time-series, safe to drop wholesale. `_field ==
"source"` filter from 614533c stays.
2. Sentinel file `/staging/last-completed-day` holds the last `$NEXT`
that succeeded. Script reads it on startup and resumes from there;
writes it after each successful chunk. A kubelet restart now picks
up where the previous attempt left off rather than retracing from
2018. The PVC survives pod restarts, so this works across the
container's full life — sentinel only resets when the Job itself
is deleted (which is also when we want to re-seed from scratch).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per-day chunking from 106d551 ran successfully for 39 minutes (got
through 2018-01-01 to 2024-06-03) before hitting:
runtime error @3:13-3:83: pivot: value "source" appears in a column
key column, but a column named "source" already exists; consider
renaming "source" to something else before pivoting
HA writes `source` as both a tag (integration name, e.g. "Home
Assistant") AND, on some entities, as a field (entity attribute) on
the same measurement. pivot() turns _field values into column names,
so when it processes a row with _field=source it tries to add a
"source" column — but the existing tag-derived "source" column is
already there. Pivot errors out.
Drop the field via `filter(fn: (r) => r._field != "source")` before
pivot. Tag `source` is preserved (more useful — present on every
point, drives Grafana datasource filtering); the field `source` is
lost (rare, unique-per-entity attribute). Net data shape stays
queryable for HA's downstream consumers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
806e979 (per-month) still EOF'd at 2024-04 — first dense month for HA
telemetry, pivot of one month at that density overwhelms legacy's
memory. Drop to per-day chunks (~3300 iterations across 9 years,
mostly empty days no-op'ing in <1 s; dense days under 1 MB pivoted).
Per-year chunking from a315ba6 still EOF'd on the current year. With
~9 years of HA telemetry the early years (2018–2024) were near-empty
and each completed in <1 s, but the full-data year (2024–2025 / current)
hit the same memory ceiling on legacy that the unchunked version did
— pivoting one year of HA metrics still doesn't fit in legacy's RAM
and the connection drops with `Post .../query: EOF`.
Switch to per-month chunks (~108 chunks for 9 years). Each pivot is
~1/12 the data so legacy holds it. Empty months no-op in <1 s. Loop
uses `date -d "+1 month"` to walk calendar months including the
in-progress one.
Pipefail still on, retries still upsert-safe, K8s Job Complete still
the post-success guard.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Last run failed every retry with `sh: 1: set: Illegal option -o pipefail`.
The influxdb:2.7.4 image's `/bin/sh` is dash (debian's POSIX shell);
`pipefail` is a bash extension. Switch the seed container's command
to `bash -c`. The init container stays on `sh` — it has no pipes and
doesn't need pipefail.
Last run completed with exit 0 but the actual stream got cut off:
Error: failed to execute query: Post ".../api/v2/query?org=renner": EOF
[21:50:31] Seed complete.
Two problems:
1. Pivoting the full bucket in one Flux query materialises the entire
result set in legacy's memory before flushing. A multi-year HA-
metrics bucket exceeds what legacy can hold; the connection drops
mid-stream with EOF. Subsequent verification showed *some* data
landed but coverage was unclear (count() queries against legacy
were timing out at 60s afterwards — server slow under load).
2. Script had `set -eu` but no `pipefail`, so `influx query`'s non-
zero exit didn't propagate through the pipe. `influx write` saw
the truncated stream, returned 0, script logged "Seed complete."
K8s marked the Job Complete; Flux saw success and stopped trying.
A genuinely broken transfer was reported as a successful seed.
Fixes:
- Loop the pivot per calendar year (2018 → current). Each year's
pivot produces a much smaller result set; legacy holds it without
OOM-ing. Year boundaries don't need to align with anything in
InfluxDB — points retain their original timestamps end-to-end.
- `set -euo pipefail` so `influx query` EOF on any year aborts the
loop with non-zero exit, the Job goes to Failed, the K8s backoff
retries.
- Drop the script-side `query|grep ^,_result` idempotency check.
Two reasons: (a) InfluxDB writes are upserts on
(timestamp,measurement,tagset,field), so a partial run leaves the
bucket in a state where re-running just re-asserts the same points
— no corruption, no duplicates; (b) the previous check would short-
circuit after a partial-success leaving later years unprocessed.
The K8s Job's `Complete` state is the actual idempotency guard:
Flux reconcile sees Complete and won't recreate.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
818c71b's stream pipeline failed with `line N: no field data found` on
every row. `influx query --raw` emits annotated CSV in *long* format
(one row per _time/_measurement/_field/_value tuple, with _field as a
plain string column and _value as the typed value). `influx write
--format csv` expects *wide* format (one row per _time, each distinct
field as its own column annotated as a field datatype). It saw _field
as a string column and _value typed but had no way to combine them
into a field-value pair, so every line was rejected.
Add `pivot(rowKey:["_time"], columnKey:["_field"], valueColumn:"_value")`
inside the Flux query to collapse the long-format rows into wide
format, plus `drop(columns:["_start","_stop"])` to strip the range-
window columns that `influx write` rejects. This is the standard
recipe for v2-to-v2 streaming export+import via the Flux API.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pipeline 876's seed Job kept failing once the SSH/CLI-version issue was
solved, this time with:
Error: failed to backup metadata: failed to download metadata
snapshot: 401 Unauthorized: read:authorizations is unauthorized
The Grafana token reused as legacy-token has all-access on the renner
org but is *not* an operator (system-wide) token. `influx backup` —
even with `--bucket NAME` — first downloads the global metadata
snapshot, which requires read:authorizations at the system level and
returns 401 for every non-operator token.
Drop `influx backup`/`influx restore` entirely. Stream the bucket
through the v2 HTTP query API instead:
influx query --host=<legacy> --raw 'from(bucket:X) |> range(start:-100y)'
| influx write --host=<local> --format csv --bucket X
InfluxData-annotated CSV is both `query --raw`'s output format and
`write --format csv`'s input format, so the pipe is direct — no
intermediate transform, no staging file (kept the staging PVC mount
for now in case we need it later for chunking on huge buckets).
read:buckets at org-scope is enough for `influx query`, write:buckets
at org-scope is enough for `influx write`; neither needs operator
scope. Pipe keeps memory bounded regardless of bucket size.
Trade-off: bucket-level metadata (retention, shard-group duration) is
NOT carried over from legacy. Chart's adminUser already pins
`retention_policy: "0s"` (forever), matching the legacy suedstr28
config, so this is a no-op for the only bucket we currently seed. Any
future bucket with non-default retention/sgd would need to be set in
the chart values explicitly.
Also drops the `influx bucket delete` step that the previous restore-
based flow needed (since `influx restore` cannot restore into an
existing bucket); `influx write` happily appends into an existing
bucket so the chart-bootstrap-created suedstr28 placeholder is the
write target directly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Last commit's `sops -e --encrypted-regex '^(data|stringData)$'`
output a malformed JSON-shaped file with only `data` + `sops` keys at
the top level — kustomize couldn't see kind/metadata, so seed KS
crashed in the build step:
accumulating resources from 'legacy-token.sops.yaml':
missing Resource metadata
Re-encrypt with full-field encryption (`--input-type yaml --output-type
yaml`, no regex) to match the existing pattern in
grafana-influxdb-token.sops.yaml. apiVersion / kind / metadata /
stringData all become ENC[...] strings; Flux's kustomize-controller
decrypts before kustomize parses, so the original Secret manifest is
visible to the build.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pipeline 876's `influxdb-seed` Job failed inside the `influx backup`
step on legacy. Logs (cluster Loki):
unknown arguments: backup --bucket suedstr28 /tmp/influxdb-seed
Usage of influx:
-version
-host 'host name'
...
The `influx` binary in legacy's `$PATH` is the InfluxDB v1 client
(flag-style CLI). Legacy's *server* is v2.7.4 (verified via
`X-Influxdb-Version: v2.7.4` on `/ping` and `version: v2.7.4` from
`/health`), but a leftover v1 client package shadows the v2 CLI in
`$PATH`. The seed Job's `ssh root@legacy "influx backup --bucket NAME
/tmp/..."` therefore hit the v1 CLI, which has no `backup` subcommand.
Pivoting away from SSH-driven backup. Both source and target are v2,
so `influx backup --host=URL --token=TOKEN --bucket=NAME PATH` works
remotely. Run `influx backup` from inside the seed Job pod (which has
the v2 CLI baked into the influxdb:2.7.4 image) against the legacy v2
HTTP API; the CLI streams the backup directly into the staging PVC.
Then `influx restore --bucket NAME /staging/` runs against the
chart-bootstrapped instance through its API. No SSH, no apt-get, no
rsync, no shell on legacy.
The Job now needs an operator-scope token for the legacy server.
Reusing the existing operator token Grafana uses (verified
operator-scope by hitting `/api/v2/authorizations` and getting 200,
which datasource-only tokens cannot do) — repackaged as
`influxdb-legacy-token` in the influxdb namespace via a new
SOPS-encrypted Secret. Same plaintext value, just scoped to the seed
Job alongside the chart's `influxdb-bootstrap` Secret. (Cross-namespace
reuse approved by the user.)
Earlier securityContext.runAsUser:0 on the pod template stays — the
v1 image has no USER directive so it ran as root anyway, but the
explicit setting is defensive in case upstream slim variants change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pipeline 876 hit BackoffLimitExceeded on `influxdb-seed`. Container
restarted ~7 times within ~6 min then the Job died — root cause:
`influxdb:2.7.4` ships with `USER influxdb` (uid 1500) by default, so
the seed container's `apt-get update` / `apt-get install openssh-client
rsync` and the `install -m 600 /root/.ssh/id_ed25519` step all hit
EACCES on the very first run and exit non-zero. CNPG pg-seed Jobs
don't see this — the `postgres:16` image runs as root by default.
Set `pod.securityContext.runAsUser: 0` (+ `runAsNonRoot: false`) so
both the init container's `influx` polls and the main container's
apt-get / SSH setup work. The InfluxDB CLI doesn't care about uid;
running as root inside the seed namespace is no worse than the pg-seed
pattern this is mirroring.
Also annotate the Job with `kustomize.toolkit.fluxcd.io/force: enabled`
so Flux will delete-then-recreate on any future spec change. Job
specs are immutable, so without `force` Flux silently no-ops the
re-apply and a Failed Job stays Failed forever (which is exactly what
just happened — the previously broken spec stuck around because Flux
couldn't update the immutable fields).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The current rsync-into-data-PVC seed shape silently breaks the chart's
admin-token bootstrap. `data-seed-job.yaml` rsyncs `/var/lib/influxdb`
from VM 108 directly into the data PVC, including `influxd.sqlite`
(InfluxDB's onboarding state). When the chart starts on that PVC the
upstream image's entrypoint sees an already-onboarded instance and
short-circuits `influx setup`, so the DOCKER_INFLUXDB_INIT_* values
sourced from the influxdb-bootstrap Secret never apply. Result: the
live admin token is the legacy VM 108 token (unknown to the cluster's
manifests) and every API call using the bootstrap-secret token returns
401 — observed in pipeline 873 with three "token required" /
"authorization not found" log lines from clients hitting the API.
Restore the original separation between transport and import, mirroring
the CNPG pg-seed pattern (ak-pg-seed, im-pg-seed, pprl-pg-seed):
- Chart's PVC template provisions an empty `influxdb-influxdb2` PVC.
HelmRelease drops `persistence.useExisting: true`/`name:` so the
chart adopts nothing. influxd's entrypoint runs `influx setup`
against the empty data dir using the bootstrap-secret values, and
the bootstrap-secret token becomes the live admin token.
- Sibling `influxdb-seed` Job (this commit) waits for the API +
bootstrap admin token to be live (init container, two-stage check
mirroring pg-seed's `wait-for-cluster`), then SSHes to legacy and
runs `influx backup --bucket suedstr28` into a temp dir, rsyncs
the directory into a new staging PVC `influxdb-seed-staging`,
deletes the empty placeholder bucket the chart's `influx setup`
just created (since `influx restore` cannot restore into an
existing bucket), and calls `influx restore --bucket suedstr28`
against the chart's API. No `--full`, so the bootstrap-secret
admin token is left untouched.
CLI flag set verified against the v2.7 docs: `influx backup -b/--bucket
NAME PATH --host URL --token TOKEN` (env: INFLUX_HOST/INFLUX_TOKEN/
INFLUX_ORG); `influx restore PATH --bucket NAME` (cannot restore into
existing bucket — must `--new-bucket` rename or delete first).
File layout mirrors CNPG: HelmRelease + bootstrap-secret + seed-job +
staging-pvc + seed-ssh all live in `kubernetes/seed/influxdb/`. The
per-app KS at `kubernetes/clusters/homelab/influxdb.yaml` and the
`kubernetes/apps/influxdb/` directory are removed; the chart now
reconciles in the seed phase alongside the seed Job. At cutover this
whole directory moves to `kubernetes/apps/influxdb/` along with the
CNPG cluster moves, when the seed phase is decommissioned.
The `.woodpecker/migrate-influxdb.yml` manual-trigger pipeline still
references the deleted `kubernetes/migrations/influxdb/seed-job.yaml`
path AND assumes the old scale-STS-to-0+rsync-into-PVC shape; needs a
separate rewrite to match this design (chart must stay UP for `influx
restore` to reach the API). Not addressed in this commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both NICs on k8s-edge default-DHCP at the Talos-default routeMetric=1024,
and the IoT VLAN's DHCP server also returns a default gateway, so two
same-priority default RouteSpecs landed (10.2.40.1 via eth0, 10.2.30.1
via eth1). Kernel installed one and rejected the other with EEXIST;
Talos's RouteSpecController retried the loser every ~30-70s, spamming
the controller-runtime log on every k8s-edge boot.
Talos v1.12.6's DHCPOptions exposes only routeMetric/ipv4/ipv6/duidv6
(verified against pkg/machinery/config/types/v1alpha1/v1alpha1_types.go
at the v1.12.6 tag) so the lever is metric. Bumping eth1's to 4096
keeps both defaults at different metrics — eth0 wins egress at 1024,
eth1 is a never-used fallback. Connected /24 on eth1 is unaffected, so
Cilium L2-IPAM (HA-iot/mosquitto-iot announcements), OTBR/matter-server
macvlan parents, and music-assistant's host-net L2 reach to renderers
keep working.
Also rewrites the opentofu comment for the second NIC to capture both
load-bearing tenants (music-assistant L2 locality for audio-sync
latency; OTBR/matter-server macvlan parent for Thread/Matter L2
presence) instead of citing the discovery protocols as the cause.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Captures the 2026-05-02 clean-rebuild outcome (pipeline 871 SUCCESS,
65m37s, e2e 67/67) and a pre-drive-replacement fio baseline against
ceph-block (RBD) and ceph-filesystem (CephFS) so the post-replacement
run has numbers to compare against. Manifest is reusable —
`kubectl apply -f .rebuild-report-2026-05-02/ceph-baseline-fio.yaml`
reruns the same workload set after the drive change.
Files:
.rebuild-report-2026-05-02/pipeline-871-final-report.md
.rebuild-report-2026-05-02/ceph-baseline-fio.yaml
.rebuild-report-2026-05-02/ceph-baseline-summary.md
.rebuild-report-2026-05-02/ceph-baseline-results.txt
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
influxdb: align seed-target PVC name with chart-consumed PVC
Pipeline 871 surfaced a PVC naming mismatch: the seed Job populates
`data-influxdb-influxdb2-0` but the influxdb2 Helm chart's standalone
PVC template would otherwise create a fresh `influxdb-influxdb2` and
the StatefulSet would mount the empty one. Switch to the chart's
`persistence.useExisting` mechanism so the chart adopts the seeded PVC
by name. See `.rebuild-report-2026-05-02/influxdb-data-relocation.md`
for the live-cluster sequencing if the running pod has already mounted
the empty chart-managed PVC.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
cni-exclusive=true in the Talos-bootstrap-time cilium ConfigMap caused
cilium-cni to rename multus's 00-multus.conf to .cilium_bak during
first boot, which is why the 2026-05-02 rebuild left OTBR's net1
attachment broken (otbr-agent died with CreateIcmp6Socket: No such
device, dataset-bootstrap Job exhausted backoffLimit). The Flux-managed
HelmRelease already uses cni.exclusive=false, so this just brings the
inline-manifest install into lockstep — kubelet picks multus-shim and
the k8s.v1.cni.cncf.io/networks annotation is honored from then on.
The fix takes effect on the next rebuild only; the live cluster still
has 00-multus.conf.cilium_bak from the bad first boot. See
.rebuild-report-2026-05-02/otbr-net1-diagnosis.md for the live-cluster
remediation options (deferred per the hands-off rebuild-test policy).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pipeline 871 surfaced a PVC naming mismatch: the seed Job wrote to
`data-influxdb-influxdb2-0` while the chart's StatefulSet mounted a
fresh `influxdb-influxdb2` (the influxdb2 chart's
`persistent-volume-claim.yaml` template generates `<release>-influxdb2`,
not the `data-<sts>-N` pattern that volumeClaimTemplates would). Result:
32.81 GB of legacy data orphaned post-rebuild, pod ran on ~196 KB of
fresh bootstrap.
Fix: set `persistence.useExisting: true` and
`persistence.name: data-influxdb-influxdb2-0` in the HelmRelease, which
tells the chart to skip its PVC template and consume the
seed-populated PVC by name from the StatefulSet's
`volumes[].persistentVolumeClaim.claimName`. Future rebuilds will
therefore seed the correct PVC and the pod will see the data
immediately.
Comments on both `helmrelease.yaml` and `seed/influxdb/pvc.yaml` were
also wrong about the chart using volumeClaimTemplates; updated to
reflect the actual chart mechanics.
See .rebuild-report-2026-05-02/influxdb-data-relocation.md for the
one-shot procedure to relocate the currently-orphaned data on the
live cluster.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the single-host k8s-1.srv.renner.uno:6443 endpoint with the
load-balanced k8s.renner.uno:6443 (10.2.40.60) so the control plane is
reachable through the VIP regardless of which CP node is up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
rsync's --info=progress2 emits progress lines via carriage returns,
which container log capture concatenates into one giant unreadable
blob. Switch to --info=stats2 + --human-readable: just the final
summary line per rsync, perfectly tail-able. Live progress is still
observable on demand via `kubectl exec ... du -sh /<mount>`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Upstream's 100m / 50Mi limits left multus-daemon getting SIGKILLed
mid-CNI-ADD on the edge node during a clean rebuild. Bootstrap
schedules home-assistant, matter-server, otbr, and music-assistant-
data-seed concurrently on edge; each pod's CNI ADD chains through
this single multus-daemon → cilium-cni delegate. Under that storm
the daemon hits its 50Mi cap and gets killed, CRI logs report
`multus-shim failed (add): netplugin failed with no error message:
signal: killed`, and containerd is left holding stale sandbox-name
reservations that wedge the affected pods until full reset.
Bump request to 128Mi / 100m, limit to 256Mi memory, drop CPU limit
(CNI ADD is bursty and CPU throttling adds latency where it hurts).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The cni-plugins tarball includes a `./` directory entry whose uid/gid
(1001:127) gets applied to /opt/cni/bin during extract. After that,
Cilium's apply-sysctl-overwrites init container fails to cp into
/hostbin with "Permission denied" — the init runs root without
DAC_OVERRIDE, so it can't write to a directory it doesn't own.
The previous cluster's Cilium kept working because its pods were
initialised before the chown happened and never re-rolled. On a
clean rebuild, the helm-managed daemonset rollout replaces the
Talos-inline pods after cni-plugins-installer has corrupted the
ownership, and the new pods crashloop on apply-sysctl-overwrites.
Always chown the directory back to 0:0, including on already-installed
nodes (re-architect the early-exit so the chown still runs). Idempotent.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The HAOS matter_server addon build.yaml pins
`build_from: ghcr.io/matter-js/python-matter-server:8.1.2` — a different
registry from the `home-assistant-libs/...` one I had been tracking.
home-assistant-libs/python-matter-server stopped at 8.1.0; matter-js
publishes 8.1.1 and 8.1.2.
Importing the OLD HAOS addon's /data into a deployment running a
different SDK build was producing a different `compressed_fabric_id`
on boot, so matter-server landed on an empty per-fabric file instead
of the imported `10489438502500755905.json` (11 nodes). Aligning to
the same image series HAOS uses keeps the operational identity stable
across the migration.
ImageRepository updated to track the matter-js registry so flux's
image automation continues to bump us in step with HAOS.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Need to see which compressed fabric ID matter-server computes from the
imported chip.json so we can compare against 10489438502500755905 (the
OLD HAOS per-fabric file). storage.py logs that filename at DEBUG
level. Revert this once the migration question is settled.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Same shape as HA Core's `seed` init at deployment.yaml:64. On first
pod startup the init container streams the most-recent backup tar from
s3://k8s-migration/home-assistant/, pulls just core_matter_server.tar.gz
out of the outer tar (≈4.5 MB, no need to stage the full 1 GB), then
decrypts the securetar-encrypted addon archive with the existing
home-assistant-backup-password and copies the addon's persistent /data
contents over the freshly-initialised PVC. Idempotent via
/data/.haos-import-applied sentinel.
The decryption step needs the `securetar` Python lib — alpine has no
dedicated CLI for it — so haos-import-script.yaml ships a 6-line
helper as a ConfigMap, mounted under /scripts and called as
`python3 /scripts/import.py <input> <output> <password>`. securetar's
public surface (verified against the lib in the HA Core image at
/usr/local/lib/python3.14/site-packages/securetar/__init__.py:909):
SecureTarFile(path, password=str, gzip=True) → tarfile.TarFile.
Cutover note: bring HAOS matter-server down before triggering the fresh
backup that this init consumes, otherwise the imported chip_counters
will lag the running addon's counters and devices will reject some
operations under Matter's counter-rollback protection.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The otbr Flux Kustomization was missing decryption.provider=sops, so
the dataset Secret was applied to the cluster as raw ENC[...]
ciphertext instead of being decrypted. Result: ot-ctl saw the SOPS
envelope as the dataset hex and rejected with Error 7: InvalidArgs.
Add decryption block matching home-assistant / speech-to-phrase /
home-assistant-zwavejs.
Also drop the stale TBD-SLZB-HOSTNAME comment — that placeholder was
filled in earlier and the dataset bootstrap now handles bring-up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
alpine/kubectl:1.31.4 doesn't exist; the repo's earliest tag is 1.33.3.
Pin to 1.35.2 to match the cluster server version (verified via Docker
Hub API + kubectl version).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bitnami/kubectl:1.31 was pulled from docker.io in early 2026 (Bitnami
moved legacy images behind their secure subscription). alpine/kubectl
is the maintained replacement and pulls cleanly via the cluster mirror.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three pieces wired together so OTBR survives container restart and
recovers from a clean PVC without manual intervention:
1. otbr-data PVC mounted at /var/lib/thread/. Persists OpenThread's
active dataset, child table, MLE/MAC frame counters, and partition
state across container restarts. Without it the BR re-elects
leadership and triggers a network reconverge on every restart.
2. otbr-dataset-bootstrap Job + sops-encrypted dataset Secret. On a
clean install (PVC empty), the Job sees no active dataset, runs
the proven bring-up sequence (ifconfig up → dataset set active
<hex> → thread start) via kubectl exec, and verifies attach.
Idempotent: exits 0 immediately if the dataset is already
configured, so subsequent Flux reconciles are no-ops.
3. Entrypoint patch + post-start activator. The wrapper now sed-
injects --auto-attach=0 into the upstream entrypoint's
OTBR_AGENT_OPTS line so otbr-agent boots into `disabled` even
when /var/lib/thread already has a saved dataset. Without this,
SLZB-06P7 firmware crashes the agent on every restart: dataset
load triggers KeyManager → otPlatRadioSetMacKey, the RCP rejects
with INVALID_STATE because the PHY is still DISABLED, and the
agent fatal-exits. After the entrypoint is up, the wrapper
activates Thread (ifconfig up → thread start) only if a dataset
is persisted; otherwise it waits for the bootstrap Job.
Active dataset TLV is the existing Nest network's (extPanId
cc4a4a491d2358c5, name NEST-PAN-339C), exported from HA's
/config/.storage/thread.datasets so this OTBR joins the same Thread
partition instead of forming a new one.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The upstream openthread/otbr entrypoint defaults REST_HOST=127.0.0.1
and passes --rest-listen-address $REST_HOST to otbr-agent, so the REST
API on :8081 was only reachable from inside the pod. From the cluster
pod network — including HA's otbr integration calling
http://otbr.home-assistant.svc:8081 — every connect got refused at the
TCP layer because nothing was listening on the pod IP.
Verified inside the pod with `ss -tlnp`: pre-fix listeners were
127.0.0.1:8081 (otbr-agent) and 127.0.0.1:80 (otbr-web). HA's
config_entry for the OTBR integration was healthy and pointed at the
right URL — the integration just couldn't connect.
Setting REST_HOST=0.0.0.0 makes otbr-agent listen on all interfaces
(eth0 cluster pod CIDR + net1 macvlan on the IoT VLAN). The Service
DNAT to the pod's eth0:8081 now lands on a real listener.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bnutzer/docker-otbr-tcp and the HA addon's
rootfs/etc/s6-overlay/s6-rc.d/socat-otbr-tcp/run are the known-working
references for SMLight network-attached Thread radios. Match their
socat command byte-for-byte:
- ignoreeof on the PTY (keeps data loop alive across otbr-agent
transient closes; previously wait-slave caused tear-down/restart)
- drop nodelay/keepalive from the TCP address (not in reference)
- lowercase pty/tcp address types
And drop uart-baudrate from RADIO_URL (PTYs ignore termios speed; the
reference omits it). Keep uart-flow-control to remain byte-identical
to the addon's URL.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two follow-ups to the in-container socat bridge:
1) Add a livenessProbe that pgreps otbr-agent. The upstream entrypoint
ends in `tail -f /var/log/syslog`, so PID 1 stays alive even when
otbr-agent has died — the pod stayed Ready for hours with no radio.
The probe makes radio failure visible as CrashLoopBackOff.
2) Match the HA addon's RADIO_URL form (uart-baudrate=460800 +
uart-flow-control). Both are no-ops on a PTY but keep the URL
byte-identical to what upstream tests; rules out URL-parser quirks
as a source of difference when comparing against a known-good
reference.
The remaining handshake failure (SLZB ACKs the TCP packets but its
EFR32 sends no spinel reply) is on the device side, not the K8s side
— see commit body / diagnosis.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add loki.process.udm_cef pipeline downstream of the udp-5514 raw
listener. Single regex extracts the CEF:0 header (vendor / product /
product_version / signature_id / event_name / cef_severity) plus the
UNIFI* extension fields (UNIFIhost / -deviceName / -Model / -Ip / -Mac
/ -Version) and msg. Curated label promotion: vendor, product,
signature_id, event_name, device_model, host (= device name). High-
cardinality fields (device_ip, device_mac, msg, full version) stay in
the extracted map and the line content remains the verbatim packet so
unparsed sender variants still land queryably.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The edge-pin existed only because mosquitto-iot used ETP=Local: with
Local, kube-proxy/Cilium will only DNAT to a backend pod on the same
node receiving the inbound packet, and the iot CiliumL2 announcement
only fires from k8s-edge. So the broker had to live on k8s-edge.
Switch the iot LB to ETP=Cluster. Inbound still arrives on k8s-edge
(only v30iot node), but Cilium forwards cluster-wide so the broker can
schedule anywhere. Source-IP preservation is lost — broker logs see a
cluster node IP, not the IoT device IP — but source-IP ACLs aren't a
security mechanism here; mosquitto authn is password+TLS.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Alloy 1.x gates syslog_format = "raw" behind experimental stability;
default refuses to load the component. Set alloy.stabilityLevel:
experimental scoped to this single-purpose receiver only — the pod-log
Alloy DaemonSet (separate HelmRelease) is untouched and stays at GA.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
UDM SIEM Server emits a payload that doesn't start with the <PRI>
syslog priority byte, so both rfc3164 and rfc5424 parsers reject it
("invalid framing. first byte: 'M'"). Switch the listener to raw
(per Alloy reference) so the bytes land verbatim. Once the actual
format is observed, add a loki.process regex stage to extract fields.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The configurator UI on :42069 and the firmware HTTP endpoint on :38080
are both reached through the ClusterIP Service in service.yaml; nothing
on this path needs the host's network namespace. Same shape as the
otbr fix in 1972c9b.
home-assistant-ns CNP comments updated: no hostNetwork siblings remain
in the home-assistant namespace. The host/remote-node ingress+egress
rules stay (still needed for kubelet probes), but the prose narrative
doesn't have to claim a sibling that no longer exists.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A sidecar approach for the spinel TCP↔PTY bridge fails because each
container has its own /dev/pts namespace; a symlink at /shared/ttyThread
pointing at /dev/pts/0 created in the sidecar is dangling in the otbr
container. Move the bridge into the otbr container's own namespace by
installing socat at startup via apt (Ubuntu 18.04 base, namespace has
no egress restrictions) and running it under a restart wrapper. The
upstream entrypoint then opens /tmp/ttyThread as a normal UART.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
UDM Network app's SIEM Server integration ships to port 5514, not 514.
Add a fifth listener pinned to RFC 5424 (UDM SIEM default) with a static
source=udm-siem label so traffic from this path is queryable independent
of host/app parsing. Add the matching UDP/5514 port to the LoadBalancer
Service.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous RADIO_URL `spinel+hdlc+forkpty://socat?STDIO,TCP:...` told
otbr-agent to fork an exec called `socat`, which the upstream
openthread/otbr image does not ship. otbr-agent died silently after
logging the radio URL and the pod looked Ready only because the
entrypoint ends in `tail -f /var/log/syslog`. Run socat as a sidecar
that bridges TCP→PTY into an emptyDir, and point otbr-agent at the PTY
via the standard `spinel+hdlc+uart:///shared/ttyThread`.
Also attach OTBR to v30iot via Multus (10.2.30.227, mirroring
matter-server's .226) and set BACKBONE_INTERFACE=net1, so the BR
advertises mDNS/SRP on the IoT VLAN where Matter commissioners and
Thread devices can actually reach it; the Cilium-managed eth0 cannot
multicast there.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The two prior label-promotion attempts (loki.relabel forward, then
loki.process+stage.labels) both saw zero __syslog_* fields because the
syslog source strips its internal __syslog_* labels BEFORE forwarding.
Per the Alloy reference, the correct hook is the `relabel_rules`
argument on loki.source.syslog itself — it consumes a loki.relabel
component's `.rules` export and applies them inside the source, before
the strip.
Convert loki.relabel "syslog" to rules-only (forward_to=[]) and
reference its .rules from each of the four sources. Drop the
intermediate loki.process. forward_to on each source goes straight to
loki.write.default.receiver.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Without --primary-interface set, python-matter-server logs "Using
'None' as primary interface" and the Matter SDK picks the kernel
default route interface (eth0, Cilium pod CIDR). Matter device
discovery on the IoT VLAN needs link-local probes from the macvlan
attachment, so the dedicated 10.2.30.226 IP on net1 was decorative
without this flag.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
loki.relabel reads from existing labels, but loki.source.syslog populates
__syslog_* into the entry's extracted map (parser scope), not as labels —
so the relabel rules never produced host/severity/facility/app/source_ip,
even for compliant senders. Empirically verified against probe messages.
Switch to loki.process with stage.labels, which is the correct hook for
extracted-map → label promotion. All four syslog sources now forward into
loki.process.syslog instead of loki.relabel.syslog.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SLZB-OS sends bare <PRI>MSG syslog packets with no RFC 3164 header, so
__syslog_message_hostname is empty and the host label stays unset. Adding
back source_ip from __syslog_connection_ip_address gives per-device
attribution if Cilium's socket-LB preserves the client IP through the
LB IP under ETP=Cluster (empirical — depends on Cilium config).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Selective tar extraction by bare name failed — the upstream
containernetworking/plugins tarball stores entries with a `./` prefix,
so `tar -xzf … macvlan static host-local` matched nothing and the init
container CrashLoopBackOff'd with `Not found in archive` for every
binary.
Just extract everything; the plugins are ~5-10MB total and any unused
binary is harmless next to Cilium's CNI bins.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Talos's /opt/cni/bin only carries what Cilium installs (cilium-cni,
bandwidth, …); the standard containernetworking/plugins binaries are
missing. Without them multus's macvlan attachment for matter-server
fails at sandbox setup with:
failed to find plugin "macvlan" in path [/opt/cni/bin]
There is no `siderolabs/cni-plugins` Talos system extension (verified
against github.com/siderolabs/extensions/tree/main/network — only
bird2/cloudflared/lldpd/nebula/netbird/newt/tailscale/zerotier).
Building a custom Talos system extension would mean a new OCI image,
schematic regeneration, installer image rebuild, and rolling node
reboots for one missing binary; not worth that surface area.
A small DaemonSet that drops the official containernetworking/plugins
v1.5.1 release tarball into /opt/cni/bin via hostPath is the same
mechanism Cilium and Multus already use successfully on this Talos
cluster — no reboot, no schematic change. Kept as a separate DS rather
than an extra initContainer in multus-daemonset.yaml so the upstream
multus manifest can be re-vendored cleanly.
Pinned to v1.5.1; bump together with the multus version when refreshing
upstream pins.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
cilium-cni's install step backs up any CNI config alphabetically before
its own 05-cilium.conflist into *.cilium_bak, which kept turning
multus's 00-multus.conf into 00-multus.conf.cilium_bak — so kubelet
never saw multus and pods with k8s.v1.cni.cncf.io/networks annotations
silently came up without their additional NICs (verified against
matter-server: pod ran 1/1 Ready but had no net1 macvlan attachment).
Setting exclusive=false leaves multus's config in place so the chain
becomes kubelet → multus → cilium (via clusterNetwork delegation).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
k.rennerandClaude Opus 4.7