influx CLI 2.7.4 opens --errors-file with O_WRONLY and fails with
"no such file or directory" if the file doesn't exist. touch it
before calling influx write.
Bump v7 → v8.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Seed Kustomization tried to create the PVC in `home-assistant` ns but
the ns lived in the `home-assistant` app Kustomization, which depends
on seed — deadlock. Matches the music-assistant pattern now: namespace
(with privileged PSA labels for hostNetwork+USB) lives in
infrastructure/namespaces, apps just consume it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
hostPath volumes with type=CharDevice refuse to mount if the path is
absent, so the placeholder TBD paths would have left both pods stuck
in ContainerCreating during canary — blocking the whole "does the
seeded /config boot HA on K8s?" test.
Strip stromzaehler + ZWA-2 volumes for now; zwavejs scaled to 0
(nothing to do until its stick is present). Adding the usb=[...] block
to vm_153_k8s_edge.tf + the hostPath volumes back + replicas:1 all land
in the same cutover commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Land HA Core on k8s-edge alongside Music Assistant. /config comes from
HAOS's own backup framework writing directly to Garage (S3 Compatible
HACS integration on VM 103, bucket k8s-migration/home-assistant/),
which the seed Job picks up — newest .tar under the prefix, extract
outer HAOS tar, extract inner homeassistant.tar.gz onto /config, yq-
rewrite http.trusted_proxies for the ingress-nginx pod CIDR.
Three Flux Kustomizations in clusters/homelab/home-assistant.yaml:
- home-assistant-seed (Job-gated, 4h timeout, wait:true)
- home-assistant-zwavejs (wait:false — ZWA-2 hostPath is TBD until
the USB physically moves to saturn at cutover)
- home-assistant (depends on seed, wait:false — stromzaehler USB
same caveat)
Ingress is home.renner.uno only; Caddy keeps fronting production until
cutover. HA auths itself via the openid HACS integration against
auth.renner.uno — no proxy-layer forward-auth needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
v6's first pod ran 6h39m before dying on a single bad line — a Home
Assistant `release_summary_str` value with unbalanced quotes (literal
newline embedded in a quoted LP string). Without error tolerance, one
bad row nukes the whole import and forces a full restart.
--errors-file captures rejected rows to /scratch/rejected.lp and lets
influx write keep going. Summary is logged at the end for visibility
into what got dropped.
Bump v6 → v7.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The SSH-to-nas-1 approach was wrong — the CI container doesn't need
shell access to produce a backup, it just needs the backup to exist
somewhere it can read. Route HA's backup framework at Garage directly
via the PhantomPhoton/S3-Compatible HACS integration; the K8s seed Job
picks the newest .tar under s3://k8s-migration/home-assistant/.
Removes:
- ansible/plays/backup-home-assistant.yml (nas-1 tar → Garage)
- .woodpecker/migrate-home-assistant.yml (pipeline that drove it)
The seed Job update (kubernetes/apps/home-assistant-seed/seed-job.yaml)
rides along in the upcoming K8s apps commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
nas-1 requires password SSH (DSM doesn't support key auth for ladmin),
and Ansible refuses to pass a password without sshpass on PATH.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The CNPG instance manager's reconcile loop blocks on "Unknown plugin:
barman-cloud.cloudnative-pg.io" when a Cluster omits a plugin declaration,
which in turn prevents TLS cert loading on the status endpoint and leaves
the pod stuck 0/1 Ready.
Match the house pattern (authentik-db, paperless-db, immich-db):
- Declare the plugin in Cluster spec with barmanObjectName: coder-pg-store
- ObjectStore coder-pg-store → s3://k8s-cnpg/coder-pg on Garage (nas-1)
- Daily base backup at 02:30 UTC with immediate first backup
- barman-cloud-creds reused from authentik-db (same Garage creds,
namespace rewritten to coder)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Prior version tried to SSH root@homeassistant.iot.renner.uno:22 and tar
/config, which fails because HAOS doesn't run SSH on 22 and that's the
wrong way to back up HAOS anyway — it bypasses supervisor metadata.
Instead, pick the newest .tar under /volume1/Homeassistant Backup on
nas-1 (produced by the existing HAOS → Synology backup pipeline) and
stream it to Garage untouched. Zero HA-side impact — no stop, no restart.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Stops HA core on VM 103, streams tar of /config (excluding logs/tts/deps)
into s3://k8s-migration/home-assistant/home-assistant-config.tar.gz, restarts
HA. Manual trigger, rerunnable. K8s-side seed Job + manifests land in a
follow-up once the tarball is in place.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Coder's OIDC init was crashlooping with "x509: certificate signed by
unknown authority" when fetching auth.renner.uno's OpenID discovery
document, because every ingress in this cluster uses the
letsencrypt-staging ClusterIssuer today.
Mount the staging roots (Pretend Pear X1 + Bogus Broccoli X2) via
coder.certs.secrets; the chart drops them at /etc/ssl/certs/ where
Go's crypto/x509 picks them up on Linux.
Remove once the cluster migrates to letsencrypt-prod.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces EOL Kasm VM 124 with a K8s-native platform for dynamically
provisioned desktop environments. Kasm was ruled out — its Helm chart
is Technical Preview and does not include the agent, Proxmox is being
decommissioned, and Talos refuses DinD.
HelmRelease `coder` (chart >=2.32.0 from helm.coder.com/v2), CNPG
Cluster `coder-pg` (2 instances, ceph-block 10Gi) as external DB,
SOPS-encrypted Authentik OIDC credentials, ingress at coder.renner.uno
behind letsencrypt-staging. Workspace templates follow separately.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
emptyDir backs onto node ephemeral-storage. With only ~4.5 GiB free per
node, the 10.5 GiB download triggered kubelet eviction (Reason: Evicted,
Exit 137) long before the file finished.
Generic ephemeral volume claims a 20 GiB RWO PVC on ceph-block for the
pod's lifetime — Ceph-backed, auto-deleted with the pod, and won't
compete with node-level ephemeral pressure.
Bump Job v5 → v6.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Streaming aws s3 cp | influx write kept the S3 GET TCP connection open
for hours at ~17 MB/min (influx write's ingest pace), and every transient
network hiccup aborted the transfer with an 'unexpected EOF' — aws-cli has
no resume semantics for streaming-to-stdout.
Staging to a 15 GiB pod-scoped emptyDir decouples the two phases: the
S3 GET now completes in minutes, and influx write runs against a stable
local file for as long as ingest takes.
Bump Job v4 → v5 for fresh template.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`influx write` 2.7.4 doesn't expose --batch-size; the flag triggers
an immediate CLI parse error and the Job crashlooped. Remove it and
rely on the server-side INFLUXD_STORAGE_WRITE_TIMEOUT=60s knob to
absorb slow TSM writes.
Bump Job v3 → v4 for fresh template.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Partial seeding produced enough shards that opening them on startup
pushes memory past the 2Gi limit → OOMKilled. Meanwhile the chart's
default zero-delay liveness probe kills the pod before shard open
finishes, causing a CrashLoopBackOff.
- resources.limits.memory: 2Gi → 6Gi
- livenessProbe: initialDelaySeconds 600, failureThreshold 6
- readinessProbe: initialDelaySeconds 30
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Root cause: `influx write` batches of 5000 points routinely blew past
InfluxDB's default 10s storage-write timeout against the cold-cache
RWO Ceph-block PVC under concurrent TSM compaction. Server responded
500 → client aborted → `aws s3 cp` got broken pipe.
Two-sided fix:
- Seed Job: --batch-size 1000 so each HTTP write finishes well
inside any plausible write deadline.
- InfluxDB server: INFLUXD_STORAGE_WRITE_TIMEOUT=60s to tolerate
transient ingest+compaction contention.
Bump Job name v2 → v3 to force a clean re-roll; pod template is
immutable.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Kubernetes Job pod templates are immutable, so the hostname fix in
716dd7f never reached the live v1 Job. Bumping the name forces Flux
to create a fresh Job with the correct influxdb-influxdb2 service
name — matches the re-seed procedure documented in the file header.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
kube-influxdb2 chart produces service name <release>-<chart> (so
`influxdb-influxdb2`, not `influxdb`). Verified with `influx ping`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Same class of bug as the backup side — seed was staging the 10 GiB
gzipped line protocol to pod /tmp before importing. Pipe `aws s3 cp -`
straight into `influx write` via stdin. set -o pipefail so any stage
failing kills the Job.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
aws-cli 2.9.19 in the influxdb:2.7.4 base image predates
AWS_ENDPOINT_URL env support (added in 2.15+). The Job's wait loop
was silently hitting real AWS instead of Garage, so head-object
always returned not-found. Factor the CLI into `$AWS` with the
explicit endpoint baked in.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
kube-prometheus-stack default RollingUpdate deadlocks on Multi-Attach
when Grafana is a single replica on RWO ceph-block — new pod can't
mount the volume while the old pod is still Running. Use Recreate so
the old pod terminates first.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Pin Loki datasource UID to `loki` so dashboards referencing that UID resolve
- Add `prometheus-legacy` datasource pointing at LXC 227 so host-level
dashboards (Alloy, pve host node_exporter, docker) keep working until
the legacy Prometheus retires at global cut-over
- Add `influxdb` datasource pointing at VM 108 (Stage A); the
suedstr28/renner bucket is the heatpump-roi dashboard's source.
Token from grafana-influxdb-token SOPS secret (same read-only token
the legacy Grafana role uses). Stage B flip to the in-cluster
InfluxDB is deferred to the HA migration PR.
- Enable kube-prometheus-stack's Grafana sidecar on the
`grafana_dashboard: "1"` label so per-dashboard ConfigMaps from
kubernetes/apps/monitoring/dashboards/ get auto-loaded
DASHBOARD-MIGRATION.md tracks the 27-dashboard port queue (bucketed
k8s-native / host-legacy / mixed). Empty dashboards/ kustomization is
ready to accept per-dashboard PRs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add garage-migration-creds secret (reused key from other seed apps) and
the kustomization resource list so Flux actually applies the seed-job
landed in the export-lp commit. Seed waits for
s3://k8s-migration/influxdb/suedstr28.lp.gz then runs `influx write`
against the in-cluster InfluxDB.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
influxdata/influxdb2 HelmRelease in its own namespace. 50Gi ceph-block
PVC, single replica (OSS). Admin password + token come from a SOPS-
managed cluster-internal secret so credentials are deterministic across
cluster rebuilds; the seed Job reuses the same token for `influx write`.
Clusters/homelab wires two Flux Kustomizations: influxdb (the chart) and
influxdb-seed (depends on influxdb being healthy so the seed only runs
once the API is reachable).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pipeline 672 filled /tmp with the partial export before failing,
leaving influxd stopped and HA writes failing. Pipe `influxd inspect
export-lp --output-path - --compress` straight to `mc pipe` — no local
staging, so the export survives any bucket size regardless of /tmp.
Matches the streaming pattern in backup-music-assistant.yml.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Ubuntu influxdb2 2.7.4 package uses the legacy /var/lib/influxdb path
(no '2' suffix) — confirmed by probe in pipeline 671. Drop the probe
block, set engine_path accordingly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pipeline 670 failed the stat assert — the assumed engine path doesn't
exist. Dump the data dir candidates, systemd unit, /etc/default/influxdb2,
and /etc/influxdb/config.toml to find where VM 108 actually stores its
TSM files before running export-lp.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`influx backup` is observably broken in 2.7.x — multiple upstream threads
(incl. 2.7.11, June 2025) report the same 'unexpected EOF' during metadata
SQLite download with a root-scope token, disk space fine, no resolution
published. Even the All-Access token permission model is documented as
under-scoped for backup.
Pivot to `influxd inspect export-lp`: reads engine TSM files on disk,
emits gzipped line-protocol for the target bucket only. Seed Job imports
with `influx write --compression gzip --file`. Server stopped for a few
seconds on VM 108 to read a consistent TSM snapshot; HA write retries
cover the gap.
Trade-off: loses dashboards/tasks/auth state from VM 108, which we don't
want to carry across (K8s Grafana has its own dashboards, HA is the sole
writer, K8s InfluxDB uses its own bootstrap admin from SOPS).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the auth-list probe with a direct curl to
/api/v2/backup/metadata so the HTTP status + first 2KB of body are
visible. Unexpected-EOF on the CLI suggests the server is returning
an error payload the CLI can't parse; curl will show what it actually is.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pipeline 668 got past auth (local admin token worked) but `influx backup`
failed with `unexpected EOF` while saving the metadata sqlite. CLI 2.7.5
vs server 2.7.4 protocol skew is the most likely cause.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add a prefix+length dump and `influx auth list` probe so we can
distinguish three cases behind the 401: (a) Woodpecker secret didn't
propagate, (b) token missing read:authorizations despite All Access
label, (c) wrong InfluxDB instance on localhost:8086.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Stop probing, commit to the `influx backup` approach: pull the v2 CLI
static binary to /tmp on VM 108, run a scoped backup of the
suedstr28/renner bucket, upload tarball to Garage, clean up. Token
read from INFLUX_BACKUP_TOKEN (Woodpecker secret) — the scoped-bucket
backup still needs operator read on metadata, so the token is an
operator-scope token on VM 108, not the Grafana read token.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
VM 108 runs InfluxDB 2.7.4 server but only has the legacy `influxdb-client`
(v1) package installed. Probe output (pipeline 663):
influxdb-client 1.6.7~rc0-1ubuntu0.22.04.3
influxdb2 2.7.4-1
Pull the matching v2 CLI tarball to /tmp, run scoped `influx backup`,
clean up. No persistent package install, no apt config changes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
audiobookshelf-seed wrote `.restore-v3-done` while the app's
wait-for-restore init polled `.restore-done` — the pod has been stuck
in Init:0/2 since the last rebuild despite a successful seed.
Strip the `-vN-` infix from all four seed sentinels (audiobookshelf,
immich, music-assistant, paperless) so the app/seed contract no longer
drifts each Job bump. Short-circuit path accepts either new or legacy
sentinel and normalizes forward, so PVCs seeded under the old name
unblock on next Job run without a re-extract.
Job object names (`*-data-seed-vN`) are unchanged — those are the
manifest-bump trick for forcing re-runs, orthogonal to the sentinel.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Last run showed `influx version` is the 1.x client (single-dash flags,
no `backup` subcommand). Before committing to a migration tool, dump
package list, binary locations, influxd version, and the :8086 health
endpoint to pick the right path (1.x → 2.x upgrade vs scoped backup
vs line-protocol export).
Play fails at the end so the debug output is visible in CI logs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Remove no_log on the backup task (it was hiding the underlying failure)
and move the token to INFLUX_TOKEN env so the command line stays clean.
Add a which + version probe so the CLI path and install are logged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Online `influx backup` scoped to the Home Assistant bucket, tarballed
and uploaded to s3://k8s-migration/influxdb/. Re-runnable — each run
overwrites the tarball with current VM 108 state, so a rebuilt cluster
picks up everything HA wrote since the last run.
Uses the existing read-only influxdb_token from the grafana-observability
role defaults (single source of truth; the same token already backs the
legacy Grafana → InfluxDB datasource).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The K8s Deployment landed with an empty PVC, so MA came up
unconfigured — no providers, no players, no library index. Adds the
seed infrastructure that matches audiobookshelf / paperless:
* Woodpecker .woodpecker/migrate-music-assistant.yml — manual trigger
that runs an Ansible playbook against LXC 229, stops the
docker-compose stack, tars /opt/music-assistant/data, streams the
tarball to s3://k8s-migration/music-assistant/, and restarts the
container.
* kubernetes/apps/music-assistant-seed/ — seed Job polls Garage for
the tarball (up to 3 h), wipes first-run remnants under /data,
extracts, writes /data/.restore-v1-done. PVC definition moves here
with prune-disabled and is bumped 10Gi → 20Gi.
* MA Deployment gains a wait-for-restore init container blocking on
the sentinel; the inline PVC definition is removed.
* Flux: new music-assistant-seed Kustomization; music-assistant
Kustomization now dependsOn it.
Run order: push → Flux reconciles seed (Job waits) → trigger
migrate-music-assistant pipeline → tarball lands → Job extracts →
Deployment unblocks with restored state.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Provider validator treats all and vmid as mutually exclusive at config
level, so the transition must be driven by specifying vmid alone.
Provider will clear PVE's all=1 internally when the API call sets the
vmid mode.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
First plan showed `all = true` unchanged and `vmid` added — PVE with both
set is ambiguous. Driving `all -> false` makes the selection mode
unambiguous.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Imports the existing cluster-wide backup job (backup-51b113b6-676a) into
OpenTofu. The job previously lived only in PVE's UI, which let a stray
manual edit cascade into an I/O freeze that took down Rook-Ceph and the
K8s control plane.
Switches from all+exclude to an explicit vmid include list — bpg/proxmox
doesn't expose vzdump's exclude attribute. Same 37 guests are covered;
K8s VMs 150-153 remain outside the list by design (Flux + Velero/CNPG
are the K8s-side protection path).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Host-networked Deployment on the edge node (workload-class=pinned,
topology.kubernetes.io/zone=edge) so mDNS / uPnP / SlimProto reach
Chromecasts, DLNA renderers, and Squeezeboxes on v30iot with
native-kernel network path (lowest latency/jitter). State on a
ceph-block PVC, retained on Kustomization retirement.
Prereqs bundled in:
* rook-ceph operator: csi.pluginTolerations so the CSI node plugin
lands on the pinned edge node and MA can mount RBD.
* ingress-isolation: allow ingress-nginx egress to remote-node, since
hostNetwork pods take node identity and namespace selectors don't
match them.
* music-assistant namespace labeled pod-security enforce=privileged
(required for hostNetwork + SYS_ADMIN/DAC_READ_SEARCH + AppArmor
unconfined, matching the LXC docker-compose).
LXC 229 stays running for now; decommission once K8s side is verified.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Music Assistant needs L2 reach to the IoT VLAN for mDNS / uPnP /
Chromecast / SlimProto discovery. Adds an optional
extra_network_bridges list to the proxmox_vm module and uses it to
give VM 153 a second NIC on v30iot, keeping v40srv as the primary
cluster interface.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ingress-nginx-egress CNP whitelists target namespaces; immich was
missing from the list, so ingress → immich-server pod times out.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Immich v2.7.5 defaults to listening on IPv6 localhost (`[::1]:3001`),
which makes the pod unreachable from the in-cluster Service — ingress
returns 504 gateway timeout because nginx can't establish a connection
to the upstream. Set IMMICH_HOST=0.0.0.0 to bind on all interfaces.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
k.rennerandClaude Opus 4.7