Pipeline 758 hit `Operation Error: timeout on preparing data upload`
30 min after Backup creation. The CLI flag --item-operation-timeout
(from commit 68bba53) was the wrong knob — it controls the per-item
operation timeout client-side, not the server-side prepare phase.
The actual flag is the Velero server's --data-mover-prepare-timeout
(default 30 min), exposed via the helm chart as
configuration.dataMoverPrepareTimeout. CSI VolumeSnapshotContent
provisioning + Kopia repo init on the immich library (~250 GiB,
millions of small files) takes longer than 30 min on this cluster's
ceph-cluster-csi pace.
Also keep configuration.defaultItemOperationTimeout: 4h for safety.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pipeline 757 hit rsync exit 23 after processing all 189k entries —
"some files/attrs were not transferred". For a live immich library
that's a handful of files with weird perms or that vanished mid-sync,
not a real failure. rsync had already moved everything else. With
`set -eu` the script aborts instead of proceeding to the velero backup
create step.
Treat 23 as a warning, fail on any other non-zero code.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pipeline 756 was stuck for 22 minutes in `find /dst/data -mindepth 1
-delete` and barely made progress (deleted ~4 GiB out of 250). CephFS
bulk-delete on millions of small files is metadata-bound; at observed
rate it would have taken 20+ hours before tar could even start.
immich VM (immich.srv.renner.uno) ships rsync — same as paperless +
audiobookshelf — so use the same `rsync -aH --delete` pattern. rsync
deletes per-file as it syncs and is incremental on re-runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pipeline 754 hit `Operation Error: timeout on preparing data upload`
on immich's 250 GB library PVC. Velero's CSI plugin defaults to 30 min
for the prepare phase (CSI VolumeSnapshotContent provisioning + Kopia
DataUpload setup). Fast Ceph snapshot creation is plenty for paperless
or music-assistant but slow for big PVCs.
Add --item-operation-timeout=4h to every velero backup create call so
the prepare phase has room to finish on the largest PVCs without
falsely failing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Apply the same shape that landed in migrate-immich (commit e9792f9) to
the other four migrate pipelines:
- Whole step in one shell block so trap EXIT covers any termination
(success / error / SIGTERM-on-cancel) — pipeline cancellation no
longer leaves Flux suspended, the Deployment scaled to 0, or the
staging Job + SSH Secret stranded.
- Heartbeat every ~3 min during Job-wait: elapsed, pod phase,
extracted size. Stops Woodpecker's idle-output watchdog from
killing silent steps.
- velero backup create runs without --wait — poll separately so the
long upload phase (especially for audiobookshelf 70 GB and immich
250 GB) keeps emitting heartbeats.
- Job manifests get ttlSecondsAfterFinished: 600 as a self-cleaning
artefact in case trap doesn't run (SIGKILL etc.).
- migrate-influxdb additionally scales the StatefulSet back to 1 in
the cleanup function (Helm controller doesn't notice live replica
drift, so otherwise the pod stays at replicas=0).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pipeline 751 was killed by Woodpecker's idle-output watchdog after
exactly 32 minutes — the wait loop polled the Job's Complete/Failed
conditions every 30 s but produced no output between polls. With
silent stdout, the agent classifies the step as stuck.
Pipeline 753 had the same pattern.
Refactor:
- Whole step is now ONE shell script in a single `- |` block. A
cleanup() function runs on EXIT (any reason: success, error,
SIGTERM-on-cancel) so a cancelled pipeline doesn't leave Flux
suspended, the Deployment at replicas=0, or the staging Job +
SSH Secret stranded.
- Heartbeat every 6 polls (~3 min) during Job-wait: prints elapsed,
pod phase, and `du -sh /dst/data` so a 4 h transfer has visible
progress.
- velero backup create runs without --wait — poll separately with
the same heartbeat shape, summing DataUpload bytes_done/total.
- Job manifest gets ttlSecondsAfterFinished: 600 so even SIGKILL
on the Woodpecker runner leaves a self-cleaning artefact.
Same pattern still needs to land on paperless / audiobookshelf /
music-assistant / influxdb pipelines.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`kubectl logs -f --pod-running-timeout=10m` is unreliable as a wait
barrier when the pod spends time in ContainerCreating (volume attach
on RWX PVCs especially for the 300 GiB books PVC). It can return
early without streaming, leaving the subsequent `kubectl wait
--for=condition=complete --timeout=30s` to hit a Job that's still
running and exit 1.
Pipeline 750 (migrate-audiobookshelf) hit this: rsync started fine
in the staging Job but the pipeline reported failure after 88 s —
even though the Job kept running and the velero backup create that
followed actually got submitted (Backup ended up in
WaitingForPluginOperations).
Replace the logs-and-wait dance with a 4 h-capped poll loop that
reads the Job's Complete / Failed conditions directly. On Failed,
print describe + last 100 log lines for diagnosis. On Complete,
print the last 200 log lines for visibility.
Same change applied to all 5 migrate-* pipelines.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bootstrap-restore Job:
- Wait for BackupStorageLocation default to reach phase=Available
before iterating schedules so the BSL controller has had a chance
to sync Backup CRs from Garage.
- Per-schedule retry: if the visible Backup count is 0 on first read,
sleep 10 s × 10 retries before declaring "no Backups produced yet".
This unblocks the BSL-sync-race regression where rebuild 741 skipped
authentik-daily, music-assistant-daily, and immich-daily.
- Use wc -l | tr -d ' ' for backup count instead of grep -c which can
double-emit on empty input (matches the migrate-* pipeline pattern).
opentofu k8s control plane:
- Bump VM memory 24 GB → 48 GB on k8s-1/k8s-2/k8s-3. Rebuild 741
showed sustained API-server timeouts under Velero data-mover
coordination load (DataDownload list calls timing out with
"the server was unable to return a response in the time allotted"),
causing audiobookshelf-books restore to PartiallyFail and
migrate-immich seed Backup to also PartiallyFail. Hosts have
plenty of free RAM, so increasing CP headroom is the cheap fix.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Helm controller doesn't notice live replica drift on a deployed
HelmRelease, so resuming the Flux Kustomization alone left
influxdb-influxdb2 at replicas=0 after the seed-pvcs step's scale-down.
Pod had to be brought back manually via kubectl scale during pipeline
740 verification. Add explicit scale --replicas=1 at the end of the
pipeline so subsequent runs come back automatically.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the influxdb-seed Flux Kustomization (which ran a one-shot Job
calling `influx restore` against an empty bucket on bootstrap) with
the same producer-equals-consumer pattern used by paperless / abs /
immich / music-assistant. Velero captures the file-level data PVC
directly; on cluster bootstrap the StatefulSet starts against a
populated /var/lib/influxdb2 and the helm chart's empty-bucket setup
is skipped.
migrate-influxdb rewritten:
- SSH stops the legacy influxdb.service on VM 108 (consistency for
BoltDB + WAL + TSM file-level snapshot), tar-streams /var/lib/
influxdb into the influxdb-influxdb2 PVC, restarts the service via
a trap so observability comes back even on rsync failure.
- Wipe-and-write into k8s-velero with label
velero.io/schedule-name=influxdb-daily.
- Drop kubernetes/apps/influxdb-seed/ entirely + the legacy backup-
influxdb Ansible playbook. influxdb Flux Kustomization now depends
on velero-bootstrap-restore.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
migrate-immich:
- Replace backup-library Ansible step with seed-pvcs that ssh-tar-
streams /mnt/media/immich from immich.srv.renner.uno into the
library PVC (~250 GiB) and writes a Velero Backup labelled
velero.io/schedule-name=immich-daily. Library mount path is /data
inside the immich-server pod (same PVC).
- DB seed (backup-legacy-db + seed-barman) untouched — already
aligned via Barman producer-equals-consumer in s3://k8s-cnpg/im-pg.
- Drop seed initContainer + garage-migration-creds Secret + the
legacy backup-immich Ansible playbook. Reduce immich Flux
Kustomization timeout 4h → 30m.
migrate-music-assistant fix:
- Pipeline 736 failed because the legacy MA LXC has no rsync binary
(`bash: line 1: rsync: command not found`). Switch to ssh+tar
streaming with explicit `find /dst/data -mindepth 1 -delete` to
mimic rsync --delete. Universal tar + ssh dependency only.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
migrate-music-assistant now rsyncs /opt/music-assistant/data from the
legacy LXC (music-assistant.iot.renner.uno) into the RWO data PVC and
writes a Velero Backup labelled velero.io/schedule-name=music-assistant-
daily so velero-bootstrap-restore picks it up identically to a daily
Schedule artefact.
Staging Job inherits the edge-zone nodeSelector + workload-class
toleration so it lands on k8s-edge alongside the RWO PVC (Multi-Attach
impossible by construction). SSHs to the legacy LXC, stops the docker-
compose stack, rsyncs, restarts it via a trap so the service comes
back even if the rsync fails midway.
Drop seed initContainer + garage-migration-creds Secret + the legacy
backup-music-assistant Ansible playbook. Reduce Flux Kustomization
timeout 4h → 30m and add velero-bootstrap-restore as a dependency.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three new daily Velero Schedules covering the namespaces still riding
the legacy seed-initContainer-from-k8s-migration path. Each restricts
to PVCs only (immich + authentik exclude CNPG cluster PVCs since the
DB is covered by Barman in the *-db namespace). Schedules stagger
through 04:45 → 05:15 → 05:45 local to spread node-agent load.
Also extends the velero-bootstrap-restore Job's SCHEDULES env so it
iterates over the new entries on cluster boot. With the schedule
labels in place, the migrate-* pipelines can write seed Backups
labelled velero.io/schedule-name=<app>-daily and bootstrap-restore
will pick them up identically to a daily Schedule artefact.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
migrate-audiobookshelf now writes the PVC seed as a Velero Backup
labelled velero.io/schedule-name=audiobookshelf-daily. velero-bootstrap-
restore picks it up identically to a daily Schedule Backup.
The seed-pvcs step suspends Flux, scales audiobookshelf to 0, applies
an in-cluster Job that SSHs to the legacy VM, stops the audiobookshelf
systemd unit (so the SQLite DB in /usr/share/audiobookshelf/config is
consistent), rsyncs config + metadata + the ~70 GB books library into
the real PVCs, then restarts the unit. Each migrate run wipes prior
audiobookshelf-daily Backups before writing the fresh seed — symmetric
with paperless.
Drop the seed initContainer from the audiobookshelf Deployment, the
now-unused garage-migration-creds Secret, and the backup-audiobookshelf
Ansible playbook. The k8s-migration bucket is no longer consulted on
bootstrap. Reduce Flux Kustomization timeout from 4h → 30m (no in-pod
extract anymore).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The instrumented wait_kustomization helper from #731 finally let us see
*why* paperless / monitoring / home-assistant Kustomizations were stalling.
Three independent bugs:
1. monitoring stalled forever with
PrometheusRule/monitoring/garage: dry-run failed: no matches for
kind "PrometheusRule" in version "monitoring.coreos.com..."
Chicken-and-egg: the rule's CRD comes from kube-prometheus-stack's
HelmRelease in the same Kustomization, so Flux's server-side dry-run
on the rule fails before the chart can install. Move the rule into a
new `monitoring-rules` Kustomization that dependsOn `monitoring` —
chart installs (CRDs and all) first, then the rule applies cleanly.
2. home-assistant stalled with
Service/home-assistant-iot: namespace not specified
iot-service.yaml was missing `metadata.namespace: home-assistant`
while every other manifest in the dir has it. Added.
3. paperless stalled with
Deployment/paperless/paperless status: 'F[ailed early due to ...]
The truncated message was cut off at 120 chars in the per-iteration
progress line so we couldn't see the actual reason. Bump truncation
to 240 chars and dump full `flux get kustomization` + recent warning
events on TIMEOUT so the next run gives a definitive answer instead
of needing a second cluster trip.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A Flux Kustomization that owns an Ingress with a cert-manager.io/cluster-
issuer annotation is marked Ready as soon as the Ingress + Certificate
manifests are applied. The actual ACME DNS-01 challenge runs asynchronously
afterward (~30s-2min per Certificate via Cloudflare), so e2e's TLS check
can hit the ingress-nginx fake default cert ("issuer=Acme Co, CN=Kubernetes
Ingress Controller Fake Certificate") for any Certificate still Issuing.
Add a `wait_certificates` poll at the end of wait-apps that blocks until
every Certificate cluster-wide reports Ready=True. Same progress-printing
pattern as wait_kustomization (state-change + 60s heartbeat). Budget is
15m, well above realistic Cloudflare ACME issuance times even with several
Certs in flight.
Closes the gap on #730's `dash.k.renner.uno fake cert` and grafana
`/api/health returns nginx HTML` failures (the Service backend was likely
fine but the Ingress was being served the default cert because the
Certificate hadn't finished issuing yet).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
#730 dropped 'Failed: 8' with three observability checks reporting empty
bodies (loki/prom/am). Root cause was opacity in the harness, not the
cluster: kubectl exec was being run with 2>/dev/null, so any failure (DNS
glitch in the helper pod, transient connection refused against the in-
cluster Service, missing endpoint, ...) was silenced. The check function
captured an empty $out, the grep failed, and the failure summary tail
was '-- ' with no diagnostic. Prometheus had 86+ min to come up before
e2e ran, so the data is up — the test is broken.
Fix:
- Hard-fail (exit 1) if e2e-helper doesn't reach Ready in 180s, with a
full kubectl describe output, instead of continuing silently into
obs checks against a missing pod.
- Sanity-check `kubectl exec` works against e2e-helper *before* trusting
it for obs queries.
- Bump the helper pod's sleep 600 → sleep 1800 so it can't expire mid-e2e.
- Replace the inline kubectl-exec curls with a `helper_curl` wrapper that
- captures stderr (no more 2>/dev/null swallowing the actual error)
- retries 5x with 5s backoff for transient DNS/connection failures
- prints `attempt=N code=XXX body=...` per attempt so the failure tail
shows exactly what happened
- is exported (export -f) so the `bash -c` subshells the check helper
uses can see it.
Loki/Prom/AM checks now read the response body from /tmp/helper_body
written by the wrapper, and the per-attempt diagnostic lands in the check
output regardless of pass/fail.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pipeline #730's wait-apps still produced an incomplete cluster for e2e:
the new wait_kustomization helper warned-and-continued on monitoring
(10m) and paperless (15m) before they finished reconciling under the
load of immich's parallel 250GB extract. e2e then caught the half-
built state via observability + grafana TLS + paperless 503 failures.
Bumping timeouts costs nothing on the happy path — Flux reconciles
all Kustomizations in parallel, so the wait just observes; longer
budgets only matter when something's actually slow.
- monitoring: 60 → 180 (10m → 30m). kube-prometheus-stack pulls many
images and stands up Prom + AM + Grafana + Loki + node-exporter +
state-metrics; first install on a contended cluster routinely
takes 15-25 min.
- paperless: 90 → 180 (15m → 30m). Velero CSI snapshot restore for
the data + media PVCs runs concurrently with all the other restores.
- home-assistant: 90 → 180 (15m → 30m). Same Velero contention plus
HA Core's native restore on first start.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The wait-apps phase only blocked on `apps`, `authentik`, `paperless`, and
`audiobookshelf`. After moving the music-assistant / immich / home-assistant
seed bodies into pod initContainers (and given monitoring is what serves the
Loki/Prom/AM/Grafana endpoints the e2e suite hits), wait-apps was returning
success while immich was still mid-extract and kube-prometheus-stack hadn't
issued certs yet — so e2e ran against a half-built cluster and flaked on
observability + grafana TLS + immich pod still in Init:0/2.
Refactor:
- factor the per-Kustomization wait into a `wait_kustomization` helper that
prints the elapsed time, current Ready/Message status on every state
change, and a 60-s heartbeat so a long silent reconcile is still visible
- add waits for: monitoring, music-assistant, home-assistant, immich
- give immich the longest budget (1440 × 10s = 240 min) — matches the prior
immich-seed Kustomization's 4h ceiling for the ~250GB library extract
Order is cosmetic (Flux reconciles in parallel), so the call sequence is
fast → slow to surface progress earlier in CI logs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`grep -c . || echo 0` double-emits when stdin is empty: grep -c returns
"0" *and* exits 1, then `|| echo 0` runs and emits another "0", making
$count = "0\n0". The `[ "$count" -eq 0 ]` check then fails (non-numeric)
and the loop runs to its 60-iteration limit (5 min wasted) before the
script proceeds to create the new seed.
Switch to a `count_backups()` helper using `wc -l | tr -d ' '` which
yields a clean integer for empty *and* populated input. Velero
deletion finished cluster-side correctly during pipeline 728 — the
stuck loop was purely the count parsing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Velero CLI only supports -o table|json|yaml. The wipe step's earlier
`velero backup get -l ... -o name` errored on stderr and returned an
empty stdout, so `grep -c .` counted 0 lines and the wipe was skipped.
Discovered after pipeline 727 left three pre-existing stale paperless-
daily Backups in place alongside the new seed.
Switch to the same counting pattern velero-bootstrap-restore already
uses: default table output, skip the header row, count remaining lines.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Woodpecker substitutes ${var} during YAML parsing, replacing references
that aren't pipeline-level variables with empty strings. Pipeline 726
failed at velero CLI download because ${VELERO_VERSION} resolved to ""
and the URL became velero--linux-amd64.tar.gz.
- Use $VAR (no braces) for env-var refs in seed-pvcs commands, matching
the seed-barman convention already in this file.
- Hardcode paperless.srv.renner.uno in the in-cluster Job spec instead
of templating it via ${LEGACY_HOST} (Woodpecker would template that
too, even inside a kubectl-apply heredoc).
- Replace ${var:-0} shell-default forms with `|| echo 0` at assignment
time so the script never relies on default-expansion syntax that
could clash with Woodpecker templating.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
migrate-paperless now writes the PVC seed as a Velero Backup labelled
velero.io/schedule-name=paperless-daily, so velero-bootstrap-restore
picks it up identically to a daily Schedule Backup. Single source per
data class on cluster bootstrap: k8s-velero for PVCs, k8s-cnpg for DB.
The seed-pvcs step suspends Flux, scales paperless to 0, applies an
in-cluster Job that rsyncs paperless_data + paperless_media from the
legacy VM, deletes prior paperless-daily Backups (Velero's deletion
controller cleans the Kopia data in Garage), then velero backup create
--snapshot-move-data --wait. Mirrors the wipe-then-write pattern that
seed-barman already uses for the CNPG bucket, so each migrate run is
authoritative for both halves.
Drop the seed initContainer from the paperless Deployment, the now-
unused garage-migration-creds Secret, and the backup-paperless Ansible
playbook. The k8s-migration bucket is no longer consulted on bootstrap.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bumps block_size from 20M to 64M. Bench (2026-04-26) on the live
20M deployment showed end-to-end cold reads at 57.6 MiB/s vs disk-
buffered floor at 67.1 MiB/s — software overhead is small (~14%),
HDD seeks dominate. Larger blocks reduce the seek count per object
proportionally; expected gain is ~15-25% on cold restores.
Only affects NEW uploads — existing objects retain their 20M (and
older 1M) chunking. CNPG WAL alignment is unchanged: WAL segments
are 16 MiB, block_size is a max, so each WAL still uploads as one
block.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Same pattern as the other seed-Job → initContainer refactors today
(c9ea0af / b9717e9 / 74994aa / e44d86b), with the home-assistant-
specific twist that the new initContainer doesn't *do* the restore —
it just stages /config/backups/<file>.tar and /config/.HA_RESTORE
manifest, and HA Core's native backup_restore.py performs the actual
decrypt + restore on its own startup. The trusted-proxies sidecar
still gates on /config/.HA_RESTORE_RESULT (HA's post-restore signal)
exactly as before.
Why this matters: home-assistant-config is RWO ceph-block. The previous
seed Job worked only because both the Job and the app pod were pinned
to `topology.kubernetes.io/zone: edge` AND there was a single edge node,
guaranteeing co-located RBD attach. Production may have multiple edge
nodes, at which point the seed and app could land on different ones and
hit the same Multi-Attach race that broke music-assistant. Moving the
seed into the HA pod itself makes Multi-Attach impossible by
construction (one pod, one kubelet, one RBD attach).
The sentinel `/config/backups/.k8s-seed-applied` survives HA's restore
(which preserves /config/backups/), so subsequent pod restarts short-
circuit in <1s. Cluster rebuild → empty PVC → no sentinel → full
download + manifest staging → HA Core does the restore on first boot.
Refactor:
- add `seed` initContainer to deployment.yaml (the HA pod previously
had no initContainers — it relied solely on Flux dependsOn ordering
for the standalone seed Job)
- move backup-password.sops.yaml + garage-migration-creds.sops.yaml +
pvcs.yaml from kubernetes/apps/home-assistant-seed/ into
kubernetes/apps/home-assistant/
- delete kubernetes/apps/home-assistant-seed/
- collapse the standalone home-assistant-seed Flux Kustomization;
replace `dependsOn: home-assistant-seed` on the app Kustomization
with the standard infrastructure deps the seed Kustomization had,
and bump timeout 30m → 4h to absorb the 3h backup-wait window
The trusted-proxies sidecar, deployment-level nodeSelector/tolerations,
and HA Core privileged container security context are unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The seed had `ttlSecondsAfterFinished: 3600` and no idempotency check —
so every hour Flux re-applied a fresh Job that downloaded the multi-GiB
backup tarball, *deleted the populated bucket*, and ran `influx restore`
again. "Idempotent" only because the second restore wrote the same data,
but with a window of empty bucket between delete and restore-finish.
Unlike the file-based seeds (music-assistant / audiobookshelf / immich /
paperless), this one operates against the live InfluxDB API after the
chart is up — so the initContainer-in-app-pod pattern doesn't fit. The
fix is API-state idempotency plus persistent Job lifecycle:
- drop `ttlSecondsAfterFinished: 3600` so the Job stays in the cluster
after first success; Flux's wait/healthCheckExprs already gate on
status.succeeded == 1, and a persistent Job won't be re-applied on
subsequent reconciles since the manifest matches actual state
- add a query at the top of the seed body: if the bucket already has
any data point, exit 0 before touching anything destructive. This is
defense-in-depth in case the Job is ever manually re-run on a
populated cluster
Re-running the restore after a fresh backup: kubectl delete the Job
manually; Flux recreates it on next reconcile.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
kube-prometheus-stack's Prometheus resource has ruleSelector
matchLabels: release=kube-prometheus-stack. Without this label the
PrometheusRule was created in-cluster but never loaded into Prometheus
— dead silent alerts. Verified post-fix that the rule appears in
/api/v1/rules.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Same shape as the prior seed-Job → initContainer refactors:
music-assistant (c9ea0af), audiobookshelf (b9717e9), immich (74994aa).
Add a `seed` initContainer to the paperless Deployment that owns the
S3-wait + extract + chown for the data and media RWX PVCs. paperless-
consume is the live drop-folder (not seeded); paperless-redis is mounted
by the redis pod (not paperless), so the seed never touched it.
Paperless used RWX (ceph-filesystem) for its data PVCs, so the
Multi-Attach race that broke music-assistant didn't surface here either —
but the refactor removes one more standalone Flux Kustomization, kills
the hourly Job churn, and matches the pattern across all migrated apps.
Refactor:
- paperless.yaml: add `seed` initContainer (the deployment had no
initContainers before — the seed Job's sentinel was relied on purely
via Flux dependsOn ordering). Mount paths /data + /media match the
legacy Job's; the app container's /usr/src/paperless/{data,media}
mounts are unchanged
- move garage-migration-creds.sops.yaml + pvcs.yaml from
kubernetes/apps/paperless-seed/ into kubernetes/apps/paperless/
- delete kubernetes/apps/paperless-seed/
- collapse the Flux Kustomizations: drop paperless-seed, replace
`dependsOn: paperless-seed` with velero-bootstrap-restore on the app
Kustomization (paperless-db dep stays — separate concern)
- raise app Kustomization timeout 1h → 2h to absorb the worst-case
tarball wait + extract
Also drop the now-stale "during the migration window" comments on the
audiobookshelf and music-assistant pvcs.yaml — the seed Kustomizations
those comments referenced are gone.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Same shape as music-assistant (c9ea0af) and audiobookshelf (b9717e9):
the standalone seed Job had ttlSecondsAfterFinished: 3600 and was being
recreated by Flux every hour for no purpose (sentinel-shortcut would
re-exit within 1s, but the kubelet/image-pull/PVC-mount cycle still
spent resources). Move the seed body into a `seed` initContainer in the
immich-server pod.
Immich uses RWX (ceph-filesystem) for the library volume, so the
Multi-Attach race that broke music-assistant didn't surface here — but
the refactor still applies for consistency, kills the hourly Job churn,
and removes one more Flux Kustomization.
Path note: the legacy seed Job mounted the library at /library; the
immich-server pod mounts it at /data. The new initContainer uses /data
to match the rest of the pod (it's the same PVC, just a different
in-container mount path).
Refactor:
- replace `wait-for-restore` initContainer in server.yaml with the
self-contained `seed` initContainer (preserves runAsRoot +
CHOWN/DAC_OVERRIDE/FOWNER for the recursive 1000:1000 chown)
- move garage-migration-creds.sops.yaml + pvcs.yaml from
kubernetes/apps/immich-seed/ into kubernetes/apps/immich/
- delete kubernetes/apps/immich-seed/
- collapse the standalone immich-seed Flux Kustomization; replace
`dependsOn: immich-seed` on the app Kustomization with
velero-bootstrap-restore (immich-db dep stays — it's a separate
CNPG bootstrap-recovery concern)
- raise app Kustomization timeout 1h → 4h to absorb the worst-case
tarball wait
immich-machine-learning is unchanged — it doesn't touch the library.
fetch-le-staging-roots initContainer is unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Same shape as the music-assistant fix (commit c9ea0af): the standalone
seed Job had ttlSecondsAfterFinished: 3600, so Flux churned a fresh seed
pod every hour with no purpose (sentinel-shortcut would just re-exit
within 1s, but each cycle still added kubelet, image-pull, and PVC-mount
load). Move the seed body verbatim into a `seed` initContainer in the
app pod.
Audiobookshelf uses RWX (ceph-filesystem) for all three of its data
volumes, so the Multi-Attach race that broke music-assistant didn't
surface here — but the refactor still applies for consistency, kills
the hourly Job churn, and removes one more Flux Kustomization.
Refactor:
- replace `wait-for-restore` initContainer with the self-contained
`seed` initContainer (preserves runAsRoot + CHOWN/DAC_OVERRIDE/FOWNER
needed for the recursive 1000:1000 chown over /audiobooks)
- move garage-migration-creds.sops.yaml + pvcs.yaml from
kubernetes/apps/audiobookshelf-seed/ into kubernetes/apps/audiobookshelf/
- delete kubernetes/apps/audiobookshelf-seed/
- collapse the two Flux Kustomizations into one; replace
`dependsOn: audiobookshelf-seed` with the standard infrastructure +
velero-bootstrap-restore deps
`fetch-le-staging-roots` initContainer is unchanged. Init order is
seed → fetch-le-staging-roots → app container.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Prometheus now scrapes Garage's admin endpoint on nas-1
(s3.nas-1.renner.uno:39033 via DSM reverse proxy, plain HTTP) with
bearer-token auth from a SOPS-encrypted Secret mounted into the
kube-prometheus-stack pod. Adds GarageDown / GarageScrapeFailing
alerts; richer rules to follow once the metric surface is wiki-documented.
install-garage-synology.yml: secret writes are now compare-then-write
and the force-recreate task fires on any rotated token, not just
garage.toml changes. Without this, a new metrics_token would never
reach the running Garage process.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The standalone seed Job had `ttlSecondsAfterFinished: 3600`, so Flux
recreated it ~1h after each successful run. The recreated pod had no
node affinity and got scheduled to k8s-3, while the music-assistant app
is pinned to k8s-edge (`topology.kubernetes.io/zone: edge` for hostNetwork
on the IoT VLAN). The shared PVC is RWO Ceph-RBD — cross-node attach is
refused, so the seed pod stuck in Multi-Attach forever and the
music-assistant-seed Kustomization stayed Reconciling.
Move the seed body verbatim into a `seed` initContainer in the app
Deployment. Same pod = same kubelet RBD mount, no Multi-Attach is even
possible. The existing `[ -f /data/.restore-done ]` sentinel keeps
ordinary pod restarts to <1s; on a fresh cluster rebuild the
initContainer does the full extract before the app container starts —
exactly the per-rebuild semantics we want.
Refactor:
- replace `wait-for-restore` initContainer with a self-contained `seed`
initContainer that owns the s3 wait/extract/sentinel logic
- move garage-migration-creds.sops.yaml + pvcs.yaml from
kubernetes/apps/music-assistant-seed/ into kubernetes/apps/music-assistant/
- delete kubernetes/apps/music-assistant-seed/
- collapse the two Flux Kustomizations into one; replace the
`dependsOn: music-assistant-seed` chain with the standard
infrastructure deps it inherits
- raise the app Kustomization timeout to 4h to cover the worst-case
3h-tarball-wait window the seed Job had
The same latent race exists in audiobookshelf-seed and paperless-seed
(they happen to co-locate on k8s-3 with their app pods today, so they
work by luck). Worth a follow-up to apply the same pattern there.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous http_body piped curl through `tee | grep -q`. grep -q has no
stdout, so when curl had a transient TLS/connection failure (or returned
an unexpected body), the failure summary line was a bare `-- ` with no
code, no body, no diagnostic.
Investigated after grafana /api/health flaked twice in CI today while
working fine when reprobed seconds later. Confirmed via ingress access
log that every request was 200 with the expected body — the fail was
runner-side opacity, not a real DB issue.
Now: 3x retry with 2s backoff, --max-time bumped 10s -> 15s, each attempt
logs `attempt=N code=XXX body=...` to /tmp/check.out so on a real failure
the actual response is in the summary tail.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Existing e2e was readiness-only — every probe could pass on a cluster that
had restored zero data, lost OIDC config, or stopped shipping logs. Add
sections that exercise actual behavior:
- Data presence (post-restore): paperless documents_document, authentik
users + applications, audiobookshelf library entries — threshold-based
(`>= N`) so adds/removes don't flake the test.
- Observability data flow: Loki query for K8s log volume in last 5m,
Prometheus count(up == 1) >= 25, Alertmanager has no active critical
alerts. Run from a short-lived helper pod (curlimages/curl) so we don't
exec into product pods.
- Authentik OIDC discovery: per-app .well-known/openid-configuration must
return JSON with issuer + authorization_endpoint. Catches Authentik
config drift that wouldn't show up on the live/ready probes.
- Ceph RGW S3 round-trip: PUT/GET/DELETE via the existing
paperless/migration-bucket OBC, run as a Job so credentials come from
envFrom and never leak to argv.
Also add .woodpecker/e2e.yml so the same script runs on cron between
rebuilds (configure schedule via Woodpecker Cron tab; suggested daily).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PCI-passthrough OSD disks survive VM destroy/recreate but the Rook mon DB
on Talos EPHEMERAL does not. Without wiping, the new cluster's fresh FSID
mismatches the BlueStore signature on /dev/nvme0n1 and ceph-volume skips
each OSD with "belonging to a different ceph cluster", leaving 0 OSDs.
Add a wipe-osd-disks step between talos-bootstrap and export-access that
runs `talosctl wipe disk nvme0n1 --method FAST` against k8s-1/2/3
(k8s-edge has no OSD).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The ansible.cfg in the repo is ignored at runtime because the working
dir is world-writable in the CI container, so its roles_path,
host_key_checking and vault_password_file settings don't apply. The
local-exec provisioner that runs the per-LXC install playbook needs
all three configured via env vars instead.
Matches the env pattern used by update-all.yml / deploy-caddy.yml:
- ANSIBLE_ROLES_PATH=../roles:./roles → finds the common, docker-install
and per-service roles (otherwise it only searches plays/roles/)
- ANSIBLE_HOST_KEY_CHECKING=false → first-contact SSH to a freshly
created LXC won't have the key fingerprint cached
- VAULT_PASS + /tmp/.vault_pass +
ANSIBLE_VAULT_PASSWORD_FILE → group_vars/all/vault.yml is
encrypted; common role fails to load vars without it
- ANSIBLE_FORCE_COLOR=true → pretty logs, matches house style
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
community.proxmox.proxmox dynamic inventory plugin reads token_id and
token_secret from PROXMOX_TOKEN_* env vars (the inventory yaml has
those keys commented out, expecting envs). Without them the plugin
fails: \"You must specify either a password or both token_id and
token_secret.\" Matches the pattern used in update-all.yml.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
