Pipeline 846 cold-start was clean end-to-end EXCEPT immich-server
crash-looped at runtime with:
PostgresError: permission denied to create extension "vector"
HINT: Must be superuser to create this extension.
Newer immich (>= the version we're on) issues `CREATE EXTENSION
IF NOT EXISTS vector` (pgvector) at startup as part of the vchord
migration path. The legacy dump only ships pgvecto.rs (`vectors`) so
pg_restore doesn't create pgvector. The immich role isn't a
superuser, and pgvector is not a trusted extension, so the runtime
CREATE fails.
Pre-create the extensions immich uses, as postgres, in the seed Job
right after CREATE DATABASE. The dump's own `CREATE EXTENSION
IF NOT EXISTS` calls then short-circuit (extension already exists),
and so does immich's runtime CREATE — no privilege check on the
non-trusted extension because it's already installed.
List: cube, earthdistance (legacy geo helpers), vector (pgvector,
runtime), vectors (pgvecto.rs, dump), vchord (VectorChord, runtime
migration target).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pipeline 849's agent step failed at "Not logged in · Please run
/login" despite CLAUDE_CODE_OAUTH_TOKEN being correctly injected
from the Woodpecker secret. Per Claude Code's authentication.md:
"Bare mode does not read CLAUDE_CODE_OAUTH_TOKEN. If your script
passes --bare, authenticate with ANTHROPIC_API_KEY or an
apiKeyHelper instead."
The headless guide had recommended --bare for CI, but didn't flag
this incompatibility with subscription auth. We use the
subscription-backed OAuth token, not an API key, so --bare is the
wrong choice here.
Trade-off: without --bare, claude loads its default config
(CLAUDE.md, hooks, MCP, skills) on startup. None of those are
present in the ci-claude image's HOME so the loss is theoretical
for now, but if any of them get added we'd need to revisit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
aquasec/trivy:latest doesn't ship jq, so the per-finding summary
lines in iac-scan and image-scans were failing on "command not found"
and (because they ran as the last command in their respective `- |`
blocks despite the `|| true`) tripping the step's exit code.
Trivy steps now produce JSON only — no jq dependency on that image.
The human-readable per-finding summary moves to security-fix-agent,
which runs on ci-claude (jq baked in) and prints both IaC misconfigs
and image vulns to the log before the gate check, so the log is
still useful even when the agent exits early on no findings.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Push-triggered scans were causing every code change to launch the
agent loop, which is both noisy (mostly nothing-changed runs) and
fork-bombs the bot's own PR branches into rerunning the full
workflow on every commit. The agent's branch=master step gate
prevented an actual fork-bomb, but the scan steps themselves were
still wasting CI on every push.
Schedule + manual is enough: the cron is the primary cadence, and
manual covers ad-hoc triggers when something specific needs scanning.
The agent's branch=master gate stays in place as defense-in-depth.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The cron security-scan now ends in an autonomous fix loop:
iac-scan → trivy-iac.json
image-scans → trivy-image-N.json (per scanned image)
security-fix-agent (master only):
1. Gates on >=1 CRITICAL/HIGH finding across all JSON outputs.
2. Creates branch security-fix/<date>-<short-sha>.
3. Hands the workspace to Claude Code in headless --bare mode
(subscription-backed via CLAUDE_CODE_OAUTH_TOKEN, no API key)
with a narrow tool allowlist (Read/Edit/Write/Glob/Grep —
no Bash). The agent reads the JSON, applies surgical fixes
under ansible/, opentofu/, kubernetes/, ci/images/, and writes
a Markdown summary to /tmp/security-fix-summary.md.
4. If the agent produced a diff, the step commits as
"Security Fix Agent <security-bot@renner.uno>" and pushes to
Gitea, then opens a PR via the pulls API using a dedicated
security_bot_token (NOT gitea_admin) so the PR is visibly
machine-authored.
The step is gated to branch=master so it doesn't fire on its own
PR branches — those still trigger the scan workflow as a re-validation
gate, but skip the agent (no fork-bombing).
Trivy now writes JSON output (--format json -o), and findings appear
both in the JSON and as one-line summaries in the step log via jq.
The legacy --exit-code 1 was kept off so the workflow always reaches
the agent step on master; on PR branches the scans are informational
and a human reviews the agent's PR.
Required Woodpecker secrets (added by user, not by this commit):
- claude_code_oauth_token (claude setup-token output)
- security_bot_token (Gitea PAT for the security-bot user)
Required Gitea setup (also out of band):
- User "security-bot" with email security-bot@renner.uno
- Write collaborator on gitea_admin/homelab
Agent constraints live in ci/agents/security-fix/prompt.md so they're
diffable + reviewable separately from the pipeline plumbing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fifth image in the ci-* family. Bakes Node.js 22 + the Claude Code
CLI (@anthropic-ai/claude-code) plus git, openssh-client, curl, jq.
Used by the upcoming security-fix-agent step in security-scan.yml,
which runs Claude headlessly inside CI to propose fixes for Trivy
findings and open a PR.
Auth pattern (deferred to the consumer commit): subscription-backed
CLAUDE_CODE_OAUTH_TOKEN secret, injected as an env var, no API key.
build-ci-images.yml gains a build-claude step that mirrors the
existing four (kaniko, parallel, registry-mirror cache, date + latest
tags pushed to gitea.renner.uno/gitea_admin/).
Pinning: Containerfile uses CLAUDE_CODE_VERSION=latest at image-build
time. The resulting date-tagged image freezes whatever was current
when build-ci-images ran. Bumping = trigger build-ci-images, get a
new date tag, bump the consumer pin.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Container registry blob storage filled the gitea LXC's rootfs.
build-ci-images pipeline 841 had three of four kaniko image pushes
fail with HTTP 500 on the gitea v2 registry endpoint; gitea host
journal showed "no space left on device" across alloy + tmp writes
at the same instant.
4G is way too tight for a registry with the ci-{ansible,k8s,tofu,
caddy-builder} family at ~200-500MB each, especially with kaniko
keeping the previous date-tagged copy until GC. 16G gives headroom
for several months of date-tagged image churn before the next bump.
bpg's proxmox_virtual_environment_container does in-place rootfs
resize for LXC (`pct resize`), no restart needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pipeline 832 (build-ci-images) ran 4 kaniko builds in parallel
across 2 agents (woodpecker_agent_count=2). build-ansible got OOM-
killed — 2 GiB per agent split across 2 concurrent kaniko jobs is
~1 GiB per build, and ansible + collections + pip deps under
kaniko's tmpfs layer cache exceeded that.
4 GiB per agent gives ~2 GiB per concurrent build, which has
enough headroom for the largest of the four images. If we ever
hit the same wall again the next bump is doubling the agent
count, not the memory — kaniko parallelism doesn't help much
beyond ~2 GiB.
BPG's proxmox_virtual_environment_container does in-place memory
updates for LXC (cgroup-level, no restart), so the apply is safe
to run while the agents are servicing other jobs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The build step pulled golang:1.26 from Docker Hub on every run and
then `go install github.com/caddyserver/xcaddy/cmd/xcaddy@latest`,
so the xcaddy version was whatever HEAD shipped that day. The
release step pulled alpine:latest and `apk add curl jq` for nothing
more than two API calls against gitea.
- build: golang:1.26 + go install xcaddy@latest
→ ci-caddy-builder:2026-05-01 (xcaddy v0.4.5 baked in)
- release: alpine:latest + apk add curl jq
→ ci-k8s:2026-04-29 (curl + jq baked in)
ci-caddy-builder was just produced by build-ci-images pipeline 832
(commit 60b9698). xcaddy version bumps now happen by editing
ci/images/caddy-builder/Containerfile and bumping the date pin
here, same flow as the other ci-* consumers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fourth image in the ci-* family (alongside ci-ansible, ci-k8s,
ci-tofu). Bakes the Go toolchain (golang:1.26 base) plus xcaddy
pinned to v0.4.5 so the build-caddy pipeline doesn't have to
`go install ...@latest` on every run.
build-ci-images.yml gains a build-caddy-builder step that mirrors
the existing three (kaniko, parallel, pull-through-cached registry,
date + latest tags pushed to gitea.renner.uno/gitea_admin/).
The build-caddy.yml consumer swap is a separate commit so it can
land after the first build-ci-images run produces the image.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The recreate-vms step still pulled ghcr.io/opentofu/opentofu:1.11
and ran `apk add --no-cache git openssh-client curl` on every run —
the only step in this pipeline that hadn't been migrated to a
prebuilt ci-* image. Each invocation was an apk roundtrip plus a
GHCR pull, and the resulting tool versions were whatever Alpine
edge happened to ship that day.
Swap to gitea.renner.uno/gitea_admin/ci-tofu:2026-04-29, which
already bakes git, openssh-client, curl, and tofu 1.11.0. The
inline `apk add` line goes away; everything else in the step
keeps working unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The RustFS → Garage tfstate cutover is done (commit fb82f5d). The
pipeline was always documented as one-shot and the source bucket on
RustFS is read-only / wedged anyway, so re-running it accomplishes
nothing.
Removing it also takes out two more inline `apk add` calls
(aws-cli, ca-certificates, jq) that nudged us toward upstream Alpine
on every run.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two fixes after pipeline 829's wipe-backups step blew up:
- aws-cli is not in ci-k8s:2026-04-29 — wipe ran "aws s3 rm ..." and
got "/bin/sh: aws: not found", then exited 0 because set -eu doesn't
catch failures inside pipes / command substitutions.
ci/images/k8s/Containerfile gains aws-cli; pushing this triggers the
build-ci-images pipeline, which builds gitea.renner.uno/gitea_admin/
ci-k8s:2026-05-01 (date tag).
The wipe-backups step is bumped to that tag and switches to
set -euo pipefail with an explicit "remaining objects must be 0"
check after each rm — so a failed wipe halts the rebuild before the
new cluster's seed Jobs run against stale state.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cold-start rebuilds need s3://k8s-cnpg/* and s3://k8s-velero/* empty
before the new cluster's seed Jobs run, otherwise the seed gate skips
seeding (sees a stale base backup) and CNPG bootstrap.recovery tries
to recover from data that doesn't match the new cluster, sticks in
"Setting up primary" until backoffLimit exhausts.
Wiping by hand from a workstation has a race: between `aws s3 rm` and
the next pipeline triggering, the still-alive old cluster's barman
plugin can lay down a fresh scheduled-backup base, re-poisoning the
bucket.
Wiping inside the pipeline AFTER recreate-vms closes the race —
recreate-vms destroys the old VMs (and therefore its barman + Velero
pods), so by the time we run `aws s3 rm` there is no writer left.
Step is opt-in via WIPE_BACKUPS=true to avoid clobbering the bucket
on routine warm-restart rebuilds:
woodpecker pipeline create gitea_admin/homelab \
--branch master --var PIPELINE=rebuild-k8s \
--var WIPE_BACKUPS=true
Requires Woodpecker secrets: garage_cnpg_key_id, garage_cnpg_secret_key,
garage_velero_key_id, garage_velero_secret_key (RW perms on the
respective buckets via the cnpg-key / velero-key Garage keys).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous "drop --role + REASSIGN" approach broke ownership: with
--no-owner --no-privileges the dump's "ALTER TABLE ... OWNER TO X" /
"GRANT ... TO X" statements are stripped, so user tables end up
owned by postgres in PGDATA. REASSIGN OWNED BY postgres TO <app>
then errors with "cannot reassign ownership of objects ... required
by the database system" — the whole REASSIGN rolls back and tables
stay postgres-owned. Recovered cluster has tables the app role
can't write to.
Cleaner: keep pg_restore running as postgres (no --role) AND keep
the dump's ownership/ACL statements. As postgres (superuser),
ALTER TABLE OWNER TO <app> and GRANT ... TO <app> just succeed,
regardless of extension schemas. The resulting PGDATA matches the
legacy DB's ownership/ACL state exactly. CNPG bootstrap.recovery
streams it back unchanged.
Applied to authentik / paperless / immich seed Jobs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the schema-permission workarounds (pre-create extensions,
GRANT USAGE on vectors/vchord) with the cleaner shape: drop --role
from pg_restore so it runs as postgres (superuser), then REASSIGN
ownership to the app role.
Why the workarounds existed:
pg_restore --role=immich → SET ROLE immich after connecting →
every CREATE EXTENSION / CREATE TABLE referencing extension-owned
schemas (vectors.vector, ll_to_earth, …) hit "permission denied".
Patched per-failure with pre-create + GRANT USAGE — fragile, dump-
specific, fumbles around the actual issue.
Why dropping --role works:
Without --role pg_restore stays as postgres for the whole replay —
superuser owns nothing of consequence in the schemas, so every
CREATE/COPY/ALTER inside the dump just works. After the load,
REASSIGN OWNED BY postgres TO <app> + GRANT ALL ON SCHEMA public
hands ownership back to the app role for runtime.
This is what the seed job is supposed to do: produce a functional
PGDATA snapshot. The barman base backup of that PGDATA is then
self-contained — CNPG bootstrap.recovery streams it back as-is,
no schema fumbling on the recovery side.
Applied symmetrically to authentik / paperless / immich seed Jobs.
The immich-specific pre-create/GRANT blocks added in earlier commits
are removed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
immich seed Job: pre-creating extensions wasn't enough — vectors and
vchord create their own schemas (`vectors`, `vchord`) owned by postgres.
pg_restore under SET ROLE immich then hits "permission denied for
schema vectors" when CREATE TABLE face_search references
vectors.vector(512). Add GRANT USAGE, CREATE on those schemas to
TARGET_OWNER right after CREATE EXTENSION.
bootstrap.sh wait-apps: pipeline 825 timed out at music-assistant (10m).
On cold-start the wait timer for music-assistant starts while
velero-bootstrap-restore is still reconciling (~15min), eating most of
the 10m budget on the dependency. Bump to 30m (180 iters × 10s).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
pg_restore runs with --role=immich which issues SET ROLE immich after
connecting, so every CREATE EXTENSION in the legacy dump fails with
"permission denied to create extension" (immich is not superuser).
Dependent vector tables (face_search, smart_search) then fail to
restore, and CNPG bootstrap.recovery on the live cluster ends up with
a backup that has the schema but no vector tables.
Per the wiki pattern (pg-restore-no-superuser-if-not-exists): the dump
emits `CREATE EXTENSION IF NOT EXISTS`, which short-circuits the
privilege check when the extension already exists. So pre-create the
required extensions as postgres before pg_restore:
cube (dependency of earthdistance)
earthdistance (geo helpers)
vectors (pgvecto.rs — legacy vector extension)
vchord (VectorChord — current vector extension)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ghcr.io/corentingiraud/cnpg-pgvector-vectorchord:16-migration inherits
the CNPG postgresql base image which sets the postgres user as the
default container user (security default). The seed script needs root
for `apt-get install` (barman, awscli), `chown postgres:postgres
$PGDATA`, and `su postgres -c initdb`. Without runAsUser=0 it fails
immediately with "Permission denied" on /var/lib/apt/lists/lock.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The im-pg seed Job's ephemeral Postgres ran the standard postgres:16
image, which doesn't ship pgvecto.rs or vchord. The legacy dump's
CREATE EXTENSION vectors / CREATE EXTENSION vchord silently failed
during pg_restore (only "warnings"), leaving the face_search /
smart_search vector tables un-restored. The barman-cloud-backup
captured this incomplete schema; the live im-pg Cluster recovered
from it; immich-server then crash-looped trying to CREATE EXTENSION
vector itself as the non-superuser immich role
("permission denied to create extension").
Match the seed Job's image and shared_preload_libraries to the live
im-pg Cluster spec (postgres.yaml):
image: ghcr.io/corentingiraud/cnpg-pgvector-vectorchord:16-migration
shared_preload_libraries = 'vectors.so,vchord.so'
Now the seed's ephemeral PG can load pgvecto.rs catalog objects from
the dump and execute the dump's CREATE EXTENSION calls, so the backup
written to Garage actually contains the vector tables.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three bugs in the cold-start seed Jobs found by pipeline 819's first
authentik attempt:
1. barman-cloud-backup connects to PG via SQL to start/stop the backup;
without --host/--port/--user it falls back to the OS user (root),
which has no PG role → FATAL: role "root" does not exist. Add
--host 127.0.0.1 --port 5432 --user postgres.
2. Idempotency gate "any object under <BARMAN_DEST>/" is too loose:
a failed prior attempt left a single marker object in Garage; the
retry pod's gate said "1 object — seed already done" and exited 0,
marking the Job Complete with no real backup. Tighten to check
<BARMAN_DEST>/<BARMAN_SERVER>/base/ — only a successful backup
writes under base/.
3. archive_timeout was unset; archive_command fires async after
pg_switch_wal so end_wal of the base backup may not upload before
we stop PG. Added archive_timeout='60s' and a post-pg_switch_wal
poll on PGDATA/pg_wal/archive_status/*.ready draining (matches the
old migrate-* pipeline pattern).
Also added a final barman-cloud-backup-list call as a sanity check
that a backup actually landed before declaring Seed complete.
Applied symmetrically to authentik / paperless / immich seed-job.yaml.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds kubernetes/apps/{authentik,paperless,immich}/seed-job.yaml — a
single k8s Job per app that:
1. Idempotency-gates on `aws s3 ls s3://k8s-cnpg/<cluster>/` being
empty (any object means a prior seed or live cluster archived).
2. legacy-dump init container: ssh + pg_dump (or docker exec
pg_dump for paperless/immich which run their legacy DB in
docker-compose) → emptyDir.
3. main container: ephemeral Postgres (initdb + WAL archive_command
wired to barman-cloud-wal-archive) + pg_restore the dump +
barman-cloud-backup writes a base backup to s3://k8s-cnpg/<cluster>/.
The Cluster's bootstrap.recovery now has a producer: the seed Job
populates Garage; CNPG's recovery Job retries on empty bucket, then
succeeds once the seed lands the base backup + end_wal. No more
chicken-and-egg where the in-pod init container needed the cluster
already up.
In-pod init containers stripped of DB seed:
- authentik: removed entirely (no PVC seed needed). HelmRelease loses
serviceAccountName, initContainers, ssh volume.
- paperless / immich: rewritten to PVC rsync only, sentinel-guarded
(`/data/.seed-done`) so subsequent restarts skip the rsync. No
kubectl, no Backup CR creation, no serviceAccountName.
seed-rbac.yaml deleted from authentik / paperless / immich (the only
RBAC consumer was the in-pod init container's kubectl calls). influxdb
keeps its seed-rbac (its init container still creates Velero Backup CRs).
bootstrap.sh wait-apps timeouts bumped for cold-start reality:
apps 5m → 15m (Ceph mon/mgr/OSD bring-up on fresh cluster)
authentik 5m → 20m (seed Job + CNPG recovery from the seed)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
authentik-db, paperless-db, immich-db are no longer separate Flux
Kustomizations — their CNPG Cluster, ObjectStore, ObjectBucketClaim,
Barman creds, and ScheduledBackup move into the app's Kustomization
(kubernetes/apps/<app>/) and the *-db Flux KS in
kubernetes/clusters/homelab/<app>.yaml is dropped.
Why: the prior split made the app KS dep-block on the *-db KS reaching
Ready, which kstatus evaluated against the CNPG Cluster's Ready
condition. On a cold-start with empty Garage, the Cluster never
reaches Ready (bootstrap.recovery has nothing to recover), so the app
KS never deploys and the seed initContainer that's supposed to
populate the DB never runs — chicken-and-egg. Merging removes the
inter-KS gate; the app Pod (with seed initContainer) and the Cluster
come up in the same KS unit.
Bootstrap mode change to make the seed actually viable lands in the
follow-on commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
wait-apps was burning the full ~5h timeout chain (sum of per-KS
timeouts up to 1440x10s for immich) before reaching e2e even when
authentik/paperless/immich were doomed by an empty CNPG bucket.
Removing || true lets set -euo pipefail halt at the first KS
timeout, surfacing the cold-start failure in minutes instead of
hours.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
DSM reverse proxy is HTTP-only and breaks the Noise handshake; expose
RPC directly so the CLI can connect from a workstation. rpc_secret
remains the auth boundary.
The seed initContainer's first failure (no rsync on legacy authentik-
db-1) wedged the Deployment rollout: failing pod kept the RS at 1
replica, Helm's --wait timed out, Helm failed and didn't apply later
fixes. The Deployment is stuck at the original broken template, the
broken RS is recreated by the controller.
disableWait makes Helm just apply the rendered manifests and exit.
The Deployment controller rolls out asynchronously; the seed fixes
land without Helm getting blocked.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The seed initContainer creates Backup CRs with no labels (chart
operator doesn't auto-label them either). The gate's
"-l cnpg.io/cluster=\$CLUSTER" selector therefore returned 0 every
time, so every pod restart re-ran the full seed (pg_restore against
the live cluster, app-session impact, etc.).
Each app namespace owns at most one CNPG cluster, so just count any
completed Backup in the namespace. Fixes the re-seed loop the cold-
start test exposed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
authentik-db-1 doesn't have rsync installed, so the seed initContainer's
"rsync \$LEGACY:\$LEGACY_DUMP /tmp/legacy.dump" spawned a remote rsync
that failed with "command not found". Switch this single-file dump
transfer to scp (always available via ssh).
The other seeds (paperless, immich, audiobookshelf, influxdb) keep
rsync because their legacy hosts have it; music-assistant already
uses ssh+tar for the same reason.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both are RWO ceph-block PVCs whose running app pods hold them RW on
the same node where Velero's CSI snapshot-exposer tries to mount the
snapshot read-only. The kernel rejects the second RBD mount with
"cannot mount /dev/rbdN read-only", leaving daily Backup pods stuck
in ContainerCreating indefinitely.
paperless-redis is a cache so it's safe to skip; music-assistant-data
is repopulatable from the legacy LXC via the seed initContainer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Verified against the upstream chart values.yaml / templates:
- authentik chart exposes server.serviceAccountName (string, references
an existing SA), not the bitnami-style top-level serviceAccount.{create,
name}. server.initContainers and server.volumes were already correct.
- influxdb2 chart has NO .Values.initContainers hook in its StatefulSet
template. Add the seed initContainer via spec.postRenderers (Kustomize
patch on the rendered StatefulSet "influxdb-influxdb2"). The SSH
Secret volume goes through the chart's supported top-level `volumes:`
value (not `extraVolumes`).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replicates the paperless pattern (49a1e75) for immich, audiobookshelf,
music-assistant, authentik, and influxdb:
- Seed work moves into an initContainer in each app's Pod template
(Deployment for immich/audiobookshelf/music-assistant; HelmRelease
values for authentik server + influxdb StatefulSet).
- Same Pod owns the PVC → no RWO/RWX contention regardless of access
mode. Main container only starts after seed completes (initContainer
ordering enforces "app starts after seed").
- Idempotency gate: completed CNPG Backup (DB apps) or completed
Velero Backup (PVC apps); skip-fast on warm starts.
- SOPS-encrypted SSH secrets and seed-RBAC moved into kubernetes/apps/
<app>/. PVC-only apps' RBAC is cross-namespace (Role+Binding in
velero ns; SA in app ns).
- Drops kubernetes/migrations/ entirely and the five
kubernetes/clusters/homelab/<app>-seed.yaml Flux Kustomizations.
- Removes <app>-seed from each app Kustomization's dependsOn.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drops the standalone seed-Job pattern (kubernetes/migrations/paperless/
+ kubernetes/clusters/homelab/paperless-seed.yaml) for paperless. The
seed work moves into an initContainer in the paperless Deployment:
- idempotency-gated against completed CNPG Backups
- rsyncs data + media PVCs from the legacy VM
- pg_dump on legacy + rsync the file + pg_restore into pprl-pg
- creates a fresh CNPG Backup (fire-and-forget)
Same Pod owns the PVCs, so RWO/RWX access mode is irrelevant — no
contention with the existing app pod. initContainer ordering enforces
"main container starts after seed completes" by Pod semantics, not
Flux dependsOn timing.
SOPS-encrypted SSH secret moves to apps/paperless/. RBAC for the seed
SA moves into apps/paperless/seed-rbac.yaml.
This is the pattern; immich/audiobookshelf/music-assistant/authentik/
influxdb to follow once the shape is reviewed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Same fix as the pg_dump rewrite: stream-through-ssh of a multi-GB
data dir has no compression/parallelism/resume. rsync handles it
properly. Adds rsync to the apk install (ci-k8s doesn't ship it).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous "ssh ... pg_dump > /tmp/legacy.dump" piped the entire
dump through an SSH stdout tunnel — no compression, no parallelism,
no resume. For multi-GB dumps that's the bottleneck.
Run pg_dump on the legacy host writing to /tmp/<db>-seed-dump there,
then rsync the file. rsync handles the transfer efficiently and we
get progress + (in principle) resume on retry. Cleanup ssh removes
the legacy temp file. Authentik also gains rsync in its apk install.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 1 rsyncs books while the legacy audiobookshelf systemd unit
stays up — no --delete so the running app's reads don't race. Books
are append-mostly so phase 1 carries the bulk of the data.
Phase 2 stops the unit briefly for a consistent snapshot of config +
metadata (SQLite, can't be rsynced live) and a delta of books with
--delete. Legacy downtime drops from hours to minutes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
audiobookshelf, music-assistant, influxdb seed-ssh.sops.yaml now hold
the SOPS-encrypted SSH key, so the Flux Kustomizations can reconcile
the seed Jobs end-to-end without operator intervention.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pipelines previously scaled the app to 0 + suspended Flux *before*
checking whether the Job would no-op. If a completed Backup already
exists, the Job's idempotency gate trips and the seed exits 0 — but
we'd already taken the app down for it.
Move the same gate query to the very top of each pipeline (right after
kubeconfig setup) and exit 0 if a Backup exists. The app stays up
when the seed isn't needed; only the cold path quiesces.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ci-k8s already ships kubectl/sops/openssh/curl/bash/tar. Replaces
postgres:16 + apt-get install + curl-install kubectl on DB seeds
(paperless/immich/authentik) and alpine:3.21 + apk add + curl-install
kubectl on PVC seeds (audiobookshelf/music-assistant/influxdb).
Net per-Job startup: a few seconds of apk for postgresql16-client
(DB seeds) or rsync (audiobookshelf), nothing for music-assistant /
influxdb. Same registry-mirror cache hit either way; far less network
on cold start.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Same shape as paperless/immich/authentik: idempotency-gated in-cluster
Job that rsyncs/streams legacy data into the live PVCs and creates a
Velero Backup CR labelled with the schedule. Pipelines collapse to
trigger-and-wait. Flux Kustomizations gate the app on the seed.
PVC-only apps gate on Velero Backup CRs (which live in the velero
namespace and survive an app-namespace wipe), which is more robust than
the CNPG-Backup gate used for DB apps. Cross-namespace RBAC: SA in
<app> namespace, Role+Binding in velero namespace.
music-assistant inherits the edge-zone nodeSelector + workload-class
toleration from its app pod (RWO PVC pinned to k8s-edge). influxdb
quiesces the StatefulSet rather than a Deployment.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
