konairius/homelab
10
0
Fork 0
Code Issues 9 Pull Requests 2 Packages Projects Releases 1 Wiki Activity

refactor(paperless): seed becomes initContainer in the app Pod

Browse Source 
Drops the standalone seed-Job pattern (kubernetes/migrations/paperless/
+ kubernetes/clusters/homelab/paperless-seed.yaml) for paperless. The
seed work moves into an initContainer in the paperless Deployment:
- idempotency-gated against completed CNPG Backups
- rsyncs data + media PVCs from the legacy VM
- pg_dump on legacy + rsync the file + pg_restore into pprl-pg
- creates a fresh CNPG Backup (fire-and-forget)

Same Pod owns the PVCs, so RWO/RWX access mode is irrelevant — no
contention with the existing app pod. initContainer ordering enforces
"main container starts after seed completes" by Pod semantics, not
Flux dependsOn timing.

SOPS-encrypted SSH secret moves to apps/paperless/. RBAC for the seed
SA moves into apps/paperless/seed-rbac.yaml.

This is the pattern; immich/audiobookshelf/music-assistant/authentik/
influxdb to follow once the shape is reviewed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
k.renner
2026-04-29 22:46:17 +02:00
co-authored by Claude Opus 4.7
parent 85a11380d9
commit 49a1e75d08
8 changed files with 159 additions and 276 deletions
Download Patch File Download Diff File Expand all files Collapse all files
kubernetes/apps/paperless/kustomization.yaml
+2
View File
@@ -4,6 +4,8 @@ resources:
- pvcs.yaml
- secrets.sops.yaml
- ftp-scanner-secret.sops.yaml
- seed-rbac.yaml
- seed-ssh.sops.yaml
- redis.yaml
- gotenberg.yaml
- tika.yaml
kubernetes/apps/paperless/paperless.yaml
+122
View File
@@ -45,11 +45,129 @@ spec:
app.kubernetes.io/name: paperless
spec:
enableServiceLinks: false
serviceAccountName: paperless-seed
securityContext:
fsGroup: 1000
fsGroupChangePolicy: OnRootMismatch
seccompProfile:
type: RuntimeDefault
initContainers:
# Atomic seed: idempotency-gated rsync of the data + media PVCs
# from the legacy paperless VM and pg_restore against the live
# pprl-pg cluster. Replaces the standalone seed Job pattern so
# the PVC is mounted by a single Pod (no RWO contention) and
# the main container only starts after seed completes (init-
# container ordering enforces "app starts after seed").
- name: seed
image: gitea.renner.uno/gitea_admin/ci-k8s:2026-04-29
env:
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: CLUSTER
value: pprl-pg
- name: PGHOST
value: pprl-pg-rw.paperless.svc
- name: PGPORT
value: "5432"
- name: PGDATABASE
value: paperless
- name: PGUSER
valueFrom:
secretKeyRef:
name: pprl-pg-app
key: username
- name: PGPASSWORD
valueFrom:
secretKeyRef:
name: pprl-pg-app
key: password
- name: LEGACY_HOST
value: paperless.srv.renner.uno
command:
- bash
- -c
- |
set -euo pipefail
log() { printf '[%s] %s\n' "$(date -u +%H:%M:%S)" "$*"; }
log "Installing rsync + postgresql-client ..."
apk add --no-cache rsync postgresql16-client >/dev/null
EXISTING=$(kubectl get backups.postgresql.cnpg.io -n "$NAMESPACE" \
-l "cnpg.io/cluster=$CLUSTER" \
-o jsonpath='{range .items[?(@.status.phase=="completed")]}{.metadata.name}{"\n"}{end}' \
2>/dev/null | grep -c . || true)
if [ "$EXISTING" -gt 0 ]; then
log "$EXISTING completed CNPG Backup(s) for $CLUSTER — skipping seed"
exit 0
fi
log "No completed CNPG Backup for $CLUSTER — proceeding with seed"
mkdir -p /root/.ssh
install -m 600 /ssh/id_ed25519 /root/.ssh/id_ed25519
printf 'Host *\n\tStrictHostKeyChecking no\n\tUserKnownHostsFile /dev/null\n' > /root/.ssh/config
LEGACY="root@$LEGACY_HOST"
# ---- 1. PVC rsync ----
log "rsync paperless_data → /usr/src/paperless/data ..."
rsync -aH --delete --info=progress2 \
"$LEGACY:/var/lib/docker/volumes/paperless_data/_data/" /usr/src/paperless/data/
log "rsync paperless_media → /usr/src/paperless/media ..."
rsync -aH --delete --info=progress2 \
"$LEGACY:/var/lib/docker/volumes/paperless_media/_data/" /usr/src/paperless/media/
chown -R 1000:1000 /usr/src/paperless/data /usr/src/paperless/media
du -sh /usr/src/paperless/data /usr/src/paperless/media
# ---- 2. Legacy DB dump (run on legacy, rsync the file) ----
log "Locating paperless DB container on $LEGACY_HOST ..."
DB_CT=$(ssh "$LEGACY" "docker ps --filter label=com.docker.compose.service=db --format '{{.Names}}' | head -1")
[ -n "$DB_CT" ] || { log "no DB container found on $LEGACY_HOST"; exit 1; }
LEGACY_DUMP=/tmp/$PGDATABASE-seed-dump
log "Running pg_dump on $LEGACY_HOST → $LEGACY_DUMP ..."
ssh "$LEGACY" "docker exec -i $DB_CT pg_dump --clean --if-exists --no-owner --no-privileges --format=custom -U $PGUSER $PGDATABASE > $LEGACY_DUMP"
log "rsync dump file from legacy ..."
rsync -aH --info=progress2 "$LEGACY:$LEGACY_DUMP" /tmp/legacy.dump
ssh "$LEGACY" "rm -f $LEGACY_DUMP" || true
test -s /tmp/legacy.dump || { log "dump empty — refusing to restore"; exit 1; }
log "Dump size: $(du -h /tmp/legacy.dump | cut -f1)"
# ---- 3. pg_restore into live cluster (commit point) ----
log "pg_restore → $PGHOST/$PGDATABASE as $PGUSER ..."
pg_restore --clean --if-exists --no-owner --no-privileges \
--role="$PGUSER" \
--dbname="$PGDATABASE" \
/tmp/legacy.dump \
|| log "pg_restore completed with warnings (expected)"
psql -c "REASSIGN OWNED BY postgres TO $PGUSER;" || true
psql -c "GRANT ALL ON SCHEMA public TO $PGUSER;" || true
# ---- 4. Trigger CNPG Backup (fire-and-forget) ----
BACKUP_NAME="$CLUSTER-seed-$(date -u +%Y%m%d-%H%M%S)"
log "Creating CNPG Backup $BACKUP_NAME ..."
cat <<YAML | kubectl apply -f -
apiVersion: postgresql.cnpg.io/v1
kind: Backup
metadata:
name: $BACKUP_NAME
namespace: $NAMESPACE
spec:
cluster:
name: $CLUSTER
method: plugin
pluginConfiguration:
name: barman-cloud.cloudnative-pg.io
YAML
log "Seed complete: PVC + DB committed; Backup CR submitted."
volumeMounts:
- name: data
mountPath: /usr/src/paperless/data
- name: media
mountPath: /usr/src/paperless/media
- name: ssh
mountPath: /ssh
readOnly: true
containers:
- name: paperless
image: ghcr.io/paperless-ngx/paperless-ngx:2.20.14
@@ -123,6 +241,10 @@ spec:
- name: consume
persistentVolumeClaim:
claimName: paperless-consume
- name: ssh
secret:
secretName: paperless-seed-ssh
defaultMode: 0400
---
apiVersion: v1
kind: Service
kubernetes/apps/paperless/seed-rbac.yaml
+35
View File
@@ -0,0 +1,35 @@
# RBAC for the seed initContainer in the paperless Pod (paperless.yaml).
# Lets the Pod's ServiceAccount query CNPG Backup CRs (idempotency gate)
# and create them post-seed.
apiVersion: v1
kind: ServiceAccount
metadata:
name: paperless-seed
namespace: paperless
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: paperless-seed
namespace: paperless
rules:
- apiGroups: [postgresql.cnpg.io]
resources: [backups]
verbs: [get, list, watch, create]
- apiGroups: [postgresql.cnpg.io]
resources: [clusters]
verbs: [get, list, watch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: paperless-seed
namespace: paperless
subjects:
- kind: ServiceAccount
name: paperless-seed
namespace: paperless
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: paperless-seed
kubernetes/migrations/paperless/seed-ssh.sops.yaml → kubernetes/apps/paperless/seed-ssh.sops.yaml
View File
kubernetes/clusters/homelab/paperless-seed.yaml
-41
View File
@@ -1,41 +0,0 @@
# Flux Kustomization for the paperless seed Job. Suspended until the
# operator commits the SOPS-encrypted SSH secret — see
# kubernetes/migrations/paperless/kustomization.yaml for the recipe.
#
# To enable auto-trigger:
# 1. Commit kubernetes/migrations/paperless/seed-ssh.sops.yaml and
# uncomment its entry in that directory's kustomization.yaml.
# 2. Set spec.suspend: false below.
# 3. Add `- name: paperless-seed` to dependsOn in
# kubernetes/clusters/homelab/paperless.yaml so the app waits for
# the seed Job to Complete before reconciling.
#
# After enable: a fresh paperless namespace (post-rebuild) will trigger
# Flux to apply the Job; the Job's idempotency gate
# (kubectl get backups …) no-ops if a completed CNPG Backup already
# exists, and seeds otherwise.
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: paperless-seed
namespace: flux-system
spec:
interval: 10m
retryInterval: 1m
# The seed Job's worst case is a multi-hour rsync of paperless data +
# media plus pg_restore. Cap generously; the Job has its own internal
# wait deadlines.
timeout: 5h
sourceRef:
kind: GitRepository
name: flux-system
path: ./kubernetes/migrations/paperless
prune: true
wait: true
decryption:
provider: sops
secretRef:
name: sops-age
dependsOn:
- name: paperless-db
- name: velero-bootstrap-restore
kubernetes/clusters/homelab/paperless.yaml
-1
View File
@@ -52,5 +52,4 @@ spec:
name: sops-age
dependsOn:
- name: paperless-db
- name: paperless-seed
- name: velero-bootstrap-restore
kubernetes/migrations/paperless/kustomization.yaml
-36
View File
@@ -1,36 +0,0 @@
# Flux applies this kustomization via kubernetes/clusters/homelab/paperless-seed.yaml.
#
# To enable Flux auto-trigger of the seed Job (so on a fresh cluster
# rebuild the seed runs without operator intervention), do this once:
#
# 1. Build a Secret manifest with the legacy-VM SSH key:
#
# cat <<EOF >/tmp/seed-ssh.yaml
# apiVersion: v1
# kind: Secret
# metadata:
# name: paperless-seed-ssh
# namespace: paperless
# type: Opaque
# data:
# id_ed25519: $(base64 -w0 < ~/.ssh/your-legacy-key)
# EOF
#
# 2. Encrypt with the repo's age recipient and commit:
#
# sops -e /tmp/seed-ssh.yaml > kubernetes/migrations/paperless/seed-ssh.sops.yaml
# rm /tmp/seed-ssh.yaml
#
# 3. Uncomment the resource entry below.
# 4. Set spec.suspend: false in kubernetes/clusters/homelab/paperless-seed.yaml
# and add `- name: paperless-seed` to dependsOn in
# kubernetes/clusters/homelab/paperless.yaml.
#
# Until that's done, the migrate-paperless pipeline still works — it
# creates paperless-seed-ssh on-the-fly from the Woodpecker secret and
# applies seed-job.yaml directly (bypasses Flux).
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- seed-job.yaml
- seed-ssh.sops.yaml
kubernetes/migrations/paperless/seed-job.yaml
-198
View File
@@ -1,198 +0,0 @@
# Self-contained atomic seed for Paperless: idempotency-gated rsync of
# data + media PVCs from the legacy VM, pg_restore into the live pprl-pg
# CNPG cluster, then a CNPG Backup CR so future cold-start
# bootstrap.recovery has a base backup.
#
# Idempotency: the Job no-ops if any completed CNPG Backup exists for
# pprl-pg. So:
# - First run (empty bucket): seeds + creates Backup.
# - Re-runs after success: kubectl-applying the same manifest is a
# no-op because Jobs are immutable; the existing Complete Job stays.
# - After namespace wipe: Job resource is gone, manifest re-apply
# creates a fresh Job, and the kubectl-get-backups check sees no
# completed Backup → seeds again.
#
# Atomicity: the DB pg_restore and the Backup CR are the LAST operations.
# If rsync fails, pg_restore is never reached, and no Backup is created
# → next reconcile sees "no completed Backup" and the Job re-runs.
#
# Required Secret in the paperless namespace (created on-the-fly by the
# pipeline today; SOPS-committable as a follow-up to enable Flux
# auto-trigger):
# - paperless-seed-ssh (id_ed25519 for legacy VM access)
# CNPG-managed (always present once the cluster exists):
# - pprl-pg-app (username + password for the database owner)
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: paperless-seed
namespace: paperless
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: paperless-seed
namespace: paperless
rules:
- apiGroups: [postgresql.cnpg.io]
resources: [backups]
verbs: [get, list, watch, create]
- apiGroups: [postgresql.cnpg.io]
resources: [clusters]
verbs: [get, list, watch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: paperless-seed
namespace: paperless
subjects:
- kind: ServiceAccount
name: paperless-seed
namespace: paperless
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: paperless-seed
---
apiVersion: batch/v1
kind: Job
metadata:
name: paperless-seed
namespace: paperless
spec:
backoffLimit: 0
ttlSecondsAfterFinished: 600
template:
spec:
restartPolicy: Never
serviceAccountName: paperless-seed
containers:
- name: seed
image: gitea.renner.uno/gitea_admin/ci-k8s:2026-04-29
env:
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: CLUSTER
value: pprl-pg
- name: PGHOST
value: pprl-pg-rw.paperless.svc
- name: PGPORT
value: "5432"
- name: PGDATABASE
value: paperless
- name: PGUSER
valueFrom:
secretKeyRef:
name: pprl-pg-app
key: username
- name: PGPASSWORD
valueFrom:
secretKeyRef:
name: pprl-pg-app
key: password
- name: LEGACY_HOST
value: paperless.srv.renner.uno
command:
- bash
- -c
- |
set -euo pipefail
log() { printf '[%s] %s\n' "$(date -u +%H:%M:%S)" "$*"; }
# ci-k8s already has kubectl/sops/openssh/curl; we only
# add rsync + postgresql-client for the seed work.
log "Installing rsync + postgresql-client ..."
apk add --no-cache rsync postgresql16-client >/dev/null
# ---- Idempotency: skip if any completed Backup exists ----
# The user's gate: "trigger if no backup exists". Once the
# cluster has a completed Backup, future runs no-op so a
# Flux re-apply doesn't clobber live data.
EXISTING=$(kubectl get backups.postgresql.cnpg.io -n "$NAMESPACE" \
-l "cnpg.io/cluster=$CLUSTER" \
-o jsonpath='{range .items[?(@.status.phase=="completed")]}{.metadata.name}{"\n"}{end}' \
2>/dev/null | grep -c . || true)
if [ "$EXISTING" -gt 0 ]; then
log "$EXISTING completed CNPG Backup(s) for $CLUSTER — skipping seed"
exit 0
fi
log "No completed CNPG Backup for $CLUSTER — proceeding with seed"
# ---- SSH ----
mkdir -p /root/.ssh
install -m 600 /ssh/id_ed25519 /root/.ssh/id_ed25519
printf 'Host *\n\tStrictHostKeyChecking no\n\tUserKnownHostsFile /dev/null\n' > /root/.ssh/config
LEGACY="root@$LEGACY_HOST"
# ---- 1. PVC rsync ----
log "rsync paperless_data → /dst/data ..."
rsync -aH --delete --info=progress2 \
"$LEGACY:/var/lib/docker/volumes/paperless_data/_data/" /dst/data/
log "rsync paperless_media → /dst/media ..."
rsync -aH --delete --info=progress2 \
"$LEGACY:/var/lib/docker/volumes/paperless_media/_data/" /dst/media/
log "Normalizing PVC ownership to 1000:1000 ..."
chown -R 1000:1000 /dst/data /dst/media
du -sh /dst/data /dst/media
# ---- 2. Legacy DB dump (run on legacy, rsync the file) ----
log "Locating paperless DB container on $LEGACY_HOST ..."
DB_CT=$(ssh "$LEGACY" "docker ps --filter label=com.docker.compose.service=db --format '{{.Names}}' | head -1")
[ -n "$DB_CT" ] || { log "no DB container found on $LEGACY_HOST"; exit 1; }
LEGACY_DUMP=/tmp/$PGDATABASE-seed-dump
log "Running pg_dump on $LEGACY_HOST → $LEGACY_DUMP ..."
ssh "$LEGACY" "docker exec -i $DB_CT pg_dump --clean --if-exists --no-owner --no-privileges --format=custom -U $PGUSER $PGDATABASE > $LEGACY_DUMP"
log "rsync dump file from legacy ..."
rsync -aH --info=progress2 "$LEGACY:$LEGACY_DUMP" /tmp/legacy.dump
ssh "$LEGACY" "rm -f $LEGACY_DUMP" || true
test -s /tmp/legacy.dump || { log "dump empty — refusing to restore"; exit 1; }
log "Dump size: $(du -h /tmp/legacy.dump | cut -f1)"
# ---- 3. pg_restore into live cluster ----
log "pg_restore → $PGHOST/$PGDATABASE as $PGUSER ..."
pg_restore --clean --if-exists --no-owner --no-privileges \
--role="$PGUSER" \
--dbname="$PGDATABASE" \
/tmp/legacy.dump \
|| log "pg_restore completed with warnings (expected)"
psql -c "REASSIGN OWNED BY postgres TO $PGUSER;" || true
psql -c "GRANT ALL ON SCHEMA public TO $PGUSER;" || true
# ---- 4. Trigger CNPG Backup (the on-disk commit point) ----
BACKUP_NAME="$CLUSTER-seed-$(date -u +%Y%m%d-%H%M%S)"
log "Creating CNPG Backup $BACKUP_NAME ..."
cat <<YAML | kubectl apply -f -
apiVersion: postgresql.cnpg.io/v1
kind: Backup
metadata:
name: $BACKUP_NAME
namespace: $NAMESPACE
spec:
cluster:
name: $CLUSTER
method: plugin
pluginConfiguration:
name: barman-cloud.cloudnative-pg.io
YAML
log "Backup $BACKUP_NAME submitted; CNPG operator runs it asynchronously."
log "Seed complete: data is in the live cluster, app can start."
volumeMounts:
- { name: data, mountPath: /dst/data }
- { name: media, mountPath: /dst/media }
- { name: ssh, mountPath: /ssh, readOnly: true }
volumes:
- name: data
persistentVolumeClaim:
claimName: paperless-data
- name: media
persistentVolumeClaim:
claimName: paperless-media
- name: ssh
secret:
secretName: paperless-seed-ssh
defaultMode: 0400