A VolumeSnapshot of influxdb-influxdb2 on ceph-block-snapclass, to be
merged and ready before homelab #1130 (2.7.4 -> 2.9.1) so the upgrade has
a restore point beyond `influxd downgrade`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
InfluxDB 2.9 hashes tokens on startup by default. The HA and Grafana
token Jobs read tokens back via `influx auth list`, which returns no
token once hashed, and a downgrade below 2.8 would erase all tokens.
Set INFLUXD_USE_HASHED_TOKENS=false so the 2.9.1 bump stays reversible
with `influxd downgrade 2.4`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Draining k8s-3 for the Talos 1.13.10 upgrade stopped on four pods
(wp-pg-1, ha-pg-1, gf-pg-1, gitea-0) whose PDBs allow zero disruptions,
and each needed a manual --disable-eviction.
CNPG: ha-pg, gf-pg, wp-pg and tr-pg go to instances: 2. With a replica,
CNPG switches the primary over ahead of a drain, the same way ak-pg,
im-pg and pprl-pg already do. With one instance it creates a primary PDB
and 'will prevent draining the node where the instance is housed'
(CNPG docs, kubernetes_upgrade). smartShutdownTimeout 10 / stopDelay 60
as on ak-pg, which bound switchover time and the barman-plugin demote
hang (cloudnative-pg#9770). Network policies already allow intra-cluster
replication, operator :8000 and WAL-restore egress in all four
namespaces; nothing references a specific instance name.
Single-replica RWO workloads (gitea, woodpecker-server,
protonmail-bridge, syncthing, influxdb): drop the minAvailable:1 PDBs.
They cannot fail over, and eviction already stops the pod gracefully
with SIGTERM and its grace period; the PDB only forced a manual
--disable-eviction on every drain. influxdb's comes from the chart
(pdb.create, influxdb2 2.1.2 defaults).
The 'single-instance cluster, the archive is the only off-node copy'
comments no longer hold and are removed (im-pg's was already stale).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The seed->apps ownership handoffs are complete, so the prune: disabled comments
on the folded CNPG DB objects no longer read 'ahead of the ... handoff' (future
tense). Normalize the 17 comments across authentik/immich/paperless/woodpecker/
influxdb to standing-guard/past tense. Comment-only; no manifest changes.
Closes the tense-normalization item flagged on the immich fold review (#405).
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses the follow-up review on #390: three more comments still described the
removed influxdb-seed Job, some in the present tense (contradicting the note the
prior commit added).
- helmrelease.yaml adminUser comment: drop the 'sibling influxdb-seed Job reads
the secret / deletes the bucket' description; state the admin token comes from
the influxdb-bootstrap Secret (OpenBao via ESO).
- helmrelease.yaml resource/timeout rationale: reword 'after a partial seed' and
'batches from the seed Job' to the seed-independent conditions (large dataset /
large write batches) that still justify the 6Gi limit + 60s write timeout.
- ha-token-job.yaml: drop 'Same wait pattern as the seed Job's wait-for-influxd';
describe the two-stage wait directly.
Verified: kubectl kustomize apps/influxdb builds; the only remaining 'seed'
strings are accurate migration-rationale comments and the OpenBao KV path.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Step 2 of the InfluxDB seed->apps handoff. #388 already landed
prune: disabled on the live HelmRelease/HelmRepository/ExternalSecret, so
retiring seed-influxdb here cannot trigger a helm uninstall — GC skips the
annotated objects and the new influxdb KS adopts them in place.
- git mv kubernetes/seed/influxdb/ -> kubernetes/apps/influxdb/ (annotations
from #388 travel with the files).
- Add kubernetes/clusters/homelab/influxdb.yaml (KS 'influxdb', path
apps/influxdb, deps infrastructure/crds/ceph). Delete seed-influxdb.yaml.
Review fixes from the closed atomic PR (#387):
- bootstrap.sh: wait on 'influxdb' instead of the deleted 'seed-influxdb'
(would otherwise spin the full 600s timeout on cold bootstrap).
- e2e-test.sh: swap the seed-influxdb readiness check for 'influxdb'.
- Fix stale comment paths kubernetes/seed/influxdb/... -> apps/influxdb/... in
network-policies/home-assistant-ns.yaml and apps/monitoring/
helmrelease-prometheus.yaml.
- Trim the helmrelease.yaml comment that described the removed seed-job.yaml,
and the ha-token-job.yaml 'at cutover moves to apps/influxdb' note (this is
that cutover).
Left intentionally: externalsecret.yaml key 'seed/influxdb/bootstrap' is an
OpenBao KV path, not a repo path — moving it needs a vault-side secret
migration, out of scope here.
Verified: kubectl kustomize apps/influxdb builds all objects with the right
prune/force annotations; bash -n passes on bootstrap.sh + e2e-test.sh; no
remaining seed-influxdb references in the repo (only the OpenBao KV path).
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The current rsync-into-data-PVC seed shape silently breaks the chart's
admin-token bootstrap. `data-seed-job.yaml` rsyncs `/var/lib/influxdb`
from VM 108 directly into the data PVC, including `influxd.sqlite`
(InfluxDB's onboarding state). When the chart starts on that PVC the
upstream image's entrypoint sees an already-onboarded instance and
short-circuits `influx setup`, so the DOCKER_INFLUXDB_INIT_* values
sourced from the influxdb-bootstrap Secret never apply. Result: the
live admin token is the legacy VM 108 token (unknown to the cluster's
manifests) and every API call using the bootstrap-secret token returns
401 — observed in pipeline 873 with three "token required" /
"authorization not found" log lines from clients hitting the API.
Restore the original separation between transport and import, mirroring
the CNPG pg-seed pattern (ak-pg-seed, im-pg-seed, pprl-pg-seed):
- Chart's PVC template provisions an empty `influxdb-influxdb2` PVC.
HelmRelease drops `persistence.useExisting: true`/`name:` so the
chart adopts nothing. influxd's entrypoint runs `influx setup`
against the empty data dir using the bootstrap-secret values, and
the bootstrap-secret token becomes the live admin token.
- Sibling `influxdb-seed` Job (this commit) waits for the API +
bootstrap admin token to be live (init container, two-stage check
mirroring pg-seed's `wait-for-cluster`), then SSHes to legacy and
runs `influx backup --bucket suedstr28` into a temp dir, rsyncs
the directory into a new staging PVC `influxdb-seed-staging`,
deletes the empty placeholder bucket the chart's `influx setup`
just created (since `influx restore` cannot restore into an
existing bucket), and calls `influx restore --bucket suedstr28`
against the chart's API. No `--full`, so the bootstrap-secret
admin token is left untouched.
CLI flag set verified against the v2.7 docs: `influx backup -b/--bucket
NAME PATH --host URL --token TOKEN` (env: INFLUX_HOST/INFLUX_TOKEN/
INFLUX_ORG); `influx restore PATH --bucket NAME` (cannot restore into
existing bucket — must `--new-bucket` rename or delete first).
File layout mirrors CNPG: HelmRelease + bootstrap-secret + seed-job +
staging-pvc + seed-ssh all live in `kubernetes/seed/influxdb/`. The
per-app KS at `kubernetes/clusters/homelab/influxdb.yaml` and the
`kubernetes/apps/influxdb/` directory are removed; the chart now
reconciles in the seed phase alongside the seed Job. At cutover this
whole directory moves to `kubernetes/apps/influxdb/` along with the
CNPG cluster moves, when the seed phase is decommissioned.
The `.woodpecker/migrate-influxdb.yml` manual-trigger pipeline still
references the deleted `kubernetes/migrations/influxdb/seed-job.yaml`
path AND assumes the old scale-STS-to-0+rsync-into-PVC shape; needs a
separate rewrite to match this design (chart must stay UP for `influx
restore` to reach the API). Not addressed in this commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pipeline 871 surfaced a PVC naming mismatch: the seed Job wrote to
`data-influxdb-influxdb2-0` while the chart's StatefulSet mounted a
fresh `influxdb-influxdb2` (the influxdb2 chart's
`persistent-volume-claim.yaml` template generates `<release>-influxdb2`,
not the `data-<sts>-N` pattern that volumeClaimTemplates would). Result:
32.81 GB of legacy data orphaned post-rebuild, pod ran on ~196 KB of
fresh bootstrap.
Fix: set `persistence.useExisting: true` and
`persistence.name: data-influxdb-influxdb2-0` in the HelmRelease, which
tells the chart to skip its PVC template and consume the
seed-populated PVC by name from the StatefulSet's
`volumes[].persistentVolumeClaim.claimName`. Future rebuilds will
therefore seed the correct PVC and the pod will see the data
immediately.
Comments on both `helmrelease.yaml` and `seed/influxdb/pvc.yaml` were
also wrong about the chart using volumeClaimTemplates; updated to
reflect the actual chart mechanics.
See .rebuild-report-2026-05-02/influxdb-data-relocation.md for the
one-shot procedure to relocate the currently-orphaned data on the
live cluster.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The seed KS owns influxdb-seed-ssh Secret in the influxdb namespace, but
the namespace was previously created by apps/influxdb/namespace.yaml —
which dependsOn seed → seed never reconciles → circular.
Hoist the namespace into infrastructure/namespaces/namespaces.yaml
(consistent with all the other app namespaces) and drop it from the
apps/influxdb Kustomization.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the per-app race-y CNPG bootstrap.recovery + velero-bootstrap-restore
pattern with a single seed Flux KS that populates databases and PVCs from
legacy hosts before any app KS reconciles.
Changes:
- New kubernetes/seed/<app>/ tree owning, per app:
- CNPG Cluster CRs with bootstrap.initdb (was: bootstrap.recovery) +
enableSuperuserAccess so seed Jobs can connect as postgres
- DB seed Jobs: ssh legacy 'pg_dump -Fc' | pg_restore -d <db> against
the running cluster (was: ephemeral PG + barman-cloud-backup → S3)
- PVC seed Jobs: rsync/ssh+tar from legacy → app PVCs as standalone
Jobs (was: in-pod init containers, blocking the app Deployment's
progressDeadlineSeconds for hours on a 250 GiB rsync)
- All ObjectStores, ScheduledBackups, barman-creds, PVCs, seed-ssh
Secrets relocated here
- New top-level kubernetes/clusters/homelab/seed.yaml Flux KS:
dependsOn infrastructure-cnpg-barman, timeout 4h, healthCheckExprs
cover CNPG Cluster phase + native Job succeeded.
- Stripped from kubernetes/apps/<app>/: Cluster CRs, seed Jobs, init
container rsync blocks, ssh volumes, seed ServiceAccount references.
~454 lines deleted, ~33 added.
- App KSs (immich, authentik, paperless, audiobookshelf,
music-assistant, influxdb): dependsOn velero-bootstrap-restore →
dependsOn seed.
- bootstrap.sh wait-apps: new wait_kustomization seed (4h) before
per-app waits; per-app timeouts dropped to 60s since they're
near-instant once seed is Ready.
- Deleted kubernetes/clusters/homelab/velero-bootstrap-restore.yaml;
manifests stay in kubernetes/infrastructure/velero-bootstrap-restore/
for post-cutover re-add (it remains the cold-rebuild path once
legacy hosts are decommissioned and seed phase is removed).
Eliminates the documented race failure modes:
- cnpg-bootstrap-recovery-no-initdb-fallback (recovery Job pounds empty
bucket until seed lands → multiple Error pods)
- velero-bootstrap-restore-bsl-sync-race (Job iterates schedules in an
empty bucket for ~15 min)
- immich-server progressDeadlineSeconds blowup (250 GiB rsync init
exceeds the 600s default and Flux marks the Deployment Failed)
Seed is migration-only: removed entirely once all services have cut
over from legacy. At that point the Cluster + ObjectStore +
ScheduledBackup + PVC manifests move back into kubernetes/apps/<app>/
and velero-bootstrap-restore.yaml is re-added at the cluster level.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Verified against the upstream chart values.yaml / templates:
- authentik chart exposes server.serviceAccountName (string, references
an existing SA), not the bitnami-style top-level serviceAccount.{create,
name}. server.initContainers and server.volumes were already correct.
- influxdb2 chart has NO .Values.initContainers hook in its StatefulSet
template. Add the seed initContainer via spec.postRenderers (Kustomize
patch on the rendered StatefulSet "influxdb-influxdb2"). The SSH
Secret volume goes through the chart's supported top-level `volumes:`
value (not `extraVolumes`).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replicates the paperless pattern (49a1e75) for immich, audiobookshelf,
music-assistant, authentik, and influxdb:
- Seed work moves into an initContainer in each app's Pod template
(Deployment for immich/audiobookshelf/music-assistant; HelmRelease
values for authentik server + influxdb StatefulSet).
- Same Pod owns the PVC → no RWO/RWX contention regardless of access
mode. Main container only starts after seed completes (initContainer
ordering enforces "app starts after seed").
- Idempotency gate: completed CNPG Backup (DB apps) or completed
Velero Backup (PVC apps); skip-fast on warm starts.
- SOPS-encrypted SSH secrets and seed-RBAC moved into kubernetes/apps/
<app>/. PVC-only apps' RBAC is cross-namespace (Role+Binding in
velero ns; SA in app ns).
- Drops kubernetes/migrations/ entirely and the five
kubernetes/clusters/homelab/<app>-seed.yaml Flux Kustomizations.
- Removes <app>-seed from each app Kustomization's dependsOn.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Partial seeding produced enough shards that opening them on startup
pushes memory past the 2Gi limit → OOMKilled. Meanwhile the chart's
default zero-delay liveness probe kills the pod before shard open
finishes, causing a CrashLoopBackOff.
- resources.limits.memory: 2Gi → 6Gi
- livenessProbe: initialDelaySeconds 600, failureThreshold 6
- readinessProbe: initialDelaySeconds 30
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Root cause: `influx write` batches of 5000 points routinely blew past
InfluxDB's default 10s storage-write timeout against the cold-cache
RWO Ceph-block PVC under concurrent TSM compaction. Server responded
500 → client aborted → `aws s3 cp` got broken pipe.
Two-sided fix:
- Seed Job: --batch-size 1000 so each HTTP write finishes well
inside any plausible write deadline.
- InfluxDB server: INFLUXD_STORAGE_WRITE_TIMEOUT=60s to tolerate
transient ingest+compaction contention.
Bump Job name v2 → v3 to force a clean re-roll; pod template is
immutable.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
influxdata/influxdb2 HelmRelease in its own namespace. 50Gi ceph-block
PVC, single replica (OSS). Admin password + token come from a SOPS-
managed cluster-internal secret so credentials are deterministic across
cluster rebuilds; the seed Job reuses the same token for `influx write`.
Clusters/homelab wires two Flux Kustomizations: influxdb (the chart) and
influxdb-seed (depends on influxdb being healthy so the seed only runs
once the API is reachable).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
