chore(llama-server): grow the model cache to 120Gi for candidate evaluation

The embedding model was picked from a recommendation and never compared. MTEB's German retrieval results put the deployed Qwen3-Embedding-0.6B well down the field, so a dozen candidates are being measured against the real archive with scripts/embed-eval-real.sh — and they will not fit in 40Gi alongside the two deployed models. Growing the shared cache rather than using a scratch volume is deliberate: on a slow connection the download is the long pole, and a shared cache lets the large candidates pull in the background while the small ones are already being measured. It also means nothing is fetched twice across runs, and a candidate that wins is already local when it is promoted. 120Gi against 309 GiB currently free in cephfs-data0. CephFS has allowVolumeExpansion, so this grows online without disturbing the running pods. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Merge pull request 'refactor(immich): finish the VectorChord move, restore the replica' (#747 ) from immich-drop-pgvectors into master
2026-09-02 11:10:04 +02:00 · 2026-09-02 07:44:49 +00:00 · 2026-09-02 09:41:48 +02:00 · 2026-09-02 06:48:52 +00:00
2 changed files with 28 additions and 16 deletions
@@ -10,15 +10,15 @@ metadata:
    # Kept permanently as a standing guard.
    kustomize.toolkit.fluxcd.io/prune: disabled
 spec:
-  # Single instance during pgvecto.rs → VectorChord migration window;
-  # pg_basebackup tar format chokes on the long pg_vectors/indexes/<uuid>
-  # paths, blocking replica join. Bump back to 2 once Immich's in-app
-  # vchord migration drops the vectors extension.
-  instances: 1
-  # See ak-pg cluster.yaml for rationale. Single-instance, so no failover, but
-  # the barman-plugin shutdown wedge (cloudnative-pg#9770) takes the whole DB
-  # down until SIGKILL — bounding terminationGracePeriod (stopDelay) matters
-  # even without a standby. smartShutdownTimeout keeps pod restarts quick.
+  # A replica is viable because nothing here writes pg_vectors any more. Both
+  # vector indexes run on vchordrq and the vectors extension is gone, so the
+  # long pg_vectors/indexes/<uuid> paths are absent — pg_basebackup's tar
+  # format cannot represent them, and that is what blocks a replica join.
+  instances: 2
+  # See ak-pg cluster.yaml for rationale. The barman-plugin shutdown wedge
+  # (cloudnative-pg#9770) holds an instance up until SIGKILL, so bounding
+  # terminationGracePeriod (stopDelay) is what keeps a failover from waiting
+  # on it. smartShutdownTimeout keeps pod restarts quick.
  smartShutdownTimeout: 10
  stopDelay: 60
  imageName: ghcr.io/corentingiraud/cnpg-pgvector-vectorchord:16-migration
@@ -36,8 +36,10 @@ spec:
      database: immich
      owner: immich
  postgresql:
+    # vchord only. The image also ships pgvecto.rs (vectors.so), but its
+    # extension is not installed and preloading it would recreate pg_vectors
+    # in PGDATA for nothing. pgvector needs no preloading.
    shared_preload_libraries:
-      - vectors.so
      - vchord.so
    parameters:
      shared_buffers: "512MB"
@@ -1,10 +1,20 @@
 ---
-# Holds the GGUF weights, downloaded once by the Deployment's initContainer.
-# Prune-disabled so retiring this Kustomization does not throw away several
-# GiB of model files that would have to be pulled from Hugging Face again.
+# Holds the GGUF weights, downloaded once by llama-server itself via --hf-repo.
+# Prune-disabled so retiring this Kustomization does not throw away tens of GiB
+# of model files that would have to be pulled from Hugging Face again.
 #
-# ReadWriteMany so a second replica or a one-off benchmark Job can mount the
-# same weights; the models are read-only at inference time.
+# ReadWriteMany so a second replica, a one-off benchmark Job, or a candidate
+# model being evaluated can mount the same weights; the models are read-only at
+# inference time.
+#
+# Sized well beyond the two deployed models. The cache is shared by every
+# candidate under evaluation (scripts/embed-eval-real.sh), and on a slow
+# connection the download is the long pole — a shared cache lets the large
+# candidates pull in the background while the small ones are already being
+# measured, and means a model is never fetched twice across runs.
+#
+# CephFS supports online expansion (allowVolumeExpansion on the storage class),
+# so growing this does not disturb the running pods.
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
@@ -18,4 +28,4 @@ spec:
  storageClassName: ceph-filesystem
  resources:
    requests:
-      storage: 40Gi
+      storage: 120Gi
Author	SHA1	Message	Date
claude-botandClaude Opus 5	e99e502da7	chore(llama-server): grow the model cache to 120Gi for candidate evaluation ci/woodpecker/pr/ci-scripts-test Pipeline was successful Details ci/woodpecker/pr/ci-manifests-test Pipeline was successful Details ci/woodpecker/pr/pr-review Pipeline was successful Details The embedding model was picked from a recommendation and never compared. MTEB's German retrieval results put the deployed Qwen3-Embedding-0.6B well down the field, so a dozen candidates are being measured against the real archive with scripts/embed-eval-real.sh — and they will not fit in 40Gi alongside the two deployed models. Growing the shared cache rather than using a scratch volume is deliberate: on a slow connection the download is the long pole, and a shared cache lets the large candidates pull in the background while the small ones are already being measured. It also means nothing is fetched twice across runs, and a candidate that wins is already local when it is promoted. 120Gi against 309 GiB currently free in cephfs-data0. CephFS has allowVolumeExpansion, so this grows online without disturbing the running pods. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 11:10:04 +02:00
k.renner	bf2b520fc4	Merge pull request 'refactor(immich): finish the VectorChord move, restore the replica' (#747 ) from immich-drop-pgvectors into master ci/woodpecker/cron/issue-fix Pipeline was successful Details ci/woodpecker/cron/alert-monitor Pipeline was successful Details Reviewed-on: k.renner/homelab#747	2026-09-02 07:44:49 +00:00
claude-botandClaude Opus 5	780401fe91	refactor(immich): finish the VectorChord move, restore the replica ci/woodpecker/pr/ci-scripts-test Pipeline was successful Details ci/woodpecker/pr/ci-manifests-test Pipeline was successful Details ci/woodpecker/pr/pr-review Pipeline was successful Details Immich's in-app migration to VectorChord is complete: both vector indexes, face_index and clip_index, run on the vchordrq access method, and nothing uses pgvecto.rs. The manifest still described a migration window. `DROP EXTENSION vectors` was run against the immich database before this. It was safe to do without CASCADE — nothing outside the extension depended on it, which a rolled-back `BEGIN; DROP EXTENSION vectors; ROLLBACK;` confirmed first. The embedding columns are `public.vector(512)`, which belongs to pgvector, not to pgvecto.rs's type in the `vectors` schema. Afterwards both vchordrq indexes are intact at 96 MB and 79 MB and the cluster stays healthy. With the extension gone, vectors.so has nothing to serve and preloading it only recreates pg_vectors in PGDATA. pgvector needs no preloading, so vchord.so is the whole list. instances goes back to 2. The blocker was never the extension as such — it was that pg_basebackup's tar format cannot represent the long pg_vectors/indexes/<uuid> paths pgvecto.rs creates, so a replica could not join. That directory is now empty, 20 KiB in total, so the constraint is gone. The image keeps its `:16-migration` tag. That is the upstream build's own name for a multi-extension image, not a statement about this cluster, and it still provides the two extensions in use. Moving to a different image is a separate change with its own risk. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 09:41:48 +02:00
k.renner	f9a9dc7ff8	Merge pull request 'docs: correct migration records that outlived their migrations' (#746 ) from fix-stale-migration-docs into master ci/woodpecker/cron/alert-monitor Pipeline was successful Details ci/woodpecker/cron/issue-fix Pipeline was successful Details Reviewed-on: k.renner/homelab#746	2026-09-02 06:48:52 +00:00