fix(llama-embed): raise the memory limit to match its own context and batch

#739 raised the context from 2048 to 4096 and the micro-batch from an effective
512 to 1024, both to match how Paperless chunks documents. It left memory at
2Gi/4Gi, which was sized for the smaller values.

The pod ran fine interactively and was OOMKilled (exit 137) partway through the
first sustained run, around 300 documents into an embedding pass. Nothing in the
server log marked it — the process was killed outright, so the last line is a
half-finished task and the reason is only visible in the container's
lastState.terminated.

This would have surfaced when Paperless first indexed the archive, i.e. on the
first real use rather than under test.

The weights are only ~600 MiB. What sets the footprint is four slots at 4096
context with a 1024-token micro-batch, plus Vulkan allocating the iGPU working
set from system RAM where it counts against this cgroup — the same effect that
put llama-server at 32Gi for 19 GiB of weights.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kubernetes/RUNBOOK-baremetal-conversion.md

@@ -2,12 +2,7 @@
 
 Converts one MS-01 from "Proxmox host running a Talos VM" to "Talos on bare
 metal", and swaps its Ceph OSD onto a Samsung PM983 at the same time. Written
-after doing jupiter on 2026-08-25.
-
-All three nodes are now on bare metal — `talosctl get systeminformation`
-reports Minisforum MS-01 hardware with distinct chassis serials for k8s-1,
-k8s-2 and k8s-3, with no hypervisor in between. Kept as the procedure for a
-rebuild, not as outstanding work.
+after doing jupiter on 2026-08-25; saturn and neptune remain.
 
 Everything here that reads like an over-specific warning is something that
 actually happened. The conversion itself is straightforward — the failures were

kubernetes/apps/home-assistant/MIGRATION-recorder-postgres.md

@@ -111,23 +111,9 @@ statistics             max_id=13323904    max_start_ts=1787162400.0
 statistics_short_term  max_id=159205762   max_start_ts=1787166000.0
 ```
 
-## Phase 2c — cutover: DONE
+## Phase 2c — cutover (REMAINING)
 
-Verified 2026-09-02 against the live database:
-
-| table | rows | span |
-|---|---|---|
-| `statistics` | 11,092,586 | 2024-03-13 20:00 → current |
-| `statistics_short_term` | 1,883,709 | rolling ~7 days |
-| `statistics_meta` | 1,480 | |
-
-The 2024-03-13 origin is the proof that matters. It is unchanged from the
-pre-migration count in "What is at risk" above, so the history was copied
-across engines rather than started from zero — which is the failure this
-document exists to prevent. `states` carries current rows, so `db_url` points
-at Postgres.
-
-The procedure below is kept as the record of what was done.
+Do not start until phase 1 is verified.
 
 **Migrate statistics only.** Copying all 18M `states` rows is slow, needs a long
 HA outage, and buys a table that is discarded within 7 days anyway.

kubernetes/apps/home-assistant/deployment.yaml

@@ -56,9 +56,10 @@ spec:
         # Re-rendered on every pod start, so a CNPG password rotation takes
         # effect rather than silently leaving a stale URI behind.
         #
-        # Load-bearing: `db_url: !secret recorder_db_url` in the homeassistant
-        # repo's configuration.yaml reads this key, so HA does not start
-        # without it. Removing this init container takes the recorder down.
+        # INERT until the recorder is repointed: nothing reads this key until
+        # `db_url: !secret recorder_db_url` lands in configuration.yaml in the
+        # homeassistant repo. That ordering is required — HA refuses to start
+        # when a referenced secret is missing.
         - name: render-db-url
           image: busybox:1.36
           command:

kubernetes/apps/home-assistant/ha-pg-cluster.yaml

@@ -18,12 +18,12 @@
 # That is the point of the move — it attacks the multiplier rather than trimming
 # what gets recorded, which would only hide the problem.
 #
-# This is the live recorder database. HA writes here; the SQLite file is no
-# longer the system of record. The 2.4 years of long-term statistics were
-# copied across engines rather than regenerated — HA does not migrate data
-# between database engines, so an empty Postgres would have silently started
-# from zero and taken the energy dashboard's history with it.
-# MIGRATION-recorder-postgres.md holds the sequence that was followed.
+# STAGING
+# Phase 1 (this manifest): the cluster comes up EMPTY and HA keeps using SQLite.
+# Nothing depends on it, so a failure here is inert. Verify Ready, WAL archiving
+# to Hetzner, and a successful base backup before going further.
+# Phase 2: migrate the statistics tables and repoint recorder.db_url.
+# See MIGRATION-recorder-postgres.md for the sequence and the rollback.
 apiVersion: postgresql.cnpg.io/v1
 kind: Cluster
 metadata:

kubernetes/apps/home-assistant/kustomization.yaml

@@ -60,8 +60,9 @@ resources:
   - zigbee2mqtt-gate-service.yaml
   - zigbee2mqtt-ingress.yaml
   # Postgres backend for the recorder (MIGRATION-recorder-postgres.md).
-  # Live: recorder.db_url in the homeassistant repo points here, and the
-  # long-term statistics carry their full history from 2024.
+  # PHASE 1: this cluster comes up EMPTY and nothing consumes it — HA is still
+  # on SQLite. Deploying it is therefore inert; phase 2 migrates the statistics
+  # tables and repoints recorder.db_url in the homeassistant repo.
   - ha-pg-hetzner-creds-externalsecret.yaml
   - ha-pg-objectstore.yaml
   - ha-pg-cluster.yaml

kubernetes/apps/llama-server/embed.yaml

@@ -91,14 +91,24 @@ spec:
             capabilities:
               drop:
                 - ALL
+          # The weights are only ~600 MiB, but that is not what sets the
+          # footprint: four slots at 4096 context with a 1024-token micro-batch,
+          # and Vulkan allocating the iGPU's working set from system RAM where
+          # it counts against this cgroup.
+          #
+          # 4Gi was sized for the earlier 2048 context and 512 micro-batch. Both
+          # were raised to match Paperless's chunking without raising this, and
+          # the pod then survived interactive use and was OOMKilled (137) partway
+          # through the first sustained run — roughly 300 documents in. Indexing
+          # the archive would have hit it on day one.
           resources:
             requests:
               cpu: 250m
-              memory: 2Gi
+              memory: 4Gi
               gpu.intel.com/i915: "1"
             limits:
               cpu: "2"
-              memory: 4Gi
+              memory: 10Gi
               gpu.intel.com/i915: "1"
           # ~600 MiB rather than ~5 GiB, so a much shorter budget than the chat
           # model needs, but still sized for a cold download.