docs: correct migration records that outlived their migrations

Both documents described work as outstanding that the cluster shows finished. A record that says "REMAINING" long after the fact is worse than no record: it is read as current state, and the next person plans around it. ## Home Assistant recorder Phase 2c is done, and done correctly. Against the live database: statistics 11,092,586 2024-03-13 20:00 -> current statistics_short_term 1,883,709 rolling ~7 days statistics_meta 1,480 The 2024-03-13 origin is the part that matters. It is unchanged from the 10,793,712-row count recorded on 2026-08-19, so the history was copied across engines rather than started from zero — the exact failure the document was written to prevent. `states` carries current rows, so db_url points at Postgres. The three "INERT"/"PHASE 1" comments went with it. All three described a cluster that nothing consumed; it is now the recorder's live database, and the render-db-url init container is load-bearing rather than inert — HA will not start without the secret it renders. ## Bare-metal conversion `talosctl get systeminformation` reports Minisforum MS-01 hardware with distinct chassis serials for all three nodes and no hypervisor, so saturn and neptune are not outstanding. Node creationTimestamp is not evidence either way: k8s-3 still carries its 2026-05-09 object and is on metal regardless. The runbook stays as the procedure for a rebuild. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-02 08:37:19 +02:00
8 changed files with 33 additions and 217 deletions
@@ -2,7 +2,12 @@

 Converts one MS-01 from "Proxmox host running a Talos VM" to "Talos on bare
 metal", and swaps its Ceph OSD onto a Samsung PM983 at the same time. Written
-after doing jupiter on 2026-08-25; saturn and neptune remain.
+after doing jupiter on 2026-08-25.
+
+All three nodes are now on bare metal — `talosctl get systeminformation`
+reports Minisforum MS-01 hardware with distinct chassis serials for k8s-1,
+k8s-2 and k8s-3, with no hypervisor in between. Kept as the procedure for a
+rebuild, not as outstanding work.

 Everything here that reads like an over-specific warning is something that
 actually happened. The conversion itself is straightforward — the failures were
@@ -111,9 +111,23 @@ statistics             max_id=13323904    max_start_ts=1787162400.0
 statistics_short_term  max_id=159205762   max_start_ts=1787166000.0
 ```

-## Phase 2c — cutover (REMAINING)
+## Phase 2c — cutover: DONE

-Do not start until phase 1 is verified.
+Verified 2026-09-02 against the live database:
+
+| table | rows | span |
+|---|---|---|
+| `statistics` | 11,092,586 | 2024-03-13 20:00 → current |
+| `statistics_short_term` | 1,883,709 | rolling ~7 days |
+| `statistics_meta` | 1,480 | |
+
+The 2024-03-13 origin is the proof that matters. It is unchanged from the
+pre-migration count in "What is at risk" above, so the history was copied
+across engines rather than started from zero — which is the failure this
+document exists to prevent. `states` carries current rows, so `db_url` points
+at Postgres.
+
+The procedure below is kept as the record of what was done.

 **Migrate statistics only.** Copying all 18M `states` rows is slow, needs a long
 HA outage, and buys a table that is discarded within 7 days anyway.
@@ -56,10 +56,9 @@ spec:
        # Re-rendered on every pod start, so a CNPG password rotation takes
        # effect rather than silently leaving a stale URI behind.
        #
-        # INERT until the recorder is repointed: nothing reads this key until
-        # `db_url: !secret recorder_db_url` lands in configuration.yaml in the
-        # homeassistant repo. That ordering is required — HA refuses to start
-        # when a referenced secret is missing.
+        # Load-bearing: `db_url: !secret recorder_db_url` in the homeassistant
+        # repo's configuration.yaml reads this key, so HA does not start
+        # without it. Removing this init container takes the recorder down.
        - name: render-db-url
          image: busybox:1.36
          command:
@@ -18,12 +18,12 @@
 # That is the point of the move — it attacks the multiplier rather than trimming
 # what gets recorded, which would only hide the problem.
 #
-# STAGING
-# Phase 1 (this manifest): the cluster comes up EMPTY and HA keeps using SQLite.
-# Nothing depends on it, so a failure here is inert. Verify Ready, WAL archiving
-# to Hetzner, and a successful base backup before going further.
-# Phase 2: migrate the statistics tables and repoint recorder.db_url.
-# See MIGRATION-recorder-postgres.md for the sequence and the rollback.
+# This is the live recorder database. HA writes here; the SQLite file is no
+# longer the system of record. The 2.4 years of long-term statistics were
+# copied across engines rather than regenerated — HA does not migrate data
+# between database engines, so an empty Postgres would have silently started
+# from zero and taken the energy dashboard's history with it.
+# MIGRATION-recorder-postgres.md holds the sequence that was followed.
 apiVersion: postgresql.cnpg.io/v1
 kind: Cluster
 metadata:
@@ -60,9 +60,8 @@ resources:
  - zigbee2mqtt-gate-service.yaml
  - zigbee2mqtt-ingress.yaml
  # Postgres backend for the recorder (MIGRATION-recorder-postgres.md).
-  # PHASE 1: this cluster comes up EMPTY and nothing consumes it — HA is still
-  # on SQLite. Deploying it is therefore inert; phase 2 migrates the statistics
-  # tables and repoints recorder.db_url in the homeassistant repo.
+  # Live: recorder.db_url in the homeassistant repo points here, and the
+  # long-term statistics carry their full history from 2024.
  - ha-pg-hetzner-creds-externalsecret.yaml
  - ha-pg-objectstore.yaml
  - ha-pg-cluster.yaml
@@ -1,177 +0,0 @@
-#!/usr/bin/env bash
-# Retrieval evaluation for an embedding model, on German documents of the kind
-# this homelab actually files.
-#
-# Companion to scripts/ai-eval.sh, but it cannot work the same way. A chat
-# answer can be read and judged; an embedding is 1024 opaque numbers. What is
-# measurable is whether the right document comes back first, so this builds a
-# small labelled corpus, embeds it, and scores real questions against it.
-#
-# Metrics:
-#   Recall@1  the correct document ranked first — what Paperless's RAG actually
-#             feeds to the classifier
-#   Recall@3  it at least made the shortlist
-#   MRR       mean reciprocal rank, so near-misses are visible rather than
-#             collapsing to pass/fail
-#   margin    cosine gap between the correct document and the best wrong one.
-#             A model that is right by 0.002 is guessing; one that is right by
-#             0.15 has actually separated them.
-#
-# The corpus deliberately contains confusable pairs — two Allianz policies, four
-# documents that are all "Rechnungen", a car insurance and a vehicle
-# registration sharing a number plate. An easy corpus would rate everything 100%
-# and tell you nothing.
-#
-# Usage:
-#   scripts/embed-eval.sh                                    # cluster default
-#   scripts/embed-eval.sh -m some-model
-#   scripts/embed-eval.sh -e https://host/v1/embeddings -m model -k KEY
-#
-# Env equivalents: EMBED_EVAL_ENDPOINT, EMBED_EVAL_MODEL, EMBED_EVAL_KEY.
-#
-# To compare two models on our llama-embed, -m is not enough: swap --hf-repo in
-# kubernetes/apps/llama-server/embed.yaml, let it roll out, then re-run. Note
-# that changing the model also invalidates any vectors Paperless has stored, so
-# comparisons are best made before an index exists.
-#
-# Exit code is the number of queries whose correct document did not rank first.
-set -uo pipefail
-
-ENDPOINT="${EMBED_EVAL_ENDPOINT:-http://llama-embed.ai.svc.cluster.local:8080/v1/embeddings}"
-MODEL="${EMBED_EVAL_MODEL:-qwen3-embedding-0.6b}"
-APIKEY="${EMBED_EVAL_KEY:-}"
-# Run curl from inside this pod when the endpoint is a cluster-internal name.
-# llama-embed has no ingress by design, so there is nothing to reach from a
-# workstation.
-VIA_POD="${EMBED_EVAL_VIA_POD:-}"
-
-while [ $# -gt 0 ]; do
-    case "$1" in
-        -e|--endpoint) ENDPOINT="$2"; shift 2 ;;
-        -m|--model)    MODEL="$2";    shift 2 ;;
-        -k|--key)      APIKEY="$2";   shift 2 ;;
-        -p|--via-pod)  VIA_POD="$2";  shift 2 ;;
-        -h|--help)     sed -n '2,34p' "$0"; exit 0 ;;
-        *) echo "unknown argument: $1 (try --help)" >&2; exit 2 ;;
-    esac
-done
-
-HERE="$(cd "$(dirname "$0")" && pwd)"
-CASES="$HERE/embed-eval"
-WORK="$(mktemp -d)"
-trap 'rm -rf "$WORK"' EXIT
-
-for f in "$CASES/corpus.jsonl" "$CASES/queries.jsonl"; do
-    [ -r "$f" ] || { echo "missing fixture: $f" >&2; exit 2; }
-done
-
-# Default to reaching the Service from a pod that is allowed to, unless the
-# caller pointed us somewhere externally reachable.
-if [ -z "$VIA_POD" ] && [ "${ENDPOINT#*svc.cluster.local}" != "$ENDPOINT" ]; then
-    VIA_POD="deploy/llama-server -n ai"
-fi
-
-# embed <text> -> the vector as a JSON array on stdout
-embed() {
-    local payload out
-    payload=$(jq -nc --arg m "$MODEL" --arg t "$1" '{model:$m,input:$t}')
-    if [ -n "$VIA_POD" ]; then
-        # stderr is kept and surfaced on failure rather than discarded: a
-        # silently swallowed kubectl or curl error is indistinguishable from a
-        # model that returned nothing.
-        # shellcheck disable=SC2086
-        out=$(kubectl exec $VIA_POD -- curl -sS -m 120 \
-              -H 'Content-Type: application/json' \
-              ${APIKEY:+-H "Authorization: Bearer $APIKEY"} \
-              -d "$payload" "$ENDPOINT" 2>"$WORK/err.txt")
-    else
-        out=$(curl -sS -m 120 -H 'Content-Type: application/json' \
-              ${APIKEY:+-H "Authorization: Bearer $APIKEY"} \
-              -d "$payload" "$ENDPOINT")
-    fi
-    printf '%s' "$out" | jq -c '.data[0].embedding // empty'
-}
-
-echo "endpoint: $ENDPOINT"
-echo "model:    $MODEL"
-[ -n "$VIA_POD" ] && echo "via pod:  $VIA_POD"
-echo ""
-
-# ---- embed the corpus ------------------------------------------------------
-DOCS=$(wc -l < "$CASES/corpus.jsonl" | tr -d ' ')
-echo "embedding $DOCS documents..."
-: > "$WORK/vecs.jsonl"
-start=$(date +%s)
-while IFS= read -r line; do
-    id=$(printf '%s' "$line" | jq -r '.id')
-    text=$(printf '%s' "$line" | jq -r '.text')
-    vec=$(embed "$text")
-    if [ -z "$vec" ]; then
-        echo "FAILED to embed '$id' — endpoint returned no vector." >&2
-        [ -s "$WORK/err.txt" ] && { echo "stderr:" >&2; cat "$WORK/err.txt" >&2; }
-        exit 2
-    fi
-    jq -nc --arg id "$id" --argjson v "$vec" '{id:$id, vec:$v}' >> "$WORK/vecs.jsonl"
-done < "$CASES/corpus.jsonl"
-elapsed=$(( $(date +%s) - start ))
-DIM=$(head -1 "$WORK/vecs.jsonl" | jq '.vec | length')
-echo "  $DOCS documents, ${DIM} dimensions, ${elapsed}s"
-echo ""
-
-# ---- score the queries -----------------------------------------------------
-# Cosine similarity in jq. The vectors llama.cpp returns are normalised, but the
-# full form costs nothing and keeps this correct against servers that do not.
-rank() { # queryvec -> "id score" lines, best first
-    jq -s -r --argjson q "$1" '
-        def dot($a; $b): reduce range(0; $a|length) as $i (0; . + ($a[$i] * $b[$i]));
-        def norm($a): (reduce ($a[]) as $x (0; . + ($x * $x))) | sqrt;
-        map({id: .id, s: (dot(.vec; $q) / ((norm(.vec) * norm($q)) + 1e-12))})
-        | sort_by(-.s)[]
-        | "\(.id) \(.s)"
-    ' "$WORK/vecs.jsonl"
-}
-
-MISS=0; TOTAL=0; HIT1=0; HIT3=0; MRRSUM=0
-printf '%-46s %-22s %5s %8s\n' QUERY EXPECTED RANK MARGIN
-printf '%s\n' "------------------------------------------------------------------------------------"
-while IFS= read -r line; do
-    q=$(printf '%s' "$line" | jq -r '.q')
-    want=$(printf '%s' "$line" | jq -r '.want')
-    qvec=$(embed "$q")
-    if [ -z "$qvec" ]; then
-        echo "FAILED to embed query: $q" >&2
-        [ -s "$WORK/err.txt" ] && { echo "stderr:" >&2; cat "$WORK/err.txt" >&2; }
-        exit 2
-    fi
-    rank "$qvec" > "$WORK/ranked.txt"
-
-    pos=$(grep -n "^$want " "$WORK/ranked.txt" | cut -d: -f1)
-    pos=${pos:-999}
-    TOTAL=$((TOTAL + 1))
-    [ "$pos" -eq 1 ] && HIT1=$((HIT1 + 1)) || MISS=$((MISS + 1))
-    [ "$pos" -le 3 ] && HIT3=$((HIT3 + 1))
-    MRRSUM=$(awk -v s="$MRRSUM" -v p="$pos" 'BEGIN{printf "%.6f", s + 1/p}')
-
-    # Margin: the correct document's score minus the best incorrect one. On a
-    # miss this is negative, which says how badly it lost.
-    want_s=$(awk -v w="$want" '$1==w{print $2; exit}' "$WORK/ranked.txt")
-    best_other=$(awk -v w="$want" '$1!=w{print $2; exit}' "$WORK/ranked.txt")
-    margin=$(awk -v a="${want_s:-0}" -v b="${best_other:-0}" 'BEGIN{printf "%+.4f", a-b}')
-
-    if [ "$pos" -eq 1 ]; then mark=" "; else mark="!"; fi
-    printf '%s%-45.45s %-22s %5s %8s\n' "$mark" "$q" "$want" "$pos" "$margin"
-    if [ "$pos" -ne 1 ]; then
-        printf '   got instead: %s\n' "$(head -1 "$WORK/ranked.txt" | cut -d' ' -f1)"
-    fi
-done < "$CASES/queries.jsonl"
-
-R1=$(awk -v h="$HIT1" -v t="$TOTAL" 'BEGIN{printf "%.0f", 100*h/t}')
-R3=$(awk -v h="$HIT3" -v t="$TOTAL" 'BEGIN{printf "%.0f", 100*h/t}')
-MRR=$(awk -v s="$MRRSUM" -v t="$TOTAL" 'BEGIN{printf "%.3f", s/t}')
-
-echo ""
-echo "========================================"
-printf '  RESULT model=%s dim=%s recall@1=%s%% recall@3=%s%% mrr=%s\n' \
-    "$MODEL" "$DIM" "$R1" "$R3" "$MRR"
-echo "========================================"
-exit "$MISS"
@@ -1,12 +0,0 @@
-{"id":"stadtwerke-strom","text":"Stadtwerke Muenchen GmbH. Jahresrechnung Strom. Rechnungsnummer 4711. Abrechnungszeitraum 01.01.2025 bis 31.12.2025. Zaehlernummer 1EMH0012345678. Verbrauch 2450 kWh zum Arbeitspreis von 31,2 Cent je kWh. Grundpreis 9,90 EUR monatlich. Zu zahlender Betrag 812,40 EUR faellig am 15.02.2026."}
-{"id":"telekom-mobilfunk","text":"Telekom Deutschland GmbH. Ihre Mobilfunkrechnung fuer Januar. Kundennummer 0891234567. Rufnummer 0170 1234567. Tarif MagentaMobil M mit 20 GB Datenvolumen. Grundpreis 39,95 EUR. Verbindungen ins Ausland 4,20 EUR. Rechnungsbetrag 44,15 EUR wird per Lastschrift eingezogen."}
-{"id":"finanzamt-steuer","text":"Finanzamt Muenchen Abteilung II. Bescheid ueber Einkommensteuer und Solidaritaetszuschlag fuer 2025. Steuernummer 143/256/78901. Zu versteuerndes Einkommen 54.320 EUR. Festgesetzte Einkommensteuer 8.412 EUR. Bereits geleistete Vorauszahlungen 9.100 EUR. Erstattungsbetrag 688 EUR. Rechtsbehelfsbelehrung siehe Seite 3."}
-{"id":"allianz-hausrat","text":"Allianz Versicherungs-AG. Versicherungsschein Hausratversicherung. Policennummer HR-88231. Versicherte Wohnflaeche 96 Quadratmeter. Versicherungssumme 65.000 EUR. Eingeschlossen sind Feuer, Leitungswasser, Sturm und Einbruchdiebstahl. Fahrraddiebstahl bis 1.500 EUR mitversichert. Jahresbeitrag 214 EUR."}
-{"id":"allianz-kfz","text":"Allianz Versicherungs-AG. Beitragsrechnung Kraftfahrtversicherung. Policennummer KFZ-55190. Amtliches Kennzeichen M-AB 1234. Fahrzeug VW Golf VIII. Vollkasko mit 500 EUR Selbstbeteiligung, Teilkasko 150 EUR. Schadenfreiheitsklasse SF 12. Jahresbeitrag 638 EUR."}
-{"id":"mietvertrag","text":"Mietvertrag ueber Wohnraum. Vermieter Immobilien Wagner GbR, Mieter Konstantin Renner. Mietobjekt Dreizimmerwohnung im zweiten Obergeschoss, 78 Quadratmeter, Lindenstrasse 14. Mietbeginn 01.04.2024. Kaltmiete 1.180 EUR monatlich, Betriebskostenvorauszahlung 240 EUR. Kaution drei Nettokaltmieten."}
-{"id":"arbeitsvertrag","text":"Arbeitsvertrag zwischen der Muster Technik GmbH und dem Arbeitnehmer. Taetigkeit als Softwareentwickler ab 01.09.2024. Bruttojahresgehalt 78.000 EUR zahlbar in zwoelf Monatsraten. Woechentliche Arbeitszeit 40 Stunden. Urlaubsanspruch 30 Arbeitstage im Kalenderjahr. Probezeit sechs Monate."}
-{"id":"aok-krankenkasse","text":"AOK Bayern. Beitragsbescheid zur gesetzlichen Kranken- und Pflegeversicherung. Versichertennummer A123456789. Beitragspflichtige Einnahmen 4.500 EUR monatlich. Allgemeiner Beitragssatz 14,6 Prozent zuzueglich Zusatzbeitrag 1,7 Prozent. Pflegeversicherung 3,4 Prozent. Monatlicher Gesamtbeitrag 887,50 EUR."}
-{"id":"sparkasse-kontoauszug","text":"Stadtsparkasse Muenchen. Kontoauszug Nummer 3 zum Girokonto IBAN DE89 7015 0000 0012 3456 78. Alter Kontostand 2.340,12 EUR. Gutschrift Gehalt 4.210,00 EUR. Lastschrift Miete 1.420,00 EUR. Dauerauftrag Sparplan 300,00 EUR. Neuer Kontostand 4.830,12 EUR."}
-{"id":"amazon-bestellung","text":"Amazon EU S.a.r.l. Bestellbestaetigung. Bestellnummer 302-4455667-8899001. Artikel Logitech MX Master 3S Wireless Maus, Menge 1, Preis 89,99 EUR. Versand durch Amazon, Lieferung voraussichtlich am 04.09. Zahlungsart Kreditkarte. Rechnungsbetrag inklusive Mehrwertsteuer 89,99 EUR."}
-{"id":"handwerker-rechnung","text":"Sanitaer Bergmeister GmbH. Rechnung fuer erbrachte Leistungen. Rechnungsnummer 2026-0417. Austausch eines defekten Heizungsthermostats sowie Entlueftung von drei Heizkoerpern. Arbeitszeit 2,5 Stunden zu 68 EUR. Material 42,30 EUR. Nettobetrag 212,30 EUR, Umsatzsteuer 19 Prozent, Gesamtbetrag 252,64 EUR."}
-{"id":"kfz-zulassung","text":"Landeshauptstadt Muenchen Kreisverwaltungsreferat. Zulassungsbescheinigung Teil I. Amtliches Kennzeichen M-AB 1234. Fahrzeugklasse Personenkraftwagen. Hersteller Volkswagen, Handelsbezeichnung Golf. Erstzulassung 14.03.2022. Hubraum 1498 Kubikzentimeter, Leistung 110 kW. Naechste Hauptuntersuchung 03.2028."}
@@ -1,12 +0,0 @@
-{"q":"Wie hoch war mein Stromverbrauch im letzten Jahr?","want":"stadtwerke-strom","note":"paraphrase: asks about consumption, the document says Verbrauch kWh"}
-{"q":"Was kostet mein Handyvertrag monatlich?","want":"telekom-mobilfunk","note":"paraphrase: Handyvertrag vs Mobilfunkrechnung/Tarif"}
-{"q":"Bescheid vom Finanzamt ueber die Einkommensteuer","want":"finanzamt-steuer","note":"near-literal, should be easy"}
-{"q":"Welchen Versicherungsschutz habe ich fuer mein Auto?","want":"allianz-kfz","note":"HARD: two Allianz policies. Must not pick the Hausrat one."}
-{"q":"Ist mein Fahrrad gegen Diebstahl versichert?","want":"allianz-hausrat","note":"HARD: same sender as kfz, and bicycles are a detail deep in the text"}
-{"q":"Wie hoch ist die Kaltmiete meiner Wohnung?","want":"mietvertrag","note":"must beat sparkasse, which also mentions Miete as a Lastschrift"}
-{"q":"Wie viele Urlaubstage stehen mir zu?","want":"arbeitsvertrag","note":"paraphrase: Urlaubsanspruch"}
-{"q":"Was zahle ich monatlich an die Krankenkasse?","want":"aok-krankenkasse","note":"Krankenkasse vs Kranken- und Pflegeversicherung"}
-{"q":"Wie viel Geld ist auf meinem Girokonto?","want":"sparkasse-kontoauszug","note":"Kontostand"}
-{"q":"Rechnung fuer die Reparatur der Heizung","want":"handwerker-rechnung","note":"HARD: several documents are Rechnungen; must pick the Sanitaer one"}
-{"q":"Wann muss mein Auto zum TUEV?","want":"kfz-zulassung","note":"HARD: TUEV never appears; the document says Hauptuntersuchung. Pure semantics, and must not be pulled to the Kfz-Versicherung by the shared plate number."}
-{"q":"Wann wurde meine Maus geliefert?","want":"amazon-bestellung","note":"colloquial vs Bestellbestaetigung/Lieferung"}