influxdb(seed): per-month chunks instead of per-year

Per-year chunking from a315ba6 still EOF'd on the current year. With
~9 years of HA telemetry the early years (2018–2024) were near-empty
and each completed in <1 s, but the full-data year (2024–2025 / current)
hit the same memory ceiling on legacy that the unchunked version did
— pivoting one year of HA metrics still doesn't fit in legacy's RAM
and the connection drops with `Post .../query: EOF`.

Switch to per-month chunks (~108 chunks for 9 years). Each pivot is
~1/12 the data so legacy holds it. Empty months no-op in <1 s. Loop
uses `date -d "+1 month"` to walk calendar months including the
in-progress one.

Pipefail still on, retries still upsert-safe, K8s Job Complete still
the post-success guard.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-05 01:16:09 +02:00
co-authored by Claude Opus 4.7
parent 53c47396a4
commit 806e9794ad
+26 -23
View File
@@ -96,7 +96,7 @@ spec:
log() { printf '[%s] %s\n' "$(date -u +%H:%M:%S)" "$*"; }
# Stream legacy bucket → local bucket through the v2 HTTP
# API in per-year chunks. Annotated-CSV is the wire format
# API in per-month chunks. Annotated-CSV is the wire format
# on both ends, but `influx query` emits LONG format (one
# row per _time/_measurement/_field/_value tuple) while
# `influx write --format csv` requires WIDE format (one row
@@ -105,35 +105,37 @@ spec:
# columns; `drop()` strips the _start/_stop range columns
# that `influx write` rejects.
#
# Why chunked: pivoting the entire bucket at once
# materialises the full result set in legacy's memory
# before flushing — for a multi-year HA-metrics bucket
# that exceeds available memory and the connection drops
# mid-stream (`Post .../query: EOF`). One year at a time
# keeps each pivot under a few hundred MB.
# Why chunked, and why monthly: pivoting in-memory in
# legacy means the whole result set has to fit before it
# flushes. The previous per-year shape EOF'd on the
# current year (full year of HA telemetry won't pivot in
# legacy's RAM). Per-month windows keep each pivot to
# tens of MB regardless of year, so legacy never OOMs.
# Empty months are no-ops (~1 s each).
#
# No script-side idempotency check: InfluxDB writes are
# upserts on (timestamp,measurement,tagset,field), so
# re-running this Job re-applies the same data without
# corrupting it. The K8s Job's `Complete` state is the
# post-success guard against re-runs — Flux reconcile
# sees Complete and leaves the Job alone.
# No script-side idempotency: InfluxDB writes are upserts
# on (timestamp,measurement,tagset,field), so a retry
# after partial-failure just re-asserts the same points.
# The K8s Job's `Complete` state is the post-success guard
# — Flux reconcile sees Complete and leaves the Job alone.
#
# `pipefail` is on so an `influx query` EOF mid-pipe
# propagates a non-zero exit code (without it, `influx
# write` swallows the empty stream and returns 0,
# producing a silent partial-success).
START_YEAR=2018
END_YEAR=$(date -u +%Y)
for YEAR in $(seq "$START_YEAR" "$END_YEAR"); do
NEXT=$((YEAR + 1))
log "Streaming $YEAR-01-01 → ${NEXT}-01-01 ..."
# `pipefail` (bash-only, hence the `command: bash` above):
# an `influx query` EOF inside the pipe propagates a non-
# zero exit, the loop aborts, the Job retries.
START="2018-01-01T00:00:00Z"
# Stop at the start of the month AFTER current → covers
# the in-progress month inclusively.
END=$(date -u -d "$(date -u +%Y-%m-01) +1 month" +%Y-%m-%dT00:00:00Z)
CUR="$START"
while [[ "$CUR" < "$END" ]]; do
NEXT=$(date -u -d "$CUR +1 month" +%Y-%m-%dT00:00:00Z)
log "Streaming $CUR → $NEXT ..."
influx query --raw \
--host "$LEGACY_HOST" \
--token "$LEGACY_TOKEN" \
--org "$LEGACY_ORG" \
"from(bucket: \"$LEGACY_BUCKET\")
|> range(start: ${YEAR}-01-01T00:00:00Z, stop: ${NEXT}-01-01T00:00:00Z)
|> range(start: $CUR, stop: $NEXT)
|> pivot(rowKey: [\"_time\"], columnKey: [\"_field\"], valueColumn: \"_value\")
|> drop(columns: [\"_start\", \"_stop\"])" \
| influx write \
@@ -142,6 +144,7 @@ spec:
--org "$INFLUX_ORG" \
--bucket "$INFLUX_BUCKET" \
--format csv
CUR="$NEXT"
done
log "Seed complete."