influxdb(seed): per-month chunks instead of per-year
Per-year chunking from a315ba6 still EOF'd on the current year. With
~9 years of HA telemetry the early years (2018–2024) were near-empty
and each completed in <1 s, but the full-data year (2024–2025 / current)
hit the same memory ceiling on legacy that the unchunked version did
— pivoting one year of HA metrics still doesn't fit in legacy's RAM
and the connection drops with `Post .../query: EOF`.
Switch to per-month chunks (~108 chunks for 9 years). Each pivot is
~1/12 the data so legacy holds it. Empty months no-op in <1 s. Loop
uses `date -d "+1 month"` to walk calendar months including the
in-progress one.
Pipefail still on, retries still upsert-safe, K8s Job Complete still
the post-success guard.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -96,7 +96,7 @@ spec:
|
||||
log() { printf '[%s] %s\n' "$(date -u +%H:%M:%S)" "$*"; }
|
||||
|
||||
# Stream legacy bucket → local bucket through the v2 HTTP
|
||||
# API in per-year chunks. Annotated-CSV is the wire format
|
||||
# API in per-month chunks. Annotated-CSV is the wire format
|
||||
# on both ends, but `influx query` emits LONG format (one
|
||||
# row per _time/_measurement/_field/_value tuple) while
|
||||
# `influx write --format csv` requires WIDE format (one row
|
||||
@@ -105,35 +105,37 @@ spec:
|
||||
# columns; `drop()` strips the _start/_stop range columns
|
||||
# that `influx write` rejects.
|
||||
#
|
||||
# Why chunked: pivoting the entire bucket at once
|
||||
# materialises the full result set in legacy's memory
|
||||
# before flushing — for a multi-year HA-metrics bucket
|
||||
# that exceeds available memory and the connection drops
|
||||
# mid-stream (`Post .../query: EOF`). One year at a time
|
||||
# keeps each pivot under a few hundred MB.
|
||||
# Why chunked, and why monthly: pivoting in-memory in
|
||||
# legacy means the whole result set has to fit before it
|
||||
# flushes. The previous per-year shape EOF'd on the
|
||||
# current year (full year of HA telemetry won't pivot in
|
||||
# legacy's RAM). Per-month windows keep each pivot to
|
||||
# tens of MB regardless of year, so legacy never OOMs.
|
||||
# Empty months are no-ops (~1 s each).
|
||||
#
|
||||
# No script-side idempotency check: InfluxDB writes are
|
||||
# upserts on (timestamp,measurement,tagset,field), so
|
||||
# re-running this Job re-applies the same data without
|
||||
# corrupting it. The K8s Job's `Complete` state is the
|
||||
# post-success guard against re-runs — Flux reconcile
|
||||
# sees Complete and leaves the Job alone.
|
||||
# No script-side idempotency: InfluxDB writes are upserts
|
||||
# on (timestamp,measurement,tagset,field), so a retry
|
||||
# after partial-failure just re-asserts the same points.
|
||||
# The K8s Job's `Complete` state is the post-success guard
|
||||
# — Flux reconcile sees Complete and leaves the Job alone.
|
||||
#
|
||||
# `pipefail` is on so an `influx query` EOF mid-pipe
|
||||
# propagates a non-zero exit code (without it, `influx
|
||||
# write` swallows the empty stream and returns 0,
|
||||
# producing a silent partial-success).
|
||||
START_YEAR=2018
|
||||
END_YEAR=$(date -u +%Y)
|
||||
for YEAR in $(seq "$START_YEAR" "$END_YEAR"); do
|
||||
NEXT=$((YEAR + 1))
|
||||
log "Streaming $YEAR-01-01 → ${NEXT}-01-01 ..."
|
||||
# `pipefail` (bash-only, hence the `command: bash` above):
|
||||
# an `influx query` EOF inside the pipe propagates a non-
|
||||
# zero exit, the loop aborts, the Job retries.
|
||||
START="2018-01-01T00:00:00Z"
|
||||
# Stop at the start of the month AFTER current → covers
|
||||
# the in-progress month inclusively.
|
||||
END=$(date -u -d "$(date -u +%Y-%m-01) +1 month" +%Y-%m-%dT00:00:00Z)
|
||||
CUR="$START"
|
||||
while [[ "$CUR" < "$END" ]]; do
|
||||
NEXT=$(date -u -d "$CUR +1 month" +%Y-%m-%dT00:00:00Z)
|
||||
log "Streaming $CUR → $NEXT ..."
|
||||
influx query --raw \
|
||||
--host "$LEGACY_HOST" \
|
||||
--token "$LEGACY_TOKEN" \
|
||||
--org "$LEGACY_ORG" \
|
||||
"from(bucket: \"$LEGACY_BUCKET\")
|
||||
|> range(start: ${YEAR}-01-01T00:00:00Z, stop: ${NEXT}-01-01T00:00:00Z)
|
||||
|> range(start: $CUR, stop: $NEXT)
|
||||
|> pivot(rowKey: [\"_time\"], columnKey: [\"_field\"], valueColumn: \"_value\")
|
||||
|> drop(columns: [\"_start\", \"_stop\"])" \
|
||||
| influx write \
|
||||
@@ -142,6 +144,7 @@ spec:
|
||||
--org "$INFLUX_ORG" \
|
||||
--bucket "$INFLUX_BUCKET" \
|
||||
--format csv
|
||||
CUR="$NEXT"
|
||||
done
|
||||
|
||||
log "Seed complete."
|
||||
|
||||
Reference in New Issue
Block a user