influxdb(seed): drop chunk size to per-day
806e979 (per-month) still EOF'd at 2024-04 — first dense month for HA
telemetry, pivot of one month at that density overwhelms legacy's
memory. Drop to per-day chunks (~3300 iterations across 9 years,
mostly empty days no-op'ing in <1 s; dense days under 1 MB pivoted).
This commit is contained in:
@@ -105,13 +105,15 @@ spec:
|
||||
# columns; `drop()` strips the _start/_stop range columns
|
||||
# that `influx write` rejects.
|
||||
#
|
||||
# Why chunked, and why monthly: pivoting in-memory in
|
||||
# Why chunked, and why daily: pivoting in-memory in
|
||||
# legacy means the whole result set has to fit before it
|
||||
# flushes. The previous per-year shape EOF'd on the
|
||||
# current year (full year of HA telemetry won't pivot in
|
||||
# legacy's RAM). Per-month windows keep each pivot to
|
||||
# tens of MB regardless of year, so legacy never OOMs.
|
||||
# Empty months are no-ops (~1 s each).
|
||||
# flushes. Per-year EOF'd on the current year; per-month
|
||||
# also EOF'd on dense months (HA-telemetry from 2024+ is
|
||||
# too big at one-month granularity). Per-day windows keep
|
||||
# each pivot tiny regardless of write rate, so legacy
|
||||
# never OOMs. Empty days are no-ops (~1 s each); the
|
||||
# outer loop is ~3300 iterations across 9 years but most
|
||||
# complete in <1 s.
|
||||
#
|
||||
# No script-side idempotency: InfluxDB writes are upserts
|
||||
# on (timestamp,measurement,tagset,field), so a retry
|
||||
@@ -123,12 +125,11 @@ spec:
|
||||
# an `influx query` EOF inside the pipe propagates a non-
|
||||
# zero exit, the loop aborts, the Job retries.
|
||||
START="2018-01-01T00:00:00Z"
|
||||
# Stop at the start of the month AFTER current → covers
|
||||
# the in-progress month inclusively.
|
||||
END=$(date -u -d "$(date -u +%Y-%m-01) +1 month" +%Y-%m-%dT00:00:00Z)
|
||||
# Stop at tomorrow → covers today inclusively.
|
||||
END=$(date -u -d "tomorrow" +%Y-%m-%dT00:00:00Z)
|
||||
CUR="$START"
|
||||
while [[ "$CUR" < "$END" ]]; do
|
||||
NEXT=$(date -u -d "$CUR +1 month" +%Y-%m-%dT00:00:00Z)
|
||||
NEXT=$(date -u -d "$CUR +1 day" +%Y-%m-%dT00:00:00Z)
|
||||
log "Streaming $CUR → $NEXT ..."
|
||||
influx query --raw \
|
||||
--host "$LEGACY_HOST" \
|
||||
|
||||
Reference in New Issue
Block a user