Fix: config-sync sidecar OOMKill (disable git auto-gc + raise limit) #88

2026-06-29T19:46:52Z

k.renner commented

2026-06-29 19:46:52 +00:00

Problem

After #87 merged and rolled out, the config-sync sidecar OOMKilled (exit 137) at the 64Mi limit and crashlooped — it never completed a sync, so config is still not auto-deploying.

Root cause

The homeassistant repo's .git is ~1.1 GB, dominated by a 1 GB backup tar committed into history (backups/home-assistant…tar). On first git fetch, git auto-gc tried to repack that entire pack inside the 64Mi sidecar → instant OOM (it died ~7s after start). Measured peak RSS for the actual fetch/checkout of config blobs is ~8 MB, so the sync itself is cheap — it was the repack that blew up.

Fix

Disable gc.auto in config-sync.sh (+ cap pack.windowMemory, pack.deltaCacheSize, core.deltaBaseCacheLimit) so the sidecar never repacks and only touches small config blobs.
Raise the sidecar memory limit 64Mi → 256Mi for index-pack headroom on the first fetch.

Follow-up (separate, not in this PR)

Purge the 1 GB backup tar from the homeassistant repo history (git-filter-repo/BFG) and .gitignore backups/. It bloats every clone and the config PVC, and is the underlying reason git is heavy here.

Validation

sh -n clean, kubectl kustomize builds. Once merged, Flux rolls a new pod; I'll confirm the sidecar reaches steady state, syncs /config to main, and that homeassistant.reload_all makes the new add_notification_to_queue (audience/title/data) live.

🤖 Generated with Claude Code

## Problem After #87 merged and rolled out, the `config-sync` sidecar **OOMKilled (exit 137) at the 64Mi limit and crashlooped** — it never completed a sync, so config is still not auto-deploying. ## Root cause The homeassistant repo's `.git` is **~1.1 GB**, dominated by a **1 GB backup tar committed into history** (`backups/home-assistant…tar`). On first `git fetch`, git **auto-gc** tried to repack that entire pack inside the 64Mi sidecar → instant OOM (it died ~7s after start). Measured peak RSS for the actual fetch/checkout of config blobs is ~8 MB, so the sync itself is cheap — it was the repack that blew up. ## Fix - **Disable `gc.auto`** in `config-sync.sh` (+ cap `pack.windowMemory`, `pack.deltaCacheSize`, `core.deltaBaseCacheLimit`) so the sidecar never repacks and only touches small config blobs. - Raise the sidecar memory limit **64Mi → 256Mi** for index-pack headroom on the first fetch. ## Follow-up (separate, not in this PR) Purge the 1 GB backup tar from the homeassistant repo history (git-filter-repo/BFG) and `.gitignore` `backups/`. It bloats every clone and the config PVC, and is the underlying reason git is heavy here. ## Validation `sh -n` clean, `kubectl kustomize` builds. Once merged, Flux rolls a new pod; I'll confirm the sidecar reaches steady state, syncs `/config` to `main`, and that `homeassistant.reload_all` makes the new `add_notification_to_queue` (audience/title/data) live. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

k.renner added 1 commit 2026-06-29 19:46:52 +00:00

fix(home-assistant): stop config-sync sidecar OOM (disable git auto-gc, raise limit) 8d7ecdda11

The config-sync sidecar OOMKilled (exit 137) at the 64Mi limit and crashlooped,
so it never synced. Root cause: this repo's .git is ~1.1 GB (a 1 GB backup tar
lives in history), and git auto-gc tried to repack that whole pack inside the
tiny sidecar on first fetch.

- Disable gc.auto in the sync script and cap pack window / delta caches, so
  fetch/checkout only ever touch the small config blobs and never repack.
- Raise the sidecar memory limit 64Mi -> 256Mi for index-pack headroom on the
  first fetch.

Follow-up (not here): purge the 1 GB backup tar from the homeassistant repo
history and .gitignore backups/ — it bloats every clone and the config PVC.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GgbyKgrN1yYWrkAA32rJE5

k.renner merged commit 35d584090b into master

2026-06-29 19:47:51 +00:00

k.renner referenced this issue from a commit

2026-06-29 19:47:52 +00:00

Merge pull request 'Fix: config-sync sidecar OOMKill (disable git auto-gc + raise limit)' (#88) from fix-config-sync-oom into master

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#88