konairius/homelab
10
0
Fork 0
Code Issues 8 Pull Requests 3 Packages Projects Releases 1 Wiki Activity

[recoverability][medium] No PodDisruptionBudgets anywhere (incl. single-instance Immich DB) #201

New Issue
Closed
opened 2026-07-14 09:48:08 +00:00 by claude-bot · 1 comment
claude-bot commented 2026-07-14 09:48:08 +00:00
Member
Copy Link
Copy Source

Finding

Zero PodDisruptionBudget objects exist in the repo. Single-replica stateful workloads with no eviction protection: Gitea StatefulSet, Woodpecker server, paperless protonmail-bridge/syncthing, CNPG im-pg (Immich's live DB, currently instances:1), CNPG gf-pg, plus all single-replica app Deployments.

Verified

grep 'kind: PodDisruptionBudget' across kubernetes/ returns nothing.

Impact

A node drain / maintenance can evict single-replica stateful pods (including the live Immich database) with no protection.

Fix

Add PDBs for critical single-replica stateful workloads and quorum-based infra (Ceph mons, CNPG).

Automated security review 2026-07-14. Verified.

## Finding Zero PodDisruptionBudget objects exist in the repo. Single-replica stateful workloads with no eviction protection: Gitea StatefulSet, Woodpecker server, paperless protonmail-bridge/syncthing, CNPG im-pg (Immich's live DB, currently instances:1), CNPG gf-pg, plus all single-replica app Deployments. ## Verified grep 'kind: PodDisruptionBudget' across kubernetes/ returns nothing. ## Impact A node drain / maintenance can evict single-replica stateful pods (including the live Immich database) with no protection. ## Fix Add PDBs for critical single-replica stateful workloads and quorum-based infra (Ceph mons, CNPG). --- _Automated security review 2026-07-14. Verified._
claude-bot commented 2026-07-14 10:11:44 +00:00
Author
Member
Copy Link
Copy Source

Fixed in PR #234 (branch fix/issue-201).

Added minAvailable: 1 PodDisruptionBudgets for the 4 data-bearing single-replica StatefulSets that had no eviction protection:

  • gitea (apps)
  • woodpecker-server (apps) — selector scoped with component: server so it doesn't catch the stateless agents
  • protonmail-bridge (paperless)
  • syncthing (paperless)

On a 1-replica workload minAvailable: 1 makes the eviction API refuse voluntary evictions, so a node drain blocks and surfaces the pod instead of yanking a live SQLite DB / RWO-PVC pod mid-write (involuntary disruptions like node crash are unaffected).

Deliberately excluded the other candidates from the finding, with rationale (full detail in the PR):

  • CNPG im-pg / gf-pg (instances:1): CloudNativePG intentionally creates no primary PDB for single-instance clusters and handles graceful primary shutdown itself. A manual maxUnavailable: 0 would wedge drains with no HA benefit and fight the operator.
  • Ceph mons: the Rook operator already self-manages quorum-preserving mon/OSD PDBs (managePodBudgets default-on). Hand-writing one would be redundant/overwritten.
  • Stateless single-replica Deployments (redis/tika/gotenberg caches, whisper, etc.): a blocking PDB gives no recoverability benefit, only blocks drains.

Validated: kubectl kustomize builds clean for all three app dirs, rendering exactly 4 PDBs with correct selectors.

Open question for review: minAvailable: 1 blocks voluntary evictions entirely (drains then need --disable-eviction or a scale-down first). That friction is the intended protection here — but if you'd rather not block drains for any specific one, say so and I'll drop it.

Fixed in PR #234 (branch `fix/issue-201`). Added `minAvailable: 1` PodDisruptionBudgets for the 4 data-bearing single-replica StatefulSets that had no eviction protection: - `gitea` (apps) - `woodpecker-server` (apps) — selector scoped with `component: server` so it doesn't catch the stateless agents - `protonmail-bridge` (paperless) - `syncthing` (paperless) On a 1-replica workload `minAvailable: 1` makes the eviction API refuse voluntary evictions, so a node drain blocks and surfaces the pod instead of yanking a live SQLite DB / RWO-PVC pod mid-write (involuntary disruptions like node crash are unaffected). Deliberately **excluded** the other candidates from the finding, with rationale (full detail in the PR): - **CNPG `im-pg` / `gf-pg` (instances:1):** CloudNativePG intentionally creates no primary PDB for single-instance clusters and handles graceful primary shutdown itself. A manual `maxUnavailable: 0` would wedge drains with no HA benefit and fight the operator. - **Ceph mons:** the Rook operator already self-manages quorum-preserving mon/OSD PDBs (managePodBudgets default-on). Hand-writing one would be redundant/overwritten. - **Stateless single-replica Deployments** (redis/tika/gotenberg caches, whisper, etc.): a blocking PDB gives no recoverability benefit, only blocks drains. Validated: `kubectl kustomize` builds clean for all three app dirs, rendering exactly 4 PDBs with correct selectors. Open question for review: `minAvailable: 1` blocks voluntary evictions entirely (drains then need `--disable-eviction` or a scale-down first). That friction is the intended protection here — but if you'd rather not block drains for any specific one, say so and I'll drop it.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#201