Compare commits

...
Author SHA1 Message Date
claude-botandClaude Opus 5.5 6edde49b48 feat(k8s): make single-instance workloads survive node drains
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Draining k8s-3 for the Talos 1.13.10 upgrade stopped on four pods
(wp-pg-1, ha-pg-1, gf-pg-1, gitea-0) whose PDBs allow zero disruptions,
and each needed a manual --disable-eviction.

CNPG: ha-pg, gf-pg, wp-pg and tr-pg go to instances: 2. With a replica,
CNPG switches the primary over ahead of a drain, the same way ak-pg,
im-pg and pprl-pg already do. With one instance it creates a primary PDB
and 'will prevent draining the node where the instance is housed'
(CNPG docs, kubernetes_upgrade). smartShutdownTimeout 10 / stopDelay 60
as on ak-pg, which bound switchover time and the barman-plugin demote
hang (cloudnative-pg#9770). Network policies already allow intra-cluster
replication, operator :8000 and WAL-restore egress in all four
namespaces; nothing references a specific instance name.

Single-replica RWO workloads (gitea, woodpecker-server,
protonmail-bridge, syncthing, influxdb): drop the minAvailable:1 PDBs.
They cannot fail over, and eviction already stops the pod gracefully
with SIGTERM and its grace period; the PDB only forced a manual
--disable-eviction on every drain. influxdb's comes from the chart
(pdb.create, influxdb2 2.1.2 defaults).

The 'single-instance cluster, the archive is the only off-node copy'
comments no longer hold and are removed (im-pg's was already stale).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 19:06:16 +02:00
k.renner 47fbc917fd Merge pull request 'ci(wiki-maintenance): run the audit as the vault's script, commit only vault content' (#925) from ci/wiki-audit-script into master
ci/woodpecker/manual/wiki-maintenance Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#925
2026-09-23 16:06:52 +00:00
12 changed files with 18 additions and 103 deletions
-1
View File
@@ -6,7 +6,6 @@ kind: Kustomization
# and verified. See MIGRATION-gitea-woodpecker.md.
resources:
- statefulset.yaml
- pdb.yaml
- service.yaml
- ingressroute-ssh.yaml
- ingress.yaml
-23
View File
@@ -1,23 +0,0 @@
---
# Eviction protection for the single-replica Gitea StatefulSet.
#
# Gitea runs on SQLite (single writer) and its repos + DB live on a single
# ReadWriteOnce PVC, so it is deliberately replicas:1 and cannot fail over.
# With no PDB, a node drain / maintenance evicts the pod abruptly, mid-write,
# with zero coordination. minAvailable:1 on a 1-replica workload makes the
# eviction API refuse *voluntary* evictions: `kubectl drain` blocks and
# surfaces this pod, forcing a conscious decision (drain with
# --disable-eviction, or scale down first) instead of yanking the live DB.
# It does NOT block involuntary disruptions (node crash, OOM).
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: gitea
namespace: gitea
labels:
app.kubernetes.io/name: gitea
spec:
minAvailable: 1
selector:
matchLabels:
app.kubernetes.io/name: gitea
+3 -1
View File
@@ -4,7 +4,9 @@ metadata:
name: gf-pg
namespace: monitoring
spec:
instances: 1
instances: 2
smartShutdownTimeout: 10
stopDelay: 60
imageName: ghcr.io/cloudnative-pg/postgresql:16
storage:
storageClass: ceph-block
@@ -30,7 +30,9 @@ metadata:
name: ha-pg
namespace: home-assistant
spec:
instances: 1
instances: 2
smartShutdownTimeout: 10
stopDelay: 60
imageName: ghcr.io/cloudnative-pg/postgresql:16
storage:
storageClass: ceph-block
@@ -124,8 +126,7 @@ spec:
# padding rather than data.
#
# The cost is RPO: up to 30min of WAL is unrecoverable if this cluster and
# its volume are lost outright. This is a single-instance cluster, so the
# archive is the only off-node copy.
# its volumes are lost outright.
archive_timeout: "30min"
# Fewer checkpoints => fewer full-page writes, since a page is only
# fully-logged on its first modification after a checkpoint.
+1 -2
View File
@@ -59,8 +59,7 @@ spec:
# padding rather than data.
#
# The cost is RPO: up to 30min of WAL is unrecoverable if this cluster and
# its volume are lost outright. This is a single-instance cluster, so the
# archive is the only off-node copy.
# its volumes are lost outright.
archive_timeout: "30min"
# Write-reduction tuning. These databases are largely idle, and the
# Postgres defaults favour throughput over storage that wears out.
@@ -61,6 +61,9 @@ spec:
enabled: true
storageClass: ceph-block
size: 50Gi
# A minAvailable:1 PDB on a single replica blocks every node drain.
pdb:
create: false
# Bootstrap admin/org/bucket on the fresh PVC. Admin password + token come
# from the influxdb-bootstrap Secret (OpenBao via ESO), deterministic across
# rebuilds so the bootstrap token is the live admin token. `influx setup`
@@ -11,7 +11,6 @@ resources:
- protonmail-bridge.yaml
- ftp-scanner.yaml
- syncthing.yaml
- pdb.yaml
- network-policy.yaml
# pprl-pg CNPG Postgres + data/media/consume/redis PVCs (moved here from the
# retired seed-paperless KS; each carries prune: disabled so ownership
-37
View File
@@ -1,37 +0,0 @@
---
# Eviction protection for the single-replica stateful paperless sidecars.
#
# protonmail-bridge and syncthing each hold state on a single RWO PVC and run
# replicas:1. minAvailable:1 makes the eviction API refuse voluntary evictions,
# so a node drain blocks and surfaces these pods instead of evicting them
# abruptly. Involuntary disruptions (node crash) are not affected.
#
# NOTE: the other single-replica paperless pods (paperless, redis, gotenberg,
# tika, ftp-scanner) are intentionally omitted — they are caches / stateless
# converters where a blocking PDB would only wedge node drains with no
# recoverability benefit.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: protonmail-bridge
namespace: paperless
labels:
app.kubernetes.io/name: protonmail-bridge
spec:
minAvailable: 1
selector:
matchLabels:
app.kubernetes.io/name: protonmail-bridge
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: syncthing
namespace: paperless
labels:
app.kubernetes.io/name: syncthing
spec:
minAvailable: 1
selector:
matchLabels:
app.kubernetes.io/name: syncthing
+2 -3
View File
@@ -4,7 +4,7 @@ metadata:
name: tr-pg
namespace: trmnl
spec:
instances: 1
instances: 2
smartShutdownTimeout: 10
stopDelay: 60
# PG 18, NOT the repo-standard 16. Terminus ships a committed
@@ -47,8 +47,7 @@ spec:
# padding rather than data.
#
# The cost is RPO: up to 30min of WAL is unrecoverable if this cluster and
# its volume are lost outright. This is a single-instance cluster, so the
# archive is the only off-node copy.
# its volumes are lost outright.
archive_timeout: "30min"
# Write-reduction tuning. These databases are largely idle, and the
# Postgres defaults favour throughput over storage that wears out.
+4 -3
View File
@@ -22,7 +22,9 @@ metadata:
# migration deliberately withheld this while a rollback still existed.
kustomize.toolkit.fluxcd.io/prune: disabled
spec:
instances: 1
instances: 2
smartShutdownTimeout: 10
stopDelay: 60
imageName: ghcr.io/cloudnative-pg/postgresql:16
storage:
storageClass: ceph-block
@@ -91,8 +93,7 @@ spec:
# padding rather than data.
#
# The cost is RPO: up to 30min of WAL is unrecoverable if this cluster and
# its volume are lost outright. This is a single-instance cluster, so the
# archive is the only off-node copy.
# its volumes are lost outright.
archive_timeout: "30min"
resources:
requests:
@@ -3,7 +3,6 @@ kind: Kustomization
resources:
- rbac.yaml
- server.yaml
- pdb.yaml
- agent.yaml
- service.yaml
- ingress.yaml
@@ -22,7 +21,7 @@ resources:
# OpenBao access, no NetworkPolicy needed. server.yaml/agent.yaml keep their
# Reloader annotations; externalsecret.yaml keeps the 10m refreshInterval.
# --- namespace migration, stages 1-2 (issue #433) ---
# Stage 2 has moved the workloads above (server, agent, service, ingress, pdb,
# Stage 2 has moved the workloads above (server, agent, service, ingress,
# rbac, secrets, policies) into the `woodpecker` namespace and pointed the
# server's DSN at wp-pg-rw.woodpecker.svc. Stage 3 retired the `apps`
# originals; these four now describe the surviving cluster. History and the
-27
View File
@@ -1,27 +0,0 @@
---
# Eviction protection for the single-replica Woodpecker server StatefulSet.
#
# The server is SQLite-backed (single writer) on a single RWO PVC, so it is
# deliberately replicas:1. minAvailable:1 makes the eviction API refuse
# voluntary evictions so a node drain blocks and surfaces this pod rather
# than yanking the live CI DB mid-write. Involuntary disruptions are not
# affected.
#
# The selector MUST include app.kubernetes.io/component: server: the agent
# Deployment in this same namespace shares app.kubernetes.io/name: woodpecker
# (component: agent), and a name-only selector would wrongly also cover the
# stateless, disposable agents.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: woodpecker-server
namespace: woodpecker
labels:
app.kubernetes.io/name: woodpecker
app.kubernetes.io/component: server
spec:
minAvailable: 1
selector:
matchLabels:
app.kubernetes.io/name: woodpecker
app.kubernetes.io/component: server