Compare commits
10
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
68c8c86046 | ||
|
|
0d65fb0982 | ||
|
|
ece3c93ad5 | ||
|
|
b34450b22e | ||
|
|
b4f44b2002 | ||
|
|
3faddcadcd | ||
|
|
6ae734eef4 | ||
|
|
39885b9400 | ||
|
|
57bb39e7ee | ||
|
|
17f913ba2f |
@@ -121,7 +121,9 @@ spec:
|
||||
persistentVolumeClaim:
|
||||
claimName: audiobookshelf-books
|
||||
- name: le-ca
|
||||
emptyDir: {}
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 16Mi
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
|
||||
@@ -43,6 +43,27 @@ spec:
|
||||
parameters:
|
||||
shared_buffers: "256MB"
|
||||
max_connections: "400"
|
||||
# Write-reduction tuning. These databases are largely idle, and the
|
||||
# Postgres defaults favour throughput over storage that wears out.
|
||||
#
|
||||
# checkpoint_timeout is the lever. The 5min default pushes every dirty
|
||||
# page to disk 288 times a day, so a small database whose same few pages
|
||||
# are re-dirtied constantly (sessions, tokens, job state) gets rewritten
|
||||
# over and over. Measured on ak-pg: 25.8M checkpoint buffers flushed
|
||||
# (~206 GB) against a 1046 MB database -- the whole thing roughly twice a
|
||||
# day. At 15min those pages coalesce and are written a third as often.
|
||||
#
|
||||
# max_wal_size must rise with it, or checkpoints trigger on size instead
|
||||
# and the timeout change does nothing. It also bounds crash recovery to
|
||||
# at most 2GB of WAL replay.
|
||||
#
|
||||
# NOT changed: synchronous_commit stays at the default `on`. It is off for
|
||||
# ha-pg because a state recorder is telemetry; these hold credentials,
|
||||
# documents and job state, where losing the last few hundred ms of commits
|
||||
# on an unclean shutdown is not an acceptable trade.
|
||||
checkpoint_timeout: "15min"
|
||||
max_wal_size: "2GB"
|
||||
wal_compression: "on"
|
||||
# Async replication makes CNPG switchover slow because the standby has
|
||||
# to chase missing WAL via barman-cloud and replay queued records
|
||||
# before it can promote. With one synchronous standby, the standby is
|
||||
|
||||
@@ -29,6 +29,27 @@ spec:
|
||||
parameters:
|
||||
shared_buffers: "128MB"
|
||||
max_connections: "50"
|
||||
# Write-reduction tuning. These databases are largely idle, and the
|
||||
# Postgres defaults favour throughput over storage that wears out.
|
||||
#
|
||||
# checkpoint_timeout is the lever. The 5min default pushes every dirty
|
||||
# page to disk 288 times a day, so a small database whose same few pages
|
||||
# are re-dirtied constantly (sessions, tokens, job state) gets rewritten
|
||||
# over and over. Measured on ak-pg: 25.8M checkpoint buffers flushed
|
||||
# (~206 GB) against a 1046 MB database -- the whole thing roughly twice a
|
||||
# day. At 15min those pages coalesce and are written a third as often.
|
||||
#
|
||||
# max_wal_size must rise with it, or checkpoints trigger on size instead
|
||||
# and the timeout change does nothing. It also bounds crash recovery to
|
||||
# at most 2GB of WAL replay.
|
||||
#
|
||||
# NOT changed: synchronous_commit stays at the default `on`. It is off for
|
||||
# ha-pg because a state recorder is telemetry; these hold credentials,
|
||||
# documents and job state, where losing the last few hundred ms of commits
|
||||
# on an unclean shutdown is not an acceptable trade.
|
||||
checkpoint_timeout: "15min"
|
||||
max_wal_size: "2GB"
|
||||
wal_compression: "on"
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
|
||||
@@ -66,7 +66,52 @@ Confirm the backup actually landed in the bucket rather than trusting the CR
|
||||
status — the WordPress backup failed nightly for three days against a bucket
|
||||
that was declared but never created (see `opentofu/object_storage.tf`).
|
||||
|
||||
## Phase 2 — migrate the data (NOT in this PR)
|
||||
## Phase 2b — bulk statistics copy: DONE 2026-08-19
|
||||
|
||||
Completed with **HA running** — the SQLite file was read through a read-only
|
||||
mount, and SQLite's WAL mode gives readers a consistent snapshot, so no downtime
|
||||
was needed for the bulk copy.
|
||||
|
||||
```
|
||||
statistics_meta: sqlite=1449 pg=1449 OK
|
||||
statistics: sqlite=10799904 pg=10799904 OK (153.9s)
|
||||
statistics_short_term: sqlite=1969804 pg=1969804 OK ( 27.0s)
|
||||
|
||||
statistics span in Postgres: 2024-03-13 -> 2026-08-19
|
||||
entities with long-term history: 1449
|
||||
rows carrying sum (energy dashboard): 2627859
|
||||
database size: 2252 MB (PVC is 20Gi)
|
||||
```
|
||||
|
||||
`ANALYZE` was run on all three tables. That is not housekeeping: after a bulk
|
||||
`COPY` the planner has no statistics, and HA queries these tables constantly for
|
||||
history and energy views, so without it every view seq-scans 10.8M rows.
|
||||
|
||||
Two schema details that cost a failed first attempt, recorded so the top-up does
|
||||
not repeat them:
|
||||
|
||||
- `has_mean` / `has_sum` are INTEGER in SQLite and BOOLEAN in Postgres. Postgres
|
||||
will not coerce; convert explicitly and preserve NULL.
|
||||
- The legacy `created` / `start` / `last_reset` DATETIME columns are 100% NULL in
|
||||
both bulk tables (superseded by the `_ts` variants), which is what makes a
|
||||
tab-separated `COPY` safe — nothing in those tables needs escaping.
|
||||
|
||||
The schema itself was created with HA's own models
|
||||
(`Base.metadata.create_all`), **not** hand-written DDL, and `schema_changes` was
|
||||
left EMPTY on purpose. HA's `_initialize_database` detects a new database by
|
||||
looking for an index on `events.time_fired_ts`, then writes `SCHEMA_VERSION`
|
||||
itself. Pre-stamping that row would skip HA's own initialisation path.
|
||||
|
||||
### High-water marks for the cutover top-up
|
||||
|
||||
Rows written after the bulk copy are not yet in Postgres. Resume from:
|
||||
|
||||
```
|
||||
statistics max_id=13323904 max_start_ts=1787162400.0
|
||||
statistics_short_term max_id=159205762 max_start_ts=1787166000.0
|
||||
```
|
||||
|
||||
## Phase 2c — cutover (REMAINING)
|
||||
|
||||
Do not start until phase 1 is verified.
|
||||
|
||||
@@ -90,7 +135,18 @@ HA outage, and buys a table that is discarded within 7 days anyway.
|
||||
SELECT setval('statistics_short_term_id_seq', (SELECT MAX(id) FROM statistics_short_term));
|
||||
```
|
||||
5. **Repoint** `recorder.db_url` in the `homeassistant` repo — that change lives
|
||||
there, not here:
|
||||
there, not here. The `recorder_db_url` secret it references is already
|
||||
rendered into `secrets.yaml` by the `render-db-url` init container in
|
||||
`deployment.yaml` (phase 2a), so this order is mandatory: **the init
|
||||
container must be deployed first**, or HA refuses to start on a missing
|
||||
secret.
|
||||
|
||||
The init container percent-encodes the credentials and validates the
|
||||
resulting YAML rather than counting lines. That is not incidental — the CNPG
|
||||
password is random, and a naive `"postgresql://user:pass@host/db"` string
|
||||
makes `secrets.yaml` unparseable the moment the password contains a quote,
|
||||
which takes out *every* secret HA loads, not just this one. Verified against
|
||||
a deliberately hostile password before merge.
|
||||
```yaml
|
||||
recorder:
|
||||
db_url: !secret recorder_db_url # postgresql://homeassistant:<pw>@ha-pg-rw.home-assistant.svc.cluster.local/homeassistant
|
||||
|
||||
@@ -36,6 +36,64 @@ spec:
|
||||
# (sml2mqtt bridge owns the USB device on edge), so it no longer needs
|
||||
# the edge USB device and schedules on the general cluster pool. This
|
||||
# gets the heavy HA pod (8Gi) off the soon-to-be-tiny edge box.
|
||||
initContainers:
|
||||
# Puts CNPG's connection URI into /config/secrets.yaml as
|
||||
# `recorder_db_url`, for the recorder's move off SQLite
|
||||
# (MIGRATION-recorder-postgres.md).
|
||||
#
|
||||
# HA has no way to read a db_url from the environment — `!env_var` was
|
||||
# removed from its YAML loader — so `!secret` is the only mechanism, and
|
||||
# secrets.yaml is PVC state that config-sync deliberately does not
|
||||
# manage. Hence a small init container rather than a env: entry.
|
||||
#
|
||||
# The URI comes STRAIGHT from the ha-pg-app Secret. CNPG builds and
|
||||
# percent-encodes it, so there is nothing to escape here: the value is
|
||||
# restricted to [A-Za-z0-9:/@._%-], verified against the live secret.
|
||||
# An earlier version of this assembled the URI from user/password/dbname
|
||||
# and needed percent-encoding plus YAML round-trip validation to be
|
||||
# safe. That was solving a problem CNPG had already solved.
|
||||
#
|
||||
# Re-rendered on every pod start, so a CNPG password rotation takes
|
||||
# effect rather than silently leaving a stale URI behind.
|
||||
#
|
||||
# INERT until the recorder is repointed: nothing reads this key until
|
||||
# `db_url: !secret recorder_db_url` lands in configuration.yaml in the
|
||||
# homeassistant repo. That ordering is required — HA refuses to start
|
||||
# when a referenced secret is missing.
|
||||
- name: render-db-url
|
||||
image: busybox:1.36
|
||||
command:
|
||||
- sh
|
||||
- -c
|
||||
- |
|
||||
set -eu
|
||||
S=/config/secrets.yaml
|
||||
T=/config/.secrets.render.tmp
|
||||
[ -f "$S" ] || : > "$S"
|
||||
# Filter-then-append, never rewrite in place: this file holds
|
||||
# every other HA secret (influxdb_token and friends).
|
||||
grep -v '^recorder_db_url:' "$S" > "$T" || true
|
||||
printf 'recorder_db_url: "%s"\n' "$DB_URI" >> "$T"
|
||||
# Refuse to shrink the file. Cheap guard against the one failure
|
||||
# that would actually hurt: losing the other keys.
|
||||
[ "$(wc -l < "$T")" -ge "$(wc -l < "$S")" ] || {
|
||||
echo "FATAL: rewrite would shrink secrets.yaml; aborting" >&2; exit 1; }
|
||||
chmod 600 "$T"
|
||||
mv "$T" "$S"
|
||||
echo "recorder_db_url rendered"
|
||||
env:
|
||||
- name: DB_URI
|
||||
valueFrom:
|
||||
secretKeyRef: { name: ha-pg-app, key: fqdn-uri }
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /config
|
||||
resources:
|
||||
requests:
|
||||
cpu: 10m
|
||||
memory: 16Mi
|
||||
limits:
|
||||
memory: 32Mi
|
||||
containers:
|
||||
- name: home-assistant
|
||||
image: ghcr.io/home-assistant/home-assistant:2026.8.2 # {"$imagepolicy": "flux-system:home-assistant"}
|
||||
@@ -221,9 +279,13 @@ spec:
|
||||
# Writable emptyDir for the nginx entrypoint's envsubst output
|
||||
# (/etc/nginx/conf.d/default.conf rendered from the .template).
|
||||
- name: proxy-conf-d
|
||||
emptyDir: {}
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 16Mi
|
||||
# code-server's settings/extensions cache. Not on the config PVC
|
||||
# so installed extensions don't pollute HA's /config listing.
|
||||
# Trade-off: extensions reset across pod restarts.
|
||||
- name: vscode-state
|
||||
emptyDir: {}
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 16Mi
|
||||
|
||||
@@ -55,7 +55,21 @@ spec:
|
||||
owner: homeassistant
|
||||
postgresql:
|
||||
parameters:
|
||||
shared_buffers: "256MB"
|
||||
# 512MB, not the 256MB this shipped with. Measured after 15h on Postgres:
|
||||
# buffers_backend was 629,704 of 939,025 total page writes — 67% written
|
||||
# by backend processes rather than the checkpointer or bgwriter. That
|
||||
# ratio means the working set does not fit in shared_buffers, so backends
|
||||
# evict dirty pages themselves and the same pages are written out
|
||||
# repeatedly instead of coalescing in memory.
|
||||
#
|
||||
# The recorder touches a wide set of pages continuously (~30 state
|
||||
# changes/sec across states, state_attributes and five indexes), which is
|
||||
# exactly the workload that punishes a small buffer pool.
|
||||
#
|
||||
# Memory headroom checked before raising: the instance was using 383Mi of
|
||||
# a 1Gi limit with 256MB of that being shared_buffers, so the limit goes
|
||||
# to 1.5Gi to keep the same margin.
|
||||
shared_buffers: "512MB"
|
||||
max_connections: "50"
|
||||
# The rest of this block is tuned to MINIMISE WRITE VOLUME, which is the
|
||||
# entire reason for the migration. Defaults would give a correct database
|
||||
@@ -87,4 +101,4 @@ spec:
|
||||
cpu: 200m
|
||||
memory: 512Mi
|
||||
limits:
|
||||
memory: 1Gi
|
||||
memory: 1536Mi
|
||||
|
||||
@@ -313,7 +313,9 @@ spec:
|
||||
configMap:
|
||||
name: mosquitto-config
|
||||
- name: auth
|
||||
emptyDir: {}
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 16Mi
|
||||
- name: tls
|
||||
secret:
|
||||
secretName: mosquitto-tls
|
||||
|
||||
@@ -344,9 +344,13 @@ spec:
|
||||
configMap:
|
||||
name: zigbee2mqtt-interview-watch-script
|
||||
- name: interview-watch-state
|
||||
emptyDir: {}
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 16Mi
|
||||
- name: discovery-guard-script
|
||||
configMap:
|
||||
name: zigbee2mqtt-discovery-guard-script
|
||||
- name: discovery-guard-state
|
||||
emptyDir: {}
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 16Mi
|
||||
|
||||
@@ -43,6 +43,25 @@ spec:
|
||||
shared_buffers: "512MB"
|
||||
max_wal_size: "2GB"
|
||||
wal_compression: "on"
|
||||
# Write-reduction tuning. These databases are largely idle, and the
|
||||
# Postgres defaults favour throughput over storage that wears out.
|
||||
#
|
||||
# checkpoint_timeout is the lever. The 5min default pushes every dirty
|
||||
# page to disk 288 times a day, so a small database whose same few pages
|
||||
# are re-dirtied constantly (sessions, tokens, job state) gets rewritten
|
||||
# over and over. Measured on ak-pg: 25.8M checkpoint buffers flushed
|
||||
# (~206 GB) against a 1046 MB database -- the whole thing roughly twice a
|
||||
# day. At 15min those pages coalesce and are written a third as often.
|
||||
#
|
||||
# max_wal_size must rise with it, or checkpoints trigger on size instead
|
||||
# and the timeout change does nothing. It also bounds crash recovery to
|
||||
# at most 2GB of WAL replay.
|
||||
#
|
||||
# NOT changed: synchronous_commit stays at the default `on`. It is off for
|
||||
# ha-pg because a state recorder is telemetry; these hold credentials,
|
||||
# documents and job state, where losing the last few hundred ms of commits
|
||||
# on an unclean shutdown is not an acceptable trade.
|
||||
checkpoint_timeout: "15min"
|
||||
resources:
|
||||
requests:
|
||||
cpu: 500m
|
||||
|
||||
@@ -147,7 +147,9 @@ spec:
|
||||
- key: immich-config.json
|
||||
path: immich-config.json
|
||||
- name: le-ca
|
||||
emptyDir: {}
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 16Mi
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
|
||||
@@ -26,32 +26,8 @@ spec:
|
||||
description: "The Ceph cluster has held HEALTH_WARN for six hours, so it is not a self-clearing blip. Get the detail with `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail`."
|
||||
|
||||
- alert: CephHealthError
|
||||
# HEALTH_ERR used to mean "data availability or redundancy is at
|
||||
# risk". Ceph 19.2.6 broke that assumption: it added the
|
||||
# AUTH_INSECURE_* cephx checks, two of them ERR-severity, which flag
|
||||
# the legacy `aes` key type every keyring in this cluster still uses.
|
||||
# That is a security-posture finding, not a durability one — see
|
||||
# CephAuthInsecureKeys below, which is what actually tracks it.
|
||||
#
|
||||
# The problem with leaving `ceph_health_status == 2` alone is not
|
||||
# noise, it is blindness: while the auth checks hold the cluster at
|
||||
# HEALTH_ERR, a genuine PG_AVAILABILITY or OSD_FULL is
|
||||
# indistinguishable from them, because the aggregate is already
|
||||
# saturated. So page on the ERR checks that are *not* auth ones.
|
||||
#
|
||||
# Written as "unless every active ERR check is an AUTH_INSECURE_ one"
|
||||
# rather than a direct match so that it fails open: if
|
||||
# ceph_health_detail goes missing or Ceph renames a check, both counts
|
||||
# come back empty, the `unless` removes nothing, and this still pages.
|
||||
# The `> 0` matters — the mgr exports all 31 known checks, the
|
||||
# inactive ones at 0, so an unfiltered count would compare constants.
|
||||
expr: |
|
||||
ceph_health_status == 2
|
||||
unless on (cluster) (
|
||||
count by (cluster) (ceph_health_detail{severity="HEALTH_ERR"} > 0)
|
||||
==
|
||||
count by (cluster) (ceph_health_detail{severity="HEALTH_ERR", name=~"AUTH_INSECURE_.*"} > 0)
|
||||
)
|
||||
# HEALTH_ERR means data availability or redundancy is at risk.
|
||||
expr: ceph_health_status == 2
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
@@ -59,22 +35,6 @@ spec:
|
||||
summary: "Ceph is HEALTH_ERR"
|
||||
description: "The Ceph cluster is in HEALTH_ERR — availability or redundancy is at risk. Inspect immediately with `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail` and `ceph osd df tree`."
|
||||
|
||||
- alert: CephAuthInsecureKeys
|
||||
# Carries the condition CephHealthError deliberately stops paging for,
|
||||
# so excluding it there narrows the alert rather than silencing it.
|
||||
# Warning, not critical: nothing is at risk of being lost, but this
|
||||
# does not self-clear either — it needs a planned cephx key rotation
|
||||
# across client.admin, the CSI provisioner/node keys and the RGW key.
|
||||
# Rotating those carelessly breaks RBD/CephFS provisioning
|
||||
# cluster-wide, so this stays a human decision, not a 3am one.
|
||||
expr: count(ceph_health_detail{name=~"AUTH_INSECURE_.*"} > 0) > 0
|
||||
for: 1h
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Ceph cephx keys use insecure key types"
|
||||
description: "Ceph reports {{ $value }} active AUTH_INSECURE_* health checks — cephx entities still using the legacy `aes` key type, flagged since the 19.2.6 upgrade. No data is at risk. Enumerate with `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail` and plan rotation against the 19.2.6 release notes; durable settings belong in `spec.cephConfig` in kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml."
|
||||
|
||||
- alert: CephMonDiskLow
|
||||
# Named explicitly rather than left to CephHealthWarn, because the
|
||||
# mons sit on the node's /var alongside container storage: this
|
||||
|
||||
@@ -64,7 +64,9 @@ spec:
|
||||
secret:
|
||||
secretName: pve-exporter-config
|
||||
- name: tmp
|
||||
emptyDir: {}
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 16Mi
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
|
||||
@@ -34,6 +34,27 @@ spec:
|
||||
parameters:
|
||||
shared_buffers: "256MB"
|
||||
max_connections: "200"
|
||||
# Write-reduction tuning. These databases are largely idle, and the
|
||||
# Postgres defaults favour throughput over storage that wears out.
|
||||
#
|
||||
# checkpoint_timeout is the lever. The 5min default pushes every dirty
|
||||
# page to disk 288 times a day, so a small database whose same few pages
|
||||
# are re-dirtied constantly (sessions, tokens, job state) gets rewritten
|
||||
# over and over. Measured on ak-pg: 25.8M checkpoint buffers flushed
|
||||
# (~206 GB) against a 1046 MB database -- the whole thing roughly twice a
|
||||
# day. At 15min those pages coalesce and are written a third as often.
|
||||
#
|
||||
# max_wal_size must rise with it, or checkpoints trigger on size instead
|
||||
# and the timeout change does nothing. It also bounds crash recovery to
|
||||
# at most 2GB of WAL replay.
|
||||
#
|
||||
# NOT changed: synchronous_commit stays at the default `on`. It is off for
|
||||
# ha-pg because a state recorder is telemetry; these hold credentials,
|
||||
# documents and job state, where losing the last few hundred ms of commits
|
||||
# on an unclean shutdown is not an acceptable trade.
|
||||
checkpoint_timeout: "15min"
|
||||
max_wal_size: "2GB"
|
||||
wal_compression: "on"
|
||||
# See ak-pg cluster.yaml for the rationale on synchronous replication.
|
||||
synchronous:
|
||||
method: any
|
||||
|
||||
@@ -218,7 +218,9 @@ spec:
|
||||
persistentVolumeClaim:
|
||||
claimName: paperless-consume
|
||||
- name: bridge-cert
|
||||
emptyDir: {}
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 16Mi
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
|
||||
@@ -33,6 +33,26 @@ spec:
|
||||
shared_buffers: "128MB"
|
||||
max_connections: "50"
|
||||
wal_compression: "on"
|
||||
# Write-reduction tuning. These databases are largely idle, and the
|
||||
# Postgres defaults favour throughput over storage that wears out.
|
||||
#
|
||||
# checkpoint_timeout is the lever. The 5min default pushes every dirty
|
||||
# page to disk 288 times a day, so a small database whose same few pages
|
||||
# are re-dirtied constantly (sessions, tokens, job state) gets rewritten
|
||||
# over and over. Measured on ak-pg: 25.8M checkpoint buffers flushed
|
||||
# (~206 GB) against a 1046 MB database -- the whole thing roughly twice a
|
||||
# day. At 15min those pages coalesce and are written a third as often.
|
||||
#
|
||||
# max_wal_size must rise with it, or checkpoints trigger on size instead
|
||||
# and the timeout change does nothing. It also bounds crash recovery to
|
||||
# at most 2GB of WAL replay.
|
||||
#
|
||||
# NOT changed: synchronous_commit stays at the default `on`. It is off for
|
||||
# ha-pg because a state recorder is telemetry; these hold credentials,
|
||||
# documents and job state, where losing the last few hundred ms of commits
|
||||
# on an unclean shutdown is not an acceptable trade.
|
||||
checkpoint_timeout: "15min"
|
||||
max_wal_size: "2GB"
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
|
||||
@@ -362,6 +362,22 @@ spec:
|
||||
name: trmnl-renderer-schedule
|
||||
- name: tmp
|
||||
emptyDir:
|
||||
# RAM-backed, NOT disk. Without `medium: Memory` an emptyDir lands
|
||||
# on the node's filesystem, and this one was the single largest
|
||||
# writer in the cluster: 249 GB over 7.3 days = ~34 GB/day onto
|
||||
# /dev/vda5, more than every Ceph OSD and every database combined
|
||||
# (measured 2026-08-20 from the container's cgroup io.stat).
|
||||
#
|
||||
# The cause is cadence, not volume. schedules.template.json renders
|
||||
# every 60 s, so Chrome is launched 1440 times a day and each launch
|
||||
# creates, fills and deletes a /tmp/puppeteer_dev_chrome_profile-*
|
||||
# directory. At rest /tmp holds under 8 MB; it is pure churn for
|
||||
# data that never outlives the render.
|
||||
#
|
||||
# tmpfs pages count against the container's memory limit, which is
|
||||
# why the cap stays at 512Mi: measured usage is 129Mi against a 2Gi
|
||||
# limit, so even the full cap leaves ample headroom.
|
||||
medium: Memory
|
||||
sizeLimit: 512Mi
|
||||
---
|
||||
apiVersion: v1
|
||||
|
||||
@@ -56,6 +56,27 @@ spec:
|
||||
parameters:
|
||||
shared_buffers: "128MB"
|
||||
max_connections: "50"
|
||||
# Write-reduction tuning. These databases are largely idle, and the
|
||||
# Postgres defaults favour throughput over storage that wears out.
|
||||
#
|
||||
# checkpoint_timeout is the lever. The 5min default pushes every dirty
|
||||
# page to disk 288 times a day, so a small database whose same few pages
|
||||
# are re-dirtied constantly (sessions, tokens, job state) gets rewritten
|
||||
# over and over. Measured on ak-pg: 25.8M checkpoint buffers flushed
|
||||
# (~206 GB) against a 1046 MB database -- the whole thing roughly twice a
|
||||
# day. At 15min those pages coalesce and are written a third as often.
|
||||
#
|
||||
# max_wal_size must rise with it, or checkpoints trigger on size instead
|
||||
# and the timeout change does nothing. It also bounds crash recovery to
|
||||
# at most 2GB of WAL replay.
|
||||
#
|
||||
# NOT changed: synchronous_commit stays at the default `on`. It is off for
|
||||
# ha-pg because a state recorder is telemetry; these hold credentials,
|
||||
# documents and job state, where losing the last few hundred ms of commits
|
||||
# on an unclean shutdown is not an acceptable trade.
|
||||
checkpoint_timeout: "15min"
|
||||
max_wal_size: "2GB"
|
||||
wal_compression: "on"
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
|
||||
@@ -156,4 +156,6 @@ spec:
|
||||
# each pod enrolls its own peer and a restart re-enrolls cleanly
|
||||
# (the old peer is reaped).
|
||||
- name: state
|
||||
emptyDir: {}
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 16Mi
|
||||
|
||||
@@ -164,7 +164,9 @@ spec:
|
||||
name: rook-config-override
|
||||
optional: true
|
||||
- name: ceph-config
|
||||
emptyDir: {}
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 16Mi
|
||||
tolerations:
|
||||
- key: "node.kubernetes.io/unreachable"
|
||||
operator: "Exists"
|
||||
|
||||
@@ -85,4 +85,6 @@ spec:
|
||||
name: restore-test-script
|
||||
defaultMode: 0755
|
||||
- name: tmp
|
||||
emptyDir: {}
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 16Mi
|
||||
|
||||
Reference in New Issue
Block a user