perf(grafana-db): raise archive_timeout to 30min on gf-pg

A WAL segment is a fixed 16 MiB file. archive_timeout forces a segment
switch on a timer so the archive never trails reality by more than that
interval, and the switched-out segment is written and archived at its full
16 MiB no matter how few bytes of real WAL it holds. CNPG defaults this to
5min, which permits up to 288 segments/day = 4.6 GB/day per cluster.

Measured on gf-pg: the database is 21 MB and generates 4.5 MB/day of actual
WAL, while its RBD volume takes 211 MB/hour -- against 192 MB/hour predicted
from 12 switches of 16 MiB. Essentially every byte written to this volume is
padding, roughly 1100x amplification.

This resolves an anomaly that has been open since the start of the endurance
work: gf-pg appeared to write gigabytes per day for a 21 MB database, and
neither WAL volume nor data changes could account for it. Segment padding
does, exactly.

Applied to gf-pg alone as a staged rollout. It is the lowest-risk cluster
and gives the cleanest signal -- its real WAL is small enough that any change
in volume is unambiguously attributable to this parameter. The same default
affects all seven clusters and is worth an estimated ~24 GB/day of logical
writes fleet-wide, but that only goes ahead once this one is verified.

The cost is RPO. On total loss of the cluster, up to 30min of Grafana state
becomes unrecoverable instead of 5min. That state is dashboards, annotations
and preferences, changing at ~4.5 MB/day, so the window is nearly always
empty. Clusters holding credentials or job state get evaluated separately
rather than inheriting this value by default.

archive_timeout is in CNPG's GlobalDefaultSettings, not its
FixedConfigurationParameters, so it is user-overridable -- verified against
pkg/postgres/configuration.go rather than assumed.

Context: drive-endurance work on three QLC NVMe OSDs at 81-87% wear.

.woodpecker/alert-monitor.yml

@@ -42,7 +42,7 @@ when:
 
 steps:
   - name: fetch-alerts
-    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-05-01
+    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-08-10
     environment:
       SOPS_AGE_KEY:
         from_secret: sops_age_key
@@ -135,8 +135,38 @@ steps:
         TITLES=$$(jq -c '[.[].title]' .ci-alert-monitor/open-issues.json)
         NOISE_BODY=$$(jq -r 'map(select(.title == "Alert monitor: noise log")) | .[0].body // ""' \
           .ci-alert-monitor/open-issues.json)
-        FPS=$$(printf '%s' "$$NOISE_BODY" | grep -oE '\b[0-9a-f]{16}\b' \
+        # Suppression entries are exactly the structured lines the write-back
+        # step emits: "- <fp> <ALERTNAME> — <YYYY-MM-DD>: <reason>". Parse THAT
+        # shape rather than grepping the whole body for 16-hex tokens. A reason
+        # routinely cites another alert's fingerprint in prose ("see the issue
+        # filed for fingerprint <fp>"), and the body-wide grep turned every such
+        # citation into a silent, permanent suppression of an alert that was
+        # never classified as noise — verified against the live noise log, which
+        # yielded 22 fingerprints for 21 real entries, the extra one being
+        # 2ddc41edb689fc33, an alert that HAD been filed as a genuine issue.
+        #
+        # Entries also expire after NOISE_TTL_DAYS. A noise verdict says "this
+        # occurrence was benign", not "this alert can never be interesting
+        # again" — but the fingerprint is sha256(alertname|scope) and therefore
+        # stable across recurrences, so without an expiry a single blip mutes
+        # that alertname+scope pair forever. Re-triaging a recurrence weeks
+        # later is cheap; missing a real outage because a July blip muted
+        # openbao's KubePodNotReady is not.
+        #
+        # Lines that fail to parse are simply dropped, i.e. they stop
+        # suppressing — this fails toward MORE alerting, never less.
+        #
+        # date is BusyBox here, not GNU: `date -d '30 days ago'` errors out, so
+        # the cutoff goes through epoch arithmetic and `date -d @<epoch>`, which
+        # this image does support.
+        NOISE_TTL_DAYS=$${NOISE_TTL_DAYS:-30}
+        CUTOFF=$$(date -u -d @$$(( $$(date -u +%s) - NOISE_TTL_DAYS * 86400 )) +%F)
+        FPS=$$(printf '%s\n' "$$NOISE_BODY" \
+          | sed -nE 's/^- ([0-9a-f]{16}) [^ ]+ — ([0-9]{4}-[0-9]{2}-[0-9]{2}):.*/\2 \1/p' \
+          | awk -v c="$$CUTOFF" 'BEGIN{gsub(/-/,"",c)} { d=$$1; gsub(/-/,"",d); if (d+0 >= c+0) print $$2 }' \
           | jq -R -s -c 'split("\n") | map(select(length > 0))')
+        echo "Noise suppressions in effect: $$(printf '%s' "$${FPS:-[]}" | jq length)" \
+             "(TTL $${NOISE_TTL_DAYS}d, entries dated >= $$CUTOFF)"
         # Synthesize pseudo-alerts for BLIPS: lookback series whose label set
         # is not currently firing at all (checked against the PRE-filter raw
         # snapshot, so live-but-null-routed alerts don't reappear as blips).
@@ -260,7 +290,7 @@ steps:
               '{body: ($$old + "\n" + $$add)}' > /tmp/gitea-patch.json
             gitea_write PATCH "$$API/issues/$$NOISE_NUM" /tmp/gitea-patch.json
           else
-            printf '%s\n\n' "Fingerprints the alert-monitor triage agent classified as noise. The dedup step skips any alert whose fingerprint appears below — delete a line to make that alert eligible for triage again. Keep this issue open; it is the suppression state." > /tmp/noise-header.txt
+            printf '%s\n\n' "Fingerprints the alert-monitor triage agent classified as noise. The dedup step skips any alert whose fingerprint appears below — delete a line to make that alert eligible for triage again. Keep this issue open; it is the suppression state. Entries EXPIRE after NOISE_TTL_DAYS (default 30) counted from the date in the line, so a one-off blip stops suppressing its alertname+scope pair once the window passes; re-add happens automatically if it is triaged as noise again. Only lines in the exact '- <fingerprint> <ALERTNAME> — <YYYY-MM-DD>: <reason>' form suppress anything — a fingerprint mentioned in prose does not." > /tmp/noise-header.txt
             jq -n --rawfile hdr /tmp/noise-header.txt --rawfile add /tmp/noise-lines.txt \
               '{title: "Alert monitor: noise log", body: ($$hdr + $$add)}' > /tmp/gitea-create.json
             gitea_write POST "$$API/issues" /tmp/gitea-create.json

.woodpecker/ci-manifests-test.yml

@@ -60,7 +60,7 @@ steps:
   # run that never touched them. Tracked in #461.
   - name: kustomize-build
     depends_on: [] # see note above `steps:`
-    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-07-30
+    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-08-10
     commands:
       - |
         set -eu
@@ -115,7 +115,7 @@ steps:
   # An earlier revision of this comment also asserted that several deploy-*.yml
   # files emit "Additional property timeout is not allowed" warnings. That was
   # observed under the v3.9.0 CLI this step used to pin, and it is NOT known to
-  # still hold on v3.16.0 — review raised the likelihood that the warning was
+  # still hold on current CLIs — review raised the likelihood that the warning was
   # schema drift in a 7-minor-old CLI rather than real debt in those files, which
   # is one of the reasons this pin moved to latest stable. It cannot be confirmed
   # from a green run either, because the loop below sends lint output to /tmp/out
@@ -123,9 +123,15 @@ steps:
   # The claim is dropped rather than restated: what this step guarantees is the
   # exit-code contract above, which is what the gate actually rests on.
   # Latest stable, per the standing rule that any software uses the latest
-  # stable version. Matching the deployed woodpecker-server (v3.16.0) is a
-  # CONSEQUENCE of that, not a separate requirement — the server is bumped by
-  # image-automation within the 3.x range, so both land on the same release.
+  # stable version. Matching the deployed woodpecker-server is a CONSEQUENCE of
+  # that, not a separate requirement — the server is bumped by image-automation
+  # within the 3.x range, so both should land on the same release.
+  #
+  # They do not land together, though: the server is a Kubernetes manifest and
+  # is bumped automatically, this pipeline file is not and cannot be (see
+  # infrastructure/image-automation/README.md). So the server moves first and
+  # this trails until someone bumps it by hand — which is what happened at
+  # v3.16.0 -> v3.17.0, caught by the update audit (#426) rather than by CI.
   #
   # It matters here because the CLI compiles the config against its own schema:
   # one behind the server can pass a config the server then rejects, or reject
@@ -152,7 +158,7 @@ steps:
   # whenever it trails the deployed server. If you bump the server, bump here.
   - name: woodpecker-lint
     depends_on: [] # see note above `steps:`
-    image: woodpeckerci/woodpecker-cli:v3.16.0-alpine
+    image: woodpeckerci/woodpecker-cli:v3.17.0-alpine
     commands:
       - |
         set -eu
@@ -240,7 +246,7 @@ steps:
   # this step fails permanently on its own documentation.
   - name: no-digest-pins
     depends_on: [] # see note above `steps:`
-    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-07-30
+    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-08-10
     commands:
       - |
         set -eu

.woodpecker/e2e-test.yml

@@ -4,7 +4,7 @@ when:
 
 steps:
   - name: e2e
-    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-05-01
+    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-08-10
     environment:
       SOPS_AGE_KEY:
         from_secret: sops_age_key

.woodpecker/e2e.yml

@@ -30,7 +30,7 @@ steps:
       - sh scripts/ci/test-model-selection.sh
 
   - name: e2e-test
-    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-05-01
+    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-08-10
     environment:
       SOPS_AGE_KEY:
         from_secret: sops_age_key

.woodpecker/issue-fix.yml

@@ -49,7 +49,7 @@ steps:
     # (the issue-pick logic needs jq, which only the ci-claude image has);
     # minting a 1h read-only token per run is cheap. Degrades to
     # snapshot-only if the ci-diagnostics ServiceAccount isn't reconciled.
-    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-05-01
+    image: gitea.renner.uno/gitea_admin/ci-k8s:2026-08-10
     environment:
       SOPS_AGE_KEY:
         from_secret: sops_age_key

kubernetes/apps/audiobookshelf/deployment.yaml

@@ -121,7 +121,9 @@ spec:
           persistentVolumeClaim:
             claimName: audiobookshelf-books
         - name: le-ca
-          emptyDir: {}
+          emptyDir:
+            medium: Memory
+            sizeLimit: 16Mi
 ---
 apiVersion: v1
 kind: Service

kubernetes/apps/authentik/cluster.yaml

@@ -43,6 +43,30 @@ spec:
     parameters:
       shared_buffers: "256MB"
       max_connections: "400"
+      # Write-reduction tuning. These databases are largely idle, and the
+      # Postgres defaults favour throughput over storage that wears out.
+      #
+      # checkpoint_timeout is the lever. The 5min default pushes every dirty
+      # page to disk 288 times a day, so a small database whose same few pages
+      # are re-dirtied constantly (sessions, tokens, job state) gets rewritten
+      # over and over. Measured on ak-pg: 25.8M checkpoint buffers flushed
+      # (~206 GB) against a 1046 MB database -- the whole thing roughly twice a
+      # day. At 15min those pages coalesce and are written a third as often.
+      #
+      # max_wal_size must rise with it, or checkpoints trigger on size instead
+      # and the timeout change does nothing. It also bounds crash recovery to
+      # at most 2GB of WAL replay.
+      #
+      # NOT changed: synchronous_commit stays at the default `on`. It is off for
+      # ha-pg because a state recorder is telemetry; these hold credentials,
+      # documents and job state, where losing the last few hundred ms of commits
+      # on an unclean shutdown is not an acceptable trade.
+      checkpoint_timeout: "15min"
+      max_wal_size: "2GB"
+      # zstd, not "on": the boolean spelling resolves to pglz (2016-era, weaker
+      # and slower than the lz4/zstd this build also accepts). Compresses the
+      # full-page images that dominate WAL volume after each checkpoint.
+      wal_compression: "zstd"
     # Async replication makes CNPG switchover slow because the standby has
     # to chase missing WAL via barman-cloud and replay queued records
     # before it can promote. With one synchronous standby, the standby is

kubernetes/apps/grafana-db/postgres.yaml

@@ -29,6 +29,52 @@ spec:
     parameters:
       shared_buffers: "128MB"
       max_connections: "50"
+      # Write-reduction tuning. These databases are largely idle, and the
+      # Postgres defaults favour throughput over storage that wears out.
+      #
+      # checkpoint_timeout is the lever. The 5min default pushes every dirty
+      # page to disk 288 times a day, so a small database whose same few pages
+      # are re-dirtied constantly (sessions, tokens, job state) gets rewritten
+      # over and over. Measured on ak-pg: 25.8M checkpoint buffers flushed
+      # (~206 GB) against a 1046 MB database -- the whole thing roughly twice a
+      # day. At 15min those pages coalesce and are written a third as often.
+      #
+      # max_wal_size must rise with it, or checkpoints trigger on size instead
+      # and the timeout change does nothing. It also bounds crash recovery to
+      # at most 2GB of WAL replay.
+      #
+      # NOT changed: synchronous_commit stays at the default `on`. It is off for
+      # ha-pg because a state recorder is telemetry; these hold credentials,
+      # documents and job state, where losing the last few hundred ms of commits
+      # on an unclean shutdown is not an acceptable trade.
+      checkpoint_timeout: "15min"
+      max_wal_size: "2GB"
+      # zstd, not "on": the boolean spelling resolves to pglz (2016-era, weaker
+      # and slower than the lz4/zstd this build also accepts). Compresses the
+      # full-page images that dominate WAL volume after each checkpoint.
+      wal_compression: "zstd"
+      # THE dominant write source on this cluster, by three orders of magnitude.
+      #
+      # A WAL segment is a fixed 16 MiB file. archive_timeout forces a segment
+      # switch on a timer so the archive never lags reality by more than that
+      # interval -- and the switched-out segment is written and archived at its
+      # full 16 MiB regardless of how few bytes of real WAL it contains.
+      #
+      # CNPG defaults this to 5min, i.e. up to 288 segments/day = 4.6 GB/day.
+      # Measured here: this database is 21 MB and generates 4.5 MB/day of actual
+      # WAL, while its RBD volume takes 211 MB/hour. Essentially every byte
+      # written is padding -- roughly 1100x amplification.
+      #
+      # 30min cuts that to 48 segments/day. The cost is RPO: on total loss of
+      # the cluster, up to 30min of Grafana state is unrecoverable rather than
+      # 5min. Grafana's state here is dashboards, annotations and preferences,
+      # and the write rate says it changes ~4.5 MB/day -- so that window is
+      # nearly always empty anyway.
+      #
+      # Applied to gf-pg FIRST, alone, as a staged rollout: it is the lowest-risk
+      # cluster and has the cleanest signal, since its real WAL is small enough
+      # that any change in volume is unambiguously this parameter.
+      archive_timeout: "30min"
   resources:
     requests:
       cpu: 100m

kubernetes/apps/home-assistant/MIGRATION-recorder-postgres.md

@@ -0,0 +1,179 @@
+# Home Assistant recorder: SQLite → Postgres
+
+## Why
+
+The recorder's SQLite database is the single largest client writer in the
+cluster. Measured 2026-08-19 (cgroup `io.stat` on the home-assistant container,
+corroborated by kernel diskstats and Ceph pool counters):
+
+| | |
+|---|---|
+| HA writes to its RBD volume | **40 GB/day** |
+| share of all client writes | **~26%** |
+| actual row data underneath | **~5 GB/day** |
+
+The 8× gap is the write *pattern*, not the volume:
+
+```
+~30 state changes/sec  (107k rows/hour, 2.57M/day)
+  ×2   SQLite WAL + checkpoint — every dirtied page written twice
+  ×4   ext4 journal + commit, because synchronous=FULL fsyncs every commit
+       (measured: 819 KB of 4 KiB writes → 3.28 MB at the block device)
+  = ~15 KB written per recorded state change
+```
+
+Postgres replaces scattered fsync'd page rewrites with sequential batched WAL.
+This attacks the multiplier rather than trimming what gets recorded — excluding
+entities would hide the problem, not fix it.
+
+**Context:** this is one of three write-reduction changes. The other two are
+merged and verified — `size 3→2` on the Ceph pools (#519) and
+`mon_compact_on_trim=false` (#520, which alone removed ~228 GB/day). See #434.
+
+## What is at risk
+
+`states` is disposable — `purge_keep_days: 7`, so it regenerates in a week.
+
+**`statistics` is not.** As of 2026-08-19:
+
+| table | rows | note |
+|---|---|---|
+| `statistics` | 10,793,712 | **spans 2024-03-13 → today, 2.4 years** |
+| `statistics_short_term` | 1,902,624 | 5-minute buckets |
+| `statistics_meta` | 1,448 | entity ↔ id mapping, required by both |
+| `states` | 18,289,190 | disposable, 7-day retention |
+| `events` | 483,989 | disposable |
+
+**Home Assistant does not migrate data between database engines.** Repointing
+`db_url` at an empty Postgres silently starts from zero and the energy dashboard
+loses its entire history. The long-term statistics are irreplaceable — they are
+not in InfluxDB, which HA does not read for the energy dashboard.
+
+## Phase 1 — deploy the cluster (this PR)
+
+`ha-pg` comes up **empty** and nothing consumes it. HA stays on SQLite, so a
+failure here is inert and there is nothing to roll back.
+
+Verify before going further:
+
+```sh
+kubectl -n home-assistant get cluster ha-pg          # Cluster in healthy state, 1/1
+kubectl -n home-assistant get backup                 # ha-pg-daily Completed (immediate: true)
+kubectl -n home-assistant logs ha-pg-1 -c postgres | grep -i archive   # WAL archiving, no errors
+```
+
+Confirm the backup actually landed in the bucket rather than trusting the CR
+status — the WordPress backup failed nightly for three days against a bucket
+that was declared but never created (see `opentofu/object_storage.tf`).
+
+## Phase 2b — bulk statistics copy: DONE 2026-08-19
+
+Completed with **HA running** — the SQLite file was read through a read-only
+mount, and SQLite's WAL mode gives readers a consistent snapshot, so no downtime
+was needed for the bulk copy.
+
+```
+statistics_meta:        sqlite=1449       pg=1449       OK
+statistics:             sqlite=10799904   pg=10799904   OK   (153.9s)
+statistics_short_term:  sqlite=1969804    pg=1969804    OK   ( 27.0s)
+
+statistics span in Postgres: 2024-03-13 -> 2026-08-19
+entities with long-term history: 1449
+rows carrying sum (energy dashboard): 2627859
+database size: 2252 MB   (PVC is 20Gi)
+```
+
+`ANALYZE` was run on all three tables. That is not housekeeping: after a bulk
+`COPY` the planner has no statistics, and HA queries these tables constantly for
+history and energy views, so without it every view seq-scans 10.8M rows.
+
+Two schema details that cost a failed first attempt, recorded so the top-up does
+not repeat them:
+
+- `has_mean` / `has_sum` are INTEGER in SQLite and BOOLEAN in Postgres. Postgres
+  will not coerce; convert explicitly and preserve NULL.
+- The legacy `created` / `start` / `last_reset` DATETIME columns are 100% NULL in
+  both bulk tables (superseded by the `_ts` variants), which is what makes a
+  tab-separated `COPY` safe — nothing in those tables needs escaping.
+
+The schema itself was created with HA's own models
+(`Base.metadata.create_all`), **not** hand-written DDL, and `schema_changes` was
+left EMPTY on purpose. HA's `_initialize_database` detects a new database by
+looking for an index on `events.time_fired_ts`, then writes `SCHEMA_VERSION`
+itself. Pre-stamping that row would skip HA's own initialisation path.
+
+### High-water marks for the cutover top-up
+
+Rows written after the bulk copy are not yet in Postgres. Resume from:
+
+```
+statistics             max_id=13323904    max_start_ts=1787162400.0
+statistics_short_term  max_id=159205762   max_start_ts=1787166000.0
+```
+
+## Phase 2c — cutover (REMAINING)
+
+Do not start until phase 1 is verified.
+
+**Migrate statistics only.** Copying all 18M `states` rows is slow, needs a long
+HA outage, and buys a table that is discarded within 7 days anyway.
+
+1. **Snapshot first.** Velero backup of the HA PVC, plus a `.backup` copy of
+   `home-assistant_v2.db`. This is the rollback.
+2. **Stop the recorder** — scale `deploy/home-assistant` to 0. The database must
+   be quiescent; copying a live SQLite file gives a torn read.
+3. **Create the schema.** Start HA once against the empty Postgres so it runs
+   its own migrations. Never hand-create the tables — the schema version must
+   match the HA version exactly or HA will refuse to start or silently migrate.
+4. **Copy** `statistics_meta` → `statistics` → `statistics_short_term`, in that
+   order (the other two carry `metadata_id` foreign keys into the first).
+   Sequences must be reset afterwards, or the first insert collides on a
+   duplicate primary key:
+   ```sql
+   SELECT setval('statistics_id_seq', (SELECT MAX(id) FROM statistics));
+   SELECT setval('statistics_meta_id_seq', (SELECT MAX(id) FROM statistics_meta));
+   SELECT setval('statistics_short_term_id_seq', (SELECT MAX(id) FROM statistics_short_term));
+   ```
+5. **Repoint** `recorder.db_url` in the `homeassistant` repo — that change lives
+   there, not here. The `recorder_db_url` secret it references is already
+   rendered into `secrets.yaml` by the `render-db-url` init container in
+   `deployment.yaml` (phase 2a), so this order is mandatory: **the init
+   container must be deployed first**, or HA refuses to start on a missing
+   secret.
+
+   The init container percent-encodes the credentials and validates the
+   resulting YAML rather than counting lines. That is not incidental — the CNPG
+   password is random, and a naive `"postgresql://user:pass@host/db"` string
+   makes `secrets.yaml` unparseable the moment the password contains a quote,
+   which takes out *every* secret HA loads, not just this one. Verified against
+   a deliberately hostile password before merge.
+   ```yaml
+   recorder:
+     db_url: !secret recorder_db_url    # postgresql://homeassistant:<pw>@ha-pg-rw.home-assistant.svc.cluster.local/homeassistant
+   ```
+   The password is in the CNPG-generated `ha-pg-app` Secret. It must not be
+   committed — render it into `secrets.yaml` the same way the Z2M MQTT password
+   is handled in `zigbee2mqtt-deployment.yaml`.
+6. **Verify** the energy dashboard renders history back to 2024, not just today.
+   That is the check that proves the statistics migration worked; "HA starts and
+   records" does not.
+
+### Rollback
+
+Revert `db_url` to the SQLite path and restore the `.backup` copy. Keep the
+SQLite file until the energy dashboard has been confirmed over a full day —
+deleting it early removes the only rollback that preserves the 2.4 years.
+
+## After the migration
+
+- The HA config PVC shrinks from 5.5 GB to a few hundred MB. Do not shrink the
+  PVC itself: `ceph-block` allows expansion, not reduction.
+- Re-measure. `kubectl -n home-assistant exec deploy/home-assistant -c
+  home-assistant -- cat /sys/fs/cgroup/io.stat` was 40 GB/day before. If it does
+  not drop substantially, the migration did not achieve its purpose and that
+  should be recorded rather than assumed.
+- `synchronous_commit: off` in `ha-pg-cluster.yaml` is the single biggest
+  contributor to the reduction. It is safe for telemetry (Postgres stays crash
+  consistent; at most a few hundred ms of commits are lost on unclean shutdown)
+  but it is a deliberate trade and should not be "corrected" later without
+  understanding why it is set.

kubernetes/apps/home-assistant/deployment.yaml

@@ -36,6 +36,64 @@ spec:
       # (sml2mqtt bridge owns the USB device on edge), so it no longer needs
       # the edge USB device and schedules on the general cluster pool. This
       # gets the heavy HA pod (8Gi) off the soon-to-be-tiny edge box.
+      initContainers:
+        # Puts CNPG's connection URI into /config/secrets.yaml as
+        # `recorder_db_url`, for the recorder's move off SQLite
+        # (MIGRATION-recorder-postgres.md).
+        #
+        # HA has no way to read a db_url from the environment — `!env_var` was
+        # removed from its YAML loader — so `!secret` is the only mechanism, and
+        # secrets.yaml is PVC state that config-sync deliberately does not
+        # manage. Hence a small init container rather than a env: entry.
+        #
+        # The URI comes STRAIGHT from the ha-pg-app Secret. CNPG builds and
+        # percent-encodes it, so there is nothing to escape here: the value is
+        # restricted to [A-Za-z0-9:/@._%-], verified against the live secret.
+        # An earlier version of this assembled the URI from user/password/dbname
+        # and needed percent-encoding plus YAML round-trip validation to be
+        # safe. That was solving a problem CNPG had already solved.
+        #
+        # Re-rendered on every pod start, so a CNPG password rotation takes
+        # effect rather than silently leaving a stale URI behind.
+        #
+        # INERT until the recorder is repointed: nothing reads this key until
+        # `db_url: !secret recorder_db_url` lands in configuration.yaml in the
+        # homeassistant repo. That ordering is required — HA refuses to start
+        # when a referenced secret is missing.
+        - name: render-db-url
+          image: busybox:1.36
+          command:
+            - sh
+            - -c
+            - |
+              set -eu
+              S=/config/secrets.yaml
+              T=/config/.secrets.render.tmp
+              [ -f "$S" ] || : > "$S"
+              # Filter-then-append, never rewrite in place: this file holds
+              # every other HA secret (influxdb_token and friends).
+              grep -v '^recorder_db_url:' "$S" > "$T" || true
+              printf 'recorder_db_url: "%s"\n' "$DB_URI" >> "$T"
+              # Refuse to shrink the file. Cheap guard against the one failure
+              # that would actually hurt: losing the other keys.
+              [ "$(wc -l < "$T")" -ge "$(wc -l < "$S")" ] || {
+                echo "FATAL: rewrite would shrink secrets.yaml; aborting" >&2; exit 1; }
+              chmod 600 "$T"
+              mv "$T" "$S"
+              echo "recorder_db_url rendered"
+          env:
+            - name: DB_URI
+              valueFrom:
+                secretKeyRef: { name: ha-pg-app, key: fqdn-uri }
+          volumeMounts:
+            - name: config
+              mountPath: /config
+          resources:
+            requests:
+              cpu: 10m
+              memory: 16Mi
+            limits:
+              memory: 32Mi
       containers:
         - name: home-assistant
           image: ghcr.io/home-assistant/home-assistant:2026.8.2 # {"$imagepolicy": "flux-system:home-assistant"}
@@ -221,9 +279,13 @@ spec:
         # Writable emptyDir for the nginx entrypoint's envsubst output
         # (/etc/nginx/conf.d/default.conf rendered from the .template).
         - name: proxy-conf-d
-          emptyDir: {}
+          emptyDir:
+            medium: Memory
+            sizeLimit: 16Mi
         # code-server's settings/extensions cache. Not on the config PVC
         # so installed extensions don't pollute HA's /config listing.
         # Trade-off: extensions reset across pod restarts.
         - name: vscode-state
-          emptyDir: {}
+          emptyDir:
+            medium: Memory
+            sizeLimit: 16Mi

kubernetes/apps/home-assistant/ha-pg-cluster.yaml

@@ -0,0 +1,118 @@
+---
+# Postgres backend for the Home Assistant recorder.
+#
+# WHY THIS EXISTS
+# The recorder's SQLite database is the single largest client writer in the
+# cluster. Measured 2026-08-19 (cgroup io.stat on the home-assistant container,
+# corroborated by kernel diskstats): 40 GB/day to its RBD volume, ~26% of all
+# client writes, against only ~5 GB/day of actual row data. The gap is the write
+# PATTERN, not the volume:
+#
+#   ~30 state changes/sec (107k rows/hour, 2.57M/day)
+#     x2  SQLite WAL + checkpoint — every dirtied page is written twice
+#     x4  ext4 journal + commit, because synchronous=FULL fsyncs every commit
+#         (measured: 819 KB of 4 KiB writes -> 3.28 MB at the block device)
+#     = ~15 KB written per recorded state change
+#
+# Postgres replaces scattered fsync'd page rewrites with sequential, batched WAL.
+# That is the point of the move — it attacks the multiplier rather than trimming
+# what gets recorded, which would only hide the problem.
+#
+# STAGING
+# Phase 1 (this manifest): the cluster comes up EMPTY and HA keeps using SQLite.
+# Nothing depends on it, so a failure here is inert. Verify Ready, WAL archiving
+# to Hetzner, and a successful base backup before going further.
+# Phase 2: migrate the statistics tables and repoint recorder.db_url.
+# See MIGRATION-recorder-postgres.md for the sequence and the rollback.
+apiVersion: postgresql.cnpg.io/v1
+kind: Cluster
+metadata:
+  name: ha-pg
+  namespace: home-assistant
+spec:
+  instances: 1
+  imageName: ghcr.io/cloudnative-pg/postgresql:16
+  storage:
+    storageClass: ceph-block
+    # Sized for the migrated statistics plus 7 days of states at the current
+    # rate: statistics 10.8M rows + statistics_short_term 1.9M (~1.8 GB with
+    # indexes in Postgres) and ~18M states rows for the purge_keep_days:7 window
+    # (~4 GB). 20Gi leaves room for autovacuum bloat and index churn.
+    # ceph-block has allowVolumeExpansion: true, so this can grow in place.
+    size: 20Gi
+  plugins:
+    - name: barman-cloud.cloudnative-pg.io
+      isWALArchiver: true
+      parameters:
+        barmanObjectName: ha-pg-store
+  bootstrap:
+    initdb:
+      # Explicit, not left to CNPG's `app` default: the generated ha-pg-app
+      # Secret and the -app Service point at whatever is named here, and the
+      # recorder db_url in phase 2 must match. Same trap that made the wp-pg
+      # namespace migration point credentials at an empty database (#433).
+      database: homeassistant
+      owner: homeassistant
+  postgresql:
+    parameters:
+      # 512MB, not the 256MB this shipped with. Measured after 15h on Postgres:
+      # buffers_backend was 629,704 of 939,025 total page writes — 67% written
+      # by backend processes rather than the checkpointer or bgwriter. That
+      # ratio means the working set does not fit in shared_buffers, so backends
+      # evict dirty pages themselves and the same pages are written out
+      # repeatedly instead of coalescing in memory.
+      #
+      # The recorder touches a wide set of pages continuously (~30 state
+      # changes/sec across states, state_attributes and five indexes), which is
+      # exactly the workload that punishes a small buffer pool.
+      #
+      # Memory headroom checked before raising: the instance was using 383Mi of
+      # a 1Gi limit with 256MB of that being shared_buffers, so the limit goes
+      # to 1.5Gi to keep the same margin.
+      shared_buffers: "512MB"
+      max_connections: "50"
+      # The rest of this block is tuned to MINIMISE WRITE VOLUME, which is the
+      # entire reason for the migration. Defaults would give a correct database
+      # that writes far more than it needs to on replicated Ceph storage.
+      #
+      # synchronous_commit off: the recorder is telemetry, not transactional
+      # record-keeping. This does NOT risk corruption — Postgres stays crash
+      # consistent; it only means the last few hundred ms of commits can be lost
+      # on an unclean shutdown, which for state history is a non-event. In
+      # exchange every commit stops forcing an fsync, which is precisely the
+      # cost that makes SQLite expensive here.
+      synchronous_commit: "off"
+      # Compress full-page images in WAL. Recorder writes are many small row
+      # inserts scattered across pages, so full-page writes after each
+      # checkpoint dominate WAL volume; compression cuts that directly.
+      #
+      # Measured on this cluster: full-page images are 62.4% of WAL bytes, so
+      # this is the single largest lever on recorder write volume.
+      #
+      # zstd, not "on": the boolean spelling resolves to pglz, the original
+      # 2016-era algorithm, which is both slower and weaker than the two
+      # alternatives PostgreSQL 16 accepts here (lz4, zstd). zstd gives the
+      # best ratio on 8 KiB page images, and the CPU cost lands on a cluster
+      # that is IO-bound on QLC NVMe, not CPU-bound.
+      wal_compression: "zstd"
+      # Fewer checkpoints => fewer full-page writes, since a page is only
+      # fully-logged on its first modification after a checkpoint. Doubling the
+      # interval halves how often each hot page pays that 8 KiB toll.
+      #
+      # Recovery cost is bounded by how much WAL a single interval produces,
+      # not by the interval itself: at the measured rate 30min is well under
+      # 200MB of replay, i.e. seconds. max_wal_size stays 2GB precisely so the
+      # timeout stays the binding trigger -- if size were the binding one, the
+      # interval change would do nothing.
+      checkpoint_timeout: "30min"
+      max_wal_size: "2GB"
+      min_wal_size: "512MB"
+      # full_page_writes is deliberately LEFT ON (the default). RBD can tear a
+      # page across a 4 MiB object boundary, and turning this off would trade
+      # write volume for the risk of an unrecoverable database. Not worth it.
+  resources:
+    requests:
+      cpu: 200m
+      memory: 512Mi
+    limits:
+      memory: 1536Mi

kubernetes/apps/home-assistant/ha-pg-hetzner-creds-externalsecret.yaml

@@ -0,0 +1,38 @@
+---
+# Barman S3 credentials for ha-pg.
+#
+# FOLLOW-UP (#397): sourced from seed/paperless/hetzner-creds, which is not where
+# they belong. woodpecker/hetzner-creds-externalsecret.yaml has the identical
+# problem and the same note. The values are the same today, so this is correct
+# but coupled: rotating paperless's Hetzner credentials would silently change
+# the backup credentials for Home Assistant and Woodpecker too.
+#
+# The fix is one dedicated path per consumer (seed/home-assistant/hetzner-creds)
+# and a one-line change to the `key:` below. Creating it needs an OpenBao write:
+#
+#   bao kv put homelab/seed/home-assistant/hetzner-creds \
+#     @<(bao kv get -format=json homelab/seed/paperless/hetzner-creds | jq '.data.data')
+#
+# Note the mount is `homelab`, so the path is homelab/seed/... — #397 records a
+# 403 on `bao kv list seed`, which was a wrong-path error rather than a
+# permissions problem.
+#
+# The namespace allowlist on the openbao ClusterSecretStore
+# (infrastructure/post-crds/openbao-clustersecretstore.yaml) must include this
+# namespace or ESO silently never syncs (#505). home-assistant is already
+# allowlisted — it has several ExternalSecrets in this directory already.
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: hetzner-s3-creds
+  namespace: home-assistant
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: openbao
+  target:
+    name: hetzner-s3-creds
+  dataFrom:
+    - extract:
+        key: seed/paperless/hetzner-creds

kubernetes/apps/home-assistant/ha-pg-objectstore.yaml

@@ -0,0 +1,35 @@
+---
+# Barman object store for ha-pg: base backups + continuous WAL archiving to
+# Hetzner Object Storage, same target bucket as the other CNPG clusters.
+#
+# Its own prefix. Two clusters archiving WAL to one destinationPath interleave
+# their streams and corrupt both — see the note in
+# kubernetes/apps/woodpecker/objectstore.yaml, which learned that the hard way.
+apiVersion: barmancloud.cnpg.io/v1
+kind: ObjectStore
+metadata:
+  name: ha-pg-store
+  namespace: home-assistant
+spec:
+  configuration:
+    destinationPath: s3://renner-uno-k8s-cnpg/ha-pg
+    endpointURL: https://fsn1.your-objectstorage.com
+    s3Credentials:
+      accessKeyId:
+        name: hetzner-s3-creds
+        key: ACCESS_KEY_ID
+      secretAccessKey:
+        name: hetzner-s3-creds
+        key: ACCESS_SECRET_KEY
+      region:
+        name: hetzner-s3-creds
+        key: REGION
+    wal:
+      compression: gzip
+    data:
+      compression: gzip
+  # Matches the 30d retention the other CNPG clusters use and the
+  # renner-uno-k8s-cnpg bucket's noncurrent-version expiry in
+  # opentofu/object_storage.tf, so old S3 versions don't outlive the backups
+  # that superseded them.
+  retentionPolicy: "30d"

kubernetes/apps/home-assistant/ha-pg-scheduled-backup.yaml

@@ -0,0 +1,23 @@
+---
+# Daily base backups for ha-pg. Continuous WAL archiving runs separately
+# (ha-pg-cluster.yaml spec.plugins[].isWALArchiver); without periodic base
+# backups the prefix would have no recovery point of its own, and
+# ha-pg-store's retentionPolicy: 30d would have nothing to prune against, so the
+# WAL chain would grow without bound.
+#
+# 02:50 UTC, offset from the other CNPG clusters (02:30, wp-pg at 02:40) so they
+# do not all hit the Hetzner bucket at once.
+apiVersion: postgresql.cnpg.io/v1
+kind: ScheduledBackup
+metadata:
+  name: ha-pg-daily
+  namespace: home-assistant
+spec:
+  schedule: "0 50 2 * * *"
+  immediate: true
+  backupOwnerReference: self
+  cluster:
+    name: ha-pg
+  method: plugin
+  pluginConfiguration:
+    name: barman-cloud.cloudnative-pg.io

kubernetes/apps/home-assistant/kustomization.yaml

@@ -52,6 +52,14 @@ resources:
   # CNP ingress rule.
   - zigbee2mqtt-gate-service.yaml
   - zigbee2mqtt-ingress.yaml
+  # Postgres backend for the recorder (MIGRATION-recorder-postgres.md).
+  # PHASE 1: this cluster comes up EMPTY and nothing consumes it — HA is still
+  # on SQLite. Deploying it is therefore inert; phase 2 migrates the statistics
+  # tables and repoints recorder.db_url in the homeassistant repo.
+  - ha-pg-hetzner-creds-externalsecret.yaml
+  - ha-pg-objectstore.yaml
+  - ha-pg-cluster.yaml
+  - ha-pg-scheduled-backup.yaml
 configMapGenerator:
   # nginx config for the reverse-proxy sidecar. Hash suffix disabled so
   # the cm name stays stable; editing proxy.conf does not auto-roll the
@@ -79,5 +87,13 @@ configMapGenerator:
     namespace: home-assistant
     files:
       - watch.sh=zigbee2mqtt-interview-watch.sh
+  # Z2M discovery guard sidecar script (gitea issue #381). Hash suffix disabled
+  # (name stays stable), so editing zigbee2mqtt-discovery-guard.sh does not
+  # auto-roll the pod — restart manually with
+  # `kubectl rollout restart deploy/zigbee2mqtt -n home-assistant`.
+  - name: zigbee2mqtt-discovery-guard-script
+    namespace: home-assistant
+    files:
+      - guard.sh=zigbee2mqtt-discovery-guard.sh
 generatorOptions:
   disableNameSuffixHash: true

kubernetes/apps/home-assistant/mosquitto-bootstrap.yaml

@@ -373,8 +373,9 @@ spec:
 # Broker user `zigbee2mqtt` — the Zigbee2MQTT bridge (ZHA replacement). Scoped
 # to its own `zigbee2mqtt/#` subtree plus `write homeassistant/#` (so it can
 # publish HA MQTT-discovery configs, exactly like the `shellies` user) and
-# `read homeassistant/status` (Z2M re-publishes discovery when HA restarts —
-# ACL in mosquitto-deployment.yaml). Own credential so its lifecycle is
+# `read homeassistant/status` (HA's birth/will topic; on `online` Z2M
+# republishes cached device states, NOT discovery configs — ACL and the full
+# reasoning in mosquitto-deployment.yaml). Own credential so its lifecycle is
 # independent of the others. The Z2M init container reads this password and
 # renders it into /app/data/secret.yaml.
 apiVersion: batch/v1

kubernetes/apps/home-assistant/mosquitto-config.yaml

@@ -5,13 +5,31 @@ metadata:
   namespace: home-assistant
 data:
   mosquitto.conf: |
-    # No on-disk persistence. Retained messages + sessions are re-populated by
-    # clients on reconnect (evcc republishes state, Shellies re-send their
-    # announce/online, HA re-publishes discovery), so the broker keeps no
-    # durable state. This drops the Ceph PVC dependency (the broker starts even
-    # if Ceph is degraded) and lets it reschedule anywhere in seconds — no RWO
-    # volume to fence/detach/reattach. Trade-off: a broker restart briefly
-    # empties retained state until clients reconnect (self-healing).
+    # No on-disk persistence, and no PVC of any kind for this broker. The
+    # broker keeps no durable state, so it starts even if Ceph is degraded and
+    # reschedules anywhere in seconds — no RWO volume to fence/detach/reattach.
+    #
+    # Most clients re-populate their retained state on reconnect: evcc
+    # republishes state, Shellies re-send their announce/online, HA re-publishes
+    # its own discovery.
+    #
+    # Zigbee2MQTT is the EXCEPTION and does not self-heal. Its HA discovery
+    # configs are published retained, and retention is the only mechanism it
+    # relies on to survive a broker restart — verified against Z2M 2.12.1
+    # source: discover() runs only from start(), the MQTT reconnect path
+    # (lib/mqtt.ts onConnect) republishes bridge/state and nothing else, and the
+    # homeassistant/status "online" handler (lib/extension/homeassistant.ts
+    # onMQTTMessage) republishes bridge/state + cached device states but never
+    # the discovery configs. There is no bridge/request endpoint that forces a
+    # rediscovery. Upstream has declined to add one (Koenkk/zigbee2mqtt #27458,
+    # #22977, #16950, all closed NOT_PLANNED).
+    #
+    # Consequence: a mosquitto restart that is not followed by a Zigbee2MQTT
+    # restart leaves every Z2M entity unavailable in HA until Z2M is restarted
+    # (`kubectl rollout restart deploy/zigbee2mqtt -n home-assistant`). This is
+    # a live operational hazard, tracked in gitea issue #381 — it caused a ~14 h
+    # outage of every Z2M entity on 2026-07-23. Enabling persistence is NOT the
+    # answer: this broker deliberately has no PVC (see above).
     persistence false
     log_dest stdout
     log_type error

kubernetes/apps/home-assistant/mosquitto-deployment.yaml

@@ -25,7 +25,13 @@ spec:
     # a broker serving throughout an image bump / reschedule (clients on the old
     # pod just reconnect to the new one); maxSurge:1 permits the brief two-pod
     # overlap — two independent stateless brokers behind the LB, fine since there
-    # is no shared state and discovery self-heals.
+    # is no shared state.
+    #
+    # NOTE: connectivity survives the roll, but RETAINED STATE DOES NOT — the
+    # new pod starts with an empty retained store, so a roll is equivalent to a
+    # broker restart for every client that depends on retention. Zigbee2MQTT
+    # does not recover from that on its own; see mosquitto-config.yaml and
+    # gitea issue #381.
     type: RollingUpdate
     rollingUpdate:
       maxUnavailable: 0
@@ -64,7 +70,9 @@ spec:
           # Pinned tag, auto-bumped within the 2.x line by Flux image-automation
           # (infrastructure/image-automation/mosquitto.yaml). The marker tells
           # Flux where to write; patch/minor updates land as git commits and
-          # roll the broker (brief restart; discovery self-heals via HA).
+          # roll the broker. HA re-publishes its own discovery afterwards, but
+          # Zigbee2MQTT does not — an image bump therefore needs the same
+          # follow-up as any other broker restart (see mosquitto-config.yaml).
           image: eclipse-mosquitto:2.0.22 # {"$imagepolicy": "flux-system:mosquitto"}
           # The shim runs as root (image sets no USER) and prepares the
           # auth files for the broker, which drops to user mosquitto
@@ -148,8 +156,16 @@ spec:
               # Owns its own `zigbee2mqtt/#` subtree; may WRITE `homeassistant/#`
               # to publish HA MQTT-discovery configs (write-only there, like the
               # shellies user, to deny discovery-tree recon from a compromised
-              # bridge); and READs `homeassistant/status` so it re-publishes
-              # discovery when HA restarts (birth/will).
+              # bridge); and READs `homeassistant/status`, HA's birth/will topic.
+              #
+              # That read grant does NOT buy a discovery re-publish: on
+              # `online`, Z2M 2.12.1 republishes bridge/state and cached device
+              # states only (lib/extension/homeassistant.ts onMQTTMessage) —
+              # discover() runs solely from start(). The grant is still correct,
+              # because the cached-state republish is what makes HA show current
+              # values right after an HA restart. Restoring discovery after a
+              # BROKER restart is handled by the discovery-guard sidecar in
+              # zigbee2mqtt-deployment.yaml.
               user zigbee2mqtt
               topic readwrite zigbee2mqtt/#
               topic write homeassistant/#
@@ -297,7 +313,9 @@ spec:
           configMap:
             name: mosquitto-config
         - name: auth
-          emptyDir: {}
+          emptyDir:
+            medium: Memory
+            sizeLimit: 16Mi
         - name: tls
           secret:
             secretName: mosquitto-tls

kubernetes/apps/home-assistant/zigbee2mqtt-deployment.yaml

@@ -272,6 +272,64 @@ spec:
               memory: 16Mi
             limits:
               memory: 64Mi
+        # Re-runs Zigbee2MQTT's HA discovery after the broker loses its retained
+        # store. mosquitto runs `persistence false` with no PVC, so every broker
+        # restart wipes the retained discovery configs, and nothing on the Z2M
+        # side restores them — not reconnect, not HA's birth message, and there
+        # is no rediscovery endpoint. Without this, a broker restart leaves every
+        # Z2M entity unavailable in HA until someone restarts Z2M by hand (gitea
+        # issue #381; ~14 h outage on 2026-07-23). The full reasoning, including
+        # why bridge/groups is the sentinel and why the guard is tuned for
+        # sensitivity, is in the script header.
+        - name: discovery-guard
+          # Same image and rationale as interview-watch above: reused purely for
+          # its bundled mosquitto_sub/mosquitto_pub, nothing installed at
+          # runtime. Verified present in this exact image: sh, mosquitto_sub
+          # (incl. -C and -W), mosquitto_pub, date -u +%s, mkdir, cat.
+          # Deliberately NOT carrying a Flux $imagepolicy marker, so it does not
+          # drag the broker's image-automation policy along with it.
+          image: eclipse-mosquitto:2.0.22
+          imagePullPolicy: IfNotPresent
+          command: [sh, /script/guard.sh]
+          env:
+            # Reuses the Z2M broker credential: its ACL is `readwrite
+            # zigbee2mqtt/#`, which covers reading bridge/state/bridge/groups and
+            # writing bridge/request/restart. No new broker user needed.
+            - name: MQTT_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: mosquitto-zigbee2mqtt-credentials
+                  key: password
+            # Must track zigbee2mqtt-config.yaml, same as interview-watch.
+            - name: MQTT_HOST
+              value: mosquitto
+            - name: MQTT_USER
+              value: zigbee2mqtt
+            - name: BASE_TOPIC
+              value: zigbee2mqtt
+            - name: STATEDIR
+              value: /state
+          securityContext:
+            runAsNonRoot: true
+            runAsUser: 1883
+            runAsGroup: 1883
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop: [ALL]
+          volumeMounts:
+            - name: discovery-guard-script
+              mountPath: /script
+              readOnly: true
+            # Restart cooldown stamp. Its own emptyDir rather than the image's
+            # /tmp, so it stays writable as non-root.
+            - name: discovery-guard-state
+              mountPath: /state
+          resources:
+            requests:
+              cpu: 10m
+              memory: 16Mi
+            limits:
+              memory: 64Mi
       volumes:
         - name: data
           persistentVolumeClaim:
@@ -286,4 +344,13 @@ spec:
           configMap:
             name: zigbee2mqtt-interview-watch-script
         - name: interview-watch-state
-          emptyDir: {}
+          emptyDir:
+            medium: Memory
+            sizeLimit: 16Mi
+        - name: discovery-guard-script
+          configMap:
+            name: zigbee2mqtt-discovery-guard-script
+        - name: discovery-guard-state
+          emptyDir:
+            medium: Memory
+            sizeLimit: 16Mi

kubernetes/apps/home-assistant/zigbee2mqtt-discovery-guard.sh

@@ -0,0 +1,162 @@
+#!/bin/sh
+# Restores Home Assistant MQTT discovery after the broker loses its retained
+# store, which mosquitto does on every restart because it runs `persistence
+# false` and has no PVC (a deliberate, settled decision — see
+# mosquitto-config.yaml).
+#
+# WHY THIS EXISTS
+# Zigbee2MQTT publishes its HA discovery configs retained and relies on broker
+# retention alone to survive a broker restart. Verified against Z2M 2.12.1
+# source, nothing on the Z2M side restores them:
+#   - discover() runs only from HomeAssistant.start().
+#   - lib/mqtt.ts onConnect() republishes bridge/state and resubscribes; the
+#     republishRetainedTimer is armed only inside connect() and never re-armed
+#     on reconnect.
+#   - The homeassistant/status "online" handler republishes bridge/state plus
+#     cached device states, never the discovery configs.
+#   - There is no bridge/request endpoint that forces a rediscovery; upstream
+#     declined to add one (Koenkk/zigbee2mqtt #27458, #22977, #16950).
+# So a broker restart not followed by a Z2M restart leaves every Z2M entity
+# unavailable in HA indefinitely. That cost ~14 h of dead entities on
+# 2026-07-23 (gitea issue #381).
+#
+# DETECTION — why bridge/groups
+# bridge/state and the four bridge/{info,devices,groups,definitions} topics are
+# all retained, but only bridge/state is republished on reconnect (mqtt.ts
+# onConnect). So "bridge/state present + one of the others absent" means: Z2M is
+# talking to this broker right now, but nothing has re-run start() since the
+# retained store was wiped. That is exactly the broken state.
+#
+# The absent-half partner must not be re-publishable by ordinary runtime
+# events, or it self-heals and the guard goes blind while HA is still broken:
+#   bridge/devices     — republished on announce/join/leave/interview. Worst.
+#   bridge/definitions — republished on devicesChanged and OTA.
+#   bridge/info        — republished on devicesChanged, which reaches inbound
+#                        radio traffic via the deviceExposesChanged callback
+#                        handed to every fromZigbee converter (receive.ts).
+#   bridge/health      — self-heals on its own 10 min timer. Useless here.
+#   bridge/groups      — only re-published by deliberate admin actions: group
+#                        membership edits, group add/remove/rename, and scene
+#                        changes (which come solely from an explicit
+#                        <device>/set scene command, never an inbound report).
+# Hence bridge/groups. It is also safe with no groups defined: publishGroups()
+# always publishes, emitting `[]`, which is still a non-empty retained payload.
+#
+# DELIBERATELY NOT AND-ed with bridge/info being absent. That would cut false
+# positives but introduce false negatives, and the costs are wildly asymmetric:
+# a false positive is one ~60 s Z2M restart, a false negative is every Zigbee
+# entity dead until a human notices. This guard is tuned for sensitivity.
+#
+# Two known ways to get a spurious positive, both handled by CONFIRM_DELAY:
+#   - Startup race. bridge/state is published at mqtt.connect() but
+#     bridge/groups only at the end of BridgeExtension.start(), so the signature
+#     is legitimately true for a few seconds during every normal Z2M boot.
+#   - QoS 0. The bridge/* topics are published with no qos option, so they go
+#     out at-most-once and a publish can in principle be dropped.
+#
+# ACTION
+# Publishes bridge/request/restart, which is a full in-process restart: index.js
+# restart() -> stop(true) tears down every extension, then start() builds a new
+# Controller and re-runs HomeAssistant.start(), hence discover(). It is NOT a
+# herdsman-only restart. discover() republishes only configs that are missing or
+# whose payload changed, which after a wipe is all of them.
+#
+# Credentials: reuses the existing zigbee2mqtt broker user, whose ACL is
+# `readwrite zigbee2mqtt/#` — enough to read bridge/state and bridge/groups and
+# to write bridge/request/restart. No new broker user, and no read grant on
+# homeassistant/# (checking for the discovery configs directly would have needed
+# one, which the ACL denies on purpose).
+set -eu
+
+MQTT_HOST="${MQTT_HOST:-mosquitto}"
+MQTT_PORT="${MQTT_PORT:-1883}"
+MQTT_USER="${MQTT_USER:-zigbee2mqtt}"
+BASE_TOPIC="${BASE_TOPIC:-zigbee2mqtt}"
+# Poll cadence. Worst-case time-to-repair is CHECK_INTERVAL + CONFIRM_DELAY.
+CHECK_INTERVAL="${CHECK_INTERVAL:-300}"
+# Re-check gap before acting, to ride out a normal Z2M boot and a dropped QoS 0
+# publish. Must comfortably exceed a Z2M cold start.
+CONFIRM_DELAY="${CONFIRM_DELAY:-90}"
+# Floor between two restarts, so a Z2M that will not come back healthy is not
+# restarted in a loop.
+COOLDOWN="${COOLDOWN:-1800}"
+# How long to wait for a retained message to land before calling it absent.
+READ_TIMEOUT="${READ_TIMEOUT:-5}"
+# Hard ceiling on a single client invocation, as a backstop to the protocol-level
+# waits below. Every MQTT call here is bounded with `timeout`, because neither
+# client bounds a TCP connect that HANGS rather than being refused — and a stalled
+# connect is most likely precisely when this guard runs, around a broker restart.
+# Without it the loop would block forever and stop logging, which is exactly the
+# silent-failure mode the guard exists to prevent. Same reasoning and the same
+# PUB_TIMEOUT default as zigbee2mqtt-interview-watch.sh.
+#
+# NOT `mosquitto_pub -W`: that flag does not exist for mosquitto_pub (it is a
+# mosquitto_sub option) and makes every publish fail on a usage error.
+# mosquitto_sub keeps its -W, which bounds the wait for a message once CONNECTED;
+# the `timeout` wrapper is what covers the connect phase, so it is set above -W.
+PUB_TIMEOUT="${PUB_TIMEOUT:-10}"
+STATEDIR="${STATEDIR:-/state}"
+STAMP="$STATEDIR/last-restart"
+
+log() { echo "[$(date -u '+%Y-%m-%dT%H:%M:%SZ')] discovery-guard: $*"; }
+
+# Echoes the retained payload of $1, or nothing if the topic holds none.
+# A retained message is delivered immediately on subscribe, so the wait is only
+# ever paid when the topic is genuinely empty.
+retained() {
+  timeout "$((READ_TIMEOUT + 5))" mosquitto_sub -h "$MQTT_HOST" -p "$MQTT_PORT" \
+    -u "$MQTT_USER" -P "$MQTT_PASSWORD" \
+    -t "$1" -C 1 -W "$READ_TIMEOUT" 2>/dev/null || true
+}
+
+# True when Z2M is connected to this broker but has not re-run start() since the
+# retained store was last wiped.
+discovery_lost() {
+  case "$(retained "$BASE_TOPIC/bridge/state")" in
+    *online*) ;;
+    *) return 1 ;;
+  esac
+  [ -z "$(retained "$BASE_TOPIC/bridge/groups")" ]
+}
+
+in_cooldown() {
+  [ -f "$STAMP" ] || return 1
+  last=$(cat "$STAMP" 2>/dev/null || echo 0)
+  now=$(date -u +%s)
+  [ $((now - last)) -lt "$COOLDOWN" ]
+}
+
+mkdir -p "$STATEDIR"
+log "watching $BASE_TOPIC/bridge/groups on $MQTT_HOST:$MQTT_PORT" \
+    "(interval ${CHECK_INTERVAL}s, confirm ${CONFIRM_DELAY}s, cooldown ${COOLDOWN}s)"
+
+while true; do
+  sleep "$CHECK_INTERVAL"
+
+  discovery_lost || continue
+
+  log "bridge/groups missing while Z2M is online — retained store looks wiped;" \
+      "re-checking in ${CONFIRM_DELAY}s before acting"
+  sleep "$CONFIRM_DELAY"
+
+  if ! discovery_lost; then
+    log "recovered on its own, no action taken"
+    continue
+  fi
+
+  if in_cooldown; then
+    log "confirmed, but a restart was requested less than ${COOLDOWN}s ago —" \
+        "skipping. Z2M may be failing to come up; check its logs."
+    continue
+  fi
+
+  log "confirmed — requesting a Zigbee2MQTT restart to re-run discovery"
+  if timeout "$PUB_TIMEOUT" mosquitto_pub -h "$MQTT_HOST" -p "$MQTT_PORT" \
+       -u "$MQTT_USER" -P "$MQTT_PASSWORD" \
+       -t "$BASE_TOPIC/bridge/request/restart" -m '' 2>/dev/null; then
+    date -u +%s > "$STAMP"
+    log "restart requested"
+  else
+    log "failed to publish the restart request; will retry next cycle"
+  fi
+done

kubernetes/apps/immich/cluster.yaml

@@ -42,7 +42,29 @@ spec:
     parameters:
       shared_buffers: "512MB"
       max_wal_size: "2GB"
-      wal_compression: "on"
+      # zstd, not "on": the boolean spelling resolves to pglz (2016-era, weaker
+      # and slower than the lz4/zstd this build also accepts). Compresses the
+      # full-page images that dominate WAL volume after each checkpoint.
+      wal_compression: "zstd"
+      # Write-reduction tuning. These databases are largely idle, and the
+      # Postgres defaults favour throughput over storage that wears out.
+      #
+      # checkpoint_timeout is the lever. The 5min default pushes every dirty
+      # page to disk 288 times a day, so a small database whose same few pages
+      # are re-dirtied constantly (sessions, tokens, job state) gets rewritten
+      # over and over. Measured on ak-pg: 25.8M checkpoint buffers flushed
+      # (~206 GB) against a 1046 MB database -- the whole thing roughly twice a
+      # day. At 15min those pages coalesce and are written a third as often.
+      #
+      # max_wal_size must rise with it, or checkpoints trigger on size instead
+      # and the timeout change does nothing. It also bounds crash recovery to
+      # at most 2GB of WAL replay.
+      #
+      # NOT changed: synchronous_commit stays at the default `on`. It is off for
+      # ha-pg because a state recorder is telemetry; these hold credentials,
+      # documents and job state, where losing the last few hundred ms of commits
+      # on an unclean shutdown is not an acceptable trade.
+      checkpoint_timeout: "15min"
   resources:
     requests:
       cpu: 500m

kubernetes/apps/immich/server.yaml

@@ -147,7 +147,9 @@ spec:
               - key: immich-config.json
                 path: immich-config.json
         - name: le-ca
-          emptyDir: {}
+          emptyDir:
+            medium: Memory
+            sizeLimit: 16Mi
 ---
 apiVersion: v1
 kind: Service

kubernetes/apps/monitoring-rules/cnpg-podmonitors.yaml

@@ -73,3 +73,52 @@ spec:
       interval: 30s
   podTargetLabels:
     - role
+---
+# wp-pg, restored into the `woodpecker` namespace (issue #433). It has never
+# been scraped: `apps` had no PodMonitor for it AND its baseline network policy
+# never allowed the monitoring namespace, so `cnpg_collector_up` has series for
+# the four clusters above but not this one. The namespace move fixes the policy
+# half; without this PodMonitor there is still no scrape target at all.
+#
+# Deliberately not adding one for the cluster still in `apps`: it is retired in
+# stage 3, and a PodMonitor there would only be reconciled to be deleted again.
+apiVersion: monitoring.coreos.com/v1
+kind: PodMonitor
+metadata:
+  name: cnpg-wp-pg
+  namespace: woodpecker
+  labels:
+    app.kubernetes.io/name: cloudnative-pg
+    release: kube-prometheus-stack
+spec:
+  selector:
+    matchLabels:
+      cnpg.io/cluster: wp-pg
+  podMetricsEndpoints:
+    - port: metrics
+      interval: 30s
+  podTargetLabels:
+    - role
+---
+# ha-pg — Postgres backend for the Home Assistant recorder
+# (kubernetes/apps/home-assistant/ha-pg-cluster.yaml). Added with the cluster
+# rather than after it, so the migration has metrics from the first moment
+# instead of discovering the gap later, which is how wp-pg's scrape hole was
+# found (#433).
+apiVersion: monitoring.coreos.com/v1
+kind: PodMonitor
+metadata:
+  name: cnpg-ha-pg
+  namespace: home-assistant
+  labels:
+    app.kubernetes.io/name: cloudnative-pg
+    release: kube-prometheus-stack
+spec:
+  selector:
+    matchLabels:
+      cnpg.io/cluster: ha-pg
+  podMetricsEndpoints:
+    - port: metrics
+      interval: 30s
+  podTargetLabels:
+    - role

kubernetes/apps/monitoring-rules/kustomization.yaml

@@ -4,6 +4,8 @@ kind: Kustomization
 # come from the kube-prometheus-stack chart. Flux applies this Kustomization
 # only after `monitoring` is Ready (chart installed → CRDs present).
 resources:
+  - prometheusrule-ceph.yaml
+  - prometheusrule-cilium.yaml
   - prometheusrule-hetzner-backup.yaml
   - prometheusrule-loki.yaml
   - prometheusrule-netbird-availability.yaml
@@ -11,6 +13,10 @@ resources:
   - prometheusrule-netbird-ingress.yaml
   - prometheusrule-velero.yaml
   - prometheusrule-woodpecker-rotation.yaml
+  # CNPG backup health. Seven Postgres clusters ship to Hetzner and nothing
+  # alerted if that stopped — HetznerBackupEndpointDown only probes the S3
+  # endpoint, which proves reachability, not that anything was written.
+  - prometheusrule-cnpg-backup.yaml
   - alloy-servicemonitor.yaml
   - cnpg-podmonitors.yaml
   - home-assistant-servicemonitor.yaml

kubernetes/apps/monitoring-rules/prometheusrule-ceph.yaml

@@ -0,0 +1,56 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: ceph
+  namespace: monitoring
+  labels:
+    # kube-prometheus-stack's Prometheus resource selects on this label;
+    # without it the rule is created but never loaded into Prometheus.
+    release: kube-prometheus-stack
+spec:
+  groups:
+    - name: ceph.rules
+      rules:
+        - alert: CephHealthWarn
+          # ceph_health_status: 0 = HEALTH_OK, 1 = HEALTH_WARN, 2 = HEALTH_ERR.
+          # 6h is deliberately long. Ceph warns for self-clearing conditions —
+          # a slow BlueStore op during the Sunday restore sweep, a mon briefly
+          # below its disk threshold — and paging on those trains people to
+          # ignore the alert. What matters is a warning that does not clear.
+          expr: ceph_health_status == 1
+          for: 6h
+          labels:
+            severity: warning
+          annotations:
+            summary: "Ceph has been HEALTH_WARN for 6h"
+            description: "The Ceph cluster has held HEALTH_WARN for six hours, so it is not a self-clearing blip. Get the detail with `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail`."
+
+        - alert: CephHealthError
+          # HEALTH_ERR means data availability or redundancy is at risk.
+          expr: ceph_health_status == 2
+          for: 5m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Ceph is HEALTH_ERR"
+            description: "The Ceph cluster is in HEALTH_ERR — availability or redundancy is at risk. Inspect immediately with `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail` and `ceph osd df tree`."
+
+        - alert: CephMonDiskLow
+          # Named explicitly rather than left to CephHealthWarn, because the
+          # mons sit on the node's /var alongside container storage: this
+          # fires for a full node, not for Ceph growth. The mon's own data is
+          # ~106 MB, so 30% of a 60 GB shared filesystem says nothing about
+          # Ceph.
+          # Image GC does not clear this — measured on k8s-2, the reclaimable
+          # content store is a small fraction of /var/lib/containerd next to
+          # ~20 GB of overlayfs snapshots held by running containers. Look at
+          # what actually grew before reaching for a threshold.
+          expr: |
+            100 * node_filesystem_avail_bytes{mountpoint="/var"}
+              / node_filesystem_size_bytes{mountpoint="/var"} < 30
+          for: 30m
+          labels:
+            severity: warning
+          annotations:
+            summary: "{{ $labels.instance }} /var below 30% free — Ceph mon will warn"
+            description: "/var on {{ $labels.instance }} is under 30% free, which is Ceph's mon_data_avail_warn threshold, so the mon on this node will report MON_DISK_LOW. Find the actual consumer with `talosctl -n <node> usage /var/lib --depth 1` and then `/var/lib/containerd --depth 1` — note that overlayfs snapshots belong to running containers and are not reclaimable by image GC."

kubernetes/apps/monitoring-rules/prometheusrule-cilium.yaml

@@ -0,0 +1,44 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: cilium
+  namespace: monitoring
+  labels:
+    # kube-prometheus-stack's Prometheus resource selects on this label;
+    # without it the rule is created but never loaded into Prometheus.
+    release: kube-prometheus-stack
+spec:
+  groups:
+    - name: cilium.rules
+      # Both alerts are map-agnostic on purpose. cilium_bpf_map_pressure is
+      # emitted per map type — policy, conntrack, NAT, LB — distinguished by
+      # the map_name label, and running out of any of them degrades the
+      # datapath. The policy map is merely the one that has bitten us.
+      # Cilium only exports the metric for maps above
+      # policyMapPressureMetricsThreshold (chart default 0.1), so quiet maps
+      # produce no series rather than a flood of near-zero ones.
+      rules:
+        - alert: CiliumBPFMapPressureHigh
+          # cilium_bpf_map_pressure is used/capacity for the fullest map of
+          # that name on the agent. A full map is not a slow path — it is a
+          # hard failure: the agent cannot program the datapath, so endpoint
+          # regeneration fails and every new pod on the node hangs in sandbox
+          # creation until something frees space.
+          expr: cilium_bpf_map_pressure > 0.8
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Cilium BPF map {{ $labels.map_name }} is {{ $value | humanizePercentage }} full on {{ $labels.node }}"
+            description: "The BPF map {{ $labels.map_name }} on node {{ $labels.node }} has been over 80% full for 15 minutes. At 100% the Cilium agent can no longer program the datapath and new pods on this node fail sandbox creation. Raise the corresponding size in kubernetes/infrastructure/cilium/helmrelease.yaml (values.bpf) — the map is sized at creation, so the change needs an agent restart."
+
+        - alert: CiliumBPFMapFull
+          # Pressure reaches (and can exceed) 1.0 when the agent wants more
+          # entries than the map holds. Pods are already failing to start.
+          expr: cilium_bpf_map_pressure >= 1
+          for: 0m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Cilium BPF map {{ $labels.map_name }} is full on {{ $labels.node }}"
+            description: "The BPF map {{ $labels.map_name }} on node {{ $labels.node }} is at or over capacity. Endpoint regeneration is failing with 'no space left on device' and pods scheduled to this node will hang in ContainerCreating. Check `{namespace=\"kube-system\",pod=~\"cilium-.+\"} |= \"no space left on device\"` in Loki for the affected endpoints."

kubernetes/apps/monitoring-rules/prometheusrule-cnpg-backup.yaml

@@ -0,0 +1,78 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: cnpg-backup
+  namespace: monitoring
+  labels:
+    # kube-prometheus-stack's Prometheus resource selects on this label;
+    # without it the rule is created but never loaded into Prometheus.
+    release: kube-prometheus-stack
+spec:
+  groups:
+    - name: cnpg-backup.rules
+      rules:
+        # WHY THIS FILE EXISTS
+        # Seven Postgres clusters back up to Hetzner and NOTHING alerted if that
+        # stopped. The only related rule was HetznerBackupEndpointDown, which
+        # probes the S3 endpoint — it proves the bucket is reachable, not that
+        # anything was written to it. Same failure shape as the WordPress backup
+        # that failed nightly for three days against a bucket that was declared
+        # in tofu but never applied: the status surface looked fine because
+        # nothing checked the thing that mattered.
+
+        - alert: CNPGWalArchiveBacklog
+          # `ready` counts WAL segments sitting in pg_wal/archive_status waiting
+          # to be archived. Steady state is 0 across every cluster (verified
+          # 2026-08-19). A sustained backlog means the archiver is failing:
+          # point-in-time recovery is silently broken from that moment, and
+          # pg_wal keeps growing until it fills the PVC and takes Postgres down.
+          #
+          # This is the load-bearing alert of the two. It depends only on the
+          # instance-level collector, so unlike the backup-age metrics below it
+          # is unaffected by the missing Backup CRs (#522).
+          #
+          # Threshold: >5 segments sustained for 30m. Brief non-zero values are
+          # normal under write bursts — a segment is 16 MiB, so 5 is roughly
+          # 80 MiB of un-archived WAL, well below anything that threatens the
+          # volume but far above idle noise.
+          expr: cnpg_collector_pg_wal_archive_status{value="ready"} > 5
+          for: 30m
+          labels:
+            severity: warning
+          annotations:
+            summary: "CNPG WAL archiving is falling behind on {{ $labels.namespace }}/{{ $labels.pod }}"
+            description: "{{ $value }} WAL segments have been waiting to archive for over 30 minutes. Point-in-time recovery is broken for anything after the last archived segment, and pg_wal will grow until the PVC fills. Check the barman-cloud sidecar logs and the ObjectStore credentials: `kubectl -n {{ $labels.namespace }} logs {{ $labels.pod }} -c plugin-barman-cloud`."
+
+        - alert: CNPGBackupFailed
+          # cnpg_collector_last_failed_backup_timestamp is a unix timestamp, 0
+          # when no failure is recorded. Guard on > 0 so the epoch-zero default
+          # is not read as "failed in 1970".
+          expr: |
+            cnpg_collector_last_failed_backup_timestamp > 0
+            and
+            (time() - cnpg_collector_last_failed_backup_timestamp) < 86400
+          for: 5m
+          labels:
+            severity: warning
+          annotations:
+            summary: "CNPG backup failed in the last 24h on {{ $labels.namespace }}/{{ $labels.pod }}"
+            description: "A base backup for {{ $labels.namespace }}/{{ $labels.pod }} failed within the last 24 hours. Verify what is actually in the bucket rather than trusting CR status — see the barman-cloud-backup-list recipe in kubernetes/apps/home-assistant/MIGRATION-recorder-postgres.md."
+
+        # DELIBERATELY ABSENT: a backup-age alert, which is the one you would
+        # most want here.
+        #
+        # The natural expression is
+        #   (time() - cnpg_collector_last_available_backup_timestamp) > 48*3600
+        # and it CANNOT be used today: that metric reads exactly 0 on every
+        # instance in the cluster (verified 2026-08-19 across gf-pg, ak-pg-1/2,
+        # pprl-pg-2/3, im-pg, wp-pg, ha-pg). CNPG has no record of any
+        # successful backup anywhere, because the Backup CRs it derives that
+        # from are being deleted within a minute of completing — see #522.
+        #
+        # Backups themselves DO work; that was verified independently by
+        # listing the bucket with barman-cloud-backup-list and restoring a WAL
+        # segment back out of S3. The gap is in the record, not the data.
+        #
+        # Adding the rule now would fire permanently for all seven clusters and
+        # be silenced within a day, which is worse than not having it. Add it as
+        # part of resolving #522, once the metric reflects reality.

kubernetes/apps/monitoring-rules/prometheusrule-woodpecker-rotation.yaml

@@ -12,37 +12,37 @@ spec:
   groups:
     - name: woodpecker-rotation.rules
       # Convergence safety net for the automated WOODPECKER_AGENT_SECRET rotation
-      # (kubernetes/apps/woodpecker/rotate-agent-secret.yaml). That rotation Job is
+      # (kubernetes/infrastructure/openbao/rotation/woodpecker.yaml). That rotation Job is
       # fire-and-forget: it writes OpenBao and exits WITHOUT verifying ESO re-synced
       # or that both consumers restarted healthy. These alerts close that gap. ESO
       # metrics are not scraped in this cluster, so everything here is built on
-      # kube-state-metrics. The CronJob/Job series stay empty until the rotation
-      # feature (branch feat/woodpecker-secret-auto-rotation) is merged — harmless.
+      # kube-state-metrics. The rotation has since landed: the CronJob lives in the
+      # openbao namespace, not apps, which is where the last two rules look.
       rules:
         - alert: WoodpeckerServerNotReady
           # 0 ready replicas well past the 10m ESO refresh + Reloader restart cycle.
-          expr: kube_statefulset_status_replicas_ready{namespace="apps",statefulset="woodpecker-server"} == 0
+          expr: kube_deployment_status_replicas_ready{namespace="woodpecker",deployment="woodpecker-server"} == 0
           for: 20m
           labels:
             severity: critical
           annotations:
             summary: "Woodpecker CI server has no ready replica"
             description: >-
-              StatefulSet woodpecker-server (ns apps) has had 0 ready replicas for 20m;
-              all CI is down (agents cannot reach woodpecker.apps.svc:9000). If this
-              followed a secret rotation, the server pod failed to return after its
-              Reloader-triggered restart — check `kubectl -n apps describe pod
-              woodpecker-server-0` (PVC attach / crashloop) and the woodpecker-secret
-              ExternalSecret sync status.
+              Deployment woodpecker-server (ns woodpecker) has had 0 ready replicas
+              for 20m; all CI is down (agents cannot reach
+              woodpecker.woodpecker.svc:9000). If this followed a secret rotation, the
+              server pod failed to return after its Reloader-triggered restart — check
+              `kubectl -n woodpecker describe deploy woodpecker-server` and the
+              woodpecker-secret ExternalSecret sync status.
         - alert: WoodpeckerAgentNotReady
-          expr: kube_deployment_status_replicas_available{namespace="apps",deployment="woodpecker-agent"} == 0
+          expr: kube_deployment_status_replicas_available{namespace="woodpecker",deployment="woodpecker-agent"} == 0
           for: 20m
           labels:
             severity: warning
           annotations:
             summary: "Woodpecker CI agent has no available replica"
             description: >-
-              Deployment woodpecker-agent (ns apps) has had 0 available replicas for
+              Deployment woodpecker-agent (ns woodpecker) has had 0 available replicas for
               20m; no pipelines will run. If this followed a secret rotation, the agent
               did not return after its Reloader restart, or its WOODPECKER_AGENT_SECRET
               no longer matches the server's — check the agent pod logs for a gRPC auth
@@ -50,29 +50,29 @@ spec:
         - alert: WoodpeckerSecretRotationJobFailed
           # Named, clearer signal than the generic KubeJobFailed for the OpenBao-write
           # step. Fires while a failed rotation Job is retained (ttl 3d).
-          expr: kube_job_status_failed{namespace="apps",job_name=~"woodpecker-secret-rotate.*"} > 0
+          expr: kube_job_status_failed{namespace="openbao",job_name=~"woodpecker-secret-rotate.*"} > 0
           for: 15m
           labels:
             severity: warning
           annotations:
             summary: "Woodpecker agent-secret rotation Job failed"
             description: >-
-              A woodpecker-secret-rotate Job (ns apps) failed at the OpenBao-write step;
+              A woodpecker-secret-rotate Job (ns openbao) failed at the OpenBao-write step;
               the agent secret was NOT rotated. Most common cause: the one-time OpenBao
               `rotator` role/policy bootstrap was never applied (see
               kubernetes/infrastructure/openbao/policies/rotator.hcl). Check
-              `kubectl -n apps logs job/<name>`.
+              `kubectl -n openbao logs job/<name>`.
         - alert: WoodpeckerSecretRotationStale
           # Quarterly cadence (~91d) + grace. The series is ABSENT until the first
           # successful run, so this cannot false-fire before rotation has ever worked.
-          expr: (time() - kube_cronjob_status_last_successful_time{namespace="apps",cronjob="woodpecker-secret-rotate"}) > 100 * 24 * 3600
+          expr: (time() - kube_cronjob_status_last_successful_time{namespace="openbao",cronjob="woodpecker-secret-rotate"}) > 100 * 24 * 3600
           for: 1h
           labels:
             severity: warning
           annotations:
             summary: "Woodpecker agent secret has not rotated in >100 days"
             description: >-
-              The quarterly woodpecker-secret-rotate CronJob (ns apps) has not completed
+              The quarterly woodpecker-secret-rotate CronJob (ns openbao) has not completed
               successfully in over 100 days — rotation has silently stopped (CronJob
               suspended/deleted, or every run failing). Investigate the CronJob and its
               recent Jobs.

kubernetes/apps/monitoring/helmrelease-prometheus.yaml

@@ -402,6 +402,52 @@ spec:
         limits:
           memory: 256Mi
 
+    # Drop the apiserver's high-cardinality latency histograms.
+    #
+    # Measured 2026-08-20: Prometheus held 298,236 active series, of which
+    # ~108,000 — over a third — were apiserver/etcd `_bucket` histograms. Each
+    # histogram fans out to one series per bucket x verb x resource x scope, so
+    # a single metric like apiserver_request_duration_seconds_bucket was 23,810
+    # series on its own. For comparison, every Home Assistant metric combined is
+    # ~4,100.
+    #
+    # That cardinality is the dominant driver of Prometheus write volume
+    # (~11 GB/day of TSDB + WAL, doubled again by Ceph replication), and none of
+    # it is observable value here: per-bucket apiserver latency quantiles are for
+    # clusters with SLAs and dozens of nodes, not three.
+    #
+    # ONLY metrics with ZERO rule dependencies are dropped. Verified against the
+    # live rule set before writing this — each metric checked individually for
+    # references in alert and recording rules:
+    #
+    #   apiserver_request_duration_seconds_bucket        rules=0   23,810 series
+    #   apiserver_request_body_size_bytes_bucket         rules=0   23,200
+    #   etcd_request_duration_seconds_bucket             rules=0   15,840
+    #   apiserver_watch_list_duration_seconds_bucket     rules=0   11,538
+    #   apiserver_watch_cache_read_wait_seconds_bucket   rules=0    8,232
+    #   apiserver_response_sizes_bucket                  rules=0    7,592
+    #   apiserver_watch_events_sizes_bucket              rules=0    5,220
+    #
+    # apiserver_request_sli_duration_seconds_bucket is DELIBERATELY KEPT: 23
+    # recording rules depend on it, including the apiserver availability SLO and
+    # the KubeAPIErrorBudgetBurn burn-rate series. Dropping it would silently
+    # break that alerting, which is the failure mode this comment exists to
+    # prevent if someone later "tidies up" this regex.
+    #
+    # Only the `_bucket` series are dropped. The matching `_count` and `_sum`
+    # remain, so request rates and mean latencies still work — what is lost is
+    # per-bucket quantiles.
+    #
+    # etcd_* appears here even though kubeEtcd is disabled below: these are the
+    # apiserver's own etcd-client metrics, exposed on the apiserver endpoint, not
+    # a separate etcd scrape.
+    kubeApiServer:
+      serviceMonitor:
+        metricRelabelings:
+          - action: drop
+            sourceLabels: [__name__]
+            regex: (apiserver_request_duration_seconds_bucket|apiserver_request_body_size_bytes_bucket|etcd_request_duration_seconds_bucket|apiserver_watch_list_duration_seconds_bucket|apiserver_watch_cache_read_wait_seconds_bucket|apiserver_response_sizes_bucket|apiserver_watch_events_sizes_bucket)
+
     # Disable etcd/scheduler/controller-manager scraping (Talos doesn't expose them by default)
     kubeEtcd:
       enabled: false

kubernetes/apps/monitoring/pve-exporter-deployment.yaml

@@ -64,7 +64,9 @@ spec:
           secret:
             secretName: pve-exporter-config
         - name: tmp
-          emptyDir: {}
+          emptyDir:
+            medium: Memory
+            sizeLimit: 16Mi
 ---
 apiVersion: v1
 kind: Service

kubernetes/apps/paperless/MIGRATION-data-to-block.md

@@ -0,0 +1,97 @@
+# paperless-data: CephFS → RBD
+
+## Why
+
+`paperless-data` is `ReadWriteMany` on `ceph-filesystem`, but **only the paperless
+Deployment mounts it**. Verified 2026-08-20:
+
+```
+$ pods mounting paperless-data
+paperless-68cb8b8744-7hjb8      <- one, and only one
+
+paperless-data     ReadWriteMany  ceph-filesystem
+paperless-media    ReadWriteMany  ceph-filesystem   <- genuinely shared (ftp-scanner)
+paperless-consume  ReadWriteMany  ceph-filesystem   <- genuinely shared (ftp-scanner)
+```
+
+Its contents are single-writer state — the 50 MB Whoosh index,
+`classification_model.pickle`, `celerybeat-schedule.db`, `log/`,
+`migration_lock`, `protonmail.pem`. 75 files. Nothing else reads or writes them.
+
+It is on CephFS by default, not by requirement.
+
+## What it costs
+
+Every access goes through the MDS. Measured with **zero CI pipelines running**:
+
+```
+paperless-data client   request_load_avg 22   (all other sessions 0-1)
+MDS over 180s           4,052 client requests  = 22.5/s
+                        2,718 journal events   = 15/s
+                            2 dir commits      <- almost none: not file writes
+```
+
+So it is cap/inode churn from repeated access, not data being written. That is
+most of the constant **7-8 GB/day** written to `cephfs-metadata`, and since that
+pool is `size: 3`, roughly **21-24 GB/day** reaches the drives.
+
+`cephfs-metadata` deliberately stays at size 3. Losing it destroys the entire
+filesystem rather than the objects on one OSD, and the 275 GiB in `cephfs-data0`
+depends on it. Cutting the writes at source is correct; cutting the replication
+is not.
+
+## Sequence
+
+Staged deliberately: the PVC lands first and empty, so merging it changes
+nothing. The copy and the repoint are separate.
+
+### Phase 1 — provision (this PR)
+
+`paperless-data-block` (RWO, `ceph-block`, 10Gi) is created and mounted by
+nothing. Inert.
+
+Verify: `kubectl -n paperless get pvc paperless-data-block` → Bound.
+
+### Phase 2 — copy
+
+The Whoosh index must come across intact. Paperless *can* rebuild it
+(`document_index reindex`), but that reprocesses every document, so copying is
+both faster and less disruptive.
+
+1. Scale paperless to 0. The source must be quiescent — `celerybeat-schedule.db`
+   and the Whoosh index are both live-written, and copying them under an active
+   writer gives a torn read.
+2. Run a one-off pod that mounts BOTH PVCs and copies:
+   ```
+   cp -a /src/. /dst/
+   ```
+   The RWO target pins the pod to one node; the RWX source will follow.
+3. Verify file count and index size match:
+   ```
+   find /src -type f | wc -l   ==   find /dst -type f | wc -l
+   du -sh /src/index           ==   du -sh /dst/index
+   ```
+
+### Phase 3 — repoint
+
+Change the Deployment's `data` volume to `claimName: paperless-data-block`,
+merge, let it roll.
+
+Verify afterwards:
+- paperless starts, search returns results (proves the Whoosh index survived)
+- `ceph tell mds.cephfs-b session ls` no longer shows a busy `paperless-data`
+  client
+- `rate(ceph_pool_wr_bytes{pool_id="3"})` drops from its 7-8 GB/day baseline
+
+**Do not delete `paperless-data` until search has been confirmed working.** It is
+the rollback, and it is `prune: disabled` so Flux will not remove it.
+
+## Expected effect
+
+~7-8 GB/day off `cephfs-metadata`, so ~21-24 GB/day off the drives once
+replication is counted. Against ~432 GB/day cluster-wide that is roughly 5% —
+worth doing, but it is a correctness fix (a single-writer volume does not belong
+on a shared filesystem) more than a large win.
+
+It also removes 22 req/sec of MDS load, which benefits the workloads that
+genuinely need CephFS — Woodpecker CI workspaces share the same MDS.

kubernetes/apps/paperless/cluster.yaml

@@ -34,6 +34,30 @@ spec:
     parameters:
       shared_buffers: "256MB"
       max_connections: "200"
+      # Write-reduction tuning. These databases are largely idle, and the
+      # Postgres defaults favour throughput over storage that wears out.
+      #
+      # checkpoint_timeout is the lever. The 5min default pushes every dirty
+      # page to disk 288 times a day, so a small database whose same few pages
+      # are re-dirtied constantly (sessions, tokens, job state) gets rewritten
+      # over and over. Measured on ak-pg: 25.8M checkpoint buffers flushed
+      # (~206 GB) against a 1046 MB database -- the whole thing roughly twice a
+      # day. At 15min those pages coalesce and are written a third as often.
+      #
+      # max_wal_size must rise with it, or checkpoints trigger on size instead
+      # and the timeout change does nothing. It also bounds crash recovery to
+      # at most 2GB of WAL replay.
+      #
+      # NOT changed: synchronous_commit stays at the default `on`. It is off for
+      # ha-pg because a state recorder is telemetry; these hold credentials,
+      # documents and job state, where losing the last few hundred ms of commits
+      # on an unclean shutdown is not an acceptable trade.
+      checkpoint_timeout: "15min"
+      max_wal_size: "2GB"
+      # zstd, not "on": the boolean spelling resolves to pglz (2016-era, weaker
+      # and slower than the lz4/zstd this build also accepts). Compresses the
+      # full-page images that dominate WAL volume after each checkpoint.
+      wal_compression: "zstd"
     # See ak-pg cluster.yaml for the rationale on synchronous replication.
     synchronous:
       method: any

kubernetes/apps/paperless/paperless.yaml

@@ -208,9 +208,25 @@ spec:
             seccompProfile:
               type: RuntimeDefault
       volumes:
+        # paperless-data-block (RWO, ceph-block), NOT paperless-data
+        # (RWX, ceph-filesystem). Only this Deployment ever mounted the volume,
+        # and its contents are single-writer state — the Whoosh index,
+        # classification model, celerybeat schedule, logs. Being on CephFS meant
+        # every access went through the MDS: 22 requests/sec sustained with zero
+        # CI running, ~15 journal events/sec against 2 directory commits per
+        # 180s, i.e. cap/inode churn rather than file writes. That was most of
+        # the constant 7-8 GB/day on cephfs-metadata, and that pool is size 3.
+        #
+        # `consume` and `media` below STAY on CephFS: the ftp-scanner writes to
+        # those from another node, so they genuinely need RWX.
+        #
+        # paperless-data is retained (prune: disabled) as the rollback. Do not
+        # delete it until search has been confirmed working — that is what
+        # proves the Whoosh index survived the copy, not merely that paperless
+        # starts. Migration record: MIGRATION-data-to-block.md.
         - name: data
           persistentVolumeClaim:
-            claimName: paperless-data
+            claimName: paperless-data-block
         - name: media
           persistentVolumeClaim:
             claimName: paperless-media
@@ -218,7 +234,9 @@ spec:
           persistentVolumeClaim:
             claimName: paperless-consume
         - name: bridge-cert
-          emptyDir: {}
+          emptyDir:
+            medium: Memory
+            sizeLimit: 16Mi
 ---
 apiVersion: v1
 kind: Service

kubernetes/apps/paperless/pvcs.yaml

@@ -18,6 +18,45 @@ spec:
     requests:
       storage: 10Gi
 ---
+# Replacement for paperless-data, on RBD instead of CephFS.
+#
+# WHY: paperless-data is declared ReadWriteMany on ceph-filesystem, but only ONE
+# pod ever mounts it — the paperless Deployment itself. Its contents are all
+# single-writer state: the 50 MB Whoosh index, classification_model.pickle,
+# celerybeat-schedule.db, logs, migration_lock. Nothing shares it. Only `consume`
+# and `media` genuinely need RWX, because the FTP scanner writes to those from a
+# different node.
+#
+# COST OF LEAVING IT: every access goes through the MDS. Measured 2026-08-20 with
+# zero CI running, this volume's client was doing 22 MDS requests/sec sustained,
+# and the MDS logged ~15 journal events/sec against only 2 directory commits per
+# 180s — i.e. cap and inode churn from repeated access, not file writes. That is
+# most of the constant 7-8 GB/day written to cephfs-metadata, which is a size 3
+# pool, so ~21-24 GB/day actually hits the drives.
+#
+# cephfs-metadata deliberately STAYS at size 3 (see storageclass.yaml): losing
+# the metadata pool destroys the whole filesystem, not just the objects on one
+# OSD, so the 275 GiB in cephfs-data0 rides on it. Reducing the writes at source
+# is the right fix; reducing the replication is not.
+#
+# MIGRATION: this PVC is created EMPTY and nothing mounts it yet. The data copy
+# and the deployment repoint are a separate, staged step — see
+# MIGRATION-data-to-block.md. Merging this alone is inert.
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: paperless-data-block
+  namespace: paperless
+  annotations:
+    kustomize.toolkit.fluxcd.io/prune: disabled
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: ceph-block
+  resources:
+    requests:
+      storage: 10Gi
+---
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:

kubernetes/apps/trmnl/cluster.yaml

@@ -32,7 +32,30 @@ spec:
     parameters:
       shared_buffers: "128MB"
       max_connections: "50"
-      wal_compression: "on"
+      # zstd, not "on": the boolean spelling resolves to pglz (2016-era, weaker
+      # and slower than the lz4/zstd this build also accepts). Compresses the
+      # full-page images that dominate WAL volume after each checkpoint.
+      wal_compression: "zstd"
+      # Write-reduction tuning. These databases are largely idle, and the
+      # Postgres defaults favour throughput over storage that wears out.
+      #
+      # checkpoint_timeout is the lever. The 5min default pushes every dirty
+      # page to disk 288 times a day, so a small database whose same few pages
+      # are re-dirtied constantly (sessions, tokens, job state) gets rewritten
+      # over and over. Measured on ak-pg: 25.8M checkpoint buffers flushed
+      # (~206 GB) against a 1046 MB database -- the whole thing roughly twice a
+      # day. At 15min those pages coalesce and are written a third as often.
+      #
+      # max_wal_size must rise with it, or checkpoints trigger on size instead
+      # and the timeout change does nothing. It also bounds crash recovery to
+      # at most 2GB of WAL replay.
+      #
+      # NOT changed: synchronous_commit stays at the default `on`. It is off for
+      # ha-pg because a state recorder is telemetry; these hold credentials,
+      # documents and job state, where losing the last few hundred ms of commits
+      # on an unclean shutdown is not an acceptable trade.
+      checkpoint_timeout: "15min"
+      max_wal_size: "2GB"
   resources:
     requests:
       cpu: 100m

kubernetes/apps/trmnl/renderer.yaml

@@ -362,6 +362,22 @@ spec:
             name: trmnl-renderer-schedule
         - name: tmp
           emptyDir:
+            # RAM-backed, NOT disk. Without `medium: Memory` an emptyDir lands
+            # on the node's filesystem, and this one was the single largest
+            # writer in the cluster: 249 GB over 7.3 days = ~34 GB/day onto
+            # /dev/vda5, more than every Ceph OSD and every database combined
+            # (measured 2026-08-20 from the container's cgroup io.stat).
+            #
+            # The cause is cadence, not volume. schedules.template.json renders
+            # every 60 s, so Chrome is launched 1440 times a day and each launch
+            # creates, fills and deletes a /tmp/puppeteer_dev_chrome_profile-*
+            # directory. At rest /tmp holds under 8 MB; it is pure churn for
+            # data that never outlives the render.
+            #
+            # tmpfs pages count against the container's memory limit, which is
+            # why the cap stays at 512Mi: measured usage is 129Mi against a 2Gi
+            # limit, so even the full cap leaves ample headroom.
+            medium: Memory
             sizeLimit: 512Mi
 ---
 apiVersion: v1

kubernetes/apps/woodpecker/MIGRATION-namespace.md

@@ -0,0 +1,388 @@
+# Woodpecker: `apps` → `woodpecker` namespace migration
+
+Moves the Woodpecker service — server, agent and the `wp-pg` CNPG backend — out
+of the shared `apps` namespace into one of its own. Distinct from
+`MIGRATION.md`, which covers the earlier SQLite → CNPG move.
+
+## Why
+
+`apps` holds gitea, gatus, external-dns, ical-proxy and woodpecker together,
+and its `default-deny-ingress` uses `endpointSelector: {}`. Every allow rule it
+carries therefore applies to *every* workload in the namespace.
+
+That is not academic. Issue #433 — `wp-pg` stuck at `Ready=False` with
+`Instance Status Extraction Error` for three weeks — is caused by the CNPG
+operator in `database` being unable to reach `https://<pod-ip>:8000/pg/status`.
+The only namespace-level fix is `fromEndpoints: namespace=database`, which
+would expose gitea, gatus, external-dns and ical-proxy to the whole `database`
+namespace to solve a problem affecting one pod on one port.
+
+`cnpg_collector_up` has series for authentik, immich, monitoring and paperless
+but not for `wp-pg`, for the same reason: `apps` never allowed the monitoring
+namespace either.
+
+A namespace per service keeps each grant on the workload that needs it.
+
+**`woodpecker-ci` is not the target.** It enforces `pod-security: privileged`
+and exists solely for ephemeral pipeline step pods running repo-supplied code.
+Putting the database there would let any build step reach it through the
+intra-namespace allow.
+
+## Staging
+
+Rollback is trivial through stage 2. The `apps` cluster is untouched until
+stage 3, so reverting means pointing the DSN back and scaling up.
+
+| Stage | Contents | CI impact |
+|---|---|---|
+| **1** (this PR) | `woodpecker` namespace, its network policies, ExternalSecret, ObjectStores, and `wp-pg` restored from barman | none — parallel, nothing points at it |
+| **runbook** | verify the restore holds real data | none |
+| **2** | move server + agent, flip DSN, RBAC subject, rotator policies, docs | **downtime** |
+| **3** | delete the `apps` originals, add the prune guard to the survivor | none |
+
+## Stage 1 design notes
+
+**Two ObjectStores, deliberately.** `wp-pg-store-source` points at
+`s3://renner-uno-k8s-cnpg/wp-pg` — the prefix the live cluster archives to —
+and is read-only. `wp-pg-store` points at a **new** prefix,
+`s3://renner-uno-k8s-cnpg/wp-pg-woodpecker`, for ongoing archiving.
+
+Both clusters writing one `destinationPath` would interleave their WAL streams
+and corrupt both. Splitting them also leaves the pre-migration backup intact
+and independently restorable.
+
+**`serverName: wp-pg`** on the `externalCluster` names the cluster that
+*produced* the backup, not the one being created.
+
+**The namespace is declared in `infrastructure/namespaces/namespaces.yaml`,**
+not alongside these workloads. Its CiliumNetworkPolicies ship with the
+`infrastructure-crds` KS (`infrastructure/post-crds` includes
+`../network-policies/`), which `dependsOn` `infrastructure` — so the namespace
+exists before any policy targets it. Creating it in the `woodpecker` workload
+KS would invert that order, since that KS runs after both.
+
+**Ingress from `woodpecker-ci` is deliberately absent in stage 1.** Step pods do
+call the server back on `:8000`, but the CNPG instance manager serves its status
+API on `:8000` too, and the baseline policy uses `endpointSelector: {}`. A
+`toPorts` block narrows the port, not the destination workload — so putting that
+rule on the baseline would hand arbitrary repo-supplied build steps a route to
+wp-pg's instance manager. It lands in stage 2 as its own policy selecting the
+server pod.
+
+**No `prune: disabled` guard on the restored cluster.** The original carries one
+because it holds the only copy of the data. This one is reconstructable by
+re-running the restore, and a prune guard on a migration artifact is how orphans
+survive an abandoned migration. It gains the guard in stage 3.
+
+**Barman credentials still come from `seed/paperless/hetzner-creds`.** Issue
+#397 wants woodpecker to own `seed/woodpecker/hetzner-creds`, and this would
+have been the natural moment — but creating that path needs OpenBao write
+access that was not available (`bao kv list seed` → 403). Repointing at a path
+that may not exist would leave the restored cluster unable to archive WAL,
+which is the thing this stage exists to prove. #397 stays open; the swap is one
+line once the path exists.
+
+## Expect the `woodpecker` Kustomization to go not-Ready during the restore
+
+`kubernetes/clusters/homelab/woodpecker.yaml` sets `wait: true` and a
+`healthCheckExprs` entry matching `postgresql.cnpg.io/v1 Cluster` **with no
+name or namespace qualifier**. Flux evaluates it against every Cluster in that
+Kustomization's inventory — which, after this stage, is both the live
+`apps/wp-pg` and the restoring `woodpecker/wp-pg`.
+
+So while the restore runs, the whole `woodpecker` KS — the one that owns
+production Woodpecker — reports not-Ready, and a reconcile timeout in that
+window will trip the `flux-failures` Alert. **That is expected and is not
+Woodpecker breaking.** Nothing `dependsOn: woodpecker`, so it does not cascade.
+
+If the restore is going to sit for a long time, or the noise is unwelcome, the
+options are to suspend the KS while it completes, or to split the migration
+resources into their own `wait: false` Kustomization. For a ~660 MB database the
+window should be minutes.
+
+## Verifying stage 1
+
+Restore is asynchronous. Watch it reach a healthy state:
+
+```sh
+kubectl -n woodpecker get cluster wp-pg -w
+kubectl -n woodpecker get pods
+```
+
+Expect `Cluster in healthy state`. If it sticks in bootstrap, the recovery job's
+logs carry the reason:
+
+```sh
+kubectl -n woodpecker logs -l cnpg.io/jobRole=full-recovery --tail=100
+```
+
+**First check the generated app credentials target the right database.** CNPG
+defaults `bootstrap.recovery.database` to `app`; the restored data uses
+`woodpecker`. If this reads anything but `woodpecker`, the cluster has
+provisioned an empty `app` database and the real data is unreachable through
+the `-app` Secret and Service:
+
+```sh
+kubectl -n woodpecker get secret wp-pg-app -o jsonpath='{.data.dbname}' | base64 -d; echo
+kubectl -n woodpecker get secret wp-pg-app -o jsonpath='{.data.username}' | base64 -d; echo
+```
+
+**Then confirm the restore actually holds Woodpecker's data** — a cluster that
+comes up healthy but empty is the failure mode worth catching:
+
+```sh
+PGPASS=$(kubectl -n woodpecker get secret wp-pg-app -o jsonpath='{.data.password}' | base64 -d)
+kubectl -n woodpecker exec -it wp-pg-1 -- env PGPASSWORD="$PGPASS" \
+  psql -U woodpecker -d woodpecker -c '\dt'
+kubectl -n woodpecker exec -it wp-pg-1 -- env PGPASSWORD="$PGPASS" \
+  psql -U woodpecker -d woodpecker -c 'select count(*) from repos;'
+```
+
+Row counts should match the live cluster:
+
+```sh
+PGPASS_OLD=$(kubectl -n apps get secret wp-pg-app -o jsonpath='{.data.password}' | base64 -d)
+kubectl -n apps exec -it wp-pg-1 -- env PGPASSWORD="$PGPASS_OLD" \
+  psql -U woodpecker -d woodpecker -c 'select count(*) from repos;'
+```
+
+They will not match exactly if builds ran after the backup the restore used.
+
+**And stage 2 does not close that gap.** An earlier draft of this document
+claimed it quiesced the server first; it does not — Flux simply prunes the
+`apps` workloads and creates the `woodpecker` ones, so anything written to the
+old database between the restore and the cutover stays only in the old database.
+
+Measured on the live cutover: the restore captured through pipeline `9488`,
+four more (`9491`, `9492`, `9494`, `9496`) landed in `apps/wp-pg` before the
+swap, and the new cluster resumed at `9521`. Four pipeline records and their
+log entries exist only in the old cluster. No ID collision, no functional
+breakage — Woodpecker's own manifest calls this database "small and
+non-critical (state is largely reconstructable)" — but it is real history loss
+and should be a decision, not a surprise.
+
+If that gap matters for a future migration, quiesce before cutting over:
+
+```sh
+flux suspend kustomization woodpecker
+kubectl -n <old-ns> scale deploy/woodpecker-server deploy/woodpecker-agent --replicas=0
+# re-run the restore so it captures the final state, then resume
+```
+
+The old cluster keeps its `prune: disabled` guard until stage 3, so the missing
+rows remain recoverable until then.
+
+Confirm the two fixes this move exists for:
+
+```sh
+# #433: operator can now reach the instance
+kubectl -n woodpecker get cluster wp-pg -o jsonpath='{.status.conditions}'
+# metrics now scraped — needs BOTH the PodMonitor (cnpg-podmonitors.yaml) and
+# the monitoring ingress allow; wp-pg had neither in `apps`, which is why it is
+# absent from cnpg_collector_up while ak-pg/im-pg/gf-pg/pprl-pg are present
+kubectl get --raw '/api/v1/namespaces/monitoring/services/kube-prometheus-stack-prometheus:9090/proxy/api/v1/query?query=cnpg_collector_up{namespace="woodpecker"}'
+```
+
+Confirm the scrape target is actually registered, not just that the metric is
+missing for some other reason:
+
+```sh
+kubectl -n woodpecker get podmonitor cnpg-wp-pg
+```
+
+**Check the policy semantics by hand.** `kubectl --dry-run=server` validates
+manifest shape, not Cilium behaviour, so it cannot catch a rule that is
+syntactically fine but scoped wider than intended.
+
+`cilium-dbg policy trace` does **not** exist in Cilium 1.20 — it was removed,
+and `cilium-dbg policy` now offers only `get`, `selectors`, `subject-selectors`
+and `wait`. Read the realized BPF policy map instead, which is better evidence
+anyway: it is the datapath as programmed, not a simulation of it, and it
+carries live byte/packet counters.
+
+Find the agent on wp-pg's node and the endpoint id:
+
+```sh
+kubectl -n woodpecker get pod wp-pg-1 -o jsonpath='{.spec.nodeName}'
+kubectl -n kube-system get pod -l k8s-app=cilium \
+  --field-selector spec.nodeName=<node> --no-headers -o custom-columns=N:.metadata.name
+
+kubectl -n kube-system exec <cilium-pod> -c cilium-agent -- \
+  cilium-dbg endpoint list -o json \
+  | jq -r '.[] | select((.status.labels."security-relevant"[]?|tostring)
+      | test("cnpg.io/cluster=wp-pg")) | "\(.id) \(.status.networking.addressing[0].ipv4)"'
+```
+
+Then, on that endpoint id — who may reach `:8000`:
+
+```sh
+kubectl -n kube-system exec <cilium-pod> -c cilium-agent -- \
+  cilium-dbg bpf policy get <endpoint-id> | awk '$4 ~ /8000/ || NR<=1'
+```
+
+Expect exactly one row, the CNPG operator, with non-zero counters once it has
+polled:
+
+```
+Allow  Ingress  k8s:app.kubernetes.io/instance=cloudnative-pg  8000/TCP  11756  58
+```
+
+And confirm CI step pods reach **ICMP only** — never a TCP port:
+
+```sh
+kubectl -n kube-system exec <cilium-pod> -c cilium-agent -- \
+  cilium-dbg bpf policy get <endpoint-id> | grep woodpecker-ci | awk '{print $4}' | sort -u
+```
+
+Every line must be an ICMP/ICMPv6 type. A bare port number here means the
+namespace baseline has grown a rule it should not have.
+
+Note the operator's allow is also confirmed indirectly and more strongly by
+`Ready=True` on the Cluster: that condition only goes true once the operator has
+successfully polled this exact endpoint and port.
+
+And that archiving goes to the new prefix, leaving the old one alone:
+
+```sh
+kubectl -n woodpecker get cluster wp-pg -o jsonpath='{.status.conditions[?(@.type=="ContinuousArchiving")]}'
+```
+
+## Stage 2 — the cutover
+
+Moves server, agent, service, ingress, pdb, the agent ServiceAccount, the
+`woodpecker-secret` ExternalSecret and the rotator policy into `woodpecker`,
+points the server's DSN at `wp-pg-rw.woodpecker.svc`, repoints the agent's gRPC
+at `woodpecker.woodpecker.svc:9000`, moves the openbao rotators' egress, and
+adds the `woodpecker-ci` → server `:8000` policy that stage 1 deliberately
+deferred — scoped to the server pod, not the namespace baseline.
+
+**This is the downtime window.** Flux prunes the `apps` copies and creates the
+`woodpecker` ones; `ci.renner.uno` is down in between, and in-flight pipelines
+die with their agent.
+
+**The outage outlasts the pods becoming Ready, because of TLS.** Certificates
+are namespace-scoped, so moving the Ingress does not move `woodpecker-tls` —
+cert-manager issues a fresh one in the new namespace. Until it completes,
+Traefik serves a default certificate and clients fail before they get an HTTP
+status at all:
+
+```
+curl: (60) SSL: no alternative certificate subject name matches target hostname 'ci.renner.uno'
+```
+
+That is expected, not a routing or policy fault. Watch it clear with:
+
+```sh
+kubectl -n woodpecker get certificate woodpecker-tls -w
+```
+
+`letsencrypt-cluster` solves **DNS-01** via Cloudflare, so there is no in-cluster
+ACME solver pod and no HTTP challenge to admit through the network policies —
+worth knowing before scoping any ingress rule, since an HTTP-01 issuer would
+need the solver reachable too.
+
+The four CNPG files (`cluster.yaml`, `objectstore.yaml`, `scheduled-backup.yaml`,
+`hetzner-creds-externalsecret.yaml`) still describe the **original** apps-namespace
+cluster and are deliberately untouched — they are the rollback. Stage 3 retires
+them.
+
+### Verifying stage 2
+
+```sh
+kubectl -n woodpecker get pods
+kubectl -n apps get pods -l app.kubernetes.io/name=woodpecker   # expect none
+```
+
+The server must reach the restored database, not the old one:
+
+```sh
+kubectl -n woodpecker logs deploy/woodpecker-server --tail=40 | grep -i "database\|migrat\|error"
+kubectl -n woodpecker exec deploy/woodpecker-server -- env | grep DATASOURCE
+```
+
+Then confirm the UI and a real pipeline, which is the only check that exercises
+server → agent → step pod → server end to end:
+
+```sh
+curl -sS -o /dev/null -w '%{http_code}\n' https://ci.renner.uno/
+woodpecker pipeline create --branch master --var PIPELINE=e2e-test k.renner/homelab
+```
+
+Step pods must still reach the server on `:8000` while still being unable to
+reach wp-pg on any TCP port — re-run the `bpf policy get` checks above against
+both the server's and wp-pg's endpoint ids.
+
+Check the rotators can still reach the server, since their egress moved:
+
+```sh
+kubectl -n openbao get cronjob
+kubectl -n openbao logs -l app.kubernetes.io/component=secret-rotator --tail=20
+```
+
+### Rolling back stage 2
+
+Revert the PR. Flux recreates the `apps` workloads and the DSN points back at
+`wp-pg-rw.apps.svc`, which never stopped running. Data written to the new
+cluster during the window is lost, so roll back promptly or not at all — if
+pipelines have run against `woodpecker/wp-pg`, prefer fixing forward.
+
+## Stage 3 — retiring the old cluster
+
+Deletes the four `apps` manifests, promotes the `migration-*` set onto their
+names, and gives the surviving Cluster the `prune: disabled` data guard it was
+deliberately denied while a rollback still existed.
+
+**Merging stage 3 does not delete anything.** The old resources carry
+`kustomize.toolkit.fluxcd.io/prune: disabled`, so dropping them from git only
+stops Flux managing them — they keep running, and keep occupying their PVC.
+That is the point: the PR is reversible, the teardown is a separate, deliberate
+act.
+
+Before deleting anything, confirm the new cluster is genuinely standing on its
+own — that it has taken at least one base backup to its own prefix, so recovery
+no longer depends on the retired cluster's data:
+
+```sh
+kubectl -n woodpecker get backup.postgresql.cnpg.io
+kubectl -n woodpecker get cluster wp-pg -o jsonpath='{.status.conditions}'
+```
+
+`LastBackupSucceeded` must be `True`, not merely `BackupStarted`.
+
+Then retire the old set:
+
+```sh
+kubectl -n apps delete scheduledbackup.postgresql.cnpg.io wp-pg-daily
+kubectl -n apps delete cluster.postgresql.cnpg.io wp-pg          # deletes its PVCs
+kubectl -n apps delete objectstore.barmancloud.cnpg.io wp-pg-store
+kubectl -n apps delete externalsecret hetzner-s3-creds
+```
+
+Delete the ScheduledBackup first: leaving it while the Cluster goes away
+produces failing backup jobs against a cluster that no longer exists.
+
+**What this destroys and what it does not.** It destroys the old cluster's PVCs
+— including the four pipeline records that only ever existed there (see the
+stage-1 verification section). It does **not** touch
+`s3://renner-uno-k8s-cnpg/wp-pg`: that prefix keeps its backups until their own
+retention expires, and `wp-pg-store-source` is retained precisely so it stays
+restorable. That is the real safety net, not the running cluster.
+
+### After the teardown
+
+The `apps` namespace should have no woodpecker-related objects left:
+
+```sh
+kubectl -n apps get cluster,objectstore,scheduledbackup,externalsecret 2>/dev/null | grep -i "wp-pg\|hetzner" || echo "clean"
+```
+
+## Rolling back stage 1
+
+Delete the four resources and the namespace. Nothing references them, and the
+`apps` cluster is untouched:
+
+```sh
+kubectl delete ns woodpecker
+```
+
+Then revert this PR so Flux stops recreating them.

kubernetes/apps/woodpecker/agent.yaml

@@ -2,7 +2,7 @@ apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: woodpecker-agent
-  namespace: apps
+  namespace: woodpecker
   labels:
     app.kubernetes.io/name: woodpecker
     app.kubernetes.io/component: agent
@@ -12,7 +12,7 @@ metadata:
     secret.reloader.stakater.com/reload: woodpecker-secret
 spec:
   # 1: the in-cluster server owns ci.renner.uno now, so this agent registers with
-  # it (woodpecker.apps.svc:9000) and runs pipelines on the kubernetes backend,
+  # it (woodpecker.woodpecker.svc:9000) and runs pipelines on the kubernetes backend,
   # replacing the retired docker agents (LXC 910/911).
   replicas: 1
   selector:
@@ -37,7 +37,7 @@ spec:
           image: woodpeckerci/woodpecker-agent:v3.17.0 # {"$imagepolicy": "flux-system:woodpecker-agent"}
           env:
             - name: WOODPECKER_SERVER
-              value: woodpecker.apps.svc.cluster.local:9000 # in-cluster gRPC, replaces LXC :9000
+              value: woodpecker.woodpecker.svc.cluster.local:9000 # in-cluster gRPC, replaces LXC :9000
             - name: WOODPECKER_MAX_WORKFLOWS
               value: "2"
             # --- kubernetes backend: pipeline steps run as native pods (retires

kubernetes/apps/woodpecker/cluster.yaml

@@ -1,19 +1,27 @@
+---
+# Woodpecker's Postgres backend. Serves the server at wp-pg-rw.woodpecker.svc.
+#
+# Originally created in `apps` with bootstrap.initdb; moved here by restoring
+# from barman into the new namespace rather than by editing a namespace field,
+# because PVCs are namespace-scoped and a Cluster cannot be relocated in place
+# (issue #433, MIGRATION-namespace.md).
+#
+# The bootstrap.recovery stanza and its externalCluster are retained rather than
+# reverted to initdb. CNPG honours `bootstrap` only when PGDATA is absent, so on
+# this live cluster it is inert — and keeping it means a cold rebuild of an
+# empty cluster restores from backup instead of coming up empty, which is the
+# gap kubernetes/RECOVERY.md exists to close for the other CNPG clusters.
 apiVersion: postgresql.cnpg.io/v1
 kind: Cluster
 metadata:
   name: wp-pg
-  namespace: apps
+  namespace: woodpecker
   annotations:
-    # CRITICAL data-guard: prune-disabled so the seed-woodpecker -> dedicated
-    # woodpecker KS ownership handoff can never prune this Cluster (which would
-    # delete its data PVCs). Landed here first, via seed-woodpecker, so it's
-    # live before the move. Kept permanently as a standing guard.
+    # Data guard, earned at stage 3 now that this is the only copy: prevents a
+    # Flux prune from deleting the Cluster and with it the data PVCs. The
+    # migration deliberately withheld this while a rollback still existed.
     kustomize.toolkit.fluxcd.io/prune: disabled
 spec:
-  # Single instance: the Woodpecker CI DB is small and non-critical (state is
-  # largely reconstructable), so HA isn't warranted — but WAL archiving to
-  # object storage still gives point-in-time backups. Mirrors grafana-db's
-  # single-instance shape, with barman enabled like immich/paperless.
   instances: 1
   imageName: ghcr.io/cloudnative-pg/postgresql:16
   storage:
@@ -25,13 +33,53 @@ spec:
       parameters:
         barmanObjectName: wp-pg-store
   bootstrap:
-    initdb:
+    recovery:
+      source: wp-pg-apps
+      # Must mirror the initdb values of the cluster this restores from
+      # (apps/wp-pg: database woodpecker, owner woodpecker). CNPG defaults
+      # `database` to `app` and `owner` to whatever `database` is, and those
+      # names do not exist in the physically restored data — it would provision
+      # an empty `app` database, point the generated wp-pg-app Secret and the
+      # -app Service at it, and leave the real data unreachable through them.
+      # Same pattern as every recovery example in kubernetes/RECOVERY.md.
       database: woodpecker
       owner: woodpecker
+  externalClusters:
+    - name: wp-pg-apps
+      plugin:
+        name: barman-cloud.cloudnative-pg.io
+        parameters:
+          barmanObjectName: wp-pg-store-source
+          # The cluster that produced the backup, not this one.
+          serverName: wp-pg
   postgresql:
     parameters:
       shared_buffers: "128MB"
       max_connections: "50"
+      # Write-reduction tuning. These databases are largely idle, and the
+      # Postgres defaults favour throughput over storage that wears out.
+      #
+      # checkpoint_timeout is the lever. The 5min default pushes every dirty
+      # page to disk 288 times a day, so a small database whose same few pages
+      # are re-dirtied constantly (sessions, tokens, job state) gets rewritten
+      # over and over. Measured on ak-pg: 25.8M checkpoint buffers flushed
+      # (~206 GB) against a 1046 MB database -- the whole thing roughly twice a
+      # day. At 15min those pages coalesce and are written a third as often.
+      #
+      # max_wal_size must rise with it, or checkpoints trigger on size instead
+      # and the timeout change does nothing. It also bounds crash recovery to
+      # at most 2GB of WAL replay.
+      #
+      # NOT changed: synchronous_commit stays at the default `on`. It is off for
+      # ha-pg because a state recorder is telemetry; these hold credentials,
+      # documents and job state, where losing the last few hundred ms of commits
+      # on an unclean shutdown is not an acceptable trade.
+      checkpoint_timeout: "15min"
+      max_wal_size: "2GB"
+      # zstd, not "on": the boolean spelling resolves to pglz (2016-era, weaker
+      # and slower than the lz4/zstd this build also accepts). Compresses the
+      # full-page images that dominate WAL volume after each checkpoint.
+      wal_compression: "zstd"
   resources:
     requests:
       cpu: 100m

kubernetes/apps/woodpecker/externalsecret.yaml

@@ -2,7 +2,7 @@ apiVersion: external-secrets.io/v1
 kind: ExternalSecret
 metadata:
   name: woodpecker-secret
-  namespace: apps
+  namespace: woodpecker
 spec:
   # 10m (was 1h) so an automated rotation (rotate-agent-secret.yaml writes a new
   # value to OpenBao) propagates into the k8s Secret within ~10m without a manual

kubernetes/apps/woodpecker/hetzner-creds-externalsecret.yaml

@@ -1,11 +1,20 @@
+---
+# Barman S3 credentials for wp-pg.
+#
+# Sourced from seed/paperless/hetzner-creds, which is not where they belong.
+# Issue #397 wants woodpecker to own seed/woodpecker/hetzner-creds; creating
+# that path needs OpenBao write access (`bao kv list seed` → 403 for the
+# identities available here), so it stays open. The values are identical, so it
+# is a one-line swap of the `key:` below once the path exists.
+#
+# The namespace allowlist on the openbao ClusterSecretStore
+# (infrastructure/post-crds/openbao-clustersecretstore.yaml) must include this
+# namespace, or ESO silently never syncs — see #505.
 apiVersion: external-secrets.io/v1
 kind: ExternalSecret
 metadata:
   name: hetzner-s3-creds
-  namespace: apps
-  annotations:
-    # Prune-disabled — a standing guard kept from the seed->dedicated-KS handoff (barman S3 creds).
-    kustomize.toolkit.fluxcd.io/prune: disabled
+  namespace: woodpecker
 spec:
   refreshInterval: 1h
   secretStoreRef:
@@ -14,10 +23,5 @@ spec:
   target:
     name: hetzner-s3-creds
   dataFrom:
-    # Reuses the paperless copy of the shared Hetzner S3 credentials (identical
-    # ACCESS_KEY_ID / ACCESS_SECRET_KEY / REGION) to avoid seeding a new OpenBao
-    # path for this migration. FOLLOW-UP: give woodpecker its own
-    # seed/woodpecker/hetzner-creds path once OpenBao can be written, and repoint
-    # this here (the value is the same, so it's a drop-in swap).
     - extract:
         key: seed/paperless/hetzner-creds

kubernetes/apps/woodpecker/ingress.yaml

@@ -2,7 +2,7 @@ apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
   name: woodpecker
-  namespace: apps
+  namespace: woodpecker
   labels:
     app.kubernetes.io/name: woodpecker
   annotations:

kubernetes/apps/woodpecker/kustomization.yaml

@@ -21,11 +21,20 @@ resources:
   # (kubernetes/infrastructure/openbao/rotation/woodpecker.yaml) — intra-ns
   # OpenBao access, no NetworkPolicy needed. server.yaml/agent.yaml keep their
   # Reloader annotations; externalsecret.yaml keeps the 10m refreshInterval.
-  # wp-pg CNPG Postgres backend (moved here from the retired seed-woodpecker KS;
-  # each object carries prune: disabled so ownership transferred without a
-  # teardown). The dedicated `woodpecker` KS depends on infrastructure-cnpg-barman
-  # for the barman plugin behind the ObjectStore/ScheduledBackup.
-  - cluster.yaml
-  - objectstore.yaml
-  - scheduled-backup.yaml
+  # --- namespace migration, stages 1-2 (issue #433) ---
+  # Stage 2 has moved the workloads above (server, agent, service, ingress, pdb,
+  # rbac, secrets, policies) into the `woodpecker` namespace and pointed the
+  # server's DSN at wp-pg-rw.woodpecker.svc. Stage 3 retired the `apps`
+  # originals; these four now describe the surviving cluster. History and the
+  # reasoning behind the three stages: MIGRATION-namespace.md.
+  #
+  # The `woodpecker` Namespace itself is declared in infrastructure/namespaces,
+  # not here: it must exist before the `infrastructure-crds` KS applies the
+  # namespace's CiliumNetworkPolicies, and this KS runs after both of those.
+  #
+  # This KS depends on infrastructure-cnpg-barman for the barman plugin behind
+  # the ObjectStore and ScheduledBackup.
   - hetzner-creds-externalsecret.yaml
+  - objectstore.yaml
+  - cluster.yaml
+  - scheduled-backup.yaml

kubernetes/apps/woodpecker/networkpolicy.yaml

@@ -1,20 +1,23 @@
 # Secret rotation: the review-bot and claude-oauth rotators (openbao ns) reach
 # the Woodpecker SERVER to upsert org-level CI secrets (review_bot_token,
 # claude_code_oauth_token). They use the in-cluster Service
-# (woodpecker.apps.svc:8000) — the public ci.renner.uno name hairpins off the
-# ingress LB from a pod. The apps default-deny-ingress only allows same-ns /
-# host / a few named sources, so without this the rotator's connect to :8000 is
+# (woodpecker.woodpecker.svc:8000) — the public ci.renner.uno name hairpins off the
+# ingress LB from a pod. This namespace's baseline
+# (network-policies/woodpecker-ns.yaml) admits only same-namespace, monitoring,
+# host/remote-node and ICMP, so without this the rotator's connect to :8000 is
 # dropped. Scoped to the woodpecker SERVER pod :8000 and the two rotator labels;
 # the egress halves are rotator-review-bot-egress + rotator-claude-oauth-egress
 # in network-policies/openbao-ns.yaml.
 #
-# This is additive to the apps-ns baseline allows (allow-ingress.yaml selects all
-# apps pods), so the server's existing ingress — Traefik, agents — is unaffected.
+# Cilium unions allow rules, so this composes with the baseline and with the
+# sibling policies below (traefik, CI steps) rather than replacing them. Note
+# allow-ingress.yaml does NOT apply here — it is hard-scoped to `namespace:
+# apps`, which is why traefik needs its own rule in this file now.
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
 metadata:
   name: woodpecker-allow-rotators
-  namespace: apps
+  namespace: woodpecker
 spec:
   endpointSelector:
     matchLabels:
@@ -34,3 +37,25 @@ spec:
         - ports:
             - port: "8000"
               protocol: TCP
+---
+# Pipeline step pods call the server back on :8000 to stream logs and report
+# status. Deferred from stage 1 on purpose, and scoped to the SERVER pod rather
+# than granted on the namespace baseline.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: woodpecker-allow-ci-steps
+  namespace: woodpecker
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: woodpecker
+      app.kubernetes.io/component: server
+  ingress:
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": woodpecker-ci
+      toPorts:
+        - ports:
+            - port: "8000"
+              protocol: TCP

kubernetes/apps/woodpecker/objectstore.yaml

@@ -1,18 +1,20 @@
 ---
+# RECOVERY SOURCE — read-only. The prefix the retired `apps/wp-pg` archived to.
+#
+# Kept after the migration on purpose. Nothing writes to it, and the backups
+# under it age out on their own retention, but while they last this is a second,
+# independent restore point that predates the move — worth having until the new
+# prefix has a meaningful backup history of its own.
+#
+# serverName on the consuming externalCluster must be `wp-pg`, the name of the
+# cluster that produced these backups.
 apiVersion: barmancloud.cnpg.io/v1
 kind: ObjectStore
 metadata:
-  name: wp-pg-store
-  namespace: apps
-  annotations:
-    # Prune-disabled — a standing guard kept from the seed->dedicated-KS handoff (backup target).
-    kustomize.toolkit.fluxcd.io/prune: disabled
+  name: wp-pg-store-source
+  namespace: woodpecker
 spec:
   configuration:
-    # Same offsite Hetzner Object Storage bucket as the other CNPG clusters,
-    # under a per-cluster prefix. See apps/paperless/objectstore.yaml for the
-    # SSE / addressing rationale (Hetzner is SSE-C only; transit is TLS,
-    # bucket private).
     destinationPath: s3://renner-uno-k8s-cnpg/wp-pg
     endpointURL: https://fsn1.your-objectstorage.com
     s3Credentials:
@@ -29,4 +31,36 @@ spec:
       compression: gzip
     data:
       compression: gzip
+  # No retentionPolicy: this store is a read source during the migration, and
+  # the old cluster in `apps` owns the lifecycle of the data underneath it.
+---
+# ARCHIVING TARGET — a DIFFERENT prefix from the source, deliberately.
+#
+# Two clusters archiving WAL to one destinationPath interleave their streams and
+# corrupt both. That mattered during the parallel-run window; it still matters
+# now, because the source prefix above is retained and must stay frozen for the
+# independent restore point it provides.
+apiVersion: barmancloud.cnpg.io/v1
+kind: ObjectStore
+metadata:
+  name: wp-pg-store
+  namespace: woodpecker
+spec:
+  configuration:
+    destinationPath: s3://renner-uno-k8s-cnpg/wp-pg-woodpecker
+    endpointURL: https://fsn1.your-objectstorage.com
+    s3Credentials:
+      accessKeyId:
+        name: hetzner-s3-creds
+        key: ACCESS_KEY_ID
+      secretAccessKey:
+        name: hetzner-s3-creds
+        key: ACCESS_SECRET_KEY
+      region:
+        name: hetzner-s3-creds
+        key: REGION
+    wal:
+      compression: gzip
+    data:
+      compression: gzip
   retentionPolicy: "30d"

kubernetes/apps/woodpecker/pdb.yaml

@@ -15,7 +15,7 @@ apiVersion: policy/v1
 kind: PodDisruptionBudget
 metadata:
   name: woodpecker-server
-  namespace: apps
+  namespace: woodpecker
   labels:
     app.kubernetes.io/name: woodpecker
     app.kubernetes.io/component: server

kubernetes/apps/woodpecker/rbac.yaml

@@ -17,7 +17,7 @@ apiVersion: v1
 kind: ServiceAccount
 metadata:
   name: woodpecker-agent
-  namespace: apps
+  namespace: woodpecker
   labels:
     app.kubernetes.io/name: woodpecker
     app.kubernetes.io/component: agent
@@ -60,4 +60,7 @@ roleRef:
 subjects:
   - kind: ServiceAccount
     name: woodpecker-agent
-    namespace: apps
+    # The agent's SA lives in `woodpecker`; this Role/RoleBinding stays in
+    # `woodpecker-ci`, because that is the namespace it grants pod
+    # create/delete in. Cross-namespace subject, deliberately.
+    namespace: woodpecker

kubernetes/apps/woodpecker/scheduled-backup.yaml

@@ -1,18 +1,19 @@
 ---
-# Daily base backup at 02:35 UTC (offset from the other clusters' 02:30 to
-# avoid a thundering herd against the Hetzner bucket). WAL archiving runs
-# continuously (see cluster.yaml spec.plugins[].isWALArchiver). immediate:true
-# takes the first base backup as soon as this CR applies.
+# Daily base backups onto the wp-pg-woodpecker prefix. Continuous WAL archiving
+# runs separately (cluster.yaml spec.plugins[].isWALArchiver); without periodic
+# base backups the prefix would have no recovery point of its own and
+# wp-pg-store's retentionPolicy: 30d would have nothing to prune against, so the
+# WAL chain would grow without bound.
+#
+# 02:40 UTC, offset from the other CNPG clusters' 02:30 so they do not all hit
+# the Hetzner bucket together.
 apiVersion: postgresql.cnpg.io/v1
 kind: ScheduledBackup
 metadata:
   name: wp-pg-daily
-  namespace: apps
-  annotations:
-    # Prune-disabled — a standing guard kept from the seed->dedicated-KS handoff (backup schedule).
-    kustomize.toolkit.fluxcd.io/prune: disabled
+  namespace: woodpecker
 spec:
-  schedule: "0 35 2 * * *"
+  schedule: "0 40 2 * * *"
   immediate: true
   backupOwnerReference: self
   cluster:

kubernetes/apps/woodpecker/secret.example.yaml

@@ -14,7 +14,7 @@ apiVersion: v1
 kind: Secret
 metadata:
   name: woodpecker-secret
-  namespace: apps
+  namespace: woodpecker
   labels:
     app.kubernetes.io/name: woodpecker
 type: Opaque

kubernetes/apps/woodpecker/server.yaml

@@ -2,7 +2,7 @@ apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: woodpecker-server
-  namespace: apps
+  namespace: woodpecker
   labels:
     app.kubernetes.io/name: woodpecker
     app.kubernetes.io/component: server
@@ -84,7 +84,7 @@ spec:
             - name: WOODPECKER_DATABASE_DRIVER
               value: postgres
             - name: WOODPECKER_DATABASE_DATASOURCE
-              value: "host=wp-pg-rw.apps.svc.cluster.local port=5432 user=$(WP_DB_USER) password=$(WP_DB_PASS) dbname=woodpecker sslmode=require"
+              value: "host=wp-pg-rw.woodpecker.svc.cluster.local port=5432 user=$(WP_DB_USER) password=$(WP_DB_PASS) dbname=woodpecker sslmode=require"
           # AGENT_SECRET + GITEA_CLIENT + GITEA_SECRET come from the SOPS secret.
           envFrom:
             - secretRef:

kubernetes/apps/woodpecker/service.yaml

@@ -2,13 +2,13 @@ apiVersion: v1
 kind: Service
 metadata:
   name: woodpecker
-  namespace: apps
+  namespace: woodpecker
   labels:
     app.kubernetes.io/name: woodpecker
 spec:
   # Flipped ExternalName -> ClusterIP: ci.renner.uno (via the Ingress, by name)
   # and the in-cluster agent both target this Service now. http:8000 serves the
-  # UI/API; grpc:9000 is how the agent (woodpecker.apps.svc:9000) reaches the
+  # UI/API; grpc:9000 is how the agent (woodpecker.woodpecker.svc:9000) reaches the
   # server. Selects the server pod only (not the agent).
   type: ClusterIP
   selector:

kubernetes/infrastructure/cilium/helmrelease.yaml

@@ -17,7 +17,11 @@ spec:
   chart:
     spec:
       chart: cilium
-      version: ">=1.19.0"
+      # Patch-only range: Flux reconciles every 30m and would otherwise take a
+      # CNI minor or major unattended. Widening this is a deliberate, reviewed
+      # act — a minor bump changes datapath behaviour and rolls 2 of 3 agents
+      # at once.
+      version: ">=1.20.0 <1.21.0"
       sourceRef:
         kind: HelmRepository
         name: cilium
@@ -29,6 +33,16 @@ spec:
     remediation:
       retries: 3
   values:
+    # Which pod labels feed security-identity computation. Patterns are
+    # start-anchored regexes; a leading "!" excludes while keeping the default
+    # set. Woodpecker stamps every step pod with woodpecker-ci.org/task-uuid
+    # and step=<name>, so without this each build step is its own identity —
+    # and every identity costs one entry per allowed (direction, proto, port)
+    # in every endpoint's policy map, cluster-wide. Nothing selects on these
+    # labels in policy, so they buy identity churn and nothing else.
+    # Keep in lockstep with kubernetes/talos/patches/cilium-cni.yaml and
+    # kubernetes/talos/cilium.yaml.
+    labels: '!woodpecker-ci\.org/.* !step$'
     ipv6:
       enabled: true
     ipam:
@@ -46,6 +60,19 @@ spec:
       serviceMonitor:
         enabled: true
     bpf:
+      # Size of each endpoint's own cilium_policy_v3_<id> map, holding one
+      # entry per (identity, direction, protocol, port) the endpoint is
+      # allowed. When it fills, endpoint regeneration fails with "no space
+      # left on device", the CNI ADD call never returns, and the pod is stuck
+      # in sandbox creation — so this must stay comfortably above the busiest
+      # endpoint's entry count, tracked by cilium_bpf_map_pressure.
+      # Takes effect on agent restart; the map is sized at creation.
+      # Bootstrap installs Cilium from the inline manifest in
+      # kubernetes/talos/patches/cilium-cni.yaml, so bpf-policy-map-max there
+      # must move with this value or a cold rebuild starts with the old size.
+      # kubernetes/talos/cilium.yaml carries a third copy of the same key.
+      # Nothing enforces any of this; grep bpf-policy-map-max before changing it.
+      policyMapMax: 65536
       # Let pods reach the cluster's own LoadBalancer external IPs; Cilium
       # redirects the socket directly to the backend pod instead of sending
       # traffic out to the VIP and back (which fails with L2 announcements).

kubernetes/infrastructure/cnpg-barman-plugin/manifest.yaml

@@ -917,11 +917,27 @@ subjects:
   name: plugin-barman-cloud
   namespace: database
 ---
+# stringData, NOT base64 `data`. An image reference is not a secret, and
+# encoding it here had two real costs: no reviewer sees a version change in a
+# diff, and no $imagepolicy marker can be attached to a base64 blob. The result
+# was that automation advanced the controller to v0.14.0 while this sidecar sat
+# at v0.12.0 — a two-minor skew in the component that performs every base backup
+# and WAL archive, invisible until someone read the pod spec (#535).
+#
+# The Secret KIND is kept because the controller reads it via secretKeyRef
+# (deployment env SIDECAR_IMAGE); changing it to a ConfigMap would need a matching
+# edit to that reference and diverge further from upstream. stringData is
+# accepted identically by the API server and serialised into `data` on write.
+#
+# The name carries a kustomize hash suffix from upstream's secretGenerator. It is
+# static here, so editing the value does NOT change the name and the deployment's
+# secretKeyRef stays valid.
+#
+# IF YOU RE-IMPORT THE UPSTREAM MANIFEST: upstream ships this as base64 `data`
+# with no marker. Re-apply this block, or the skew silently returns.
 apiVersion: v1
-data:
-  SIDECAR_IMAGE: |
-    Z2hjci5pby9jbG91ZG5hdGl2ZS1wZy9wbHVnaW4tYmFybWFuLWNsb3VkLXNpZGVjYXI6dj
-    AuMTIuMA==
+stringData:
+  SIDECAR_IMAGE: ghcr.io/cloudnative-pg/plugin-barman-cloud-sidecar:v0.14.0 # {"$imagepolicy": "flux-system:cnpg-barman-plugin-sidecar"}
 kind: Secret
 metadata:
   name: plugin-barman-cloud-kg5c2795f7

kubernetes/infrastructure/image-automation/cnpg-barman-plugin.yaml

@@ -28,3 +28,40 @@ spec:
   policy:
     semver:
       range: "0.x"
+---
+# The SIDECAR image, tracked separately from the controller above.
+#
+# These are two DIFFERENT images that must move together:
+#   plugin-barman-cloud          the controller, in the database namespace
+#   plugin-barman-cloud-sidecar  injected into every Postgres pod
+#
+# Only the controller was tracked, so automation walked it from v0.12.0 to
+# v0.14.0 while the sidecar stayed at v0.12.0 — a two-minor skew in the
+# component that performs every base backup and WAL archive (#535). The skew was
+# invisible because the sidecar tag was base64-encoded inside a Secret, where no
+# $imagepolicy marker can reach it and no reviewer would see it in a diff.
+apiVersion: image.toolkit.fluxcd.io/v1
+kind: ImageRepository
+metadata:
+  name: cnpg-barman-plugin-sidecar
+  namespace: flux-system
+spec:
+  image: ghcr.io/cloudnative-pg/plugin-barman-cloud-sidecar
+  interval: 1h
+---
+# Same filter and range as the controller policy deliberately: the two images
+# are released together and must not be allowed to drift apart again.
+apiVersion: image.toolkit.fluxcd.io/v1
+kind: ImagePolicy
+metadata:
+  name: cnpg-barman-plugin-sidecar
+  namespace: flux-system
+spec:
+  imageRepositoryRef:
+    name: cnpg-barman-plugin-sidecar
+  filterTags:
+    pattern: '^v(?P<ver>\d+\.\d+\.\d+)$'
+    extract: '$ver'
+  policy:
+    semver:
+      range: "0.x"

kubernetes/infrastructure/namespaces/namespaces.yaml

@@ -160,3 +160,26 @@ apiVersion: v1
 kind: Namespace
 metadata:
   name: wordpress
+---
+# The Woodpecker service itself — server, agent and its wp-pg Postgres backend,
+# moving out of the shared `apps` namespace (issue #433). Distinct from
+# `woodpecker-ci`, which enforces pod-security: privileged and holds only
+# ephemeral pipeline step pods.
+#
+# `apps` uses an endpointSelector:{} default-deny, so every allow it carries
+# applies to gitea, gatus, external-dns and ical-proxy alike — granting the
+# CNPG operator access to wp-pg there would have opened all of `apps` to all of
+# `database`. A namespace per service keeps that grant on the one workload.
+#
+# Declared here rather than alongside the workloads so it is created by the
+# `infrastructure` KS. The namespace's CiliumNetworkPolicies ship with the
+# `infrastructure-crds` KS (infrastructure/post-crds includes
+# ../network-policies/), which itself dependsOn `infrastructure` — so the
+# namespace exists before any policy targets it. Creating it in the `woodpecker`
+# workload KS would invert that order, since that KS runs after both.
+#
+# No pod-security labels: the cluster default applies.
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: woodpecker

kubernetes/infrastructure/netbird-ingress/daemonset.yaml

@@ -156,4 +156,6 @@ spec:
         # each pod enrolls its own peer and a restart re-enrolls cleanly
         # (the old peer is reaped).
         - name: state
-          emptyDir: {}
+          emptyDir:
+            medium: Memory
+            sizeLimit: 16Mi

kubernetes/infrastructure/network-policies/audiobookshelf-ns.yaml

@@ -31,43 +31,20 @@ spec:
         - fields:
             - { type: 0,   family: IPv4 }
             - { type: 3,   family: IPv4 }
-            - { type: 4,   family: IPv4 }
-            - { type: 5,   family: IPv4 }
             - { type: 8,   family: IPv4 }
-            - { type: 9,   family: IPv4 }
-            - { type: 10,  family: IPv4 }
             - { type: 11,  family: IPv4 }
             - { type: 12,  family: IPv4 }
-            - { type: 13,  family: IPv4 }
-            - { type: 14,  family: IPv4 }
-            - { type: 15,  family: IPv4 }
-            - { type: 16,  family: IPv4 }
-            - { type: 17,  family: IPv4 }
-            - { type: 18,  family: IPv4 }
-            - { type: 30,  family: IPv4 }
             - { type: 1,   family: IPv6 }
             - { type: 2,   family: IPv6 }
             - { type: 3,   family: IPv6 }
             - { type: 4,   family: IPv6 }
             - { type: 128, family: IPv6 }
             - { type: 129, family: IPv6 }
-            - { type: 130, family: IPv6 }
-            - { type: 131, family: IPv6 }
-            - { type: 132, family: IPv6 }
             - { type: 133, family: IPv6 }
             - { type: 134, family: IPv6 }
             - { type: 135, family: IPv6 }
             - { type: 136, family: IPv6 }
-            - { type: 137, family: IPv6 }
-            - { type: 138, family: IPv6 }
-            - { type: 141, family: IPv6 }
-            - { type: 142, family: IPv6 }
             - { type: 143, family: IPv6 }
-            - { type: 148, family: IPv6 }
-            - { type: 149, family: IPv6 }
-            - { type: 151, family: IPv6 }
-            - { type: 152, family: IPv6 }
-            - { type: 153, family: IPv6 }
   egress:
     - toEndpoints:
         - matchLabels:
@@ -103,40 +80,17 @@ spec:
         - fields:
             - { type: 0,   family: IPv4 }
             - { type: 3,   family: IPv4 }
-            - { type: 4,   family: IPv4 }
-            - { type: 5,   family: IPv4 }
             - { type: 8,   family: IPv4 }
-            - { type: 9,   family: IPv4 }
-            - { type: 10,  family: IPv4 }
             - { type: 11,  family: IPv4 }
             - { type: 12,  family: IPv4 }
-            - { type: 13,  family: IPv4 }
-            - { type: 14,  family: IPv4 }
-            - { type: 15,  family: IPv4 }
-            - { type: 16,  family: IPv4 }
-            - { type: 17,  family: IPv4 }
-            - { type: 18,  family: IPv4 }
-            - { type: 30,  family: IPv4 }
             - { type: 1,   family: IPv6 }
             - { type: 2,   family: IPv6 }
             - { type: 3,   family: IPv6 }
             - { type: 4,   family: IPv6 }
             - { type: 128, family: IPv6 }
             - { type: 129, family: IPv6 }
-            - { type: 130, family: IPv6 }
-            - { type: 131, family: IPv6 }
-            - { type: 132, family: IPv6 }
             - { type: 133, family: IPv6 }
             - { type: 134, family: IPv6 }
             - { type: 135, family: IPv6 }
             - { type: 136, family: IPv6 }
-            - { type: 137, family: IPv6 }
-            - { type: 138, family: IPv6 }
-            - { type: 141, family: IPv6 }
-            - { type: 142, family: IPv6 }
             - { type: 143, family: IPv6 }
-            - { type: 148, family: IPv6 }
-            - { type: 149, family: IPv6 }
-            - { type: 151, family: IPv6 }
-            - { type: 152, family: IPv6 }
-            - { type: 153, family: IPv6 }

kubernetes/infrastructure/network-policies/authentik-ns.yaml

@@ -38,43 +38,20 @@ spec:
         - fields:
             - { type: 0,   family: IPv4 }
             - { type: 3,   family: IPv4 }
-            - { type: 4,   family: IPv4 }
-            - { type: 5,   family: IPv4 }
             - { type: 8,   family: IPv4 }
-            - { type: 9,   family: IPv4 }
-            - { type: 10,  family: IPv4 }
             - { type: 11,  family: IPv4 }
             - { type: 12,  family: IPv4 }
-            - { type: 13,  family: IPv4 }
-            - { type: 14,  family: IPv4 }
-            - { type: 15,  family: IPv4 }
-            - { type: 16,  family: IPv4 }
-            - { type: 17,  family: IPv4 }
-            - { type: 18,  family: IPv4 }
-            - { type: 30,  family: IPv4 }
             - { type: 1,   family: IPv6 }
             - { type: 2,   family: IPv6 }
             - { type: 3,   family: IPv6 }
             - { type: 4,   family: IPv6 }
             - { type: 128, family: IPv6 }
             - { type: 129, family: IPv6 }
-            - { type: 130, family: IPv6 }
-            - { type: 131, family: IPv6 }
-            - { type: 132, family: IPv6 }
             - { type: 133, family: IPv6 }
             - { type: 134, family: IPv6 }
             - { type: 135, family: IPv6 }
             - { type: 136, family: IPv6 }
-            - { type: 137, family: IPv6 }
-            - { type: 138, family: IPv6 }
-            - { type: 141, family: IPv6 }
-            - { type: 142, family: IPv6 }
             - { type: 143, family: IPv6 }
-            - { type: 148, family: IPv6 }
-            - { type: 149, family: IPv6 }
-            - { type: 151, family: IPv6 }
-            - { type: 152, family: IPv6 }
-            - { type: 153, family: IPv6 }
   egress:
     - toEndpoints:
         - matchLabels:
@@ -103,40 +80,17 @@ spec:
         - fields:
             - { type: 0,   family: IPv4 }
             - { type: 3,   family: IPv4 }
-            - { type: 4,   family: IPv4 }
-            - { type: 5,   family: IPv4 }
             - { type: 8,   family: IPv4 }
-            - { type: 9,   family: IPv4 }
-            - { type: 10,  family: IPv4 }
             - { type: 11,  family: IPv4 }
             - { type: 12,  family: IPv4 }
-            - { type: 13,  family: IPv4 }
-            - { type: 14,  family: IPv4 }
-            - { type: 15,  family: IPv4 }
-            - { type: 16,  family: IPv4 }
-            - { type: 17,  family: IPv4 }
-            - { type: 18,  family: IPv4 }
-            - { type: 30,  family: IPv4 }
             - { type: 1,   family: IPv6 }
             - { type: 2,   family: IPv6 }
             - { type: 3,   family: IPv6 }
             - { type: 4,   family: IPv6 }
             - { type: 128, family: IPv6 }
             - { type: 129, family: IPv6 }
-            - { type: 130, family: IPv6 }
-            - { type: 131, family: IPv6 }
-            - { type: 132, family: IPv6 }
             - { type: 133, family: IPv6 }
             - { type: 134, family: IPv6 }
             - { type: 135, family: IPv6 }
             - { type: 136, family: IPv6 }
-            - { type: 137, family: IPv6 }
-            - { type: 138, family: IPv6 }
-            - { type: 141, family: IPv6 }
-            - { type: 142, family: IPv6 }
             - { type: 143, family: IPv6 }
-            - { type: 148, family: IPv6 }
-            - { type: 149, family: IPv6 }
-            - { type: 151, family: IPv6 }
-            - { type: 152, family: IPv6 }
-            - { type: 153, family: IPv6 }

kubernetes/infrastructure/network-policies/crowdsec-ns.yaml

@@ -36,40 +36,17 @@ spec:
         - fields:
             - { type: 0,   family: IPv4 }
             - { type: 3,   family: IPv4 }
-            - { type: 4,   family: IPv4 }
-            - { type: 5,   family: IPv4 }
             - { type: 8,   family: IPv4 }
-            - { type: 9,   family: IPv4 }
-            - { type: 10,  family: IPv4 }
             - { type: 11,  family: IPv4 }
             - { type: 12,  family: IPv4 }
-            - { type: 13,  family: IPv4 }
-            - { type: 14,  family: IPv4 }
-            - { type: 15,  family: IPv4 }
-            - { type: 16,  family: IPv4 }
-            - { type: 17,  family: IPv4 }
-            - { type: 18,  family: IPv4 }
-            - { type: 30,  family: IPv4 }
             - { type: 1,   family: IPv6 }
             - { type: 2,   family: IPv6 }
             - { type: 3,   family: IPv6 }
             - { type: 4,   family: IPv6 }
             - { type: 128, family: IPv6 }
             - { type: 129, family: IPv6 }
-            - { type: 130, family: IPv6 }
-            - { type: 131, family: IPv6 }
-            - { type: 132, family: IPv6 }
             - { type: 133, family: IPv6 }
             - { type: 134, family: IPv6 }
             - { type: 135, family: IPv6 }
             - { type: 136, family: IPv6 }
-            - { type: 137, family: IPv6 }
-            - { type: 138, family: IPv6 }
-            - { type: 141, family: IPv6 }
-            - { type: 142, family: IPv6 }
             - { type: 143, family: IPv6 }
-            - { type: 148, family: IPv6 }
-            - { type: 149, family: IPv6 }
-            - { type: 151, family: IPv6 }
-            - { type: 152, family: IPv6 }
-            - { type: 153, family: IPv6 }

kubernetes/infrastructure/network-policies/database-ns.yaml

@@ -32,40 +32,17 @@ spec:
         - fields:
             - { type: 0,   family: IPv4 }
             - { type: 3,   family: IPv4 }
-            - { type: 4,   family: IPv4 }
-            - { type: 5,   family: IPv4 }
             - { type: 8,   family: IPv4 }
-            - { type: 9,   family: IPv4 }
-            - { type: 10,  family: IPv4 }
             - { type: 11,  family: IPv4 }
             - { type: 12,  family: IPv4 }
-            - { type: 13,  family: IPv4 }
-            - { type: 14,  family: IPv4 }
-            - { type: 15,  family: IPv4 }
-            - { type: 16,  family: IPv4 }
-            - { type: 17,  family: IPv4 }
-            - { type: 18,  family: IPv4 }
-            - { type: 30,  family: IPv4 }
             - { type: 1,   family: IPv6 }
             - { type: 2,   family: IPv6 }
             - { type: 3,   family: IPv6 }
             - { type: 4,   family: IPv6 }
             - { type: 128, family: IPv6 }
             - { type: 129, family: IPv6 }
-            - { type: 130, family: IPv6 }
-            - { type: 131, family: IPv6 }
-            - { type: 132, family: IPv6 }
             - { type: 133, family: IPv6 }
             - { type: 134, family: IPv6 }
             - { type: 135, family: IPv6 }
             - { type: 136, family: IPv6 }
-            - { type: 137, family: IPv6 }
-            - { type: 138, family: IPv6 }
-            - { type: 141, family: IPv6 }
-            - { type: 142, family: IPv6 }
             - { type: 143, family: IPv6 }
-            - { type: 148, family: IPv6 }
-            - { type: 149, family: IPv6 }
-            - { type: 151, family: IPv6 }
-            - { type: 152, family: IPv6 }
-            - { type: 153, family: IPv6 }

kubernetes/infrastructure/network-policies/evcc-ns.yaml

@@ -27,43 +27,20 @@ spec:
         - fields:
             - { type: 0,   family: IPv4 }
             - { type: 3,   family: IPv4 }
-            - { type: 4,   family: IPv4 }
-            - { type: 5,   family: IPv4 }
             - { type: 8,   family: IPv4 }
-            - { type: 9,   family: IPv4 }
-            - { type: 10,  family: IPv4 }
             - { type: 11,  family: IPv4 }
             - { type: 12,  family: IPv4 }
-            - { type: 13,  family: IPv4 }
-            - { type: 14,  family: IPv4 }
-            - { type: 15,  family: IPv4 }
-            - { type: 16,  family: IPv4 }
-            - { type: 17,  family: IPv4 }
-            - { type: 18,  family: IPv4 }
-            - { type: 30,  family: IPv4 }
             - { type: 1,   family: IPv6 }
             - { type: 2,   family: IPv6 }
             - { type: 3,   family: IPv6 }
             - { type: 4,   family: IPv6 }
             - { type: 128, family: IPv6 }
             - { type: 129, family: IPv6 }
-            - { type: 130, family: IPv6 }
-            - { type: 131, family: IPv6 }
-            - { type: 132, family: IPv6 }
             - { type: 133, family: IPv6 }
             - { type: 134, family: IPv6 }
             - { type: 135, family: IPv6 }
             - { type: 136, family: IPv6 }
-            - { type: 137, family: IPv6 }
-            - { type: 138, family: IPv6 }
-            - { type: 141, family: IPv6 }
-            - { type: 142, family: IPv6 }
             - { type: 143, family: IPv6 }
-            - { type: 148, family: IPv6 }
-            - { type: 149, family: IPv6 }
-            - { type: 151, family: IPv6 }
-            - { type: 152, family: IPv6 }
-            - { type: 153, family: IPv6 }
   egress:
     - toEndpoints:
         - matchLabels:
@@ -102,40 +79,17 @@ spec:
         - fields:
             - { type: 0,   family: IPv4 }
             - { type: 3,   family: IPv4 }
-            - { type: 4,   family: IPv4 }
-            - { type: 5,   family: IPv4 }
             - { type: 8,   family: IPv4 }
-            - { type: 9,   family: IPv4 }
-            - { type: 10,  family: IPv4 }
             - { type: 11,  family: IPv4 }
             - { type: 12,  family: IPv4 }
-            - { type: 13,  family: IPv4 }
-            - { type: 14,  family: IPv4 }
-            - { type: 15,  family: IPv4 }
-            - { type: 16,  family: IPv4 }
-            - { type: 17,  family: IPv4 }
-            - { type: 18,  family: IPv4 }
-            - { type: 30,  family: IPv4 }
             - { type: 1,   family: IPv6 }
             - { type: 2,   family: IPv6 }
             - { type: 3,   family: IPv6 }
             - { type: 4,   family: IPv6 }
             - { type: 128, family: IPv6 }
             - { type: 129, family: IPv6 }
-            - { type: 130, family: IPv6 }
-            - { type: 131, family: IPv6 }
-            - { type: 132, family: IPv6 }
             - { type: 133, family: IPv6 }
             - { type: 134, family: IPv6 }
             - { type: 135, family: IPv6 }
             - { type: 136, family: IPv6 }
-            - { type: 137, family: IPv6 }
-            - { type: 138, family: IPv6 }
-            - { type: 141, family: IPv6 }
-            - { type: 142, family: IPv6 }
             - { type: 143, family: IPv6 }
-            - { type: 148, family: IPv6 }
-            - { type: 149, family: IPv6 }
-            - { type: 151, family: IPv6 }
-            - { type: 152, family: IPv6 }
-            - { type: 153, family: IPv6 }

kubernetes/infrastructure/network-policies/external-secrets-ns.yaml

@@ -28,43 +28,20 @@ spec:
         - fields:
             - { type: 0,   family: IPv4 }
             - { type: 3,   family: IPv4 }
-            - { type: 4,   family: IPv4 }
-            - { type: 5,   family: IPv4 }
             - { type: 8,   family: IPv4 }
-            - { type: 9,   family: IPv4 }
-            - { type: 10,  family: IPv4 }
             - { type: 11,  family: IPv4 }
             - { type: 12,  family: IPv4 }
-            - { type: 13,  family: IPv4 }
-            - { type: 14,  family: IPv4 }
-            - { type: 15,  family: IPv4 }
-            - { type: 16,  family: IPv4 }
-            - { type: 17,  family: IPv4 }
-            - { type: 18,  family: IPv4 }
-            - { type: 30,  family: IPv4 }
             - { type: 1,   family: IPv6 }
             - { type: 2,   family: IPv6 }
             - { type: 3,   family: IPv6 }
             - { type: 4,   family: IPv6 }
             - { type: 128, family: IPv6 }
             - { type: 129, family: IPv6 }
-            - { type: 130, family: IPv6 }
-            - { type: 131, family: IPv6 }
-            - { type: 132, family: IPv6 }
             - { type: 133, family: IPv6 }
             - { type: 134, family: IPv6 }
             - { type: 135, family: IPv6 }
             - { type: 136, family: IPv6 }
-            - { type: 137, family: IPv6 }
-            - { type: 138, family: IPv6 }
-            - { type: 141, family: IPv6 }
-            - { type: 142, family: IPv6 }
             - { type: 143, family: IPv6 }
-            - { type: 148, family: IPv6 }
-            - { type: 149, family: IPv6 }
-            - { type: 151, family: IPv6 }
-            - { type: 152, family: IPv6 }
-            - { type: 153, family: IPv6 }
   egress:
     - toEndpoints:
         - matchLabels:
@@ -103,40 +80,17 @@ spec:
         - fields:
             - { type: 0,   family: IPv4 }
             - { type: 3,   family: IPv4 }
-            - { type: 4,   family: IPv4 }
-            - { type: 5,   family: IPv4 }
             - { type: 8,   family: IPv4 }
-            - { type: 9,   family: IPv4 }
-            - { type: 10,  family: IPv4 }
             - { type: 11,  family: IPv4 }
             - { type: 12,  family: IPv4 }
-            - { type: 13,  family: IPv4 }
-            - { type: 14,  family: IPv4 }
-            - { type: 15,  family: IPv4 }
-            - { type: 16,  family: IPv4 }
-            - { type: 17,  family: IPv4 }
-            - { type: 18,  family: IPv4 }
-            - { type: 30,  family: IPv4 }
             - { type: 1,   family: IPv6 }
             - { type: 2,   family: IPv6 }
             - { type: 3,   family: IPv6 }
             - { type: 4,   family: IPv6 }
             - { type: 128, family: IPv6 }
             - { type: 129, family: IPv6 }
-            - { type: 130, family: IPv6 }
-            - { type: 131, family: IPv6 }
-            - { type: 132, family: IPv6 }
             - { type: 133, family: IPv6 }
             - { type: 134, family: IPv6 }
             - { type: 135, family: IPv6 }
             - { type: 136, family: IPv6 }
-            - { type: 137, family: IPv6 }
-            - { type: 138, family: IPv6 }
-            - { type: 141, family: IPv6 }
-            - { type: 142, family: IPv6 }
             - { type: 143, family: IPv6 }
-            - { type: 148, family: IPv6 }
-            - { type: 149, family: IPv6 }
-            - { type: 151, family: IPv6 }
-            - { type: 152, family: IPv6 }
-            - { type: 153, family: IPv6 }

kubernetes/infrastructure/network-policies/home-assistant-ns.yaml

@@ -157,44 +157,21 @@ spec:
       icmps:
         - fields:
             - { type: 0,   family: IPv4 }   # Echo Reply
-            - { type: 3,   family: IPv4 }   # Destination Unreachable
-            - { type: 4,   family: IPv4 }   # Source Quench
-            - { type: 5,   family: IPv4 }   # Redirect
+            - { type: 3,   family: IPv4 }   # Destination Unreachable — PMTUD
             - { type: 8,   family: IPv4 }   # Echo Request
-            - { type: 9,   family: IPv4 }   # Router Advertisement
-            - { type: 10,  family: IPv4 }   # Router Solicitation
             - { type: 11,  family: IPv4 }   # Time Exceeded
             - { type: 12,  family: IPv4 }   # Parameter Problem
-            - { type: 13,  family: IPv4 }   # Timestamp
-            - { type: 14,  family: IPv4 }   # Timestamp Reply
-            - { type: 15,  family: IPv4 }   # Information Request
-            - { type: 16,  family: IPv4 }   # Information Reply
-            - { type: 17,  family: IPv4 }   # Address Mask Request
-            - { type: 18,  family: IPv4 }   # Address Mask Reply
-            - { type: 30,  family: IPv4 }   # Traceroute
             - { type: 1,   family: IPv6 }   # Destination Unreachable
-            - { type: 2,   family: IPv6 }   # Packet Too Big
+            - { type: 2,   family: IPv6 }   # Packet Too Big — PMTUD, mandatory
             - { type: 3,   family: IPv6 }   # Time Exceeded
             - { type: 4,   family: IPv6 }   # Parameter Problem
             - { type: 128, family: IPv6 }   # Echo Request
             - { type: 129, family: IPv6 }   # Echo Reply
-            - { type: 130, family: IPv6 }   # Multicast Listener Query
-            - { type: 131, family: IPv6 }   # Multicast Listener Report
-            - { type: 132, family: IPv6 }   # Multicast Listener Done
             - { type: 133, family: IPv6 }   # Router Solicitation
             - { type: 134, family: IPv6 }   # Router Advertisement
             - { type: 135, family: IPv6 }   # Neighbor Solicitation
             - { type: 136, family: IPv6 }   # Neighbor Advertisement
-            - { type: 137, family: IPv6 }   # Redirect
-            - { type: 138, family: IPv6 }   # Router Renumbering
-            - { type: 141, family: IPv6 }   # Inverse ND Solicitation
-            - { type: 142, family: IPv6 }   # Inverse ND Advertisement
             - { type: 143, family: IPv6 }   # MLDv2 Listener Report
-            - { type: 148, family: IPv6 }   # Certification Path Solicitation
-            - { type: 149, family: IPv6 }   # Certification Path Advertisement
-            - { type: 151, family: IPv6 }   # Multicast Router Advertisement
-            - { type: 152, family: IPv6 }   # Multicast Router Solicitation
-            - { type: 153, family: IPv6 }   # Multicast Router Termination
   egress:
     # Cluster-internal callers HA initiates connections to: mosquitto,
     # whisper, matter-server, etc.
@@ -253,43 +230,20 @@ spec:
         - fields:
             - { type: 0,   family: IPv4 }
             - { type: 3,   family: IPv4 }
-            - { type: 4,   family: IPv4 }
-            - { type: 5,   family: IPv4 }
             - { type: 8,   family: IPv4 }
-            - { type: 9,   family: IPv4 }
-            - { type: 10,  family: IPv4 }
             - { type: 11,  family: IPv4 }
             - { type: 12,  family: IPv4 }
-            - { type: 13,  family: IPv4 }
-            - { type: 14,  family: IPv4 }
-            - { type: 15,  family: IPv4 }
-            - { type: 16,  family: IPv4 }
-            - { type: 17,  family: IPv4 }
-            - { type: 18,  family: IPv4 }
-            - { type: 30,  family: IPv4 }
             - { type: 1,   family: IPv6 }
             - { type: 2,   family: IPv6 }
             - { type: 3,   family: IPv6 }
             - { type: 4,   family: IPv6 }
             - { type: 128, family: IPv6 }
             - { type: 129, family: IPv6 }
-            - { type: 130, family: IPv6 }
-            - { type: 131, family: IPv6 }
-            - { type: 132, family: IPv6 }
             - { type: 133, family: IPv6 }
             - { type: 134, family: IPv6 }
             - { type: 135, family: IPv6 }
             - { type: 136, family: IPv6 }
-            - { type: 137, family: IPv6 }
-            - { type: 138, family: IPv6 }
-            - { type: 141, family: IPv6 }
-            - { type: 142, family: IPv6 }
             - { type: 143, family: IPv6 }
-            - { type: 148, family: IPv6 }
-            - { type: 149, family: IPv6 }
-            - { type: 151, family: IPv6 }
-            - { type: 152, family: IPv6 }
-            - { type: 153, family: IPv6 }
 ---
 # Scoped world ingress — replaces the former namespace-wide `fromEntities:
 # world` allow (see the note in the default-deny-ingress policy above). IoT-VLAN
@@ -404,3 +358,67 @@ spec:
         - ports:
             - port: "8123"
               protocol: TCP
+---
+# ha-pg (CNPG) — instance-manager access for the CloudNativePG operator.
+#
+# DEDICATED policy with its own endpointSelector, NOT another rule on the
+# namespace-wide default-deny above. That policy has endpointSelector:{}, so any
+# rule added there applies to EVERY pod in the namespace — and :8000 is served
+# both by the CNPG instance manager and by other workloads, so a namespace-wide
+# :8000 grant would expose far more than intended. Same reasoning as
+# wp-pg-allow-cnpg-operator in woodpecker-ns.yaml, which was corrected during
+# review for exactly this (#433).
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: ha-pg-allow-cnpg-operator
+  namespace: home-assistant
+spec:
+  endpointSelector:
+    matchLabels:
+      cnpg.io/cluster: ha-pg
+  ingress:
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": database
+            app.kubernetes.io/name: cloudnative-pg
+      toPorts:
+        - ports:
+            - port: "8000"
+              protocol: TCP
+    # ICMP to the ha-pg pods. REQUIRED because the L4 rules in this policy
+    # select those endpoints, and Cilium drops ICMP to an endpoint selected by
+    # any layer 4 policy unless the traffic is CT-related. Without this, PMTUD
+    # Destination Unreachable / Packet Too Big toward ha-pg are dropped
+    # silently, and the symptom is stalled transfers rather than a policy
+    # denial — the failure mode documented in woodpecker-ns.yaml.
+    - fromEntities:
+        - cluster
+      icmps:
+        - fields:
+            - { type: 3,   family: IPv4 }   # Destination Unreachable — PMTUD
+            - { type: 1,   family: IPv6 }   # Destination Unreachable
+            - { type: 2,   family: IPv6 }   # Packet Too Big — PMTUD, mandatory
+---
+# ha-pg — Postgres access for the Home Assistant recorder.
+#
+# Scoped to the HA pod on both ends: only home-assistant reaches ha-pg, and only
+# on 5432. Nothing else in this namespace has any business in the recorder
+# database.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: ha-pg-allow-home-assistant
+  namespace: home-assistant
+spec:
+  endpointSelector:
+    matchLabels:
+      cnpg.io/cluster: ha-pg
+  ingress:
+    - fromEndpoints:
+        - matchLabels:
+            app.kubernetes.io/name: home-assistant
+      toPorts:
+        - ports:
+            - port: "5432"
+              protocol: TCP

kubernetes/infrastructure/network-policies/immich-ns.yaml

@@ -42,40 +42,17 @@ spec:
         - fields:
             - { type: 0,   family: IPv4 }
             - { type: 3,   family: IPv4 }
-            - { type: 4,   family: IPv4 }
-            - { type: 5,   family: IPv4 }
             - { type: 8,   family: IPv4 }
-            - { type: 9,   family: IPv4 }
-            - { type: 10,  family: IPv4 }
             - { type: 11,  family: IPv4 }
             - { type: 12,  family: IPv4 }
-            - { type: 13,  family: IPv4 }
-            - { type: 14,  family: IPv4 }
-            - { type: 15,  family: IPv4 }
-            - { type: 16,  family: IPv4 }
-            - { type: 17,  family: IPv4 }
-            - { type: 18,  family: IPv4 }
-            - { type: 30,  family: IPv4 }
             - { type: 1,   family: IPv6 }
             - { type: 2,   family: IPv6 }
             - { type: 3,   family: IPv6 }
             - { type: 4,   family: IPv6 }
             - { type: 128, family: IPv6 }
             - { type: 129, family: IPv6 }
-            - { type: 130, family: IPv6 }
-            - { type: 131, family: IPv6 }
-            - { type: 132, family: IPv6 }
             - { type: 133, family: IPv6 }
             - { type: 134, family: IPv6 }
             - { type: 135, family: IPv6 }
             - { type: 136, family: IPv6 }
-            - { type: 137, family: IPv6 }
-            - { type: 138, family: IPv6 }
-            - { type: 141, family: IPv6 }
-            - { type: 142, family: IPv6 }
             - { type: 143, family: IPv6 }
-            - { type: 148, family: IPv6 }
-            - { type: 149, family: IPv6 }
-            - { type: 151, family: IPv6 }
-            - { type: 152, family: IPv6 }
-            - { type: 153, family: IPv6 }

kubernetes/infrastructure/network-policies/influxdb-ns.yaml

@@ -42,40 +42,17 @@ spec:
         - fields:
             - { type: 0,   family: IPv4 }
             - { type: 3,   family: IPv4 }
-            - { type: 4,   family: IPv4 }
-            - { type: 5,   family: IPv4 }
             - { type: 8,   family: IPv4 }
-            - { type: 9,   family: IPv4 }
-            - { type: 10,  family: IPv4 }
             - { type: 11,  family: IPv4 }
             - { type: 12,  family: IPv4 }
-            - { type: 13,  family: IPv4 }
-            - { type: 14,  family: IPv4 }
-            - { type: 15,  family: IPv4 }
-            - { type: 16,  family: IPv4 }
-            - { type: 17,  family: IPv4 }
-            - { type: 18,  family: IPv4 }
-            - { type: 30,  family: IPv4 }
             - { type: 1,   family: IPv6 }
             - { type: 2,   family: IPv6 }
             - { type: 3,   family: IPv6 }
             - { type: 4,   family: IPv6 }
             - { type: 128, family: IPv6 }
             - { type: 129, family: IPv6 }
-            - { type: 130, family: IPv6 }
-            - { type: 131, family: IPv6 }
-            - { type: 132, family: IPv6 }
             - { type: 133, family: IPv6 }
             - { type: 134, family: IPv6 }
             - { type: 135, family: IPv6 }
             - { type: 136, family: IPv6 }
-            - { type: 137, family: IPv6 }
-            - { type: 138, family: IPv6 }
-            - { type: 141, family: IPv6 }
-            - { type: 142, family: IPv6 }
             - { type: 143, family: IPv6 }
-            - { type: 148, family: IPv6 }
-            - { type: 149, family: IPv6 }
-            - { type: 151, family: IPv6 }
-            - { type: 152, family: IPv6 }
-            - { type: 153, family: IPv6 }

kubernetes/infrastructure/network-policies/kustomization.yaml

@@ -21,6 +21,7 @@ resources:
   - openbao-ns.yaml
   - external-secrets-ns.yaml
   - woodpecker-ci-ns.yaml
+  - woodpecker-ns.yaml
   - trmnl-ns.yaml
   - wordpress-ns.yaml
   - allow-cilium-health.yaml

kubernetes/infrastructure/network-policies/monitoring-ns.yaml

@@ -67,40 +67,17 @@ spec:
         - fields:
             - { type: 0,   family: IPv4 }
             - { type: 3,   family: IPv4 }
-            - { type: 4,   family: IPv4 }
-            - { type: 5,   family: IPv4 }
             - { type: 8,   family: IPv4 }
-            - { type: 9,   family: IPv4 }
-            - { type: 10,  family: IPv4 }
             - { type: 11,  family: IPv4 }
             - { type: 12,  family: IPv4 }
-            - { type: 13,  family: IPv4 }
-            - { type: 14,  family: IPv4 }
-            - { type: 15,  family: IPv4 }
-            - { type: 16,  family: IPv4 }
-            - { type: 17,  family: IPv4 }
-            - { type: 18,  family: IPv4 }
-            - { type: 30,  family: IPv4 }
             - { type: 1,   family: IPv6 }
             - { type: 2,   family: IPv6 }
             - { type: 3,   family: IPv6 }
             - { type: 4,   family: IPv6 }
             - { type: 128, family: IPv6 }
             - { type: 129, family: IPv6 }
-            - { type: 130, family: IPv6 }
-            - { type: 131, family: IPv6 }
-            - { type: 132, family: IPv6 }
             - { type: 133, family: IPv6 }
             - { type: 134, family: IPv6 }
             - { type: 135, family: IPv6 }
             - { type: 136, family: IPv6 }
-            - { type: 137, family: IPv6 }
-            - { type: 138, family: IPv6 }
-            - { type: 141, family: IPv6 }
-            - { type: 142, family: IPv6 }
             - { type: 143, family: IPv6 }
-            - { type: 148, family: IPv6 }
-            - { type: 149, family: IPv6 }
-            - { type: 151, family: IPv6 }
-            - { type: 152, family: IPv6 }
-            - { type: 153, family: IPv6 }

kubernetes/infrastructure/network-policies/openbao-ns.yaml

@@ -141,9 +141,9 @@ spec:
 ---
 # Egress for the review-bot rotator. Its CronJob has two steps: the init step
 # mints review-bot's PAT against gitea (apps :3000), the main step upserts the
-# Woodpecker org secret against the woodpecker server (apps :8000). Both run
+# Woodpecker org secret against the woodpecker server (woodpecker :8000). Both run
 # under the same pod labels, so one egress policy covers both hops. Complements
-# gitea-allow-review-bot-rotator + woodpecker-allow-rotators (apps side).
+# gitea-allow-review-bot-rotator (apps) + woodpecker-allow-rotators (woodpecker).
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
 metadata:
@@ -165,16 +165,16 @@ spec:
               protocol: TCP
     - toEndpoints:
         - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": apps
+            "k8s:io.kubernetes.pod.namespace": woodpecker
             app.kubernetes.io/name: woodpecker
       toPorts:
         - ports:
             - port: "8000"
               protocol: TCP
 ---
-# Egress for the claude-oauth sync rotator: woodpecker server only (apps :8000).
+# Egress for the claude-oauth sync rotator: woodpecker server only (woodpecker :8000).
 # It never touches gitea (the OAuth value is seeded into OpenBao by hand, not
-# minted). Complements woodpecker-allow-rotators (apps side).
+# minted). Complements woodpecker-allow-rotators (woodpecker side).
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
 metadata:
@@ -188,7 +188,7 @@ spec:
   egress:
     - toEndpoints:
         - matchLabels:
-            "k8s:io.kubernetes.pod.namespace": apps
+            "k8s:io.kubernetes.pod.namespace": woodpecker
             app.kubernetes.io/name: woodpecker
       toPorts:
         - ports:

kubernetes/infrastructure/network-policies/paperless-ns.yaml

@@ -37,43 +37,20 @@ spec:
         - fields:
             - { type: 0,   family: IPv4 }
             - { type: 3,   family: IPv4 }
-            - { type: 4,   family: IPv4 }
-            - { type: 5,   family: IPv4 }
             - { type: 8,   family: IPv4 }
-            - { type: 9,   family: IPv4 }
-            - { type: 10,  family: IPv4 }
             - { type: 11,  family: IPv4 }
             - { type: 12,  family: IPv4 }
-            - { type: 13,  family: IPv4 }
-            - { type: 14,  family: IPv4 }
-            - { type: 15,  family: IPv4 }
-            - { type: 16,  family: IPv4 }
-            - { type: 17,  family: IPv4 }
-            - { type: 18,  family: IPv4 }
-            - { type: 30,  family: IPv4 }
             - { type: 1,   family: IPv6 }
             - { type: 2,   family: IPv6 }
             - { type: 3,   family: IPv6 }
             - { type: 4,   family: IPv6 }
             - { type: 128, family: IPv6 }
             - { type: 129, family: IPv6 }
-            - { type: 130, family: IPv6 }
-            - { type: 131, family: IPv6 }
-            - { type: 132, family: IPv6 }
             - { type: 133, family: IPv6 }
             - { type: 134, family: IPv6 }
             - { type: 135, family: IPv6 }
             - { type: 136, family: IPv6 }
-            - { type: 137, family: IPv6 }
-            - { type: 138, family: IPv6 }
-            - { type: 141, family: IPv6 }
-            - { type: 142, family: IPv6 }
             - { type: 143, family: IPv6 }
-            - { type: 148, family: IPv6 }
-            - { type: 149, family: IPv6 }
-            - { type: 151, family: IPv6 }
-            - { type: 152, family: IPv6 }
-            - { type: 153, family: IPv6 }
   egress:
     - toEndpoints:
         - matchLabels:
@@ -109,43 +86,20 @@ spec:
         - fields:
             - { type: 0,   family: IPv4 }
             - { type: 3,   family: IPv4 }
-            - { type: 4,   family: IPv4 }
-            - { type: 5,   family: IPv4 }
             - { type: 8,   family: IPv4 }
-            - { type: 9,   family: IPv4 }
-            - { type: 10,  family: IPv4 }
             - { type: 11,  family: IPv4 }
             - { type: 12,  family: IPv4 }
-            - { type: 13,  family: IPv4 }
-            - { type: 14,  family: IPv4 }
-            - { type: 15,  family: IPv4 }
-            - { type: 16,  family: IPv4 }
-            - { type: 17,  family: IPv4 }
-            - { type: 18,  family: IPv4 }
-            - { type: 30,  family: IPv4 }
             - { type: 1,   family: IPv6 }
             - { type: 2,   family: IPv6 }
             - { type: 3,   family: IPv6 }
             - { type: 4,   family: IPv6 }
             - { type: 128, family: IPv6 }
             - { type: 129, family: IPv6 }
-            - { type: 130, family: IPv6 }
-            - { type: 131, family: IPv6 }
-            - { type: 132, family: IPv6 }
             - { type: 133, family: IPv6 }
             - { type: 134, family: IPv6 }
             - { type: 135, family: IPv6 }
             - { type: 136, family: IPv6 }
-            - { type: 137, family: IPv6 }
-            - { type: 138, family: IPv6 }
-            - { type: 141, family: IPv6 }
-            - { type: 142, family: IPv6 }
             - { type: 143, family: IPv6 }
-            - { type: 148, family: IPv6 }
-            - { type: 149, family: IPv6 }
-            - { type: 151, family: IPv6 }
-            - { type: 152, family: IPv6 }
-            - { type: 153, family: IPv6 }
 ---
 # Allow FTP access (from scanners on the LAN) to the ftp-scanner pod.
 apiVersion: cilium.io/v2

kubernetes/infrastructure/network-policies/rook-ceph-ns.yaml

@@ -41,40 +41,17 @@ spec:
         - fields:
             - { type: 0,   family: IPv4 }
             - { type: 3,   family: IPv4 }
-            - { type: 4,   family: IPv4 }
-            - { type: 5,   family: IPv4 }
             - { type: 8,   family: IPv4 }
-            - { type: 9,   family: IPv4 }
-            - { type: 10,  family: IPv4 }
             - { type: 11,  family: IPv4 }
             - { type: 12,  family: IPv4 }
-            - { type: 13,  family: IPv4 }
-            - { type: 14,  family: IPv4 }
-            - { type: 15,  family: IPv4 }
-            - { type: 16,  family: IPv4 }
-            - { type: 17,  family: IPv4 }
-            - { type: 18,  family: IPv4 }
-            - { type: 30,  family: IPv4 }
             - { type: 1,   family: IPv6 }
             - { type: 2,   family: IPv6 }
             - { type: 3,   family: IPv6 }
             - { type: 4,   family: IPv6 }
             - { type: 128, family: IPv6 }
             - { type: 129, family: IPv6 }
-            - { type: 130, family: IPv6 }
-            - { type: 131, family: IPv6 }
-            - { type: 132, family: IPv6 }
             - { type: 133, family: IPv6 }
             - { type: 134, family: IPv6 }
             - { type: 135, family: IPv6 }
             - { type: 136, family: IPv6 }
-            - { type: 137, family: IPv6 }
-            - { type: 138, family: IPv6 }
-            - { type: 141, family: IPv6 }
-            - { type: 142, family: IPv6 }
             - { type: 143, family: IPv6 }
-            - { type: 148, family: IPv6 }
-            - { type: 149, family: IPv6 }
-            - { type: 151, family: IPv6 }
-            - { type: 152, family: IPv6 }
-            - { type: 153, family: IPv6 }

kubernetes/infrastructure/network-policies/spegel-ns.yaml

@@ -26,43 +26,20 @@ spec:
         - fields:
             - { type: 0,   family: IPv4 }
             - { type: 3,   family: IPv4 }
-            - { type: 4,   family: IPv4 }
-            - { type: 5,   family: IPv4 }
             - { type: 8,   family: IPv4 }
-            - { type: 9,   family: IPv4 }
-            - { type: 10,  family: IPv4 }
             - { type: 11,  family: IPv4 }
             - { type: 12,  family: IPv4 }
-            - { type: 13,  family: IPv4 }
-            - { type: 14,  family: IPv4 }
-            - { type: 15,  family: IPv4 }
-            - { type: 16,  family: IPv4 }
-            - { type: 17,  family: IPv4 }
-            - { type: 18,  family: IPv4 }
-            - { type: 30,  family: IPv4 }
             - { type: 1,   family: IPv6 }
             - { type: 2,   family: IPv6 }
             - { type: 3,   family: IPv6 }
             - { type: 4,   family: IPv6 }
             - { type: 128, family: IPv6 }
             - { type: 129, family: IPv6 }
-            - { type: 130, family: IPv6 }
-            - { type: 131, family: IPv6 }
-            - { type: 132, family: IPv6 }
             - { type: 133, family: IPv6 }
             - { type: 134, family: IPv6 }
             - { type: 135, family: IPv6 }
             - { type: 136, family: IPv6 }
-            - { type: 137, family: IPv6 }
-            - { type: 138, family: IPv6 }
-            - { type: 141, family: IPv6 }
-            - { type: 142, family: IPv6 }
             - { type: 143, family: IPv6 }
-            - { type: 148, family: IPv6 }
-            - { type: 149, family: IPv6 }
-            - { type: 151, family: IPv6 }
-            - { type: 152, family: IPv6 }
-            - { type: 153, family: IPv6 }
   egress:
     # Peer P2P
     - toEndpoints:
@@ -92,40 +69,17 @@ spec:
         - fields:
             - { type: 0,   family: IPv4 }
             - { type: 3,   family: IPv4 }
-            - { type: 4,   family: IPv4 }
-            - { type: 5,   family: IPv4 }
             - { type: 8,   family: IPv4 }
-            - { type: 9,   family: IPv4 }
-            - { type: 10,  family: IPv4 }
             - { type: 11,  family: IPv4 }
             - { type: 12,  family: IPv4 }
-            - { type: 13,  family: IPv4 }
-            - { type: 14,  family: IPv4 }
-            - { type: 15,  family: IPv4 }
-            - { type: 16,  family: IPv4 }
-            - { type: 17,  family: IPv4 }
-            - { type: 18,  family: IPv4 }
-            - { type: 30,  family: IPv4 }
             - { type: 1,   family: IPv6 }
             - { type: 2,   family: IPv6 }
             - { type: 3,   family: IPv6 }
             - { type: 4,   family: IPv6 }
             - { type: 128, family: IPv6 }
             - { type: 129, family: IPv6 }
-            - { type: 130, family: IPv6 }
-            - { type: 131, family: IPv6 }
-            - { type: 132, family: IPv6 }
             - { type: 133, family: IPv6 }
             - { type: 134, family: IPv6 }
             - { type: 135, family: IPv6 }
             - { type: 136, family: IPv6 }
-            - { type: 137, family: IPv6 }
-            - { type: 138, family: IPv6 }
-            - { type: 141, family: IPv6 }
-            - { type: 142, family: IPv6 }
             - { type: 143, family: IPv6 }
-            - { type: 148, family: IPv6 }
-            - { type: 149, family: IPv6 }
-            - { type: 151, family: IPv6 }
-            - { type: 152, family: IPv6 }
-            - { type: 153, family: IPv6 }

kubernetes/infrastructure/network-policies/trmnl-ns.yaml

@@ -42,43 +42,20 @@ spec:
         - fields:
             - { type: 0,   family: IPv4 }
             - { type: 3,   family: IPv4 }
-            - { type: 4,   family: IPv4 }
-            - { type: 5,   family: IPv4 }
             - { type: 8,   family: IPv4 }
-            - { type: 9,   family: IPv4 }
-            - { type: 10,  family: IPv4 }
             - { type: 11,  family: IPv4 }
             - { type: 12,  family: IPv4 }
-            - { type: 13,  family: IPv4 }
-            - { type: 14,  family: IPv4 }
-            - { type: 15,  family: IPv4 }
-            - { type: 16,  family: IPv4 }
-            - { type: 17,  family: IPv4 }
-            - { type: 18,  family: IPv4 }
-            - { type: 30,  family: IPv4 }
             - { type: 1,   family: IPv6 }
             - { type: 2,   family: IPv6 }
             - { type: 3,   family: IPv6 }
             - { type: 4,   family: IPv6 }
             - { type: 128, family: IPv6 }
             - { type: 129, family: IPv6 }
-            - { type: 130, family: IPv6 }
-            - { type: 131, family: IPv6 }
-            - { type: 132, family: IPv6 }
             - { type: 133, family: IPv6 }
             - { type: 134, family: IPv6 }
             - { type: 135, family: IPv6 }
             - { type: 136, family: IPv6 }
-            - { type: 137, family: IPv6 }
-            - { type: 138, family: IPv6 }
-            - { type: 141, family: IPv6 }
-            - { type: 142, family: IPv6 }
             - { type: 143, family: IPv6 }
-            - { type: 148, family: IPv6 }
-            - { type: 149, family: IPv6 }
-            - { type: 151, family: IPv6 }
-            - { type: 152, family: IPv6 }
-            - { type: 153, family: IPv6 }
 ---
 # The TRMNL panel polls /api/display over cleartext HTTP on the
 # terminus-device LoadBalancer. The firmware cannot be handed the internal

kubernetes/infrastructure/network-policies/velero-ns.yaml

@@ -28,40 +28,17 @@ spec:
         - fields:
             - { type: 0,   family: IPv4 }
             - { type: 3,   family: IPv4 }
-            - { type: 4,   family: IPv4 }
-            - { type: 5,   family: IPv4 }
             - { type: 8,   family: IPv4 }
-            - { type: 9,   family: IPv4 }
-            - { type: 10,  family: IPv4 }
             - { type: 11,  family: IPv4 }
             - { type: 12,  family: IPv4 }
-            - { type: 13,  family: IPv4 }
-            - { type: 14,  family: IPv4 }
-            - { type: 15,  family: IPv4 }
-            - { type: 16,  family: IPv4 }
-            - { type: 17,  family: IPv4 }
-            - { type: 18,  family: IPv4 }
-            - { type: 30,  family: IPv4 }
             - { type: 1,   family: IPv6 }
             - { type: 2,   family: IPv6 }
             - { type: 3,   family: IPv6 }
             - { type: 4,   family: IPv6 }
             - { type: 128, family: IPv6 }
             - { type: 129, family: IPv6 }
-            - { type: 130, family: IPv6 }
-            - { type: 131, family: IPv6 }
-            - { type: 132, family: IPv6 }
             - { type: 133, family: IPv6 }
             - { type: 134, family: IPv6 }
             - { type: 135, family: IPv6 }
             - { type: 136, family: IPv6 }
-            - { type: 137, family: IPv6 }
-            - { type: 138, family: IPv6 }
-            - { type: 141, family: IPv6 }
-            - { type: 142, family: IPv6 }
             - { type: 143, family: IPv6 }
-            - { type: 148, family: IPv6 }
-            - { type: 149, family: IPv6 }
-            - { type: 151, family: IPv6 }
-            - { type: 152, family: IPv6 }
-            - { type: 153, family: IPv6 }

kubernetes/infrastructure/network-policies/woodpecker-ci-ns.yaml

@@ -45,40 +45,17 @@ spec:
         - fields:
             - { type: 0,   family: IPv4 }
             - { type: 3,   family: IPv4 }
-            - { type: 4,   family: IPv4 }
-            - { type: 5,   family: IPv4 }
             - { type: 8,   family: IPv4 }
-            - { type: 9,   family: IPv4 }
-            - { type: 10,  family: IPv4 }
             - { type: 11,  family: IPv4 }
             - { type: 12,  family: IPv4 }
-            - { type: 13,  family: IPv4 }
-            - { type: 14,  family: IPv4 }
-            - { type: 15,  family: IPv4 }
-            - { type: 16,  family: IPv4 }
-            - { type: 17,  family: IPv4 }
-            - { type: 18,  family: IPv4 }
-            - { type: 30,  family: IPv4 }
             - { type: 1,   family: IPv6 }
             - { type: 2,   family: IPv6 }
             - { type: 3,   family: IPv6 }
             - { type: 4,   family: IPv6 }
             - { type: 128, family: IPv6 }
             - { type: 129, family: IPv6 }
-            - { type: 130, family: IPv6 }
-            - { type: 131, family: IPv6 }
-            - { type: 132, family: IPv6 }
             - { type: 133, family: IPv6 }
             - { type: 134, family: IPv6 }
             - { type: 135, family: IPv6 }
             - { type: 136, family: IPv6 }
-            - { type: 137, family: IPv6 }
-            - { type: 138, family: IPv6 }
-            - { type: 141, family: IPv6 }
-            - { type: 142, family: IPv6 }
             - { type: 143, family: IPv6 }
-            - { type: 148, family: IPv6 }
-            - { type: 149, family: IPv6 }
-            - { type: 151, family: IPv6 }
-            - { type: 152, family: IPv6 }
-            - { type: 153, family: IPv6 }

kubernetes/infrastructure/network-policies/woodpecker-ns.yaml

@@ -0,0 +1,156 @@
+---
+# Baseline ingress policy for the `woodpecker` namespace (wp-pg now; server and
+# agent arrive in stage 2).
+#
+# Unlike the `apps` baseline this replaces, every allow here lands on a
+# namespace holding one service, so a namespace-scoped grant is not a blanket
+# grant across unrelated workloads. That is the point of the move: in `apps`,
+# letting the CNPG operator reach wp-pg would also have opened gitea, gatus,
+# external-dns and ical-proxy to the whole database namespace (issue #433).
+#
+# CAUTION when adding rules here: this policy has `endpointSelector: {}`, so
+# every rule below applies to EVERY pod in the namespace. A `toPorts` block
+# narrows the port, NOT the destination workload — that is governed solely by
+# the enclosing policy's endpointSelector. Anything meant for one workload
+# belongs in its own policy with its own selector, as
+# `wp-pg-allow-cnpg-operator` below does.
+#
+# No egress section, so egress is unrestricted. Deliberate for now and revisited
+# in stage 2: wp-pg needs DNS plus `world` for WAL archiving to Hetzner S3, and
+# the server/agent add gitea, the Kubernetes API and outbound plugin pulls.
+# Writing that ruleset once, with the full workload set present, beats guessing
+# it here and silently breaking the WAL archiving this stage exists to verify.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: default-deny-ingress
+  namespace: woodpecker
+spec:
+  endpointSelector: {}
+  ingress:
+    # Intra-namespace: server <-> agent, server -> wp-pg.
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": woodpecker
+    # NOT here: ingress from the `ingress` namespace. Traefik terminates
+    # ci.renner.uno and forwards to the server, but that namespace also holds
+    # netbird-ingress, and this policy's endpointSelector:{} would extend the
+    # grant to wp-pg and the agent as well. Scoped to traefik -> server in the
+    # companion policy at the bottom of this file.
+    # Prometheus scrape. Namespace-wide and port-unrestricted, matching the
+    # other *-ns baselines; this is what covers wp-pg's :9187 metrics.
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": monitoring
+    # Kubelet probes and hostNetwork siblings.
+    - fromEntities:
+        - host
+        - remote-node
+    # ICMP — from any cluster pod. See home-assistant-ns.yaml for the rationale.
+    # INGRESS ONLY: do NOT add an egress section to this CNP.
+    #
+    # Needed here specifically because wp-pg-allow-cnpg-operator below attaches
+    # an L4 rule (toPorts :8000) to the wp-pg pods, and Cilium blocks ICMP to an
+    # endpoint once any layer 4 policy selects it unless the traffic is
+    # CT-related. Without this, PMTUD Destination Unreachable and Packet Too Big
+    # toward wp-pg are dropped silently — the failure shows up as stalled
+    # transfers, not as a policy denial.
+    #
+    # Namespace-wide is right for this one, unlike the :8000 case noted below:
+    # these are diagnostic and path-MTU messages that expose no API surface, so
+    # granting them across the namespace concedes nothing a per-workload rule
+    # would withhold.
+    - fromEndpoints:
+        - matchExpressions:
+            - key: io.kubernetes.pod.namespace
+              operator: Exists
+      icmps:
+        - fields:
+            - { type: 0,   family: IPv4 }
+            - { type: 3,   family: IPv4 }
+            - { type: 8,   family: IPv4 }
+            - { type: 11,  family: IPv4 }
+            - { type: 12,  family: IPv4 }
+            - { type: 1,   family: IPv6 }
+            - { type: 2,   family: IPv6 }
+            - { type: 3,   family: IPv6 }
+            - { type: 4,   family: IPv6 }
+            - { type: 128, family: IPv6 }
+            - { type: 129, family: IPv6 }
+            - { type: 133, family: IPv6 }
+            - { type: 134, family: IPv6 }
+            - { type: 135, family: IPv6 }
+            - { type: 136, family: IPv6 }
+            - { type: 143, family: IPv6 }
+    #
+    # NOT here: ingress from `woodpecker-ci`. Pipeline step pods do call the
+    # server back on :8000 to stream logs, but the CNPG instance manager serves
+    # its status API on :8000 too — so granting that on this endpointSelector:{}
+    # policy would hand arbitrary repo-supplied build steps a route to wp-pg's
+    # instance manager. It lands in stage 2 as its own policy selecting the
+    # server pod, the way woodpecker-allow-rotators already scopes its :8000
+    # rule in apps/woodpecker/networkpolicy.yaml.
+---
+# The CNPG operator polls each instance's status endpoint over HTTPS
+# (https://<pod-ip>:8000/pg/status) to populate Cluster status. Without it the
+# Cluster reports "Instance Status Extraction Error: HTTP communication issue"
+# while Postgres serves normally, and the operator loses failover and
+# switchover decisions — issue #433, where the operator's dial timed out
+# because the apps baseline never allowed the database namespace.
+#
+# Scoped to the wp-pg endpoints and that one port: the operator has no reason to
+# reach the woodpecker server or agent, and the barman and mariadb operators
+# sharing its namespace have no reason to reach anything here at all.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: wp-pg-allow-cnpg-operator
+  namespace: woodpecker
+spec:
+  endpointSelector:
+    matchLabels:
+      cnpg.io/cluster: wp-pg
+  ingress:
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": database
+            app.kubernetes.io/name: cloudnative-pg
+      toPorts:
+        - ports:
+            - port: "8000"
+              protocol: TCP
+---
+# Traefik terminates ci.renner.uno and forwards to the server on :8000.
+#
+# Lives here, beside the baseline, rather than with the workload policies in
+# apps/woodpecker/networkpolicy.yaml — deliberately. The baseline's `ingress`
+# admit was removed in the same commit, and these two ship in the same
+# Kustomization (infrastructure-crds), so the swap lands in one reconcile. Split
+# across Kustomizations it would not: infrastructure-crds reconciles before the
+# `woodpecker` KS, so the admit would be withdrawn minutes before its
+# replacement arrived, and ci.renner.uno would 503 in the gap.
+#
+# Scoped to traefik and to the server pod. The `ingress` namespace also holds
+# netbird-ingress, and the baseline's endpointSelector:{} would have extended a
+# namespace-level admit to wp-pg and the agent. The `apps` policy this replaces
+# (allow-ingress.yaml) matches traefik by label for the same reason, and does
+# NOT apply here — it is hard-scoped to `namespace: apps`.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: woodpecker-allow-traefik
+  namespace: woodpecker
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: woodpecker
+      app.kubernetes.io/component: server
+  ingress:
+    - fromEndpoints:
+        - matchLabels:
+            "k8s:io.kubernetes.pod.namespace": ingress
+            app.kubernetes.io/name: traefik
+      toPorts:
+        - ports:
+            - port: "8000"
+              protocol: TCP

kubernetes/infrastructure/network-policies/wordpress-ns.yaml

@@ -53,43 +53,20 @@ spec:
         - fields:
             - { type: 0,   family: IPv4 }
             - { type: 3,   family: IPv4 }
-            - { type: 4,   family: IPv4 }
-            - { type: 5,   family: IPv4 }
             - { type: 8,   family: IPv4 }
-            - { type: 9,   family: IPv4 }
-            - { type: 10,  family: IPv4 }
             - { type: 11,  family: IPv4 }
             - { type: 12,  family: IPv4 }
-            - { type: 13,  family: IPv4 }
-            - { type: 14,  family: IPv4 }
-            - { type: 15,  family: IPv4 }
-            - { type: 16,  family: IPv4 }
-            - { type: 17,  family: IPv4 }
-            - { type: 18,  family: IPv4 }
-            - { type: 30,  family: IPv4 }
             - { type: 1,   family: IPv6 }
             - { type: 2,   family: IPv6 }
             - { type: 3,   family: IPv6 }
             - { type: 4,   family: IPv6 }
             - { type: 128, family: IPv6 }
             - { type: 129, family: IPv6 }
-            - { type: 130, family: IPv6 }
-            - { type: 131, family: IPv6 }
-            - { type: 132, family: IPv6 }
             - { type: 133, family: IPv6 }
             - { type: 134, family: IPv6 }
             - { type: 135, family: IPv6 }
             - { type: 136, family: IPv6 }
-            - { type: 137, family: IPv6 }
-            - { type: 138, family: IPv6 }
-            - { type: 141, family: IPv6 }
-            - { type: 142, family: IPv6 }
             - { type: 143, family: IPv6 }
-            - { type: 148, family: IPv6 }
-            - { type: 149, family: IPv6 }
-            - { type: 151, family: IPv6 }
-            - { type: 152, family: IPv6 }
-            - { type: 153, family: IPv6 }
   egress:
     - toEndpoints:
         - matchLabels:
@@ -128,40 +105,17 @@ spec:
         - fields:
             - { type: 0,   family: IPv4 }
             - { type: 3,   family: IPv4 }
-            - { type: 4,   family: IPv4 }
-            - { type: 5,   family: IPv4 }
             - { type: 8,   family: IPv4 }
-            - { type: 9,   family: IPv4 }
-            - { type: 10,  family: IPv4 }
             - { type: 11,  family: IPv4 }
             - { type: 12,  family: IPv4 }
-            - { type: 13,  family: IPv4 }
-            - { type: 14,  family: IPv4 }
-            - { type: 15,  family: IPv4 }
-            - { type: 16,  family: IPv4 }
-            - { type: 17,  family: IPv4 }
-            - { type: 18,  family: IPv4 }
-            - { type: 30,  family: IPv4 }
             - { type: 1,   family: IPv6 }
             - { type: 2,   family: IPv6 }
             - { type: 3,   family: IPv6 }
             - { type: 4,   family: IPv6 }
             - { type: 128, family: IPv6 }
             - { type: 129, family: IPv6 }
-            - { type: 130, family: IPv6 }
-            - { type: 131, family: IPv6 }
-            - { type: 132, family: IPv6 }
             - { type: 133, family: IPv6 }
             - { type: 134, family: IPv6 }
             - { type: 135, family: IPv6 }
             - { type: 136, family: IPv6 }
-            - { type: 137, family: IPv6 }
-            - { type: 138, family: IPv6 }
-            - { type: 141, family: IPv6 }
-            - { type: 142, family: IPv6 }
             - { type: 143, family: IPv6 }
-            - { type: 148, family: IPv6 }
-            - { type: 149, family: IPv6 }
-            - { type: 151, family: IPv6 }
-            - { type: 152, family: IPv6 }
-            - { type: 153, family: IPv6 }

kubernetes/infrastructure/openbao/rotation/claude-oauth.yaml

@@ -14,8 +14,10 @@
 # immediately:
 #   kubectl -n openbao create job --from=cronjob/claude-oauth-secret-sync claude-oauth-sync-now
 #
-# woodpecker is reached in-cluster at woodpecker.apps.svc:8000. Needs the netpol
-# halves woodpecker-allow-rotators (apps/*) + rotator-claude-oauth-egress
+# woodpecker is reached in-cluster at woodpecker.woodpecker.svc:8000 (it moved
+# out of `apps` in the namespace migration — see
+# kubernetes/apps/woodpecker/MIGRATION-namespace.md). Needs the netpol
+# halves woodpecker-allow-rotators (woodpecker ns) + rotator-claude-oauth-egress
 # (openbao-ns.yaml). OpenBao policy/role rotator-claude-oauth + the bootstraps
 # (woodpecker-bot token, the seeded OAuth value) are provisioned by hand — see
 # policies/rotator-claude-oauth.hcl.
@@ -89,7 +91,7 @@ spec:
                 - name: BAO_FIELD
                   value: setup-token
                 - name: WOODPECKER_URL
-                  value: http://woodpecker.apps.svc.cluster.local:8000
+                  value: http://woodpecker.woodpecker.svc.cluster.local:8000
                 - name: WOODPECKER_ORG
                   value: k.renner
                 - name: WOODPECKER_SECRET_NAME

kubernetes/infrastructure/openbao/rotation/review-bot.yaml

@@ -25,9 +25,10 @@
 # but a failed sync should be fixed before the next run rather than left.
 #
 # gitea is reached in-cluster at gitea.apps.svc:3000, woodpecker at
-# woodpecker.apps.svc:8000 (both hairpin off the ingress LB from a pod). Needs
+# woodpecker.woodpecker.svc:8000 (it moved out of `apps` in the namespace
+# migration — see kubernetes/apps/woodpecker/MIGRATION-namespace.md). Needs
 # the netpol halves: gitea-allow-review-bot-rotator + woodpecker-allow-rotators
-# (apps/*) and rotator-review-bot-egress (openbao-ns.yaml).
+# (woodpecker ns) and rotator-review-bot-egress (openbao-ns.yaml).
 #
 # OpenBao policy/role rotator-review-bot + the bootstraps (review-bot password,
 # woodpecker-bot token, seed review-bot#token) are provisioned by hand — see
@@ -147,7 +148,7 @@ spec:
                 - name: BAO_FIELD
                   value: token
                 - name: WOODPECKER_URL
-                  value: http://woodpecker.apps.svc.cluster.local:8000
+                  value: http://woodpecker.woodpecker.svc.cluster.local:8000
                 - name: WOODPECKER_ORG
                   value: k.renner
                 - name: WOODPECKER_SECRET_NAME

kubernetes/infrastructure/post-crds/openbao-clustersecretstore.yaml

@@ -30,6 +30,7 @@ spec:
         - paperless
         - trmnl
         - velero
+        - woodpecker # wp-pg barman creds; woodpecker moving out of `apps` (#433)
         - wordpress
   provider:
     vault: # ESO's provider name; speaks OpenBao fine

kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml

@@ -5,7 +5,7 @@ metadata:
   namespace: rook-ceph
 spec:
   cephVersion:
-    image: quay.io/ceph/ceph:v19.2.5 # {"$imagepolicy": "flux-system:ceph"}
+    image: quay.io/ceph/ceph:v19.2.6 # {"$imagepolicy": "flux-system:ceph"}
     allowUnsupported: false
   dataDirHostPath: /var/lib/rook
   mon:
@@ -24,18 +24,113 @@ spec:
   # gives headroom for parallel restores during cluster rebuild too
   # (paperless + audiobookshelf + immich + ... all clone at once).
   cephConfig:
+    mon:
+      # Ceph's 30% default assumes the mon has a filesystem to itself, where
+      # 30% free is a meaningful reserve. Here the mons live on
+      # dataDirHostPath /var/lib/rook — the node's /var, shared with ~29 GB
+      # of containerd image and snapshot storage — and the mon DB itself is
+      # ~106 MB. So the default measures container churn, not Ceph: k8s-2
+      # trips MON_DISK_LOW whenever the image cache fills, and image GC does
+      # not clear it (most of /var/lib/containerd is overlayfs snapshots held
+      # by running containers, which GC cannot reclaim — see #434).
+      # 20% of a 60 GB /var still leaves ~12 GB against a 106 MB database.
+      # mon_data_avail_crit stays at the 5% default, and CephMonDiskLow in
+      # monitoring-rules watches /var directly for the real risk, which is the
+      # node filling rather than the mon growing.
+      #
+      # BARE-METAL MIGRATION: revisit this. Give the mons their own volume and
+      # the default 30% becomes meaningful again, at which point this override
+      # should be dropped rather than carried over. It cannot be done on the
+      # current cluster — every StorageClass is Ceph-backed, so a mon PVC
+      # would be circular, and Rook only applies mon.volumeClaimTemplate to
+      # newly created mons, never migrating existing ones.
+      mon_data_avail_warn: "20"
+      # Stop compacting the whole RocksDB prefix on every paxos trim.
+      #
+      # Upstream default is true — "Compact a certain prefix (including paxos)
+      # when we trim its old states" — which is fine at low paxos rates and
+      # pathological here. Measured 2026-08-19:
+      #
+      #   paxos transactions      ~99,000/day  (1.15/s, sampled over 60 s)
+      #   paxos logical data       811 MB/day  (21.9 GB over 27 d uptime)
+      #   ACTUAL mon disk writes    80 GB/day  per mon, 240 GB/day for three
+      #                                        (cgroup io.stat: 2.15 TB/27 d)
+      #   => ~98x write amplification, for a ~106 MB database
+      #
+      # That was ~62% of all OS-disk writes on these nodes and roughly a
+      # quarter of everything the cluster writes — more than every client
+      # workload combined. With paxos_trim_min at 250, ~99k transactions/day
+      # means hundreds of forced range compactions per day, each rewriting the
+      # store.
+      #
+      # The churn itself is 99% cluster log and mgr stats, not real cluster
+      # activity: logm holds 6.7M paxos versions and mgrstat 4.4M, against
+      # osdmap's 9.6k. Every Rook-operator poll (mgr stat / versions / status /
+      # report / osd dump — 23 audit entries per minute per mon) becomes a
+      # cluster-log entry, hence a paxos transaction, replicated to all three
+      # mons. Reducing that polling is the complementary fix and is tracked
+      # separately; this setting removes the amplification regardless of rate.
+      #
+      # This does NOT disable compaction. RocksDB's own levelled compaction
+      # still runs and still reclaims trimmed keys; what stops is the extra
+      # forced compact_range on every trim. Store growth is therefore bounded
+      # by normal compaction, and there are two backstops if that assumption is
+      # wrong: mon_data_size_warn (15 GB default) raises HEALTH_WARN, which
+      # CephHealthWarn in monitoring-rules already alerts on, and
+      # mon_compact_on_start below compacts fully on every mon restart.
+      #
+      # Watch after applying: `du -sh /var/lib/rook/mon-*/store.db` on a node,
+      # or the mon store size in `ceph status`. If it grows past ~1 GB and does
+      # not come back down, revisit rather than leaving it.
+      mon_compact_on_trim: "false"
+      # Compensating control for the above: a full compaction on mon start.
+      # Upstream default is false. Mons restart on upgrades, node reboots and
+      # reschedules, so this gives a periodic full reclaim without paying for
+      # one on every trim. Cheap here — the store is ~106 MB.
+      mon_compact_on_start: "true"
     mgr:
       mgr/volumes/max_concurrent_clones: "16"
-    # QLC tune for the Crucial P2 NVMe per host. Baseline 2026-05-05:
-    # 71.6 MB/s avg / 0.89 s avg latency on a 5-min replicated 4 MiB
-    # write bench (post-SLC-cache). bdev_*_discard need an OSD restart
-    # to take effect (init-time); the bluestore_* params are runtime.
-    # See ceph-speed-baseline-2026-05-05 in the wiki.
+    # Discard/TRIM for the QLC NVMe per host (CT1000P3PSSD8, Crucial P3 Plus —
+    # earlier comments here said "P2", which is a different drive). These need
+    # an OSD restart to take effect (init-time). Baseline 2026-05-05:
+    # 71.6 MB/s avg / 0.89 s avg latency on a 5-min replicated 4 MiB write
+    # bench (post-SLC-cache). See ceph-speed-baseline-2026-05-05 in the wiki.
+    #
+    # bluestore_prefer_deferred_size_ssd / bluestore_deferred_batch_ops_ssd are
+    # DELIBERATELY LEFT AT THE CEPH DEFAULTS (0 and 16) and must not be raised
+    # again without re-reading this.
+    #
+    # They were previously set to the HDD-style 65536 / 64 as a throughput tune.
+    # prefer_deferred_size defers every write SMALLER than the threshold: the
+    # write goes first into RocksDB's WAL and is applied to the object store
+    # afterwards. That is a double write, and at 65536 it caught essentially
+    # every write this cluster makes — SQLite pages are 4 KiB, Postgres pages
+    # 8 KiB. Ceph disables deferral on SSD (default 0) precisely because solid
+    # state does not need the random-write batching that helps spinning rust.
+    #
+    # Two reasons it was the wrong trade here:
+    #   - The benchmark that justified it cannot have exercised it. It used
+    #     4 MiB writes, 64x ABOVE the 64 KiB cutoff, so not one of those writes
+    #     was ever deferred. The setting was validated by a workload it does
+    #     not affect.
+    #   - It doubles writes onto QLC, which has the lowest endurance of any
+    #     NAND type. Measured 2026-08-19: all three OSD drives at 80/82/87%
+    #     endurance used, advancing ~0.13 points/day, i.e. rated life exhausted
+    #     between Nov 2026 and Jan 2027 — and simultaneously, since size=3 on
+    #     3 OSDs means every drive absorbs 100% of cluster writes (see #434).
+    #
+    # This cluster is write-light (~117 GB/day at the RBD client layer), so
+    # throughput was never the binding constraint. Endurance is. If small-write
+    # latency ever becomes the real problem, fix it with more OSDs rather than
+    # by paying a 2x write tax on every 4 KiB page.
     osd:
       bdev_enable_discard: "true"
       bdev_async_discard: "true"
-      bluestore_prefer_deferred_size_ssd: "65536"
-      bluestore_deferred_batch_ops_ssd: "64"
+      # Ceph defaults, set explicitly so the previous override is actively
+      # overwritten rather than merely removed — `ceph config` retains a value
+      # that is only deleted from this file.
+      bluestore_prefer_deferred_size_ssd: "0"
+      bluestore_deferred_batch_ops_ssd: "16"
   dashboard:
     enabled: true
     ssl: false

kubernetes/infrastructure/rook-ceph-cluster/kustomization.yaml

@@ -2,6 +2,7 @@ apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
   - cephcluster.yaml
+  - toolbox.yaml
   - storageclass.yaml
   - volumesnapshotclass.yaml
   - objectstore.yaml

kubernetes/infrastructure/rook-ceph-cluster/storageclass.yaml

@@ -5,8 +5,31 @@ metadata:
   namespace: rook-ceph
 spec:
   failureDomain: host
+  # size 2, NOT 3. With 3 OSDs on 3 hosts, size=3 meant every write landed on
+  # every drive in the cluster: per-drive write volume equalled total cluster
+  # write volume, and all three drives wore at an identical rate and would have
+  # reached end-of-rated-life together (measured 2026-08-19: 80/82/87% endurance
+  # used, ~0.13 points/day, all three exhausted between Nov 2026 and Jan 2027 —
+  # see #434). Replication does not protect against that failure mode, because
+  # wear-out here is correlated by construction rather than independent.
+  #
+  # size 2 cuts per-drive write volume by a third and desynchronises the wear.
+  #
+  # min_size 1 is REQUIRED and deliberate. Ceph does not lower min_size when
+  # size is reduced — it was 2 here, and leaving it would make the pool 2/2,
+  # where any single OSD down (every rolling node reboot, every OSD restart)
+  # blocks IO for its PGs and hangs every RBD volume in the cluster.
+  #
+  # The trade this accepts: while the pool is degraded (one OSD down), a second
+  # drive failing loses data, and with all three drives at 80%+ endurance that
+  # window is not hypothetical. Mitigations that already exist: CNPG barman and
+  # Velero both ship offsite to Hetzner (opentofu/object_storage.tf), so the
+  # exposure is the RPO of those backups, not total loss. Keep node reboots
+  # serialised and wait for HEALTH_OK between them.
   replicated:
-    size: 3
+    size: 2
+  parameters:
+    min_size: "1"
 ---
 apiVersion: storage.k8s.io/v1
 kind: StorageClass
@@ -64,13 +87,23 @@ metadata:
   name: cephfs
   namespace: rook-ceph
 spec:
+  # metadataPool STAYS at size 3 while the data pool drops to 2. It holds
+  # 467 MiB against the data pool's 275 GiB, so it contributes nothing
+  # measurable to drive wear — the entire reason for the size reduction — while
+  # losing it would destroy the whole filesystem, not just the objects on the
+  # lost OSD. Paying 3x on half a gigabyte to keep that intact is the right
+  # trade; the wear saving comes from data0.
   metadataPool:
     replicated:
       size: 3
   dataPools:
     - name: data0
+      # size 2 / min_size 1 for the same reasons as replicapool above — see
+      # the comment there for the full rationale and the accepted trade.
       replicated:
-        size: 3
+        size: 2
+      parameters:
+        min_size: "1"
   metadataServer:
     activeCount: 1
     activeStandby: true

kubernetes/infrastructure/rook-ceph-cluster/toolbox.yaml

@@ -0,0 +1,174 @@
+# Upstream Rook toolbox, taken verbatim from
+# https://raw.githubusercontent.com/rook/rook/v1.20.4/deploy/examples/toolbox.yaml
+# (matching the operator image docker.io/rook/ceph:v1.20.4), with three
+# deliberate changes:
+#   - the image is pinned to the cluster's own Ceph version and carries the
+#     same imagepolicy marker as cephcluster.yaml, so the client never drifts
+#     ahead of the mons it talks to
+#   - resource requests/limits, since this is a long-lived idle pod
+#   - no other divergence; keep it that way so upstream diffs stay readable
+#
+# This is the only admin path to Ceph internals. Without it `ceph status`,
+# `ceph health detail`, `ceph osd df tree`, `ceph osd perf` and `ceph device
+# ls` are unreachable during an incident: the mons, OSDs and mgr mount no
+# admin keyring, and the operator's /etc/ceph is empty.
+#
+# It mounts the client.admin keyring from the rook-ceph-mon secret, so it is
+# a full-privilege credential holder, not a read-only one — Rook offers no
+# lesser-privileged toolbox. Ingress to it is gated by the namespace
+# default-deny-ingress CNP (network-policies/rook-ceph-ns.yaml).
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: rook-ceph-tools
+  namespace: rook-ceph
+  labels:
+    app: rook-ceph-tools
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: rook-ceph-tools
+  template:
+    metadata:
+      labels:
+        app: rook-ceph-tools
+    spec:
+      dnsPolicy: ClusterFirstWithHostNet
+      serviceAccountName: rook-ceph-default
+      containers:
+        - name: rook-ceph-tools
+          image: quay.io/ceph/ceph:v19.2.6 # {"$imagepolicy": "flux-system:ceph"}
+          command:
+            - /bin/bash
+            - -c
+            - |
+              # Replicate the script from toolbox.sh inline so the ceph image
+              # can be run directly, instead of requiring the rook toolbox
+              CEPH_CONFIG="/etc/ceph/ceph.conf"
+              MON_CONFIG="/etc/rook/mon-endpoints"
+              KEYRING_FILE="/etc/ceph/keyring"
+              CONFIG_OVERRIDE="/etc/rook-config-override/config"
+
+              # create a ceph config file in its default location so ceph/rados tools can be used
+              # without specifying any arguments
+              write_endpoints() {
+                endpoints=$(cat ${MON_CONFIG})
+
+                # filter out the mon names
+                # external cluster can have numbers or hyphens in mon names, handling them in regex
+                # shellcheck disable=SC2001
+                mon_endpoints=$(echo "${endpoints}"| sed 's/[a-z0-9_-]\+=//g')
+
+                DATE=$(date)
+                echo "$DATE writing mon endpoints to ${CEPH_CONFIG}: ${endpoints}"
+                  cat <<EOF > ${CEPH_CONFIG}
+              [global]
+              mon_host = ${mon_endpoints}
+
+              [client.admin]
+              keyring = ${KEYRING_FILE}
+              EOF
+
+                # Merge the config override if it exists and is not empty
+                if [ -f "${CONFIG_OVERRIDE}" ] && [ -s "${CONFIG_OVERRIDE}" ]; then
+                  echo "$DATE merging config override from ${CONFIG_OVERRIDE}"
+                  echo "" >> ${CEPH_CONFIG}
+                  cat ${CONFIG_OVERRIDE} >> ${CEPH_CONFIG}
+                  echo "" >> ${CEPH_CONFIG}
+                fi
+              }
+
+              # watch the endpoints config file and update if the mon endpoints ever change
+              watch_endpoints() {
+                # get the timestamp for the target of the soft link
+                real_path=$(realpath ${MON_CONFIG})
+                initial_time=$(stat -c %Z "${real_path}")
+                while true; do
+                  real_path=$(realpath ${MON_CONFIG})
+                  latest_time=$(stat -c %Z "${real_path}")
+
+                  if [[ "${latest_time}" != "${initial_time}" ]]; then
+                    write_endpoints
+                    initial_time=${latest_time}
+                  fi
+
+                  sleep 10
+                done
+              }
+
+              # read the secret from an env var (for backward compatibility), or from the secret file
+              ceph_secret=${ROOK_CEPH_SECRET}
+              if [[ "$ceph_secret" == "" ]]; then
+                ceph_secret=$(cat /var/lib/rook-ceph-mon/secret.keyring)
+              fi
+
+              # create the keyring file
+              cat <<EOF > ${KEYRING_FILE}
+              [${ROOK_CEPH_USERNAME}]
+              key = ${ceph_secret}
+              EOF
+
+              # write the initial config file
+              write_endpoints
+
+              # continuously update the mon endpoints if they fail over
+              watch_endpoints
+          imagePullPolicy: IfNotPresent
+          tty: true
+          securityContext:
+            runAsNonRoot: true
+            runAsUser: 2016
+            runAsGroup: 2016
+            capabilities:
+              drop: ["ALL"]
+          env:
+            - name: ROOK_CEPH_USERNAME
+              valueFrom:
+                secretKeyRef:
+                  name: rook-ceph-mon
+                  key: ceph-username
+          resources:
+            requests:
+              cpu: 10m
+              memory: 128Mi
+            limits:
+              memory: 512Mi
+          volumeMounts:
+            - mountPath: /etc/ceph
+              name: ceph-config
+            - name: mon-endpoint-volume
+              mountPath: /etc/rook
+            - name: ceph-admin-secret
+              mountPath: /var/lib/rook-ceph-mon
+              readOnly: true
+            - name: rook-config-override
+              mountPath: /etc/rook-config-override
+              readOnly: true
+      volumes:
+        - name: ceph-admin-secret
+          secret:
+            secretName: rook-ceph-mon
+            optional: false
+            items:
+              - key: ceph-secret
+                path: secret.keyring
+        - name: mon-endpoint-volume
+          configMap:
+            name: rook-ceph-mon-endpoints
+            items:
+              - key: data
+                path: mon-endpoints
+        - name: rook-config-override
+          configMap:
+            name: rook-config-override
+            optional: true
+        - name: ceph-config
+          emptyDir:
+            medium: Memory
+            sizeLimit: 16Mi
+      tolerations:
+        - key: "node.kubernetes.io/unreachable"
+          operator: "Exists"
+          effect: "NoExecute"
+          tolerationSeconds: 5

kubernetes/infrastructure/velero/restore-test-cronjob.yaml

@@ -85,4 +85,6 @@ spec:
                 name: restore-test-script
                 defaultMode: 0755
             - name: tmp
-              emptyDir: {}
+              emptyDir:
+                medium: Memory
+                sizeLimit: 16Mi

kubernetes/talos/cilium.yaml

@@ -249,6 +249,11 @@ data:
 
   identity-heartbeat-timeout: "30m0s"
   identity-gc-interval: "15m0s"
+  # Exclude Woodpecker's per-build labels from identity computation, so CI
+  # steps share one identity instead of minting one each. Keep in lockstep
+  # with patches/cilium-cni.yaml and values.labels in
+  # kubernetes/infrastructure/cilium/helmrelease.yaml.
+  labels: '!woodpecker-ci\.org/.* !step$'
   cilium-endpoint-gc-interval: "5m0s"
   nodes-gc-interval: "5m0s"
 
@@ -302,8 +307,9 @@ data:
   # sizing of the TCP CT, non-TCP CT, NAT and policy BPF maps.
   bpf-map-dynamic-size-ratio: "0.0025"
   # bpf-policy-map-max specifies the maximum number of entries in endpoint
-  # policy map (per endpoint)
-  bpf-policy-map-max: "16384"
+  # policy map (per endpoint). Keep in lockstep with patches/cilium-cni.yaml and
+  # values.bpf.policyMapMax in kubernetes/infrastructure/cilium/helmrelease.yaml.
+  bpf-policy-map-max: "65536"
   # bpf-policy-stats-map-max specifies the maximum number of entries in global
   # policy stats map
   bpf-policy-stats-map-max: "65536"

kubernetes/talos/patches/cilium-cni.yaml

@@ -267,6 +267,11 @@ cluster:
         
           identity-heartbeat-timeout: "30m0s"
           identity-gc-interval: "15m0s"
+          # Exclude Woodpecker's per-build labels from identity computation, so
+          # CI steps share one identity instead of minting one each. Keep in
+          # lockstep with values.labels in
+          # kubernetes/infrastructure/cilium/helmrelease.yaml.
+          labels: '!woodpecker-ci\.org/.* !step$'
           cilium-endpoint-gc-interval: "5m0s"
           nodes-gc-interval: "5m0s"
         
@@ -320,8 +325,12 @@ cluster:
           # sizing of the TCP CT, non-TCP CT, NAT and policy BPF maps.
           bpf-map-dynamic-size-ratio: "0.0025"
           # bpf-policy-map-max specifies the maximum number of entries in endpoint
-          # policy map (per endpoint)
-          bpf-policy-map-max: "16384"
+          # policy map (per endpoint). Keep in lockstep with values.bpf.policyMapMax
+          # in kubernetes/infrastructure/cilium/helmrelease.yaml, and with the third
+          # copy in kubernetes/talos/cilium.yaml: bootstrap installs Cilium from this
+          # inline manifest, so a lower value here means a cold rebuild comes up with
+          # a map that overflows until Flux reconciles.
+          bpf-policy-map-max: "65536"
           # bpf-policy-stats-map-max specifies the maximum number of entries in global
           # policy stats map
           bpf-policy-stats-map-max: "65536"

kubernetes/talos/patches/cluster.yaml

@@ -26,16 +26,20 @@ machine:
     nodeIP:
       validSubnets:
         - 10.2.40.0/24
-    # Lower the image GC thresholds from the kubelet defaults (85/80)
-    # so containerd's image cache is pruned before /var crosses Ceph's
-    # mon_data_avail_warn line (mon DB warns when its hostpath has
-    # <30% free). k8s-2 and k8s-3 sit at 77-79% used because of pod
-    # density (75 pods on k8s-2 vs 42 on k8s-1) — unused image layers
-    # accumulate untouched until the 85% high watermark, which the
-    # node never hits, so GC never fires. 70/60 keeps /var safely
-    # under 30%-free without raising the VM disk again. Spegel P2P
-    # pull-through softens the re-pull cost when GC'd images are
-    # needed back.
+    # Lower the image GC thresholds from the kubelet defaults (85/80) to
+    # bound containerd's image cache, which is the largest single consumer
+    # of /var. Spegel P2P pull-through softens the re-pull cost when GC'd
+    # images are needed back.
+    #
+    # These thresholds do NOT control whether Ceph's mon reports
+    # MON_DISK_LOW, and must not be tuned for that. Measured on k8s-2:
+    # image GC frees only the content store (~8.6 GB), while the bulk of
+    # /var/lib/containerd is ~20 GB of overlayfs snapshots — unpacked
+    # layers shared with running containers, which GC cannot reclaim.
+    # Dropping high to 60 deleted 28 images and moved free space by 0.02
+    # percentage points. The mon's own data is ~106 MB; its 30%-free
+    # warning is measuring a filesystem it merely shares, so the lever for
+    # it is elsewhere (mon volume, /var size, or mon_data_avail_warn).
     extraConfig:
       imageGCHighThresholdPercent: 70
       imageGCLowThresholdPercent: 60

opentofu/dns_renner_uno.tf

@@ -1,8 +1,20 @@
 locals {
   renner_uno_records = {
-    # A Records
-    "@"   = { type = "A", value = "159.69.42.237" }
-    "www" = { type = "A", value = "159.69.42.237" }
+    # No "@" / "www" entries. Both used to pin renner.uno and www.renner.uno
+    # to 159.69.42.237, the Hetzner VM that hosted the WordPress site until it
+    # moved into the cluster (kubernetes/apps/wordpress). Neither record is
+    # actually live on the UDM any more — verified 2026-08-16: a LAN query for
+    # www.renner.uno returns the public answer (proxy.renner.uno) — so this
+    # block was drift waiting to be re-applied.
+    #
+    # The correct end state is exactly this absence: www.renner.uno is labelled
+    # dns.renner.uno/public, so external-dns-unifi deliberately writes no UDM
+    # record and LAN clients follow the public path (wildcard -> proxy) and
+    # hairpin back in. Re-adding an A record here would point the house at a
+    # server that is being decommissioned.
+    #
+    # The apex stays with Cloudflare, where a redirect rule sends
+    # renner.uno/* to https://www.renner.uno/*.
 
     # CNAME Records (External Targets)
     "nut" = { type = "CNAME", value = "nut.srv.renner.uno" }

opentofu/lxc_800_rustfs_decommission.tf

@@ -1,43 +0,0 @@
-# TEMPORARY shims v2 — rustfs LXCs 800/801/802, decommission in progress.
-#
-# Stage 2 (#270) failed: both destroy attempts died instantly with
-# "context deadline exceeded" on the pre-destroy status GET, while the PVE
-# API answers the same call in <1s. Cause: the bpg provider reads
-# `timeout_delete` from STATE at destroy time, and importing a resource
-# whose config declares no timeouts stored 0 — a zero-second context.
-# (The import/plan read path uses its own timeout, which is why stage 1
-# planned clean.)
-#
-# Fix: declare the timeouts explicitly, apply ONCE so they land in state
-# (in-place no-API update), then delete this file again and the next apply
-# destroys the containers. No lifecycle ignore_changes this time — the
-# timeout update must actually apply; the other attributes mirror the real
-# containers so the plan stays in-place.
-#
-#   1. merge; run PIPELINE=deploy-infrastructure (plan MUST show in-place
-#      update only, no replace), then TOFU_APPLY=true
-#   2. follow-up PR deletes this file; plan shows 3 destroys; apply.
-
-locals {
-  rustfs_decommission = {
-    "800" = "saturn"
-    "801" = "jupiter"
-    "802" = "neptune"
-  }
-}
-
-resource "proxmox_virtual_environment_container" "rustfs_decommission" {
-  for_each = local.rustfs_decommission
-
-  node_name     = each.value
-  vm_id         = tonumber(each.key)
-  started       = false
-  unprivileged  = true
-  start_on_boot = true
-
-  timeout_create = 1800
-  timeout_clone  = 1800
-  timeout_update = 1800
-  timeout_delete = 1800
-  timeout_start  = 300
-}

opentofu/providers.tf

@@ -59,13 +59,46 @@ provider "unifi" {
 
 # UniFi Network v2 API (policy-table NAT rules). Same controller and API key
 # as the unifi provider above; the v2 endpoints are simply not covered by it.
-# UniFi PUT quirk: updates may require _id in the body, which restapi does
-# not send — if an in-place update 400s, taint the object so CI recreates it.
+# UniFi PUT quirk — DISPROVEN 2026-08-19, left here because it was wrong for
+# long enough to do damage. The claim was that a UniFi PUT may require _id in
+# the body, which restapi does not send, and that a 400 should be answered by
+# tainting the object so CI recreates it. Tested directly against a live rule
+# with exactly the body tofu sends:
+#
+#   PUT /proxy/network/v2/api/site/default/nat/6a61c7d7bbd7b2ca90981f51
+#   body: the object, _id removed          -> HTTP 200, object returned intact
+#   rule re-read and diffed before/after   -> byte-identical
+#   rule count                             -> 60 before, 60 after
+#
+# Confirmed again at scale by apply 5675: all 60 rules "Modifications complete
+# after 0s", no 400s, ci.renner.uno and immich.renner.uno both still 200.
+#
+# This mattered. A warning that applying might break firewall rules is a
+# warning nobody argues with, so the apply did not get run — and the
+# renner-uno-k8s-mariadb bucket added in b976a67b sat uncreated for three days
+# while the WordPress backup failed nightly with "The specified bucket does not
+# exist". Applying these NAT rules is safe. Do not taint them.
+#
+# create_returns_object, NOT write_returns_object. Both make the provider store
+# the API response into `data`; write_returns_object does it for PUT as well as
+# POST. Since `data` in unifi_nat_hairpin.tf never sets the UDM-assigned `_id`,
+# every write put a field into state that config does not have, so all 60 rules
+# planned as "will be updated in-place" with `- _id` as the whole diff — and
+# re-dirtied themselves on the next write, forever.
+#
+# No provider-side ignore can suppress that: `data` is a configured attribute,
+# compared by Terraform core before provider logic runs. Tried and disproven —
+# ignore_changes_to = ["_id"] (plan 5661), ignore_all_server_changes (5662),
+# and ignore_server_additions on restapi 3.x (5667). All three left the 60.
+#
+# create_returns_object keeps what the provider actually needs — the POST
+# response is how it learns `_id` for id_attribute below — without letting
+# updates rewrite `data`.
 provider "restapi" {
-  uri                  = "${var.unifi_api_url}/proxy/network/v2/api/site/default"
-  insecure             = var.unifi_insecure
-  write_returns_object = true
-  id_attribute         = "_id"
+  uri                   = "${var.unifi_api_url}/proxy/network/v2/api/site/default"
+  insecure              = var.unifi_insecure
+  create_returns_object = true
+  id_attribute          = "_id"
   headers = {
     "X-API-KEY"    = var.unifi_api_key
     "Content-Type" = "application/json"

opentofu/unifi_nat_hairpin.tf

@@ -107,17 +107,51 @@ resource "restapi_object" "hairpin" {
   for_each = local.hairpin_rules
 
   path = "/nat"
+
+  # KNOWN PERMANENT DIFF — cosmetic. Do not try to fix it with a provider
+  # setting; four have been tried and none work. Recorded here so the next
+  # person does not spend a day rediscovering that.
+  #
+  # Every plan shows all 60 of these as "will be updated in-place" with the
+  # entire diff being one line:
+  #
+  #   ~ data = jsonencode( ~ { - _id = "6a61c7d7bbd7b2ca90981f51"
+  #                              # (15 unchanged attributes hidden) } )
+  #
+  # The UDM assigns _id and returns it; the provider puts the response into
+  # `data`; the `data` below never sets it. So state carries a field the config
+  # does not. It survives an apply: 5675 rewrote all 60, and the very next plan
+  # (5677) showed the same 60 again.
+  #
+  # Tried, with the plan that disproved each — do not repeat these:
+  #
+  #   ignore_changes_to = ["_id"]                    5661  60 remained
+  #   ignore_all_server_changes = true               5662  60 remained
+  #   restapi 3.x + ignore_server_additions = true   5667  60 remained, and
+  #                                                        added an api_data diff
+  #   write_returns_object -> create_returns_object  5677  60 remained
+  #
+  # The first three cannot work in principle: `data` is a configured attribute,
+  # so Terraform core diffs config against state before any provider-side ignore
+  # logic runs. The fourth was the better theory — that PUT responses were
+  # re-polluting `data` — and it is wrong too, which means the response is
+  # written into `data` on the READ path. Settling it properly needs the
+  # provider source, not more guesses at the schema.
+  #
+  # It is safe to apply through. Verified twice: a single PUT without _id
+  # returns 200 and leaves the rule byte-identical (see providers.tf), and
+  # apply 5675 modified all 60 with no errors and no service impact.
   data = jsonencode({
-    description  = "Hairpin ${each.key}"
-    type         = "DNAT"
-    enabled      = true
-    exclude      = false
-    in_interface = each.value.network_id
-    ip_address   = each.value.endpoint.target
-    ip_version   = each.value.endpoint.ip_version
-    protocol     = "tcp_udp"
-    rule_index   = each.value.rule_index
-    logging      = false
+    description              = "Hairpin ${each.key}"
+    type                     = "DNAT"
+    enabled                  = true
+    exclude                  = false
+    in_interface             = each.value.network_id
+    ip_address               = each.value.endpoint.target
+    ip_version               = each.value.endpoint.ip_version
+    protocol                 = "tcp_udp"
+    rule_index               = each.value.rule_index
+    logging                  = false
     is_predefined            = false
     pppoe_use_base_interface = false
     setting_preference       = "manual"