feat(rook-ceph): rotate CSI and rbd-mirror-peer keys to aes256k (#527)

The last five CephX entities on aes are the four CSI clients and
client.rbd-mirror-peer. The CSI keys waited on node kernel support:
Rook's key-rotation doc asks for kernel 7.0+ for aes256k over kernel
mounts. Talos v1.13.10 (now on all nodes) backports those libceph
patches to 6.18 (siderolabs/pkgs 2da2cdc); CONFIG_CRYPTO_KRB5 and
CONFIG_CRYPTO_KRB5ENC are set on every node. ceph-csi is v3.17.1, the
minimum the doc names; Rook v1.20.7.

csi: KeyGeneration 2, keyType aes256k, keepPriorKeyCountMax 1. Rotated
keys only apply to new mounts; the prior key stays valid for PVCs
mounted before the rotation, per the Rook doc.

rbdMirrorPeer: KeyGeneration 2, aes256k. No CephRBDMirror and no peers
exist, so rotating the peer token affects nothing.

allowedCiphers / mon_auth_allow_insecure_key are NOT changed here: that
is the lockout-capable step and follows only once ceph health shows no
aes entity left.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

ci/images/k8s/Containerfile

@@ -1,6 +1,6 @@
 FROM alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d
 
-ARG TALOS_VERSION=v1.13.9
+ARG TALOS_VERSION=v1.13.10
 ARG VELERO_VERSION=v1.16.0
 ARG KUBECTL_VERSION=v1.36.3
 ARG FLUX_VERSION=2.9.5
@@ -18,7 +18,7 @@ RUN apk add --no-cache \
 
 RUN curl -sfL -o /usr/local/bin/talosctl \
       "https://github.com/siderolabs/talos/releases/download/${TALOS_VERSION}/talosctl-linux-amd64" \
- && echo "7e1d4b7d5846964bdcf63a794e3c8161bb6ef2983d5ace58ea5322f3bf32a27e  /usr/local/bin/talosctl" | sha256sum -c - \
+ && echo "28e9ccd9aebd690fa03d8cf2582221d87ea38b2a1704f411d75a03589c58f2aa  /usr/local/bin/talosctl" | sha256sum -c - \
  && chmod +x /usr/local/bin/talosctl
 
 RUN curl -sfL -o /usr/local/bin/kubectl \

ci/images/vmtest/Containerfile

@@ -14,7 +14,7 @@ FROM alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d650
 # kubernetes/talos/patches/cluster.yaml. The phase reads the version from that
 # file at run time and passes it to the provisioner; this pin only decides which
 # client talks to it.
-ARG TALOS_VERSION=v1.13.9
+ARG TALOS_VERSION=v1.13.10
 ARG KUBECTL_VERSION=v1.36.2
 ARG FLUX_VERSION=2.9.2
 ARG KOPIA_VERSION=0.23.1
@@ -49,7 +49,7 @@ RUN apk add --no-cache \
 
 RUN curl -sfL -o /usr/local/bin/talosctl \
       "https://github.com/siderolabs/talos/releases/download/${TALOS_VERSION}/talosctl-linux-amd64" \
- && echo "7e1d4b7d5846964bdcf63a794e3c8161bb6ef2983d5ace58ea5322f3bf32a27e  /usr/local/bin/talosctl" | sha256sum -c - \
+ && echo "28e9ccd9aebd690fa03d8cf2582221d87ea38b2a1704f411d75a03589c58f2aa  /usr/local/bin/talosctl" | sha256sum -c - \
  && chmod +x /usr/local/bin/talosctl
 
 RUN curl -sfL -o /usr/local/bin/kubectl \

kubernetes/RUNBOOK-cold-rebuild.md

@@ -107,7 +107,7 @@ config, and the nodes are addressed by name.
 ## Step 1b — boot all three machines from install media
 
 ```
-https://factory.talos.dev/image/0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9/v1.13.9/metal-amd64.iso
+https://factory.talos.dev/image/0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9/v1.13.10/metal-amd64.iso
 ```
 
 Write it to a USB stick and boot each MS-01 from it. Each comes up in

kubernetes/apps/gitea/kustomization.yaml

@@ -6,7 +6,6 @@ kind: Kustomization
 # and verified. See MIGRATION-gitea-woodpecker.md.
 resources:
   - statefulset.yaml
-  - pdb.yaml
   - service.yaml
   - ingressroute-ssh.yaml
   - ingress.yaml

kubernetes/apps/gitea/pdb.yaml

@@ -1,23 +0,0 @@
----
-# Eviction protection for the single-replica Gitea StatefulSet.
-#
-# Gitea runs on SQLite (single writer) and its repos + DB live on a single
-# ReadWriteOnce PVC, so it is deliberately replicas:1 and cannot fail over.
-# With no PDB, a node drain / maintenance evicts the pod abruptly, mid-write,
-# with zero coordination. minAvailable:1 on a 1-replica workload makes the
-# eviction API refuse *voluntary* evictions: `kubectl drain` blocks and
-# surfaces this pod, forcing a conscious decision (drain with
-# --disable-eviction, or scale down first) instead of yanking the live DB.
-# It does NOT block involuntary disruptions (node crash, OOM).
-apiVersion: policy/v1
-kind: PodDisruptionBudget
-metadata:
-  name: gitea
-  namespace: gitea
-  labels:
-    app.kubernetes.io/name: gitea
-spec:
-  minAvailable: 1
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: gitea

kubernetes/apps/grafana-db/postgres.yaml

@@ -4,7 +4,9 @@ metadata:
   name: gf-pg
   namespace: monitoring
 spec:
-  instances: 1
+  instances: 2
+  smartShutdownTimeout: 10
+  stopDelay: 60
   imageName: ghcr.io/cloudnative-pg/postgresql:16
   storage:
     storageClass: ceph-block

kubernetes/apps/home-assistant/ha-pg-cluster.yaml

@@ -30,7 +30,9 @@ metadata:
   name: ha-pg
   namespace: home-assistant
 spec:
-  instances: 1
+  instances: 2
+  smartShutdownTimeout: 10
+  stopDelay: 60
   imageName: ghcr.io/cloudnative-pg/postgresql:16
   storage:
     storageClass: ceph-block
@@ -124,8 +126,7 @@ spec:
       # padding rather than data.
       #
       # The cost is RPO: up to 30min of WAL is unrecoverable if this cluster and
-      # its volume are lost outright. This is a single-instance cluster, so the
-      # archive is the only off-node copy.
+      # its volumes are lost outright.
       archive_timeout: "30min"
       # Fewer checkpoints => fewer full-page writes, since a page is only
       # fully-logged on its first modification after a checkpoint.

kubernetes/apps/immich/cluster.yaml

@@ -59,8 +59,7 @@ spec:
       # padding rather than data.
       #
       # The cost is RPO: up to 30min of WAL is unrecoverable if this cluster and
-      # its volume are lost outright. This is a single-instance cluster, so the
-      # archive is the only off-node copy.
+      # its volumes are lost outright.
       archive_timeout: "30min"
       # Write-reduction tuning. These databases are largely idle, and the
       # Postgres defaults favour throughput over storage that wears out.

kubernetes/apps/influxdb/helmrelease.yaml

@@ -61,6 +61,9 @@ spec:
       enabled: true
       storageClass: ceph-block
       size: 50Gi
+    # A minAvailable:1 PDB on a single replica blocks every node drain.
+    pdb:
+      create: false
     # Bootstrap admin/org/bucket on the fresh PVC. Admin password + token come
     # from the influxdb-bootstrap Secret (OpenBao via ESO), deterministic across
     # rebuilds so the bootstrap token is the live admin token. `influx setup`

kubernetes/apps/paperless/kustomization.yaml

@@ -11,7 +11,6 @@ resources:
   - protonmail-bridge.yaml
   - ftp-scanner.yaml
   - syncthing.yaml
-  - pdb.yaml
   - network-policy.yaml
   # pprl-pg CNPG Postgres + data/media/consume/redis PVCs (moved here from the
   # retired seed-paperless KS; each carries prune: disabled so ownership

kubernetes/apps/paperless/pdb.yaml

@@ -1,37 +0,0 @@
----
-# Eviction protection for the single-replica stateful paperless sidecars.
-#
-# protonmail-bridge and syncthing each hold state on a single RWO PVC and run
-# replicas:1. minAvailable:1 makes the eviction API refuse voluntary evictions,
-# so a node drain blocks and surfaces these pods instead of evicting them
-# abruptly. Involuntary disruptions (node crash) are not affected.
-#
-# NOTE: the other single-replica paperless pods (paperless, redis, gotenberg,
-# tika, ftp-scanner) are intentionally omitted — they are caches / stateless
-# converters where a blocking PDB would only wedge node drains with no
-# recoverability benefit.
-apiVersion: policy/v1
-kind: PodDisruptionBudget
-metadata:
-  name: protonmail-bridge
-  namespace: paperless
-  labels:
-    app.kubernetes.io/name: protonmail-bridge
-spec:
-  minAvailable: 1
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: protonmail-bridge
----
-apiVersion: policy/v1
-kind: PodDisruptionBudget
-metadata:
-  name: syncthing
-  namespace: paperless
-  labels:
-    app.kubernetes.io/name: syncthing
-spec:
-  minAvailable: 1
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: syncthing

kubernetes/apps/trmnl/cluster.yaml

@@ -4,7 +4,7 @@ metadata:
   name: tr-pg
   namespace: trmnl
 spec:
-  instances: 1
+  instances: 2
   smartShutdownTimeout: 10
   stopDelay: 60
   # PG 18, NOT the repo-standard 16. Terminus ships a committed
@@ -47,8 +47,7 @@ spec:
       # padding rather than data.
       #
       # The cost is RPO: up to 30min of WAL is unrecoverable if this cluster and
-      # its volume are lost outright. This is a single-instance cluster, so the
-      # archive is the only off-node copy.
+      # its volumes are lost outright.
       archive_timeout: "30min"
       # Write-reduction tuning. These databases are largely idle, and the
       # Postgres defaults favour throughput over storage that wears out.

kubernetes/apps/woodpecker/cluster.yaml

@@ -22,7 +22,9 @@ metadata:
     # migration deliberately withheld this while a rollback still existed.
     kustomize.toolkit.fluxcd.io/prune: disabled
 spec:
-  instances: 1
+  instances: 2
+  smartShutdownTimeout: 10
+  stopDelay: 60
   imageName: ghcr.io/cloudnative-pg/postgresql:16
   storage:
     storageClass: ceph-block
@@ -91,8 +93,7 @@ spec:
       # padding rather than data.
       #
       # The cost is RPO: up to 30min of WAL is unrecoverable if this cluster and
-      # its volume are lost outright. This is a single-instance cluster, so the
-      # archive is the only off-node copy.
+      # its volumes are lost outright.
       archive_timeout: "30min"
   resources:
     requests:

kubernetes/apps/woodpecker/kustomization.yaml

@@ -3,7 +3,6 @@ kind: Kustomization
 resources:
   - rbac.yaml
   - server.yaml
-  - pdb.yaml
   - agent.yaml
   - service.yaml
   - ingress.yaml
@@ -22,7 +21,7 @@ resources:
   # OpenBao access, no NetworkPolicy needed. server.yaml/agent.yaml keep their
   # Reloader annotations; externalsecret.yaml keeps the 10m refreshInterval.
   # --- namespace migration, stages 1-2 (issue #433) ---
-  # Stage 2 has moved the workloads above (server, agent, service, ingress, pdb,
+  # Stage 2 has moved the workloads above (server, agent, service, ingress,
   # rbac, secrets, policies) into the `woodpecker` namespace and pointed the
   # server's DSN at wp-pg-rw.woodpecker.svc. Stage 3 retired the `apps`
   # originals; these four now describe the surviving cluster. History and the

kubernetes/apps/woodpecker/pdb.yaml

@@ -1,27 +0,0 @@
----
-# Eviction protection for the single-replica Woodpecker server StatefulSet.
-#
-# The server is SQLite-backed (single writer) on a single RWO PVC, so it is
-# deliberately replicas:1. minAvailable:1 makes the eviction API refuse
-# voluntary evictions so a node drain blocks and surfaces this pod rather
-# than yanking the live CI DB mid-write. Involuntary disruptions are not
-# affected.
-#
-# The selector MUST include app.kubernetes.io/component: server: the agent
-# Deployment in this same namespace shares app.kubernetes.io/name: woodpecker
-# (component: agent), and a name-only selector would wrongly also cover the
-# stateless, disposable agents.
-apiVersion: policy/v1
-kind: PodDisruptionBudget
-metadata:
-  name: woodpecker-server
-  namespace: woodpecker
-  labels:
-    app.kubernetes.io/name: woodpecker
-    app.kubernetes.io/component: server
-spec:
-  minAvailable: 1
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: woodpecker
-      app.kubernetes.io/component: server

kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml

@@ -21,17 +21,24 @@ spec:
     modules:
       - name: pg_autoscaler
         enabled: true
-  # Rotates daemon keys to aes256k for CVE-2025-30156 (#527). Opt-in; disabled
-  # by default. Increment keyGeneration to rotate again.
-  #
-  # Daemon keys only — aes256k over the CSI mount path needs kernel 7.0+ and
-  # these nodes run 6.18, so CSI stays on aes and the client-key checks stay
-  # raised. keyType is unset so Rook picks it.
+  # CephX keys on aes256k for CVE-2025-30156 (#527). Increment keyGeneration
+  # to rotate again. CSI kernel mounts need aes256k in libceph: Talos
+  # >= v1.13.10 backports it to 6.18 (siderolabs/pkgs 2da2cdc).
   security:
     cephx:
       daemon:
         keyRotationPolicy: KeyGeneration
         keyGeneration: 2
+      csi:
+        keyRotationPolicy: KeyGeneration
+        keyGeneration: 2
+        keyType: aes256k
+        # Keeps the prior key valid for PVCs mounted before the rotation.
+        keepPriorKeyCountMax: 1
+      rbdMirrorPeer:
+        keyRotationPolicy: KeyGeneration
+        keyGeneration: 2
+        keyType: aes256k
   # CephFS cloner pool raised from the default 4. Velero clones a
   # VolumeSnapshot per PVC, and 4 threads stall a multi-PVC backup with
   # `rados: "all cloner threads are busy"`.

kubernetes/talos/patches/cluster.yaml

@@ -24,7 +24,7 @@ machine:
   # The Kubernetes version is a separate decision, moved by
   # `talosctl upgrade-k8s`.
   install:
-    image: factory.talos.dev/metal-installer/0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9:v1.13.9
+    image: factory.talos.dev/metal-installer/0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9:v1.13.10
   features:
     rbac: true
     # stableHostname is NOT here: it is the v1alpha1 spelling of what the