feat(rook-ceph): rotate CSI and rbd-mirror-peer keys to aes256k (#527)

The last five CephX entities on aes are the four CSI clients and
client.rbd-mirror-peer. The CSI keys waited on node kernel support:
Rook's key-rotation doc asks for kernel 7.0+ for aes256k over kernel
mounts. Talos v1.13.10 (now on all nodes) backports those libceph
patches to 6.18 (siderolabs/pkgs 2da2cdc); CONFIG_CRYPTO_KRB5 and
CONFIG_CRYPTO_KRB5ENC are set on every node. ceph-csi is v3.17.1, the
minimum the doc names; Rook v1.20.7.

csi: KeyGeneration 2, keyType aes256k, keepPriorKeyCountMax 1. Rotated
keys only apply to new mounts; the prior key stays valid for PVCs
mounted before the rotation, per the Rook doc.

rbdMirrorPeer: KeyGeneration 2, aes256k. No CephRBDMirror and no peers
exist, so rotating the peer token affects nothing.

allowedCiphers / mon_auth_allow_insecure_key are NOT changed here: that
is the lockout-capable step and follows only once ceph health shows no
aes entity left.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

.woodpecker/wiki-maintenance.yml

@@ -1,12 +1,11 @@
 # Nightly wiki maintenance for the Notes vault (k.renner/notes), replacing the
-# workstation's wiki-maintenance.timer: distill the raw/ backlog into wiki/,
-# then audit wiki/. Each agent is its own `claude` run, not a subagent of an
-# orchestrator, so nothing depends on background tasks in print mode (the
-# workstation job died on their 600 s ceiling). The pipeline commits and pushes
-# as Claude Bot; the push webhook then refreshes the wiki-index service.
+# workstation's wiki-maintenance.timer: distill the raw/ backlog into wiki/
+# (a `claude` run of the wiki-distill agent), then audit wiki/ with the vault's
+# own script (structural checks, no model). The pipeline commits and pushes as
+# Claude Bot; the push webhook then refreshes the wiki-index service.
 #
-# The agents and their retrieval scripts address the vault as ~/Notes, so each
-# agent step links $HOME/Notes to the checkout instead of rewriting them.
+# The agent and its retrieval scripts address the vault as ~/Notes, so the
+# distill step links $HOME/Notes to the checkout instead of rewriting them.
 when:
   - event: cron
     cron: wiki-maintenance
@@ -64,31 +63,9 @@ steps:
   - name: audit
     image: gitea.renner.uno/gitea_admin/ci-claude:2.1.280
     pull: true
-    environment:
-      CLAUDE_CODE_OAUTH_TOKEN:
-        from_secret: claude_code_oauth_token
-      TYPESAFE_API_KEY:
-        from_secret: typesafe_api_key
-      WIKI_INDEX_BASIC_AUTH:
-        from_secret: wiki_index_basic_auth
     commands:
-      - |
-        set -eu
-        ln -sfn "$$PWD/notes" "$$HOME/Notes"
-        mkdir -p "$$HOME/.config/wiki-index"
-        printf '%s' "$$WIKI_INDEX_BASIC_AUTH" | tr -d '\r\n' > "$$HOME/.config/wiki-index/basic-auth"
-        chmod 600 "$$HOME/.config/wiki-index/basic-auth"
-      - |
-        set -eu
-        MODEL=$$(sh scripts/ci/pick-model.sh "structural audit of ~1700 markdown notes: frontmatter completeness, broken links, orphans; mechanical checks, little reasoning")
-        if [ -n "$$MODEL" ]; then echo "audit: selected model $$MODEL"; else echo "audit: using the CLI default model"; fi
-        cd notes
-        cat ../ci/agents/wiki-maintenance/audit.md | claude $${MODEL:+--model $$MODEL} \
-          --agent wiki-audit \
-          --permission-mode acceptEdits \
-          --allowedTools "Read,Write,Glob,Grep,Bash(/root/Notes/bin/wiki_rank.py:*),Bash(/root/Notes/bin/wiki_ask.py:*)" \
-          --settings ../ci/agents/wiki-maintenance/settings.json \
-          -p
+      # Findings are report lines in output/, not a pipeline failure.
+      - python3 -B notes/bin/wiki_audit.py --vault "$$PWD/notes"
 
   - name: push-notes
     image: gitea.renner.uno/gitea_admin/ci-claude:2.1.280
@@ -102,7 +79,9 @@ steps:
         cd notes
         git config user.email "claude-bot@renner.uno"
         git config user.name "Claude Bot"
-        if git diff --quiet && git diff --cached --quiet && [ -z "$$(git ls-files --others --exclude-standard)" ]; then
+        # Only the vault content is committed; anything an agent leaves elsewhere
+        # in the checkout (run 10362: scratch scripts in the root) stays behind.
+        if [ -z "$$(git status --porcelain -- wiki raw output)" ]; then
           echo "wiki-maintenance: no changes"
           exit 0
         fi
@@ -112,7 +91,7 @@ steps:
         # leaves the PR open for a human and fails the run.
         BRANCH="wiki-maintenance/$$(date -u +%F)-$$CI_PIPELINE_NUMBER"
         git checkout -q -b "$$BRANCH"
-        git add -A
+        git add -A -- wiki raw output
         git commit -q -m "wiki maintenance $$(date -u +%F): distill + audit (CI run $$CI_PIPELINE_NUMBER)"
         TOKEN=$$(printf '%s' "$$WIKI_BOT_TOKEN" | tr -d '\r\n')
         git -c http.extraheader="AUTHORIZATION: token $$TOKEN" push origin "HEAD:$$BRANCH"

ci/agents/wiki-maintenance/audit.md

@@ -1,5 +0,0 @@
-You are running as the scheduled wiki-maintenance job in CI. The Notes vault is checked out at ~/Notes (the repository k.renner/notes).
-
-Run the standard audit pass over ~/Notes/wiki/ and write the dated report to ~/Notes/output/. Do not modify any wiki note. Do not commit or push; the pipeline does that.
-
-End with two lines: notes scanned, and issues found by category.

ci/agents/wiki-maintenance/settings.json

@@ -1,5 +1,5 @@
 {
-  "//": "Permission scope for the wiki-maintenance agents (wiki-distill, wiki-audit) in CI.",
+  "//": "Permission scope for the wiki-maintenance agent (wiki-distill) in CI.",
   "//2": "The step's environment holds a Gitea push token, a TypeSafe key and the wiki-index",
   "//3": "client credential, so the Read tool is kept away from /proc (env dumps), /root outside",
   "//4": "the vault symlink target, /etc and the checkout's .git/. The vault itself is the work.",

ci/images/k8s/Containerfile

@@ -1,6 +1,6 @@
 FROM alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d
 
-ARG TALOS_VERSION=v1.13.9
+ARG TALOS_VERSION=v1.13.10
 ARG VELERO_VERSION=v1.16.0
 ARG KUBECTL_VERSION=v1.36.3
 ARG FLUX_VERSION=2.9.5
@@ -18,7 +18,7 @@ RUN apk add --no-cache \
 
 RUN curl -sfL -o /usr/local/bin/talosctl \
       "https://github.com/siderolabs/talos/releases/download/${TALOS_VERSION}/talosctl-linux-amd64" \
- && echo "7e1d4b7d5846964bdcf63a794e3c8161bb6ef2983d5ace58ea5322f3bf32a27e  /usr/local/bin/talosctl" | sha256sum -c - \
+ && echo "28e9ccd9aebd690fa03d8cf2582221d87ea38b2a1704f411d75a03589c58f2aa  /usr/local/bin/talosctl" | sha256sum -c - \
  && chmod +x /usr/local/bin/talosctl
 
 RUN curl -sfL -o /usr/local/bin/kubectl \

ci/images/vmtest/Containerfile

@@ -14,7 +14,7 @@ FROM alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d650
 # kubernetes/talos/patches/cluster.yaml. The phase reads the version from that
 # file at run time and passes it to the provisioner; this pin only decides which
 # client talks to it.
-ARG TALOS_VERSION=v1.13.9
+ARG TALOS_VERSION=v1.13.10
 ARG KUBECTL_VERSION=v1.36.2
 ARG FLUX_VERSION=2.9.2
 ARG KOPIA_VERSION=0.23.1
@@ -49,7 +49,7 @@ RUN apk add --no-cache \
 
 RUN curl -sfL -o /usr/local/bin/talosctl \
       "https://github.com/siderolabs/talos/releases/download/${TALOS_VERSION}/talosctl-linux-amd64" \
- && echo "7e1d4b7d5846964bdcf63a794e3c8161bb6ef2983d5ace58ea5322f3bf32a27e  /usr/local/bin/talosctl" | sha256sum -c - \
+ && echo "28e9ccd9aebd690fa03d8cf2582221d87ea38b2a1704f411d75a03589c58f2aa  /usr/local/bin/talosctl" | sha256sum -c - \
  && chmod +x /usr/local/bin/talosctl
 
 RUN curl -sfL -o /usr/local/bin/kubectl \

kubernetes/NETWORK-traffic-classes.md

@@ -1,117 +1,99 @@
 # Traffic classes on the k8s nodes
 
-Issue #576. Three of the four classes are separated; management is not.
+Issue #576. Five classes, each on its own VLAN.
 
 ## Where each class runs
 
 | class | link | mechanism |
 |---|---|---|
 | Ceph replication | `ceph-san`, VLAN 49 | the whole link is handed to the OSD pod (Multus `host-device`) |
-| world→pod | `ingress0`, VLAN 47 | BGP session sourced there, so the advertised next hop is that address |
-| pod→world | `ingress0`, VLAN 47 | routing rule on the pod source prefix → table 47 |
-| pod↔pod | `pod0`, VLAN 48 | routing rule on pod source **and** destination → table 48 |
-| management | `mgmt0`, VLAN 40 | not started |
+| pod↔pod | `pod0`, VLAN 48 | pod-sourced traffic is routed from table 47; bird puts the peers' pod CIDRs there via `pod0` |
+| world↔pod | `ingress0`, VLAN 47 | BGP session sourced there (next hop of every LoadBalancer VIP); pod egress takes table 47's default |
+| node↔node | `node0`, VLAN 46 | the node InternalIP and etcd live there; so do apiserver↔kubelet, cilium-health and host→remote-pod traffic |
+| world↔node | `mgmt0`, VLAN 40 | API VIP for kubectl, talosctl from outside, node egress (DNS, NTP, image pulls) |
 
 Hard constraint, set by the owner: inter-node and pod traffic never traverses
-the UDM. Everything above stays on the 10G switch.
+the UDM. Everything but world↔node stays on the 10G switch. VLAN 46 and 48 are
+statically addressed, without a gateway; VLAN 46 is isolated in UniFi.
 
-## Why policy routing rather than moving the node IP
+## Pod traffic: one table
 
-Cilium reaches a peer's pods at that peer's Kubernetes InternalIP, in native
-and tunnel mode alike. `--ipv4-node` is overwritten from the Node object, and
-`nodePort.directRoutingDevice` selects the BPF NodePort device, not a next hop.
-Both were established by test (#837, #838), not by reading.
+Two rules, one per family, send every packet with a pod source to table 47
+(`talos/patches/site-homelab.yaml`, priorities 1000/1001, below main). The
+table decides by longest prefix; no rule carves out exceptions.
 
-So the InternalIP decides where pod traffic goes — unless the lookup happens in
-a different table. The rules do that, and nothing about the node's identity
-moves. The routing decision runs before Cilium's masquerade hook, so a rule
-matching the pod source sees the pod address, and the masquerade then rewrites
-it to the egress link's.
+bird (`kubernetes/infrastructure/bird/`) owns table 47 and fills it with:
 
-## Rule order
+- the node's attached networks — `mgmt0`, `ingress0`, `pod0`, `node0` —
+  taken from the interfaces;
+- the peers' pod CIDRs, learned over an iBGP mesh on `pod0`, next hop the
+  peer's `pod0` address;
+- the node's own pod CIDR via `cilium_host`, originated rather than read back
+  from the kernel (reading it back made BIRD depend on the shape of Cilium's
+  route, and KRT rejects the IPv4 one);
+- the egress default via the UDM on `ingress0`, from the node annotations
+  `net.renner.uno/egress-gw-v4/v6`.
 
-Lower priority runs first.
+`node0` has to be in the table: a pod reaching a node address — the
+`kubernetes` ClusterIP, which socket LB rewrites to the apiserver's advertise
+address, or a hostNetwork pod — would otherwise take the default to the UDM.
 
-| prio | match | table |
-|---|---|---|
-| 980 | src pod, dst pod | 48 |
-| 981 | src pod, dst pod (v4) | 48 |
-| 990 | dst `fd00::/8` | main |
-| 991 | dst `10.0.0.0/8` | main |
-| 1000 | src pod | 47 |
-| 1001 | src pod (v4) | 47 |
+The routing decision runs before Cilium's masquerade hook, so the rule sees the
+pod address and the masquerade then rewrites it to the egress link's.
 
-980/981 match source **and** destination. Source alone would also catch the
-node's own traffic to a pod, which would then leave with a `pod0` source — an
-address `CiliumNode.spec.addresses` does not list, so Cilium classifies it as
-`world` rather than `host`.
-
-990/991 are Cilium's `ipv4/ipv6NativeRoutingCIDR`, and they are what keeps
-cluster-internal traffic out of table 47. Without them a pod's packet to a
-service matches the source rule, finds only a default route, and leaves for the
-UDM — measured, in-cluster DNS stopped resolving.
-
-Traffic to a pod on the same node falls through on its own: table 48 holds only
-the peers' CIDRs, so the lookup misses and 990/991 send it to main.
-
-## Where each table comes from
-
-- **47** — two default routes declared per node in `talos/patches/k8s-*.yaml`.
-  The IPv6 gateway is the UDM's configured address on the segment, not the
-  RA-learned link-local; `accept_ra_defrtr` stays `0` on `ingress0` so the RA
-  default does not land in main.
-- **48** — filled by the `bird` DaemonSet: an iBGP mesh over `pod0`, one route
-  per peer pod CIDR with the peer's `pod0` address as next hop.
-
-## The bird mesh
-
-`kubernetes/infrastructure/bird/`. Each node advertises only its own pod CIDR
-and installs what it learns into table 48. Cilium's routes in main are
-untouched, which is why the two coexist.
+Node-originated traffic does not match the rules and uses main, where Cilium
+routes each peer's pod CIDR via the peer's InternalIP — on `node0`.
 
 No address is written into this repo twice. Each node publishes its `pod0`
-addresses as `nodeAnnotations`, beside the `VLANConfig` that assigns them; the
-pod CIDRs come from the Node objects. An init container renders both from the
-API with `kubectl -o go-template-file`, excludes the node itself with a field
-selector, and ends with `bird -p` so BIRD's own parser gates startup.
+addresses as `nodeAnnotations` beside the `VLANConfig` that assigns them; the
+pod CIDRs come from the Node objects. bird's init container renders both from
+the API and ends with `bird -p`, so BIRD's own parser gates startup.
 
-The node's own CIDR is **originated**, not read back from the kernel. Reading
-it back made BIRD depend on the shape Cilium installs, and the IPv4 route is
-rejected outright: its next hop sits inside the prefix it describes, which KRT
-refuses with `strange next-hop`. The IPv6 route is a device route and passed,
-so the failure was silent on one family only.
+## Node↔node
+
+`kubelet.nodeIP.validSubnets` and `cluster.etcd.advertisedSubnets` point at
+VLAN 46. Talos picks one InternalIP per family, IPv4 first — the service CIDR
+is IPv4, and the apiserver's advertise address must match it.
+
+The API VIP stays on `mgmt0` for kubectl. No in-cluster client uses it:
+kubelet, scheduler, controller-manager and Cilium go through KubePrism
+(`localhost:7445`), which balances over every address of every control-plane
+member; pods use the ClusterIP.
+
+Talos never updates an existing etcd member's peer URLs
+(siderolabs/talos#12720). Changing `advertisedSubnets` means, per member:
+stage the config, `talosctl etcd forfeit-leadership`, `talosctl etcd leave`,
+reboot. `leave` removes only `/var/lib/etcd`; the node rejoins as a learner
+and is promoted. EPHEMERAL does not need a wipe.
+
+## Moving node IPs: the mixed state
+
+While some nodes have their InternalIP on the new network and some on the old,
+host→remote-pod traffic leaves with a node address that is not an InternalIP:
+the kernel picks the source from the route to the peer's InternalIP. Cilium
+classifies that as `world`, and every ingress policy that admits only
+`host`/`remote-node` (rook-ceph, monitoring, …) drops it. Kernel Ceph clients
+lose the OSDs on the other side; RBD mounts hang.
+
+Bridge it for the duration with a temporary `CiliumClusterwideNetworkPolicy`
+admitting the node subnets of both networks, both families, with
+`enableDefaultDeny: false` so it restricts nothing else, and delete it once
+every node has moved. Keep all OSD nodes schedulable while one is drained:
+draining two takes Ceph below its data availability.
 
 ## Failure behaviour
 
-Benign at every layer. An empty or absent table 48 makes 980/981 fall through
-to main, and pod traffic returns to VLAN 40 — unseparated, not broken. The same
-holds if the DaemonSet cannot start, which is the case during a cold rebuild
-until Gitea serves its image.
-
-## Management, the remaining class
-
-Not started. The levers differ per service and none of them is the node IP:
-
-| service | can leave the node IP? | lever |
-|---|---|---|
-| etcd | yes | `cluster.etcd.advertisedSubnets`; needs a reboot per node |
-| kube-apiserver endpoint | yes | it is a VIP — `Layer2VIPConfig.link` |
-| Talos API | n/a | `apid` answers on every node address; a firewall question |
-| kubelet | no | it *is* the node IP |
-
-Two approaches were rejected by the owner and are not to be proposed again:
-moving the node IP and etcd to `pod0` (#838), and relabelling VLAN 40 as the
-cluster VLAN while moving etcd and the API VIP to a new management VLAN.
-Replacing the CNI (#839, Antrea) was shelved.
-
-Note that the reasoning which produced that list also concluded pod↔pod was
-blocked, which policy routing disproved. Treat it as a record of decisions, not
-as an exhaustive analysis.
+If bird stops, table 47 empties, the rules find nothing and fall through to
+main: pod traffic returns to the node networks, unseparated rather than
+misrouted. The same holds during a cold rebuild until Gitea serves bird's
+image. The default route lives in table 47, not in the Talos config, so a
+default cannot outlive the specific routes and send pod traffic to the UDM.
 
 ## Sources
 
-Cilium v1.20.1 `pkg/datapath/linux/node.go`, `pkg/node/types/node.go`,
-`pkg/node/sync/local_node_sync.go`; BIRD 2.17 user manual (kernel protocol,
-`learn all`, `strict bind`, `direct`); Talos v1.13 `RoutingRuleConfig` and
-`machine.nodeAnnotations`. PRs #831, #833, #835, #836, #837, #842, #852, #854,
-#855; closed unmerged: #830, #838, #839.
+Cilium v1.20 `pkg/datapath/linux/node.go`; BIRD 2.17 user manual (kernel
+protocol, `strict bind`, `direct`); Talos v1.13 `RoutingRuleConfig`,
+`machine.nodeAnnotations`, configuration patching (lists are appended,
+`$patch: delete`), etcd `LeaveCluster`; KubePrism docs;
+siderolabs/talos#12720. PRs #831, #833, #835, #836, #837, #842, #852, #854,
+#855, #857, #858, #918, #919, #922; closed unmerged: #830, #838, #839.

kubernetes/RUNBOOK-cold-rebuild.md

@@ -107,7 +107,7 @@ config, and the nodes are addressed by name.
 ## Step 1b — boot all three machines from install media
 
 ```
-https://factory.talos.dev/image/0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9/v1.13.9/metal-amd64.iso
+https://factory.talos.dev/image/0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9/v1.13.10/metal-amd64.iso
 ```
 
 Write it to a USB stick and boot each MS-01 from it. Each comes up in

kubernetes/apps/gitea/kustomization.yaml

@@ -6,7 +6,6 @@ kind: Kustomization
 # and verified. See MIGRATION-gitea-woodpecker.md.
 resources:
   - statefulset.yaml
-  - pdb.yaml
   - service.yaml
   - ingressroute-ssh.yaml
   - ingress.yaml

kubernetes/apps/gitea/pdb.yaml

@@ -1,23 +0,0 @@
----
-# Eviction protection for the single-replica Gitea StatefulSet.
-#
-# Gitea runs on SQLite (single writer) and its repos + DB live on a single
-# ReadWriteOnce PVC, so it is deliberately replicas:1 and cannot fail over.
-# With no PDB, a node drain / maintenance evicts the pod abruptly, mid-write,
-# with zero coordination. minAvailable:1 on a 1-replica workload makes the
-# eviction API refuse *voluntary* evictions: `kubectl drain` blocks and
-# surfaces this pod, forcing a conscious decision (drain with
-# --disable-eviction, or scale down first) instead of yanking the live DB.
-# It does NOT block involuntary disruptions (node crash, OOM).
-apiVersion: policy/v1
-kind: PodDisruptionBudget
-metadata:
-  name: gitea
-  namespace: gitea
-  labels:
-    app.kubernetes.io/name: gitea
-spec:
-  minAvailable: 1
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: gitea

kubernetes/apps/grafana-db/postgres.yaml

@@ -4,7 +4,9 @@ metadata:
   name: gf-pg
   namespace: monitoring
 spec:
-  instances: 1
+  instances: 2
+  smartShutdownTimeout: 10
+  stopDelay: 60
   imageName: ghcr.io/cloudnative-pg/postgresql:16
   storage:
     storageClass: ceph-block

kubernetes/apps/home-assistant/ha-pg-cluster.yaml

@@ -30,7 +30,9 @@ metadata:
   name: ha-pg
   namespace: home-assistant
 spec:
-  instances: 1
+  instances: 2
+  smartShutdownTimeout: 10
+  stopDelay: 60
   imageName: ghcr.io/cloudnative-pg/postgresql:16
   storage:
     storageClass: ceph-block
@@ -124,8 +126,7 @@ spec:
       # padding rather than data.
       #
       # The cost is RPO: up to 30min of WAL is unrecoverable if this cluster and
-      # its volume are lost outright. This is a single-instance cluster, so the
-      # archive is the only off-node copy.
+      # its volumes are lost outright.
       archive_timeout: "30min"
       # Fewer checkpoints => fewer full-page writes, since a page is only
       # fully-logged on its first modification after a checkpoint.

kubernetes/apps/immich/cluster.yaml

@@ -59,8 +59,7 @@ spec:
       # padding rather than data.
       #
       # The cost is RPO: up to 30min of WAL is unrecoverable if this cluster and
-      # its volume are lost outright. This is a single-instance cluster, so the
-      # archive is the only off-node copy.
+      # its volumes are lost outright.
       archive_timeout: "30min"
       # Write-reduction tuning. These databases are largely idle, and the
       # Postgres defaults favour throughput over storage that wears out.

kubernetes/apps/influxdb/helmrelease.yaml

@@ -61,6 +61,9 @@ spec:
       enabled: true
       storageClass: ceph-block
       size: 50Gi
+    # A minAvailable:1 PDB on a single replica blocks every node drain.
+    pdb:
+      create: false
     # Bootstrap admin/org/bucket on the fresh PVC. Admin password + token come
     # from the influxdb-bootstrap Secret (OpenBao via ESO), deterministic across
     # rebuilds so the bootstrap token is the live admin token. `influx setup`

kubernetes/apps/monitoring-rules/prometheusrule-openbao.yaml

@@ -57,10 +57,14 @@ spec:
           # the ClusterSecretStore goes InvalidProviderConfig and no
           # ExternalSecret can refresh.
           #
-          # No window: below quorum the secret store is down and every
-          # ExternalSecret stalls with it, so waiting only delays the page.
+          # Short window: readiness is only a proxy for quorum. A one-second
+          # node NotReady flap makes the node lifecycle controller mark that
+          # node's pods not-ready for ~30s while raft keeps its leader and
+          # serves (issue #920, during a rolling reboot with one replica
+          # already evicted). A really sealed or crashed majority persists,
+          # and existing Secrets keep their last values, so 2m costs little.
           expr: max by (namespace, statefulset) (kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"}) < 2
-          for: 0m
+          for: 2m
           labels:
             severity: critical
           annotations:

kubernetes/apps/paperless/kustomization.yaml

@@ -11,7 +11,6 @@ resources:
   - protonmail-bridge.yaml
   - ftp-scanner.yaml
   - syncthing.yaml
-  - pdb.yaml
   - network-policy.yaml
   # pprl-pg CNPG Postgres + data/media/consume/redis PVCs (moved here from the
   # retired seed-paperless KS; each carries prune: disabled so ownership

kubernetes/apps/paperless/pdb.yaml

@@ -1,37 +0,0 @@
----
-# Eviction protection for the single-replica stateful paperless sidecars.
-#
-# protonmail-bridge and syncthing each hold state on a single RWO PVC and run
-# replicas:1. minAvailable:1 makes the eviction API refuse voluntary evictions,
-# so a node drain blocks and surfaces these pods instead of evicting them
-# abruptly. Involuntary disruptions (node crash) are not affected.
-#
-# NOTE: the other single-replica paperless pods (paperless, redis, gotenberg,
-# tika, ftp-scanner) are intentionally omitted — they are caches / stateless
-# converters where a blocking PDB would only wedge node drains with no
-# recoverability benefit.
-apiVersion: policy/v1
-kind: PodDisruptionBudget
-metadata:
-  name: protonmail-bridge
-  namespace: paperless
-  labels:
-    app.kubernetes.io/name: protonmail-bridge
-spec:
-  minAvailable: 1
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: protonmail-bridge
----
-apiVersion: policy/v1
-kind: PodDisruptionBudget
-metadata:
-  name: syncthing
-  namespace: paperless
-  labels:
-    app.kubernetes.io/name: syncthing
-spec:
-  minAvailable: 1
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: syncthing

kubernetes/apps/trmnl/cluster.yaml

@@ -4,7 +4,7 @@ metadata:
   name: tr-pg
   namespace: trmnl
 spec:
-  instances: 1
+  instances: 2
   smartShutdownTimeout: 10
   stopDelay: 60
   # PG 18, NOT the repo-standard 16. Terminus ships a committed
@@ -47,8 +47,7 @@ spec:
       # padding rather than data.
       #
       # The cost is RPO: up to 30min of WAL is unrecoverable if this cluster and
-      # its volume are lost outright. This is a single-instance cluster, so the
-      # archive is the only off-node copy.
+      # its volumes are lost outright.
       archive_timeout: "30min"
       # Write-reduction tuning. These databases are largely idle, and the
       # Postgres defaults favour throughput over storage that wears out.

kubernetes/apps/woodpecker/cluster.yaml

@@ -22,7 +22,9 @@ metadata:
     # migration deliberately withheld this while a rollback still existed.
     kustomize.toolkit.fluxcd.io/prune: disabled
 spec:
-  instances: 1
+  instances: 2
+  smartShutdownTimeout: 10
+  stopDelay: 60
   imageName: ghcr.io/cloudnative-pg/postgresql:16
   storage:
     storageClass: ceph-block
@@ -91,8 +93,7 @@ spec:
       # padding rather than data.
       #
       # The cost is RPO: up to 30min of WAL is unrecoverable if this cluster and
-      # its volume are lost outright. This is a single-instance cluster, so the
-      # archive is the only off-node copy.
+      # its volumes are lost outright.
       archive_timeout: "30min"
   resources:
     requests:

kubernetes/apps/woodpecker/kustomization.yaml

@@ -3,7 +3,6 @@ kind: Kustomization
 resources:
   - rbac.yaml
   - server.yaml
-  - pdb.yaml
   - agent.yaml
   - service.yaml
   - ingress.yaml
@@ -22,7 +21,7 @@ resources:
   # OpenBao access, no NetworkPolicy needed. server.yaml/agent.yaml keep their
   # Reloader annotations; externalsecret.yaml keeps the 10m refreshInterval.
   # --- namespace migration, stages 1-2 (issue #433) ---
-  # Stage 2 has moved the workloads above (server, agent, service, ingress, pdb,
+  # Stage 2 has moved the workloads above (server, agent, service, ingress,
   # rbac, secrets, policies) into the `woodpecker` namespace and pointed the
   # server's DSN at wp-pg-rw.woodpecker.svc. Stage 3 retired the `apps`
   # originals; these four now describe the surviving cluster. History and the

kubernetes/apps/woodpecker/pdb.yaml

@@ -1,27 +0,0 @@
----
-# Eviction protection for the single-replica Woodpecker server StatefulSet.
-#
-# The server is SQLite-backed (single writer) on a single RWO PVC, so it is
-# deliberately replicas:1. minAvailable:1 makes the eviction API refuse
-# voluntary evictions so a node drain blocks and surfaces this pod rather
-# than yanking the live CI DB mid-write. Involuntary disruptions are not
-# affected.
-#
-# The selector MUST include app.kubernetes.io/component: server: the agent
-# Deployment in this same namespace shares app.kubernetes.io/name: woodpecker
-# (component: agent), and a name-only selector would wrongly also cover the
-# stateless, disposable agents.
-apiVersion: policy/v1
-kind: PodDisruptionBudget
-metadata:
-  name: woodpecker-server
-  namespace: woodpecker
-  labels:
-    app.kubernetes.io/name: woodpecker
-    app.kubernetes.io/component: server
-spec:
-  minAvailable: 1
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: woodpecker
-      app.kubernetes.io/component: server

kubernetes/infrastructure/bird/bird.conf

@@ -23,7 +23,7 @@ protocol device { scan time 10; }
 # down. The usual advice is not to export these to the kernel because it
 # maintains them itself — true of main, not of a table the kernel never fills.
 protocol direct attached {
-  interface "mgmt0", "ingress0", "pod0";
+  interface "mgmt0", "ingress0", "pod0", "node0";
   ipv4 { table master4; };
   ipv6 { table master6; };
 }

kubernetes/infrastructure/bird/daemonset.yaml

@@ -48,8 +48,8 @@ spec:
         tier: node
         app: bird
     spec:
-      # Carries pod east-west once the routing rules land. Without it table 48
-      # stays empty and the rules fall through to main, i.e. back to VLAN 40.
+      # Owns table 47, where all pod traffic is routed. Without it the table
+      # stays empty and the rules fall through to main, unseparated.
       priorityClassName: system-node-critical
       # Writes host routes and peers on host addresses.
       hostNetwork: true

kubernetes/infrastructure/k8s-gateway/helmrelease.yaml

@@ -1,11 +1,16 @@
 apiVersion: source.toolkit.fluxcd.io/v1
-kind: HelmRepository
+kind: OCIRepository
 metadata:
   name: k8s-gateway
   namespace: kube-system
 spec:
   interval: 24h
-  url: https://k8s-gateway.github.io/k8s_gateway/
+  url: oci://codeberg.org/k8s-gateway/charts/k8s-gateway
+  ref:
+    semver: ">=2.4.0"
+  layerSelector:
+    mediaType: "application/vnd.cncf.helm.chart.content.v1.tar+gzip"
+    operation: copy
 ---
 # In-cluster authoritative resolver for renner.uno.
 #
@@ -28,14 +33,10 @@ metadata:
   namespace: kube-system
 spec:
   interval: 30m
-  chart:
-    spec:
-      chart: k8s-gateway
-      version: ">=2.4.0"
-      sourceRef:
-        kind: HelmRepository
-        name: k8s-gateway
-        namespace: kube-system
+  chartRef:
+    kind: OCIRepository
+    name: k8s-gateway
+    namespace: kube-system
   install:
     remediation:
       retries: 3

kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml

@@ -21,17 +21,24 @@ spec:
     modules:
       - name: pg_autoscaler
         enabled: true
-  # Rotates daemon keys to aes256k for CVE-2025-30156 (#527). Opt-in; disabled
-  # by default. Increment keyGeneration to rotate again.
-  #
-  # Daemon keys only — aes256k over the CSI mount path needs kernel 7.0+ and
-  # these nodes run 6.18, so CSI stays on aes and the client-key checks stay
-  # raised. keyType is unset so Rook picks it.
+  # CephX keys on aes256k for CVE-2025-30156 (#527). Increment keyGeneration
+  # to rotate again. CSI kernel mounts need aes256k in libceph: Talos
+  # >= v1.13.10 backports it to 6.18 (siderolabs/pkgs 2da2cdc).
   security:
     cephx:
       daemon:
         keyRotationPolicy: KeyGeneration
         keyGeneration: 2
+      csi:
+        keyRotationPolicy: KeyGeneration
+        keyGeneration: 2
+        keyType: aes256k
+        # Keeps the prior key valid for PVCs mounted before the rotation.
+        keepPriorKeyCountMax: 1
+      rbdMirrorPeer:
+        keyRotationPolicy: KeyGeneration
+        keyGeneration: 2
+        keyType: aes256k
   # CephFS cloner pool raised from the default 4. Velero clones a
   # VolumeSnapshot per PVC, and 4 threads stall a multi-PVC backup with
   # `rados: "all cloner threads are busy"`.

kubernetes/talos/patches/cluster.yaml

@@ -24,7 +24,7 @@ machine:
   # The Kubernetes version is a separate decision, moved by
   # `talosctl upgrade-k8s`.
   install:
-    image: factory.talos.dev/metal-installer/0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9:v1.13.9
+    image: factory.talos.dev/metal-installer/0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9:v1.13.10
   features:
     rbac: true
     # stableHostname is NOT here: it is the v1alpha1 spelling of what the
@@ -58,12 +58,12 @@ machine:
       imageGCHighThresholdPercent: 70
       imageGCLowThresholdPercent: 60
 cluster:
-  # etcd advertises on the management VLAN until the node↔node move (#576
-  # step 3). Pinned so the node0 addresses cannot be picked on a reboot: Talos
-  # never updates an existing member's peer URLs.
+  # etcd peers on node0, VLAN 46 (#576). Pinned: Talos never updates an
+  # existing member's peer URLs, so changing this subnet means leave + rejoin
+  # per member (talosctl etcd leave, reboot).
   etcd:
     advertisedSubnets:
-      - 10.2.40.0/24
+      - 10.2.46.0/24
   # controlPlane.endpoint is NOT here: `talosctl gen config` takes it as its
   # second argument, and bootstrap.sh already passes https://k8s.renner.uno:6443
   # there. Setting it again in this patch restated the same value in a second

kubernetes/talos/patches/k8s-1.yaml

@@ -124,8 +124,8 @@ kind: LinkConfig
 name: uplink
 up: true
 ---
-# Management, VLAN 40. Carries the node IP, etcd, the API and Ceph's public
-# network.
+# Management, VLAN 40: world↔node. The API VIP, admin access to the nodes, and
+# node egress (DNS, NTP, image pulls). Node↔node traffic is on node0 (#576).
 apiVersion: v1alpha1
 kind: VLANConfig
 name: mgmt0
@@ -181,7 +181,7 @@ addresses:
 ---
 # Node network, VLAN 46: traffic among the three nodes themselves (#576) — the
 # node IP, etcd, apiserver↔kubelet. Closed like pod0: statically addressed, no
-# gateway, never routed via the UDM. Unused until the node IP moves here.
+# gateway, never routed via the UDM.
 apiVersion: v1alpha1
 kind: VLANConfig
 name: node0

kubernetes/talos/patches/k8s-2.yaml

@@ -83,7 +83,7 @@ kind: LinkConfig
 name: uplink
 up: true
 ---
-# Management, VLAN 40. Node IP, etcd, the API and Ceph's public network.
+# Management, VLAN 40: world↔node — see k8s-1.yaml.
 apiVersion: v1alpha1
 kind: VLANConfig
 name: mgmt0

kubernetes/talos/patches/site-homelab.yaml

@@ -7,19 +7,15 @@
 # part of the config that could not be true anywhere but here.
 machine:
   kubelet:
-    # Constrain kubelet's auto-picked node IP to the mgmt LAN. Without this,
-    # kubelet picks the first non-loopback IP, which could land on the v30iot
-    # NIC (ens19, 10.2.30.0/24). Control-plane traffic and hostNetwork pod IPs
-    # must stay on mgmt LAN.
-    # fd40::/64 is the mgmt LAN ULA, and it is listed so the Node object carries
-    # an IPv6 InternalIP at all. Without one, Cilium falls back to detecting the
-    # node's IPv6 from mgmt0 and picks the SLAAC address from the dynamic
-    # provider prefix, which it then probes until the agent restarts (#789).
-    # The GUA is not a control-plane address: BGP peers on fd40::/64 too.
+    # The node IP lives on node0, VLAN 46: the closed node↔node network (#576).
+    # Both families are listed so the Node object carries an IPv6 InternalIP at
+    # all; without one Cilium detects the node's IPv6 itself and picks a SLAAC
+    # address from the dynamic provider prefix (#789). Talos picks one address
+    # per family, IPv4 first, whatever the order here.
     nodeIP:
       validSubnets:
-        - 10.2.40.0/24
-        - fd40::/64
+        - fd02:46::/64
+        - 10.2.46.0/24
   # The UDM on the ingress segment. Published here rather than as a route
   # because bird owns table 47 and originates the default from it; this is the
   # one place the address is stated.