fix(cilium): roll the agent DaemonSet when cilium-config changes

Cilium reads datapath options once, at agent start, and the chart puts no
config checksum on the pod template. A changed ConfigMap therefore leaves every
agent running the previous datapath until something restarts it, and nothing
does — the DaemonSet is not rolled by a ConfigMap-only Helm change.

This already cost an outage. vlan-bpf-bypass landed in cilium-config and every
v30iot macvlan workload stayed broken until the agents were deleted by hand,
one node at a time (issue #592). The merge looked successful: HelmRelease
Ready, release upgraded, key present in the ConfigMap, and nothing working.

policyMapMax carries the same exposure. Its comment says "takes effect on agent
restart", which is documentation standing in for automation — as does the
equivalent note in the two bootstrap copies.

Reloader is already deployed and used across the repo for the same purpose.
Named per-ConfigMap rather than blanket, matching the existing convention.

The trade, stated plainly: any cilium-config change now rolls the CNI
cluster-wide, node by node. That is the same disruption a datapath change
already required; the difference is that it happens on its own rather than
being forgotten.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kubernetes/infrastructure/cilium/helmrelease.yaml

@@ -33,6 +33,24 @@ spec:
     remediation:
       retries: 3
   values:
+    # Roll the agent DaemonSet when cilium-config changes. Cilium reads
+    # datapath options once, at agent start, and the chart puts no config
+    # checksum on the pod template — so a changed ConfigMap leaves every agent
+    # running the previous datapath until something restarts it. Nothing did.
+    #
+    # This is not hypothetical. vlan-bpf-bypass reached cilium-config and every
+    # v30iot macvlan workload stayed broken until the agents were deleted by
+    # hand, one node at a time (issue #592). policyMapMax has the same
+    # exposure, and its comment below says "takes effect on agent restart" —
+    # documentation standing in for automation.
+    #
+    # The trade: any cilium-config change now rolls the CNI cluster-wide,
+    # node by node. That is disruptive-ish, but it is what a datapath change
+    # already required — the difference is that it now happens on its own
+    # instead of being forgotten. Reloader is already deployed for exactly
+    # this pattern; named per-ConfigMap rather than blanket, as elsewhere.
+    annotations:
+      configmap.reloader.stakater.com/reload: cilium-config
     # Which pod labels feed security-identity computation. Patterns are
     # start-anchored regexes; a leading "!" excludes while keeping the default
     # set. Woodpecker stamps every step pod with woodpecker-ci.org/task-uuid

kubernetes/infrastructure/rook-ceph-cluster/storageclass.yaml

@@ -5,31 +5,24 @@ metadata:
   namespace: rook-ceph
 spec:
   failureDomain: host
-  # size 2, NOT 3. With 3 OSDs on 3 hosts, size=3 meant every write landed on
-  # every drive in the cluster: per-drive write volume equalled total cluster
-  # write volume, and all three drives wore at an identical rate and would have
-  # reached end-of-rated-life together (measured 2026-08-19: 80/82/87% endurance
-  # used, ~0.13 points/day, all three exhausted between Nov 2026 and Jan 2027 —
-  # see #434). Replication does not protect against that failure mode, because
-  # wear-out here is correlated by construction rather than independent.
+  # size 3. This pool sat at 2 while the OSDs were Crucial P3 QLC drives: with
+  # 3 OSDs on 3 hosts, size=3 put every write on every drive, so per-drive write
+  # volume equalled total cluster write volume and all three wore at an
+  # identical rate (measured 2026-08-19: 80/82/87% endurance used, ~0.13
+  # points/day, all three projected to exhaust between Nov 2026 and Jan 2027 —
+  # see #434). Replication does not protect against that, because the wear was
+  # correlated by construction rather than independent.
   #
-  # size 2 cuts per-drive write volume by a third and desynchronises the wear.
+  # The OSDs are Samsung PM983 enterprise drives now, at 2-3% of rated
+  # endurance with a far higher write budget, so wear-out is no longer the
+  # binding constraint and the correlation argument no longer applies.
   #
-  # min_size 1 is REQUIRED and deliberate. Ceph does not lower min_size when
-  # size is reduced — it was 2 here, and leaving it would make the pool 2/2,
-  # where any single OSD down (every rolling node reboot, every OSD restart)
+  # min_size is left at the size-3 default of 2. The explicit min_size 1
+  # override existed only because Ceph does not lower min_size when size is
+  # reduced, which would have left the pool 2/2 — where any single OSD down
   # blocks IO for its PGs and hangs every RBD volume in the cluster.
-  #
-  # The trade this accepts: while the pool is degraded (one OSD down), a second
-  # drive failing loses data, and with all three drives at 80%+ endurance that
-  # window is not hypothetical. Mitigations that already exist: CNPG barman and
-  # Velero both ship offsite to Hetzner (opentofu/object_storage.tf), so the
-  # exposure is the RPO of those backups, not total loss. Keep node reboots
-  # serialised and wait for HEALTH_OK between them.
   replicated:
-    size: 2
-  parameters:
-    min_size: "1"
+    size: 3
 ---
 apiVersion: storage.k8s.io/v1
 kind: StorageClass
@@ -88,10 +81,9 @@ metadata:
   namespace: rook-ceph
 spec:
   # size 3 with no min_size override. This pool holds ~253 MiB against the data
-  # pool's 280 GiB, so it contributes nothing measurable to drive wear — the
-  # entire reason data0 sits at 2 — while losing it would destroy the whole
-  # filesystem, not just the objects on the lost OSD. Paying 3x on a quarter of
-  # a gigabyte to keep that intact is the right trade.
+  # pool's 280 GiB, and losing it would destroy the whole filesystem rather
+  # than just the objects on the lost OSD, so it is the last pool that should
+  # ever run thin.
   #
   # It drops to 2/1 whenever one of the three hosts is out for longer than a
   # reboot. size 3 on two remaining hosts leaves every PG undersized at two
@@ -104,12 +96,10 @@ spec:
       size: 3
   dataPools:
     - name: data0
-      # size 2 / min_size 1 for the same reasons as replicapool above — see
-      # the comment there for the full rationale and the accepted trade.
+      # size 3 for the same reasons as replicapool above — see the comment
+      # there for why this pool sat at 2 while the Crucials backed the OSDs.
       replicated:
-        size: 2
-      parameters:
-        min_size: "1"
+        size: 3
   metadataServer:
     activeCount: 1
     activeStandby: true