docs(infra): trim the remaining oversized comments

crowdsec, netbird-ingress, woodpecker-ns, san-network-attachment, cilium-bgp and the openbao rotation kustomization. Comments only; the diff filtered for non-comment lines is empty and infrastructure/ still builds. Every instruction survives, because that is the half worth having: do not switch crowdsec back to password auth, or a wiped LAPI crashloops the agents and wedges the Kustomization; do not move netbird-ingress off the rootless image, or socket-LB stops seeing its traffic and the VIPs blackhole; do not pin the rotators by digest, or a quarterly job fails to pull a manifest that was garbage-collected months earlier and looks like it simply did not run. What went is the incident each was learned from, and one duplicate: the san-network-attachment header re-derived the provider: host argument that cephcluster.yaml already makes, so it now points there instead. Two more stale claims found on the way through: - woodpecker-ns.yaml described the namespace as holding wp-pg alone with 'server and agent arriving in stage 2'. Both have been running there for some time. - cilium-bgp.yaml said each node sources its BGP session from eth0. The management interface is mgmt0; eth0 was the VM-era name. infrastructure/ is now 2553 comment lines.
docs(infra): trim the cilium and traefik helm release comments
2026-08-28 06:50:54 +02:00 · 2026-08-28 06:47:39 +02:00 · 2026-08-28 04:45:23 +00:00 · 2026-08-28 06:43:03 +02:00 · 2026-08-28 04:41:06 +00:00 · 2026-08-28 06:39:44 +02:00
34 changed files with 728 additions and 809 deletions
@@ -1,18 +0,0 @@
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: barman-cloud-creds
-  namespace: authentik
-  annotations:
-    # Prune-disabled — a standing guard kept from the seed->authentik-KS handoff (barman creds).
-    kustomize.toolkit.fluxcd.io/prune: disabled
-spec:
-  refreshInterval: 1h
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: barman-cloud-creds # keeps the old Secret's name — consumers unchanged
-  dataFrom:
-    - extract:
-        key: seed/authentik/barman-creds
@@ -15,4 +15,4 @@ spec:
    name: hetzner-s3-creds # keeps the old Secret's name — consumers unchanged
  dataFrom:
    - extract:
-        key: seed/authentik/hetzner-creds
+        key: infrastructure/hetzner-s3
@@ -27,5 +27,4 @@ resources:
  - cluster.yaml
  - objectstore.yaml
  - scheduled-backup.yaml
-  - barman-creds-externalsecret.yaml
  - hetzner-creds-externalsecret.yaml
@@ -1,21 +1,8 @@
 ---
 # Barman S3 credentials for ha-pg.
 #
-# FOLLOW-UP (#397): sourced from seed/paperless/hetzner-creds, which is not where
-# they belong. woodpecker/hetzner-creds-externalsecret.yaml has the identical
-# problem and the same note. The values are the same today, so this is correct
-# but coupled: rotating paperless's Hetzner credentials would silently change
-# the backup credentials for Home Assistant and Woodpecker too.
-#
-# The fix is one dedicated path per consumer (seed/home-assistant/hetzner-creds)
-# and a one-line change to the `key:` below. Creating it needs an OpenBao write:
-#
-#   bao kv put homelab/seed/home-assistant/hetzner-creds \
-#     @<(bao kv get -format=json homelab/seed/paperless/hetzner-creds | jq '.data.data')
-#
-# Note the mount is `homelab`, so the path is homelab/seed/... — #397 records a
-# 403 on `bao kv list seed`, which was a wrong-path error rather than a
-# permissions problem.
+# One shared path for one shared credential — see the note in
+# ../paperless/hetzner-creds-externalsecret.yaml.
 #
 # The namespace allowlist on the openbao ClusterSecretStore
 # (infrastructure/post-crds/openbao-clustersecretstore.yaml) must include this
@@ -35,4 +22,4 @@ spec:
    name: hetzner-s3-creds
  dataFrom:
    - extract:
-        key: seed/paperless/hetzner-creds
+        key: infrastructure/hetzner-s3
@@ -1,18 +0,0 @@
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: barman-cloud-creds
-  namespace: immich
-  annotations:
-    # Prune-disabled — a standing guard kept from the seed->immich-KS handoff (barman creds).
-    kustomize.toolkit.fluxcd.io/prune: disabled
-spec:
-  refreshInterval: 1h
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: barman-cloud-creds # keeps the old Secret's name — consumers unchanged
-  dataFrom:
-    - extract:
-        key: seed/immich/barman-creds
@@ -15,4 +15,4 @@ spec:
    name: hetzner-s3-creds # keeps the old Secret's name — consumers unchanged
  dataFrom:
    - extract:
-        key: seed/immich/hetzner-creds
+        key: infrastructure/hetzner-s3
@@ -13,6 +13,5 @@ resources:
  - cluster.yaml
  - objectstore.yaml
  - scheduled-backup.yaml
-  - barman-creds-externalsecret.yaml
  - hetzner-creds-externalsecret.yaml
  - pvcs.yaml
@@ -6,6 +6,7 @@ kind: Kustomization
 resources:
  - prometheusrule-ceph.yaml
  - prometheusrule-cilium.yaml
+  - prometheusrule-disk-wear.yaml
  - prometheusrule-hetzner-backup.yaml
  - prometheusrule-loki.yaml
  - prometheusrule-mariadb-backup.yaml
@@ -24,4 +25,5 @@ resources:
  - home-assistant-servicemonitor.yaml
  - immich-servicemonitor.yaml
  - loki-servicemonitor.yaml
+  - smartctl-exporter-servicemonitor.yaml
  - probes-blackbox.yaml
@@ -0,0 +1,92 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: disk-wear
+  namespace: monitoring
+  labels:
+    # kube-prometheus-stack's Prometheus resource selects on this label;
+    # without it the rule is created but never loaded into Prometheus.
+    release: kube-prometheus-stack
+spec:
+  groups:
+    - name: disk-wear.rules
+      rules:
+        # Wear percentages are self-reported: 100 means the rated endurance is
+        # spent, not that the drive stops. Planning thresholds, not predictions.
+        - alert: NvmeDriveWearHigh
+          # Long `for` because wear is monotonic and cannot flap.
+          expr: smartctl_device_percentage_used >= 80
+          for: 1h
+          labels:
+            severity: warning
+          annotations:
+            summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} is at {{ $value }}% rated wear"
+            description: >-
+              {{ $value }}% of rated write endurance used. Nothing is failing
+              yet. Urgency depends on what the drive carries — the Disk Health
+              dashboard shows the role. Losing an OSD costs a rebalance;
+              EPHEMERAL is /var and rebuilds, apart from the node's etcd member
+              data, which has to be removed and re-added.
+        - alert: NvmeDriveWearCritical
+          expr: smartctl_device_percentage_used >= 95
+          for: 1h
+          labels:
+            severity: critical
+          annotations:
+            summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} is at {{ $value }}% rated wear"
+            description: >-
+              The drive is at or past the end of its rated endurance. It will
+              likely keep working for a while, but it is now operating outside
+              what the manufacturer stands behind, and replacement should be
+              scheduled rather than considered.
+        # Wear is a budget; spare exhaustion is damage.
+        - alert: NvmeDriveSpareExhausted
+          expr: smartctl_device_available_spare <= smartctl_device_available_spare_threshold
+          for: 15m
+          labels:
+            severity: critical
+          annotations:
+            summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} has exhausted its spare area"
+            description: >-
+              Available spare has fallen to the manufacturer's threshold. The
+              drive has run out of reserve blocks to remap failures into.
+              Replace it.
+        - alert: NvmeDriveCriticalWarning
+          expr: smartctl_device_critical_warning > 0
+          for: 5m
+          labels:
+            severity: critical
+          annotations:
+            summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} reports critical warning {{ $value }}"
+            description: >-
+              The drive has raised its own critical warning flag. Read the
+              detail with `smartctl -a` on the node; the value is a bitfield,
+              not a count.
+        - alert: NvmeDriveMediaErrors
+          # On the increase, not the total: pre-existing errors would otherwise
+          # alert forever. Standing counts are visible on the dashboard only.
+          expr: increase(smartctl_device_media_errors[6h]) > 0
+          for: 5m
+          labels:
+            severity: warning
+          annotations:
+            summary: "NVMe {{ $labels.device }} on {{ $labels.instance }} recorded {{ $value }} new media errors"
+            description: >-
+              The drive reported unrecoverable data-integrity errors in the
+              last six hours. A healthy drive never increments this. Check
+              whether Ceph has logged scrub errors on the same node.
+        # The check that the checks are running: no series fires no alert.
+        # Counted against node-exporter's inventory, not a fixed fleet size.
+        - alert: SmartctlExporterIncomplete
+          expr: count(smartctl_device_percentage_used) < count(node_nvme_info) or absent(smartctl_device_percentage_used)
+          for: 30m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Fewer NVMe drives report SMART than node-exporter can see"
+            description: >-
+              Some drives visible to node-exporter are not reporting SMART, or
+              none are. Either a DaemonSet pod is down or a drive stopped
+              answering — both mean the wear alerts above have stopped watching
+              part of the fleet, which looks identical to everything being
+              fine.
@@ -0,0 +1,25 @@
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: smartctl-exporter
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/name: smartctl-exporter
+    # kube-prometheus-stack's Prometheus resource selects ServiceMonitors
+    # by this label; without it the resource exists but is never
+    # discovered.
+    release: kube-prometheus-stack
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: smartctl-exporter
+  endpoints:
+    # smartctl polls the drives every 60s; scraping faster gains nothing.
+    - port: http-metrics
+      path: /metrics
+      interval: 60s
+      relabelings:
+        # Without this a series carries only the exporter pod's IP, which does
+        # not identify the machine the drive is in.
+        - sourceLabels: [__meta_kubernetes_pod_node_name]
+          targetLabel: node
@@ -1,6 +1,6 @@
 {
  "annotations": {"list": []},
-  "description": "NVMe throughput, latency and temperature per node. Every panel joins node_disk_* to node_nvme_info and node_uname_info on purpose: device names are NOT stable across nodes (nvme0 is the Kingston system disk on k8s-1 and k8s-3, but the Crucial on k8s-2), so a legend keyed on the device alone compares different hardware. Endurance/wear is deliberately absent — see the note at the bottom and issue #549.",
+  "description": "NVMe health and performance per node. The top row answers 'is anything wrong'; the graphs below answer 'why'. Panels resolve each drive through node_nvme_info and node_uname_info rather than plotting the Linux device name, because NVMe enumeration order is not guaranteed to agree across machines — the same name can be a different drive on the next node. Which drive currently holds which role is in the inventory table, deliberately not in this text. Endurance comes from smartctl_exporter, whose series are labelled by node rather than by the exporter pod's IP — see the note at the bottom for what it still cannot tell you.",
  "editable": true,
  "fiscalYearStartMonth": 0,
  "graphTooltip": 1,
@@ -9,7 +9,89 @@
  "panels": [
    {
      "datasource": {"type": "prometheus", "uid": "prometheus"},
-      "description": "Which physical drive is which, resolved live rather than written down. The OSD column comes from ceph_disk_occupation, so it follows Rook instead of restating a device list that would go stale the next time a drive is swapped.",
+      "description": "Ceph's overall verdict. A HEALTH_WARN is frequently raised by something unrelated to the drives, so read the detail before treating it as a storage problem.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "mappings": [{"options": {"0": {"color": "green", "index": 0, "text": "HEALTH_OK"}, "1": {"color": "orange", "index": 1, "text": "HEALTH_WARN"}, "2": {"color": "red", "index": 2, "text": "HEALTH_ERR"}}, "type": "value"}],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 1}, {"color": "red", "value": 2}]},
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 5, "x": 0, "y": 0},
+      "id": 10,
+      "options": {"colorMode": "background", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
+      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "ceph_health_status", "legendFormat": "", "refId": "A"}],
+      "title": "Ceph health",
+      "type": "stat"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "OSDs the cluster knows about but which are not up. Should read zero. Counted as a difference against ceph_osd_metadata rather than compared to a fixed number, so adding or removing an OSD does not turn this panel into a false alarm. An OSD that is up but out shows as a Ceph health warning instead — it is a state, not a fault, and it does not belong in a red-or-green box.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "red", "value": 1}]},
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 5, "x": 5, "y": 0},
+      "id": 11,
+      "options": {"colorMode": "background", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
+      "targets": [
+        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "sum(ceph_osd_metadata) - sum(ceph_osd_up)", "legendFormat": "", "refId": "A"}
+      ],
+      "title": "OSDs down",
+      "type": "stat"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Raw Ceph capacity in use. The cluster average is the optimistic number: Ceph stops writing when a single OSD crosses the full ratio, so per-OSD headroom is what actually bites.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "mappings": [],
+          "max": 1,
+          "min": 0,
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 0.75}, {"color": "red", "value": 0.85}]},
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 5, "x": 10, "y": 0},
+      "id": 12,
+      "options": {"colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
+      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "ceph_cluster_total_used_bytes / ceph_cluster_total_bytes", "legendFormat": "", "refId": "A"}],
+      "title": "Ceph capacity used",
+      "type": "stat"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Fullest EPHEMERAL volume across the fleet — Talos's /var. Its contents are meant to be disposable: container images, kubelet state and pod scratch all rebuild. Filling it is still a problem, because a node with no room to write stops being useful long before anything is lost, and nothing rebalances /var the way Ceph rebalances a full OSD.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "mappings": [],
+          "max": 1,
+          "min": 0,
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 0.7}, {"color": "red", "value": 0.85}]},
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 5, "x": 15, "y": 0},
+      "id": 13,
+      "options": {"colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
+      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "max(1 - node_filesystem_avail_bytes{mountpoint=\"/var\"} / node_filesystem_size_bytes{mountpoint=\"/var\"})", "legendFormat": "", "refId": "A"}],
+      "title": "EPHEMERAL used (worst node)",
+      "type": "stat"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Which physical drive is which, resolved live rather than written down — this table is the answer to 'what is in the machines', so that no panel description has to be. The OSD column is joined from ceph_disk_occupation, so it follows Rook; a drive with no OSD is carrying either the system disk or EPHEMERAL.",
      "fieldConfig": {
        "defaults": {
          "color": {"mode": "thresholds"},
@@ -19,27 +101,121 @@
        },
        "overrides": []
      },
-      "gridPos": {"h": 9, "w": 24, "x": 0, "y": 0},
+      "gridPos": {"h": 9, "w": 24, "x": 0, "y": 4},
      "id": 1,
      "options": {"cellHeight": "sm", "footer": {"countRows": false, "fields": "", "reducer": ["sum"], "show": false}, "showHeader": true, "sortBy": [{"desc": false, "displayName": "node"}]},
      "targets": [
        {
          "datasource": {"type": "prometheus", "uid": "prometheus"},
-          "expr": "node_nvme_info * on (instance) group_left(nodename) node_uname_info",
+          "expr": "(label_replace(node_nvme_info * on (instance) group_left(nodename) node_uname_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") * on (nodename, nvmedev) group_left(ceph_daemon) label_replace(label_replace(ceph_disk_occupation, \"nodename\", \"$1\", \"instance\", \"(.*)\"), \"nvmedev\", \"$1\", \"devices\", \"(nvme[0-9]+)n[0-9]+\")) or (label_replace(node_nvme_info * on (instance) group_left(nodename) node_uname_info, \"nvmedev\", \"$1\", \"device\", \"(nvme[0-9]+)\") unless on (nodename, nvmedev) label_replace(label_replace(ceph_disk_occupation, \"nodename\", \"$1\", \"instance\", \"(.*)\"), \"nvmedev\", \"$1\", \"devices\", \"(nvme[0-9]+)n[0-9]+\"))",
          "format": "table",
          "instant": true,
          "legendFormat": "",
          "refId": "A"
        }
      ],
-      "title": "Drive inventory",
+      "title": "Drive inventory (identity and role)",
      "transformations": [
        {
          "id": "organize",
          "options": {
-            "excludeByName": {"Time": true, "Value": true, "__name__": true, "cntlid": true, "container": true, "endpoint": true, "instance": true, "job": true, "namespace": true, "pod": true, "service": true, "domainname": true, "machine": true, "release": true, "sysname": true, "version": true, "state": true},
+            "excludeByName": {"Time": true, "Value": true, "__name__": true, "cntlid": true, "container": true, "endpoint": true, "instance": true, "job": true, "namespace": true, "pod": true, "service": true, "domainname": true, "machine": true, "release": true, "sysname": true, "version": true, "state": true, "nvmedev": true},
            "indexByName": {},
-            "renameByName": {"nodename": "node", "device": "device", "model": "model", "serial": "serial", "firmware_revision": "firmware"}
+            "renameByName": {"nodename": "node", "device": "device", "model": "model", "serial": "serial", "firmware_revision": "firmware", "ceph_daemon": "ceph OSD"}
+          }
+        }
+      ],
+      "type": "table"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Rated endurance consumed, worst drive in the fleet. This is the drive's own accounting against what the manufacturer warrants, not a failure prediction — a drive past 100 usually keeps working and starts eating its spare area. Read it for planning; read the SMART table for whether anything is failing now.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "mappings": [],
+          "max": 100,
+          "min": 0,
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 80}, {"color": "red", "value": 95}]},
+          "unit": "percent"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 4, "w": 4, "x": 20, "y": 0},
+      "id": 14,
+      "options": {"colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": {"calcs": ["lastNotNull"], "fields": "", "values": false}, "textMode": "auto"},
+      "targets": [{"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "max(smartctl_device_percentage_used)", "legendFormat": "", "refId": "A"}],
+      "title": "Worst NVMe wear",
+      "type": "stat"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "Rated endurance consumed per drive. The line is nearly flat by nature — wear moves on a scale of weeks — so the useful reading is the spread between drives and the slope, not today's value. Joined to smartctl_device for the model name, because the device label alone is not comparable across nodes.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "dashed"}},
+          "mappings": [],
+          "max": 100,
+          "min": 0,
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 80}, {"color": "red", "value": 95}]},
+          "unit": "percent"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 0, "y": 13},
+      "id": 15,
+      "options": {"legend": {"calcs": ["last"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "smartctl_device_percentage_used * on (node, device) group_left(model_name) smartctl_device",
+          "legendFormat": "{{node}} {{model_name}}",
+          "refId": "A"
+        }
+      ],
+      "title": "NVMe wear",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "description": "The failure-now signals, as opposed to the endurance budget above. Spare below its threshold means the drive has run out of blocks to remap defects into; media errors are unrecoverable data-integrity failures and should never increment; the critical warning is a bitfield the drive raises itself. Written bytes is the counter behind the wear percentage, useful for projecting when a drive will reach it.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "thresholds"},
+          "custom": {"align": "auto", "cellOptions": {"type": "auto"}, "inspect": false},
+          "mappings": [],
+          "thresholds": {"mode": "absolute", "steps": [{"color": "text", "value": null}]}
+        },
+        "overrides": [
+          {"matcher": {"id": "byName", "options": "wear"}, "properties": [{"id": "unit", "value": "percent"}, {"id": "custom.cellOptions", "value": {"type": "color-text"}}, {"id": "thresholds", "value": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 80}, {"color": "red", "value": 95}]}}]},
+          {"matcher": {"id": "byName", "options": "spare"}, "properties": [{"id": "unit", "value": "percent"}]},
+          {"matcher": {"id": "byName", "options": "media errors"}, "properties": [{"id": "custom.cellOptions", "value": {"type": "color-text"}}, {"id": "thresholds", "value": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "red", "value": 1}]}}]},
+          {"matcher": {"id": "byName", "options": "critical warning"}, "properties": [{"id": "custom.cellOptions", "value": {"type": "color-text"}}, {"id": "thresholds", "value": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "red", "value": 1}]}}]},
+          {"matcher": {"id": "byName", "options": "powered on"}, "properties": [{"id": "unit", "value": "s"}]},
+          {"matcher": {"id": "byName", "options": "written"}, "properties": [{"id": "unit", "value": "bytes"}]}
+        ]
+      },
+      "gridPos": {"h": 9, "w": 12, "x": 12, "y": 13},
+      "id": 16,
+      "options": {"cellHeight": "sm", "footer": {"countRows": false, "fields": "", "reducer": ["sum"], "show": false}, "showHeader": true, "sortBy": [{"desc": true, "displayName": "wear"}]},
+      "targets": [
+        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "smartctl_device_percentage_used * on (node, device) group_left(model_name) smartctl_device", "format": "table", "instant": true, "refId": "A"},
+        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "smartctl_device_available_spare * on (node, device) group_left(model_name) smartctl_device", "format": "table", "instant": true, "refId": "B"},
+        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "smartctl_device_media_errors * on (node, device) group_left(model_name) smartctl_device", "format": "table", "instant": true, "refId": "C"},
+        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "smartctl_device_critical_warning * on (node, device) group_left(model_name) smartctl_device", "format": "table", "instant": true, "refId": "D"},
+        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "smartctl_device_power_on_seconds * on (node, device) group_left(model_name) smartctl_device", "format": "table", "instant": true, "refId": "E"},
+        {"datasource": {"type": "prometheus", "uid": "prometheus"}, "expr": "smartctl_device_bytes_written * on (node, device) group_left(model_name) smartctl_device", "format": "table", "instant": true, "refId": "F"}
+      ],
+      "title": "SMART health",
+      "transformations": [
+        {"id": "merge", "options": {}},
+        {
+          "id": "organize",
+          "options": {
+            "excludeByName": {"Time": true, "__name__": true, "container": true, "endpoint": true, "instance": true, "job": true, "namespace": true, "pod": true, "service": true},
+            "indexByName": {},
+            "renameByName": {"node": "node", "device": "device", "model_name": "model", "Value #A": "wear", "Value #B": "spare", "Value #C": "media errors", "Value #D": "critical warning", "Value #E": "powered on", "Value #F": "written"}
          }
        }
      ],
@@ -58,7 +234,7 @@
        },
        "overrides": []
      },
-      "gridPos": {"h": 9, "w": 12, "x": 0, "y": 9},
+      "gridPos": {"h": 9, "w": 12, "x": 0, "y": 22},
      "id": 2,
      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
      "targets": [
@@ -91,7 +267,7 @@
        },
        "overrides": []
      },
-      "gridPos": {"h": 9, "w": 12, "x": 12, "y": 9},
+      "gridPos": {"h": 9, "w": 12, "x": 12, "y": 22},
      "id": 3,
      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
      "targets": [
@@ -124,7 +300,7 @@
        },
        "overrides": []
      },
-      "gridPos": {"h": 9, "w": 12, "x": 0, "y": 18},
+      "gridPos": {"h": 9, "w": 12, "x": 0, "y": 31},
      "id": 4,
      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
      "targets": [
@@ -162,7 +338,7 @@
          }
        ]
      },
-      "gridPos": {"h": 9, "w": 12, "x": 12, "y": 18},
+      "gridPos": {"h": 9, "w": 12, "x": 12, "y": 31},
      "id": 5,
      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
      "targets": [
@@ -184,7 +360,7 @@
    },
    {
      "datasource": {"type": "prometheus", "uid": "prometheus"},
-      "description": "Composite temperature reported by each controller, read from hwmon in sysfs. This is the one health value available without an admin ioctl, which is why it is the only SMART-adjacent series on this dashboard.",
+      "description": "Composite temperature reported by each controller, read from hwmon in sysfs. This is the one health value available without an admin ioctl, which is why it is the only SMART-adjacent series on this dashboard. The thresholds are generic NVMe limits rather than any particular drive's rating — check the datasheet before treating a crossing as urgent.",
      "fieldConfig": {
        "defaults": {
          "color": {"mode": "palette-classic"},
@@ -195,7 +371,7 @@
        },
        "overrides": []
      },
-      "gridPos": {"h": 9, "w": 12, "x": 0, "y": 27},
+      "gridPos": {"h": 9, "w": 12, "x": 0, "y": 40},
      "id": 6,
      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
      "targets": [
@@ -222,7 +398,7 @@
        },
        "overrides": []
      },
-      "gridPos": {"h": 9, "w": 12, "x": 12, "y": 27},
+      "gridPos": {"h": 9, "w": 12, "x": 12, "y": 40},
      "id": 7,
      "options": {"legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
      "targets": [
@@ -244,11 +420,40 @@
    },
    {
      "datasource": {"type": "prometheus", "uid": "prometheus"},
-      "gridPos": {"h": 8, "w": 24, "x": 0, "y": 36},
+      "description": "EPHEMERAL fill level per node. Growth here is usually container images and pod scratch space rather than anything Ceph does.",
+      "fieldConfig": {
+        "defaults": {
+          "color": {"mode": "palette-classic"},
+          "custom": {"axisCenteredZero": false, "axisColorMode": "text", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 10, "gradientMode": "none", "hideFrom": {"legend": false, "tooltip": false, "viz": false}, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": {"type": "linear"}, "showPoints": "never", "spanNulls": false, "stacking": {"group": "A", "mode": "none"}, "thresholdsStyle": {"mode": "dashed"}},
+          "mappings": [],
+          "max": 1,
+          "min": 0,
+          "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}, {"color": "orange", "value": 0.7}, {"color": "red", "value": 0.85}]},
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "gridPos": {"h": 8, "w": 24, "x": 0, "y": 49},
+      "id": 9,
+      "options": {"legend": {"calcs": ["last", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true}, "tooltip": {"mode": "multi", "sort": "desc"}},
+      "targets": [
+        {
+          "datasource": {"type": "prometheus", "uid": "prometheus"},
+          "expr": "(1 - node_filesystem_avail_bytes{mountpoint=\"/var\"} / node_filesystem_size_bytes{mountpoint=\"/var\"}) * on (instance) group_left(nodename) node_uname_info",
+          "legendFormat": "{{nodename}} /var",
+          "refId": "A"
+        }
+      ],
+      "title": "EPHEMERAL usage",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {"type": "prometheus", "uid": "prometheus"},
+      "gridPos": {"h": 11, "w": 24, "x": 0, "y": 57},
      "id": 8,
      "options": {
        "code": {"language": "plaintext", "showLineNumbers": false, "showMiniMap": false},
-        "content": "### What this dashboard cannot tell you\n\n**Endurance is missing.** There is no wear, no written-bytes total, no power-on hours and no media-error count here, because no exporter in this cluster produces them. NVMe reports those over an admin ioctl that needs elevated privileges, and temperature is the only health value the kernel publishes through sysfs. Tracked as **#549** — do not read a healthy dashboard as a healthy drive.\n\n**The three roles are not interchangeable.** Each node carries a Kingston OM8PGP41024Q as the Talos system disk, a Samsung PM983 that Rook claims as the Ceph OSD, and a Crucial P3 that was retired from Ceph and now carries the EPHEMERAL volume. The Crucial drives were replaced *because* they were worn, so the drive with the least endurance left is the one holding container state — and it is the one nothing is watching.\n\n**Device names are not identity.** `nvme0` is the Kingston on k8s-1 and k8s-3 and the Crucial on k8s-2. Every panel here resolves the model through `node_nvme_info` for that reason; a query written against a bare device name compares different hardware and will keep doing so silently.",
+        "content": "### What this dashboard cannot tell you\n\nThese notes are about mechanisms, not about the drives currently fitted. The inventory table above is the live answer to what is in the machines; nothing here restates it, because prose does not get re-read when hardware is swapped.\n\n**Wear is a budget, not a prediction.** The percentage is what the drive reports against its own rated endurance. Drives do not stop at 100 — they usually keep working and start consuming spare blocks — so read it for planning, and read available spare and media errors for whether something is failing now. Those are different questions and the alert rules treat them as such.\n\n**A green top row is not a healthy drive.** Endurance says nothing about the drive that dies of a controller fault tomorrow, and this dashboard cannot see one coming.\n\nCeph keeps its own wear figures for OSDs in the devicehealth module, which is a useful cross-check but covers OSDs only: `kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph device ls`. A drive removed from Ceph is purged from that registry and its history goes with it.\n\n**Roles are assigned in configuration, not by device name.** Talos selects the system disk and the EPHEMERAL volume by serial, Rook selects the OSD by `/dev/disk/by-id` path. Two consequences outlive any particular set of drives: a broad selector such as `disk.transport == 'nvme'` matches every drive in the box, with the installer taking the OSD as the failure mode — and a drive retired from Ceph is not necessarily out of service, and putting a worn one on EPHEMERAL is a sensible destination rather than a hazard — that volume is built to be rebuilt. Check the inventory table for what a given drive holds before reading its wear as urgent.\n\n**A dying NVMe here hangs rather than errors.** Talos boots with `nvme_core.io_timeout=4294967295`, so NVMe commands never time out. Expect a stall, not an I/O error, and expect the latency panel to go flat rather than spike.\n\n**Kernel messages are not retained.** Talos kernel logs do not reach Loki — the only log jobs are Kubernetes pods and a systemd-journal from elsewhere. `talosctl -n <node> dmesg` is the only source, it is a ring buffer, and a reboot empties it. An NVMe error from before the last reboot left no trace anywhere.",
        "mode": "markdown"
      },
      "title": "Gaps and caveats",
@@ -15,6 +15,7 @@ resources:
  - prometheus-gate-service.yaml
  - prometheus-gate-ingress.yaml
  - ci-diagnostics-rbac.yaml
+  - smartctl-exporter.yaml
  - dashboards/
 # prometheusrule-garage.yaml lives in ./monitoring-rules/ and is applied by a
 # separate Flux Kustomization that dependsOn `monitoring` — otherwise the
@@ -0,0 +1,89 @@
+# SMART/endurance metrics for every NVMe drive in the fleet (#549).
+#
+# Runs privileged: host device access is gated by the container device cgroup,
+# and Kubernetes offers no per-device allowlist. Narrower settings do not work;
+# the alternatives that were tried are in the commit history.
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: smartctl-exporter
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/name: smartctl-exporter
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: smartctl-exporter
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: smartctl-exporter
+    spec:
+      # Control planes are also the storage nodes here, so the exporter has to
+      # tolerate the control-plane taint or it would cover nothing at all.
+      tolerations:
+        - key: node-role.kubernetes.io/control-plane
+          operator: Exists
+          effect: NoSchedule
+      containers:
+        - name: smartctl-exporter
+          image: quay.io/prometheuscommunity/smartctl-exporter:v0.14.0
+          # No --smartctl.device: the exporter scans and rescans on its own.
+          # Device names are not stable across these machines.
+          args:
+            - --web.listen-address=:9633
+          ports:
+            - name: http-metrics
+              containerPort: 9633
+              protocol: TCP
+          securityContext:
+            runAsUser: 0
+            runAsNonRoot: false
+            privileged: true
+            # The API rejects privileged together with
+            # allowPrivilegeEscalation: false.
+            allowPrivilegeEscalation: true
+            readOnlyRootFilesystem: true
+          volumeMounts:
+            # Not read-only: smartmontools opens NVMe devices read-write.
+            - name: dev
+              mountPath: /dev
+            # Scratch space, because the root filesystem is read-only.
+            - name: tmp
+              mountPath: /tmp
+          resources:
+            requests:
+              cpu: 10m
+              memory: 32Mi
+            limits:
+              memory: 128Mi
+          livenessProbe:
+            httpGet:
+              path: /metrics
+              port: http-metrics
+            initialDelaySeconds: 15
+            periodSeconds: 30
+      volumes:
+        - name: dev
+          hostPath:
+            path: /dev
+            type: Directory
+        - name: tmp
+          emptyDir: {}
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: smartctl-exporter
+  namespace: monitoring
+  labels:
+    app.kubernetes.io/name: smartctl-exporter
+spec:
+  clusterIP: None
+  selector:
+    app.kubernetes.io/name: smartctl-exporter
+  ports:
+    - name: http-metrics
+      port: 9633
+      targetPort: http-metrics
+      protocol: TCP
@@ -1,18 +0,0 @@
-apiVersion: external-secrets.io/v1
-kind: ExternalSecret
-metadata:
-  name: barman-cloud-creds
-  namespace: paperless
-  annotations:
-    # Prune-disabled — a standing guard kept from the seed->paperless-KS handoff (barman creds).
-    kustomize.toolkit.fluxcd.io/prune: disabled
-spec:
-  refreshInterval: 1h
-  secretStoreRef:
-    kind: ClusterSecretStore
-    name: openbao
-  target:
-    name: barman-cloud-creds # keeps the old Secret's name — consumers unchanged
-  dataFrom:
-    - extract:
-        key: seed/paperless/barman-creds
@@ -1,3 +1,9 @@
+# One Hetzner S3 key, one path, shared by every barman ObjectStore. Separate
+# per-app paths would not make rotation independent — that needs one Hetzner
+# key per consumer (#397).
+#
+# velero reads the same key from infrastructure/velero-hetzner, which holds it
+# as a credentials file rather than these three fields.
 apiVersion: external-secrets.io/v1
 kind: ExternalSecret
 metadata:
@@ -15,4 +21,4 @@ spec:
    name: hetzner-s3-creds # keeps the old Secret's name — consumers unchanged
  dataFrom:
    - extract:
-        key: seed/paperless/hetzner-creds
+        key: infrastructure/hetzner-s3
@@ -20,7 +20,6 @@ resources:
  - cluster.yaml
  - objectstore.yaml
  - scheduled-backup.yaml
-  - barman-creds-externalsecret.yaml
  - hetzner-creds-externalsecret.yaml
  - pvcs.yaml
 configMapGenerator:
@@ -12,4 +12,4 @@ spec:
    name: hetzner-s3-creds
  dataFrom:
    - extract:
-        key: seed/trmnl/hetzner-creds
+        key: infrastructure/hetzner-s3
@@ -1,11 +1,8 @@
 ---
 # Barman S3 credentials for wp-pg.
 #
-# Sourced from seed/paperless/hetzner-creds, which is not where they belong.
-# Issue #397 wants woodpecker to own seed/woodpecker/hetzner-creds; creating
-# that path needs OpenBao write access (`bao kv list seed` → 403 for the
-# identities available here), so it stays open. The values are identical, so it
-# is a one-line swap of the `key:` below once the path exists.
+# One shared path for one shared credential — see the note in
+# ../paperless/hetzner-creds-externalsecret.yaml.
 #
 # The namespace allowlist on the openbao ClusterSecretStore
 # (infrastructure/post-crds/openbao-clustersecretstore.yaml) must include this
@@ -24,4 +21,4 @@ spec:
    name: hetzner-s3-creds
  dataFrom:
    - extract:
-        key: seed/paperless/hetzner-creds
+        key: infrastructure/hetzner-s3
@@ -36,11 +36,9 @@ spec:
  target:
    name: hetzner-s3-creds
  dataFrom:
-    # Reuses the paperless copy of the shared Hetzner S3 credentials (identical
-    # ACCESS_KEY_ID / ACCESS_SECRET_KEY / REGION) to avoid seeding a new OpenBao
-    # path for this migration — same approach woodpecker took. FOLLOW-UP: give
-    # wordpress its own seed/wordpress/hetzner-creds path once OpenBao can be
-    # written, and repoint this here (the value is the same, so it's a drop-in
-    # swap).
+    # One shared path for one shared credential. There is a single Hetzner S3
+    # key behind all of these — see the note in
+    # ../paperless/hetzner-creds-externalsecret.yaml for why per-app paths were
+    # the wrong shape and what would have to change for them to be right.
    - extract:
-        key: seed/paperless/hetzner-creds
+        key: infrastructure/hetzner-s3
@@ -33,22 +33,14 @@ spec:
    remediation:
      retries: 3
  values:
-    # Roll the agent DaemonSet when cilium-config changes. Cilium reads
-    # datapath options once, at agent start, and the chart puts no config
-    # checksum on the pod template — so a changed ConfigMap leaves every agent
-    # running the previous datapath until something restarts it. Nothing did.
+    # Rolls the agent DaemonSet when cilium-config changes. Cilium reads
+    # datapath options once at agent start and the chart puts no config
+    # checksum on the pod template, so without this a changed ConfigMap leaves
+    # every agent on the previous datapath indefinitely (#592).
    #
-    # This is not hypothetical. vlan-bpf-bypass reached cilium-config and every
-    # v30iot macvlan workload stayed broken until the agents were deleted by
-    # hand, one node at a time (issue #592). policyMapMax has the same
-    # exposure, and its comment below says "takes effect on agent restart" —
-    # documentation standing in for automation.
-    #
-    # The trade: any cilium-config change now rolls the CNI cluster-wide,
-    # node by node. That is disruptive-ish, but it is what a datapath change
-    # already required — the difference is that it now happens on its own
-    # instead of being forgotten. Reloader is already deployed for exactly
-    # this pattern; named per-ConfigMap rather than blanket, as elsewhere.
+    # The trade is that any cilium-config change now rolls the CNI node by
+    # node — which is what a datapath change required anyway, only now it is
+    # not forgotten.
    annotations:
      configmap.reloader.stakater.com/reload: cilium-config
    # Which pod labels feed security-identity computation. Patterns are
@@ -69,57 +61,42 @@ spec:
    socketLB:
      enabled: true
    prometheus:
-      # The agent exported no metrics at all, so BPF map utilisation was
-      # invisible: when per-endpoint policy maps (cilium_policy_v3_*) run out
-      # of space the agent only says "no space left on device" in its log and
-      # pod sandbox creation fails (issue #452). cilium_bpf_map_pressure makes
-      # that observable — and alertable — before it stalls a rollout.
+      # Without agent metrics, BPF map utilisation is invisible: a full policy
+      # map only shows up as "no space left on device" in the log while pod
+      # sandbox creation fails (#452). cilium_bpf_map_pressure makes it
+      # alertable beforehand.
      enabled: true
      serviceMonitor:
        enabled: true
    bpf:
-      # Size of each endpoint's own cilium_policy_v3_<id> map, holding one
-      # entry per (identity, direction, protocol, port) the endpoint is
-      # allowed. When it fills, endpoint regeneration fails with "no space
-      # left on device", the CNI ADD call never returns, and the pod is stuck
-      # in sandbox creation — so this must stay comfortably above the busiest
-      # endpoint's entry count, tracked by cilium_bpf_map_pressure.
-      # Takes effect on agent restart; the map is sized at creation.
-      # Bootstrap installs Cilium from the inline manifest in
-      # kubernetes/talos/patches/cilium-cni.yaml, so bpf-policy-map-max there
-      # must move with this value or a cold rebuild starts with the old size.
-      # kubernetes/talos/cilium.yaml carries a third copy of the same key.
-      # Nothing enforces any of this; grep bpf-policy-map-max before changing it.
+      # Per-endpoint policy map size: one entry per (identity, direction,
+      # protocol, port) allowed. When it fills, endpoint regeneration fails
+      # with "no space left on device" and the pod hangs in sandbox creation.
+      # Keep it comfortably above the busiest endpoint, watched by
+      # cilium_bpf_map_pressure. Sized at map creation, so it takes an agent
+      # restart.
+      #
+      # THREE COPIES of this key exist — here, talos/patches/cilium-cni.yaml
+      # and talos/cilium.yaml. The latter two are what a cold rebuild installs
+      # from. Nothing enforces agreement; grep bpf-policy-map-max before
+      # changing it.
      policyMapMax: 65536
      # VLAN ids the eBPF datapath is allowed to pass. Without this it drops
      # VLAN-tagged frames, accounted as "VLAN traffic disallowed by VLAN
      # filter" in cilium_drop_count_total.
      #
-      # 30 is v30iot, the macvlan parent for the shelly-manager, matter-server,
-      # otbr and music-assistant attachments. On bare metal it is a VLAN
-      # sub-interface of enp2s0f0np0; inside the former Proxmox guests it was
-      # an untagged virtio NIC, which is why this was not needed before.
+      # Only VLAN 30 (v30iot) needs listing. Cilium attaches to devices that
+      # carry an address, so it sees mgmt0's frames already untagged by the
+      # 8021q layer — nothing for a VLAN filter to match. ens19 is deliberately
+      # address-less, so its frames only meet the filter on the shared parent,
+      # still tagged. Adding 40 would change nothing and would touch the
+      # datapath carrying etcd and the API.
      #
-      # VLAN 40 (mgmt0) is deliberately NOT listed, and the asymmetry is the
-      # interesting part. Cilium attaches to devices that carry an address —
-      # `cilium-dbg status --verbose` lists enp87s0, enp2s0f0np0, enp2s0f1np1
-      # and mgmt0, but NOT ens19. mgmt0 holds the node IP, so Cilium owns it as
-      # a device and sees frames the 8021q layer has already untagged; nothing
-      # for a VLAN filter to match. ens19 is deliberately address-less (see
-      # talos/patches/k8s-1.yaml — host addresses on that L2 make node replies
-      # bypass the UDM), so Cilium never attaches to it and VLAN 30 frames only
-      # meet the filter on the shared parent, still tagged.
+      # The symptom when this is wrong is not a DHCP timeout: a tc-layer drop
+      # reaches an AF_PACKET sender as ENOBUFS, so the CNI dhcp plugin reports
+      # "no buffer space available" (#592).
      #
-      # So the fix belongs on 30 alone. Listing 40 would change nothing, and
-      # would touch the datapath carrying etcd and the API for no reason.
-      #
-      # A tc-layer drop surfaces to an AF_PACKET sender as ENOBUFS, so the
-      # symptom is the CNI dhcp plugin reporting "no buffer space available"
-      # rather than a DHCP timeout. See issue #592.
-      #
-      # Same three-copy problem as policyMapMax above: vlan-bpf-bypass in
-      # kubernetes/talos/cilium.yaml and kubernetes/talos/patches/cilium-cni.yaml
-      # must move with this list. Takes effect on agent restart.
+      # Same three-copy problem as policyMapMax above. Takes an agent restart.
      vlanBypass: [30]
      # Let pods reach the cluster's own LoadBalancer external IPs; Cilium
      # redirects the socket directly to the backend pod instead of sending
@@ -36,25 +36,16 @@ spec:
  values:
    container_runtime: containerd

-    # --- mTLS between agents and the LAPI: the durable fix for the recurring
-    # KubeDaemonSetRolloutStuck auth loop (#340) ---
+    # mTLS between agents and the LAPI. Do not switch back to password auth:
+    # the LAPI runs without persistence, so its machine-registration DB is
+    # wiped on every restart, and password-registered agents then get 401
+    # "machine not found" and crashloop until re-registered by hand — which
+    # wedges the whole `infrastructure` Kustomization behind a HelmRelease
+    # that never goes Ready (#340).
    #
-    # The LAPI runs WITHOUT persistence (see persistentVolume note below), so
-    # its sqlite machine-registration DB is ephemeral and every LAPI pod restart
-    # wiped it. Under the old password flow the agents registered exactly ONCE
-    # (init container running `cscli lapi register --token`) and cached a
-    # password the wiped LAPI no longer recognised -> 401 "machine not found"
-    # -> CrashLoopBackOff on all three agents -> DaemonSet never Ready -> the
-    # crowdsec HelmRelease upgrade `--wait` times out forever -> the whole
-    # `infrastructure` Kustomization (and every app depending on it) stays
-    # unhealthy. That is the #340 outage.
-    #
-    # With client-cert auth the agent init container no longer registers at all
-    # (verified via `helm template`: the `cscli lapi register` step is gone).
-    # The agent presents a cert whose OU (agent-ou) the LAPI trusts, and a
-    # freshly restarted LAPI with an EMPTY DB auto-recreates each machine record
-    # from the presented cert on connect. So the system self-heals across LAPI
-    # restarts with no PVC, no Ceph, and no manual re-registration.
+    # With client certs the agent never registers: a restarted LAPI with an
+    # empty DB recreates each machine record from the presented cert, so this
+    # self-heals with no PVC and no intervention.
    #
    # cert-manager (already reconciled earlier in this same `infrastructure`
    # stage) provisions a fully self-contained self-signed CA (selfSigned
@@ -87,13 +78,10 @@ spec:

    lapi:
      replicas: 1
-      # The chart defaults request 500m. Measured steady-state draw is 2m, and
-      # the request is what the scheduler reserves, so the default parks half a
-      # core of unusable capacity per LAPI. That is affordable on three nodes
-      # and is not on two: with one node out for maintenance the remaining pair
-      # sit at 100% of requested CPU while actually using ~22%, and nothing new
-      # can be placed. The limit stays at the chart default so a log flood can
-      # still burst; only the reservation shrinks.
+      # Far below the chart default, which reserves half a core per LAPI for a
+      # workload that draws single-digit millicores — enough to leave the
+      # cluster unschedulable with one node out. Only the reservation shrinks;
+      # the limit stays at the chart default so a log flood can still burst.
      resources:
        requests:
          cpu: 50m
@@ -6,19 +6,12 @@
 # fails the route over across all per-node peers (no single point of
 # failure if a node/pod drops).
 #
-# Uses the ROOTLESS image (userspace wireguard-go). The netbird process
-# terminates the WireGuard tunnel in userspace and reaches cluster
-# services by opening its OWN sockets to them — NOT by L3-forwarding
-# packets. That distinction is the whole point:
-#   - Cilium socket-LB DNATs the netbird process's own connections, so
-#     forwarding to ClusterIP / LB VIPs actually works. A kernel L3
-#     routing peer fails here: socket-LB never sees forwarded packets,
-#     so they hit the raw VIP and are blackholed (confirmed via conntrack).
-#   - No NET_ADMIN / no kernel WireGuard → runs fully unprivileged
-#     (drop ALL caps, runAsNonRoot), so it satisfies Pod Security too.
-#
-# Flow: mesh peer (100.96.x.x) --wg--> a per-node pod (userspace netbird)
-#        --own socket (socket-LB DNAT)--> service ClusterIP/LB VIP
+# Must stay on the ROOTLESS image (userspace wireguard-go). The process
+# terminates the tunnel in userspace and reaches services by opening its own
+# sockets, which Cilium's socket-LB can DNAT. A kernel WireGuard peer
+# L3-forwards instead, socket-LB never sees those packets, and they are
+# blackholed at the raw VIP. Rootless also means no NET_ADMIN, so the pod
+# runs fully unprivileged.
 #
 # Egress is locked down by the netbird-ingress-egress CiliumNetworkPolicy
 # (network-policies/ingress-isolation.yaml).
@@ -41,18 +41,14 @@ spec:
        - ports:
            - port: "8080"
              protocol: TCP
-    # shelly-manager (:8080) is likewise reached only through its own
-    # Authentik gate outpost. Note this rule and the vscode one are
-    # port-identical: because this CNP's endpointSelector is {}, a
-    # `toPorts: 8080` grant here applies to EVERY pod in the namespace that
-    # listens on 8080, not just the intended one — which is exactly how
-    # shelly-manager silently rode on the old traefik→:8080 allow written
-    # for code-server, and why removing that allow in 760e187 took
-    # shelly-manager's ingress down with it. The `fromEndpoints` selector is
-    # what keeps these two apart: each gate outpost can only reach its own
-    # target, enforced on the egress side by the matching CNPs in the
-    # authentik namespace (vscode-gate-egress.yaml,
-    # shelly-manager-gate-egress.yaml).
+    # shelly-manager, behind its own Authentik gate outpost.
+    #
+    # CAUTION for every :8080 rule here: this policy's endpointSelector is {},
+    # so a toPorts grant on 8080 applies to EVERY pod in the namespace
+    # listening on that port, not just the intended one. Three apps share
+    # :8080. What keeps them apart is the fromEndpoints selector plus the
+    # matching per-gate egress CNPs in the authentik namespace. Changing a
+    # :8080 rule here affects all of them.
    - fromEndpoints:
        - matchLabels:
            "k8s:io.kubernetes.pod.namespace": authentik
@@ -61,11 +57,8 @@ spec:
        - ports:
            - port: "8080"
              protocol: TCP
-    # esphome (ESPHome Device Builder, :6052) is likewise reached only through
-    # its own Authentik gate outpost. Unlike the :8080 vscode/shelly rules
-    # above, :6052 is unique to esphome, so this namespace-wide (endpointSelector
-    # {}) toPorts grant doesn't silently apply to any other app — the
-    # fromEndpoints selector still scopes it to the esphome-gate outpost.
+    # esphome (:6052), behind its own gate outpost. Port is unique to it, so
+    # the namespace-wide grant does not reach anything else.
    - fromEndpoints:
        - matchLabels:
            "k8s:io.kubernetes.pod.namespace": authentik
@@ -74,13 +67,8 @@ spec:
        - ports:
            - port: "6052"
              protocol: TCP
-    # zigbee2mqtt (Z2M frontend, :8080) is likewise reached only through its own
-    # Authentik gate outpost. Like the vscode/shelly-manager rules above this is
-    # the SHARED :8080 (zigbee2mqtt is the third app on that port), so this
-    # namespace-wide (endpointSelector {}) grant applies to every :8080 pod — the
-    # fromEndpoints selector scopes it to the zigbee2mqtt-gate outpost, and the
-    # matching authentik-side zigbee2mqtt-gate-egress.yaml scopes egress to the
-    # zigbee2mqtt pod specifically.
+    # zigbee2mqtt frontend, behind its own gate outpost. Shared :8080 — see the
+    # caution above.
    - fromEndpoints:
        - matchLabels:
            "k8s:io.kubernetes.pod.namespace": authentik
@@ -89,11 +77,8 @@ spec:
        - ports:
            - port: "8080"
              protocol: TCP
-    # zone-configurator (Everything Presence mmWave zone configurator, :42069)
-    # is likewise reached only through its own Authentik gate outpost. Like
-    # esphome's :6052, :42069 is unique to it, so this namespace-wide (endpoint
-    # Selector {}) toPorts grant doesn't silently apply to any other app; the
-    # fromEndpoints selector scopes it to the zone-configurator-gate outpost.
+    # zone-configurator (:42069), behind its own gate outpost. Port is unique
+    # to it.
    - fromEndpoints:
        - matchLabels:
            "k8s:io.kubernetes.pod.namespace": authentik
@@ -131,25 +116,16 @@ spec:
    - fromEntities:
        - host
        - remote-node
-    # World ingress (IoT-VLAN devices via the BGP LoadBalancers) is NOT
-    # granted namespace-wide here. A blanket `fromEntities: world` on this
-    # endpointSelector:{} policy exposed EVERY pod in the namespace to the
-    # untrusted IoT VLAN / any BGP-routed source — vscode (:8080),
-    # zone-configurator (:42069), otbr, whisper, etc. — not just the two
-    # LoadBalancers that need it. World is now scoped per-target in the
-    # dedicated policies at the bottom of this file: world -> mosquitto
-    # :1883/:8883 and world -> home-assistant :8123 only. (matter-server on
-    # 10.2.x.226 rides a multus net1 interface that bypasses Cilium policy
-    # entirely, so it is unaffected either way.)
-    # ICMP — all standard ICMPv4 and ICMPv6 types from any cluster pod.
-    # Per Cilium docs, "If any layer 4 policy is specified, then ICMP
-    # will be blocked unless it's related to a connection that is
-    # otherwise allowed by the policy." The CT-RELATED path is not
-    # passing these in practice (Hubble shows ICMPv6 DestinationUnreachable
-    # being POLICY_DENIED on the egress side of established DNS flows),
-    # so allow ICMP explicitly. Same shape as commit 3c517a0 — widened
-    # from the error class to the full standard type set. Cilium's
-    # ICMPField requires an explicit type per entry; no wildcard.
+    # Do NOT add a blanket `fromEntities: world` to this policy. With
+    # endpointSelector {} it exposes every pod in the namespace to the IoT
+    # VLAN and any BGP-routed source, not just the LoadBalancers that need it.
+    # World is scoped per target in the dedicated policies at the bottom of
+    # this file. (matter-server rides a multus interface that bypasses Cilium
+    # policy entirely and is unaffected either way.)
+    # ICMP must be allowed explicitly: once any L4 policy exists Cilium blocks
+    # it unless CT-RELATED passes it, which in practice it does not. Cilium's
+    # ICMPField takes one type per entry — there is no wildcard, so the list
+    # has to be spelled out.
    - fromEndpoints:
        - matchExpressions:
            - key: io.kubernetes.pod.namespace
@@ -186,10 +162,8 @@ spec:
    - toEndpoints:
        - matchLabels:
            "k8s:io.kubernetes.pod.namespace": apps
-    # In-cluster InfluxDB: HA's HACS InfluxDB integration writes state
-    # changes here (replaces the legacy http://influxdb.srv.renner.uno
-    # target — see kubernetes/apps/influxdb/ha-token-job.yaml for the
-    # write-bucket token wiring).
+    # In-cluster InfluxDB: HA's HACS integration writes state changes here.
+    # Token wiring is in apps/influxdb/ha-token-job.yaml.
    - toEndpoints:
        - matchLabels:
            "k8s:io.kubernetes.pod.namespace": influxdb
@@ -213,9 +187,8 @@ spec:
    - toEntities:
        - host
        - remote-node
-    # World: IoT-VLAN devices, ESPHome, HA Cloud, garage S3 (initial
-    # tarball seed), legacy InfluxDB on VM 108, third-party integration
-    # endpoints (Hue, Roomba, Philips, Brother IPP, …).
+    # World: IoT-VLAN devices, HA Cloud, object storage, and third-party
+    # integration endpoints on the LAN and internet.
    - toEntities:
        - world
    # ICMP — egress mirror of the ingress ICMP allow above. Lets HA
@@ -1,12 +1,5 @@
 ---
-# Baseline ingress policy for the `woodpecker` namespace (wp-pg now; server and
-# agent arrive in stage 2).
-#
-# Unlike the `apps` baseline this replaces, every allow here lands on a
-# namespace holding one service, so a namespace-scoped grant is not a blanket
-# grant across unrelated workloads. That is the point of the move: in `apps`,
-# letting the CNPG operator reach wp-pg would also have opened gitea, gatus,
-# external-dns and ical-proxy to the whole database namespace (issue #433).
+# Baseline ingress policy for the `woodpecker` namespace.
 #
 # CAUTION when adding rules here: this policy has `endpointSelector: {}`, so
 # every rule below applies to EVERY pod in the namespace. A `toPorts` block
@@ -15,11 +8,7 @@
 # belongs in its own policy with its own selector, as
 # `wp-pg-allow-cnpg-operator` below does.
 #
-# No egress section, so egress is unrestricted. Deliberate for now and revisited
-# in stage 2: wp-pg needs DNS plus `world` for WAL archiving to Hetzner S3, and
-# the server/agent add gitea, the Kubernetes API and outbound plugin pulls.
-# Writing that ruleset once, with the full workload set present, beats guessing
-# it here and silently breaking the WAL archiving this stage exists to verify.
+# There is no egress section, so egress is unrestricted. Still open (#433).
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
 metadata:
@@ -5,25 +5,15 @@ kind: Kustomization
 # secret. The matching OpenBao policy/role for each lives in ../policies/ and is
 # bootstrapped manually (not GitOps-managed).
 #
-# IMAGE PINNING: every rotator here uses a DATED TAG of secret-rotator
-# (`:YYYY-MM-DD`), never a digest and never `:latest`. This is a registry-local
-# image, so it does not need digest pinning, and a dated tag makes "what actually
-# runs" a reviewable diff.
+# IMAGE PINNING: dated tags (`:YYYY-MM-DD`), never a digest and never
+# `:latest`. Do NOT switch to digests — untagged manifests in this registry are
+# garbage-collected once the tags move past them, so a pinned digest silently
+# stops resolving after a few unrelated rebuilds of build-ci-images.yml. These
+# jobs are quarterly, so the resulting ImagePullBackOff looks like nothing
+# happening rather than like a failure (#458).
 #
-# Digests were tried and actively broke: untagged manifests are garbage-collected
-# once :latest and the dated tags move past them, so a digest pinned today stops
-# resolving after a few rebuilds. That decay is driven by edits to
-# .woodpecker/build-ci-images.yml — a shared file — so adding an UNRELATED image
-# republishes secret-rotator and orphans the manifest every rotator here was
-# pinned to. 11 of these were pinned to digests that no longer existed and would
-# have ImagePullBackOff'd on their first scheduled run (issue #458); because they
-# are quarterly, that would have looked like nothing happening rather than like a
-# failure.
-#
-# Keep them all on one tag: it is a single binary containing every mode
-# (random | ha-token | cloudflare-roll | netbird-key | proxmox-token |
-# gitea-token | woodpecker-secret), so a split serves no purpose and just makes
-# the next bump partial.
+# Keep every rotator on the same tag: it is one binary containing all modes, so
+# splitting only makes the next bump partial.
 #
 # ADDING A ROTATOR: copy an existing file, and check you did not inherit a
 # digest pin from an older example — nothing enforces this automatically, so it
@@ -24,12 +24,10 @@ spec:
    - start: "fd99::"
      stop: "fd99::ff"
 ---
-# Cilium BGPv2 control-plane config peering each k8s node with the UDM
-# over its MGMT LAN interface — NOT over v30iot. Control-plane traffic
-# belongs on the management network; the IoT VLAN is the noisy untrusted
-# L2 segment we're explicitly trying to keep cluster infrastructure off
-# of. Each node sources from its eth0 (10.2.40.x / fd40::<host>) via
-# the kernel routing decision; no explicit localAddress override needed.
+# Cilium BGPv2 control-plane config peering each node with the UDM over the
+# management LAN — never over v30iot, which is the untrusted IoT segment.
+# The source address comes from the kernel routing decision, so no explicit
+# localAddress override is needed.
 #
 # Two parallel sessions, one per address family:
 #   - v4 session: peer 10.2.40.1, family IPv4 unicast only
@@ -1,27 +1,16 @@
 ---
 # Velero Schedules for stateful app namespaces.
 #
-# Scope is restricted to PersistentVolumeClaims via includedResources.
-# Flux owns every other K8s object (Deployment, Service, Ingress,
-# ConfigMap, Secret, …) — backing those up shadows current git state
-# on a rebuild: Velero restores stale manifests from yesterday's
-# snapshot, then the app Kustomization clobbers them with whatever git
-# says — except when the Kustomization is blocked on the restore chain,
-# leaving the cluster temporarily serving the stale spec (e.g. the
-# paperless Ingress missing prod hostnames after the 2026-04-22
-# rebuild). PVC-only is the canonical "back up data, not state" scope
-# (see ~/Notes/wiki/k8s-backup-scope.md and the project memory rule).
+# Scope is PersistentVolumeClaims only. Flux owns every other object, and
+# backing those up shadows git on a rebuild: Velero restores yesterday's
+# manifests and the cluster serves them until the Kustomization catches up.
+# Back up data, not state.
 #
-# snapshotMoveData=true keeps doing the heavy lifting via Kopia +
-# node-agent: PVC contents are uploaded to Garage and restored
+# snapshotMoveData uploads PVC contents through Kopia and the node-agent,
 # independently of the K8s API surface.
 #
-# Naming convention: <app>-daily for the app's Schedule. Restore is a
-# manual, operator-driven procedure — see kubernetes/RUNBOOK-cold-rebuild.md.
-# (There is no automatic bootstrap-restore Job; it was removed as dead code,
-# issue #200.)
-#
-# Retention: 30 days, matching the live CNPG Barman ObjectStore policy.
+# Naming: <app>-daily. Restore is manual — see RUNBOOK-cold-rebuild.md.
+# Retention matches the CNPG Barman policy.

 apiVersion: velero.io/v1
 kind: Schedule
@@ -203,22 +192,13 @@ metadata:
  labels:
    app.kubernetes.io/part-of: homelab-backup
 spec:
-  # Daily at 06:45 local (next free stagger slot). The app PVCs live in
-  # the home-assistant namespace regardless of which app/ dir owns
-  # their manifest (matter-server, otbr, whisper, speech-to-phrase,
-  # zone-configurator all set `namespace: home-assistant` via their
-  # Kustomization, not per-resource): home-assistant-config (.storage
-  # entity registry, recorder DB, secrets.yaml), esphome-data,
-  # shelly-manager-data, matter-server-data, otbr-data, whisper-data,
-  # speech-to-phrase-models, speech-to-phrase-train, zigbee2mqtt-data,
-  # zwavejs-store, zone-configurator-config. On restore,
-  # home-assistant-config has its
-  # own path (the HA pod's `seed` initContainer pulling the native HAOS
-  # backup from Garage, see the home-assistant Kustomization comment in
-  # kubernetes/clusters/homelab/home-assistant.yaml); layering a Velero
-  # PVC restore on top of that flow for the same PVC needs its own review.
-  # Restore of these schedules is manual — see
-  # kubernetes/RUNBOOK-cold-rebuild.md.
+  # Daily at 06:45 local. Sweeps every PVC in the home-assistant namespace —
+  # several apps under apps/ set `namespace: home-assistant` in their
+  # Kustomization, so their PVCs land here too.
+  #
+  # home-assistant-config has its own restore path: the HA pod's `seed`
+  # initContainer pulls the native HAOS backup. Layering a Velero PVC restore
+  # on top of that for the same PVC needs its own review.
  schedule: "45 6 * * *"
  useOwnerReferencesInBackup: false
  template:
@@ -229,16 +209,9 @@ spec:
    storageLocation: default
    ttl: 720h0m0s
    snapshotMoveData: true
-    # The ha-pg CNPG cluster now runs in this namespace (it did not when
-    # this schedule was written — the old comment here still claimed
-    # "no CNPG cluster in this namespace"). Its PVC ha-pg-1 carries the
-    # cnpg.io/cluster label and was being swept in: a crash-consistent
-    # snapshot of a running Postgres data directory is not a restorable
-    # backup, the same trap the wordpress-daily comment below describes.
-    # ha-pg is covered by Barman — daily base backup at 02:50 UTC
-    # (kubernetes/apps/home-assistant/ha-pg-scheduled-backup.yaml) plus
-    # continuous WAL archiving — so this exclusion removes a redundant
-    # 20 GiB upload, matching paperless/authentik/immich/trmnl above.
+    # Excludes the CNPG cluster's PVC. A crash-consistent snapshot of a
+    # running Postgres data directory is not a restorable backup, and Barman
+    # covers that database anyway — see ha-pg-scheduled-backup.yaml.
    labelSelector:
      matchExpressions:
        - key: cnpg.io/cluster
@@ -8,16 +8,9 @@ spec:
    image: quay.io/ceph/ceph:v19.2.6 # {"$imagepolicy": "flux-system:ceph"}
    allowUnsupported: false
  dataDirHostPath: /var/lib/rook
-  # Do not start restarting OSDs while PGs are still recovering. The default
-  # is false, so Rook would proceed regardless — and paired with the old
-  # osdMaxUpdatesInParallel default that means it can take the next OSD down
-  # while the previous one's data is still being caught up.
-  #
-  # With three OSDs and size 3 there is no spare copy to fall back on: the
-  # gate between one OSD returning and the next going down is the only thing
-  # that keeps a rolling change from becoming an outage. See
-  # storage.osdMaxUpdatesInParallel below, which bounds the same operation
-  # from the other side.
+  # Defaults to false. With three OSDs and size 3 there is no spare copy, so
+  # this gate is the only thing keeping a rolling OSD change from taking the
+  # next one down mid-recovery. Paired with storage.osdMaxUpdatesInParallel.
  upgradeOSDRequiresHealthyPGs: true
  mon:
    count: 3
@@ -39,113 +32,40 @@ spec:
      daemon:
        keyRotationPolicy: KeyGeneration
        keyGeneration: 2
-  # Bump CephFS subvolume cloner pool from 4 → 16. Velero CSI Snapshot
-  # Data Movement provisions a backupPVC per source PVC by cloning the
-  # VolumeSnapshot, and a 4-thread default stalls migrate-immich's
-  # multi-PVC Backup with `rados: "all cloner threads are busy"`. 16
-  # gives headroom for parallel restores during cluster rebuild too
-  # (paperless + audiobookshelf + immich + ... all clone at once).
+  # CephFS cloner pool raised from the default 4. Velero clones a
+  # VolumeSnapshot per PVC, and 4 threads stall a multi-PVC backup with
+  # `rados: "all cloner threads are busy"`.
  cephConfig:
    mon:
-      # The Ceph default, stated rather than left out. Removing the line would
-      # not restore it: `ceph config` keeps a value that only disappears from
-      # this file, so the old 20 would sit in the mon config database
-      # indefinitely — the same trap the bluestore settings below describe,
-      # and the one that crash-looped an OSD during the network provider
-      # change when public_network survived being deleted here.
-      #
-      # It was 20 because the mons share the node's /var with containerd, and
-      # on the VMs that was ~60 GB: 30% free measured image and snapshot churn
-      # rather than anything about Ceph, so k8s-2 tripped MON_DISK_LOW whenever
-      # the image cache filled and GC could not reclaim it (#434).
-      #
-      # Bare metal removed the premise. /var is now ~931 GB per node at 7-8%
-      # used, so 30% free is a real reserve again and the ~106 MB mon database
-      # is irrelevant against it. Reaching 70% would be a genuine signal about
-      # the node rather than noise about the image cache.
-      #
-      # mon_data_avail_crit stays at the 5% default, and CephMonDiskLow in
-      # monitoring-rules still watches /var directly for the real risk, which
-      # is the node filling rather than the mon growing.
+      # The Ceph default, stated rather than omitted. Deleting a line here does
+      # NOT restore the default: `ceph config` keeps whatever was last applied,
+      # so the old value would sit in the mon config database indefinitely.
+      # Change values in place; to actually drop one, `ceph config rm` it.
      mon_data_avail_warn: "30"
-      # Stop compacting the whole RocksDB prefix on every paxos trim.
+      # Off, against the upstream default. The forced compact_range on every
+      # paxos trim rewrote the whole store hundreds of times a day and was the
+      # single largest source of disk writes on these nodes.
      #
-      # Upstream default is true — "Compact a certain prefix (including paxos)
-      # when we trim its old states" — which is fine at low paxos rates and
-      # pathological here. Measured 2026-08-19:
-      #
-      #   paxos transactions      ~99,000/day  (1.15/s, sampled over 60 s)
-      #   paxos logical data       811 MB/day  (21.9 GB over 27 d uptime)
-      #   ACTUAL mon disk writes    80 GB/day  per mon, 240 GB/day for three
-      #                                        (cgroup io.stat: 2.15 TB/27 d)
-      #   => ~98x write amplification, for a ~106 MB database
-      #
-      # That was ~62% of all OS-disk writes on these nodes and roughly a
-      # quarter of everything the cluster writes — more than every client
-      # workload combined. With paxos_trim_min at 250, ~99k transactions/day
-      # means hundreds of forced range compactions per day, each rewriting the
-      # store.
-      #
-      # The churn itself is 99% cluster log and mgr stats, not real cluster
-      # activity: logm holds 6.7M paxos versions and mgrstat 4.4M, against
-      # osdmap's 9.6k. Every Rook-operator poll (mgr stat / versions / status /
-      # report / osd dump — 23 audit entries per minute per mon) becomes a
-      # cluster-log entry, hence a paxos transaction, replicated to all three
-      # mons. Reducing that polling is the complementary fix and is tracked
-      # separately; this setting removes the amplification regardless of rate.
-      #
-      # This does NOT disable compaction. RocksDB's own levelled compaction
-      # still runs and still reclaims trimmed keys; what stops is the extra
-      # forced compact_range on every trim. Store growth is therefore bounded
-      # by normal compaction, and there are two backstops if that assumption is
-      # wrong: mon_data_size_warn (15 GB default) raises HEALTH_WARN, which
-      # CephHealthWarn in monitoring-rules already alerts on, and
-      # mon_compact_on_start below compacts fully on every mon restart.
-      #
-      # Watch after applying: `du -sh /var/lib/rook/mon-*/store.db` on a node,
-      # or the mon store size in `ceph status`. If it grows past ~1 GB and does
-      # not come back down, revisit rather than leaving it.
+      # This does NOT disable compaction — RocksDB's levelled compaction still
+      # reclaims trimmed keys. Backstops if that proves wrong:
+      # mon_data_size_warn raises HEALTH_WARN, and mon_compact_on_start below
+      # compacts fully on every mon restart.
      mon_compact_on_trim: "false"
-      # Compensating control for the above: a full compaction on mon start.
-      # Upstream default is false. Mons restart on upgrades, node reboots and
-      # reschedules, so this gives a periodic full reclaim without paying for
-      # one on every trim. Cheap here — the store is ~106 MB.
+      # Compensating control, against the upstream default: mons restart often
+      # enough for this to give periodic full reclaim without paying for one
+      # on every trim.
      mon_compact_on_start: "true"
    mgr:
      mgr/volumes/max_concurrent_clones: "16"
-    # Discard/TRIM for the QLC NVMe per host (CT1000P3PSSD8, Crucial P3 Plus —
-    # earlier comments here said "P2", which is a different drive). These need
-    # an OSD restart to take effect (init-time). Baseline 2026-05-05:
-    # 71.6 MB/s avg / 0.89 s avg latency on a 5-min replicated 4 MiB write
-    # bench (post-SLC-cache). See ceph-speed-baseline-2026-05-05 in the wiki.
+    # Discard/TRIM. Init-time settings — an OSD restart is needed to apply.
    #
-    # bluestore_prefer_deferred_size_ssd / bluestore_deferred_batch_ops_ssd are
-    # DELIBERATELY LEFT AT THE CEPH DEFAULTS (0 and 16) and must not be raised
-    # again without re-reading this.
-    #
-    # They were previously set to the HDD-style 65536 / 64 as a throughput tune.
-    # prefer_deferred_size defers every write SMALLER than the threshold: the
-    # write goes first into RocksDB's WAL and is applied to the object store
-    # afterwards. That is a double write, and at 65536 it caught essentially
-    # every write this cluster makes — SQLite pages are 4 KiB, Postgres pages
-    # 8 KiB. Ceph disables deferral on SSD (default 0) precisely because solid
-    # state does not need the random-write batching that helps spinning rust.
-    #
-    # Two reasons it was the wrong trade here:
-    #   - The benchmark that justified it cannot have exercised it. It used
-    #     4 MiB writes, 64x ABOVE the 64 KiB cutoff, so not one of those writes
-    #     was ever deferred. The setting was validated by a workload it does
-    #     not affect.
-    #   - It doubles writes onto QLC, which has the lowest endurance of any
-    #     NAND type. Measured 2026-08-19: all three OSD drives at 80/82/87%
-    #     endurance used, advancing ~0.13 points/day, i.e. rated life exhausted
-    #     between Nov 2026 and Jan 2027 — and simultaneously, since size=3 on
-    #     3 OSDs means every drive absorbs 100% of cluster writes (see #434).
-    #
-    # This cluster is write-light (~117 GB/day at the RBD client layer), so
-    # throughput was never the binding constraint. Endurance is. If small-write
-    # latency ever becomes the real problem, fix it with more OSDs rather than
-    # by paying a 2x write tax on every 4 KiB page.
+    # bluestore_prefer_deferred_size_ssd and bluestore_deferred_batch_ops_ssd
+    # are DELIBERATELY LEFT AT THE CEPH DEFAULTS and must not be raised again.
+    # Deferral double-writes everything below the threshold, and an HDD-style
+    # value catches every write this cluster makes (4 KiB SQLite, 8 KiB
+    # Postgres pages). Ceph disables it on SSD for that reason. Throughput has
+    # never been the binding constraint here; endurance is. If small-write
+    # latency becomes a real problem, add OSDs instead.
    osd:
      bdev_enable_discard: "true"
      bdev_async_discard: "true"
@@ -159,106 +79,44 @@ spec:
    ssl: false
  monitoring:
    enabled: true
-  # Dedicated Ceph cluster network on VLAN 49, carried by the second X710
-  # SFP+ port (enp2s0f1np1) on each node, attached to the OSDs through Multus.
+  # Dedicated Ceph cluster network on VLAN 49, attached to the OSDs through
+  # Multus. The attachment is san-network-attachment.yaml.
  #
-  # WHY NOT provider: host, WHICH ALSO REACHES THAT LINK
+  # Only `cluster` is selected. Do not add `public`, and do not reach for
+  # `addressRanges` — that is honoured only under provider: host, which puts
+  # every daemon including the mons on host networking. A mon's address then
+  # becomes the node's DHCP lease, and when a lease moves the mon crash-loops
+  # on "Cannot assign requested address" with no node free to fail over to,
+  # because all three already hold the mon host ports.
  #
-  # Because it reaches it for every daemon, not just the ones that need it.
-  # The cluster network is an OSD concern: per the Ceph documentation it
-  # carries OSD heartbeat, replication and recovery, and "Ceph Monitors always
-  # operate on the public network". `addressRanges` is only honoured under
-  # provider: host, so using it put the mons on host networking as a side
-  # effect — making a mon's address the node's DHCP lease.
+  # With `public` unset, client traffic stays on the pod network and VLAN 49
+  # carries Ceph-internal traffic only — which is also why Rook's Multus
+  # prerequisites about host routing do not apply here.
  #
-  # That is not theoretical. When a lease moved, the mon could no longer bind
-  # ("Cannot assign requested address") and crash-looped. Rook detected it and
-  # started a failover, but with three mons on three nodes each holding the
-  # mon host ports, the replacement could not be scheduled anywhere — the
-  # canary sat Pending on "didn't have free ports". Two of three nodes hit
-  # this during the Talos v1.12.11 upgrade and both needed hand intervention.
-  #
-  # Only `cluster` is selected here. With `public` left unset Rook puts client
-  # traffic on the pod network, so mons, MDS, MGR and the CSI pods keep
-  # Rook-managed identities and hold no host ports — which is what removes
-  # that failure mode rather than working around it.
-  #
-  # ISOLATION IS PRESERVED
-  #
-  # Rook's Multus prerequisites about hosts routing to the Multus network
-  # apply to a Multus *public* network. Selecting only `cluster` means they do
-  # not apply, so VLAN 49 stays what it is: Ceph-internal traffic and nothing
-  # else, no router, no other participants. After this settles, the node
-  # addresses on that segment (10.99.0.1-.3) have no remaining purpose and
-  # should come out of the Talos configs — at which point the attachment's
-  # pool can take the whole /29.
-  #
-  # Reached from provider: "" rather than directly from host, which Rook's CRD
-  # rejects. See the phase A commit.
-  #
-  # ONE-TIME STEP THAT THIS FILE CANNOT EXPRESS, recorded because it is not
-  # reproducible from the manifests alone. Removing addressRanges here does
-  # not clear public_network and cluster_network from Ceph's own config
-  # database — the same trap the bluestore comment above describes: `ceph
-  # config` retains a value that is only deleted from this file. During the
-  # phase A rollout the first OSD came up on the pod network, still had
-  # public_network 10.2.40.0/24 in the config database, found no interface in
-  # it and crash-looped on "Failed to pick public address". Both keys were
-  # removed with `ceph config rm global public_network` and the same for
-  # cluster_network; Rook then wrote cluster_network afresh from the
-  # attachment below. Anyone rebuilding this cluster from scratch will not hit
-  # it — the config database starts empty — but anyone repeating a provider
-  # change on a live cluster will.
-  #
-  # Bounded by upgradeOSDRequiresHealthyPGs and storage.osdMaxUpdatesInParallel
-  # above: one OSD restarts at a time and only once PGs are clean again.
-  #
-  # The attachment itself is san-network-attachment.yaml.
+  # Changing the provider on a LIVE cluster needs a step this file cannot
+  # express: `ceph config` retains public_network and cluster_network after
+  # they disappear from here, and an OSD that comes up on the pod network with
+  # a stale public_network crash-loops on "Failed to pick public address".
+  # Clear them with `ceph config rm global public_network` (and cluster_network)
+  # before the rollout. A cluster built from scratch starts with an empty
+  # config database and never sees this.
  network:
    provider: multus
    selectors:
      cluster: rook-ceph/ceph-san
-  # Nodes are listed explicitly rather than useAllNodes, so that a node can be
-  # taken out of the storage fleet by removing it here. Under useAllNodes the
-  # operator re-provisions an OSD on any node whose device still carries a
-  # BlueStore signature, which makes purging an OSD ahead of pulling its drive
-  # impossible: the OSD is recreated within a reconcile and starts backfilling
-  # onto the drive being replaced.
+  # Nodes listed explicitly, not useAllNodes: under useAllNodes the operator
+  # re-provisions an OSD on any node whose device still carries a BlueStore
+  # signature, so purging an OSD before pulling its drive is impossible — it
+  # comes back within a reconcile and backfills onto the drive being replaced.
  #
-  # All three nodes are bare metal and see three NVMe drives each, so they name
-  # their OSDs by /dev/disk/by-id path. A bare name would be actively wrong:
-  # Talos enumerates these machines' drives in a different order than Proxmox
-  # did, and the kernel name lands on the Kingston SYSTEM disk or on EPHEMERAL
-  # depending on the boot. Rook would probably decline the system disk for
-  # carrying partitions, but that is luck rather than intent. The by-id path is
-  # derived from model and serial and does not depend on enumeration at all.
-  #
-  # k8s-3 demonstrated why: it enumerated the Kingston as nvme0n1 on one boot
-  # and the Crucial as nvme0n1 on the very next one.
-  #
-  #   k8s-1 (jupiter)
-  #   50026B7382B48B86  Kingston  — Talos system disk
-  #   2332E86B749A      Crucial   — Talos EPHEMERAL (/var)
-  #   S437NX0M400426    PM983     — this OSD
-  #
-  #   k8s-2 (saturn)
-  #   50026B7382DA5F6E  Kingston  — Talos system disk
-  #   2332E86B74BC      Crucial   — Talos EPHEMERAL (/var)
-  #   S437NX0M403161    PM983     — this OSD
-  #
-  #   k8s-3 (neptune)
-  #   50026B7382B48B90  Kingston  — Talos system disk
-  #   2326E6E8E67D      Crucial   — Talos EPHEMERAL (/var)
-  #   S437NF0M501176    PM983     — this OSD
+  # Devices are named by /dev/disk/by-id path. Kernel names are not stable
+  # across boots on these machines — k8s-3 enumerated a different drive as
+  # nvme0n1 on two consecutive boots — and a bare name can land on the system
+  # disk or on EPHEMERAL.
  storage:
-    # One OSD restarted at a time. The default is 20, which on a three-OSD
-    # cluster means all of them at once — every PG inactive, every pool
-    # unreachable, for as long as the restart takes. The default suits a
-    # cluster large enough that 20 OSDs is a small fraction of it; here it is
-    # the whole thing.
-    #
-    # This binds any change that rewrites the OSD deployment spec, not just
-    # Ceph version upgrades — a network provider change is one such.
+    # One at a time. The default of 20 would restart every OSD in this cluster
+    # simultaneously — every PG inactive until they return. Binds any change
+    # that rewrites the OSD deployment spec, not just version upgrades.
    osdMaxUpdatesInParallel: 1
    useAllNodes: false
    useAllDevices: false
@@ -274,18 +132,12 @@ spec:
          - fullpath: /dev/disk/by-id/nvme-SAMSUNG_MZQLB960HAJR-00007_S437NF0M501176
    config:
      osdsPerDevice: "1"
-  # Storage layer must never be evicted by kubelet under node pressure
-  # and must never be picked first by the Linux OOM killer. system-node-
-  # critical (PriorityValue 2000001000) is the highest built-in class:
-  # kubelet preserves these pods last, and kernel sets oom_score_adj to
-  # -997 so anything else dies before a Ceph daemon does.
+  # Highest built-in class: kubelet evicts these last and the kernel gives
+  # them the lowest oom_score_adj.
  priorityClassNames:
    all: system-node-critical
-  # memory requests == limits => Guaranteed QoS. Combined with the
-  # priority class above, the pod's oom_score_adj is the lowest possible.
-  # CPU limits are deliberately NOT set — Ceph daemons benefit from
-  # bursting under recovery/backfill, and a CPU limit would throttle
-  # them; CPU pressure on this cluster is not a real concern (~15%).
+  # requests == limits gives Guaranteed QoS. CPU limits are deliberately NOT
+  # set — Ceph daemons burst during recovery and a limit would throttle them.
  resources:
    mon:
      requests:
@@ -1,22 +1,7 @@
 ---
-# The Ceph cluster network as a Multus attachment, so OSDs reach VLAN 49
-# without every Ceph daemon being put on host networking to get there.
-#
-# WHAT THIS REPLACES, AND WHY
-#
-# `addressRanges` only takes effect under `provider: host` — Rook ignores it
-# for the default CNI provider. So carrying replication over the dedicated
-# 10G link meant setting provider: host, which is not per-daemon: it puts
-# mons on host networking too. A mon's address is then the node's DHCP lease,
-# and when that lease moves the mon cannot bind and fails. Worse, three mons
-# on three nodes each hold the mon host ports, so Rook cannot schedule a
-# replacement anywhere and the failover it starts cannot finish.
-#
-# None of that has anything to do with the goal. Per the Ceph documentation
-# the cluster network carries OSD heartbeat, replication and recovery only —
-# "Ceph Monitors always operate on the public network". Attaching it here
-# gives the OSDs the link and leaves everything else on the pod network,
-# where Rook manages identity itself.
+# The Ceph cluster network as a Multus attachment, so the OSDs reach VLAN 49
+# without every Ceph daemon landing on host networking to get there. The
+# alternative and why it was rejected is in cephcluster.yaml.
 #
 # ISOLATION
 #
@@ -7,11 +7,8 @@ spec:
  interval: 24h
  url: https://traefik.github.io/charts
 ---
-# Traefik runs alongside ingress-nginx during the migration off the
-# archived kubernetes/ingress-nginx project. Shares the `ingress` namespace
-# so the post-migration topology is just "ingress namespace, one controller".
-# Pilots: shelly-manager.home.renner.uno. ingress-nginx remains the default
-# IngressClass; this controller is opt-in via `ingressClassName: traefik`.
+# The cluster's only ingress controller, and the only IngressClass. It
+# replaced ingress-nginx, which was retired after that project was archived.
 apiVersion: helm.toolkit.fluxcd.io/v2
 kind: HelmRelease
 metadata:
@@ -44,27 +41,18 @@ spec:
  values:
    deployment:
      replicas: 2
-    # Entrypoint-default Middlewares for websecure: baseline security
-    # response headers (HSTS/CSP/X-Frame-Options/etc.) and a generous
-    # abuse/brute-force rate limit, both applied to every router on this
-    # entrypoint without per-Ingress annotations. See
-    # security-headers-middleware.yaml / rate-limit-middleware.yaml for the
-    # actual settings and rationale. This is the same entrypoint-default
-    # wiring mechanism the (currently detached) crowdsec-bouncer Middleware
-    # documents re-enabling via — independent of that CrowdSec enforcement
-    # decision, which stays off pending the client-IP-visibility fix
-    # referenced below.
+    # Entrypoint-default Middlewares for websecure: security response headers
+    # and a rate limit, applied to every router on this entrypoint without
+    # per-Ingress annotations. Settings live in
+    # security-headers-middleware.yaml and rate-limit-middleware.yaml.
    additionalArguments:
      - --entryPoints.websecure.http.middlewares=ingress-security-headers@kubernetescrd,ingress-rate-limit@kubernetescrd
-    # CrowdSec bouncer plugin (Yaegi), loaded but NOT globally attached.
-    # Enforcement was disabled after it caused an IPv6 ingress outage on
-    # first deploy: Traefik sees a single shared/SNAT'd source IP for v6
-    # ingress (no real per-client IP), so one http-admin-interface-probing
-    # hit (the e2e scan) banned that shared IP and 403'd all v6 clients.
-    # Re-attach the bouncer (additionalArguments entrypoint middleware ->
-    # ingress-crowdsec-bouncer@kubernetescrd) only after fixing client-IP
-    # visibility (forwardedHeaders/trustedIPs or the v6 masquerade) and
-    # adding an internal/LAN whitelist. The plugin + crowdsec-middleware.yaml
+    # CrowdSec bouncer plugin (Yaegi), loaded but deliberately NOT attached.
+    # Traefik sees one shared SNAT'd source for IPv6 ingress, so a single
+    # probe banned that address and 403'd every v6 client. Do not re-attach
+    # it until client-IP visibility is fixed (forwardedHeaders/trustedIPs or
+    # the v6 masquerade) and a LAN whitelist exists.
+    # The plugin + crowdsec-middleware.yaml
    # stay in place so re-enabling is a one-line change. Detection (the
    # crowdsec agents parsing Traefik logs) remains on and is unaffected.
    experimental:
@@ -5,52 +5,24 @@ machine:
    - k8s-3.srv.renner.uno
    - k8s.renner.uno
    - 10.2.40.60
-  # Install from the Image Factory metal installer, not the stock
-  # ghcr.io/siderolabs/installer that `talosctl gen config` defaults to.
-  # Extensions live in the installer image, so with the stock one the schematic
-  # is only ever a property of the boot media and nothing lands on disk —
-  # `talosctl get extensions` returns empty on a node installed that way.
+  # Image Factory metal installer, NOT the stock ghcr.io/siderolabs/installer
+  # that `talosctl gen config` defaults to. Extensions live in the installer
+  # image; install with the stock one and nothing lands on disk —
+  # `talosctl get extensions` comes back empty.
  #
-  # What the schematic carries, and why each matters here:
+  # The schematic carries intel-ucode, thunderbolt and i915, and sets
+  # net.ifnames=1, which is what the per-node VLAN parents match against.
+  # Changing which extensions are present means a new schematic hash and a
+  # reinstall or upgrade, not an edit here.
  #
-  #   siderolabs/intel-ucode   CPU microcode. On a VM the hypervisor supplied
-  #                            it; on bare metal nothing does except the BIOS.
-  #   siderolabs/thunderbolt   Inert until machine.kernel.modules asks for
-  #                            thunderbolt/thunderbolt_net. Present so enabling
-  #                            the MS-01 mesh later needs neither a new image
-  #                            nor a reinstall — which is exactly the property
-  #                            lost by installing without it.
-  #   siderolabs/i915          Intel GPU microcode and kernel modules for the
-  #                            Alder Lake-P Iris Xe in the MS-01. No workload
-  #                            requests /dev/dri today; it is here for the same
-  #                            reason as thunderbolt, so that wanting hardware
-  #                            transcoding later is a config change rather than
-  #                            another rolling upgrade of every node.
+  # Read only at install and upgrade time — editing this does nothing to a
+  # running node. The tag must match the version being upgraded to, or the
+  # upgrade is a downgrade. Talos only tests migration between adjacent minor
+  # releases: take the latest patch of each intermediate minor rather than
+  # skipping one.
  #
-  # Not included, and why, so the absences are not mistaken for oversights:
-  # the X710 (i40e) and I226 (igc) NICs need no firmware extension —
-  # intel-ice-firmware covers E810/ice only. mei is a prerequisite for Intel
-  # Arc discrete GPUs, of which there are none. iscsi-tools, nvme-cli and
-  # util-linux-tools have no consumer: storage is Ceph RBD, and device health
-  # is reported by Ceph itself rather than a node-level exporter.
-  #
-  # 0f6fb8bd… is the bare-metal schematic: same net.ifnames=1 as the Proxmox
-  # one (the predictable interface names the per-node VLAN parents match
-  # against), minus qemu-guest-agent. It supersedes a21cec8f…, which was
-  # identical but without i915 — see RUNBOOK-cold-rebuild.md.
-  #
-  # Only read at install and upgrade time. Changing it here does nothing to a
-  # running node until that node is upgraded. Bump the tag with the Talos
-  # version — it must match the running version, or the upgrade is a downgrade.
-  #
-  # Upgrades follow the documented path: migration is only tested between
-  # adjacent minor releases, so go to the latest patch of each intermediate
-  # minor rather than skipping. 1.12.11 -> 1.13.9 satisfies that — one minor
-  # step from the last 1.12 patch, no intermediate needed.
-  #
-  # Kubernetes is a separate decision and is not forced by this. Talos 1.13
-  # supports Kubernetes 1.36, 1.35, 1.34, 1.33, 1.32 and 1.31, so the cluster
-  # stays on 1.35.2 until `talosctl upgrade-k8s` moves it deliberately.
+  # The Kubernetes version is a separate decision, moved by
+  # `talosctl upgrade-k8s`.
  install:
    image: factory.talos.dev/metal-installer/0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9:v1.13.9
  features:
@@ -74,20 +46,13 @@ machine:
    nodeIP:
      validSubnets:
        - 10.2.40.0/24
-    # Lower the image GC thresholds from the kubelet defaults (85/80) to
-    # bound containerd's image cache, which is the largest single consumer
-    # of /var. Spegel P2P pull-through softens the re-pull cost when GC'd
-    # images are needed back.
+    # Lower than the kubelet defaults (85/80) to bound containerd's image
+    # cache, the largest single consumer of /var.
    #
-    # These thresholds do NOT control whether Ceph's mon reports
-    # MON_DISK_LOW, and must not be tuned for that. Measured on k8s-2:
-    # image GC frees only the content store (~8.6 GB), while the bulk of
-    # /var/lib/containerd is ~20 GB of overlayfs snapshots — unpacked
-    # layers shared with running containers, which GC cannot reclaim.
-    # Dropping high to 60 deleted 28 images and moved free space by 0.02
-    # percentage points. The mon's own data is ~106 MB; its 30%-free
-    # warning is measuring a filesystem it merely shares, so the lever for
-    # it is elsewhere (mon volume, /var size, or mon_data_avail_warn).
+    # Do NOT tune these to clear Ceph's MON_DISK_LOW. Image GC reclaims only
+    # the content store; the bulk of /var/lib/containerd is overlayfs
+    # snapshots shared with running containers, which GC cannot touch. The
+    # lever for the mon warning is elsewhere.
    extraConfig:
      imageGCHighThresholdPercent: 70
      imageGCLowThresholdPercent: 60
@@ -1,55 +1,25 @@
-# k8s-1 runs Talos directly on jupiter's MS-01 hardware; k8s-2 and k8s-3 are
-# still Proxmox VMs. Every machine-specific selector below differs from theirs
-# for that reason: there are no virtio NICs, no PCI-passthrough layer, and all
-# four physical NICs and all three NVMe drives are visible to Talos at once.
-#
-# Networking uses the v1.12 multi-document config rather than the deprecated
-# machine.network.interfaces. That is deliberate: the docs do not define what
-# happens when the same link is configured by both schemas, so this file keeps
-# every link in one schema instead of straddling them.
+# Networking uses the multi-document config, not the deprecated
+# machine.network.interfaces. Keep every link in one schema: the docs do not
+# define what happens when both configure the same link.
 machine:
  network:
    hostname: k8s-1
  sysctls:
-    # ens19 (VLAN 30, v30iot) is LINK ONLY — no host addressing. It exists as
-    # the macvlan parent for the v30iot pod attachments
-    # (matter/otbr/shelly-manager/music-assistant), whose children do their own
-    # DHCP on that L2. Host addresses were never needed and are actively
-    # harmful: a connected 10.2.30.0/24 route makes node replies to IoT-subnet
-    # clients bypass the UDM, which breaks every UDM-DNATed flow (hairpin NAT:
-    # un-translated SYN-ACK -> client kernel RST, verified live 2026-07-23).
-    # This disables v6 entirely on that link, including link-local.
+    # ens19 (VLAN 30, v30iot) is LINK ONLY — it is the macvlan parent for the
+    # v30iot pod attachments and must never get host addressing. A connected
+    # 10.2.30.0/24 route makes node replies bypass the UDM and breaks every
+    # DNATed flow.
    net.ipv6.conf.ens19.disable_ipv6: "1"
-    # Same argument, applied to the physical ports rather than a VLAN child.
-    #
-    # enp2s0f0np0 is nothing but the trunk parent for mgmt0 and ens19, and
-    # enp87s0 is unused. Both sit on the untagged fallback network, where by
-    # design nothing is supposed to run — and both were autoconfiguring
-    # themselves from its router advertisements: an fd00::/64 ULA each, plus
-    # an IPv6 default route each.
-    #
-    # The result was four IPv6 default routes per node at identical priority
-    # 1024 — mgmt0, enp2s0f0np0, enp87s0 and lo — of which only mgmt0 has a
-    # globally routable source address. With nothing to break the tie,
-    # outbound IPv6 mostly left through an interface whose only usable source
-    # was a ULA, so the cluster had no IPv6 egress at all while IPv4 worked
-    # normally. It surfaced as two NetBird probes reporting the proxies down
-    # for 30 hours; the proxies were fine and answered IPv6 in 13 ms from
-    # outside.
-    #
-    # Verified on k8s-1 before rolling to the others: disabling v6 on a VLAN
-    # parent leaves the VLAN child alone. mgmt0 kept its global address and
-    # its fd40:: address, Ceph did not notice, and every failing probe
-    # recovered on the next scrape.
+    # These two sit on the untagged fallback network, where nothing may run.
+    # Left enabled they autoconfigure a ULA and an IPv6 default route from its
+    # RAs, which ties with mgmt0's and costs the cluster IPv6 egress.
+    # Disabling v6 on a VLAN parent does not affect the VLAN child.
    net.ipv6.conf.enp2s0f0np0.disable_ipv6: "1"
    net.ipv6.conf.enp87s0.disable_ipv6: "1"
  install:
-    # Three NVMe drives are ordinary block devices to Talos here: the Kingston
-    # OM8PGP41024Q-A0 system disk, the Samsung PM983 that Rook claims as the
-    # Ceph OSD, and the retired Crucial P3 that carries EPHEMERAL (see the
-    # VolumeConfig below). Every one of them is selected by serial. Nothing in
-    # this file may use a broad selector: `disk.transport == 'nvme'` matches all
-    # three, and the failure mode is Talos taking the OSD drive.
+    # Select disks by serial only. A broad selector such as
+    # `disk.transport == 'nvme'` matches every drive in the machine, and the
+    # failure mode is the installer taking the Ceph OSD drive.
    diskSelector:
      serial: 50026B7382B48B86
  certSANs:
@@ -59,15 +29,10 @@ machine:
 cluster:
  allowSchedulingOnControlPlanes: true
 ---
-# EPHEMERAL (/var) goes on the retired Crucial P3, not the system disk.
-#
-# That drive is at 87% of its rated endurance and was this node's Ceph OSD until
-# the PM983 replaced it. /var is exactly the right role for a disk in that
-# state: containerd's image and snapshot store, pod logs and the Ceph mon store
-# are all reconstructible, so the drive is consumable here in a way it is not as
-# an OSD. It also gives /var ~931 GB instead of the 64 GB the VM-era zvol had,
-# which is the constraint that makes k8s-2 and k8s-3 trip DiskPressure whenever
-# they absorb another node's pods.
+# EPHEMERAL (/var) goes on the retired Ceph OSD drive, deliberately: its
+# contents are reconstructible, so a worn disk is consumable here in a way it
+# is not as an OSD. Note /var/lib/etcd lives here too, which is the one thing
+# on the volume that is not free to lose.
 #
 # THIS ONLY TAKES EFFECT ON FIRST PROVISION. Talos applies volume configuration
 # solely when the volume does not yet exist; a config applied after EPHEMERAL
@@ -75,8 +40,8 @@ cluster:
 # requires wiping the disk and booting install media again.
 #
 # Matched by serial deliberately. `disk.transport == 'nvme' && !system_disk`
-# would match the PM983 as well, and losing that coin flip means Talos formats
-# the OSD drive.
+# still matches the OSD drive, and losing that coin flip means Talos formats
+# it.
 apiVersion: v1alpha1
 kind: VolumeConfig
 name: EPHEMERAL
@@ -86,30 +51,25 @@ provisioning:
  minSize: 50GiB
  grow: true
 ---
-# X710 port 0 carries the switch trunk. It takes no addressing itself — the
-# port's untagged VLAN is the 192.168.8.0/24 network, which this node has no
-# business being on — so management and v30iot both ride it as tagged VLANs.
+# X710 port 0 carries the switch trunk and takes no addressing itself — its
+# untagged VLAN is a network this node has no business being on. Management
+# and v30iot ride it as tagged VLANs.
 #
-# Links are addressed by predictable kernel name. Those derive from PCI
-# topology, not probe order — though not immune: adding the third NVMe
-# renumbered the buses and moved the unused second I226 from enp89s0 to
-# enp90s0. Neither name used here was affected.
+# Links are named by predictable kernel name, which derives from PCI topology
+# and can still shift when hardware is added.
 #
-# LinkAliasConfig would be the better way to pin these, and is worth
-# reintroducing — but only once it is proven. Aliasing both X710 ports by
-# permanent MAC and using the aliases as VLAN parents produced a node with no
-# address on any interface: the aliases were never created, so every document
-# referencing them did nothing, including the static address on port 1. Verify
-# any alias-based revision with `talosctl apply-config --mode=try`, which rolls
-# back on its own, before making it permanent.
+# LinkAliasConfig would pin these better, but aliasing the X710 ports by MAC
+# and using the aliases as VLAN parents left the node with no address on any
+# interface — the aliases were never created and every document referencing
+# them silently did nothing. Prove any alias-based revision with
+# `talosctl apply-config --mode=try` first; it rolls back on its own.
 apiVersion: v1alpha1
 kind: LinkConfig
 name: enp2s0f0np0
 up: true
 ---
 # Management, VLAN 40. Carries the node IP, etcd, the API and Ceph's public
-# network. The k8s-1 VM reached 10.2.40.174 over this same physical port via
-# Proxmox's v40srv VNet (vmbr0.40), so the switch trunks VLAN 40 here.
+# network.
 apiVersion: v1alpha1
 kind: VLANConfig
 name: mgmt0
@@ -117,30 +77,15 @@ vlanID: 40
 parent: enp2s0f0np0
 up: true
 ---
-# Management addressing is DHCP and the lease is NOT reserved anywhere in this
-# repo — the k8s-{1,2,3}.srv.renner.uno reservations live on the UDM out of band
-# and opentofu/unifi.tf declares no live resources. The addresses do move under
-# a running node: k8s-3 went 10.2.40.168 -> .138 mid-flight on 2026-08-26
-# without its Node object ever re-registering (issue #635).
+# DHCP, and the lease is not reserved in this repo — reservations live on the
+# UDM out of band. Addresses do move under a running node.
 #
-# kube-apiserver does not follow when that happens. Talos starts it with
-# `--advertise-address=$(POD_IP)` and no `--endpoint-reconciler-type` override,
-# so the advertised address is fixed at pod start and the default lease
-# reconciler goes on renewing the masterlease under the OLD address. Nothing
-# expires it while the process lives, so `default/kubernetes` keeps handing out
-# a dead endpoint and the node's live address is never registered at all — 19h
-# of TargetDown on job="apiserver", a monitoring blind spot on one of three
-# apiservers, and it cleared only when the static pod restarted.
-#
-# So after any node re-IPs, check that
+# kube-apiserver does not follow a re-IP: it advertises $(POD_IP) fixed at pod
+# start and the lease reconciler keeps renewing the masterlease under the old
+# address, so `default/kubernetes` hands out a dead endpoint until the static
+# pod restarts (#635). After any node re-IPs, check that
 # `kubectl get endpoints kubernetes -n default` agrees with
-# `kubectl get nodes -o wide`, and restart the offending node
-# (`talosctl -n <node> reboot`) if it does not.
-#
-# Pinning mgmt0 with a static AddressConfig alongside this document would remove
-# the trigger, but it needs an address outside the UDM's DHCP pool — which is
-# not declared in this repo — and §6 of RUNBOOK-baremetal-conversion.md applies:
-# prove it with `talosctl apply-config --mode=try` before making it permanent.
+# `kubectl get nodes -o wide` and reboot the node if it does not.
 apiVersion: v1alpha1
 kind: DHCPv4Config
 name: mgmt0
@@ -152,10 +97,8 @@ kind: Layer2VIPConfig
 name: 10.2.40.60
 link: mgmt0
 ---
-# Named ens19 on purpose. The four v30iot macvlan NADs hard-code
-# `master: ens19`, and k8s-2/k8s-3 still present the VLAN as a virtio NIC of
-# that name. VLANConfig.name is free-form, so keeping it lets one set of NADs
-# serve both VM and bare-metal nodes for as long as the cluster is mixed.
+# Named ens19 on purpose: the v30iot macvlan NADs hard-code `master: ens19`.
+# VLANConfig.name is free-form, so renaming this link means editing them too.
 apiVersion: v1alpha1
 kind: VLANConfig
 name: ens19
@@ -163,19 +106,13 @@ vlanID: 30
 parent: enp2s0f0np0
 up: true
 ---
-# Dedicated Ceph cluster network on X710 port 1, VLAN 49 / 10.99.0.0/29,
-# untagged on the wire. Carries OSD<->OSD replication and nothing else.
+# Dedicated Ceph cluster network on X710 port 1, VLAN 49, untagged on the
+# wire. OSD-to-OSD replication and nothing else.
 #
-# The node deliberately holds NO address here. Rook attaches this link to the
-# OSD pods through a Multus macvlan (infrastructure/rook-ceph-cluster/
-# san-network-attachment.yaml), and a macvlan parent only has to be up — it
-# does not need an address of its own for its children to work.
-#
-# It used to carry 10.99.0.1/29, because reaching this link required
-# CephCluster.spec.network.provider: host, and that put every Ceph daemon on
-# host networking, mons included. Nothing needs it now, and by the rule this
-# segment is built on — Ceph-internal traffic and nothing else — the node has
-# no business on it either.
+# The node holds NO address here on purpose. Rook attaches the link to the OSD
+# pods through a Multus macvlan, and a macvlan parent only needs to be up.
+# Giving the node an address would put it on a segment reserved for
+# Ceph-internal traffic.
 apiVersion: v1alpha1
 kind: LinkConfig
 name: enp2s0f1np1
@@ -9,20 +9,14 @@ machine:
    # macvlan parent for the v30iot pod attachments; host addresses on that L2
    # make node replies bypass the UDM and break every UDM-DNATed flow.
    net.ipv6.conf.ens19.disable_ipv6: "1"
-    # Same argument for the physical ports: enp2s0f0np0 is only the trunk
-    # parent for mgmt0 and ens19, enp87s0 is unused, and both sit on the
-    # untagged fallback network where nothing is supposed to run. Both were
-    # taking an fd00::/64 and an IPv6 default route from its router
-    # advertisements, leaving four equal-priority default routes per node of
-    # which only mgmt0 has a globally routable source. See k8s-1.yaml for the
-    # full reasoning and how it was verified.
+    # Same for the physical ports on the untagged fallback network: left
+    # enabled they take an IPv6 default route from its RAs and cost the
+    # cluster IPv6 egress.
    net.ipv6.conf.enp2s0f0np0.disable_ipv6: "1"
    net.ipv6.conf.enp87s0.disable_ipv6: "1"
  install:
-    # Three NVMe drives are visible on bare metal. Select the installer target
-    # by serial so it cannot land on the OSD drive or on EPHEMERAL, whatever
-    # order the kernel enumerates them in — Talos and Proxmox do not agree on
-    # that order.
+    # Select the installer target by serial so it cannot land on the OSD drive
+    # or on EPHEMERAL, whatever order the kernel enumerates them in.
    diskSelector:
      serial: 50026B7382DA5F6E
  certSANs:
@@ -32,17 +26,15 @@ machine:
 cluster:
  allowSchedulingOnControlPlanes: true
 ---
-# EPHEMERAL (/var) goes on the retired Crucial P3, not the system disk. That
-# drive is at 81% of its rated endurance and is finished as an OSD, but
-# everything EPHEMERAL holds is reconstructible, so it is consumable in this
-# role. It also gives /var ~931 GB instead of the 64 GB the VM-era zvol had.
+# EPHEMERAL (/var) goes on the retired Ceph OSD drive, deliberately: its
+# contents are reconstructible, so a worn disk is consumable in this role.
 #
 # ONLY TAKES EFFECT ON FIRST PROVISION. Talos applies volume configuration
 # solely when the volume does not yet exist; applied later it is silently
 # ignored and relocating needs a disk wipe and a boot from install media.
 #
 # Matched by serial deliberately: a broad selector such as
-# `disk.transport == 'nvme' && !system_disk` would match the PM983 as well.
+# `disk.transport == 'nvme' && !system_disk` still matches the OSD drive.
 apiVersion: v1alpha1
 kind: VolumeConfig
 name: EPHEMERAL
@@ -85,9 +77,8 @@ kind: Layer2VIPConfig
 name: 10.2.40.60
 link: mgmt0
 ---
-# Named ens19 on purpose: the four v30iot macvlan NADs hard-code
-# `master: ens19`, and k8s-3 still presents the VLAN as a virtio NIC of that
-# name. VLANConfig.name is free-form, so one set of NADs serves both.
+# Named ens19 on purpose: the v30iot macvlan NADs hard-code `master: ens19`.
+# VLANConfig.name is free-form, so renaming this link means editing them too.
 apiVersion: v1alpha1
 kind: VLANConfig
 name: ens19
@@ -95,18 +86,11 @@ vlanID: 30
 parent: enp2s0f0np0
 up: true
 ---
-# Dedicated Ceph cluster network on X710 port 1, VLAN 49 / 10.99.0.0/29,
-# untagged on the wire. OSD<->OSD replication only. Not routed off its own
-# segment and deliberately has no default route.
+# Dedicated Ceph cluster network on X710 port 1, VLAN 49, untagged on the
+# wire. OSD-to-OSD replication only, not routed off its own segment.
 #
-# The node deliberately holds NO address here. Rook attaches this link to the
-# OSD pods through a Multus macvlan (infrastructure/rook-ceph-cluster/
-# san-network-attachment.yaml), and a macvlan parent only has to be up — it
-# does not need an address of its own for its children to work.
-#
-# It used to carry 10.99.0.2/29, because reaching this link required
-# CephCluster.spec.network.provider: host, which put every Ceph daemon on
-# host networking, mons included. Nothing needs it now.
+# The node holds NO address here on purpose. Rook attaches the link to the OSD
+# pods through a Multus macvlan, and a macvlan parent only needs to be up.
 apiVersion: v1alpha1
 kind: LinkConfig
 name: enp2s0f1np1
@@ -1,27 +1,18 @@
-# k8s-3 runs Talos directly on neptune's MS-01 hardware. Structurally identical
-# to k8s-1.yaml — see that file for the reasoning behind each choice; only the
-# serials and addresses differ. neptune was the last node converted, so all
-# three control-plane nodes now share this layout and no Proxmox remains.
+# Structurally identical to k8s-1.yaml — see that file for the reasoning
+# behind each choice; only the serials and addresses differ.
 machine:
  network:
    hostname: k8s-3
  sysctls:
    # v30iot NIC carries no host addressing — see k8s-1.yaml.
    net.ipv6.conf.ens19.disable_ipv6: "1"
-    # Same argument for the physical ports: enp2s0f0np0 is only the trunk
-    # parent for mgmt0 and ens19, enp87s0 is unused, and both sit on the
-    # untagged fallback network where nothing is supposed to run. Both were
-    # taking an fd00::/64 and an IPv6 default route from its router
-    # advertisements, leaving four equal-priority default routes per node of
-    # which only mgmt0 has a globally routable source. See k8s-1.yaml for the
-    # full reasoning and how it was verified.
+    # Same for the physical ports on the untagged fallback network — see
+    # k8s-1.yaml.
    net.ipv6.conf.enp2s0f0np0.disable_ipv6: "1"
    net.ipv6.conf.enp87s0.disable_ipv6: "1"
  install:
-    # Three NVMe drives are visible on bare metal. Select the installer target
-    # by serial so it cannot land on the OSD drive or on EPHEMERAL, whatever
-    # order the kernel enumerates them in — Talos and Proxmox do not agree on
-    # that order.
+    # Select the installer target by serial so it cannot land on the OSD drive
+    # or on EPHEMERAL, whatever order the kernel enumerates them in.
    diskSelector:
      serial: 50026B7382B48B90
  certSANs:
@@ -31,10 +22,8 @@ machine:
 cluster:
  allowSchedulingOnControlPlanes: true
 ---
-# EPHEMERAL (/var) goes on the retired Crucial P3, not the system disk. That
-# drive is at 83% of its rated endurance and is finished as an OSD, but
-# everything EPHEMERAL holds is reconstructible, so it is consumable in this
-# role. Available spare is still 100% and it reports no media errors.
+# EPHEMERAL (/var) goes on the retired Ceph OSD drive, deliberately: its
+# contents are reconstructible, so a worn disk is consumable in this role.
 #
 # ONLY TAKES EFFECT ON FIRST PROVISION. Talos applies volume configuration
 # solely when the volume does not yet exist; applied later it is silently
@@ -81,15 +70,8 @@ vlanID: 30
 parent: enp2s0f0np0
 up: true
 ---
-# Dedicated Ceph cluster network — see k8s-1.yaml.
-# The node deliberately holds NO address here. Rook attaches this link to the
-# OSD pods through a Multus macvlan (infrastructure/rook-ceph-cluster/
-# san-network-attachment.yaml), and a macvlan parent only has to be up — it
-# does not need an address of its own for its children to work.
-#
-# It used to carry 10.99.0.3/29, because reaching this link required
-# CephCluster.spec.network.provider: host, which put every Ceph daemon on
-# host networking, mons included. Nothing needs it now.
+# Dedicated Ceph cluster network — see k8s-1.yaml. The node holds NO address
+# here on purpose; the link is only a macvlan parent for the OSD pods.
 apiVersion: v1alpha1
 kind: LinkConfig
 name: enp2s0f1np1