chore(rook-ceph): return mon_data_avail_warn to the Ceph default

The override existed because the mons share the node's /var with containerd,
and on the VMs that was ~60 GB — 30% free measured image and snapshot churn
rather than anything about Ceph, so k8s-2 tripped MON_DISK_LOW whenever the
image cache filled and GC could not reclaim it (#434).

Bare metal removed the premise. /var is now ~931 GB per node at 7-8% used, so
30% free is a real reserve again and the ~106 MB mon database is irrelevant
against it. Reaching 70% would be a genuine signal about the node rather than
noise about the image cache. The comment still described the 60 GB VM /var and
ended with "BARE-METAL MIGRATION: revisit this", which is now done.

Set to 30 rather than deleted, deliberately. `ceph config` keeps a value that
only disappears from this file: `ceph config dump` currently shows
`mon advanced mon_data_avail_warn 20`, and removing the line would leave it
there indefinitely. That is the trap the bluestore settings below already
document, and the one that crash-looped an OSD during the network provider
change when public_network survived being deleted here.

mon_data_avail_crit stays at the 5% default, and CephMonDiskLow in
monitoring-rules still watches /var directly for the real risk, which is the
node filling rather than the mon growing.

kubectl diff against the live cluster shows the value and the generation bump,
nothing else. Verify after it applies with
`ceph config get mon mon_data_avail_warn` — 30 there is the proof the database
entry was overwritten and not merely edited in git.

Supersedes the remaining half of #503, which was closed as overtaken by the
bare-metal migration.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

kubernetes/infrastructure/rook-ceph-cluster/cephcluster.yaml

@@ -47,26 +47,27 @@ spec:
   # (paperless + audiobookshelf + immich + ... all clone at once).
   cephConfig:
     mon:
-      # Ceph's 30% default assumes the mon has a filesystem to itself, where
-      # 30% free is a meaningful reserve. Here the mons live on
-      # dataDirHostPath /var/lib/rook — the node's /var, shared with ~29 GB
-      # of containerd image and snapshot storage — and the mon DB itself is
-      # ~106 MB. So the default measures container churn, not Ceph: k8s-2
-      # trips MON_DISK_LOW whenever the image cache fills, and image GC does
-      # not clear it (most of /var/lib/containerd is overlayfs snapshots held
-      # by running containers, which GC cannot reclaim — see #434).
-      # 20% of a 60 GB /var still leaves ~12 GB against a 106 MB database.
-      # mon_data_avail_crit stays at the 5% default, and CephMonDiskLow in
-      # monitoring-rules watches /var directly for the real risk, which is the
-      # node filling rather than the mon growing.
+      # The Ceph default, stated rather than left out. Removing the line would
+      # not restore it: `ceph config` keeps a value that only disappears from
+      # this file, so the old 20 would sit in the mon config database
+      # indefinitely — the same trap the bluestore settings below describe,
+      # and the one that crash-looped an OSD during the network provider
+      # change when public_network survived being deleted here.
       #
-      # BARE-METAL MIGRATION: revisit this. Give the mons their own volume and
-      # the default 30% becomes meaningful again, at which point this override
-      # should be dropped rather than carried over. It cannot be done on the
-      # current cluster — every StorageClass is Ceph-backed, so a mon PVC
-      # would be circular, and Rook only applies mon.volumeClaimTemplate to
-      # newly created mons, never migrating existing ones.
-      mon_data_avail_warn: "20"
+      # It was 20 because the mons share the node's /var with containerd, and
+      # on the VMs that was ~60 GB: 30% free measured image and snapshot churn
+      # rather than anything about Ceph, so k8s-2 tripped MON_DISK_LOW whenever
+      # the image cache filled and GC could not reclaim it (#434).
+      #
+      # Bare metal removed the premise. /var is now ~931 GB per node at 7-8%
+      # used, so 30% free is a real reserve again and the ~106 MB mon database
+      # is irrelevant against it. Reaching 70% would be a genuine signal about
+      # the node rather than noise about the image cache.
+      #
+      # mon_data_avail_crit stays at the 5% default, and CephMonDiskLow in
+      # monitoring-rules still watches /var directly for the real risk, which
+      # is the node filling rather than the mon growing.
+      mon_data_avail_warn: "30"
       # Stop compacting the whole RocksDB prefix on every paxos trim.
       #
       # Upstream default is true — "Compact a certain prefix (including paxos)

kubernetes/talos/patches/k8s-1.yaml

@@ -20,6 +20,29 @@ machine:
     # un-translated SYN-ACK -> client kernel RST, verified live 2026-07-23).
     # This disables v6 entirely on that link, including link-local.
     net.ipv6.conf.ens19.disable_ipv6: "1"
+    # Same argument, applied to the physical ports rather than a VLAN child.
+    #
+    # enp2s0f0np0 is nothing but the trunk parent for mgmt0 and ens19, and
+    # enp87s0 is unused. Both sit on the untagged fallback network, where by
+    # design nothing is supposed to run — and both were autoconfiguring
+    # themselves from its router advertisements: an fd00::/64 ULA each, plus
+    # an IPv6 default route each.
+    #
+    # The result was four IPv6 default routes per node at identical priority
+    # 1024 — mgmt0, enp2s0f0np0, enp87s0 and lo — of which only mgmt0 has a
+    # globally routable source address. With nothing to break the tie,
+    # outbound IPv6 mostly left through an interface whose only usable source
+    # was a ULA, so the cluster had no IPv6 egress at all while IPv4 worked
+    # normally. It surfaced as two NetBird probes reporting the proxies down
+    # for 30 hours; the proxies were fine and answered IPv6 in 13 ms from
+    # outside.
+    #
+    # Verified on k8s-1 before rolling to the others: disabling v6 on a VLAN
+    # parent leaves the VLAN child alone. mgmt0 kept its global address and
+    # its fd40:: address, Ceph did not notice, and every failing probe
+    # recovered on the next scrape.
+    net.ipv6.conf.enp2s0f0np0.disable_ipv6: "1"
+    net.ipv6.conf.enp87s0.disable_ipv6: "1"
   install:
     # Three NVMe drives are ordinary block devices to Talos here: the Kingston
     # OM8PGP41024Q-A0 system disk, the Samsung PM983 that Rook claims as the
@@ -94,6 +117,30 @@ vlanID: 40
 parent: enp2s0f0np0
 up: true
 ---
+# Management addressing is DHCP and the lease is NOT reserved anywhere in this
+# repo — the k8s-{1,2,3}.srv.renner.uno reservations live on the UDM out of band
+# and opentofu/unifi.tf declares no live resources. The addresses do move under
+# a running node: k8s-3 went 10.2.40.168 -> .138 mid-flight on 2026-08-26
+# without its Node object ever re-registering (issue #635).
+#
+# kube-apiserver does not follow when that happens. Talos starts it with
+# `--advertise-address=$(POD_IP)` and no `--endpoint-reconciler-type` override,
+# so the advertised address is fixed at pod start and the default lease
+# reconciler goes on renewing the masterlease under the OLD address. Nothing
+# expires it while the process lives, so `default/kubernetes` keeps handing out
+# a dead endpoint and the node's live address is never registered at all — 19h
+# of TargetDown on job="apiserver", a monitoring blind spot on one of three
+# apiservers, and it cleared only when the static pod restarted.
+#
+# So after any node re-IPs, check that
+# `kubectl get endpoints kubernetes -n default` agrees with
+# `kubectl get nodes -o wide`, and restart the offending node
+# (`talosctl -n <node> reboot`) if it does not.
+#
+# Pinning mgmt0 with a static AddressConfig alongside this document would remove
+# the trigger, but it needs an address outside the UDM's DHCP pool — which is
+# not declared in this repo — and §6 of RUNBOOK-baremetal-conversion.md applies:
+# prove it with `talosctl apply-config --mode=try` before making it permanent.
 apiVersion: v1alpha1
 kind: DHCPv4Config
 name: mgmt0

kubernetes/talos/patches/k8s-2.yaml

@@ -9,6 +9,15 @@ machine:
     # macvlan parent for the v30iot pod attachments; host addresses on that L2
     # make node replies bypass the UDM and break every UDM-DNATed flow.
     net.ipv6.conf.ens19.disable_ipv6: "1"
+    # Same argument for the physical ports: enp2s0f0np0 is only the trunk
+    # parent for mgmt0 and ens19, enp87s0 is unused, and both sit on the
+    # untagged fallback network where nothing is supposed to run. Both were
+    # taking an fd00::/64 and an IPv6 default route from its router
+    # advertisements, leaving four equal-priority default routes per node of
+    # which only mgmt0 has a globally routable source. See k8s-1.yaml for the
+    # full reasoning and how it was verified.
+    net.ipv6.conf.enp2s0f0np0.disable_ipv6: "1"
+    net.ipv6.conf.enp87s0.disable_ipv6: "1"
   install:
     # Three NVMe drives are visible on bare metal. Select the installer target
     # by serial so it cannot land on the OSD drive or on EPHEMERAL, whatever

kubernetes/talos/patches/k8s-3.yaml

@@ -8,6 +8,15 @@ machine:
   sysctls:
     # v30iot NIC carries no host addressing — see k8s-1.yaml.
     net.ipv6.conf.ens19.disable_ipv6: "1"
+    # Same argument for the physical ports: enp2s0f0np0 is only the trunk
+    # parent for mgmt0 and ens19, enp87s0 is unused, and both sit on the
+    # untagged fallback network where nothing is supposed to run. Both were
+    # taking an fd00::/64 and an IPv6 default route from its router
+    # advertisements, leaving four equal-priority default routes per node of
+    # which only mgmt0 has a globally routable source. See k8s-1.yaml for the
+    # full reasoning and how it was verified.
+    net.ipv6.conf.enp2s0f0np0.disable_ipv6: "1"
+    net.ipv6.conf.enp87s0.disable_ipv6: "1"
   install:
     # Three NVMe drives are visible on bare metal. Select the installer target
     # by serial so it cannot land on the OSD drive or on EPHEMERAL, whatever