fix: [alert] VeleroBackupFailed (velero) (#657)

kubernetes/PROPOSAL-traffic-class-separation.md

@@ -1,194 +0,0 @@
-# Proposal — separate the four traffic classes onto their own VLANs
-
-Status: draft. The question this started with is **answered** — from the Cilium
-source and flag reference, not from a test — and the answer changes the shape
-of the proposal rather than confirming it.
-
-Supersedes the framing in issue #576, which described this as "give node
-management its own NIC, and Ceph public its own VLAN". That was two of four
-classes and one of them turned out already solved.
-
-## The four classes
-
-| class | what it is | where it goes today |
-|---|---|---|
-| **Ceph replication** | OSD-to-OSD heartbeat, replication, recovery | VLAN 49, its own X710 port — **separated** |
-| **cluster-internal** | pod-to-pod east-west, including Ceph client I/O | pod network, VXLAN over `mgmt0` |
-| **cluster-external** | ingress to LoadBalancer VIPs, egress to LAN and internet | `mgmt0` |
-| **management** | Talos API, etcd, kube-apiserver, kubelet | `mgmt0` |
-
-Three of four share one interface. That is not a design decision — until the
-bare-metal conversion there was one NIC to work with.
-
-Ceph client I/O living on the pod network is correct rather than a leftover:
-that traffic *is* cluster-internal. Ceph is done.
-
-## Principle
-
-**Each class gets its own VLAN. Which physical interface a VLAN lands on is a
-separate, changeable decision.**
-
-Defining classes by VLAN means the separation survives recabling, port failures
-and capacity changes, and it does not require one port per class. Defining them
-by interface would freeze a capacity decision into the architecture.
-
-## The finding that shapes everything else
-
-The original open question was whether `nodePort.directRoutingDevice` moves the
-next-hop of Cilium's direct routes, or only selects an interface. It is
-answerable from the source, and the answer is: **only the interface.**
-
-`installDirectRoute` takes the next-hop as a parameter:
-
-```go
-func installDirectRoute(log *slog.Logger, prefix netip.Prefix, nodeIP net.IP,
-    skipUnreachable bool) (routeSpec *netlink.Route, err error)
-```
-
-and `nodeUpdate` supplies it as `newNode.GetNodeIP(false)`. `GetNodeIP` returns,
-in priority order, `NodeInternalIP` → `NodeExternalIP` → anything else. The
-same `newIP4` is what tunnel mode uses as the VXLAN endpoint.
-
-`NodeInternalIP` comes from the Kubernetes Node object. `--ipv4-node` looks like
-an override but is not one: `initFromConfig()` calls `SetNodeInternalIP` and
-`initFromK8s()` runs **after** it and overwrites the value. In a Kubernetes
-cluster there is no path by which Cilium's node IP differs from the kubelet's
-`InternalIP`.
-
-`--direct-routing-device` is documented as "Device name used to connect nodes in
-direct routing mode" — a device, not an address.
-
-**So cluster-internal traffic is welded to the node IP, and the node IP is the
-kubelet InternalIP.** Two consequences:
-
-- The class cannot be moved off whatever VLAN the node IP lives on.
-- **Routing mode is irrelevant to this problem.** Tunnel mode sources the VXLAN
-  from the node IP; native mode uses it as the direct-route next-hop. Native
-  routing buys nothing here, so the disruptive, non-rolling step this proposal
-  previously contained is simply gone.
-
-## Consequence: draw the line the other way
-
-If cluster-internal cannot leave the node IP, then separation has to come from
-moving *management services* off it — and each one has to be checked
-individually, because they do not share a mechanism.
-
-| management service | can it leave the node IP? | lever |
-|---|---|---|
-| etcd | **yes** | `cluster.etcd.advertisedSubnets` (+ `listenSubnets`, which defaults to it) |
-| kube-apiserver endpoint | **yes** | already a VIP — `Layer2VIPConfig` with a `link:` field |
-| Talos API | **n/a** | `apid` answers on every node address (measured); a client and firewall choice, not a bind setting |
-| kubelet | **no** | it *is* the node IP, by definition |
-
-So four VLANs are reachable, but the boundary does not fall where the class
-names suggest: **kubelet stays with cluster-internal.** That is the honest
-result, and it is worth stating plainly rather than filing kubelet under
-management and hoping.
-
-The remaining classes keep the levers established earlier:
-
-- **cluster-external ingress** — `CiliumBGPNodeConfigOverride.spec.bgpInstances[].peers[].localAddress`.
-  For eBGP the session source is the next-hop, so pinning the session pins
-  where return traffic arrives. The session to the UDM has been running for
-  months.
-- **cluster-external egress** — eBPF masquerading follows the output device, so
-  this is a default-route decision plus `enable-masquerade-to-route-source`
-  (today `false`).
-- **Ceph replication** — Multus macvlan on VLAN 49, done.
-
-## The general escape hatch
-
-Ceph is separated because its OSDs got a second interface and stopped using the
-pod network for replication. That is the *only* mechanism that fully detaches a
-workload from the node IP, and it generalises: anything that must not ride the
-node IP gets its own `NetworkAttachmentDefinition`.
-
-It is also expensive per workload — an attachment, an IPAM pool, a pod spec
-change. Worth it for Ceph replication. Not worth it for east-west traffic in
-general.
-
-## Recommended mapping
-
-Two ways to arrange this, and they differ only in which VLAN gets relabelled:
-
-**A — move the node IP to a new cluster VLAN.** VLAN 40 keeps etcd, the API VIP
-and Talos API. Cleanest match to the class names, but it changes every node's
-identity, which is the one address the whole cluster is administered through.
-
-**B (recommended) — leave the node IP on VLAN 40 and move management off it.**
-VLAN 40 becomes the cluster VLAN; a new VLAN carries etcd and the API VIP.
-Identical end state, and no node ever changes its InternalIP.
-
-Under B:
-
-| VLAN | carries |
-|---|---|
-| 49 | Ceph replication |
-| 40 (relabelled *cluster*) | pod east-west, kubelet |
-| new *management* | etcd, kube-apiserver VIP, Talos API by convention |
-| new *external* | LoadBalancer ingress, egress default route |
-
-The property the user asked for — management carries only management traffic —
-holds under B. The label moves; the property is what matters.
-
-## Migration shape
-
-Every step is rolling and reversible. The routing-mode change is gone.
-
-1. **New VLANs** on the switch, trunked to the X710. Nothing uses them.
-2. **Addresses** on the new VLANs via Talos, one node at a time, `--mode=try`.
-   Still unused; this only proves the L2 works.
-3. **etcd** to the management VLAN via `advertisedSubnets`. One node at a time,
-   waiting for a healthy quorum between each.
-4. **API VIP** to the management VLAN via `Layer2VIPConfig.link`. DNS for
-   `k8s.renner.uno` follows.
-5. **cluster-external**: BGP `localAddress`, default route, masquerade flag.
-   Reversible by reverting the override; failure is visible immediately in
-   `cilium-dbg shell -- bgp/peers`.
-6. Verify VLAN 40 by what is *absent* from it, not by what still works.
-
-Step 3 is the one that deserves care — etcd is the only step where a mistake
-costs quorum rather than connectivity.
-
-## What this does not buy
-
-Management currently shares a 10 Gb link and is not contended. Nothing observed
-today is slow because of this. The case for the split is that traffic classes
-with different failure modes and security postures should not be
-indistinguishable — not that something is broken.
-
-## Sources and measurements
-
-Source, Cilium v1.20.1 (the running version):
-
-- `pkg/datapath/linux/node.go` — `installDirectRoute(…, nodeIP net.IP, …)`,
-  fed from `newIP4 = newNode.GetNodeIP(false)`; `createDirectRouteSpec` rejects
-  a next-hop that is not directly reachable.
-- `pkg/node/types/node.go` — `GetNodeIP` prefers `NodeInternalIP`.
-- `pkg/node/sync/local_node_sync.go` — `initFromConfig()` applies
-  `--ipv4-node`, `initFromK8s()` runs afterwards and overwrites it from the
-  Kubernetes Node.
-- `cilium-agent` flag reference — `--direct-routing-device`: "Device name used
-  to connect nodes in direct routing mode".
-
-Talos:
-
-- Multihoming guide — `cluster.etcd.advertisedSubnets`; `listenSubnets`
-  defaults to it. kube-apiserver and Talos API listen addresses are not
-  covered there.
-- `kubernetes/talos/patches/k8s-{1,2,3}.yaml` — `Layer2VIPConfig` for
-  `10.2.40.60` with `link: mgmt0`.
-
-Measured on the cluster:
-
-- `apid` answered on a node address that is not the InternalIP, so the Talos
-  API is not bound to one address.
-- k8s-1 currently holds one routable v4 address plus the VIP on `mgmt0`; its
-  InternalIP had changed since the last session, which is why nothing here
-  names an address as fixed.
-- Cilium runs `routing-mode: tunnel` with `enable-masquerade-to-route-source`
-  unset and no direct-routing device configured.
-
-Not verified: whether the kube-apiserver's `--advertise-address` follows the
-node IP independently of the VIP. It does not affect the recommendation, since
-clients reach the API through the VIP either way.

kubernetes/RUNBOOK-cold-rebuild.md

@@ -1,22 +1,10 @@
 # Cold-rebuild runbook — homelab cluster from scratch
 
 Rebuild the entire cluster from **this repo + backups only**, assuming nothing
-in the cluster survives (all three nodes wiped, Ceph gone, in-cluster Gitea
-gone).
+in the cluster survives (all VMs gone, Ceph gone, in-cluster Gitea gone).
 
 Order: **tofu → talos → flux → openbao → apps.**
 
-All three nodes are bare-metal MS-01 machines. Nothing provisions them: there
-are no VMs to recreate, and OpenTofu owns only DNS, object storage and UniFi.
-Step 1 is therefore two unrelated halves — a `tofu apply` for the platform
-resources, and physically booting three machines from install media.
-
-**Read this first: the rebuild cannot be done remotely today.** Management is a
-tagged VLAN, and a node in maintenance mode has no VLAN configuration — it can
-only DHCP untagged, so it comes up unreachable. MS-01 has no IPMI. Every node
-therefore needs a keyboard and a monitor until #576 puts VLAN 40 native on the
-I226-V port. Plan for physical access before starting.
-
 ## The circular-bootstrap problem this runbook exists to break
 
 In steady state Flux's `GitRepository flux-system` pulls manifests from the
@@ -73,20 +61,12 @@ These must exist outside the cluster or the rebuild cannot complete:
       `openbao-daily` offsite copy. In a total-loss rebuild the offsite Velero
       copy is the one that survives.
 - [ ] **Tooling:** `talosctl`, `kubectl`, `flux`, `tofu`, `bao`, plus the
-      OpenTofu backend + UniFi credentials the `opentofu/` stack needs. No
-      Proxmox credentials — there is no Proxmox any more.
-- [ ] **Install media** written and to hand: the Image Factory ISO named in
-      Step 1, on a USB stick, for all three machines.
-- [ ] **Physical access to all three nodes** — keyboard and monitor. Not
-      optional today: see the note at the top. Budget for standing in front of
-      the machines, not for driving this from a desk.
+      OpenTofu backend + Proxmox/UniFi credentials the `opentofu/` stack needs.
 
-## Step 1a — tofu: DNS, object storage, UniFi
+## Step 1 — tofu: provision the VMs + supporting infra
 
-OpenTofu no longer provisions any node. The `proxmox_vm` module still exists in
-the tree but nothing instantiates it, so `tofu apply` here creates only the
-platform resources: UniFi DNS records including `k8s.renner.uno` and the node A
-records, the Hetzner object storage, and the NAT hairpin rules.
+Recreate the Talos VMs and the platform resources OpenTofu owns (Proxmox VMs,
+UniFi DNS records incl. `k8s.renner.uno` / node A records, etc.).
 
 ```sh
 cd opentofu
@@ -94,119 +74,44 @@ tofu init -upgrade -backend-config="access_key=..." -backend-config="secret_key=
 tofu apply
 ```
 
-Run this first regardless, because Step 2 needs `k8s.renner.uno` and the node
-FQDNs to resolve — the control-plane endpoint is baked into every generated
-config, and the nodes are addressed by name.
+In CI this is `.woodpecker/deploy-infrastructure.yml` (manual, `TOFU_APPLY=true`),
+which also pre-seeds the Talos image onto every Proxmox node. Wait until the
+node FQDNs (`k8s-{1,2,3}.srv.renner.uno`) resolve and the VMs are up before
+Step 2 — `bootstrap.sh` polls DNS but will not create the records.
 
-## Step 1b — boot all three machines from install media
+**k8s-1 is not a VM and OpenTofu does not create it.** jupiter runs Talos on
+bare metal, so this step only provisions k8s-2 (saturn) and k8s-3 (neptune).
+Before Step 2, boot jupiter from the bare-metal image below so it comes up in
+maintenance mode and answers on `k8s-1.srv.renner.uno` — `bootstrap.sh` treats
+all three nodes identically from there, since `apply-config --insecure` does
+not care whether the target is a VM or bare metal.
 
 ```
 https://factory.talos.dev/image/0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9/v1.13.9/metal-amd64.iso
 ```
 
-Write it to a USB stick and boot each MS-01 from it. Each comes up in
-maintenance mode with a DHCP address on the **untagged** network — not on
-management, because that is a tagged VLAN a maintenance-mode node does not
-configure. Read the address off the console; that is what Step 2 talks to.
+That schematic is deliberately not the one `.woodpecker/deploy-infrastructure.yml`
+seeds onto the Proxmox nodes. It keeps the same `net.ifnames=1` kernel argument
+— without it the predictable interface names that `talos/patches/k8s-1.yaml`
+matches its VLAN parents against do not exist — and differs in three ways:
 
-The schematic `0f6fb8bd…` carries `net.ifnames=1` — without it the predictable
-interface names that `talos/patches/k8s-{1,2,3}.yaml` match their VLAN parents
-against do not exist, and the node ends up with no address on any interface.
-Beyond that it carries:
-
-- `siderolabs/intel-ucode` — no hypervisor supplies microcode here
-- `siderolabs/i915` — the Alder Lake-P Iris Xe. Nothing requests `/dev/dri`
-  today; present so wanting hardware transcoding later is a config change
-  rather than another rolling reinstall
-- `siderolabs/thunderbolt` — inert until `machine.kernel.modules` requests
-  `thunderbolt`/`thunderbolt_net`, present for the same reason. Note its udev
-  rule auto-authorises any Thunderbolt device attached once those modules load
-- no `qemu-guest-agent`, which has nothing to talk to on bare metal
-
-Verify the disks before going further. Talos enumerates NVMe in an order that
-is not stable across boots, and the per-node configs select by serial for
-exactly that reason — the failure mode is the installer taking the Ceph OSD
-drive. `talosctl -n <maintenance-ip> --insecure get disks` should show three
-NVMe devices whose serials match the `diskSelector` and `VolumeConfig` entries
-in `talos/patches/k8s-N.yaml`.
+- drops `siderolabs/qemu-guest-agent`, which has nothing to talk to on bare metal
+- adds `siderolabs/intel-ucode`, since no hypervisor is supplying microcode now
+- adds `siderolabs/thunderbolt`, unused so far and inert until
+  `machine.kernel.modules` requests `thunderbolt`/`thunderbolt_net`. It is
+  present only so enabling the MS-01 Thunderbolt mesh later does not require a
+  new image and a reinstall. Note its udev rule auto-authorises any Thunderbolt
+  device attached to the machine once those modules are loaded.
 
 ## Step 2 — talos: form the cluster + get kubeconfig
 
-**Do not use `kubernetes/bootstrap.sh talos`.** That phase cannot generate a
-config: it passes `talos/patches/cilium-cni.yaml` whole, and that file's
-`$patch: delete` document fails against a base with no `inlineManifests`
-(#581). Its other phases — `flux`, `wait-apps` — are still used, in Steps 3, 5
-and 6; only config generation is broken.
-
-The procedure below is what actually works, and is the same one
-`RUNBOOK-baremetal-conversion.md` uses.
-
-### Generate the configs
-
-This is a genuinely new cluster — nothing survives to copy secrets from, so
-generate them fresh. (In a *partial* rebuild, where one node still runs, pull
-secrets off it instead with `talosctl get mc v1alpha1` + `gen secrets
---from-controlplane-config`, or the new node forms its own cluster rather than
-joining.)
-
 ```sh
-talosctl gen secrets -o secrets.yaml
-
-for N in 1 2 3; do
-  talosctl gen config homelab https://k8s.renner.uno:6443 \
-    --with-secrets secrets.yaml --output-types controlplane \
-    --config-patch @talos/patches/cluster.yaml \
-    --config-patch @<cilium additive half> \
-    --config-patch @talos/patches/registry-mirror.yaml \
-    --config-patch @talos/patches/k8s-$N.yaml -o install-k8s-$N.yaml
-done
+kubernetes/bootstrap.sh talos
 ```
 
-Two things the generator gets wrong for this repo, both of which will bite
-silently:
-
-- **`cilium-cni.yaml` cannot be passed whole** — extract only its additive
-  second document. See issue #581.
-- **Strip the generated `HostnameConfig`** — it collides with
-  `machine.network.hostname` from the per-node patch.
-
-Validate each before applying:
-
-```sh
-talosctl validate --config install-k8s-$N.yaml --mode metal
-```
-
-### Apply in two stages
-
-Do **not** apply the full per-node config in maintenance mode. A document that
-references an interface which does not exist yet hangs the install: the node
-sits at `STAGE: Installing` indefinitely with `KUBELET n/a` and `APISERVER n/a`,
-and nothing times out. Concretely, the `ens19` VLAN document must be removed
-from the first-stage config — its parent exists, but the install stalls anyway.
-
-```sh
-# Stage 1: minimal config, from the maintenance-mode address off the console
-talosctl apply-config --insecure -n <maintenance-ip> --file install-k8s-$N-stage1.yaml
-# Stage 2: once the node is up on management, apply the complete config
-talosctl -n k8s-$N.srv.renner.uno apply-config --mode=try --timeout=5m --file install-k8s-$N.yaml
-```
-
-`--mode=try` rolls back on its own if the node becomes unreachable, which is the
-only safety net there is for a network change that can strand a machine with no
-IPMI. Use it for every network-affecting apply, then make it permanent once
-proven.
-
-### Bootstrap etcd, then collect access
-
-Bootstrap **exactly one** node:
-
-```sh
-talosctl -n k8s-1.srv.renner.uno bootstrap
-talosctl -n k8s-1.srv.renner.uno kubeconfig .
-```
-
-Ends when all three nodes are `Ready`. `scripts/import-cluster-access.sh` is the
-supported way to get the kubeconfig into place afterwards.
+Generates + applies Talos configs to `k8s-{1,2,3}`, bootstraps etcd/Kubernetes,
+writes `~/.talos/config` + `~/.kube/config`, and stages both into
+`kubernetes/access-staging/` for later phases. Ends when all 3 nodes are `Ready`.
 
 ## Step 3 — flux: bootstrap GitOps **from the GitHub mirror** (break-glass)
 
@@ -326,28 +231,9 @@ it: repo → **Settings → Repository → Mirror Settings** in Gitea, or
   in-cluster commits made **after** Flux/Gitea broke are not on the mirror
   (Gitea couldn't push them if Gitea itself was down). This is the intended
   recovery point, but be aware of it when reconciling drift.
-- **The rebuild cannot be driven remotely.** Management is a tagged VLAN and a
-  maintenance-mode node cannot configure VLANs, so a freshly booted machine is
-  unreachable until its config is applied — and MS-01 has no IPMI. Every node
-  needs a keyboard and a monitor. #576 fixes this by putting VLAN 40 native on
-  the I226-V port; until then, physical access is a hard prerequisite rather
-  than a convenience.
-- **`bootstrap.sh talos` is dead** — it cannot generate a config (#581), and
-  Step 2 no longer uses it. Its `flux` and `wait-apps` phases are still used in
-  Steps 3, 5 and 6 and work; do not read this as the whole script being gone.
-  It has also never had an `openbao` phase — Step 4 is manual because the
-  unseal key must not live in CI; see issue #187 on whether to wire a
-  semi-automated OpenBao restore.
-- **OpenBao seals on every restart and each replica must be unsealed
-  individually.** Three raft replicas mean the service survives losing one, but
-  a sealed server is neither a voter nor a standby — unsealing only one leaves
-  the cluster short of quorum. Step 4 covers this; it is repeated here because
-  it is the step most likely to be half-done under pressure.
-- Ceph's cluster network is a Multus attachment on VLAN 49 with whereabouts
-  IPAM, not host networking. Nothing in the rebuild needs special handling for
-  it — the nodes hold no address on that segment — but a rebuilt cluster will
-  not have the whereabouts allocations from before, which is harmless since
-  they are assigned fresh.
+- `bootstrap.sh` has no `openbao` phase — Step 4 is manual because the unseal
+  key must not live in CI. See issue #187 for the open question of whether to
+  wire a semi-automated OpenBao restore.
 - warn-and-continue waits in the `flux` phase can mask a genuinely broken
   Ceph/infra bring-up; always read the phase output rather than trusting the
   exit code.

kubernetes/infrastructure/post-crds/velero-schedules.yaml

@@ -203,16 +203,16 @@ metadata:
   labels:
     app.kubernetes.io/part-of: homelab-backup
 spec:
-  # Daily at 06:45 local (next free stagger slot). All nine PVCs live
-  # in the home-assistant namespace regardless of which app/ dir owns
+  # Daily at 06:45 local (next free stagger slot). The app PVCs live in
+  # the home-assistant namespace regardless of which app/ dir owns
   # their manifest (matter-server, otbr, whisper, speech-to-phrase,
   # zone-configurator all set `namespace: home-assistant` via their
   # Kustomization, not per-resource): home-assistant-config (.storage
   # entity registry, recorder DB, secrets.yaml), esphome-data,
   # shelly-manager-data, matter-server-data, otbr-data, whisper-data,
-  # speech-to-phrase-models, speech-to-phrase-train,
-  # zone-configurator-config. No CNPG cluster in this namespace, so no
-  # label exclusion is needed. On restore, home-assistant-config has its
+  # speech-to-phrase-models, speech-to-phrase-train, zigbee2mqtt-data,
+  # zwavejs-store, zone-configurator-config. On restore,
+  # home-assistant-config has its
   # own path (the HA pod's `seed` initContainer pulling the native HAOS
   # backup from Garage, see the home-assistant Kustomization comment in
   # kubernetes/clusters/homelab/home-assistant.yaml); layering a Velero
@@ -229,6 +229,20 @@ spec:
     storageLocation: default
     ttl: 720h0m0s
     snapshotMoveData: true
+    # The ha-pg CNPG cluster now runs in this namespace (it did not when
+    # this schedule was written — the old comment here still claimed
+    # "no CNPG cluster in this namespace"). Its PVC ha-pg-1 carries the
+    # cnpg.io/cluster label and was being swept in: a crash-consistent
+    # snapshot of a running Postgres data directory is not a restorable
+    # backup, the same trap the wordpress-daily comment below describes.
+    # ha-pg is covered by Barman — daily base backup at 02:50 UTC
+    # (kubernetes/apps/home-assistant/ha-pg-scheduled-backup.yaml) plus
+    # continuous WAL archiving — so this exclusion removes a redundant
+    # 20 GiB upload, matching paperless/authentik/immich/trmnl above.
+    labelSelector:
+      matchExpressions:
+        - key: cnpg.io/cluster
+          operator: DoesNotExist
 ---
 apiVersion: velero.io/v1
 kind: Schedule