Compare commits
9
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
  claude-botandClaude Opus 5
|
55ea98c035 | ||
|
claude-botandClaude Opus 5
|
6a50ab1daa | ||
 k.renner
|
d1e072146f | ||
|
claude-botandClaude Opus 5
|
4678f73e9f | ||
|
k.renner
|
5f8c0115b5 | ||
|
claude-botandClaude Opus 5
|
e2f3d67f79 | ||
|
k.renner
|
7b0b664a46 | ||
|
claude-botandClaude Opus 5
|
3abdeb2203 | ||
|
k.renner
|
cdebaba2d3 |
@@ -0,0 +1,194 @@
|
||||
# Proposal — separate the four traffic classes onto their own VLANs
|
||||
|
||||
Status: draft. The question this started with is **answered** — from the Cilium
|
||||
source and flag reference, not from a test — and the answer changes the shape
|
||||
of the proposal rather than confirming it.
|
||||
|
||||
Supersedes the framing in issue #576, which described this as "give node
|
||||
management its own NIC, and Ceph public its own VLAN". That was two of four
|
||||
classes and one of them turned out already solved.
|
||||
|
||||
## The four classes
|
||||
|
||||
| class | what it is | where it goes today |
|
||||
|---|---|---|
|
||||
| **Ceph replication** | OSD-to-OSD heartbeat, replication, recovery | VLAN 49, its own X710 port — **separated** |
|
||||
| **cluster-internal** | pod-to-pod east-west, including Ceph client I/O | pod network, VXLAN over `mgmt0` |
|
||||
| **cluster-external** | ingress to LoadBalancer VIPs, egress to LAN and internet | `mgmt0` |
|
||||
| **management** | Talos API, etcd, kube-apiserver, kubelet | `mgmt0` |
|
||||
|
||||
Three of four share one interface. That is not a design decision — until the
|
||||
bare-metal conversion there was one NIC to work with.
|
||||
|
||||
Ceph client I/O living on the pod network is correct rather than a leftover:
|
||||
that traffic *is* cluster-internal. Ceph is done.
|
||||
|
||||
## Principle
|
||||
|
||||
**Each class gets its own VLAN. Which physical interface a VLAN lands on is a
|
||||
separate, changeable decision.**
|
||||
|
||||
Defining classes by VLAN means the separation survives recabling, port failures
|
||||
and capacity changes, and it does not require one port per class. Defining them
|
||||
by interface would freeze a capacity decision into the architecture.
|
||||
|
||||
## The finding that shapes everything else
|
||||
|
||||
The original open question was whether `nodePort.directRoutingDevice` moves the
|
||||
next-hop of Cilium's direct routes, or only selects an interface. It is
|
||||
answerable from the source, and the answer is: **only the interface.**
|
||||
|
||||
`installDirectRoute` takes the next-hop as a parameter:
|
||||
|
||||
```go
|
||||
func installDirectRoute(log *slog.Logger, prefix netip.Prefix, nodeIP net.IP,
|
||||
skipUnreachable bool) (routeSpec *netlink.Route, err error)
|
||||
```
|
||||
|
||||
and `nodeUpdate` supplies it as `newNode.GetNodeIP(false)`. `GetNodeIP` returns,
|
||||
in priority order, `NodeInternalIP` → `NodeExternalIP` → anything else. The
|
||||
same `newIP4` is what tunnel mode uses as the VXLAN endpoint.
|
||||
|
||||
`NodeInternalIP` comes from the Kubernetes Node object. `--ipv4-node` looks like
|
||||
an override but is not one: `initFromConfig()` calls `SetNodeInternalIP` and
|
||||
`initFromK8s()` runs **after** it and overwrites the value. In a Kubernetes
|
||||
cluster there is no path by which Cilium's node IP differs from the kubelet's
|
||||
`InternalIP`.
|
||||
|
||||
`--direct-routing-device` is documented as "Device name used to connect nodes in
|
||||
direct routing mode" — a device, not an address.
|
||||
|
||||
**So cluster-internal traffic is welded to the node IP, and the node IP is the
|
||||
kubelet InternalIP.** Two consequences:
|
||||
|
||||
- The class cannot be moved off whatever VLAN the node IP lives on.
|
||||
- **Routing mode is irrelevant to this problem.** Tunnel mode sources the VXLAN
|
||||
from the node IP; native mode uses it as the direct-route next-hop. Native
|
||||
routing buys nothing here, so the disruptive, non-rolling step this proposal
|
||||
previously contained is simply gone.
|
||||
|
||||
## Consequence: draw the line the other way
|
||||
|
||||
If cluster-internal cannot leave the node IP, then separation has to come from
|
||||
moving *management services* off it — and each one has to be checked
|
||||
individually, because they do not share a mechanism.
|
||||
|
||||
| management service | can it leave the node IP? | lever |
|
||||
|---|---|---|
|
||||
| etcd | **yes** | `cluster.etcd.advertisedSubnets` (+ `listenSubnets`, which defaults to it) |
|
||||
| kube-apiserver endpoint | **yes** | already a VIP — `Layer2VIPConfig` with a `link:` field |
|
||||
| Talos API | **n/a** | `apid` answers on every node address (measured); a client and firewall choice, not a bind setting |
|
||||
| kubelet | **no** | it *is* the node IP, by definition |
|
||||
|
||||
So four VLANs are reachable, but the boundary does not fall where the class
|
||||
names suggest: **kubelet stays with cluster-internal.** That is the honest
|
||||
result, and it is worth stating plainly rather than filing kubelet under
|
||||
management and hoping.
|
||||
|
||||
The remaining classes keep the levers established earlier:
|
||||
|
||||
- **cluster-external ingress** — `CiliumBGPNodeConfigOverride.spec.bgpInstances[].peers[].localAddress`.
|
||||
For eBGP the session source is the next-hop, so pinning the session pins
|
||||
where return traffic arrives. The session to the UDM has been running for
|
||||
months.
|
||||
- **cluster-external egress** — eBPF masquerading follows the output device, so
|
||||
this is a default-route decision plus `enable-masquerade-to-route-source`
|
||||
(today `false`).
|
||||
- **Ceph replication** — Multus macvlan on VLAN 49, done.
|
||||
|
||||
## The general escape hatch
|
||||
|
||||
Ceph is separated because its OSDs got a second interface and stopped using the
|
||||
pod network for replication. That is the *only* mechanism that fully detaches a
|
||||
workload from the node IP, and it generalises: anything that must not ride the
|
||||
node IP gets its own `NetworkAttachmentDefinition`.
|
||||
|
||||
It is also expensive per workload — an attachment, an IPAM pool, a pod spec
|
||||
change. Worth it for Ceph replication. Not worth it for east-west traffic in
|
||||
general.
|
||||
|
||||
## Recommended mapping
|
||||
|
||||
Two ways to arrange this, and they differ only in which VLAN gets relabelled:
|
||||
|
||||
**A — move the node IP to a new cluster VLAN.** VLAN 40 keeps etcd, the API VIP
|
||||
and Talos API. Cleanest match to the class names, but it changes every node's
|
||||
identity, which is the one address the whole cluster is administered through.
|
||||
|
||||
**B (recommended) — leave the node IP on VLAN 40 and move management off it.**
|
||||
VLAN 40 becomes the cluster VLAN; a new VLAN carries etcd and the API VIP.
|
||||
Identical end state, and no node ever changes its InternalIP.
|
||||
|
||||
Under B:
|
||||
|
||||
| VLAN | carries |
|
||||
|---|---|
|
||||
| 49 | Ceph replication |
|
||||
| 40 (relabelled *cluster*) | pod east-west, kubelet |
|
||||
| new *management* | etcd, kube-apiserver VIP, Talos API by convention |
|
||||
| new *external* | LoadBalancer ingress, egress default route |
|
||||
|
||||
The property the user asked for — management carries only management traffic —
|
||||
holds under B. The label moves; the property is what matters.
|
||||
|
||||
## Migration shape
|
||||
|
||||
Every step is rolling and reversible. The routing-mode change is gone.
|
||||
|
||||
1. **New VLANs** on the switch, trunked to the X710. Nothing uses them.
|
||||
2. **Addresses** on the new VLANs via Talos, one node at a time, `--mode=try`.
|
||||
Still unused; this only proves the L2 works.
|
||||
3. **etcd** to the management VLAN via `advertisedSubnets`. One node at a time,
|
||||
waiting for a healthy quorum between each.
|
||||
4. **API VIP** to the management VLAN via `Layer2VIPConfig.link`. DNS for
|
||||
`k8s.renner.uno` follows.
|
||||
5. **cluster-external**: BGP `localAddress`, default route, masquerade flag.
|
||||
Reversible by reverting the override; failure is visible immediately in
|
||||
`cilium-dbg shell -- bgp/peers`.
|
||||
6. Verify VLAN 40 by what is *absent* from it, not by what still works.
|
||||
|
||||
Step 3 is the one that deserves care — etcd is the only step where a mistake
|
||||
costs quorum rather than connectivity.
|
||||
|
||||
## What this does not buy
|
||||
|
||||
Management currently shares a 10 Gb link and is not contended. Nothing observed
|
||||
today is slow because of this. The case for the split is that traffic classes
|
||||
with different failure modes and security postures should not be
|
||||
indistinguishable — not that something is broken.
|
||||
|
||||
## Sources and measurements
|
||||
|
||||
Source, Cilium v1.20.1 (the running version):
|
||||
|
||||
- `pkg/datapath/linux/node.go` — `installDirectRoute(…, nodeIP net.IP, …)`,
|
||||
fed from `newIP4 = newNode.GetNodeIP(false)`; `createDirectRouteSpec` rejects
|
||||
a next-hop that is not directly reachable.
|
||||
- `pkg/node/types/node.go` — `GetNodeIP` prefers `NodeInternalIP`.
|
||||
- `pkg/node/sync/local_node_sync.go` — `initFromConfig()` applies
|
||||
`--ipv4-node`, `initFromK8s()` runs afterwards and overwrites it from the
|
||||
Kubernetes Node.
|
||||
- `cilium-agent` flag reference — `--direct-routing-device`: "Device name used
|
||||
to connect nodes in direct routing mode".
|
||||
|
||||
Talos:
|
||||
|
||||
- Multihoming guide — `cluster.etcd.advertisedSubnets`; `listenSubnets`
|
||||
defaults to it. kube-apiserver and Talos API listen addresses are not
|
||||
covered there.
|
||||
- `kubernetes/talos/patches/k8s-{1,2,3}.yaml` — `Layer2VIPConfig` for
|
||||
`10.2.40.60` with `link: mgmt0`.
|
||||
|
||||
Measured on the cluster:
|
||||
|
||||
- `apid` answered on a node address that is not the InternalIP, so the Talos
|
||||
API is not bound to one address.
|
||||
- k8s-1 currently holds one routable v4 address plus the VIP on `mgmt0`; its
|
||||
InternalIP had changed since the last session, which is why nothing here
|
||||
names an address as fixed.
|
||||
- Cilium runs `routing-mode: tunnel` with `enable-masquerade-to-route-source`
|
||||
unset and no direct-routing device configured.
|
||||
|
||||
Not verified: whether the kube-apiserver's `--advertise-address` follows the
|
||||
node IP independently of the VIP. It does not affect the recommendation, since
|
||||
clients reach the API through the VIP either way.
|
||||
@@ -1,10 +1,22 @@
|
||||
# Cold-rebuild runbook — homelab cluster from scratch
|
||||
|
||||
Rebuild the entire cluster from **this repo + backups only**, assuming nothing
|
||||
in the cluster survives (all VMs gone, Ceph gone, in-cluster Gitea gone).
|
||||
in the cluster survives (all three nodes wiped, Ceph gone, in-cluster Gitea
|
||||
gone).
|
||||
|
||||
Order: **tofu → talos → flux → openbao → apps.**
|
||||
|
||||
All three nodes are bare-metal MS-01 machines. Nothing provisions them: there
|
||||
are no VMs to recreate, and OpenTofu owns only DNS, object storage and UniFi.
|
||||
Step 1 is therefore two unrelated halves — a `tofu apply` for the platform
|
||||
resources, and physically booting three machines from install media.
|
||||
|
||||
**Read this first: the rebuild cannot be done remotely today.** Management is a
|
||||
tagged VLAN, and a node in maintenance mode has no VLAN configuration — it can
|
||||
only DHCP untagged, so it comes up unreachable. MS-01 has no IPMI. Every node
|
||||
therefore needs a keyboard and a monitor until #576 puts VLAN 40 native on the
|
||||
I226-V port. Plan for physical access before starting.
|
||||
|
||||
## The circular-bootstrap problem this runbook exists to break
|
||||
|
||||
In steady state Flux's `GitRepository flux-system` pulls manifests from the
|
||||
@@ -61,12 +73,20 @@ These must exist outside the cluster or the rebuild cannot complete:
|
||||
`openbao-daily` offsite copy. In a total-loss rebuild the offsite Velero
|
||||
copy is the one that survives.
|
||||
- [ ] **Tooling:** `talosctl`, `kubectl`, `flux`, `tofu`, `bao`, plus the
|
||||
OpenTofu backend + Proxmox/UniFi credentials the `opentofu/` stack needs.
|
||||
OpenTofu backend + UniFi credentials the `opentofu/` stack needs. No
|
||||
Proxmox credentials — there is no Proxmox any more.
|
||||
- [ ] **Install media** written and to hand: the Image Factory ISO named in
|
||||
Step 1, on a USB stick, for all three machines.
|
||||
- [ ] **Physical access to all three nodes** — keyboard and monitor. Not
|
||||
optional today: see the note at the top. Budget for standing in front of
|
||||
the machines, not for driving this from a desk.
|
||||
|
||||
## Step 1 — tofu: provision the VMs + supporting infra
|
||||
## Step 1a — tofu: DNS, object storage, UniFi
|
||||
|
||||
Recreate the Talos VMs and the platform resources OpenTofu owns (Proxmox VMs,
|
||||
UniFi DNS records incl. `k8s.renner.uno` / node A records, etc.).
|
||||
OpenTofu no longer provisions any node. The `proxmox_vm` module still exists in
|
||||
the tree but nothing instantiates it, so `tofu apply` here creates only the
|
||||
platform resources: UniFi DNS records including `k8s.renner.uno` and the node A
|
||||
records, the Hetzner object storage, and the NAT hairpin rules.
|
||||
|
||||
```sh
|
||||
cd opentofu
|
||||
@@ -74,44 +94,119 @@ tofu init -upgrade -backend-config="access_key=..." -backend-config="secret_key=
|
||||
tofu apply
|
||||
```
|
||||
|
||||
In CI this is `.woodpecker/deploy-infrastructure.yml` (manual, `TOFU_APPLY=true`),
|
||||
which also pre-seeds the Talos image onto every Proxmox node. Wait until the
|
||||
node FQDNs (`k8s-{1,2,3}.srv.renner.uno`) resolve and the VMs are up before
|
||||
Step 2 — `bootstrap.sh` polls DNS but will not create the records.
|
||||
Run this first regardless, because Step 2 needs `k8s.renner.uno` and the node
|
||||
FQDNs to resolve — the control-plane endpoint is baked into every generated
|
||||
config, and the nodes are addressed by name.
|
||||
|
||||
**k8s-1 is not a VM and OpenTofu does not create it.** jupiter runs Talos on
|
||||
bare metal, so this step only provisions k8s-2 (saturn) and k8s-3 (neptune).
|
||||
Before Step 2, boot jupiter from the bare-metal image below so it comes up in
|
||||
maintenance mode and answers on `k8s-1.srv.renner.uno` — `bootstrap.sh` treats
|
||||
all three nodes identically from there, since `apply-config --insecure` does
|
||||
not care whether the target is a VM or bare metal.
|
||||
## Step 1b — boot all three machines from install media
|
||||
|
||||
```
|
||||
https://factory.talos.dev/image/0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9/v1.12.11/metal-amd64.iso
|
||||
https://factory.talos.dev/image/0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9/v1.13.9/metal-amd64.iso
|
||||
```
|
||||
|
||||
That schematic is deliberately not the one `.woodpecker/deploy-infrastructure.yml`
|
||||
seeds onto the Proxmox nodes. It keeps the same `net.ifnames=1` kernel argument
|
||||
— without it the predictable interface names that `talos/patches/k8s-1.yaml`
|
||||
matches its VLAN parents against do not exist — and differs in three ways:
|
||||
Write it to a USB stick and boot each MS-01 from it. Each comes up in
|
||||
maintenance mode with a DHCP address on the **untagged** network — not on
|
||||
management, because that is a tagged VLAN a maintenance-mode node does not
|
||||
configure. Read the address off the console; that is what Step 2 talks to.
|
||||
|
||||
- drops `siderolabs/qemu-guest-agent`, which has nothing to talk to on bare metal
|
||||
- adds `siderolabs/intel-ucode`, since no hypervisor is supplying microcode now
|
||||
- adds `siderolabs/thunderbolt`, unused so far and inert until
|
||||
`machine.kernel.modules` requests `thunderbolt`/`thunderbolt_net`. It is
|
||||
present only so enabling the MS-01 Thunderbolt mesh later does not require a
|
||||
new image and a reinstall. Note its udev rule auto-authorises any Thunderbolt
|
||||
device attached to the machine once those modules are loaded.
|
||||
The schematic `0f6fb8bd…` carries `net.ifnames=1` — without it the predictable
|
||||
interface names that `talos/patches/k8s-{1,2,3}.yaml` match their VLAN parents
|
||||
against do not exist, and the node ends up with no address on any interface.
|
||||
Beyond that it carries:
|
||||
|
||||
- `siderolabs/intel-ucode` — no hypervisor supplies microcode here
|
||||
- `siderolabs/i915` — the Alder Lake-P Iris Xe. Nothing requests `/dev/dri`
|
||||
today; present so wanting hardware transcoding later is a config change
|
||||
rather than another rolling reinstall
|
||||
- `siderolabs/thunderbolt` — inert until `machine.kernel.modules` requests
|
||||
`thunderbolt`/`thunderbolt_net`, present for the same reason. Note its udev
|
||||
rule auto-authorises any Thunderbolt device attached once those modules load
|
||||
- no `qemu-guest-agent`, which has nothing to talk to on bare metal
|
||||
|
||||
Verify the disks before going further. Talos enumerates NVMe in an order that
|
||||
is not stable across boots, and the per-node configs select by serial for
|
||||
exactly that reason — the failure mode is the installer taking the Ceph OSD
|
||||
drive. `talosctl -n <maintenance-ip> --insecure get disks` should show three
|
||||
NVMe devices whose serials match the `diskSelector` and `VolumeConfig` entries
|
||||
in `talos/patches/k8s-N.yaml`.
|
||||
|
||||
## Step 2 — talos: form the cluster + get kubeconfig
|
||||
|
||||
**Do not use `kubernetes/bootstrap.sh talos`.** That phase cannot generate a
|
||||
config: it passes `talos/patches/cilium-cni.yaml` whole, and that file's
|
||||
`$patch: delete` document fails against a base with no `inlineManifests`
|
||||
(#581). Its other phases — `flux`, `wait-apps` — are still used, in Steps 3, 5
|
||||
and 6; only config generation is broken.
|
||||
|
||||
The procedure below is what actually works, and is the same one
|
||||
`RUNBOOK-baremetal-conversion.md` uses.
|
||||
|
||||
### Generate the configs
|
||||
|
||||
This is a genuinely new cluster — nothing survives to copy secrets from, so
|
||||
generate them fresh. (In a *partial* rebuild, where one node still runs, pull
|
||||
secrets off it instead with `talosctl get mc v1alpha1` + `gen secrets
|
||||
--from-controlplane-config`, or the new node forms its own cluster rather than
|
||||
joining.)
|
||||
|
||||
```sh
|
||||
kubernetes/bootstrap.sh talos
|
||||
talosctl gen secrets -o secrets.yaml
|
||||
|
||||
for N in 1 2 3; do
|
||||
talosctl gen config homelab https://k8s.renner.uno:6443 \
|
||||
--with-secrets secrets.yaml --output-types controlplane \
|
||||
--config-patch @talos/patches/cluster.yaml \
|
||||
--config-patch @<cilium additive half> \
|
||||
--config-patch @talos/patches/registry-mirror.yaml \
|
||||
--config-patch @talos/patches/k8s-$N.yaml -o install-k8s-$N.yaml
|
||||
done
|
||||
```
|
||||
|
||||
Generates + applies Talos configs to `k8s-{1,2,3}`, bootstraps etcd/Kubernetes,
|
||||
writes `~/.talos/config` + `~/.kube/config`, and stages both into
|
||||
`kubernetes/access-staging/` for later phases. Ends when all 3 nodes are `Ready`.
|
||||
Two things the generator gets wrong for this repo, both of which will bite
|
||||
silently:
|
||||
|
||||
- **`cilium-cni.yaml` cannot be passed whole** — extract only its additive
|
||||
second document. See issue #581.
|
||||
- **Strip the generated `HostnameConfig`** — it collides with
|
||||
`machine.network.hostname` from the per-node patch.
|
||||
|
||||
Validate each before applying:
|
||||
|
||||
```sh
|
||||
talosctl validate --config install-k8s-$N.yaml --mode metal
|
||||
```
|
||||
|
||||
### Apply in two stages
|
||||
|
||||
Do **not** apply the full per-node config in maintenance mode. A document that
|
||||
references an interface which does not exist yet hangs the install: the node
|
||||
sits at `STAGE: Installing` indefinitely with `KUBELET n/a` and `APISERVER n/a`,
|
||||
and nothing times out. Concretely, the `ens19` VLAN document must be removed
|
||||
from the first-stage config — its parent exists, but the install stalls anyway.
|
||||
|
||||
```sh
|
||||
# Stage 1: minimal config, from the maintenance-mode address off the console
|
||||
talosctl apply-config --insecure -n <maintenance-ip> --file install-k8s-$N-stage1.yaml
|
||||
# Stage 2: once the node is up on management, apply the complete config
|
||||
talosctl -n k8s-$N.srv.renner.uno apply-config --mode=try --timeout=5m --file install-k8s-$N.yaml
|
||||
```
|
||||
|
||||
`--mode=try` rolls back on its own if the node becomes unreachable, which is the
|
||||
only safety net there is for a network change that can strand a machine with no
|
||||
IPMI. Use it for every network-affecting apply, then make it permanent once
|
||||
proven.
|
||||
|
||||
### Bootstrap etcd, then collect access
|
||||
|
||||
Bootstrap **exactly one** node:
|
||||
|
||||
```sh
|
||||
talosctl -n k8s-1.srv.renner.uno bootstrap
|
||||
talosctl -n k8s-1.srv.renner.uno kubeconfig .
|
||||
```
|
||||
|
||||
Ends when all three nodes are `Ready`. `scripts/import-cluster-access.sh` is the
|
||||
supported way to get the kubeconfig into place afterwards.
|
||||
|
||||
## Step 3 — flux: bootstrap GitOps **from the GitHub mirror** (break-glass)
|
||||
|
||||
@@ -231,9 +326,28 @@ it: repo → **Settings → Repository → Mirror Settings** in Gitea, or
|
||||
in-cluster commits made **after** Flux/Gitea broke are not on the mirror
|
||||
(Gitea couldn't push them if Gitea itself was down). This is the intended
|
||||
recovery point, but be aware of it when reconciling drift.
|
||||
- `bootstrap.sh` has no `openbao` phase — Step 4 is manual because the unseal
|
||||
key must not live in CI. See issue #187 for the open question of whether to
|
||||
wire a semi-automated OpenBao restore.
|
||||
- **The rebuild cannot be driven remotely.** Management is a tagged VLAN and a
|
||||
maintenance-mode node cannot configure VLANs, so a freshly booted machine is
|
||||
unreachable until its config is applied — and MS-01 has no IPMI. Every node
|
||||
needs a keyboard and a monitor. #576 fixes this by putting VLAN 40 native on
|
||||
the I226-V port; until then, physical access is a hard prerequisite rather
|
||||
than a convenience.
|
||||
- **`bootstrap.sh talos` is dead** — it cannot generate a config (#581), and
|
||||
Step 2 no longer uses it. Its `flux` and `wait-apps` phases are still used in
|
||||
Steps 3, 5 and 6 and work; do not read this as the whole script being gone.
|
||||
It has also never had an `openbao` phase — Step 4 is manual because the
|
||||
unseal key must not live in CI; see issue #187 on whether to wire a
|
||||
semi-automated OpenBao restore.
|
||||
- **OpenBao seals on every restart and each replica must be unsealed
|
||||
individually.** Three raft replicas mean the service survives losing one, but
|
||||
a sealed server is neither a voter nor a standby — unsealing only one leaves
|
||||
the cluster short of quorum. Step 4 covers this; it is repeated here because
|
||||
it is the step most likely to be half-done under pressure.
|
||||
- Ceph's cluster network is a Multus attachment on VLAN 49 with whereabouts
|
||||
IPAM, not host networking. Nothing in the rebuild needs special handling for
|
||||
it — the nodes hold no address on that segment — but a rebuilt cluster will
|
||||
not have the whereabouts allocations from before, which is harmless since
|
||||
they are assigned fresh.
|
||||
- warn-and-continue waits in the `flux` phase can mask a genuinely broken
|
||||
Ceph/infra bring-up; always read the phase output rather than trusting the
|
||||
exit code.
|
||||
|
||||
@@ -40,17 +40,31 @@ spec:
|
||||
description: "A Velero backup run for {{ $labels.schedule }} completed with Failed or PartiallyFailed status in the last hour. Inspect with `kubectl -n velero get backup.velero.io -l velero.io/schedule-name={{ $labels.schedule }}` and the velero pod logs."
|
||||
|
||||
- alert: VeleroBackupStale
|
||||
# Daily schedule with no successful backup in over 30h: covers
|
||||
# the case where multiple consecutive runs failed (Garage flapping
|
||||
# over the backup window) without the per-event alerts being
|
||||
# acted on.
|
||||
# Daily schedule with no successful backup in over 60h: covers the
|
||||
# case where consecutive runs failed without the per-event alerts
|
||||
# being acted on.
|
||||
#
|
||||
# 60h, not 30h. At 30h a single missed cycle pages, and a single
|
||||
# missed cycle is the normal cost of touching the nodes: Velero moves
|
||||
# data through pods, so draining a node kills whatever upload is in
|
||||
# flight and the schedule is simply lost for that day — it is not
|
||||
# retried. Rolling node work between 03:15 and 08:45 UTC therefore
|
||||
# produced a guaranteed critical alert for every schedule it
|
||||
# overlapped, which is noise about planned work rather than a signal
|
||||
# about backups.
|
||||
#
|
||||
# 60h still catches what this is for. Two consecutive failures is no
|
||||
# longer explicable by one maintenance window, and the per-event
|
||||
# VeleroBackupFailed alert fires on the first failure regardless — so
|
||||
# nothing is hidden, only the escalation to critical is deferred
|
||||
# until a second cycle confirms it.
|
||||
expr: |
|
||||
time() - velero_backup_last_successful_timestamp{schedule!=""} > 30 * 3600
|
||||
time() - velero_backup_last_successful_timestamp{schedule!=""} > 60 * 3600
|
||||
and
|
||||
velero_backup_last_successful_timestamp{schedule!=""} > 0
|
||||
for: 10m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "Velero {{ $labels.schedule }} has no successful backup in 30h+"
|
||||
description: "Last successful backup for {{ $labels.schedule }} was over 30 hours ago. With a daily schedule that means at least one full cycle missed. RPO is degrading."
|
||||
summary: "Velero {{ $labels.schedule }} has no successful backup in 60h+"
|
||||
description: "Last successful backup for {{ $labels.schedule }} was over 60 hours ago. With a daily schedule that is at least two consecutive cycles missed, which one maintenance window does not explain. RPO is degrading."
|
||||
|
||||
@@ -42,8 +42,17 @@ machine:
|
||||
# Only read at install and upgrade time. Changing it here does nothing to a
|
||||
# running node until that node is upgraded. Bump the tag with the Talos
|
||||
# version — it must match the running version, or the upgrade is a downgrade.
|
||||
#
|
||||
# Upgrades follow the documented path: migration is only tested between
|
||||
# adjacent minor releases, so go to the latest patch of each intermediate
|
||||
# minor rather than skipping. 1.12.11 -> 1.13.9 satisfies that — one minor
|
||||
# step from the last 1.12 patch, no intermediate needed.
|
||||
#
|
||||
# Kubernetes is a separate decision and is not forced by this. Talos 1.13
|
||||
# supports Kubernetes 1.36, 1.35, 1.34, 1.33, 1.32 and 1.31, so the cluster
|
||||
# stays on 1.35.2 until `talosctl upgrade-k8s` moves it deliberately.
|
||||
install:
|
||||
image: factory.talos.dev/metal-installer/0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9:v1.12.11
|
||||
image: factory.talos.dev/metal-installer/0f6fb8bd8bb53013e9743b7c7b099d96fb9c32fd2464ced32693c2dc0ea5feb9:v1.13.9
|
||||
features:
|
||||
rbac: true
|
||||
stableHostname: true
|
||||
|
||||
Reference in New Issue
Block a user