At 30h a single missed cycle pages critical, and a single missed cycle is the
normal cost of touching the nodes. Velero moves data through pods, so draining
a node kills whatever upload is in flight and the schedule is lost for that day
— it is not retried. Backup windows run 03:15 to 08:45 UTC, so any rolling node
work overlapping them produced a guaranteed critical alert per affected
schedule.
That is noise about planned work, not signal about backups. It fired for eleven
schedules during the bare-metal Talos upgrade and again today.
60h still catches what the rule exists for. Two consecutive failures is no
longer explicable by one maintenance window. Nothing is hidden in the meantime:
VeleroBackupFailed fires on the first failure regardless, so the first cycle is
still visible — only the escalation to critical waits for a second cycle to
confirm it.
Calibration against the current state: the four stale schedules sit at 46.8 to
48.5 hours since their last success, so the new threshold leaves roughly twelve
hours — exactly until today's runs. If those fail too, the alert fires, which
is the intended behaviour rather than a gap.
Verified: the rule passes a server-side dry run, the expression returns nothing
at 60h, and the same query at a lower bound returns the four affected schedules
with their ages — so the selector is known to match rather than merely silent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nodes run v1.12.11, which is the last 1.12 patch — v1.12.12 does not exist.
v1.13.9 is current.
The upgrade path is the documented one rather than a shortcut. Migration is
only tested between adjacent minor releases, and the recommendation is to go
to the latest patch of each intermediate minor. 1.12.11 -> 1.13.9 satisfies
both: one minor step, taken from the last patch of the version being left. No
intermediate hop is needed.
Kubernetes is not forced along. Talos 1.13's support matrix lists 1.36, 1.35,
1.34, 1.33, 1.32 and 1.31, and the cluster runs 1.35.2 — so it stays there
until `talosctl upgrade-k8s` moves it as a separate decision. This was the one
open question before the jump and it is answered from the matrix rather than
assumed.
Deprecations in 1.13 were checked against this cluster, not just read:
.machine.env -> EnvironmentConfig not used
.machine.network.kubespan -> KubeSpanConfig not used
EtcdConfigs/KubeletConfigs protobuf change no consumer in this repo
legacy upgrade API -> LifecycleService still functional; `talosctl
upgrade` keeps working, this
only matters once it is removed
Both the five machine-config files here and the live config on k8s-1 were
checked for the first two, because the running config is generated from a base
this repo only patches and could carry fields the patches never mention.
Neither appears in either.
The removed interactive installer and the dropped `talosctl install` do not
apply — installation here is `talosctl upgrade --image` and Image Factory
media.
Schematic is unchanged; the factory image was verified to exist for the new
tag before committing. The cold-rebuild ISO URL moves with it.
Not applied to any node. Rolling it out is drain -> upgrade -> wait for Ceph
to return to 217 PGs active+clean, one node at a time, as in the 1.12.11 run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The override existed because the mons share the node's /var with containerd,
and on the VMs that was ~60 GB — 30% free measured image and snapshot churn
rather than anything about Ceph, so k8s-2 tripped MON_DISK_LOW whenever the
image cache filled and GC could not reclaim it (#434).
Bare metal removed the premise. /var is now ~931 GB per node at 7-8% used, so
30% free is a real reserve again and the ~106 MB mon database is irrelevant
against it. Reaching 70% would be a genuine signal about the node rather than
noise about the image cache. The comment still described the 60 GB VM /var and
ended with "BARE-METAL MIGRATION: revisit this", which is now done.
Set to 30 rather than deleted, deliberately. `ceph config` keeps a value that
only disappears from this file: `ceph config dump` currently shows
`mon advanced mon_data_avail_warn 20`, and removing the line would leave it
there indefinitely. That is the trap the bluestore settings below already
document, and the one that crash-looped an OSD during the network provider
change when public_network survived being deleted here.
mon_data_avail_crit stays at the 5% default, and CephMonDiskLow in
monitoring-rules still watches /var directly for the real risk, which is the
node filling rather than the mon growing.
kubectl diff against the live cluster shows the value and the generation bump,
nothing else. Verify after it applies with
`ceph config get mon mon_data_avail_warn` — 30 there is the proof the database
entry was overwritten and not merely edited in git.
Supersedes the remaining half of #503, which was closed as overtaken by the
bare-metal migration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The pool was .4-.6 because the nodes held .1-.3 for as long as Ceph needed
host networking to reach VLAN 49 — three addresses for three OSDs, with no
spare, during the one phase where things were actually moving.
The nodes have since given those addresses up: the link is only a macvlan
parent now and does not need one of its own. The reservation has nothing left
to avoid, so the pool takes .1 through .6.
Six for three is the headroom the migration could not afford. It leaves room
for a leaked allocation to sit until the whereabouts reconciler collects it —
the case where a node is lost hard enough that the CNI DEL never runs — and
room to add a second OSD per node without touching this file.
Keeping the pool inside the /29 rather than moving it somewhere roomier on the
segment stays deliberate. There is no router here, so an address outside the
/29 is unreachable from anything using one inside it, which is what would
break a future rolling change where migrated and un-migrated OSDs still have
to talk to each other. That was nearly the shape of this attachment and it
would have taken every pool down at the first OSD.
Inert for the running cluster: a NetworkAttachmentDefinition is read when a
pod is created, so the three existing OSDs keep .4, .5 and .6 until they next
restart for some other reason.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The cluster had no IPv6 egress. Pods could not reach the IPv6 internet at all
while IPv4 from the same pod worked normally.
enp2s0f0np0 is nothing but the trunk parent for mgmt0 and ens19, and enp87s0
is unused. Both sit on the untagged fallback network, where by design nothing
is supposed to run — and both were autoconfiguring from its router
advertisements: an fd00::/64 ULA each, plus an IPv6 default route each.
That left four IPv6 default routes per node at identical priority 1024 —
mgmt0, enp2s0f0np0, enp87s0 and lo — of which only mgmt0 has a globally
routable source address. With nothing to break the tie, outbound IPv6 mostly
left through an interface whose only usable source was a ULA.
This is the same argument the file already makes for ens19, where host
addresses on a link that exists only as a macvlan parent are called "never
needed and actively harmful". It was simply never extended to the physical
ports.
How it surfaced: NetbirdRelayMemberDown and NetbirdPublicEdgeMemberDown had
been firing for 30 hours, reading as though the proxies were down. They were
not — every failing target is an IPv6 literal and the blackbox exporter is a
pod, so the probes were failing on cluster egress. The proxies answered IPv6
in 13 ms from a workstation on another VLAN throughout.
Applied to k8s-1 first and verified before rolling to the other two, because
the existing precedent disables v6 on a VLAN *child* while this targets VLAN
*parents*, and mgmt0 carries the address Ceph and Cilium use:
mgmt0 kept 2003:d6:9718:3b02:… and fd40::… unchanged
fd00:: addresses 2 -> 0
IPv6 default routes 4 -> 2 (mgmt0, lo)
failing blackbox probes 5 -> 0
Ceph 217 PGs active+clean, no OSD restart
All three nodes are already patched live; this puts it in git so the next
apply-config does not undo it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
description:"A Velero backup run for {{ $labels.schedule }} completed with Failed or PartiallyFailed status in the last hour. Inspect with `kubectl -n velero get backup.velero.io -l velero.io/schedule-name={{ $labels.schedule }}` and the velero pod logs."
- alert:VeleroBackupStale
# Daily schedule with no successful backup in over 30h: covers
# the case where multiple consecutive runs failed (Garage flapping
# over the backup window) without the per-event alerts being
# acted on.
# Daily schedule with no successful backup in over 60h: covers the
# case where consecutive runs failed without the per-event alerts
# being acted on.
#
# 60h, not 30h. At 30h a single missed cycle pages, and a single
# missed cycle is the normal cost of touching the nodes: Velero moves
# data through pods, so draining a node kills whatever upload is in
# flight and the schedule is simply lost for that day — it is not
# retried. Rolling node work between 03:15 and 08:45 UTC therefore
# produced a guaranteed critical alert for every schedule it
# overlapped, which is noise about planned work rather than a signal
# about backups.
#
# 60h still catches what this is for. Two consecutive failures is no
# longer explicable by one maintenance window, and the per-event
# VeleroBackupFailed alert fires on the first failure regardless — so
# nothing is hidden, only the escalation to critical is deferred
summary:"Velero {{ $labels.schedule }} has no successful backup in 30h+"
description:"Last successful backup for {{ $labels.schedule }} was over 30 hours ago. With a daily schedule that means at least one full cycle missed. RPO is degrading."
summary:"Velero {{ $labels.schedule }} has no successful backup in 60h+"
description:"Last successful backup for {{ $labels.schedule }} was over 60 hours ago. With a daily schedule that is at least two consecutive cycles missed, which one maintenance window does not explain. RPO is degrading."
# v30iot NIC carries no host addressing — see k8s-1.yaml.
net.ipv6.conf.ens19.disable_ipv6:"1"
# Same argument for the physical ports: enp2s0f0np0 is only the trunk
# parent for mgmt0 and ens19, enp87s0 is unused, and both sit on the
# untagged fallback network where nothing is supposed to run. Both were
# taking an fd00::/64 and an IPv6 default route from its router
# advertisements, leaving four equal-priority default routes per node of
# which only mgmt0 has a globally routable source. See k8s-1.yaml for the
# full reasoning and how it was verified.
net.ipv6.conf.enp2s0f0np0.disable_ipv6:"1"
net.ipv6.conf.enp87s0.disable_ipv6:"1"
install:
# Three NVMe drives are visible on bare metal. Select the installer target
# by serial so it cannot land on the OSD drive or on EPHEMERAL, whatever
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
issue-fix-bot
k.renner
claude-botandClaude Opus 5
security-bot