At 30h a single missed cycle pages critical, and a single missed cycle is the
normal cost of touching the nodes. Velero moves data through pods, so draining
a node kills whatever upload is in flight and the schedule is lost for that day
— it is not retried. Backup windows run 03:15 to 08:45 UTC, so any rolling node
work overlapping them produced a guaranteed critical alert per affected
schedule.
That is noise about planned work, not signal about backups. It fired for eleven
schedules during the bare-metal Talos upgrade and again today.
60h still catches what the rule exists for. Two consecutive failures is no
longer explicable by one maintenance window. Nothing is hidden in the meantime:
VeleroBackupFailed fires on the first failure regardless, so the first cycle is
still visible — only the escalation to critical waits for a second cycle to
confirm it.
Calibration against the current state: the four stale schedules sit at 46.8 to
48.5 hours since their last success, so the new threshold leaves roughly twelve
hours — exactly until today's runs. If those fail too, the alert fires, which
is the intended behaviour rather than a gap.
Verified: the rule passes a server-side dry run, the expression returns nothing
at 60h, and the same query at a lower bound returns the four affected schedules
with their ages — so the selector is known to match rather than merely silent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nodes run v1.12.11, which is the last 1.12 patch — v1.12.12 does not exist.
v1.13.9 is current.
The upgrade path is the documented one rather than a shortcut. Migration is
only tested between adjacent minor releases, and the recommendation is to go
to the latest patch of each intermediate minor. 1.12.11 -> 1.13.9 satisfies
both: one minor step, taken from the last patch of the version being left. No
intermediate hop is needed.
Kubernetes is not forced along. Talos 1.13's support matrix lists 1.36, 1.35,
1.34, 1.33, 1.32 and 1.31, and the cluster runs 1.35.2 — so it stays there
until `talosctl upgrade-k8s` moves it as a separate decision. This was the one
open question before the jump and it is answered from the matrix rather than
assumed.
Deprecations in 1.13 were checked against this cluster, not just read:
.machine.env -> EnvironmentConfig not used
.machine.network.kubespan -> KubeSpanConfig not used
EtcdConfigs/KubeletConfigs protobuf change no consumer in this repo
legacy upgrade API -> LifecycleService still functional; `talosctl
upgrade` keeps working, this
only matters once it is removed
Both the five machine-config files here and the live config on k8s-1 were
checked for the first two, because the running config is generated from a base
this repo only patches and could carry fields the patches never mention.
Neither appears in either.
The removed interactive installer and the dropped `talosctl install` do not
apply — installation here is `talosctl upgrade --image` and Image Factory
media.
Schematic is unchanged; the factory image was verified to exist for the new
tag before committing. The cold-rebuild ISO URL moves with it.
Not applied to any node. Rolling it out is drain -> upgrade -> wait for Ceph
to return to 217 PGs active+clean, one node at a time, as in the 1.12.11 run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The override existed because the mons share the node's /var with containerd,
and on the VMs that was ~60 GB — 30% free measured image and snapshot churn
rather than anything about Ceph, so k8s-2 tripped MON_DISK_LOW whenever the
image cache filled and GC could not reclaim it (#434).
Bare metal removed the premise. /var is now ~931 GB per node at 7-8% used, so
30% free is a real reserve again and the ~106 MB mon database is irrelevant
against it. Reaching 70% would be a genuine signal about the node rather than
noise about the image cache. The comment still described the 60 GB VM /var and
ended with "BARE-METAL MIGRATION: revisit this", which is now done.
Set to 30 rather than deleted, deliberately. `ceph config` keeps a value that
only disappears from this file: `ceph config dump` currently shows
`mon advanced mon_data_avail_warn 20`, and removing the line would leave it
there indefinitely. That is the trap the bluestore settings below already
document, and the one that crash-looped an OSD during the network provider
change when public_network survived being deleted here.
mon_data_avail_crit stays at the 5% default, and CephMonDiskLow in
monitoring-rules still watches /var directly for the real risk, which is the
node filling rather than the mon growing.
kubectl diff against the live cluster shows the value and the generation bump,
nothing else. Verify after it applies with
`ceph config get mon mon_data_avail_warn` — 30 there is the proof the database
entry was overwritten and not merely edited in git.
Supersedes the remaining half of #503, which was closed as overtaken by the
bare-metal migration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The pool was .4-.6 because the nodes held .1-.3 for as long as Ceph needed
host networking to reach VLAN 49 — three addresses for three OSDs, with no
spare, during the one phase where things were actually moving.
The nodes have since given those addresses up: the link is only a macvlan
parent now and does not need one of its own. The reservation has nothing left
to avoid, so the pool takes .1 through .6.
Six for three is the headroom the migration could not afford. It leaves room
for a leaked allocation to sit until the whereabouts reconciler collects it —
the case where a node is lost hard enough that the CNI DEL never runs — and
room to add a second OSD per node without touching this file.
Keeping the pool inside the /29 rather than moving it somewhere roomier on the
segment stays deliberate. There is no router here, so an address outside the
/29 is unreachable from anything using one inside it, which is what would
break a future rolling change where migrated and un-migrated OSDs still have
to talk to each other. That was nearly the shape of this attachment and it
would have taken every pool down at the first OSD.
Inert for the running cluster: a NetworkAttachmentDefinition is read when a
pod is created, so the three existing OSDs keep .4, .5 and .6 until they next
restart for some other reason.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The cluster had no IPv6 egress. Pods could not reach the IPv6 internet at all
while IPv4 from the same pod worked normally.
enp2s0f0np0 is nothing but the trunk parent for mgmt0 and ens19, and enp87s0
is unused. Both sit on the untagged fallback network, where by design nothing
is supposed to run — and both were autoconfiguring from its router
advertisements: an fd00::/64 ULA each, plus an IPv6 default route each.
That left four IPv6 default routes per node at identical priority 1024 —
mgmt0, enp2s0f0np0, enp87s0 and lo — of which only mgmt0 has a globally
routable source address. With nothing to break the tie, outbound IPv6 mostly
left through an interface whose only usable source was a ULA.
This is the same argument the file already makes for ens19, where host
addresses on a link that exists only as a macvlan parent are called "never
needed and actively harmful". It was simply never extended to the physical
ports.
How it surfaced: NetbirdRelayMemberDown and NetbirdPublicEdgeMemberDown had
been firing for 30 hours, reading as though the proxies were down. They were
not — every failing target is an IPv6 literal and the blackbox exporter is a
pod, so the probes were failing on cluster egress. The proxies answered IPv6
in 13 ms from a workstation on another VLAN throughout.
Applied to k8s-1 first and verified before rolling to the other two, because
the existing precedent disables v6 on a VLAN *child* while this targets VLAN
*parents*, and mgmt0 carries the address Ceph and Cilium use:
mgmt0 kept 2003:d6:9718:3b02:… and fd40::… unchanged
fd00:: addresses 2 -> 0
IPv6 default routes 4 -> 2 (mgmt0, lo)
failing blackbox probes 5 -> 0
Ceph 217 PGs active+clean, no OSD restart
All three nodes are already patched live; this puts it in git so the next
apply-config does not undo it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was no OpenBao alerting of any kind — no rule, no ServiceMonitor. That
is why today's seal went unnoticed for two hours: the k8s-1 Talos upgrade
sealed the single replica, the ClusterSecretStore went InvalidProviderConfig,
and nothing said so.
Three replicas make this more important rather than less. Before, a seal was a
visible outage that announced itself. Now the service survives one, so a
sealed replica is a quiet loss of redundancy that nothing surfaces until the
next drain takes quorum with it.
Watches pod readiness rather than a seal metric, because there is no seal
metric to watch. OpenBao is not scraped: no ServiceMonitor exists, and its
metrics endpoint needs a token or an explicit unauthenticated_metrics_access
telemetry setting, neither configured. Verified against the running
Prometheus — no bao_* or vault_* series at all. External Secrets exports
nothing usable either, so the ClusterSecretStore going NotReady is invisible.
Readiness carries the signal for a structural reason, not by luck: a sealed
server fails its readiness probe because it can neither serve requests nor act
as a raft standby. Sealed is therefore always not-Ready. The converse does not
hold — a crash or stuck scheduling fires the same alert — but those want a
human too.
Three rules:
OpenBaoReplicaNotReady < 3 ready for 10m, warning. The duration only
ignores ordinary rescheduling; a sealed replica
never recovers on its own.
OpenBaoQuorumLost < 2 ready for 5m, critical. Raft quorum of three
is two, so below that the store is down.
OpenBaoMetricsMissing absent() for 15m, warning. Both rules above are
threshold comparisons, so if the series disappears
they stop evaluating and the silence is
indistinguishable from health.
Verified against the running Prometheus rather than assumed: the selector
matches (the same query with `< 4` returns the series at 3), and the alerting
thresholds correctly return nothing at 3/3 ready. Testing only that it returns
nothing would not have caught a typo in the selector.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
OpenBao ran a single replica. Every node drain took the whole secret store
down with it: today the k8s-1 Talos upgrade sealed it, the ClusterSecretStore
went InvalidProviderConfig, and it stayed that way for two hours because
nothing was watching.
Three, not two. Raft needs a quorum and the quorum of two is two, so a
two-replica cluster tolerates no failures — it would stop the moment one pod
is sealed or rescheduled, which is worse than the single replica it replaced.
Three tolerates one.
What this buys: a drain seals the replica on that node, and raft elects a new
leader from the remaining standbys within seconds. The service does not go
down.
What it does not buy, stated plainly because it is easy to assume otherwise: a
sealed server can neither serve requests nor act as a standby, so that replica
stays out of the cluster until someone unseals it. Scaling removes the outage,
not the manual unseal. For rolling node maintenance the consequence matters —
a Talos upgrade drains all three nodes in turn, so unseal each replica as it
comes back, or the third drain takes the last unsealed one and quorum is lost
anyway.
retry_join is what makes a scaled replica join the existing cluster rather
than come up isolated and uninitialized; without it each new pod would need a
manual `bao operator raft join`. All three peers are listed on every node —
the stanza is a list of possible leaders, a node ignores its own entry, and
listing all three keeps the config identical across replicas regardless of
which pod currently leads. Addressed through the headless service so each
entry resolves to one specific pod.
The snapshot job moves from the plain service to openbao-active. With one
replica the plain service could only ever hit the one pod; with three it
selects any of them, and a raft snapshot has to be taken against the leader.
No affinity was added: the chart already applies a required podAntiAffinity on
kubernetes.io/hostname, so the replicas spread across the three nodes on their
own — verified against the running StatefulSet.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-26 22:49:41 +02:00
16 changed files with 293 additions and 61 deletions
description:"A Velero backup run for {{ $labels.schedule }} completed with Failed or PartiallyFailed status in the last hour. Inspect with `kubectl -n velero get backup.velero.io -l velero.io/schedule-name={{ $labels.schedule }}` and the velero pod logs."
- alert:VeleroBackupStale
# Daily schedule with no successful backup in over 30h: covers
# the case where multiple consecutive runs failed (Garage flapping
# over the backup window) without the per-event alerts being
# acted on.
# Daily schedule with no successful backup in over 60h: covers the
# case where consecutive runs failed without the per-event alerts
# being acted on.
#
# 60h, not 30h. At 30h a single missed cycle pages, and a single
# missed cycle is the normal cost of touching the nodes: Velero moves
# data through pods, so draining a node kills whatever upload is in
# flight and the schedule is simply lost for that day — it is not
# retried. Rolling node work between 03:15 and 08:45 UTC therefore
# produced a guaranteed critical alert for every schedule it
# overlapped, which is noise about planned work rather than a signal
# about backups.
#
# 60h still catches what this is for. Two consecutive failures is no
# longer explicable by one maintenance window, and the per-event
# VeleroBackupFailed alert fires on the first failure regardless — so
# nothing is hidden, only the escalation to critical is deferred
summary:"Velero {{ $labels.schedule }} has no successful backup in 30h+"
description:"Last successful backup for {{ $labels.schedule }} was over 30 hours ago. With a daily schedule that means at least one full cycle missed. RPO is degrading."
summary:"Velero {{ $labels.schedule }} has no successful backup in 60h+"
description:"Last successful backup for {{ $labels.schedule }} was over 60 hours ago. With a daily schedule that is at least two consecutive cycles missed, which one maintenance window does not explain. RPO is degrading."
# v30iot NIC carries no host addressing — see k8s-1.yaml.
net.ipv6.conf.ens19.disable_ipv6:"1"
# Same argument for the physical ports: enp2s0f0np0 is only the trunk
# parent for mgmt0 and ens19, enp87s0 is unused, and both sit on the
# untagged fallback network where nothing is supposed to run. Both were
# taking an fd00::/64 and an IPv6 default route from its router
# advertisements, leaving four equal-priority default routes per node of
# which only mgmt0 has a globally routable source. See k8s-1.yaml for the
# full reasoning and how it was verified.
net.ipv6.conf.enp2s0f0np0.disable_ipv6:"1"
net.ipv6.conf.enp87s0.disable_ipv6:"1"
install:
# Three NVMe drives are visible on bare metal. Select the installer target
# by serial so it cannot land on the OSD drive or on EPHEMERAL, whatever
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
issue-fix-bot
k.renner
claude-botandClaude Opus 5
security-bot