At 30h a single missed cycle pages critical, and a single missed cycle is the
normal cost of touching the nodes. Velero moves data through pods, so draining
a node kills whatever upload is in flight and the schedule is lost for that day
— it is not retried. Backup windows run 03:15 to 08:45 UTC, so any rolling node
work overlapping them produced a guaranteed critical alert per affected
schedule.
That is noise about planned work, not signal about backups. It fired for eleven
schedules during the bare-metal Talos upgrade and again today.
60h still catches what the rule exists for. Two consecutive failures is no
longer explicable by one maintenance window. Nothing is hidden in the meantime:
VeleroBackupFailed fires on the first failure regardless, so the first cycle is
still visible — only the escalation to critical waits for a second cycle to
confirm it.
Calibration against the current state: the four stale schedules sit at 46.8 to
48.5 hours since their last success, so the new threshold leaves roughly twelve
hours — exactly until today's runs. If those fail too, the alert fires, which
is the intended behaviour rather than a gap.
Verified: the rule passes a server-side dry run, the expression returns nothing
at 60h, and the same query at a lower bound returns the four affected schedules
with their ages — so the selector is known to match rather than merely silent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nodes run v1.12.11, which is the last 1.12 patch — v1.12.12 does not exist.
v1.13.9 is current.
The upgrade path is the documented one rather than a shortcut. Migration is
only tested between adjacent minor releases, and the recommendation is to go
to the latest patch of each intermediate minor. 1.12.11 -> 1.13.9 satisfies
both: one minor step, taken from the last patch of the version being left. No
intermediate hop is needed.
Kubernetes is not forced along. Talos 1.13's support matrix lists 1.36, 1.35,
1.34, 1.33, 1.32 and 1.31, and the cluster runs 1.35.2 — so it stays there
until `talosctl upgrade-k8s` moves it as a separate decision. This was the one
open question before the jump and it is answered from the matrix rather than
assumed.
Deprecations in 1.13 were checked against this cluster, not just read:
.machine.env -> EnvironmentConfig not used
.machine.network.kubespan -> KubeSpanConfig not used
EtcdConfigs/KubeletConfigs protobuf change no consumer in this repo
legacy upgrade API -> LifecycleService still functional; `talosctl
upgrade` keeps working, this
only matters once it is removed
Both the five machine-config files here and the live config on k8s-1 were
checked for the first two, because the running config is generated from a base
this repo only patches and could carry fields the patches never mention.
Neither appears in either.
The removed interactive installer and the dropped `talosctl install` do not
apply — installation here is `talosctl upgrade --image` and Image Factory
media.
Schematic is unchanged; the factory image was verified to exist for the new
tag before committing. The cold-rebuild ISO URL moves with it.
Not applied to any node. Rolling it out is drain -> upgrade -> wait for Ceph
to return to 217 PGs active+clean, one node at a time, as in the 1.12.11 run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The override existed because the mons share the node's /var with containerd,
and on the VMs that was ~60 GB — 30% free measured image and snapshot churn
rather than anything about Ceph, so k8s-2 tripped MON_DISK_LOW whenever the
image cache filled and GC could not reclaim it (#434).
Bare metal removed the premise. /var is now ~931 GB per node at 7-8% used, so
30% free is a real reserve again and the ~106 MB mon database is irrelevant
against it. Reaching 70% would be a genuine signal about the node rather than
noise about the image cache. The comment still described the 60 GB VM /var and
ended with "BARE-METAL MIGRATION: revisit this", which is now done.
Set to 30 rather than deleted, deliberately. `ceph config` keeps a value that
only disappears from this file: `ceph config dump` currently shows
`mon advanced mon_data_avail_warn 20`, and removing the line would leave it
there indefinitely. That is the trap the bluestore settings below already
document, and the one that crash-looped an OSD during the network provider
change when public_network survived being deleted here.
mon_data_avail_crit stays at the 5% default, and CephMonDiskLow in
monitoring-rules still watches /var directly for the real risk, which is the
node filling rather than the mon growing.
kubectl diff against the live cluster shows the value and the generation bump,
nothing else. Verify after it applies with
`ceph config get mon mon_data_avail_warn` — 30 there is the proof the database
entry was overwritten and not merely edited in git.
Supersedes the remaining half of #503, which was closed as overtaken by the
bare-metal migration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The pool was .4-.6 because the nodes held .1-.3 for as long as Ceph needed
host networking to reach VLAN 49 — three addresses for three OSDs, with no
spare, during the one phase where things were actually moving.
The nodes have since given those addresses up: the link is only a macvlan
parent now and does not need one of its own. The reservation has nothing left
to avoid, so the pool takes .1 through .6.
Six for three is the headroom the migration could not afford. It leaves room
for a leaked allocation to sit until the whereabouts reconciler collects it —
the case where a node is lost hard enough that the CNI DEL never runs — and
room to add a second OSD per node without touching this file.
Keeping the pool inside the /29 rather than moving it somewhere roomier on the
segment stays deliberate. There is no router here, so an address outside the
/29 is unreachable from anything using one inside it, which is what would
break a future rolling change where migrated and un-migrated OSDs still have
to talk to each other. That was nearly the shape of this attachment and it
would have taken every pool down at the first OSD.
Inert for the running cluster: a NetworkAttachmentDefinition is read when a
pod is created, so the three existing OSDs keep .4, .5 and .6 until they next
restart for some other reason.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The cluster had no IPv6 egress. Pods could not reach the IPv6 internet at all
while IPv4 from the same pod worked normally.
enp2s0f0np0 is nothing but the trunk parent for mgmt0 and ens19, and enp87s0
is unused. Both sit on the untagged fallback network, where by design nothing
is supposed to run — and both were autoconfiguring from its router
advertisements: an fd00::/64 ULA each, plus an IPv6 default route each.
That left four IPv6 default routes per node at identical priority 1024 —
mgmt0, enp2s0f0np0, enp87s0 and lo — of which only mgmt0 has a globally
routable source address. With nothing to break the tie, outbound IPv6 mostly
left through an interface whose only usable source was a ULA.
This is the same argument the file already makes for ens19, where host
addresses on a link that exists only as a macvlan parent are called "never
needed and actively harmful". It was simply never extended to the physical
ports.
How it surfaced: NetbirdRelayMemberDown and NetbirdPublicEdgeMemberDown had
been firing for 30 hours, reading as though the proxies were down. They were
not — every failing target is an IPv6 literal and the blackbox exporter is a
pod, so the probes were failing on cluster egress. The proxies answered IPv6
in 13 ms from a workstation on another VLAN throughout.
Applied to k8s-1 first and verified before rolling to the other two, because
the existing precedent disables v6 on a VLAN *child* while this targets VLAN
*parents*, and mgmt0 carries the address Ceph and Cilium use:
mgmt0 kept 2003:d6:9718:3b02:… and fd40::… unchanged
fd00:: addresses 2 -> 0
IPv6 default routes 4 -> 2 (mgmt0, lo)
failing blackbox probes 5 -> 0
Ceph 217 PGs active+clean, no OSD restart
All three nodes are already patched live; this puts it in git so the next
apply-config does not undo it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was no OpenBao alerting of any kind — no rule, no ServiceMonitor. That
is why today's seal went unnoticed for two hours: the k8s-1 Talos upgrade
sealed the single replica, the ClusterSecretStore went InvalidProviderConfig,
and nothing said so.
Three replicas make this more important rather than less. Before, a seal was a
visible outage that announced itself. Now the service survives one, so a
sealed replica is a quiet loss of redundancy that nothing surfaces until the
next drain takes quorum with it.
Watches pod readiness rather than a seal metric, because there is no seal
metric to watch. OpenBao is not scraped: no ServiceMonitor exists, and its
metrics endpoint needs a token or an explicit unauthenticated_metrics_access
telemetry setting, neither configured. Verified against the running
Prometheus — no bao_* or vault_* series at all. External Secrets exports
nothing usable either, so the ClusterSecretStore going NotReady is invisible.
Readiness carries the signal for a structural reason, not by luck: a sealed
server fails its readiness probe because it can neither serve requests nor act
as a raft standby. Sealed is therefore always not-Ready. The converse does not
hold — a crash or stuck scheduling fires the same alert — but those want a
human too.
Three rules:
OpenBaoReplicaNotReady < 3 ready for 10m, warning. The duration only
ignores ordinary rescheduling; a sealed replica
never recovers on its own.
OpenBaoQuorumLost < 2 ready for 5m, critical. Raft quorum of three
is two, so below that the store is down.
OpenBaoMetricsMissing absent() for 15m, warning. Both rules above are
threshold comparisons, so if the series disappears
they stop evaluating and the silence is
indistinguishable from health.
Verified against the running Prometheus rather than assumed: the selector
matches (the same query with `< 4` returns the series at 3), and the alerting
thresholds correctly return nothing at 3/3 ready. Testing only that it returns
nothing would not have caught a typo in the selector.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
OpenBao ran a single replica. Every node drain took the whole secret store
down with it: today the k8s-1 Talos upgrade sealed it, the ClusterSecretStore
went InvalidProviderConfig, and it stayed that way for two hours because
nothing was watching.
Three, not two. Raft needs a quorum and the quorum of two is two, so a
two-replica cluster tolerates no failures — it would stop the moment one pod
is sealed or rescheduled, which is worse than the single replica it replaced.
Three tolerates one.
What this buys: a drain seals the replica on that node, and raft elects a new
leader from the remaining standbys within seconds. The service does not go
down.
What it does not buy, stated plainly because it is easy to assume otherwise: a
sealed server can neither serve requests nor act as a standby, so that replica
stays out of the cluster until someone unseals it. Scaling removes the outage,
not the manual unseal. For rolling node maintenance the consequence matters —
a Talos upgrade drains all three nodes in turn, so unseal each replica as it
comes back, or the third drain takes the last unsealed one and quorum is lost
anyway.
retry_join is what makes a scaled replica join the existing cluster rather
than come up isolated and uninitialized; without it each new pod would need a
manual `bao operator raft join`. All three peers are listed on every node —
the stanza is a list of possible leaders, a node ignores its own entry, and
listing all three keeps the config identical across replicas regardless of
which pod currently leads. Addressed through the headless service so each
entry resolves to one specific pod.
The snapshot job moves from the plain service to openbao-active. With one
replica the plain service could only ever hit the one pod; with three it
selects any of them, and a raft snapshot has to be taken against the leader.
No affinity was added: the chart already applies a required podAntiAffinity on
kubernetes.io/hostname, so the replicas spread across the three nodes on their
own — verified against the running StatefulSet.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The nodes held 10.99.0.1-.3/29 on enp2s0f1np1 only because reaching that link
required CephCluster.spec.network.provider: host, which is not per-daemon and
put every Ceph daemon on host networking. Rook now attaches the link to the
OSD pods through a Multus macvlan, and a macvlan parent only needs to be up —
it does not need an address of its own for its children to work.
By the rule this segment is built on — Ceph-internal traffic and nothing else,
no router, no other participants — the node has no business holding an address
there either. This is stricter than before, not looser.
The link keeps up: true and mtu: 9000. Both still matter: the parent must be
up for the macvlan children to exist, and the MTU is what the children
inherit.
Not in this commit, deliberately: widening the attachment's whereabouts pool
from .4-.6 to the full /29. That must not land before these addresses are
actually gone from the running nodes, or whereabouts could hand .1 to an OSD
while a node still holds it. Follow-up once these configs are applied and
verified.
Also records the one-time step that the manifests cannot express. Removing
addressRanges did not clear public_network and cluster_network from Ceph's
config database — the same trap the bluestore comment in the same file
describes. The first OSD of the phase A rollout came up on the pod network,
still had public_network 10.2.40.0/24 in the config database, found no
interface in it and crash-looped on "Failed to pick public address". Both keys
were removed by hand. A rebuild from scratch will not hit this because the
config database starts empty; another provider change on a live cluster will.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Puts the OSDs back on the VLAN 49 link that phase A gave up, without putting
every Ceph daemon on host networking to get there.
The cluster network is an OSD concern. Per the Ceph documentation it carries
OSD heartbeat, replication and recovery, and "Ceph Monitors always operate on
the public network". Reaching it through addressRanges required
provider: host, which is not per-daemon — the mons went onto host networking
as a side effect, making a mon's address the node's DHCP lease.
That failure is documented rather than hypothetical. When a lease moved, the
mon could not bind ("Cannot assign requested address") and crash-looped. Rook
detected it and began a failover, but with three mons on three nodes each
holding the mon host ports the replacement could not be scheduled anywhere —
the canary sat Pending on "didn't have free ports". Two of three nodes hit it
during the Talos v1.12.11 upgrade; both needed hand intervention.
Selecting only `cluster` leaves `public` on the pod network, so mons, MDS,
MGR and the CSI pods keep Rook-managed identities and hold no host ports.
That removes the failure mode rather than working around it.
Isolation is not weakened. Rook's Multus prerequisites about hosts routing to
the Multus network apply to a Multus *public* network; selecting only
`cluster` means they do not apply here. VLAN 49 stays Ceph-internal traffic
and nothing else, with no router and no other participants.
MUST NOT be applied before phase A. Verified rather than assumed — kubectl
diff against the current cluster refuses it:
spec.network.provider: Invalid value: "multus": network provider must be
disabled (reverted to empty string) before a new provider is enabled
Second of the two rolling OSD restarts, bounded by the same two settings: one
OSD at a time, only once PGs are clean.
Follow-up, not in this commit: once this settles, the node addresses on VLAN
49 (10.99.0.1-.3) have no remaining purpose and should come out of the Talos
configs, after which the attachment's pool can take the whole /29 instead of
the three addresses it reserves today.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Intermediate state, not a destination. san-network-attachment.yaml is already
in the cluster and stays unused until phase B sets provider: multus.
The detour is forced by Rook's own CRD validation, not chosen:
rule: self == '' || oldSelf == '' || self == oldSelf
message: network provider must be disabled (reverted to empty string)
before a new provider is enabled
host -> multus is rejected outright. It has to pass through empty, and an
empty provider means the default CNI — so addressRanges, which only takes
effect under provider: host, goes with it.
What this costs while it is in place: there is no cluster network. OSD
heartbeat, replication and recovery move onto the pod network and share it
with everything else instead of the dedicated X710 SFP+ link. Slower, and
every OSD restart pays that price in backfill. Not a correctness problem —
the OSDs can still reach one another throughout, which is the property that
matters, and the one a separate subnet would have broken.
The mons do not move here. Rook leaves existing mons on the network they were
created with; they change only on failover, which is deliberate and separate.
Bounded by the two settings that landed ahead of this: one OSD restarts at a
time, and only once PGs are clean again. Expect three sequential OSD restarts
with full recovery between them, and the same again in phase B.
kubectl diff against the live cluster shows provider host -> "" and the
generation bump. The removal of addressRanges does not appear there, because
this resource carries no managedFields and client-side diff cannot see a
field deletion. It is inert either way — Rook does not read addressRanges
without provider: host — but worth confirming after the apply rather than
assuming.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rook's defaults are wrong for a cluster this size, and nothing here had
overridden them:
storage.osdMaxUpdatesInParallel default 20
upgradeOSDRequiresHealthyPGs default false
Twenty parallel OSD restarts is a small fraction of a large cluster. Here it
is all three, at once, with no gate on PG recovery in between. Every PG goes
inactive and every pool — RBD volumes and both CephFS mounts — becomes
unreachable for the duration. Not a degraded cluster, a stopped one.
With three OSDs at size 3 there is no spare copy to absorb the second
restart, so the health gate between one OSD returning and the next going down
is the only thing separating a rolling change from an outage.
This binds any change that rewrites the OSD deployment spec, not only Ceph
version upgrades. The immediate reason is the pending network provider
switch, which is exactly such a change — but the exposure predates it and
would have applied to any of them.
Lands inert: `kubectl diff` against the live cluster shows the two added
fields and the generation bump, nothing else. No OSD deployment spec changes,
so nothing restarts.
Deliberately separate from the provider switch rather than bundled with it,
so the bound is in place and observed before the operation it exists to
bound.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The pool was 10.99.1.0/24, a separate subnet on the same segment, chosen to
avoid carving up a /29 with only three addresses left. That choice would have
broken the migration it exists for.
OSDs move one at a time, so the cluster runs mixed: an OSD already on the
attachment has to reach an OSD still using its node's host address on
10.99.0.x. There is no router on this segment — by design, it carries
Ceph-internal traffic and nothing else — so a pool outside the /29 is
unreachable from the un-migrated side. The two OSDs would mark each other
down, during precisely the window that is supposed to be the safe one.
Moves the pool into 10.99.0.0/29 with range_start/range_end reserving .4-.6,
leaving .1-.3 to the nodes. That is one address per OSD with nothing spare,
which is sufficient rather than merely tolerable: osdMaxUpdatesInParallel
bounds the migration to one OSD at a time, so allocations grow 1, 2, 3 and
never overlap. Headroom returns after the switch is confirmed and the node
addresses come out of the Talos configs — then the pool takes the whole /29.
Verified: the embedded CNI JSON parses with the intended range and bounds;
the NAD passes a server-side dry run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Step 2 of moving Ceph's cluster network off host networking. Inert on its
own: a NetworkAttachmentDefinition does nothing until a pod references it,
and nothing will until cephcluster.yaml names it under
network.selectors.cluster.
Why this exists at all: `addressRanges` only takes effect under
`provider: host`, so carrying replication over the dedicated 10G link meant
setting a provider that is not per-daemon. It puts mons on host networking
too, which makes a mon's address the node's DHCP lease — and when that lease
moves the mon cannot bind. Three mons on three nodes then each hold the mon
host ports, so Rook cannot schedule the replacement it just decided to create
and the failover stalls. Both happened during the v1.12.11 upgrade, on two of
three nodes.
None of that serves the goal. Per the Ceph documentation the cluster network
carries OSD heartbeat, replication and recovery only — "Ceph Monitors always
operate on the public network". Attaching it here gives the OSDs the link and
leaves every other daemon on the pod network.
Its own subnet rather than the existing /29. The nodes hold 10.99.0.1-.3 for
as long as Ceph still uses host networking, which leaves exactly three
addresses for three OSDs — no headroom during the one phase where things
actually move, and a leaked allocation from a hard node loss would mean an
OSD without an address. OSDs never need to reach the node addresses, since
the cluster network is OSD-to-OSD, so the pool sits in its own range on the
same segment instead. A /24 for three addresses costs nothing: nothing else
can ever live on this segment, and it takes address count off the table when
OSDs are added.
After the provider switch is confirmed, the node addresses on VLAN 49 have no
remaining purpose and can come out of the Talos configs.
No gateway and no routes, because there is no router on this segment and
there must not be one. That also means no `sbr` chaining, unlike the v30iot
attachments — there is no DHCP-supplied default route to confine here.
whereabouts for IPAM: dhcp needs a server that must not exist on the SAN, and
host-local keeps state per node with no cluster-wide coordination, so two
nodes would hand out the same address on shared L2.
MTU 9000 matches the link. macvlan inherits it, but stating it means a change
to the parent cannot silently halve replication throughput with nothing in
the tree recording the expectation.
Verified: whereabouts from step 1 is running on all three nodes with its
binary installed next to macvlan; the NAD passes a server-side dry run; the
embedded CNI JSON parses and carries the intended master, mtu, ipam type and
range.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Groundwork for moving Ceph's cluster network off host networking. Nothing
references whereabouts yet, so this lands inert: no existing attachment
changes, and the DaemonSet only installs a CNI binary that nothing invokes
until a NetworkAttachmentDefinition names it.
The Ceph SAN on VLAN 49 carries Ceph-internal traffic and nothing else, by
design. Neither IPAM already available covers it:
dhcp cni-dhcp-daemon works where a DHCP server is on the wire.
There is none on the SAN and there must not be one.
host-local installed by cni-plugins-installer, but it keeps allocation
state per node with no cluster-wide coordination. On a shared
L2 segment two nodes hand out the same address.
Whereabouts coordinates through the Kubernetes API — allocations live in the
ippools and overlappingrangeipreservations CRDs. Nothing about assigning an
address touches the segment being addressed, which is the property that makes
it usable on an isolated SAN.
The DaemonSet runs hostNetwork and privileged. That is upstream's shape, and
it matches what multus-daemonset.yaml, cni-dhcp-daemon.yaml and
cni-plugins-installer.yaml already do: the pod writes a CNI binary and its
config into /opt/cni/bin and /etc/cni/net.d, which is node plumbing, and it
has to work before the attachment it serves exists. It is the fourth instance
of an existing exception, not a new one.
The reconciler is split out deliberately. It needs no host access — only the
API, where the allocations are — so it runs as an ordinary pod-network
Deployment. It matters more than it looks: the normal release path is the CNI
DEL when a pod goes away, which does not run when a node is lost hard, and on
a /29 a handful of leaked addresses is the difference between an OSD getting
one and not.
Vendored from upstream v0.9.4 with the image tag pinned — upstream ships
`:latest`. Tag rather than digest, matching the other images in this
directory. The three CRDs are vendored unmodified so a future upstream diff
stays readable. nodeslicepools is included because the ClusterRole references
it, though the node-slice controller itself is not deployed.
Verified: kustomize builds; all nine resources pass a server-side dry run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The nodes run v1.12.6; v1.12.11 is the last patch of the 1.12 series. Upstream
tests migration only between adjacent minors and recommends going to the latest
patch of each intermediate minor, so 1.12.11 is the required stop before 1.13,
not an optional one.
Adds siderolabs/i915 while the image is being rebuilt anyway. Nothing requests
/dev/dri today — every manifest was checked, the only devices any workload asks
for are Ceph RBD and the NVMe disks by-id. It is included for the reason
thunderbolt already is: the extension set is fixed at install and upgrade time,
so adding it later would mean another rolling upgrade of all three nodes rather
than a config change. The MS-01's Alder Lake-P Iris Xe is present in hardware
either way.
New schematic 0f6fb8bd…, identical to a21cec8f… plus i915. Registered with the
Image Factory and the resulting installer image verified to exist for v1.12.11.
The absences are deliberate and now written down, so they are not re-derived as
gaps later: the X710 (i40e) and I226 (igc) NICs need no firmware extension —
intel-ice-firmware covers E810/ice only; mei is a prerequisite for Intel Arc
discrete GPUs and there are none; iscsi-tools, nvme-cli and util-linux-tools
have no consumer, since storage is Ceph RBD and device health comes from Ceph
rather than a node-level exporter.
Both runbooks move to the new schematic and version.
Note for whoever applies this: talosctl upgrade --image does NOT write
.machine.install.image, so this config has to be applied separately or the node
keeps pointing at whatever it was installed with. That is how the running nodes
ended up on ghcr.io/siderolabs/installer:v1.12.6 despite this file naming a
Factory image — which is also why talosctl get extensions is empty on all
three.
Kubernetes is untouched; talosctl upgrade does not move it. It stays on v1.35.2
and is a separate upgrade-k8s step.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
description:"A Velero backup run for {{ $labels.schedule }} completed with Failed or PartiallyFailed status in the last hour. Inspect with `kubectl -n velero get backup.velero.io -l velero.io/schedule-name={{ $labels.schedule }}` and the velero pod logs."
- alert:VeleroBackupStale
# Daily schedule with no successful backup in over 30h: covers
# the case where multiple consecutive runs failed (Garage flapping
# over the backup window) without the per-event alerts being
# acted on.
# Daily schedule with no successful backup in over 60h: covers the
# case where consecutive runs failed without the per-event alerts
# being acted on.
#
# 60h, not 30h. At 30h a single missed cycle pages, and a single
# missed cycle is the normal cost of touching the nodes: Velero moves
# data through pods, so draining a node kills whatever upload is in
# flight and the schedule is simply lost for that day — it is not
# retried. Rolling node work between 03:15 and 08:45 UTC therefore
# produced a guaranteed critical alert for every schedule it
# overlapped, which is noise about planned work rather than a signal
# about backups.
#
# 60h still catches what this is for. Two consecutive failures is no
# longer explicable by one maintenance window, and the per-event
# VeleroBackupFailed alert fires on the first failure regardless — so
# nothing is hidden, only the escalation to critical is deferred
summary:"Velero {{ $labels.schedule }} has no successful backup in 30h+"
description:"Last successful backup for {{ $labels.schedule }} was over 30 hours ago. With a daily schedule that means at least one full cycle missed. RPO is degrading."
summary:"Velero {{ $labels.schedule }} has no successful backup in 60h+"
description:"Last successful backup for {{ $labels.schedule }} was over 60 hours ago. With a daily schedule that is at least two consecutive cycles missed, which one maintenance window does not explain. RPO is degrading."
# v30iot NIC carries no host addressing — see k8s-1.yaml.
net.ipv6.conf.ens19.disable_ipv6:"1"
# Same argument for the physical ports: enp2s0f0np0 is only the trunk
# parent for mgmt0 and ens19, enp87s0 is unused, and both sit on the
# untagged fallback network where nothing is supposed to run. Both were
# taking an fd00::/64 and an IPv6 default route from its router
# advertisements, leaving four equal-priority default routes per node of
# which only mgmt0 has a globally routable source. See k8s-1.yaml for the
# full reasoning and how it was verified.
net.ipv6.conf.enp2s0f0np0.disable_ipv6:"1"
net.ipv6.conf.enp87s0.disable_ipv6:"1"
install:
# Three NVMe drives are visible on bare metal. Select the installer target
# by serial so it cannot land on the OSD drive or on EPHEMERAL, whatever
@@ -73,10 +82,16 @@ parent: enp2s0f0np0
up:true
---
# Dedicated Ceph cluster network — see k8s-1.yaml.
# The node deliberately holds NO address here. Rook attaches this link to the
# OSD pods through a Multus macvlan (infrastructure/rook-ceph-cluster/
# san-network-attachment.yaml), and a macvlan parent only has to be up — it
# does not need an address of its own for its children to work.
#
# It used to carry 10.99.0.3/29, because reaching this link required
# CephCluster.spec.network.provider: host, which put every Ceph daemon on
# host networking, mons included. Nothing needs it now.
apiVersion:v1alpha1
kind:LinkConfig
name:enp2s0f1np1
up:true
mtu:9000
addresses:
- address:10.99.0.3/29
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
issue-fix-bot
k.renner
claude-botandClaude Opus 5
security-bot