TargetDown{job="apiserver"} fired for ~19h because default/kubernetes kept
advertising 10.2.40.168:6443 after k8s-3's management DHCP lease moved off
that address while the node kept running. I confirmed the mechanism live:
kube-apiserver is started by Talos with --advertise-address=$(POD_IP) and no --endpoint-reconciler-type override, so the advertised address is frozen at
pod start and the default lease reconciler keeps renewing the masterlease under
the old address until the static pod restarts. The underlying trigger is not
safely fixable in this repo (see Notes), so this PR makes no config change —
it records the failure mode and the operator remedy as a comment on the very
document that causes it, DHCPv4Config/mgmt0 in k8s-1.yaml, which is where
the repo keeps the reasoning for the whole network layout (k8s-2/k8s-3 point
at that file).
Changes
kubernetes/talos/patches/k8s-1.yaml — comment above the mgmt0 DHCPv4Config document: leases are unreserved and do move under a running
node; kube-apiserver does not follow, so the kubernetes Endpoints go stale
until a restart; the post-re-IP check (kubectl get endpoints kubernetes -n default vs kubectl get nodes -o wide) and remedy (talosctl -n <node> reboot); and why static addressing was not adopted here. Comments only — no
behavioural change, nothing for Talos to apply.
Verification
Everything below was observed this run via ./.ci-diag/kubectl.
The alert is genuinely recovered, not suppressed. get nodes -o wide → k8s-1 10.2.40.169, k8s-2 10.2.40.212, k8s-3 10.2.40.139; get endpoints kubernetes -n default → exactly .139, .169, .212. They agree, so no stale entry remains.
All three targets are up. Prometheus proxy query?query=up{job="apiserver"} → 1 for 10.2.40.169:6443, 10.2.40.212:6443, 10.2.40.139:6443. No dead instance in the series.
Mechanism confirmed on a live pod (the issue had only checked k8s-2): get pod kube-apiserver-k8s-3 -n kube-system -o jsonpath='{.spec.containers[0].command}'
contains --advertise-address=$(POD_IP) and no --endpoint-reconciler-type
flag → default lease reconciler, address fixed at start.
The address really did move under a running node: get node k8s-3 -o jsonpath='{.metadata.creationTimestamp}...' →
created 2026-05-09T06:57:12Z, InternalIP 10.2.40.139. The Node object is
109 days old and never re-registered, so the InternalIP changed in place —
the precondition for the stale-masterlease failure.
No pinning exists in the repo:mgmt0 carries only DHCPv4Config in all
three patch files; opentofu/unifi.tf is entirely commented-out examples; no
node MAC addresses appear anywhere in the repo (grep for MAC-shaped strings
finds only Zigbee IEEEs, a Miele appliance and one check script).
Reviewer check: nothing should change at runtime — the edit is comments inside a
Talos machine-config patch, which is not reconciled by Flux and only read at talosctl apply-config time. TargetDown is already resolved and should stay
resolved; the value of the change is that the next occurrence is diagnosable in
minutes instead of nineteen hours.
Notes
Why no functional fix. Both real remedies are out of reach of a safe,
verifiable repo change:
Static AddressConfig on mgmt0 would need addresses outside the UDM's
DHCP pool, and that pool is not declared anywhere in this repo — I cannot
pick one without risking a conflict. It also rewrites the addressing of all
three etcd members, and the repo already records (k8s-1.yaml and §6 of RUNBOOK-baremetal-conversion.md) a network-document change that left a
node with no address on any interface. That is decision rule 7 territory:
a change I cannot verify and whose failure mode is a stranded control plane.
DHCP reservations in opentofu/unifi.tf would need the nodes' MAC
addresses; they are not in the repo and this read-only ServiceAccount cannot
obtain them. The file also holds no live resources today, so this would be
the first UniFi-provider resource, applied against the live UDM.
Either is a reasonable follow-up, but both need the owner to supply input the
repo does not contain (free addresses / MACs), so I left them out rather than
guess.
Clearing a stale entry is inherently a live operation — restarting the
affected kube-apiserver, e.g. talosctl -n <node> reboot. In this incident
that already happened at 18:37Z, which is why the alert resolved.
One thing the issue got wrong: it cites k8s-3.yaml:81-82 as still
carrying a static 10.99.0.3/29 on the Ceph link. That address was removed in 96acff3 ("chore(talos): drop the node addresses on the Ceph SAN"); the
current file has a bare LinkConfig. The conclusion it drew from that line —
that no interface anywhere is statically addressed — is now more true, not
less.
The issue's own hedge is right: the masterlease itself lives in etcd under /registry/masterleases/ and this ServiceAccount has no path to it. I did not
read the leases; I confirmed the flags and the address history that make the
lease explanation the only one consistent with the evidence.
No alert silencing of any kind: no PrometheusRule, threshold or inhibition was
touched. TargetDown remains routed normally.
The issue's closing paragraph flags a separate active incident
(ClusterSecretStore/openbao Kubernetes auth failing, ExternalSecrets and
Flux kustomizations blocked). It is unrelated to this alert and out of scope
here, but it is worth its own issue if one does not exist yet.
## What this PR does
`TargetDown{job="apiserver"}` fired for ~19h because `default/kubernetes` kept
advertising `10.2.40.168:6443` after k8s-3's management DHCP lease moved off
that address while the node kept running. I confirmed the mechanism live:
kube-apiserver is started by Talos with `--advertise-address=$(POD_IP)` and no
`--endpoint-reconciler-type` override, so the advertised address is frozen at
pod start and the default lease reconciler keeps renewing the masterlease under
the old address until the static pod restarts. **The underlying trigger is not
safely fixable in this repo** (see Notes), so this PR makes no config change —
it records the failure mode and the operator remedy as a comment on the very
document that causes it, `DHCPv4Config/mgmt0` in `k8s-1.yaml`, which is where
the repo keeps the reasoning for the whole network layout (`k8s-2`/`k8s-3` point
at that file).
## Changes
- `kubernetes/talos/patches/k8s-1.yaml` — comment above the `mgmt0`
`DHCPv4Config` document: leases are unreserved and do move under a running
node; kube-apiserver does not follow, so the `kubernetes` Endpoints go stale
until a restart; the post-re-IP check (`kubectl get endpoints kubernetes -n
default` vs `kubectl get nodes -o wide`) and remedy (`talosctl -n <node>
reboot`); and why static addressing was not adopted here. Comments only — no
behavioural change, nothing for Talos to apply.
## Verification
Everything below was observed this run via `./.ci-diag/kubectl`.
- **The alert is genuinely recovered, not suppressed.**
`get nodes -o wide` → `k8s-1 10.2.40.169`, `k8s-2 10.2.40.212`,
`k8s-3 10.2.40.139`; `get endpoints kubernetes -n default` → exactly
`.139, .169, .212`. They agree, so no stale entry remains.
- **All three targets are up.** Prometheus proxy
`query?query=up{job="apiserver"}` → `1` for `10.2.40.169:6443`,
`10.2.40.212:6443`, `10.2.40.139:6443`. No dead instance in the series.
- **Mechanism confirmed on a live pod** (the issue had only checked k8s-2):
`get pod kube-apiserver-k8s-3 -n kube-system -o jsonpath='{.spec.containers[0].command}'`
contains `--advertise-address=$(POD_IP)` and no `--endpoint-reconciler-type`
flag → default lease reconciler, address fixed at start.
- **The address really did move under a running node:**
`get node k8s-3 -o jsonpath='{.metadata.creationTimestamp}...'` →
created `2026-05-09T06:57:12Z`, `InternalIP 10.2.40.139`. The Node object is
109 days old and never re-registered, so the InternalIP changed in place —
the precondition for the stale-masterlease failure.
- **No pinning exists in the repo:** `mgmt0` carries only `DHCPv4Config` in all
three patch files; `opentofu/unifi.tf` is entirely commented-out examples; no
node MAC addresses appear anywhere in the repo (`grep` for MAC-shaped strings
finds only Zigbee IEEEs, a Miele appliance and one check script).
Reviewer check: nothing should change at runtime — the edit is comments inside a
Talos machine-config patch, which is not reconciled by Flux and only read at
`talosctl apply-config` time. `TargetDown` is already resolved and should stay
resolved; the value of the change is that the next occurrence is diagnosable in
minutes instead of nineteen hours.
## Notes
- **Why no functional fix.** Both real remedies are out of reach of a safe,
verifiable repo change:
- *Static `AddressConfig` on `mgmt0`* would need addresses outside the UDM's
DHCP pool, and that pool is not declared anywhere in this repo — I cannot
pick one without risking a conflict. It also rewrites the addressing of all
three etcd members, and the repo already records (`k8s-1.yaml` and §6 of
`RUNBOOK-baremetal-conversion.md`) a network-document change that left a
node with no address on any interface. That is decision rule 7 territory:
a change I cannot verify and whose failure mode is a stranded control plane.
- *DHCP reservations in `opentofu/unifi.tf`* would need the nodes' MAC
addresses; they are not in the repo and this read-only ServiceAccount cannot
obtain them. The file also holds no live resources today, so this would be
the first UniFi-provider resource, applied against the live UDM.
Either is a reasonable follow-up, but both need the owner to supply input the
repo does not contain (free addresses / MACs), so I left them out rather than
guess.
- **Clearing a stale entry is inherently a live operation** — restarting the
affected kube-apiserver, e.g. `talosctl -n <node> reboot`. In this incident
that already happened at 18:37Z, which is why the alert resolved.
- **One thing the issue got wrong:** it cites `k8s-3.yaml:81-82` as still
carrying a static `10.99.0.3/29` on the Ceph link. That address was removed in
`96acff3` ("chore(talos): drop the node addresses on the Ceph SAN"); the
current file has a bare `LinkConfig`. The conclusion it drew from that line —
that no interface anywhere is statically addressed — is now *more* true, not
less.
- The issue's own hedge is right: the masterlease itself lives in etcd under
`/registry/masterleases/` and this ServiceAccount has no path to it. I did not
read the leases; I confirmed the flags and the address history that make the
lease explanation the only one consistent with the evidence.
- No alert silencing of any kind: no PrometheusRule, threshold or inhibition was
touched. `TargetDown` remains routed normally.
- The issue's closing paragraph flags a separate active incident
(`ClusterSecretStore/openbao` Kubernetes auth failing, ExternalSecrets and
Flux kustomizations blocked). It is unrelated to this alert and out of scope
here, but it is worth its own issue if one does not exist yet.
Closes #635
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
What this PR does
TargetDown{job="apiserver"}fired for ~19h becausedefault/kuberneteskeptadvertising
10.2.40.168:6443after k8s-3's management DHCP lease moved offthat address while the node kept running. I confirmed the mechanism live:
kube-apiserver is started by Talos with
--advertise-address=$(POD_IP)and no--endpoint-reconciler-typeoverride, so the advertised address is frozen atpod start and the default lease reconciler keeps renewing the masterlease under
the old address until the static pod restarts. The underlying trigger is not
safely fixable in this repo (see Notes), so this PR makes no config change —
it records the failure mode and the operator remedy as a comment on the very
document that causes it,
DHCPv4Config/mgmt0ink8s-1.yaml, which is wherethe repo keeps the reasoning for the whole network layout (
k8s-2/k8s-3pointat that file).
Changes
kubernetes/talos/patches/k8s-1.yaml— comment above themgmt0DHCPv4Configdocument: leases are unreserved and do move under a runningnode; kube-apiserver does not follow, so the
kubernetesEndpoints go staleuntil a restart; the post-re-IP check (
kubectl get endpoints kubernetes -n defaultvskubectl get nodes -o wide) and remedy (talosctl -n <node> reboot); and why static addressing was not adopted here. Comments only — nobehavioural change, nothing for Talos to apply.
Verification
Everything below was observed this run via
./.ci-diag/kubectl.get nodes -o wide→k8s-1 10.2.40.169,k8s-2 10.2.40.212,k8s-3 10.2.40.139;get endpoints kubernetes -n default→ exactly.139, .169, .212. They agree, so no stale entry remains.query?query=up{job="apiserver"}→1for10.2.40.169:6443,10.2.40.212:6443,10.2.40.139:6443. No dead instance in the series.get pod kube-apiserver-k8s-3 -n kube-system -o jsonpath='{.spec.containers[0].command}'contains
--advertise-address=$(POD_IP)and no--endpoint-reconciler-typeflag → default lease reconciler, address fixed at start.
get node k8s-3 -o jsonpath='{.metadata.creationTimestamp}...'→created
2026-05-09T06:57:12Z,InternalIP 10.2.40.139. The Node object is109 days old and never re-registered, so the InternalIP changed in place —
the precondition for the stale-masterlease failure.
mgmt0carries onlyDHCPv4Configin allthree patch files;
opentofu/unifi.tfis entirely commented-out examples; nonode MAC addresses appear anywhere in the repo (
grepfor MAC-shaped stringsfinds only Zigbee IEEEs, a Miele appliance and one check script).
Reviewer check: nothing should change at runtime — the edit is comments inside a
Talos machine-config patch, which is not reconciled by Flux and only read at
talosctl apply-configtime.TargetDownis already resolved and should stayresolved; the value of the change is that the next occurrence is diagnosable in
minutes instead of nineteen hours.
Notes
Why no functional fix. Both real remedies are out of reach of a safe,
verifiable repo change:
AddressConfigonmgmt0would need addresses outside the UDM'sDHCP pool, and that pool is not declared anywhere in this repo — I cannot
pick one without risking a conflict. It also rewrites the addressing of all
three etcd members, and the repo already records (
k8s-1.yamland §6 ofRUNBOOK-baremetal-conversion.md) a network-document change that left anode with no address on any interface. That is decision rule 7 territory:
a change I cannot verify and whose failure mode is a stranded control plane.
opentofu/unifi.tfwould need the nodes' MACaddresses; they are not in the repo and this read-only ServiceAccount cannot
obtain them. The file also holds no live resources today, so this would be
the first UniFi-provider resource, applied against the live UDM.
Either is a reasonable follow-up, but both need the owner to supply input the
repo does not contain (free addresses / MACs), so I left them out rather than
guess.
Clearing a stale entry is inherently a live operation — restarting the
affected kube-apiserver, e.g.
talosctl -n <node> reboot. In this incidentthat already happened at 18:37Z, which is why the alert resolved.
One thing the issue got wrong: it cites
k8s-3.yaml:81-82as stillcarrying a static
10.99.0.3/29on the Ceph link. That address was removed in96acff3("chore(talos): drop the node addresses on the Ceph SAN"); thecurrent file has a bare
LinkConfig. The conclusion it drew from that line —that no interface anywhere is statically addressed — is now more true, not
less.
The issue's own hedge is right: the masterlease itself lives in etcd under
/registry/masterleases/and this ServiceAccount has no path to it. I did notread the leases; I confirmed the flags and the address history that make the
lease explanation the only one consistent with the evidence.
No alert silencing of any kind: no PrometheusRule, threshold or inhibition was
touched.
TargetDownremains routed normally.The issue's closing paragraph flags a separate active incident
(
ClusterSecretStore/openbaoKubernetes auth failing, ExternalSecrets andFlux kustomizations blocked). It is unrelated to this alert and out of scope
here, but it is worth its own issue if one does not exist yet.
Closes #635