fix: [alert] KubeJobFailed (velero) (#439) #440

Merged
k.renner merged 1 commits from alert-fix/issue-439 into master 2026-07-30 07:12:15 +00:00
Member

What this PR does

KubeJobFailed is firing for the Velero Kopia repo-maintenance Job
apps-default-kopia-maintain-job-1785374478669 (velero). The Job did not fail
because of Velero or the object store — it failed because cluster DNS
SERVFAILed the Hetzner S3 endpoint during a ~4-minute brownout of the
upstream resolver on 2026-07-30 01:22–01:26Z. Velero's repo-maintenance Jobs
run with backoffLimit: 0, so the first transient failure is terminal and the
Job object stays Failed — and the alert keeps firing — indefinitely.

The .:53 server block in the CoreDNS Corefile has cache 30 with no
serve_stale, so it has no resilience to an upstream blip: every non-renner.uno
name SERVFAILs cluster-wide for the duration. The neighbouring renner.uno:53
block was already hardened against this exact pathology (issue #393). This PR
adds serve_stale 1h immediate to the .:53 cache block so a multi-minute
upstream outage is served from expired cache instead of SERVFAILing.

Changes

  • kubernetes/infrastructure/coredns-config/configmap.yaml — add
    serve_stale 1h immediate to the .:53 cache 30 block, plus a comment
    recording the incident and why cluster.local is unaffected. The existing
    disable success cluster.local / disable denial cluster.local lines are
    kept unchanged.

Verification

Verified live this run:

  • Alert is still firing. Prometheus API via apiserver proxy,
    query=ALERTS{alertname="KubeJobFailed",namespace="velero"} →
    alertstate="firing", job_name="apps-default-kopia-maintain-job-1785374478669".
  • The Job is terminally failed and no-retry.
    kubectl get job -n velero -o custom-columns=... → that Job is the only one
    with FAILED=1 and BACKOFF=0; every other maintenance Job in the list
    (28 of them, including three later apps-default-* runs at 01:26 and 02:31)
    shows COMPLETIONS=1. So the fault was transient and maintenance self-recovered.
  • The literal error. Loki, {namespace="velero"} |= "server misbehaving" over
    the window → error retrieving storage config from bucket "renner-uno-k8s-velero": … lookup fsn1.your-objectstorage.com on [fd10:96::a]:53: server misbehaving.
    fd10:96::a is the in-cluster kube-dns ClusterIP, so this is cluster DNS, not Hetzner.
  • It was cluster-wide, not velero-specific. Loki,
    {namespace="kube-system",pod=~"coredns.*"} |= "SERVFAIL" over 01:20–01:30Z →
    SERVFAILs across unrelated public names from unrelated clients, e.g.
    "A IN api.getbring.com.", "AAAA IN stun.miwifi.com.",
    "AAAA IN authentik.error-reporting.a7k.io.". Confirms upstream resolution,
    not one endpoint, was broken.
  • The live Corefile matches the repo file byte-for-byte before this change
    (kubectl get cm -n kube-system coredns -o jsonpath='{.data.Corefile}'), and the
    ConfigMap carries kustomize.toolkit.fluxcd.io/name: infrastructure, so the repo
    is the source of truth and this edit will actually take effect.
  • The directive is supported. CoreDNS is registry.k8s.io/coredns/coredns:v1.13.2;
    serve_stale is already in use in the renner.uno:53 block of the running config.
  • The edited file still renders a valid Corefile —
    kubectl create --dry-run=client -o jsonpath='{.data.Corefile}' -f <file>.

For the reviewer, after Flux reconciles: the .:53 block has reload, so CoreDNS
picks the Corefile up within ~2 minutes with no restart. Confirm with
kubectl get cm -n kube-system coredns -o jsonpath='{.data.Corefile}' and
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=20 (expect a
Reloading complete line, and no plugin parse errors). At the next upstream
brownout, {namespace="kube-system",pod=~"coredns.*"} |= "SERVFAIL" in Loki
should stay near zero for names that were previously resolved.

This PR does not clear the current alert. The Job object must be deleted by
a human; backoffLimit: 0 is set by Velero's repo-maintenance controller and is
not exposed by anything in this repo:

kubectl delete job -n velero apps-default-kopia-maintain-job-1785374478669

Notes

  • Scope. I deliberately did not touch the Velero HelmRelease, the
    velero-repo-maintenance ConfigMap, or the KubeJobFailed rule. The Job's
    backoffLimit: 0 is controller-generated and the maintenance-job ConfigMap
    ({"keepLatestMaintenanceJobs": 3}) does not expose a retry knob, so there is
    no repo-side way to make these Jobs retry. Fixing the DNS fragility is the
    only durable repo-level change available.
  • One thing the issue got wrong. It stated that most SERVFAILs "return
    instantly (0.000072605s), i.e. CoreDNS's forward had marked the upstream
    unhealthy and was failing fast." In the sample I pulled, the durations are
    genuine upstream timeouts — 4.001828712s, 4.002017976s, 6.002933186s,
    2.00212761s. This does not change the fix (serve_stale covers both cases),
    but it means clients were also blocking for seconds during the burst, not
    just erroring quickly.
  • Why immediate rather than the more conservative verify. verify
    re-queries upstream before serving stale, which keeps full query pressure on a
    resolver that is likely failing because of load (the renner.uno block's
    comment cites "the UDM's dnsmasq refuses connections in bursts"). immediate
    serves from cache and refreshes in the background, shedding load and matching
    the existing precedent in this same file. Trade-off: a client can receive an
    answer up to one refresh-cycle stale in normal operation. This is the standard
    RFC 8767 stale-while-revalidate behaviour.
  • Why this is safe for in-cluster DNS. serve_stale only serves what the
    cache holds, and disable success cluster.local / disable denial cluster.local
    mean cluster.local is never cached — so Service and Pod records can never be
    served stale. The residual effect is on cached in-addr.arpa/ip6.arpa PTR
    records that fall through to upstream, where a stale answer is cosmetic
    (logging/reverse lookups).
  • Not addressed — the actual upstream fault. The issue hypothesises the UDM
    at 10.2.40.1 reached via Talos host DNS (hostDNS.enabled: true,
    forwardKubeDNSToHost: true, no explicit nameservers: in the repo). I could
    not verify this: the diagnostic ServiceAccount has no node-level access.
    Settling it needs talosctl -n k8s-1 read /etc/resolv.conf and
    talosctl -n k8s-1 logs around 01:22–01:26Z. This change makes the cluster
    tolerate the brownout; it does not stop the brownout. If the UDM is confirmed,
    the durable fix is UDM-side capacity/config, or pinning redundant
    nameservers: into kubernetes/talos/patches/.
  • I also left the issue's "optional hardening" (a redundant upstream or explicit
    policy on forward .) alone. It would partially bypass Talos host DNS, which
    is a deliberate choice in kubernetes/talos/patches/cluster.yaml, and I cannot
    verify the blast radius from here.

Closes #439

## What this PR does `KubeJobFailed` is firing for the Velero Kopia repo-maintenance Job `apps-default-kopia-maintain-job-1785374478669` (velero). The Job did not fail because of Velero or the object store — it failed because cluster DNS SERVFAILed the Hetzner S3 endpoint during a ~4-minute brownout of the *upstream* resolver on 2026-07-30 01:22–01:26Z. Velero's repo-maintenance Jobs run with `backoffLimit: 0`, so the first transient failure is terminal and the Job object stays `Failed` — and the alert keeps firing — indefinitely. The `.:53` server block in the CoreDNS Corefile has `cache 30` with no `serve_stale`, so it has no resilience to an upstream blip: every non-`renner.uno` name SERVFAILs cluster-wide for the duration. The neighbouring `renner.uno:53` block was already hardened against this exact pathology (issue #393). This PR adds `serve_stale 1h immediate` to the `.:53` cache block so a multi-minute upstream outage is served from expired cache instead of SERVFAILing. ## Changes - `kubernetes/infrastructure/coredns-config/configmap.yaml` — add `serve_stale 1h immediate` to the `.:53` `cache 30` block, plus a comment recording the incident and why cluster.local is unaffected. The existing `disable success cluster.local` / `disable denial cluster.local` lines are kept unchanged. ## Verification Verified live this run: - **Alert is still firing.** Prometheus API via apiserver proxy, `query=ALERTS{alertname="KubeJobFailed",namespace="velero"}` → `alertstate="firing"`, `job_name="apps-default-kopia-maintain-job-1785374478669"`. - **The Job is terminally failed and no-retry.** `kubectl get job -n velero -o custom-columns=...` → that Job is the only one with `FAILED=1` and `BACKOFF=0`; **every other** maintenance Job in the list (28 of them, including three later `apps-default-*` runs at 01:26 and 02:31) shows `COMPLETIONS=1`. So the fault was transient and maintenance self-recovered. - **The literal error.** Loki, `{namespace="velero"} |= "server misbehaving"` over the window → `error retrieving storage config from bucket "renner-uno-k8s-velero": … lookup fsn1.your-objectstorage.com on [fd10:96::a]:53: server misbehaving`. `fd10:96::a` is the in-cluster kube-dns ClusterIP, so this is cluster DNS, not Hetzner. - **It was cluster-wide, not velero-specific.** Loki, `{namespace="kube-system",pod=~"coredns.*"} |= "SERVFAIL"` over 01:20–01:30Z → SERVFAILs across unrelated public names from unrelated clients, e.g. `"A IN api.getbring.com."`, `"AAAA IN stun.miwifi.com."`, `"AAAA IN authentik.error-reporting.a7k.io."`. Confirms upstream resolution, not one endpoint, was broken. - **The live Corefile matches the repo file byte-for-byte** before this change (`kubectl get cm -n kube-system coredns -o jsonpath='{.data.Corefile}'`), and the ConfigMap carries `kustomize.toolkit.fluxcd.io/name: infrastructure`, so the repo is the source of truth and this edit will actually take effect. - **The directive is supported.** CoreDNS is `registry.k8s.io/coredns/coredns:v1.13.2`; `serve_stale` is already in use in the `renner.uno:53` block of the running config. - **The edited file still renders a valid Corefile** — `kubectl create --dry-run=client -o jsonpath='{.data.Corefile}' -f <file>`. For the reviewer, after Flux reconciles: the `.:53` block has `reload`, so CoreDNS picks the Corefile up within ~2 minutes with no restart. Confirm with `kubectl get cm -n kube-system coredns -o jsonpath='{.data.Corefile}'` and `kubectl logs -n kube-system -l k8s-app=kube-dns --tail=20` (expect a `Reloading complete` line, and no plugin parse errors). At the next upstream brownout, `{namespace="kube-system",pod=~"coredns.*"} |= "SERVFAIL"` in Loki should stay near zero for names that were previously resolved. **This PR does not clear the current alert.** The Job object must be deleted by a human; `backoffLimit: 0` is set by Velero's repo-maintenance controller and is not exposed by anything in this repo: ``` kubectl delete job -n velero apps-default-kopia-maintain-job-1785374478669 ``` ## Notes - **Scope.** I deliberately did not touch the Velero HelmRelease, the `velero-repo-maintenance` ConfigMap, or the `KubeJobFailed` rule. The Job's `backoffLimit: 0` is controller-generated and the maintenance-job ConfigMap (`{"keepLatestMaintenanceJobs": 3}`) does not expose a retry knob, so there is no repo-side way to make these Jobs retry. Fixing the DNS fragility is the only durable repo-level change available. - **One thing the issue got wrong.** It stated that most SERVFAILs "return instantly (0.000072605s), i.e. CoreDNS's `forward` had marked the upstream unhealthy and was failing fast." In the sample I pulled, the durations are genuine upstream timeouts — `4.001828712s`, `4.002017976s`, `6.002933186s`, `2.00212761s`. This does not change the fix (serve_stale covers both cases), but it means clients were also *blocking* for seconds during the burst, not just erroring quickly. - **Why `immediate` rather than the more conservative `verify`.** `verify` re-queries upstream before serving stale, which keeps full query pressure on a resolver that is likely failing *because* of load (the `renner.uno` block's comment cites "the UDM's dnsmasq refuses connections in bursts"). `immediate` serves from cache and refreshes in the background, shedding load and matching the existing precedent in this same file. Trade-off: a client can receive an answer up to one refresh-cycle stale in normal operation. This is the standard RFC 8767 stale-while-revalidate behaviour. - **Why this is safe for in-cluster DNS.** `serve_stale` only serves what the cache holds, and `disable success cluster.local` / `disable denial cluster.local` mean cluster.local is never cached — so Service and Pod records can never be served stale. The residual effect is on cached `in-addr.arpa`/`ip6.arpa` PTR records that fall through to upstream, where a stale answer is cosmetic (logging/reverse lookups). - **Not addressed — the actual upstream fault.** The issue hypothesises the UDM at `10.2.40.1` reached via Talos host DNS (`hostDNS.enabled: true`, `forwardKubeDNSToHost: true`, no explicit `nameservers:` in the repo). I could not verify this: the diagnostic ServiceAccount has no node-level access. Settling it needs `talosctl -n k8s-1 read /etc/resolv.conf` and `talosctl -n k8s-1 logs` around 01:22–01:26Z. This change makes the cluster tolerate the brownout; it does not stop the brownout. If the UDM is confirmed, the durable fix is UDM-side capacity/config, or pinning redundant `nameservers:` into `kubernetes/talos/patches/`. - I also left the issue's "optional hardening" (a redundant upstream or explicit `policy` on `forward .`) alone. It would partially bypass Talos host DNS, which is a deliberate choice in `kubernetes/talos/patches/cluster.yaml`, and I cannot verify the blast radius from here. Closes #439
issue-fix-bot added 1 commit 2026-07-30 02:45:04 +00:00
fix: [alert] KubeJobFailed (velero) (#439)
ci/woodpecker/pr/pr-review Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
1f29f7221e
review-bot left a comment
Member

🤖 Automated review — model CLI default (complexity: complex), commit 1f29f7221e5df4b46a72f4fb81183098f79c7b74.

Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.


Summary

Adds serve_stale 1h immediate to the .:53 block's cache 30 {} in kubernetes/infrastructure/coredns-config/configmap.yaml, plus an explanatory comment, mirroring the existing hardening already present in the neighbouring renner.uno:53 block (issue #393). This is a small, low-risk, single-directive config change. No blocking issues found.

🟡 Suggestions

  • kubernetes/infrastructure/coredns-config/configmap.yaml:59-65: The new comment (7 lines) duplicates a fair amount of narrative already covered by the PR/issue history (specific timestamp, SERVFAIL count, job name). Incident post-mortems tend to accumulate in Corefile comments over time; consider trimming to "mirrors renner.uno:53 (issue #393); cluster.local is exempt from the cache above so it's never served stale" and letting the issue/PR carry the forensic detail. Not blocking — the existing renner.uno:53 block already sets this precedent of verbose in-file comments, so this is consistent with local convention.

Tests

There's no test harness for Corefile changes in this repo (it's a ConfigMap consumed by Flux/CoreDNS at runtime), and none would reasonably be expected here — this matches the pattern of the prior renner.uno:53 serve_stale change from issue #393, which also shipped without an automated test. The PR body documents manual verification (dry-run render, byte-for-byte diff against the live ConfigMap, and confirmation that serve_stale is supported by the running CoreDNS version), which is appropriate rigor for this kind of change.

I additionally verified independently:

  • The edited Corefile block parses as valid YAML (data.Corefile is a plain block-scalar string; the edit only adds lines inside it).
  • Indentation of the new serve_stale 1h immediate line matches the sibling disable success/denial cluster.local lines exactly (11 spaces), and no tabs are present anywhere in the file.
  • serve_stale [DURATION] [VERIFY|IMMEDIATE] is valid CoreDNS cache plugin syntax, and the syntax/casing used here is identical to the already-running renner.uno:53 block, so there's no new/unproven directive being introduced.
  • disable success cluster.local / disable denial cluster.local are untouched, so cluster.local is never cached and therefore never served stale, consistent with the PR's safety claim.

Recommendation

Looks reasonable — no blocking issues; comment above is optional.

🤖 **Automated review** — model `CLI default` (complexity: complex), commit `1f29f7221e5df4b46a72f4fb81183098f79c7b74`. > Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review. --- ## Summary Adds `serve_stale 1h immediate` to the `.:53` block's `cache 30 {}` in `kubernetes/infrastructure/coredns-config/configmap.yaml`, plus an explanatory comment, mirroring the existing hardening already present in the neighbouring `renner.uno:53` block (issue #393). This is a small, low-risk, single-directive config change. No blocking issues found. ## 🟡 Suggestions - `kubernetes/infrastructure/coredns-config/configmap.yaml:59-65`: The new comment (7 lines) duplicates a fair amount of narrative already covered by the PR/issue history (specific timestamp, SERVFAIL count, job name). Incident post-mortems tend to accumulate in Corefile comments over time; consider trimming to "mirrors `renner.uno:53` (issue #393); cluster.local is exempt from the cache above so it's never served stale" and letting the issue/PR carry the forensic detail. Not blocking — the existing `renner.uno:53` block already sets this precedent of verbose in-file comments, so this is consistent with local convention. ## Tests There's no test harness for Corefile changes in this repo (it's a ConfigMap consumed by Flux/CoreDNS at runtime), and none would reasonably be expected here — this matches the pattern of the prior `renner.uno:53` `serve_stale` change from issue #393, which also shipped without an automated test. The PR body documents manual verification (dry-run render, byte-for-byte diff against the live ConfigMap, and confirmation that `serve_stale` is supported by the running CoreDNS version), which is appropriate rigor for this kind of change. I additionally verified independently: - The edited Corefile block parses as valid YAML (`data.Corefile` is a plain block-scalar string; the edit only adds lines inside it). - Indentation of the new `serve_stale 1h immediate` line matches the sibling `disable success/denial cluster.local` lines exactly (11 spaces), and no tabs are present anywhere in the file. - `serve_stale [DURATION] [VERIFY|IMMEDIATE]` is valid CoreDNS `cache` plugin syntax, and the syntax/casing used here is identical to the already-running `renner.uno:53` block, so there's no new/unproven directive being introduced. - `disable success cluster.local` / `disable denial cluster.local` are untouched, so cluster.local is never cached and therefore never served stale, consistent with the PR's safety claim. ## Recommendation Looks reasonable — no blocking issues; comment above is optional. <!-- pr-review:sha=1f29f7221e5df4b46a72f4fb81183098f79c7b74 -->
k.renner merged commit 42596daf3e into master 2026-07-30 07:12:15 +00:00
Sign in to join this conversation.