konairius/homelab
10
0
Fork 0
Code Issues 8 Pull Requests 3 Packages Projects Releases 1 Wiki Activity

[alert] KubeJobFailed (openbao) #554

New Issue
Closed
opened 2026-08-22 02:36:55 +00:00 by alert-monitor-bot · 1 comment
alert-monitor-bot commented 2026-08-22 02:36:55 +00:00
Member
Copy Link
Copy Source

What is firing

  • Alert: KubeJobFailed, severity warning, namespace openbao
  • Failed Job: claude-oauth-secret-sync-29785100 (from CronJob claude-oauth-secret-sync), reason BackoffLimitExceeded
  • Fired: 2026-08-19 02:31 UTC → 2026-08-22 02:20 UTC — ~3 days continuously

The pipeline classified this as a "blip" (resolved within the lookback window). That framing is misleading and is the main reason this is filed rather than suppressed — see Why this is not noise.

Evidence

kube_job_status_failed{namespace="openbao"} over 7 days (Prometheus query_range via the apiserver proxy) returns exactly one series for the whole week:

kube_job_status_failed{job_name="claude-oauth-secret-sync-29785100",
                       namespace="openbao", reason="BackoffLimitExceeded"} = 1

The Job pods are long gone, but their logs are in Loki ({namespace="openbao",pod=~"claude-oauth-secret-sync-29785100.*"}). Both attempts produced a single line and nothing else:

02:20:01.13  claude-oauth-secret-sync-29785100-w5jz4  error: io: failed to lookup address information: Name does not resolve
02:20:12.10  claude-oauth-secret-sync-29785100-qvcn2  error: io: failed to lookup address information: Name does not resolve

backoffLimit: 1 means 2 attempts total, ~11 s apart — both landed inside the same short window and the Job was marked failed.

context.txt does not corroborate any current symptom (no non-Running pods, all 3 nodes Ready, no Flux kustomization/helmrelease unready). That is expected: the condition is historical.

Why the alert "resolved" — it was not fixed

The Job finished (last container teardown) at 2026-08-19T02:20:13Z. The CronJob sets ttlSecondsAfterFinished: 259200 (exactly 3 d), so the TTL controller deleted the Job object at 2026-08-22T02:20:13Z. The ALERTS series' last firing sample is 2026-08-22T02:20:20Z and it is absent by 02:20:50Z.

The alert cleared because the evidence was garbage-collected, not because anything recovered. It also routes to the email receiver (kubernetes/apps/monitoring/helmrelease-prometheus.yaml — only Watchdog and KubeCPUOvercommit go to null), so it was an emailing alert that sat unhandled for three days.

Root cause

Confirmed: the container's first outbound name resolution failed. Not confirmed beyond that — the following were each ruled out with a command:

Candidate Ruled out by
Cluster DNS outage coredns_dns_requests_total steady ~47/s across the window; SERVFAIL rate 0 at 02:20 (the only spike was ~90 min earlier, 00:46–00:50); 2/2 CoreDNS pods Ready
Node outage kube_node_status_condition{condition="Ready",status="true"} = 3 throughout
Network policy blocking DNS kubernetes/infrastructure/network-policies/openbao-ns.yaml — the namespace-wide default-deny-ingress policy grants egress to kube-system/k8s-app: kube-dns on 53 UDP+TCP; Cilium policies are additive, so rotator-claude-oauth-egress narrowing to woodpecker:8000 does not remove it
Renamed/missing target service openbao.openbao.svc (39 d old) and woodpecker.woodpecker.svc (created 2026-08-18T21:25:04Z, ~5 h before the failure) both existed
Cilium datapath not yet programmed cilium-agent logs: endpoint 101 → "Reloaded endpoint BPF program" 02:20:00.755, "Successful endpoint creation" 02:20:00.758 — 380 ms before the app's DNS failure at 02:20:01.13. Same ordering on the retry. The datapath was ready.

Remaining hypothesis (unverified): a transient resolver failure local to those two pods. The error text corresponds to getaddrinfo EAI_NONAME ("name does not exist") rather than a timeout, which is odd given the names existed and CoreDNS was healthy — a stale negative answer or a search-path traversal quirk is plausible but I could not confirm it retroactively. No other workload cluster-wide logged the same string in a 3.3 h window around the failure ({namespace=~".+"} |= "does not resolve" over Loki), so it was isolated to this Job.

Impact was bounded: the runs on 2026-08-20, 08-21 and 08-22 all succeeded (status.lastSuccessfulTime: 2026-08-22T02:20:03Z). The Claude OAuth token propagated to the Woodpecker org secret a day late; nothing is currently broken.

Where to fix it

This is not a manifest bug — the configuration is correct and the same manifest has worked every day since. The actionable repo change is resilience:

  • kubernetes/infrastructure/openbao/rotation/claude-oauth.yaml:54 — backoffLimit: 1 gives only two attempts ~11 s apart, so any sub-minute DNS transient burns the entire day's sync and leaves a failed Job (and a firing alert) for the full 3-day TTL. Raising it to ~3–4 would let the Job ride out a transient; activeDeadlineSeconds: 300 (line 55) still bounds total runtime.
  • The same backoffLimit: 1 pattern is used by the sibling rotators in kubernetes/infrastructure/openbao/rotation/ — worth applying consistently.

No live-cluster intervention is needed; the Job already recovered on its own schedule. To force an immediate sync at any time:
kubectl -n openbao create job --from=cronjob/claude-oauth-secret-sync claude-oauth-sync-now

Why this is not noise

The blip's reconstructed labels are only alertname, job, namespace, severity — job_name was dropped. A noise verdict on fingerprint b2940903be750f09 would therefore permanently suppress every future KubeJobFailed in the openbao namespace, which covers the daily openbao-raft-snapshot backup Job and 15 secret-rotation CronJobs. That blast radius is unacceptable for a one-off transient.

Secondary observation for the monitoring pipeline: an alert that fires for 3 days and is only noticed in the 11-minute window after it self-clears indicates the triage scan can miss long-running alerts entirely — worth a separate look.

Note: the CI checkout is shallow (1 commit), so no repo history analysis was possible.

Filed automatically by the alert-monitor pipeline (2026-08-22). The issue-fix pipeline may pick this up and open a PR.

## What is firing - **Alert:** `KubeJobFailed`, severity `warning`, namespace `openbao` - **Failed Job:** `claude-oauth-secret-sync-29785100` (from CronJob `claude-oauth-secret-sync`), reason `BackoffLimitExceeded` - **Fired:** 2026-08-19 02:31 UTC → 2026-08-22 02:20 UTC — **~3 days continuously** The pipeline classified this as a "blip" (resolved within the lookback window). That framing is misleading and is the main reason this is filed rather than suppressed — see *Why this is not noise*. ## Evidence `kube_job_status_failed{namespace="openbao"}` over 7 days (Prometheus `query_range` via the apiserver proxy) returns exactly one series for the whole week: ``` kube_job_status_failed{job_name="claude-oauth-secret-sync-29785100", namespace="openbao", reason="BackoffLimitExceeded"} = 1 ``` The Job pods are long gone, but their logs are in Loki (`{namespace="openbao",pod=~"claude-oauth-secret-sync-29785100.*"}`). Both attempts produced a single line and nothing else: ``` 02:20:01.13 claude-oauth-secret-sync-29785100-w5jz4 error: io: failed to lookup address information: Name does not resolve 02:20:12.10 claude-oauth-secret-sync-29785100-qvcn2 error: io: failed to lookup address information: Name does not resolve ``` `backoffLimit: 1` means 2 attempts total, ~11 s apart — both landed inside the same short window and the Job was marked failed. `context.txt` does **not** corroborate any current symptom (no non-Running pods, all 3 nodes Ready, no Flux kustomization/helmrelease unready). That is expected: the condition is historical. ## Why the alert "resolved" — it was not fixed The Job finished (last container teardown) at `2026-08-19T02:20:13Z`. The CronJob sets `ttlSecondsAfterFinished: 259200` (exactly 3 d), so the TTL controller deleted the Job object at `2026-08-22T02:20:13Z`. The `ALERTS` series' last firing sample is `2026-08-22T02:20:20Z` and it is absent by `02:20:50Z`. The alert cleared because **the evidence was garbage-collected**, not because anything recovered. It also routes to the `email` receiver (`kubernetes/apps/monitoring/helmrelease-prometheus.yaml` — only `Watchdog` and `KubeCPUOvercommit` go to `null`), so it was an emailing alert that sat unhandled for three days. ## Root cause **Confirmed:** the container's first outbound name resolution failed. **Not** confirmed beyond that — the following were each ruled out with a command: | Candidate | Ruled out by | |---|---| | Cluster DNS outage | `coredns_dns_requests_total` steady ~47/s across the window; `SERVFAIL` rate 0 at 02:20 (the only spike was ~90 min earlier, 00:46–00:50); 2/2 CoreDNS pods Ready | | Node outage | `kube_node_status_condition{condition="Ready",status="true"}` = 3 throughout | | Network policy blocking DNS | `kubernetes/infrastructure/network-policies/openbao-ns.yaml` — the namespace-wide `default-deny-ingress` policy grants egress to `kube-system`/`k8s-app: kube-dns` on 53 UDP+TCP; Cilium policies are additive, so `rotator-claude-oauth-egress` narrowing to woodpecker:8000 does not remove it | | Renamed/missing target service | `openbao.openbao.svc` (39 d old) and `woodpecker.woodpecker.svc` (created `2026-08-18T21:25:04Z`, ~5 h *before* the failure) both existed | | Cilium datapath not yet programmed | cilium-agent logs: endpoint `101` → "Reloaded endpoint BPF program" 02:20:00.755, "Successful endpoint creation" 02:20:00.758 — **380 ms before** the app's DNS failure at 02:20:01.13. Same ordering on the retry. The datapath was ready. | **Remaining hypothesis (unverified):** a transient resolver failure local to those two pods. The error text corresponds to `getaddrinfo` `EAI_NONAME` ("name does not exist") rather than a timeout, which is odd given the names existed and CoreDNS was healthy — a stale negative answer or a search-path traversal quirk is plausible but I could not confirm it retroactively. No other workload cluster-wide logged the same string in a 3.3 h window around the failure (`{namespace=~".+"} |= "does not resolve"` over Loki), so it was isolated to this Job. **Impact was bounded:** the runs on 2026-08-20, 08-21 and 08-22 all succeeded (`status.lastSuccessfulTime: 2026-08-22T02:20:03Z`). The Claude OAuth token propagated to the Woodpecker org secret a day late; nothing is currently broken. ## Where to fix it This is **not a manifest bug** — the configuration is correct and the same manifest has worked every day since. The actionable repo change is resilience: - `kubernetes/infrastructure/openbao/rotation/claude-oauth.yaml:54` — `backoffLimit: 1` gives only two attempts ~11 s apart, so any sub-minute DNS transient burns the entire day's sync and leaves a failed Job (and a firing alert) for the full 3-day TTL. Raising it to ~3–4 would let the Job ride out a transient; `activeDeadlineSeconds: 300` (line 55) still bounds total runtime. - The same `backoffLimit: 1` pattern is used by the sibling rotators in `kubernetes/infrastructure/openbao/rotation/` — worth applying consistently. No live-cluster intervention is needed; the Job already recovered on its own schedule. To force an immediate sync at any time: `kubectl -n openbao create job --from=cronjob/claude-oauth-secret-sync claude-oauth-sync-now` ## Why this is not noise The blip's reconstructed labels are only `alertname`, `job`, `namespace`, `severity` — **`job_name` was dropped**. A `noise` verdict on fingerprint `b2940903be750f09` would therefore permanently suppress *every* future `KubeJobFailed` in the `openbao` namespace, which covers the daily `openbao-raft-snapshot` backup Job and 15 secret-rotation CronJobs. That blast radius is unacceptable for a one-off transient. Secondary observation for the monitoring pipeline: an alert that fires for 3 days and is only noticed in the 11-minute window after it self-clears indicates the triage scan can miss long-running alerts entirely — worth a separate look. *Note: the CI checkout is shallow (1 commit), so no repo history analysis was possible.* --- <!-- alert-monitor:fingerprint=b2940903be750f09 --> _Filed automatically by the alert-monitor pipeline (2026-08-22). The issue-fix pipeline may pick this up and open a PR._
issue-fix-bot commented 2026-08-22 02:45:07 +00:00
Member
Copy Link
Copy Source

🤖 Opened k.renner/homelab#555 with a proposed fix.

🤖 Opened https://gitea.renner.uno/k.renner/homelab/pulls/555 with a proposed fix.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#554