On 2026-08-19 the daily claude-oauth-secret-sync Job failed with BackoffLimitExceeded after both of its attempts hit the same resolver error ~11 s apart, and the resulting KubeJobFailed alert emailed and sat firing for three days until the Job's 3-day TTL deleted the evidence. The underlying resolver transient is not reproducible or attributable to any repo bug — the manifest is correct and has synced successfully every other day, and I ruled out the remaining infrastructure explanations live (below). What is a repo-level defect is the retry budget: backoffLimit: 1 gives exactly two attempts inside a ~11 s window, so any sub-minute transient burns the whole day's sync. This raises it to 4, spreading attempts over ~150 s, still inside the existing activeDeadlineSeconds: 300.
The key fact that justifies treating this file differently from its siblings: claude-oauth-secret-sync is the only daily rotator in kubernetes/infrastructure/openbao/rotation/. All 14 sibling CronJobs are quarterly (0 4 1 1,4,7,10 * and offsets), so this one takes roughly 90× as many runs and has 90× the exposure to a per-run transient.
Changes
kubernetes/infrastructure/openbao/rotation/claude-oauth.yaml — backoffLimit: 1 → 4, with a comment recording why this rotator diverges from its siblings (daily cadence) and the incident that motivated it. No other field touched.
Verification
Verified live this run with ./.ci-diag/kubectl:
The failure and its timing — Loki {namespace="openbao",pod=~"claude-oauth-secret-sync-29785100.*"}: exactly two log lines, one per attempt, error: io: failed to lookup address information: Name does not resolve, at 02:20:01.13 (pod …-w5jz4) and 02:20:12.10 (pod …-qvcn2). Confirms the issue's claim: 2 attempts, ~11 s apart, nothing else logged.
Current spec and health — get cronjob claude-oauth-secret-sync -o jsonpath: 20 4 * * * / Europe/Berlin / backoffLimit=1 / activeDeadlineSeconds=300 / ttlSecondsAfterFinished=259200, lastSuccessfulTime=2026-08-22T02:20:03Z. The Job is healthy now; nothing is currently broken.
Cadence claim — grep over rotation/*.yaml: this is the only * * * daily schedule; every sibling is quarterly. Several siblings already use backoffLimit: 2, so a value >1 is established idiom here, not a new pattern.
Manifest validity — kubectl apply --dry-run=client -f … parses the edited file (cronjob.batch/claude-oauth-secret-sync); the accompanying error is only the read-only SA being denied a ServiceAccount read.
Hypotheses I tested that the issue did not cover, both ruled out:
CoreDNS answering NXDOMAIN from a cold informer cache (would produce EAI_NONAME exactly as observed, and would not show up in the issue's SERVFAIL check): get pods -l k8s-app=kube-dns shows both pods started 2026-07-23T13:36:02Z with restarts=0 — 27 days of uptime, so no cold cache. Prometheus sum by (rcode) (increase(coredns_dns_responses_total[2m])) over 02:10–02:30 is flat across the failure (NOERROR ~1000–1500, NXDOMAIN ~4500–5900, SERVFAIL 0 throughout) — no anomaly at 02:20.
Cilium L7 DNS proxy race on endpoint startup: kubernetes/infrastructure/network-policies/openbao-ns.yaml contains no toFQDNs or L7 DNS rules — DNS egress is plain L3/L4 to kube-system/k8s-app: kube-dns on 53. No DNS proxy is in the path, so there is no proxy-rule-programming race to lose.
How a reviewer confirms the fix: after Flux reconciles, kubectl -n openbao get cronjob claude-oauth-secret-sync -o jsonpath='{.spec.jobTemplate.spec.backoffLimit}' should read 4. There is nothing to un-fire — the alert already cleared. The change is preventive: the next single-attempt transient should show up as a retried-then-succeeded Job rather than a KubeJobFailed email. To force a sync at any time: kubectl -n openbao create job --from=cronjob/claude-oauth-secret-sync claude-oauth-sync-now.
Notes
The root cause is still unconfirmed, and this PR does not claim to fix it. I could not explain the resolver failure, and neither could the issue. This is a resilience change that converts a single transient from "lost day + 3-day emailing alert" into "retried and recovered". If the failure recurs despite 5 attempts over 150 s, that is a much stronger signal of a real DNS problem and should be investigated as such — the change makes the next occurrence more diagnostic, not less.
What the issue got right: the Loki evidence, the timing, the TTL explanation for why the alert "resolved", and the backoffLimit recommendation all held up against live checks. Its ruled-out table I did not re-verify line by line, but the two hypotheses it left open I tested and closed.
Where I deliberately diverged from the issue: it suggested applying the same bump to all sibling rotators "for consistency". I did not. They run 4×/year rather than 365×/year, several are already at backoffLimit: 2, and none of them has actually failed — that would be 14 files of unverifiable churn for no observed problem. The daily cadence is precisely what makes this file different, so the divergence is the point and is now documented in the comment.
I did not reduce ttlSecondsAfterFinished: 259200, even though it is why the alert stayed up for three days. Shortening it would make the alert clear faster by deleting the evidence sooner — that is silencing, not fixing, and the long TTL is what made this incident diagnosable at all.
Alternative considered and rejected: adding dnsConfig: { options: [{name: ndots, value: "1"}] }. The observed cluster-wide NXDOMAIN:NOERROR ratio of ~4.3:1 confirms ndots:5 search-path traversal means ~4 sequential lookups per FQDN, i.e. 4× the chance of a per-lookup failure — so this would plausibly help. But it changes resolution semantics for every name the container looks up, I cannot test it from a read-only session, and I have no evidence the failure was search-path related. Per "don't break things you can't verify", I left it out; it is worth considering deliberately if this recurs.
The issue's closing note about the triage scan missing multi-day alerts (it only noticed this one in the 11-minute window after it self-cleared) is a real gap, but it concerns the alert-monitor pipeline, which is outside this PR's edit scope.
## What this PR does
On 2026-08-19 the daily `claude-oauth-secret-sync` Job failed with `BackoffLimitExceeded` after both of its attempts hit the same resolver error ~11 s apart, and the resulting `KubeJobFailed` alert emailed and sat firing for three days until the Job's 3-day TTL deleted the evidence. The underlying resolver transient is **not** reproducible or attributable to any repo bug — the manifest is correct and has synced successfully every other day, and I ruled out the remaining infrastructure explanations live (below). What *is* a repo-level defect is the retry budget: `backoffLimit: 1` gives exactly two attempts inside a ~11 s window, so any sub-minute transient burns the whole day's sync. This raises it to `4`, spreading attempts over ~150 s, still inside the existing `activeDeadlineSeconds: 300`.
The key fact that justifies treating this file differently from its siblings: **`claude-oauth-secret-sync` is the only daily rotator in `kubernetes/infrastructure/openbao/rotation/`.** All 14 sibling CronJobs are quarterly (`0 4 1 1,4,7,10 *` and offsets), so this one takes roughly 90× as many runs and has 90× the exposure to a per-run transient.
## Changes
- `kubernetes/infrastructure/openbao/rotation/claude-oauth.yaml` — `backoffLimit: 1` → `4`, with a comment recording why this rotator diverges from its siblings (daily cadence) and the incident that motivated it. No other field touched.
## Verification
Verified live this run with `./.ci-diag/kubectl`:
- **The failure and its timing** — Loki `{namespace="openbao",pod=~"claude-oauth-secret-sync-29785100.*"}`: exactly two log lines, one per attempt, `error: io: failed to lookup address information: Name does not resolve`, at `02:20:01.13` (pod `…-w5jz4`) and `02:20:12.10` (pod `…-qvcn2`). Confirms the issue's claim: 2 attempts, ~11 s apart, nothing else logged.
- **Current spec and health** — `get cronjob claude-oauth-secret-sync -o jsonpath`: `20 4 * * *` / `Europe/Berlin` / `backoffLimit=1` / `activeDeadlineSeconds=300` / `ttlSecondsAfterFinished=259200`, `lastSuccessfulTime=2026-08-22T02:20:03Z`. The Job is healthy now; nothing is currently broken.
- **Cadence claim** — `grep` over `rotation/*.yaml`: this is the only `* * *` daily schedule; every sibling is quarterly. Several siblings already use `backoffLimit: 2`, so a value >1 is established idiom here, not a new pattern.
- **Manifest validity** — `kubectl apply --dry-run=client -f …` parses the edited file (`cronjob.batch/claude-oauth-secret-sync`); the accompanying error is only the read-only SA being denied a ServiceAccount read.
Hypotheses I tested that the issue did **not** cover, both ruled out:
- **CoreDNS answering NXDOMAIN from a cold informer cache** (would produce `EAI_NONAME` exactly as observed, and would *not* show up in the issue's SERVFAIL check): `get pods -l k8s-app=kube-dns` shows both pods started `2026-07-23T13:36:02Z` with `restarts=0` — 27 days of uptime, so no cold cache. Prometheus `sum by (rcode) (increase(coredns_dns_responses_total[2m]))` over 02:10–02:30 is flat across the failure (NOERROR ~1000–1500, NXDOMAIN ~4500–5900, SERVFAIL 0 throughout) — no anomaly at 02:20.
- **Cilium L7 DNS proxy race on endpoint startup**: `kubernetes/infrastructure/network-policies/openbao-ns.yaml` contains no `toFQDNs` or L7 DNS rules — DNS egress is plain L3/L4 to `kube-system`/`k8s-app: kube-dns` on 53. No DNS proxy is in the path, so there is no proxy-rule-programming race to lose.
**How a reviewer confirms the fix:** after Flux reconciles, `kubectl -n openbao get cronjob claude-oauth-secret-sync -o jsonpath='{.spec.jobTemplate.spec.backoffLimit}'` should read `4`. There is nothing to un-fire — the alert already cleared. The change is preventive: the next single-attempt transient should show up as a retried-then-succeeded Job rather than a `KubeJobFailed` email. To force a sync at any time: `kubectl -n openbao create job --from=cronjob/claude-oauth-secret-sync claude-oauth-sync-now`.
## Notes
- **The root cause is still unconfirmed, and this PR does not claim to fix it.** I could not explain the resolver failure, and neither could the issue. This is a resilience change that converts a single transient from "lost day + 3-day emailing alert" into "retried and recovered". If the failure recurs *despite* 5 attempts over 150 s, that is a much stronger signal of a real DNS problem and should be investigated as such — the change makes the next occurrence more diagnostic, not less.
- **What the issue got right:** the Loki evidence, the timing, the TTL explanation for why the alert "resolved", and the `backoffLimit` recommendation all held up against live checks. Its ruled-out table I did not re-verify line by line, but the two hypotheses it left open I tested and closed.
- **Where I deliberately diverged from the issue:** it suggested applying the same bump to all sibling rotators "for consistency". I did not. They run 4×/year rather than 365×/year, several are already at `backoffLimit: 2`, and none of them has actually failed — that would be 14 files of unverifiable churn for no observed problem. The daily cadence is precisely what makes this file different, so the divergence is the point and is now documented in the comment.
- **I did not reduce `ttlSecondsAfterFinished: 259200`,** even though it is why the alert stayed up for three days. Shortening it would make the alert clear faster by deleting the evidence sooner — that is silencing, not fixing, and the long TTL is what made this incident diagnosable at all.
- **Alternative considered and rejected:** adding `dnsConfig: { options: [{name: ndots, value: "1"}] }`. The observed cluster-wide NXDOMAIN:NOERROR ratio of ~4.3:1 confirms `ndots:5` search-path traversal means ~4 sequential lookups per FQDN, i.e. 4× the chance of a per-lookup failure — so this would plausibly help. But it changes resolution semantics for every name the container looks up, I cannot test it from a read-only session, and I have no evidence the failure was search-path related. Per "don't break things you can't verify", I left it out; it is worth considering deliberately if this recurs.
- The issue's closing note about the triage scan missing multi-day alerts (it only noticed this one in the 11-minute window after it self-cleared) is a real gap, but it concerns the alert-monitor pipeline, which is outside this PR's edit scope.
Closes #554
🤖Automated review — model claude-haiku-4-5-20251001 (complexity: trivial), commit cf63aa61bede4f530359e32dfe1c2b847762340a.
Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.
Summary
This PR increases the retry budget (backoffLimit) for the claude-oauth-secret-sync daily CronJob from 1 to 4, based on an incident where a sub-minute transient resolver error caused the job to fail with only two attempts. The change is well-justified, narrowly scoped to this one daily rotator (which runs ~90× more often than its quarterly siblings), and includes detailed documentation of the rationale and alternatives considered. No blocking issues.
🟢 Nits
Minor — comment length (claude-oauth.yaml:54-61): The 9-line explanatory comment is quite detailed, but this is actually appropriate for a divergence from the established pattern (siblings use 1-2). It documents the incident, the exponential backoff strategy, and why alternatives were rejected. No changes needed.
Tests
Not applicable — this is a Kubernetes manifest configuration change. The PR body includes verification steps that can be performed post-merge by checking the live cluster state and testing with manual job creation. No automated tests are expected for this type of change.
Recommendation
Looks reasonable — no blocking issues; the detailed context above is purely informational.
🤖 **Automated review** — model `claude-haiku-4-5-20251001` (complexity: trivial), commit `cf63aa61bede4f530359e32dfe1c2b847762340a`.
> Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review.
---
## Summary
This PR increases the retry budget (`backoffLimit`) for the `claude-oauth-secret-sync` daily CronJob from 1 to 4, based on an incident where a sub-minute transient resolver error caused the job to fail with only two attempts. The change is well-justified, narrowly scoped to this one daily rotator (which runs ~90× more often than its quarterly siblings), and includes detailed documentation of the rationale and alternatives considered. No blocking issues.
## 🟢 Nits
**Minor — comment length** (claude-oauth.yaml:54-61): The 9-line explanatory comment is quite detailed, but this is actually appropriate for a divergence from the established pattern (siblings use 1-2). It documents the incident, the exponential backoff strategy, and why alternatives were rejected. No changes needed.
## Tests
Not applicable — this is a Kubernetes manifest configuration change. The PR body includes verification steps that can be performed post-merge by checking the live cluster state and testing with manual job creation. No automated tests are expected for this type of change.
## Recommendation
Looks reasonable — no blocking issues; the detailed context above is purely informational.
<!-- pr-review:sha=cf63aa61bede4f530359e32dfe1c2b847762340a -->
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
review-bot
What this PR does
On 2026-08-19 the daily
claude-oauth-secret-syncJob failed withBackoffLimitExceededafter both of its attempts hit the same resolver error ~11 s apart, and the resultingKubeJobFailedalert emailed and sat firing for three days until the Job's 3-day TTL deleted the evidence. The underlying resolver transient is not reproducible or attributable to any repo bug — the manifest is correct and has synced successfully every other day, and I ruled out the remaining infrastructure explanations live (below). What is a repo-level defect is the retry budget:backoffLimit: 1gives exactly two attempts inside a ~11 s window, so any sub-minute transient burns the whole day's sync. This raises it to4, spreading attempts over ~150 s, still inside the existingactiveDeadlineSeconds: 300.The key fact that justifies treating this file differently from its siblings:
claude-oauth-secret-syncis the only daily rotator inkubernetes/infrastructure/openbao/rotation/. All 14 sibling CronJobs are quarterly (0 4 1 1,4,7,10 *and offsets), so this one takes roughly 90× as many runs and has 90× the exposure to a per-run transient.Changes
kubernetes/infrastructure/openbao/rotation/claude-oauth.yaml—backoffLimit: 1→4, with a comment recording why this rotator diverges from its siblings (daily cadence) and the incident that motivated it. No other field touched.Verification
Verified live this run with
./.ci-diag/kubectl:{namespace="openbao",pod=~"claude-oauth-secret-sync-29785100.*"}: exactly two log lines, one per attempt,error: io: failed to lookup address information: Name does not resolve, at02:20:01.13(pod…-w5jz4) and02:20:12.10(pod…-qvcn2). Confirms the issue's claim: 2 attempts, ~11 s apart, nothing else logged.get cronjob claude-oauth-secret-sync -o jsonpath:20 4 * * */Europe/Berlin/backoffLimit=1/activeDeadlineSeconds=300/ttlSecondsAfterFinished=259200,lastSuccessfulTime=2026-08-22T02:20:03Z. The Job is healthy now; nothing is currently broken.grepoverrotation/*.yaml: this is the only* * *daily schedule; every sibling is quarterly. Several siblings already usebackoffLimit: 2, so a value >1 is established idiom here, not a new pattern.kubectl apply --dry-run=client -f …parses the edited file (cronjob.batch/claude-oauth-secret-sync); the accompanying error is only the read-only SA being denied a ServiceAccount read.Hypotheses I tested that the issue did not cover, both ruled out:
EAI_NONAMEexactly as observed, and would not show up in the issue's SERVFAIL check):get pods -l k8s-app=kube-dnsshows both pods started2026-07-23T13:36:02Zwithrestarts=0— 27 days of uptime, so no cold cache. Prometheussum by (rcode) (increase(coredns_dns_responses_total[2m]))over 02:10–02:30 is flat across the failure (NOERROR ~1000–1500, NXDOMAIN ~4500–5900, SERVFAIL 0 throughout) — no anomaly at 02:20.kubernetes/infrastructure/network-policies/openbao-ns.yamlcontains notoFQDNsor L7 DNS rules — DNS egress is plain L3/L4 tokube-system/k8s-app: kube-dnson 53. No DNS proxy is in the path, so there is no proxy-rule-programming race to lose.How a reviewer confirms the fix: after Flux reconciles,
kubectl -n openbao get cronjob claude-oauth-secret-sync -o jsonpath='{.spec.jobTemplate.spec.backoffLimit}'should read4. There is nothing to un-fire — the alert already cleared. The change is preventive: the next single-attempt transient should show up as a retried-then-succeeded Job rather than aKubeJobFailedemail. To force a sync at any time:kubectl -n openbao create job --from=cronjob/claude-oauth-secret-sync claude-oauth-sync-now.Notes
backoffLimitrecommendation all held up against live checks. Its ruled-out table I did not re-verify line by line, but the two hypotheses it left open I tested and closed.backoffLimit: 2, and none of them has actually failed — that would be 14 files of unverifiable churn for no observed problem. The daily cadence is precisely what makes this file different, so the divergence is the point and is now documented in the comment.ttlSecondsAfterFinished: 259200, even though it is why the alert stayed up for three days. Shortening it would make the alert clear faster by deleting the evidence sooner — that is silencing, not fixing, and the long TTL is what made this incident diagnosable at all.dnsConfig: { options: [{name: ndots, value: "1"}] }. The observed cluster-wide NXDOMAIN:NOERROR ratio of ~4.3:1 confirmsndots:5search-path traversal means ~4 sequential lookups per FQDN, i.e. 4× the chance of a per-lookup failure — so this would plausibly help. But it changes resolution semantics for every name the container looks up, I cannot test it from a read-only session, and I have no evidence the failure was search-path related. Per "don't break things you can't verify", I left it out; it is worth considering deliberately if this recurs.Closes #554
🤖 Automated review — model
claude-haiku-4-5-20251001(complexity: trivial), commitcf63aa61bede4f530359e32dfe1c2b847762340a.Summary
This PR increases the retry budget (
backoffLimit) for theclaude-oauth-secret-syncdaily CronJob from 1 to 4, based on an incident where a sub-minute transient resolver error caused the job to fail with only two attempts. The change is well-justified, narrowly scoped to this one daily rotator (which runs ~90× more often than its quarterly siblings), and includes detailed documentation of the rationale and alternatives considered. No blocking issues.🟢 Nits
Minor — comment length (claude-oauth.yaml:54-61): The 9-line explanatory comment is quite detailed, but this is actually appropriate for a divergence from the established pattern (siblings use 1-2). It documents the incident, the exponential backoff strategy, and why alternatives were rejected. No changes needed.
Tests
Not applicable — this is a Kubernetes manifest configuration change. The PR body includes verification steps that can be performed post-merge by checking the live cluster state and testing with manual job creation. No automated tests are expected for this type of change.
Recommendation
Looks reasonable — no blocking issues; the detailed context above is purely informational.