external-dns-unifi in namespace apps is in CrashLoopBackOff — the external-dns
container exits 1 roughly every 70s of uptime while the webhook sidecar stays up. The
issue attributed this to the UDM's intermittent 401: User not authenticated responses
and suggested the UNIFI_API_KEY may be stale. That diagnosis is wrong. external-dns
v0.21.0 classifies a webhook HTTP 5xx (which is what the sidecar returns when the UDM
401s or 502s) as a soft error and retries on the next cycle — verified live this run.
The only failure that is fatal is a client-side timeout on GET /records, which is a
hard error → log.Fatalf → exit 1 → CrashLoopBackOff. The UDM's static-dns endpoint
has simply gotten slower than the current --webhook-provider-read-timeout=60s ceiling
(reads observed completing at ~70s), so this raises that timeout to 180s.
Changes
kubernetes/infrastructure/external-dns-unifi/helmrelease.yaml — raise --webhook-provider-read-timeout from 60s to 180s, and rewrite the adjacent
comment to record the soft-error-vs-hard-error distinction that makes this the right
knob (the previous comment predated the soft-error behaviour and only described the
slow-read case).
Verification
Everything below was observed live this run via ./.ci-diag/kubectl.
The crash loop is ongoing (the alert had already "resolved" in Alertmanager): get pods -n apps -o wide → external-dns-unifi-89f464b66-98jjw 1/2 CrashLoopBackOff 18 (2m37s ago) 5h48m
(19 restarts by the end of this run, up from the 13 recorded in the issue).
Only the external-dns container crashes; the sidecar is healthy. get pod ... -o jsonpath={.status.containerStatuses} → external-dns: exitCode:1, reason:"Error", startedAt:11:37:51Z, finishedAt:11:39:01Z; webhook: restarts=0, running since 05:52:49Z. Container lifetime is 70s — ~10s of
startup plus exactly one 60s read timeout.
The fatal error is a client timeout, not an auth error. logs ... -c external-dns --previous → level=fatal msg="Failed to do run once: Get \"http://localhost:8888/records\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
The same log's config dump confirms the current setting is live: WebhookProviderReadTimeout:1m0s.
UDM errors are survivable; this is the key correction. logs ... -c external-dns --tail=6 --timestamps on the current container → 11:41:50 level=error ... "soft error\nfailed to get records with code 500 (consecutive soft errors: 1)" 11:42:50 level=error ... (consecutive soft errors: 2) 11:43:51 level=info msg="All records are already up to date" 11:43:51 level=info msg="Reconciliation succeeded after 2 consecutive soft errors"
Those two soft errors correspond 1:1 (same second) with the sidecar's UDM failures at 11:41:50 and 11:42:50 (returned 502). So a 401/502 costs one skipped cycle, not a
restart.
Slow reads are what kill it, and they land just over 60s. logs ... -c webhook --since=12m --timestamps shows entries where the sidecar successfully retrieved all 128 records but the client had already gone: 11:33:00 "provider retrieved records" count:128 immediately followed by error encoding records ... write: broken pipe. Same pattern at 11:30:47, 11:30:51, 11:35:01. Each abandon happens 60s after the request, so the true read time is ~70s —
over the ceiling, but only just.
How a reviewer confirms the fix. After Flux reconciles the HelmRelease and the pod is
recreated:
./.ci-diag/kubectl get pods -n apps -l app.kubernetes.io/instance=external-dns-unifi —
expect 2/2 Running with the restart count flat over 30+ min.
./.ci-diag/kubectl logs deploy/external-dns-unifi -n apps -c external-dns --tail=30 —
expect WebhookProviderReadTimeout:3m0s in the config dump, and any remaining UDM
trouble to appear as soft error ... code 500 lines followed by Reconciliation succeeded, never as level=fatal.
KubePodCrashLooping for apps/external-dns-unifi should stop firing. DNS
reconciliation is already working between crashes (All records are already up to date),
so no record churn is expected.
Notes
What the issue got wrong. It proposed that the UNIFI_API_KEY may be stale/rotated.
It is not: the same key returns count:128 successfully on most cycles and did so at 11:43:51, minutes after a 401. A revoked key would fail every time. I did not read the
Secret (the diag ServiceAccount has no Secrets access) — this is inferred from the
interleaved successes, which is sufficient to rule out a hard revocation. The issue also
framed the 401s as the cause of the crash loop; evidence item 4 shows they are not.
This does not make the UDM healthy.https://udm.renner.uno really is returning
intermittent 401: User not authenticated and 502, and its static-dns read time has
roughly doubled since the 60s value was chosen. That is outside this repo. It is still
worth a human look at the UDM's own health/logs (controller restarts, backups, or load
around those timestamps) — but it is now a degraded-reconcile problem, not a
crash-loop problem, and it will no longer page.
This is a ceiling, and the UDM is trending slower. If reads keep growing, this will
need revisiting. A more durable option is --provider-cache-time (currently ProviderCacheTime:0s), which would cut UDM polling from ~60/hour to a handful and
shrink the exposure window proportionally. I did not do it here: it changes reconcile
semantics (out-of-band UDM edits would be picked up with a delay), and I could not verify
the cache-invalidation-on-apply behaviour from read-only tooling. Worth considering
separately if the timeout bump proves insufficient.
Nothing was silenced — no PrometheusRule, threshold, or inhibition was touched. The
webhook image tag (v0.4.1) was deliberately left alone; image tags are Flux
image-automation's domain, and the sidecar is not the failing component.
## What this PR does
`external-dns-unifi` in namespace `apps` is in `CrashLoopBackOff` — the `external-dns`
container exits 1 roughly every 70s of uptime while the `webhook` sidecar stays up. The
issue attributed this to the UDM's intermittent `401: User not authenticated` responses
and suggested the `UNIFI_API_KEY` may be stale. **That diagnosis is wrong.** external-dns
v0.21.0 classifies a webhook HTTP 5xx (which is what the sidecar returns when the UDM
401s or 502s) as a *soft* error and retries on the next cycle — verified live this run.
The only failure that is fatal is a **client-side timeout** on `GET /records`, which is a
hard error → `log.Fatalf` → exit 1 → `CrashLoopBackOff`. The UDM's `static-dns` endpoint
has simply gotten slower than the current `--webhook-provider-read-timeout=60s` ceiling
(reads observed completing at ~70s), so this raises that timeout to `180s`.
## Changes
- `kubernetes/infrastructure/external-dns-unifi/helmrelease.yaml` — raise
`--webhook-provider-read-timeout` from `60s` to `180s`, and rewrite the adjacent
comment to record the soft-error-vs-hard-error distinction that makes this the right
knob (the previous comment predated the soft-error behaviour and only described the
slow-read case).
## Verification
Everything below was observed live this run via `./.ci-diag/kubectl`.
1. **The crash loop is ongoing** (the alert had already "resolved" in Alertmanager):
`get pods -n apps -o wide` → `external-dns-unifi-89f464b66-98jjw 1/2 CrashLoopBackOff 18 (2m37s ago) 5h48m`
(19 restarts by the end of this run, up from the 13 recorded in the issue).
2. **Only the `external-dns` container crashes; the sidecar is healthy.**
`get pod ... -o jsonpath={.status.containerStatuses}` →
`external-dns`: `exitCode:1, reason:"Error", startedAt:11:37:51Z, finishedAt:11:39:01Z`;
`webhook`: `restarts=0`, running since `05:52:49Z`. Container lifetime is 70s — ~10s of
startup plus exactly one 60s read timeout.
3. **The fatal error is a client timeout, not an auth error.**
`logs ... -c external-dns --previous` →
`level=fatal msg="Failed to do run once: Get \"http://localhost:8888/records\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"`
The same log's config dump confirms the current setting is live: `WebhookProviderReadTimeout:1m0s`.
4. **UDM errors are survivable; this is the key correction.**
`logs ... -c external-dns --tail=6 --timestamps` on the *current* container →
`11:41:50 level=error ... "soft error\nfailed to get records with code 500 (consecutive soft errors: 1)"`
`11:42:50 level=error ... (consecutive soft errors: 2)`
`11:43:51 level=info msg="All records are already up to date"`
`11:43:51 level=info msg="Reconciliation succeeded after 2 consecutive soft errors"`
Those two soft errors correspond 1:1 (same second) with the sidecar's UDM failures at
`11:41:50` and `11:42:50` (`returned 502`). So a 401/502 costs one skipped cycle, not a
restart.
5. **Slow reads are what kill it, and they land just over 60s.**
`logs ... -c webhook --since=12m --timestamps` shows entries where the sidecar
*successfully* retrieved all 128 records but the client had already gone:
`11:33:00 "provider retrieved records" count:128` immediately followed by
`error encoding records ... write: broken pipe`. Same pattern at `11:30:47`, `11:30:51`,
`11:35:01`. Each abandon happens 60s after the request, so the true read time is ~70s —
over the ceiling, but only just.
**How a reviewer confirms the fix.** After Flux reconciles the HelmRelease and the pod is
recreated:
- `./.ci-diag/kubectl get pods -n apps -l app.kubernetes.io/instance=external-dns-unifi` —
expect `2/2 Running` with the restart count flat over 30+ min.
- `./.ci-diag/kubectl logs deploy/external-dns-unifi -n apps -c external-dns --tail=30` —
expect `WebhookProviderReadTimeout:3m0s` in the config dump, and any remaining UDM
trouble to appear as `soft error ... code 500` lines followed by
`Reconciliation succeeded`, never as `level=fatal`.
- `KubePodCrashLooping` for `apps/external-dns-unifi` should stop firing. DNS
reconciliation is already working between crashes (`All records are already up to date`),
so no record churn is expected.
## Notes
- **What the issue got wrong.** It proposed that the `UNIFI_API_KEY` may be stale/rotated.
It is not: the same key returns `count:128` successfully on most cycles and did so at
`11:43:51`, minutes after a 401. A revoked key would fail every time. I did not read the
Secret (the diag ServiceAccount has no Secrets access) — this is inferred from the
interleaved successes, which is sufficient to rule out a hard revocation. The issue also
framed the 401s as the cause of the crash loop; evidence item 4 shows they are not.
- **This does not make the UDM healthy.** `https://udm.renner.uno` really is returning
intermittent `401: User not authenticated` and `502`, and its `static-dns` read time has
roughly doubled since the 60s value was chosen. That is outside this repo. It is still
worth a human look at the UDM's own health/logs (controller restarts, backups, or load
around those timestamps) — but it is now a degraded-reconcile problem, not a
crash-loop problem, and it will no longer page.
- **This is a ceiling, and the UDM is trending slower.** If reads keep growing, this will
need revisiting. A more durable option is `--provider-cache-time` (currently
`ProviderCacheTime:0s`), which would cut UDM polling from ~60/hour to a handful and
shrink the exposure window proportionally. I did not do it here: it changes reconcile
semantics (out-of-band UDM edits would be picked up with a delay), and I could not verify
the cache-invalidation-on-apply behaviour from read-only tooling. Worth considering
separately if the timeout bump proves insufficient.
- Nothing was silenced — no PrometheusRule, threshold, or inhibition was touched. The
webhook image tag (`v0.4.1`) was deliberately left alone; image tags are Flux
image-automation's domain, and the sidecar is not the failing component.
Closes #424
🤖Automated review — model claude-haiku-4-5-20251001 (complexity: trivial), commit 5f7e3c1268367f7f37776727fb389a90c560c4bb.
Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.
Summary
This PR fixes the external-dns-unifi CrashLoopBackOff alert by increasing the webhook read timeout from 60s to 180s. The diagnosis is evidence-based (detailed live logs provided), the fix is proportionate (2.5× headroom over observed 70s read times), and the comment accurately explains the distinction between survivable soft errors (HTTP 5xx) and fatal hard errors (client-side timeout). No blocking issues.
🟢 Nits
Line 116: The timestamp 2026-07-26 in the comment will become outdated if this is revisited later. Consider phrasing it as a relative date ("recent reads observed at") if you expect this to be read months from now, though the specific date is useful for issue traceability and the PR body provides full context.
Line 129–130: Minor: the phrasing "~2.5x headroom" is informal but clear. If this repo prefers more formal style, consider "approximately 2.5× the observed worst-case latency" — though this is a nit and the current phrasing is fine for internal documentation.
Tests
This is a Kubernetes/Helm configuration change affecting a single deployment. The PR body documents comprehensive post-deployment verification steps:
Expect pod restart count to remain flat after reconciliation
Config dump should show WebhookProviderReadTimeout:3m0s
Remaining UDM errors should appear as soft errors followed by "Reconciliation succeeded", never as fatal log.Fatalf
No new tests are needed for a configuration-only change, and the manual verification steps are adequate.
Recommendation
Looks reasonable — no blocking issues. The fix is well-diagnosed and proportionate to the problem.
🤖 **Automated review** — model `claude-haiku-4-5-20251001` (complexity: trivial), commit `5f7e3c1268367f7f37776727fb389a90c560c4bb`.
> Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review.
---
## Summary
This PR fixes the `external-dns-unifi` CrashLoopBackOff alert by increasing the webhook read timeout from 60s to 180s. The diagnosis is evidence-based (detailed live logs provided), the fix is proportionate (2.5× headroom over observed 70s read times), and the comment accurately explains the distinction between survivable soft errors (HTTP 5xx) and fatal hard errors (client-side timeout). No blocking issues.
## 🟢 Nits
**Line 116:** The timestamp `2026-07-26` in the comment will become outdated if this is revisited later. Consider phrasing it as a relative date ("recent reads observed at") if you expect this to be read months from now, though the specific date is useful for issue traceability and the PR body provides full context.
**Line 129–130:** Minor: the phrasing "~2.5x headroom" is informal but clear. If this repo prefers more formal style, consider "approximately 2.5× the observed worst-case latency" — though this is a nit and the current phrasing is fine for internal documentation.
## Tests
This is a Kubernetes/Helm configuration change affecting a single deployment. The PR body documents comprehensive post-deployment verification steps:
- Expect pod restart count to remain flat after reconciliation
- Config dump should show `WebhookProviderReadTimeout:3m0s`
- Remaining UDM errors should appear as soft errors followed by "Reconciliation succeeded", never as fatal log.Fatalf
No new tests are needed for a configuration-only change, and the manual verification steps are adequate.
## Recommendation
Looks reasonable — no blocking issues. The fix is well-diagnosed and proportionate to the problem.
<!-- pr-review:sha=5f7e3c1268367f7f37776727fb389a90c560c4bb -->
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
review-bot
What this PR does
external-dns-unifiin namespaceappsis inCrashLoopBackOff— theexternal-dnscontainer exits 1 roughly every 70s of uptime while the
webhooksidecar stays up. Theissue attributed this to the UDM's intermittent
401: User not authenticatedresponsesand suggested the
UNIFI_API_KEYmay be stale. That diagnosis is wrong. external-dnsv0.21.0 classifies a webhook HTTP 5xx (which is what the sidecar returns when the UDM
401s or 502s) as a soft error and retries on the next cycle — verified live this run.
The only failure that is fatal is a client-side timeout on
GET /records, which is ahard error →
log.Fatalf→ exit 1 →CrashLoopBackOff. The UDM'sstatic-dnsendpointhas simply gotten slower than the current
--webhook-provider-read-timeout=60sceiling(reads observed completing at ~70s), so this raises that timeout to
180s.Changes
kubernetes/infrastructure/external-dns-unifi/helmrelease.yaml— raise--webhook-provider-read-timeoutfrom60sto180s, and rewrite the adjacentcomment to record the soft-error-vs-hard-error distinction that makes this the right
knob (the previous comment predated the soft-error behaviour and only described the
slow-read case).
Verification
Everything below was observed live this run via
./.ci-diag/kubectl.The crash loop is ongoing (the alert had already "resolved" in Alertmanager):
get pods -n apps -o wide→external-dns-unifi-89f464b66-98jjw 1/2 CrashLoopBackOff 18 (2m37s ago) 5h48m(19 restarts by the end of this run, up from the 13 recorded in the issue).
Only the
external-dnscontainer crashes; the sidecar is healthy.get pod ... -o jsonpath={.status.containerStatuses}→external-dns:exitCode:1, reason:"Error", startedAt:11:37:51Z, finishedAt:11:39:01Z;webhook:restarts=0, running since05:52:49Z. Container lifetime is 70s — ~10s ofstartup plus exactly one 60s read timeout.
The fatal error is a client timeout, not an auth error.
logs ... -c external-dns --previous→level=fatal msg="Failed to do run once: Get \"http://localhost:8888/records\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"The same log's config dump confirms the current setting is live:
WebhookProviderReadTimeout:1m0s.UDM errors are survivable; this is the key correction.
logs ... -c external-dns --tail=6 --timestampson the current container →11:41:50 level=error ... "soft error\nfailed to get records with code 500 (consecutive soft errors: 1)"11:42:50 level=error ... (consecutive soft errors: 2)11:43:51 level=info msg="All records are already up to date"11:43:51 level=info msg="Reconciliation succeeded after 2 consecutive soft errors"Those two soft errors correspond 1:1 (same second) with the sidecar's UDM failures at
11:41:50and11:42:50(returned 502). So a 401/502 costs one skipped cycle, not arestart.
Slow reads are what kill it, and they land just over 60s.
logs ... -c webhook --since=12m --timestampsshows entries where the sidecarsuccessfully retrieved all 128 records but the client had already gone:
11:33:00 "provider retrieved records" count:128immediately followed byerror encoding records ... write: broken pipe. Same pattern at11:30:47,11:30:51,11:35:01. Each abandon happens 60s after the request, so the true read time is ~70s —over the ceiling, but only just.
How a reviewer confirms the fix. After Flux reconciles the HelmRelease and the pod is
recreated:
./.ci-diag/kubectl get pods -n apps -l app.kubernetes.io/instance=external-dns-unifi—expect
2/2 Runningwith the restart count flat over 30+ min../.ci-diag/kubectl logs deploy/external-dns-unifi -n apps -c external-dns --tail=30—expect
WebhookProviderReadTimeout:3m0sin the config dump, and any remaining UDMtrouble to appear as
soft error ... code 500lines followed byReconciliation succeeded, never aslevel=fatal.KubePodCrashLoopingforapps/external-dns-unifishould stop firing. DNSreconciliation is already working between crashes (
All records are already up to date),so no record churn is expected.
Notes
UNIFI_API_KEYmay be stale/rotated.It is not: the same key returns
count:128successfully on most cycles and did so at11:43:51, minutes after a 401. A revoked key would fail every time. I did not read theSecret (the diag ServiceAccount has no Secrets access) — this is inferred from the
interleaved successes, which is sufficient to rule out a hard revocation. The issue also
framed the 401s as the cause of the crash loop; evidence item 4 shows they are not.
https://udm.renner.unoreally is returningintermittent
401: User not authenticatedand502, and itsstatic-dnsread time hasroughly doubled since the 60s value was chosen. That is outside this repo. It is still
worth a human look at the UDM's own health/logs (controller restarts, backups, or load
around those timestamps) — but it is now a degraded-reconcile problem, not a
crash-loop problem, and it will no longer page.
need revisiting. A more durable option is
--provider-cache-time(currentlyProviderCacheTime:0s), which would cut UDM polling from ~60/hour to a handful andshrink the exposure window proportionally. I did not do it here: it changes reconcile
semantics (out-of-band UDM edits would be picked up with a delay), and I could not verify
the cache-invalidation-on-apply behaviour from read-only tooling. Worth considering
separately if the timeout bump proves insufficient.
webhook image tag (
v0.4.1) was deliberately left alone; image tags are Fluximage-automation's domain, and the sidecar is not the failing component.
Closes #424
🤖 Automated review — model
claude-haiku-4-5-20251001(complexity: trivial), commit5f7e3c1268367f7f37776727fb389a90c560c4bb.Summary
This PR fixes the
external-dns-unifiCrashLoopBackOff alert by increasing the webhook read timeout from 60s to 180s. The diagnosis is evidence-based (detailed live logs provided), the fix is proportionate (2.5× headroom over observed 70s read times), and the comment accurately explains the distinction between survivable soft errors (HTTP 5xx) and fatal hard errors (client-side timeout). No blocking issues.🟢 Nits
Line 116: The timestamp
2026-07-26in the comment will become outdated if this is revisited later. Consider phrasing it as a relative date ("recent reads observed at") if you expect this to be read months from now, though the specific date is useful for issue traceability and the PR body provides full context.Line 129–130: Minor: the phrasing "~2.5x headroom" is informal but clear. If this repo prefers more formal style, consider "approximately 2.5× the observed worst-case latency" — though this is a nit and the current phrasing is fine for internal documentation.
Tests
This is a Kubernetes/Helm configuration change affecting a single deployment. The PR body documents comprehensive post-deployment verification steps:
WebhookProviderReadTimeout:3m0sNo new tests are needed for a configuration-only change, and the manual verification steps are adequate.
Recommendation
Looks reasonable — no blocking issues. The fix is well-diagnosed and proportionate to the problem.