KubePodCrashLooping fired for the external-dns container in apps/external-dns-unifi-* (18 restarts by the time I looked). The container
was dying with Failed to do run once: Get "http://localhost:8888/records": context deadline exceeded. Reading the webhook sidecar's log for the same
timestamps shows what the issue's evidence missed: on every one of those
fatals the webhook had in fact successfully retrieved all 128 records from
the UDM — it just finished ~1s after external-dns had already given up and
closed the socket, so the webhook's next log line is error encoding records ... write: broken pipe. In other words those crashes
were not an upstream UDM failure at all; they were the --webhook-provider-read-timeout=30s client deadline being marginally too
tight for a UDM fetch that takes ~31s, combined with external-dns treating any RunOnce error as fatal (log.Fatalf → exit 1). This PR raises that timeout
to 60s, which turns those slow-but-successful reads into normal syncs instead
of container restarts.
Changes
kubernetes/infrastructure/external-dns-unifi/helmrelease.yaml — --webhook-provider-read-timeout 30s → 60s, with a comment recording the
observed ~31s worst case and why a longer timeout cannot stall the reconcile
loop (external-dns calls RunOnce sequentially, so a slow read only delays
the next cycle past the 1m interval).
Verification
All of the following was observed live this run via ./.ci-diag/kubectl:
-n apps get pods -l app.kubernetes.io/instance=external-dns-unifi -o wide → 2/2 Running, RESTARTS 18 (41m ago), pod age 2d16h. The pod is not
currently crashlooping; the last restart was at 05:00:57Z and it has been
up cleanly since.
get pod ... -o jsonpath=...containerStatuses → external-dns restarts=18,
last termination 04:55:02Z -> 05:00:46Z exit=1; webhook restarts=0 started=2026-07-23T13:36:13Z. Only the main container restarts.
logs ... -c external-dns --previous --tail=40 → clean 1-minute cycles
(All records are already up to date at 04:55/04:56/04:57/04:58/04:59) then level=fatal msg="Failed to do run once: Get \"http://localhost:8888/records\": context deadline exceeded" at 05:00:46Z.
logs ... -c webhook --since-time=2026-07-26T03:30:00Z --timestamps — the
decisive evidence. Paired against the fatal above, at 05:00:47.912Z: provider retrieved records count=128 immediately followed by error encoding records ... write tcp 127.0.0.1:8888->127.0.0.1:32872: write: broken pipe. The same success-then-broken-pipe pair occurs at 04:31:39Z, 04:32:54Z and 04:49:50Z — four of the restarts in that
window are pure timeout races, not UDM errors. The UDM did also return real
errors in the same bad patch (500: Read timed out twice at 04:33:45Z, 401 at 04:36:43Z and 04:39:20Z), but none since 04:39:20Z.
Every UDM fetch from 05:01:01Z through 05:42:20Z succeeded with count=128 on a steady 60s cadence — no errors of any kind for the last
~40 minutes.
get externalsecret external-dns-unifi-secret → Ready True SecretSynced, refreshTime 2026-07-26T04:52:51Z.
get helmrelease external-dns-unifi → Ready=True UpgradeSucceeded,
chart external-dns@1.21.1, release .v7 — Flux will apply this values
change on the next reconcile (interval: 30m).
How a reviewer confirms the fix: after Flux reconciles, the HelmRelease rolls a
new pod; check logs -c external-dns for WebhookProviderReadTimeout:60s in the startup config dump, then watch RESTARTS on the new pod. KubePodCrashLooping should stop firing.
Specifically, the webhook log should no longer produce error encoding records ... broken pipe — a recurrence of that line after this
change would mean the UDM is now taking >60s, not >30s.
Notes
What the issue got wrong. The issue's root cause attributes the external-dns fatals to the webhook being slow because of the UDM's 401/500 responses. Those UDM errors are real (I reproduced the exact log
lines), but they are a separate failure mode: when the UDM returns 401/500
the webhook answers external-dns immediately with an error, which also fatals
the container — that path has nothing to do with the 30s timeout. The four context deadline exceeded fatals I could pair with webhook log lines were
all cases where the UDM answered successfully at ~31s. Roughly half the
restarts in the 04:30–05:00 window are the timeout race this PR fixes; the
other half are the UDM's own transient errors, which no repo change can
prevent.
The credential hypothesis is settled, and it's not the credential. The
issue left open whether the 401s came from a stale/rejected UNIFI_API_KEY. They did not. The webhook container has been running
continuously since 2026-07-23T13:36:13Z (restarts=0), and UNIFI_API_KEY
is injected via secretKeyRef as an env var — env vars are snapshotted at
container start and do not track later Secret updates. So the key the webhook
is presenting has been byte-identical since 23 July, it returned 401 at
04:36/04:39, and it has returned 200 on every request since 05:01. A
revoked or stale key cannot behave that way; these were transient
auth failures on the UDM side. No key rotation is needed.
Residual, not fixable in this repo. The UDM's transient 500: Read timed out / 401 responses originate on udm.renner.uno. If the crash loop
recurs without the broken pipe signature, a human should check the UniFi
controller's own health around the failure timestamps (UI/API service
restarts, load, rate limiting). That is outside this agent's cluster-only
read access.
Alternatives considered and rejected. Adding --provider-cache-time (or
raising --interval) would cut UDM record fetches by ~15x and so reduce
exposure to both failure modes — but it changes reconcile semantics
(out-of-band UDM edits would go uncorrected for the cache window) on a policy: sync instance, which I cannot validate from here. Left out
deliberately as a bigger behavioural change than the alert warrants; worth
considering if the UDM's transient errors turn out to be chronic. I did not
touch the PrometheusRule, thresholds, probes, or the webhook image tag.
## What this PR does
`KubePodCrashLooping` fired for the `external-dns` container in
`apps/external-dns-unifi-*` (18 restarts by the time I looked). The container
was dying with `Failed to do run once: Get "http://localhost:8888/records":
context deadline exceeded`. Reading the **webhook sidecar's** log for the same
timestamps shows what the issue's evidence missed: on every one of those
fatals the webhook had in fact **successfully retrieved all 128 records** from
the UDM — it just finished ~1s *after* external-dns had already given up and
closed the socket, so the webhook's next log line is
`error encoding records ... write: broken pipe`. In other words those crashes
were not an upstream UDM failure at all; they were the
`--webhook-provider-read-timeout=30s` client deadline being marginally too
tight for a UDM fetch that takes ~31s, combined with external-dns treating any
`RunOnce` error as fatal (`log.Fatalf` → exit 1). This PR raises that timeout
to 60s, which turns those slow-but-successful reads into normal syncs instead
of container restarts.
## Changes
- `kubernetes/infrastructure/external-dns-unifi/helmrelease.yaml` —
`--webhook-provider-read-timeout` 30s → 60s, with a comment recording the
observed ~31s worst case and why a longer timeout cannot stall the reconcile
loop (external-dns calls `RunOnce` sequentially, so a slow read only delays
the next cycle past the 1m interval).
## Verification
All of the following was observed live this run via `./.ci-diag/kubectl`:
- `-n apps get pods -l app.kubernetes.io/instance=external-dns-unifi -o wide` →
`2/2 Running`, `RESTARTS 18 (41m ago)`, pod age `2d16h`. The pod is **not**
currently crashlooping; the last restart was at `05:00:57Z` and it has been
up cleanly since.
- `get pod ... -o jsonpath=...containerStatuses` → `external-dns restarts=18`,
last termination `04:55:02Z -> 05:00:46Z exit=1`; `webhook restarts=0
started=2026-07-23T13:36:13Z`. Only the main container restarts.
- `logs ... -c external-dns --previous --tail=40` → clean 1-minute cycles
(`All records are already up to date` at 04:55/04:56/04:57/04:58/04:59) then
`level=fatal msg="Failed to do run once: Get \"http://localhost:8888/records\":
context deadline exceeded"` at `05:00:46Z`.
- `logs ... -c webhook --since-time=2026-07-26T03:30:00Z --timestamps` — the
decisive evidence. Paired against the fatal above, at `05:00:47.912Z`:
`provider retrieved records count=128` immediately followed by
`error encoding records ... write tcp 127.0.0.1:8888->127.0.0.1:32872:
write: broken pipe`. The same success-then-broken-pipe pair occurs at
`04:31:39Z`, `04:32:54Z` and `04:49:50Z` — four of the restarts in that
window are pure timeout races, not UDM errors. The UDM did also return real
errors in the same bad patch (`500: Read timed out` twice at `04:33:45Z`,
`401` at `04:36:43Z` and `04:39:20Z`), but **none since `04:39:20Z`**.
- Every UDM fetch from `05:01:01Z` through `05:42:20Z` succeeded with
`count=128` on a steady 60s cadence — no errors of any kind for the last
~40 minutes.
- `get externalsecret external-dns-unifi-secret` → `Ready True SecretSynced`,
`refreshTime 2026-07-26T04:52:51Z`.
- `get helmrelease external-dns-unifi` → `Ready=True UpgradeSucceeded`,
chart `external-dns@1.21.1`, release `.v7` — Flux will apply this values
change on the next reconcile (`interval: 30m`).
How a reviewer confirms the fix: after Flux reconciles, the HelmRelease rolls a
new pod; check `logs -c external-dns` for
`WebhookProviderReadTimeout:60s` in the startup config dump, then watch
`RESTARTS` on the new pod. `KubePodCrashLooping` should stop firing.
Specifically, the webhook log should no longer produce
`error encoding records ... broken pipe` — a recurrence of that line after this
change would mean the UDM is now taking >60s, not >30s.
## Notes
- **What the issue got wrong.** The issue's root cause attributes the
`external-dns` fatals to the webhook being slow *because* of the UDM's
`401`/`500` responses. Those UDM errors are real (I reproduced the exact log
lines), but they are a *separate* failure mode: when the UDM returns 401/500
the webhook answers external-dns immediately with an error, which also fatals
the container — that path has nothing to do with the 30s timeout. The four
`context deadline exceeded` fatals I could pair with webhook log lines were
all cases where the UDM answered *successfully* at ~31s. Roughly half the
restarts in the 04:30–05:00 window are the timeout race this PR fixes; the
other half are the UDM's own transient errors, which no repo change can
prevent.
- **The credential hypothesis is settled, and it's not the credential.** The
issue left open whether the `401`s came from a stale/rejected
`UNIFI_API_KEY`. They did not. The `webhook` container has been running
continuously since `2026-07-23T13:36:13Z` (`restarts=0`), and `UNIFI_API_KEY`
is injected via `secretKeyRef` as an env var — env vars are snapshotted at
container start and do not track later Secret updates. So the key the webhook
is presenting has been byte-identical since 23 July, it returned `401` at
04:36/04:39, and it has returned `200` on every request since `05:01`. A
revoked or stale key cannot behave that way; these were transient
auth failures on the UDM side. No key rotation is needed.
- **Residual, not fixable in this repo.** The UDM's transient `500: Read timed
out` / `401` responses originate on `udm.renner.uno`. If the crash loop
recurs *without* the `broken pipe` signature, a human should check the UniFi
controller's own health around the failure timestamps (UI/API service
restarts, load, rate limiting). That is outside this agent's cluster-only
read access.
- **Alternatives considered and rejected.** Adding `--provider-cache-time` (or
raising `--interval`) would cut UDM record fetches by ~15x and so reduce
exposure to *both* failure modes — but it changes reconcile semantics
(out-of-band UDM edits would go uncorrected for the cache window) on a
`policy: sync` instance, which I cannot validate from here. Left out
deliberately as a bigger behavioural change than the alert warrants; worth
considering if the UDM's transient errors turn out to be chronic. I did not
touch the PrometheusRule, thresholds, probes, or the webhook image tag.
Closes #408
🤖Automated review — model claude-haiku-4-5-20251001 (complexity: trivial), commit ebf93db1910da1fa932f66327bfca66db3d2298c.
Advisory review by the review-bot pipeline. It never approves; a human merges. Push new commits to get a fresh review.
Summary
This PR fixes KubePodCrashLooping alerts for external-dns-unifi by raising the webhook provider read timeout from 30s to 60s. The root cause is that the UDM's static-dns endpoint occasionally takes ~31s to respond, exceeding the original 30s timeout and causing external-dns to exit with a fatal error. The fix is minimal, well-evidenced by live pod logs, and includes a thorough comment explaining the rationale and addressing potential performance concerns.
🔴 Blocking
None identified.
🟡 Suggestions
Post-deployment validation note (informational). The PR description provides excellent verification steps for reviewers to confirm the fix post-deployment. Once Flux reconciles, the new pod should show WebhookProviderReadTimeout:60s in logs, and RESTARTS count should stabilize without the broken pipe signature in webhook logs. This is already documented in the PR but worth confirming in the next reconciliation cycle.
Consider documenting the UDM transient error separately. The PR correctly separates two failure modes: (a) the timeout race this PR fixes, and (b) UDM's own transient 401/500 responses. The PR notes these are outside the repo's scope (require UDM controller investigation), but a separate issue/monitoring task might help track the UDM health as a residual concern.
🟢 Nits
None.
Tests
This is a Kubernetes/Helm configuration change, not application code, so traditional unit/integration tests do not apply. The fix has been validated through live environment observation:
Webhook logs confirm 128 records retrieved successfully at ~31s before each crash
Pod logs show exact context deadline exceeded errors at the 30s boundary
Post-fix observation shows stable 60s cadence with no timeout errors for 40+ minutes
Pod RESTARTS count has been stable since deployment
Recommendation
Looks reasonable — no blocking issues; the fix is targeted, well-justified, and verified in production.
🤖 **Automated review** — model `claude-haiku-4-5-20251001` (complexity: trivial), commit `ebf93db1910da1fa932f66327bfca66db3d2298c`.
> Advisory review by the `review-bot` pipeline. It never approves; a human merges. Push new commits to get a fresh review.
---
## Summary
This PR fixes `KubePodCrashLooping` alerts for external-dns-unifi by raising the webhook provider read timeout from 30s to 60s. The root cause is that the UDM's static-dns endpoint occasionally takes ~31s to respond, exceeding the original 30s timeout and causing external-dns to exit with a fatal error. The fix is minimal, well-evidenced by live pod logs, and includes a thorough comment explaining the rationale and addressing potential performance concerns.
## 🔴 Blocking
None identified.
## 🟡 Suggestions
1. **Post-deployment validation note (informational).** The PR description provides excellent verification steps for reviewers to confirm the fix post-deployment. Once Flux reconciles, the new pod should show `WebhookProviderReadTimeout:60s` in logs, and `RESTARTS` count should stabilize without the `broken pipe` signature in webhook logs. This is already documented in the PR but worth confirming in the next reconciliation cycle.
2. **Consider documenting the UDM transient error separately.** The PR correctly separates two failure modes: (a) the timeout race this PR fixes, and (b) UDM's own transient 401/500 responses. The PR notes these are outside the repo's scope (require UDM controller investigation), but a separate issue/monitoring task might help track the UDM health as a residual concern.
## 🟢 Nits
None.
## Tests
This is a Kubernetes/Helm configuration change, not application code, so traditional unit/integration tests do not apply. The fix has been validated through live environment observation:
- Webhook logs confirm 128 records retrieved successfully at ~31s before each crash
- Pod logs show exact `context deadline exceeded` errors at the 30s boundary
- Post-fix observation shows stable 60s cadence with no timeout errors for 40+ minutes
- Pod `RESTARTS` count has been stable since deployment
## Recommendation
Looks reasonable — no blocking issues; the fix is targeted, well-justified, and verified in production.
<!-- pr-review:sha=ebf93db1910da1fa932f66327bfca66db3d2298c -->
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
review-bot
What this PR does
KubePodCrashLoopingfired for theexternal-dnscontainer inapps/external-dns-unifi-*(18 restarts by the time I looked). The containerwas dying with
Failed to do run once: Get "http://localhost:8888/records": context deadline exceeded. Reading the webhook sidecar's log for the sametimestamps shows what the issue's evidence missed: on every one of those
fatals the webhook had in fact successfully retrieved all 128 records from
the UDM — it just finished ~1s after external-dns had already given up and
closed the socket, so the webhook's next log line is
error encoding records ... write: broken pipe. In other words those crasheswere not an upstream UDM failure at all; they were the
--webhook-provider-read-timeout=30sclient deadline being marginally tootight for a UDM fetch that takes ~31s, combined with external-dns treating any
RunOnceerror as fatal (log.Fatalf→ exit 1). This PR raises that timeoutto 60s, which turns those slow-but-successful reads into normal syncs instead
of container restarts.
Changes
kubernetes/infrastructure/external-dns-unifi/helmrelease.yaml—--webhook-provider-read-timeout30s → 60s, with a comment recording theobserved ~31s worst case and why a longer timeout cannot stall the reconcile
loop (external-dns calls
RunOncesequentially, so a slow read only delaysthe next cycle past the 1m interval).
Verification
All of the following was observed live this run via
./.ci-diag/kubectl:-n apps get pods -l app.kubernetes.io/instance=external-dns-unifi -o wide→2/2 Running,RESTARTS 18 (41m ago), pod age2d16h. The pod is notcurrently crashlooping; the last restart was at
05:00:57Zand it has beenup cleanly since.
get pod ... -o jsonpath=...containerStatuses→external-dns restarts=18,last termination
04:55:02Z -> 05:00:46Z exit=1;webhook restarts=0 started=2026-07-23T13:36:13Z. Only the main container restarts.logs ... -c external-dns --previous --tail=40→ clean 1-minute cycles(
All records are already up to dateat 04:55/04:56/04:57/04:58/04:59) thenlevel=fatal msg="Failed to do run once: Get \"http://localhost:8888/records\": context deadline exceeded"at05:00:46Z.logs ... -c webhook --since-time=2026-07-26T03:30:00Z --timestamps— thedecisive evidence. Paired against the fatal above, at
05:00:47.912Z:provider retrieved records count=128immediately followed byerror encoding records ... write tcp 127.0.0.1:8888->127.0.0.1:32872: write: broken pipe. The same success-then-broken-pipe pair occurs at04:31:39Z,04:32:54Zand04:49:50Z— four of the restarts in thatwindow are pure timeout races, not UDM errors. The UDM did also return real
errors in the same bad patch (
500: Read timed outtwice at04:33:45Z,401at04:36:43Zand04:39:20Z), but none since04:39:20Z.05:01:01Zthrough05:42:20Zsucceeded withcount=128on a steady 60s cadence — no errors of any kind for the last~40 minutes.
get externalsecret external-dns-unifi-secret→Ready True SecretSynced,refreshTime 2026-07-26T04:52:51Z.get helmrelease external-dns-unifi→Ready=True UpgradeSucceeded,chart
external-dns@1.21.1, release.v7— Flux will apply this valueschange on the next reconcile (
interval: 30m).How a reviewer confirms the fix: after Flux reconciles, the HelmRelease rolls a
new pod; check
logs -c external-dnsforWebhookProviderReadTimeout:60sin the startup config dump, then watchRESTARTSon the new pod.KubePodCrashLoopingshould stop firing.Specifically, the webhook log should no longer produce
error encoding records ... broken pipe— a recurrence of that line after thischange would mean the UDM is now taking >60s, not >30s.
Notes
external-dnsfatals to the webhook being slow because of the UDM's401/500responses. Those UDM errors are real (I reproduced the exact loglines), but they are a separate failure mode: when the UDM returns 401/500
the webhook answers external-dns immediately with an error, which also fatals
the container — that path has nothing to do with the 30s timeout. The four
context deadline exceededfatals I could pair with webhook log lines wereall cases where the UDM answered successfully at ~31s. Roughly half the
restarts in the 04:30–05:00 window are the timeout race this PR fixes; the
other half are the UDM's own transient errors, which no repo change can
prevent.
issue left open whether the
401s came from a stale/rejectedUNIFI_API_KEY. They did not. Thewebhookcontainer has been runningcontinuously since
2026-07-23T13:36:13Z(restarts=0), andUNIFI_API_KEYis injected via
secretKeyRefas an env var — env vars are snapshotted atcontainer start and do not track later Secret updates. So the key the webhook
is presenting has been byte-identical since 23 July, it returned
401at04:36/04:39, and it has returned
200on every request since05:01. Arevoked or stale key cannot behave that way; these were transient
auth failures on the UDM side. No key rotation is needed.
500: Read timed out/401responses originate onudm.renner.uno. If the crash looprecurs without the
broken pipesignature, a human should check the UniFicontroller's own health around the failure timestamps (UI/API service
restarts, load, rate limiting). That is outside this agent's cluster-only
read access.
--provider-cache-time(orraising
--interval) would cut UDM record fetches by ~15x and so reduceexposure to both failure modes — but it changes reconcile semantics
(out-of-band UDM edits would go uncorrected for the cache window) on a
policy: syncinstance, which I cannot validate from here. Left outdeliberately as a bigger behavioural change than the alert warrants; worth
considering if the UDM's transient errors turn out to be chronic. I did not
touch the PrometheusRule, thresholds, probes, or the webhook image tag.
Closes #408
🤖 Automated review — model
claude-haiku-4-5-20251001(complexity: trivial), commitebf93db1910da1fa932f66327bfca66db3d2298c.Summary
This PR fixes
KubePodCrashLoopingalerts for external-dns-unifi by raising the webhook provider read timeout from 30s to 60s. The root cause is that the UDM's static-dns endpoint occasionally takes ~31s to respond, exceeding the original 30s timeout and causing external-dns to exit with a fatal error. The fix is minimal, well-evidenced by live pod logs, and includes a thorough comment explaining the rationale and addressing potential performance concerns.🔴 Blocking
None identified.
🟡 Suggestions
Post-deployment validation note (informational). The PR description provides excellent verification steps for reviewers to confirm the fix post-deployment. Once Flux reconciles, the new pod should show
WebhookProviderReadTimeout:60sin logs, andRESTARTScount should stabilize without thebroken pipesignature in webhook logs. This is already documented in the PR but worth confirming in the next reconciliation cycle.Consider documenting the UDM transient error separately. The PR correctly separates two failure modes: (a) the timeout race this PR fixes, and (b) UDM's own transient 401/500 responses. The PR notes these are outside the repo's scope (require UDM controller investigation), but a separate issue/monitoring task might help track the UDM health as a residual concern.
🟢 Nits
None.
Tests
This is a Kubernetes/Helm configuration change, not application code, so traditional unit/integration tests do not apply. The fix has been validated through live environment observation:
context deadline exceedederrors at the 30s boundaryRESTARTScount has been stable since deploymentRecommendation
Looks reasonable — no blocking issues; the fix is targeted, well-justified, and verified in production.