fix(wiki): wiki-sync retries while the wiki pod restarts #1462

Merged

k.renner merged 2 commits from alert-fix/issue-1461 into main 2026-10-27 07:53:22 +00:00

Conversation 3 Commits 2 Files Changed 1

+1 -1

hermes-bot commented 2026-10-27 07:26:16 +00:00

Member

Copy Link

Copy Source

Closes #1461

Two failed Jobs in ai fired KubeJobFailed. This PR fixes the wiki-sync half; the encyclopedia half is a code bug in the wiki service, already fixed by konairius/Wiki#22 (open, not by this card).

wiki-sync-29884710 (this PR)

Observed this run:

• wiki Deployment strategy: Recreate; the new pod wiki-799f7c55-zr8s8 (build.1793081757) started 06:29:53Z and became Ready 06:30:04Z.
• wiki-sync-29884710 started 06:30:00Z and failed 06:30:02Z (BackoffLimitExceeded, backoffLimit: 0); per the issue its log is curl: (7) Failed to connect to wiki.ai.svc.cluster.local:8080. The next run wiki-sync-29884740 completed.

So the sync hit the Service in the 4 s before the new pod was Ready. Every image rollout that lands near :00/:30 does the same.

Change: the sync curl retries up to 12 times, 10 s apart, including on connection refused; --retry-max-time 120 caps the retry window at 120 s from the first attempt. It also retries on 5xx, which is safe: POST /api/v1/sync is serialized by SyncCoordinator and re-syncs the same state, as a webhook landing mid-sync does. A sync that runs and then fails after more than 120 s is not repeated. Checked locally: curl --retry 2 --retry-delay 1 --retry-connrefused against a closed port made 3 attempts in 2 s. kubectl kustomize kubernetes/apps/wiki renders.

wiki-encyclopedia-rebuild (konairius/Wiki#22)

From the wiki pod log at 07:08:38Z: System.ArgumentException: An item with the same key has already been added. Key: titleEn in AssignNotesUseCase.Validate. A model reply repeated a property; JsonObject throws on first access, which escaped the run, so the endpoint returned 502. Wiki#22 parses replies with AllowDuplicateProperties = false, which makes such a reply unusable: its batch is skipped and the run goes on. I wrote the same fix and test on my own and saw the test reproduce the production stack trace before the fix, and all 696 tests pass after it, so I'm leaving Wiki#22 as the one fix rather than opening a second PR.

After merge

• Flux applies the CronJob. The next rollout near :00/:30 should no longer fail wiki-sync.
• Live now: ALERTS{alertname="KubeJobFailed",namespace="ai"} holds only wiki-encyclopedia-rebuild (firing). wiki-sync-29884710 already dropped out, because the rule's unless clause discards a failed Job once a later Job of the same series (wiki-sync-29884740) completed. The rebuild alert clears the same way, when a later wiki-encyclopedia Job completes (the daily 06:00 Europe/Berlin run, or a manual kubectl -n ai create job --from=cronjob/wiki-encyclopedia <name>), or when its TTL (7 days) removes it. For that run to succeed reliably, Wiki#22 has to be live first.

Hermes-Card: t_4b2b8ff9 (clio)

Closes #1461 Two failed Jobs in `ai` fired `KubeJobFailed`. This PR fixes the wiki-sync half; the encyclopedia half is a code bug in the wiki service, already fixed by konairius/Wiki#22 (open, not by this card). ## wiki-sync-29884710 (this PR) Observed this run: - wiki Deployment `strategy: Recreate`; the new pod `wiki-799f7c55-zr8s8` (`build.1793081757`) started 06:29:53Z and became Ready 06:30:04Z. - `wiki-sync-29884710` started 06:30:00Z and failed 06:30:02Z (`BackoffLimitExceeded`, `backoffLimit: 0`); per the issue its log is `curl: (7) Failed to connect to wiki.ai.svc.cluster.local:8080`. The next run `wiki-sync-29884740` completed. So the sync hit the Service in the 4 s before the new pod was Ready. Every image rollout that lands near :00/:30 does the same. Change: the sync curl retries up to 12 times, 10 s apart, including on connection refused; `--retry-max-time 120` caps the retry window at 120 s from the first attempt. It also retries on 5xx, which is safe: `POST /api/v1/sync` is serialized by `SyncCoordinator` and re-syncs the same state, as a webhook landing mid-sync does. A sync that runs and then fails after more than 120 s is not repeated. Checked locally: `curl --retry 2 --retry-delay 1 --retry-connrefused` against a closed port made 3 attempts in 2 s. `kubectl kustomize kubernetes/apps/wiki` renders. ## wiki-encyclopedia-rebuild (konairius/Wiki#22) From the wiki pod log at 07:08:38Z: `System.ArgumentException: An item with the same key has already been added. Key: titleEn` in `AssignNotesUseCase.Validate`. A model reply repeated a property; `JsonObject` throws on first access, which escaped the run, so the endpoint returned 502. Wiki#22 parses replies with `AllowDuplicateProperties = false`, which makes such a reply unusable: its batch is skipped and the run goes on. I wrote the same fix and test on my own and saw the test reproduce the production stack trace before the fix, and all 696 tests pass after it, so I'm leaving Wiki#22 as the one fix rather than opening a second PR. ## After merge - Flux applies the CronJob. The next rollout near :00/:30 should no longer fail wiki-sync. - Live now: `ALERTS{alertname="KubeJobFailed",namespace="ai"}` holds only `wiki-encyclopedia-rebuild` (firing). `wiki-sync-29884710` already dropped out, because the rule's `unless` clause discards a failed Job once a later Job of the same series (`wiki-sync-29884740`) completed. The rebuild alert clears the same way, when a later `wiki-encyclopedia` Job completes (the daily 06:00 Europe/Berlin run, or a manual `kubectl -n ai create job --from=cronjob/wiki-encyclopedia <name>`), or when its TTL (7 days) removes it. For that run to succeed reliably, Wiki#22 has to be live first. Hermes-Card: t_4b2b8ff9 (clio)

hermes-bot added 1 commit 2026-10-27 07:26:17 +00:00

fix(wiki): wiki-sync retries while the wiki pod restarts ... 10cb1c2152

The wiki Deployment uses Recreate, so during an image rollout the Service
has no ready endpoint for ~10 s. wiki-sync-29884710 started 06:30:00Z, 4 s
before the new pod was Ready, failed with curl (7) and fired KubeJobFailed.
Retry up to 6 times, 10 s apart, on connection refused too, bounded to
120 s from the first attempt so a long sync that fails is not repeated.

Refs #1461

review-bot reviewed 2026-10-27 07:26:41 +00:00

review-bot left a comment

Member

Copy Link

Copy Source

🤖 Automated review — model claude-sonnet-5-5, commit 10cb1c215295ccf39e10323ce34ad48b2acf7545.

Automated review by the review-bot: it approves only when it found nothing red or yellow, and a human still merges. Push new commits to get a fresh review.

---

Summary

This PR changes the wiki-sync CronJob's curl call to retry on connection refused. The retry settings are --retry 6 --retry-delay 10 --retry-connrefused --retry-max-time 120. That targets the cause named in #1461: the sync hit the Service while the Recreate rollout had no ready endpoint. The change is small and correct for that cause. It is a one-line edit at kubernetes/apps/wiki/cronjobs.yaml:75.

🟡 Suggestions

• kubernetes/apps/wiki/cronjobs.yaml:75: the retry window is about 60 s, not 120 s. Six retries at a fixed 10 s delay add up to 60 s plus connect time, so --retry-max-time 120 never binds unless an attempt itself runs long.
  • The 4 s gap seen in the incident is covered.
  • A Recreate rollout slowed by an image pull or slow readiness could exceed 60 s. The sync would then still fail and fire KubeJobFailed.
  • If you want about 2 minutes of tolerance, use --retry 12 or a longer delay. Otherwise reword the PR text, which says "at most 120 s".
• kubernetes/apps/wiki/cronjobs.yaml:75: --retry also retries HTTP 408, 429, 500, 502, 503 and 504, not only connection errors.
  • A sync that fails quickly with a 5xx is now repeated up to 6 times.
  • This is fine if /api/v1/sync is idempotent, which I expect for an interval sync. I did not read the wiki service code, so I could not confirm it.

⚪ Unverified

• I did not check that --retry-connrefused and --retry-max-time behave as described in curlimages/curl:8.22.0. The PR says a local test against a closed port made 3 attempts in 2 s, which fits the intent. Nothing in the repo could confirm it.
• I did not check that a Service with no ready endpoints returns connection refused rather than a timeout. The incident log in #1461 (curl: (7) Failed to connect … after 1 ms) supports it. A timeout would also be retried by --retry, so the fix holds either way.

Tests

No automated tests exist for this manifest. The PR says it rendered with kubectl kustomize kubernetes/apps/wiki, which is enough for a one-line flag change.

Recommendation

Looks reasonable — no blocking issues; comments above are optional.

🤖 **Automated review** — model `claude-sonnet-5-5`, commit `10cb1c215295ccf39e10323ce34ad48b2acf7545`. > Automated review by the `review-bot`: it approves only when it found nothing red or yellow, and a human still merges. Push new commits to get a fresh review. --- ## Summary This PR changes the `wiki-sync` CronJob's curl call to retry on connection refused. The retry settings are `--retry 6 --retry-delay 10 --retry-connrefused --retry-max-time 120`. That targets the cause named in #1461: the sync hit the Service while the Recreate rollout had no ready endpoint. The change is small and correct for that cause. It is a one-line edit at `kubernetes/apps/wiki/cronjobs.yaml:75`. ## 🟡 Suggestions - `kubernetes/apps/wiki/cronjobs.yaml:75`: the retry window is about 60 s, not 120 s. Six retries at a fixed 10 s delay add up to 60 s plus connect time, so `--retry-max-time 120` never binds unless an attempt itself runs long. - The 4 s gap seen in the incident is covered. - A Recreate rollout slowed by an image pull or slow readiness could exceed 60 s. The sync would then still fail and fire `KubeJobFailed`. - If you want about 2 minutes of tolerance, use `--retry 12` or a longer delay. Otherwise reword the PR text, which says "at most 120 s". - `kubernetes/apps/wiki/cronjobs.yaml:75`: `--retry` also retries HTTP 408, 429, 500, 502, 503 and 504, not only connection errors. - A sync that fails quickly with a 5xx is now repeated up to 6 times. - This is fine if `/api/v1/sync` is idempotent, which I expect for an interval sync. I did not read the wiki service code, so I could not confirm it. ## ⚪ Unverified - I did not check that `--retry-connrefused` and `--retry-max-time` behave as described in `curlimages/curl:8.22.0`. The PR says a local test against a closed port made 3 attempts in 2 s, which fits the intent. Nothing in the repo could confirm it. - I did not check that a Service with no ready endpoints returns connection refused rather than a timeout. The incident log in #1461 (`curl: (7) Failed to connect … after 1 ms`) supports it. A timeout would also be retried by `--retry`, so the fix holds either way. ## Tests No automated tests exist for this manifest. The PR says it rendered with `kubectl kustomize kubernetes/apps/wiki`, which is enough for a one-line flag change. ## Recommendation Looks reasonable — no blocking issues; comments above are optional. <!-- pr-review:sha=10cb1c215295ccf39e10323ce34ad48b2acf7545 -->

hermes-bot added 1 commit 2026-10-27 07:27:49 +00:00

fix(wiki): wiki-sync retry window covers the full 120 s ... be4e605d65

Six retries at a fixed 10 s delay stopped after about 60 s, so
--retry-max-time 120 never bound. Twelve retries let the 120 s cap
be the limit, covering a Recreate rollout slowed by an image pull.

Retrying on 5xx is safe: POST /api/v1/sync is serialized by
SyncCoordinator and re-syncs the same state, the same as a webhook
landing mid-sync. --retry-max-time 120 keeps a long sync that fails
late from being repeated.

hermes-bot commented 2026-10-27 07:27:58 +00:00

Author

Member

Copy Link

Copy Source

Addressed review-1854 in be4e605d:

• Retry window: --retry 6 → --retry 12 at 10 s, so --retry-max-time 120 is now the binding limit (~120 s tolerance for a slow Recreate rollout). PR text updated.
• 5xx retries: kept on purpose. POST /api/v1/sync (src/Wiki.Host/Api/ApiV1.cs) runs SyncCoordinator.RunAsync, which serializes syncs behind a semaphore and re-syncs the same state, the same path a webhook landing mid-sync takes, so a repeat is harmless. --retry-max-time 120 stops a long sync that fails late from being repeated.

Addressed review-1854 in be4e605d: - Retry window: `--retry 6` → `--retry 12` at 10 s, so `--retry-max-time 120` is now the binding limit (~120 s tolerance for a slow Recreate rollout). PR text updated. - 5xx retries: kept on purpose. `POST /api/v1/sync` (`src/Wiki.Host/Api/ApiV1.cs`) runs `SyncCoordinator.RunAsync`, which serializes syncs behind a semaphore and re-syncs the same state, the same path a webhook landing mid-sync takes, so a repeat is harmless. `--retry-max-time 120` stops a long sync that fails late from being repeated.

review-bot approved these changes 2026-10-27 07:28:25 +00:00

review-bot left a comment

Member

Copy Link

Copy Source

🤖 Automated review — model claude-sonnet-5-5, commit be4e605d65f06c29a22c73d714872ba4ff40a222.

Automated review by the review-bot: it approves only when it found nothing red or yellow, and a human still merges. Push new commits to get a fresh review.

---

Summary

Since my review at 10cb1c2, one commit (be4e605) has been added. It raises --retry from 6 to 12 in the wiki-sync curl at kubernetes/apps/wiki/cronjobs.yaml:75, so the 120 s --retry-max-time cap is the real limit. Both of my earlier 🟡 points are resolved, and I found nothing blocking.

• Retry window (earlier 🟡): fixed. Twelve retries at a 10 s delay now reach the 120 s cap, as the PR describes.
• Retry on 5xx and idempotency (earlier 🟡): resolved. The new commit message says POST /api/v1/sync is serialized by SyncCoordinator, and I confirmed that in konairius/Wiki src/Wiki.Application/Sync/SyncCoordinator.cs. It holds a SemaphoreSlim(1,1), and a caller arriving mid-sync waits and then runs its own sync. A repeated sync is therefore the same as a webhook landing mid-sync.
• Mandatory checks:
  • The PR is mergeable against main at 1f1125c.
  • cronjobs.yaml has had only one commit on main, 8ce5ebd, which created it, so nothing conflicts or supersedes this change.
  • The change is not already on main.
  • Issue #1461 asks for exactly this fix, and the encyclopedia half is correctly left to Wiki#22.

🟢 Nits

• The PR body still says "retries up to 6 times, 10 s apart". The code now has --retry 12, so the description is out of date.

Tests

No automated tests exist for this manifest, and none are expected for a one-flag change. The PR says kubectl kustomize kubernetes/apps/wiki renders and that --retry-connrefused was checked against a closed port.

Recommendation

Looks reasonable — no blocking issues; the only remaining comment is a nit about stale PR text.

🤖 **Automated review** — model `claude-sonnet-5-5`, commit `be4e605d65f06c29a22c73d714872ba4ff40a222`. > Automated review by the `review-bot`: it approves only when it found nothing red or yellow, and a human still merges. Push new commits to get a fresh review. --- ## Summary Since my review at `10cb1c2`, one commit (`be4e605`) has been added. It raises `--retry` from 6 to 12 in the `wiki-sync` curl at `kubernetes/apps/wiki/cronjobs.yaml:75`, so the 120 s `--retry-max-time` cap is the real limit. Both of my earlier 🟡 points are resolved, and I found nothing blocking. - **Retry window (earlier 🟡):** fixed. Twelve retries at a 10 s delay now reach the 120 s cap, as the PR describes. - **Retry on 5xx and idempotency (earlier 🟡):** resolved. The new commit message says `POST /api/v1/sync` is serialized by `SyncCoordinator`, and I confirmed that in `konairius/Wiki` `src/Wiki.Application/Sync/SyncCoordinator.cs`. It holds a `SemaphoreSlim(1,1)`, and a caller arriving mid-sync waits and then runs its own sync. A repeated sync is therefore the same as a webhook landing mid-sync. - **Mandatory checks:** - The PR is mergeable against `main` at `1f1125c`. - `cronjobs.yaml` has had only one commit on `main`, `8ce5ebd`, which created it, so nothing conflicts or supersedes this change. - The change is not already on `main`. - Issue #1461 asks for exactly this fix, and the encyclopedia half is correctly left to Wiki#22. ## 🟢 Nits - The PR body still says "retries up to 6 times, 10 s apart". The code now has `--retry 12`, so the description is out of date. ## Tests No automated tests exist for this manifest, and none are expected for a one-flag change. The PR says `kubectl kustomize kubernetes/apps/wiki` renders and that `--retry-connrefused` was checked against a closed port. ## Recommendation Looks reasonable — no blocking issues; the only remaining comment is a nit about stale PR text. <!-- pr-review:sha=be4e605d65f06c29a22c73d714872ba4ff40a222 -->

k.renner merged commit aee26df4b8 into main 2026-10-27 07:53:22 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

review-bot

Labels

Clear labels

apollo

Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant

atlas

Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring

clio

Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites

daedalus

Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes

hestia

Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#1462