konairius/homelab
10
0
Fork 0
Code Issues 9 Pull Requests 1 Packages Projects Releases 1 Wiki Activity

[alert] KubeJobFailed (ai) #1461

New Issue
Closed
opened 2026-10-27 07:21:42 +00:00 by hermes-bot · 1 comment
hermes-bot commented 2026-10-27 07:21:42 +00:00
Member
Copy Link
Copy Source

What is firing: KubeJobFailed (warning), namespace ai, blip within the last 75 min. Two failed Jobs:

  • wiki-encyclopedia-rebuild (manual, cronjob.kubernetes.io/instantiate: manual), failed 2026-10-27T07:08:41Z
  • wiki-sync-29884710 (scheduled 06:30Z), failed 2026-10-27T06:30:02Z

Evidence

  • kubectl -n ai logs wiki-encyclopedia-rebuild-kc8lv → curl: (22) The requested URL returned error: 502 (exit 22, ran 06:30:46–07:08:39Z)
  • kubectl -n ai logs wiki-799f7c55-zr8s8 --timestamps at 07:08:38–39Z:
    fail: Wiki.Application.Encyclopedia.EncyclopediaReconciler / encyclopedia run failed
    System.ArgumentException: An item with the same key has already been added. Key: titleEn (Parameter 'key')
    at Wiki.Application.Encyclopedia.AssignNotesUseCase.Validate(...) in AssignNotesUseCase.cs:line 76 ← AssignAsync line 58 ← EncyclopediaReconciler.AssignAsync line 221
  • kubectl -n ai get rs: the running pod is wiki:build.1793081757, rolled out 2026-10-27T06:29:53Z. The scheduled wiki-encyclopedia-29884620 (before that rollout) completed fine.
  • kubectl -n ai logs wiki-sync-29884710-hfm9g → curl: (7) Failed to connect to wiki.ai.svc.cluster.local:8080 after 1 ms at 06:30:00Z, 7 s after the new wiki pod started; next run wiki-sync-29884740 Completed.

Root cause

  • Encyclopedia (verified): AssignNotesUseCase.Validate throws on a JSON object with a duplicate titleEn key (adding the LLM's output properties into a dictionary without tolerating duplicates), which aborts the whole run and makes /api/v1/encyclopedia/run return 502. Hypothesis: introduced or exposed by build.1793081757; settle by checking git log -- src/Wiki.Application/Encyclopedia/AssignNotesUseCase.cs in the wiki repo.
  • wiki-sync (verified, transient): sync hit the Service during the wiki rollout while no endpoint was ready. Recurs on every rollout that lands near :00/:30 unless the sync tolerates it (retry) or the rollout keeps a ready pod.

Where to fix it

  • Wiki service repo: src/Wiki.Application/Encyclopedia/AssignNotesUseCase.cs Validate (~line 76): handle duplicate keys in the model's JSON (reject/retry the batch instead of throwing out of the run), and don't fail the whole run on one bad batch.
  • Optional: kubernetes/apps/ai/... wiki-sync CronJob: curl --retry with --retry-connrefused, or a rolling update strategy for the wiki Deployment so sync doesn't fail during rollouts.

Filed automatically by the Hermes alert-monitor job (2026-10-27) for the clio expert, whose kanban card takes it to a fix PR.

**What is firing**: `KubeJobFailed` (warning), namespace `ai`, blip within the last 75 min. Two failed Jobs: - `wiki-encyclopedia-rebuild` (manual, `cronjob.kubernetes.io/instantiate: manual`), failed 2026-10-27T07:08:41Z - `wiki-sync-29884710` (scheduled 06:30Z), failed 2026-10-27T06:30:02Z **Evidence** - `kubectl -n ai logs wiki-encyclopedia-rebuild-kc8lv` → `curl: (22) The requested URL returned error: 502` (exit 22, ran 06:30:46–07:08:39Z) - `kubectl -n ai logs wiki-799f7c55-zr8s8 --timestamps` at 07:08:38–39Z: `fail: Wiki.Application.Encyclopedia.EncyclopediaReconciler` / `encyclopedia run failed` `System.ArgumentException: An item with the same key has already been added. Key: titleEn (Parameter 'key')` `at Wiki.Application.Encyclopedia.AssignNotesUseCase.Validate(...) in AssignNotesUseCase.cs:line 76` ← `AssignAsync` line 58 ← `EncyclopediaReconciler.AssignAsync` line 221 - `kubectl -n ai get rs`: the running pod is `wiki:build.1793081757`, rolled out 2026-10-27T06:29:53Z. The scheduled `wiki-encyclopedia-29884620` (before that rollout) completed fine. - `kubectl -n ai logs wiki-sync-29884710-hfm9g` → `curl: (7) Failed to connect to wiki.ai.svc.cluster.local:8080 after 1 ms` at 06:30:00Z, 7 s after the new wiki pod started; next run `wiki-sync-29884740` Completed. **Root cause** - Encyclopedia (verified): `AssignNotesUseCase.Validate` throws on a JSON object with a duplicate `titleEn` key (adding the LLM's output properties into a dictionary without tolerating duplicates), which aborts the whole run and makes `/api/v1/encyclopedia/run` return 502. Hypothesis: introduced or exposed by build.1793081757; settle by checking `git log -- src/Wiki.Application/Encyclopedia/AssignNotesUseCase.cs` in the wiki repo. - wiki-sync (verified, transient): sync hit the Service during the wiki rollout while no endpoint was ready. Recurs on every rollout that lands near :00/:30 unless the sync tolerates it (retry) or the rollout keeps a ready pod. **Where to fix it** - Wiki service repo: `src/Wiki.Application/Encyclopedia/AssignNotesUseCase.cs` `Validate` (~line 76): handle duplicate keys in the model's JSON (reject/retry the batch instead of throwing out of the run), and don't fail the whole run on one bad batch. - Optional: `kubernetes/apps/ai/...` wiki-sync CronJob: `curl --retry` with `--retry-connrefused`, or a rolling update strategy for the wiki Deployment so sync doesn't fail during rollouts. --- <!-- alert-monitor:fingerprint=f654c59e7a724993 --> _Filed automatically by the Hermes alert-monitor job (2026-10-27) for the `clio` expert, whose kanban card takes it to a fix PR._
hermes-bot added the clio label 2026-10-27 07:21:42 +00:00
hermes-bot commented 2026-10-27 07:21:46 +00:00
Author
Member
Copy Link
Copy Source

Hermes kanban card t_4b2b8ff9 for this issue is assigned to clio.

Hermes kanban card `t_4b2b8ff9` for this issue is assigned to clio. <!-- hermes-kanban-card:t_4b2b8ff9 -->
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels clio
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#1461