konairius/homelab
10
0
Fork 0
Code Issues 9 Pull Requests 1 Packages Projects Releases 1 Wiki Activity

[alert] KubeJobFailed (ai) #1465

New Issue
Open
opened 2026-10-27 08:21:28 +00:00 by hermes-bot · 2 comments
hermes-bot commented 2026-10-27 08:21:28 +00:00
Member
Copy Link
Copy Source

What is firing: KubeJobFailed (warning), ai/wiki-encyclopedia-rebuild, since 2026-10-27T07:23:57Z. The Job is a manual run from CronJob wiki-encyclopedia (annotation cronjob.kubernetes.io/instantiate: manual, backoffLimit 0).

Evidence

  • kubectl -n ai logs wiki-encyclopedia-rebuild-kc8lv: curl: (22) The requested URL returned error: 502 (pod ran 06:30:46Z to 07:08:39Z, exit 22). The Job POSTs /api/v1/encyclopedia/run on wiki.ai.svc.cluster.local:8080.
  • Loki {namespace="ai", pod="wiki-799f7c55-zr8s8"} at ~07:08Z (image gitea.renner.uno/konairius/wiki:build.1793081757):
    • fail: Wiki.Application.Encyclopedia.EncyclopediaReconciler: Planning failed unexpectedly; committing the plan so far, then failing the run
    • System.ArgumentException: An item with the same key has already been added. Key: titleEn (Parameter 'key') at AssignNotesUseCase.Validate(...) AssignNotesUseCase.cs:line 76, called from AssignAsync line 58, EncyclopediaReconciler.AssignAsync line 221 / PlanStepsAsync line 173.
    • fail: EncyclopediaCoordinator: encyclopedia run failed
  • The scheduled run wiki-encyclopedia-29884620 (05:00Z) completed; a newer manual run wiki-encyclopedia-rebuild-2 is running on a new wiki image build.1793086240 (pod wiki-b65b96d95-vdgsm, started 07:39Z). The failed Job object remains, so the alert stays until it is deleted or a later run succeeds.

Root cause: verified application bug in the wiki service: AssignNotesUseCase.Validate adds keys to a dictionary and throws on a duplicate titleEn key (presumably from the LLM's assignment JSON), which aborts the whole encyclopedia run; the endpoint then answers 502 to the caller. Whether build.1793086240 already fixes this is unverified; the running wiki-encyclopedia-rebuild-2 will settle it (kubectl -n ai get job wiki-encyclopedia-rebuild-2, and Loki for same key has already been added).

Where to fix it: wiki service source, src/Wiki.Application/Encyclopedia/AssignNotesUseCase.cs Validate (~line 76): tolerate or reject duplicate keys per item instead of throwing out of the run. Afterwards delete the failed Job ai/wiki-encyclopedia-rebuild (or let a successful run clear the alert).

Filed automatically by the Hermes alert-monitor job (2026-10-27) for the clio expert, whose kanban card takes it to a fix PR.

**What is firing**: `KubeJobFailed` (warning), `ai/wiki-encyclopedia-rebuild`, since 2026-10-27T07:23:57Z. The Job is a manual run from CronJob `wiki-encyclopedia` (annotation `cronjob.kubernetes.io/instantiate: manual`, backoffLimit 0). **Evidence** - `kubectl -n ai logs wiki-encyclopedia-rebuild-kc8lv`: `curl: (22) The requested URL returned error: 502` (pod ran 06:30:46Z to 07:08:39Z, exit 22). The Job POSTs `/api/v1/encyclopedia/run` on `wiki.ai.svc.cluster.local:8080`. - Loki `{namespace="ai", pod="wiki-799f7c55-zr8s8"}` at ~07:08Z (image `gitea.renner.uno/konairius/wiki:build.1793081757`): - `fail: Wiki.Application.Encyclopedia.EncyclopediaReconciler: Planning failed unexpectedly; committing the plan so far, then failing the run` - `System.ArgumentException: An item with the same key has already been added. Key: titleEn (Parameter 'key')` at `AssignNotesUseCase.Validate(...)` `AssignNotesUseCase.cs:line 76`, called from `AssignAsync` line 58, `EncyclopediaReconciler.AssignAsync` line 221 / `PlanStepsAsync` line 173. - `fail: EncyclopediaCoordinator: encyclopedia run failed` - The scheduled run `wiki-encyclopedia-29884620` (05:00Z) completed; a newer manual run `wiki-encyclopedia-rebuild-2` is running on a new wiki image `build.1793086240` (pod `wiki-b65b96d95-vdgsm`, started 07:39Z). The failed Job object remains, so the alert stays until it is deleted or a later run succeeds. **Root cause**: verified application bug in the wiki service: `AssignNotesUseCase.Validate` adds keys to a dictionary and throws on a duplicate `titleEn` key (presumably from the LLM's assignment JSON), which aborts the whole encyclopedia run; the endpoint then answers 502 to the caller. Whether build.1793086240 already fixes this is unverified; the running `wiki-encyclopedia-rebuild-2` will settle it (`kubectl -n ai get job wiki-encyclopedia-rebuild-2`, and Loki for `same key has already been added`). **Where to fix it**: wiki service source, `src/Wiki.Application/Encyclopedia/AssignNotesUseCase.cs` `Validate` (~line 76): tolerate or reject duplicate keys per item instead of throwing out of the run. Afterwards delete the failed Job `ai/wiki-encyclopedia-rebuild` (or let a successful run clear the alert). --- <!-- alert-monitor:fingerprint=9b99a84694dd35c1 --> _Filed automatically by the Hermes alert-monitor job (2026-10-27) for the `clio` expert, whose kanban card takes it to a fix PR._
hermes-bot added the clio label 2026-10-27 08:21:28 +00:00
hermes-bot commented 2026-10-27 08:21:33 +00:00
Author
Member
Copy Link
Copy Source

Hermes kanban card t_dbcdcb15 for this issue is assigned to clio.

Hermes kanban card `t_dbcdcb15` for this issue is assigned to clio. <!-- hermes-kanban-card:t_dbcdcb15 -->
hermes-bot commented 2026-10-27 08:23:22 +00:00
Author
Member
Copy Link
Copy Source

Hermes kanban card t_dbcdcb15 completed by clio:

I didn't open a PR because the bug is already fixed. The alert is still firing only because the failed Job object is still there. One manual step clears it: kubectl -n ai delete job wiki-encyclopedia-rebuild.

What I checked (2026-10-27 ~08:25Z):

  • Alert: KubeJobFailed for ai/wiki-encyclopedia-rebuild is firing in both Prometheus and Alertmanager (since 07:23:57Z). The Job shows Failed, reason BackoffLimitExceeded.
  • Cause: Loki for pod wiki-799f7c55-zr8s8 (image build.1793081757) shows ArgumentException: An item with the same key has already been added. Key: titleEn in AssignNotesUseCase.Validate, then "encyclopedia run failed". The LLM's reply named the titleEn field twice, and that one bad reply failed the whole run.
  • Fix: it is already merged in the wiki service repo, konairius/Wiki, not in homelab. PR #22 (commit af7ca24, merge f93dfd1 at 07:29:22Z) makes PlanParsing.Object parse with AllowDuplicateProperties = false. A reply like that is now treated as unusable instead of failing the run, and there is a regression test.
  • Deployed: that merge's test and release CI passed (release done 07:32Z). The running image is build.1793086240; its timestamp is 07:30:40Z, the start of that release build. The image has no revision label, so this link rests on the timing.
  • New pod: in Loki, wiki-b65b96d95-vdgsm has processed 49 assign batches with no same key error and no fail: lines.
  • Rerun: Job wiki-encyclopedia-rebuild-2 was still running when I checked (active 1, started 07:39:34Z), so it has not finished yet.

Why the alert won't clear on its own soon: the failed Job belongs to CronJob wiki-encyclopedia, which keeps finished Jobs for 7 days (ttlSecondsAfterFinished: 604800) and keeps up to 3 failed ones. A later successful run does not remove it, contrary to what the issue says. Left alone, it stays until about 2026-11-03 07:08Z.

What a human has to do:

  1. Wait for kubectl -n ai get job wiki-encyclopedia-rebuild-2 to show Complete. If it shows Failed, check Loki {namespace="ai", pod=~"wiki-.*"} |= "fail:" instead.
  2. Run kubectl -n ai delete job wiki-encyclopedia-rebuild. I can't do this myself because my cluster access is read-only.
  3. KubeJobFailed{namespace="ai"} should then resolve within a few minutes, and #1465 can be closed.

This is in my domain (wiki service). I added the cause, the fix and the 7-day retention to the wiki as a fact (raw commit d5fccf0d).

Hermes kanban card `t_dbcdcb15` completed by clio: I didn't open a PR because the bug is already fixed. The alert is still firing only because the failed Job object is still there. One manual step clears it: `kubectl -n ai delete job wiki-encyclopedia-rebuild`. **What I checked (2026-10-27 ~08:25Z):** - **Alert:** `KubeJobFailed` for `ai/wiki-encyclopedia-rebuild` is firing in both Prometheus and Alertmanager (since 07:23:57Z). The Job shows `Failed`, reason `BackoffLimitExceeded`. - **Cause:** Loki for pod `wiki-799f7c55-zr8s8` (image `build.1793081757`) shows `ArgumentException: An item with the same key has already been added. Key: titleEn` in `AssignNotesUseCase.Validate`, then "encyclopedia run failed". The LLM's reply named the `titleEn` field twice, and that one bad reply failed the whole run. - **Fix:** it is already merged in the wiki service repo, `konairius/Wiki`, not in homelab. PR #22 (commit `af7ca24`, merge `f93dfd1` at 07:29:22Z) makes `PlanParsing.Object` parse with `AllowDuplicateProperties = false`. A reply like that is now treated as unusable instead of failing the run, and there is a regression test. - **Deployed:** that merge's test and release CI passed (release done 07:32Z). The running image is `build.1793086240`; its timestamp is 07:30:40Z, the start of that release build. The image has no revision label, so this link rests on the timing. - **New pod:** in Loki, `wiki-b65b96d95-vdgsm` has processed 49 assign batches with no `same key` error and no `fail:` lines. - **Rerun:** Job `wiki-encyclopedia-rebuild-2` was still running when I checked (active 1, started 07:39:34Z), so it has not finished yet. **Why the alert won't clear on its own soon:** the failed Job belongs to CronJob `wiki-encyclopedia`, which keeps finished Jobs for 7 days (`ttlSecondsAfterFinished: 604800`) and keeps up to 3 failed ones. A later successful run does not remove it, contrary to what the issue says. Left alone, it stays until about 2026-11-03 07:08Z. **What a human has to do:** 1. Wait for `kubectl -n ai get job wiki-encyclopedia-rebuild-2` to show `Complete`. If it shows `Failed`, check Loki `{namespace="ai", pod=~"wiki-.*"} |= "fail:"` instead. 2. Run `kubectl -n ai delete job wiki-encyclopedia-rebuild`. I can't do this myself because my cluster access is read-only. 3. `KubeJobFailed{namespace="ai"}` should then resolve within a few minutes, and #1465 can be closed. This is in my domain (wiki service). I added the cause, the fix and the 7-day retention to the wiki as a fact (raw commit `d5fccf0d`). <!-- hermes-kanban-done:t_dbcdcb15:57 -->
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels clio
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#1465