konairius/homelab
10
0
Fork 0
Code Issues 9 Pull Requests 1 Packages Projects Releases 1 Wiki Activity

[alert] TargetDown (pve-node) #573

New Issue
Closed
opened 2026-08-25 17:34:11 +00:00 by alert-monitor-bot · 1 comment
alert-monitor-bot commented 2026-08-25 17:34:11 +00:00
Member
Copy Link
Copy Source

What is firing

TargetDown (severity: warning) for job pve-node -- "33.33% of the pve-node/ targets ... are down", i.e. 1 of the 3 statically-configured Proxmox hosts is unreachable. Firing since 2026-08-25T17:20:33Z.

Evidence

  • Live check via ./.ci-diag/kubectl get --raw ".../query?query=up%7Bjob%3D%22pve-node%22%7D": 
    up{instance="saturn.renner.uno",job="pve-node"} = 1
up{instance="neptune.renner.uno",job="pve-node"} = 1
up{instance="jupiter.renner.uno",job="pve-node"} = 0
    So the down target is specifically jupiter.
  • context.txt node list shows only k8s-2 and k8s-3 as Ready -- k8s-1 is absent -- and shows a batch of NodeShutdown/FailedDaemonPod warning events ~52m old for daemonsets across kube-system, monitoring, and ingress, all noting pods "rejected as the node is shutting down".
  • kubernetes/talos/patches/k8s-1.yaml:1 states k8s-1 "runs Talos directly on jupiter's MS-01 hardware" -- so the missing k8s node and the down PVE target are the same physical host.
  • git log shows PR #570, "chore(immich): take server and machine-learning down for the jupiter window", merged 2026-08-25T15:36:48Z (about 1h44m before this alert started). kubernetes/apps/immich/server.yaml:39-42 has replicas deliberately set to 0 with the comment: "TEMPORARY -- RESTORE TO 1 ONCE k8s-1 HAS REJOINED ... immich is offline for the jupiter conversion."

Root cause

Hypothesis, well-supported but not 100% certain: jupiter is currently offline/mid-reinstall as part of a deliberate, already-in-progress "jupiter conversion" (the same operation that took k8s-1 out of the cluster and prompted PR #570 to scale immich to 0). The PVE node-exporter on jupiter (scraped via kubernetes/apps/monitoring/helmrelease-prometheus.yaml job pve-node, target jupiter.renner.uno) is unreachable simply because the host itself is down for that work, not because of an unrelated fault.

To fully confirm intent (vs. an unplanned host crash), a human with access to the actual maintenance plan / Proxmox console should verify jupiter is up and rejoin is complete.

Where to fix

Not a repo config fix -- nothing about the scrape config (kubernetes/apps/monitoring/helmrelease-prometheus.yaml, kubernetes/apps/pve/endpointslice.yaml) is wrong; jupiter is a real, intentionally-defined target. Action needed is operational:

  1. Confirm jupiter/k8s-1 has rejoined (kubectl get nodes, and up{job="pve-node"} should show all three instances at 1).
  2. Once confirmed, restore immich-server and immich-machine-learning replica counts per the TODO comments in kubernetes/apps/immich/server.yaml and kubernetes/apps/immich/machine-learning.yaml.
  3. Close this issue -- do not mark it permanently noise-suppressed, since this alert's fingerprint is per-job (not per-instance) and would also match a genuine future outage of neptune or saturn.

Filed automatically by the alert-monitor pipeline (2026-08-25). The issue-fix pipeline may pick this up and open a PR.

## What is firing `TargetDown` (severity: warning) for job `pve-node` -- "33.33% of the pve-node/ targets ... are down", i.e. 1 of the 3 statically-configured Proxmox hosts is unreachable. Firing since `2026-08-25T17:20:33Z`. ## Evidence - Live check via `./.ci-diag/kubectl get --raw ".../query?query=up%7Bjob%3D%22pve-node%22%7D"`: ``` up{instance="saturn.renner.uno",job="pve-node"} = 1 up{instance="neptune.renner.uno",job="pve-node"} = 1 up{instance="jupiter.renner.uno",job="pve-node"} = 0 ``` So the down target is specifically **jupiter**. - `context.txt` node list shows only `k8s-2` and `k8s-3` as `Ready` -- `k8s-1` is absent -- and shows a batch of `NodeShutdown`/`FailedDaemonPod` warning events ~52m old for daemonsets across `kube-system`, `monitoring`, and `ingress`, all noting pods "rejected as the node is shutting down". - `kubernetes/talos/patches/k8s-1.yaml:1` states k8s-1 "runs Talos directly on jupiter's MS-01 hardware" -- so the missing k8s node and the down PVE target are the same physical host. - `git log` shows PR #570, "chore(immich): take server and machine-learning down for the jupiter window", merged 2026-08-25T15:36:48Z (about 1h44m before this alert started). `kubernetes/apps/immich/server.yaml:39-42` has replicas deliberately set to 0 with the comment: "TEMPORARY -- RESTORE TO 1 ONCE k8s-1 HAS REJOINED ... immich is offline for the jupiter conversion." ## Root cause **Hypothesis, well-supported but not 100% certain:** jupiter is currently offline/mid-reinstall as part of a deliberate, already-in-progress "jupiter conversion" (the same operation that took k8s-1 out of the cluster and prompted PR #570 to scale immich to 0). The PVE node-exporter on jupiter (scraped via `kubernetes/apps/monitoring/helmrelease-prometheus.yaml` job `pve-node`, target `jupiter.renner.uno`) is unreachable simply because the host itself is down for that work, not because of an unrelated fault. To fully confirm intent (vs. an unplanned host crash), a human with access to the actual maintenance plan / Proxmox console should verify jupiter is up and rejoin is complete. ## Where to fix Not a repo config fix -- nothing about the scrape config (`kubernetes/apps/monitoring/helmrelease-prometheus.yaml`, `kubernetes/apps/pve/endpointslice.yaml`) is wrong; jupiter is a real, intentionally-defined target. Action needed is operational: 1. Confirm jupiter/k8s-1 has rejoined (`kubectl get nodes`, and `up{job="pve-node"}` should show all three instances at 1). 2. Once confirmed, restore `immich-server` and `immich-machine-learning` replica counts per the TODO comments in `kubernetes/apps/immich/server.yaml` and `kubernetes/apps/immich/machine-learning.yaml`. 3. Close this issue -- do not mark it permanently noise-suppressed, since this alert's fingerprint is per-`job` (not per-instance) and would also match a genuine future outage of neptune or saturn. --- <!-- alert-monitor:fingerprint=0782449c57a0337e --> _Filed automatically by the alert-monitor pipeline (2026-08-25). The issue-fix pipeline may pick this up and open a PR._
issue-fix-bot commented 2026-08-25 18:48:50 +00:00
Member
Copy Link
Copy Source

🤖 Opened k.renner/homelab#577 with a proposed fix.

🤖 Opened https://gitea.renner.uno/k.renner/homelab/pulls/577 with a proposed fix.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#573