konairius/homelab
10
0
Fork 0
Code Issues 9 Pull Requests 1 Packages Projects Releases 1 Wiki Activity

[alert] TargetDown (pve) #579

New Issue
Closed
opened 2026-08-25 19:05:38 +00:00 by alert-monitor-bot · 1 comment
alert-monitor-bot commented 2026-08-25 19:05:38 +00:00
Member
Copy Link
Copy Source

What is firing

  • Alert: TargetDown, severity warning, job pve (Prometheus scrape pool, no namespace label)
  • Firing since: 2026-08-25 18:58:33 UTC, still active
  • Annotation: "100% of the pve/ targets are down."
  • Effect: every cluster-wide Proxmox metric (pve_up, pve_guest_info, pve_version_info, storage status) has stopped being ingested. The proxmox-cluster Grafana dashboard is blind.

Evidence

1. The target is down with an HTTP 500 from the exporter — Prometheus targets API (kubectl get --raw ".../kube-prometheus-stack-prometheus:9090/proxy/api/v1/targets?state=any&scrapePool=pve"):

scrapeUrl: http://pve-exporter.monitoring.svc.cluster.local:9221/pve?cluster=1&node=0&target=pve.apps.svc.cluster.local
health:    down
lastError: server returned HTTP status 500 INTERNAL SERVER ERROR

2. The 500 is a connection failure to the PVE API — pve-exporter logs via Loki ({namespace="monitoring",pod=~"pve-exporter.*"} |= "pve.apps.svc.cluster.local"):

requests.exceptions.ConnectionError: HTTPSConnectionPool(host='pve.apps.svc.cluster.local', port=8006):
  Max retries exceeded with url: /api2/json/cluster/status
  (Caused by NewConnectionError(... [Errno 113] Host is unreachable))

Errno 113 (EHOSTUNREACH) means no route to the backend, not an auth or TLS problem — so this is not a token-rotation failure.

3. The unreachable backend is jupiter specifically. The sibling pve-node job scrapes each node by name, and only jupiter fails (same targets API, scrapePool=pve-node):

jupiter.renner.uno  health=down  lastError=server returned HTTP status 500 INTERNAL SERVER ERROR
neptune.renner.uno  health=up    lastError=""
saturn.renner.uno   health=up    lastError=""

The jupiter traceback names the host directly: HTTPSConnectionPool(host='jupiter.renner.uno', port=8006) ... [Errno 113] Host is unreachable.

4. Timeline (Prometheus query_range, 10-min step):

Time (UTC) Observation
16:06 pve_up{id="node/jupiter"}=1, pve_up{id="qemu/150"}=1 — jupiter healthy, k8s-1 VM running
16:30–16:45 kube_node_info{node="k8s-1"} disappears — k8s-1 leaves the cluster
17:00–17:10 up{job="pve-node",instance="jupiter.renner.uno"} goes 1 → 0
18:58 TargetDown (pve) fires

up{job="pve"} has been mostly 0 since ~18:38 with brief flaps back to 1 — the flapping is the ClusterIP load balancer occasionally routing a whole scrape's worth of API calls to a surviving node.

Root cause

Two layers, both verified:

(a) jupiter is offline, and this looks deliberate — it is being converted from Proxmox to bare-metal Talos. kubernetes/talos/patches/k8s-1.yaml:1 states plainly that "k8s-1 runs Talos directly on jupiter's MS-01 hardware; k8s-2 and k8s-3 are still Proxmox VMs", and that the EPHEMERAL VolumeConfig "ONLY TAKES EFFECT ON FIRST PROVISION" — i.e. it requires a wipe and a fresh apply-config --insecure. That change merged as PR #575 at 18:33 UTC, ~1.5h after k8s-1 left the cluster and ~1.3h after jupiter's PVE API went dark. The ordering (VM leaves cluster → PVE host goes offline → bare-metal Talos config merged) is consistent with an in-flight migration. This part is a hypothesis about intent; the outage itself is verified. It can be settled from the host console or with talosctl -n 10.2.40.201 version.

(b) The monitoring config cannot survive one PVE node being gone — which is exactly what it claims to do. kubernetes/apps/pve/endpointslice.yaml is a hand-written EndpointSlice with conditions: ready: true hardcoded for all three nodes. Nothing health-checks those addresses, so 10.2.40.201 stays in rotation forever. The pve job's exporter makes several API calls per scrape; if any one lands on jupiter the whole scrape 500s. The comment at kubernetes/apps/monitoring/helmrelease-prometheus.yaml:111-116 asserts the opposite — that targeting the Service means "cluster metrics keep flowing as long as ANY node is up, instead of pausing whenever .201 (jupiter) specifically is down." That guarantee does not hold, and this outage is the proof.

So even after the migration settles, the alert will not clear on its own: jupiter is not coming back as a PVE node, and both its EndpointSlice entry and its pve-node scrape target are now permanently stale.

Not verified: I could not read the live EndpointSlice — the CI ServiceAccount lacks list on discovery.k8s.io/endpointslices in apps. All Flux kustomizations report Ready, so applied state should match the manifest. Confirm with kubectl get endpointslice -n apps -l kubernetes.io/service-name=pve -o yaml.

Where to fix it

  1. kubernetes/apps/pve/endpointslice.yaml — remove the 10.2.40.201 / jupiter endpoint (or set conditions.ready: false). This alone restores the pve job, since neptune and saturn are healthy. The file's header comment about jupiter keeping .201 also needs updating if the host is now a Talos node.
  2. kubernetes/apps/monitoring/helmrelease-prometheus.yaml:143 — drop jupiter.renner.uno from the pve-node job's static_configs. It is currently down too and will keep a second TargetDown firing.
  3. Same file, lines 111–116 — the comment now overstates the design. A static EndpointSlice gives round-robin without health checking, so any single PVE node outage breaks 100% of the cluster-collector job. If real failover is wanted, the options are a readiness-checked mechanism or accepting per-node scraping for the cluster collector too.
  4. Stale references to jupiter as a Proxmox host, worth sweeping in the same change: ansible/inventory/pve_hosts.yml:14 (still in [pve_cluster], and therefore in [ntp_cluster:children]) and ansible/plays/configure-pve-k8s-zfs-tuning.yml:15 ("jupiter→150").

Not fixable in the repo: bringing jupiter/k8s-1 back online. If the Talos reprovision is mid-flight, a human needs to finish it; if the host is unexpectedly down, it needs console/IPMI attention.

Related cluster state (context for whoever picks this up)

Other symptoms in the same snapshot trace back to k8s-1 being absent, and are expected to clear when it rejoins — flagged here only so they are not chased separately: rook-ceph-mon-c Pending with "didn't match Pod's node affinity/selector", the rook-ceph-mon-d-canary reschedule loop, woodpecker-ci pods with unbound PVCs, FreeDiskSpaceFailed on k8s-3 at 78% of 59.9 GiB, and probe_success{instance="https://immich.renner.uno"}=0. The k8s-1 patch comment notes that the 64 GB VM-era /var is precisely why k8s-2 and k8s-3 hit DiskPressure when they absorb another node's pods, which is what is happening now.

Filed automatically by the alert-monitor pipeline (2026-08-25). The issue-fix pipeline may pick this up and open a PR.

## What is firing - **Alert:** `TargetDown`, severity `warning`, job `pve` (Prometheus scrape pool, no namespace label) - **Firing since:** 2026-08-25 18:58:33 UTC, still active - **Annotation:** "100% of the pve/ targets are down." - **Effect:** every cluster-wide Proxmox metric (`pve_up`, `pve_guest_info`, `pve_version_info`, storage status) has stopped being ingested. The proxmox-cluster Grafana dashboard is blind. ## Evidence **1. The target is down with an HTTP 500 from the exporter** — Prometheus targets API (`kubectl get --raw ".../kube-prometheus-stack-prometheus:9090/proxy/api/v1/targets?state=any&scrapePool=pve"`): ``` scrapeUrl: http://pve-exporter.monitoring.svc.cluster.local:9221/pve?cluster=1&node=0&target=pve.apps.svc.cluster.local health: down lastError: server returned HTTP status 500 INTERNAL SERVER ERROR ``` **2. The 500 is a connection failure to the PVE API** — pve-exporter logs via Loki (`{namespace="monitoring",pod=~"pve-exporter.*"} |= "pve.apps.svc.cluster.local"`): ``` requests.exceptions.ConnectionError: HTTPSConnectionPool(host='pve.apps.svc.cluster.local', port=8006): Max retries exceeded with url: /api2/json/cluster/status (Caused by NewConnectionError(... [Errno 113] Host is unreachable)) ``` `Errno 113` (EHOSTUNREACH) means no route to the backend, not an auth or TLS problem — so this is **not** a token-rotation failure. **3. The unreachable backend is jupiter specifically.** The sibling `pve-node` job scrapes each node by name, and only jupiter fails (same targets API, `scrapePool=pve-node`): ``` jupiter.renner.uno health=down lastError=server returned HTTP status 500 INTERNAL SERVER ERROR neptune.renner.uno health=up lastError="" saturn.renner.uno health=up lastError="" ``` The jupiter traceback names the host directly: `HTTPSConnectionPool(host='jupiter.renner.uno', port=8006) ... [Errno 113] Host is unreachable`. **4. Timeline** (Prometheus `query_range`, 10-min step): | Time (UTC) | Observation | |---|---| | 16:06 | `pve_up{id="node/jupiter"}=1`, `pve_up{id="qemu/150"}=1` — jupiter healthy, k8s-1 VM running | | 16:30–16:45 | `kube_node_info{node="k8s-1"}` disappears — k8s-1 leaves the cluster | | 17:00–17:10 | `up{job="pve-node",instance="jupiter.renner.uno"}` goes 1 → 0 | | 18:58 | `TargetDown` (pve) fires | `up{job="pve"}` has been mostly 0 since ~18:38 with brief flaps back to 1 — the flapping is the ClusterIP load balancer occasionally routing a whole scrape's worth of API calls to a surviving node. ## Root cause Two layers, both verified: **(a) jupiter is offline, and this looks deliberate — it is being converted from Proxmox to bare-metal Talos.** `kubernetes/talos/patches/k8s-1.yaml:1` states plainly that "k8s-1 runs Talos directly on jupiter's MS-01 hardware; k8s-2 and k8s-3 are still Proxmox VMs", and that the EPHEMERAL VolumeConfig "ONLY TAKES EFFECT ON FIRST PROVISION" — i.e. it requires a wipe and a fresh `apply-config --insecure`. That change merged as PR #575 at 18:33 UTC, ~1.5h after k8s-1 left the cluster and ~1.3h after jupiter's PVE API went dark. The ordering (VM leaves cluster → PVE host goes offline → bare-metal Talos config merged) is consistent with an in-flight migration. *This part is a hypothesis about intent*; the outage itself is verified. It can be settled from the host console or with `talosctl -n 10.2.40.201 version`. **(b) The monitoring config cannot survive one PVE node being gone — which is exactly what it claims to do.** `kubernetes/apps/pve/endpointslice.yaml` is a hand-written EndpointSlice with `conditions: ready: true` hardcoded for all three nodes. Nothing health-checks those addresses, so 10.2.40.201 stays in rotation forever. The `pve` job's exporter makes several API calls per scrape; if any one lands on jupiter the whole scrape 500s. The comment at `kubernetes/apps/monitoring/helmrelease-prometheus.yaml:111-116` asserts the opposite — that targeting the Service means "cluster metrics keep flowing as long as ANY node is up, instead of pausing whenever .201 (jupiter) specifically is down." That guarantee does not hold, and this outage is the proof. So even after the migration settles, the alert will not clear on its own: jupiter is not coming back as a PVE node, and both its EndpointSlice entry and its `pve-node` scrape target are now permanently stale. *Not verified:* I could not read the live EndpointSlice — the CI ServiceAccount lacks `list` on `discovery.k8s.io/endpointslices` in `apps`. All Flux kustomizations report Ready, so applied state should match the manifest. Confirm with `kubectl get endpointslice -n apps -l kubernetes.io/service-name=pve -o yaml`. ## Where to fix it 1. `kubernetes/apps/pve/endpointslice.yaml` — remove the `10.2.40.201` / `jupiter` endpoint (or set `conditions.ready: false`). This alone restores the `pve` job, since neptune and saturn are healthy. The file's header comment about jupiter keeping `.201` also needs updating if the host is now a Talos node. 2. `kubernetes/apps/monitoring/helmrelease-prometheus.yaml:143` — drop `jupiter.renner.uno` from the `pve-node` job's `static_configs`. It is currently down too and will keep a second `TargetDown` firing. 3. Same file, lines 111–116 — the comment now overstates the design. A static EndpointSlice gives round-robin without health checking, so any single PVE node outage breaks 100% of the cluster-collector job. If real failover is wanted, the options are a readiness-checked mechanism or accepting per-node scraping for the cluster collector too. 4. Stale references to jupiter as a Proxmox host, worth sweeping in the same change: `ansible/inventory/pve_hosts.yml:14` (still in `[pve_cluster]`, and therefore in `[ntp_cluster:children]`) and `ansible/plays/configure-pve-k8s-zfs-tuning.yml:15` ("jupiter→150"). **Not fixable in the repo:** bringing jupiter/k8s-1 back online. If the Talos reprovision is mid-flight, a human needs to finish it; if the host is unexpectedly down, it needs console/IPMI attention. ## Related cluster state (context for whoever picks this up) Other symptoms in the same snapshot trace back to k8s-1 being absent, and are expected to clear when it rejoins — flagged here only so they are not chased separately: `rook-ceph-mon-c` Pending with "didn't match Pod's node affinity/selector", the `rook-ceph-mon-d-canary` reschedule loop, woodpecker-ci pods with unbound PVCs, `FreeDiskSpaceFailed` on k8s-3 at 78% of 59.9 GiB, and `probe_success{instance="https://immich.renner.uno"}=0`. The k8s-1 patch comment notes that the 64 GB VM-era `/var` is precisely why k8s-2 and k8s-3 hit DiskPressure when they absorb another node's pods, which is what is happening now. --- <!-- alert-monitor:fingerprint=4b08005b6bbf98cf --> _Filed automatically by the alert-monitor pipeline (2026-08-25). The issue-fix pipeline may pick this up and open a PR._
issue-fix-bot commented 2026-08-25 19:45:19 +00:00
Member
Copy Link
Copy Source

🤖 Opened k.renner/homelab#583 with a proposed fix.

🤖 Opened https://gitea.renner.uno/k.renner/homelab/pulls/583 with a proposed fix.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#579