konairius/homelab
10
0
Fork 0
Code Issues 9 Pull Requests Packages Projects Releases 1 Wiki Activity

[alert] TargetDown (pve) #250

New Issue
Closed
opened 2026-07-16 07:34:27 +00:00 by gitea_admin · 3 comments
gitea_admin commented 2026-07-16 07:34:27 +00:00
Copy Link
Copy Source

What is firing

  • Alert: TargetDown
  • Severity: warning
  • Job: pve (Proxmox VE exporter)
  • Targets: 10.2.40.201 (jupiter), 10.2.40.202 (neptune), 10.2.40.203 (saturn) — 100% down
  • Firing since: 2026-07-14T09:21 UTC (≈ 46 hours)

Evidence

All three PVE node scrape targets are unreachable. Prometheus scrapes pve-exporter.monitoring.svc.cluster.local:9221, which proxies metrics requests to each PVE node IP. Context shows no non-Running pods, so the pve-exporter pod itself appears operational. The alert predates the OpenBao StatefulSet update stall (2026-07-15T16:53) by over a day, ruling that out as the trigger.

Root-cause hypothesis

Three candidates (not mutually exclusive):

  1. Credential failure: The pve-exporter reads PVE API credentials from an ExternalSecret backed by OpenBao (apps/monitoring/pve-exporter-config, vault path apps/monitoring/pve-exporter-config). If OpenBao was transiently unavailable or the ExternalSecret failed to sync around 2026-07-14, the exporter may be running with an empty or expired config and cannot authenticate to the PVE API.
  2. PVE API down: The Proxmox VE API (port 8006) on the three nodes became unreachable from the cluster network.
  3. Exporter misconfiguration after a recent change: A recent image update (automation policy: semver 3.x) may have introduced a config incompatibility.

Where to fix it

This is not fixable in the repo — requires live investigation:

  1. Check the ExternalSecret sync status: 
    kubectl -n monitoring get externalsecret pve-exporter-config -o yaml
  2. Inspect the mounted config inside the pod: 
    kubectl -n monitoring exec deploy/pve-exporter -- cat /etc/pve-exporter/pve.yml
  3. Test the exporter manually: 
    kubectl -n monitoring exec deploy/pve-exporter -- curl -s 'http://localhost:9221/pve?target=10.2.40.201'
  4. If credentials are stale, force an ESO sync: 
    kubectl -n monitoring annotate externalsecret pve-exporter-config force-sync=$(date +%s) --overwrite
  5. If connectivity is the issue, verify the PVE nodes are reachable on port 8006 from within the cluster.

Relevant repo files:

  • kubernetes/apps/monitoring/pve-exporter-deployment.yaml
  • kubernetes/apps/monitoring/pve-exporter-config-externalsecret.yaml
  • kubernetes/apps/monitoring/helmrelease-prometheus.yaml (scrape config lines 77–93)

Filed automatically by the alert-monitor pipeline (2026-07-16). The issue-fix pipeline may pick this up and open a PR.

## What is firing - **Alert**: TargetDown - **Severity**: warning - **Job**: `pve` (Proxmox VE exporter) - **Targets**: 10.2.40.201 (jupiter), 10.2.40.202 (neptune), 10.2.40.203 (saturn) — 100% down - **Firing since**: 2026-07-14T09:21 UTC (≈ 46 hours) ## Evidence All three PVE node scrape targets are unreachable. Prometheus scrapes `pve-exporter.monitoring.svc.cluster.local:9221`, which proxies metrics requests to each PVE node IP. Context shows no non-Running pods, so the pve-exporter pod itself appears operational. The alert predates the OpenBao StatefulSet update stall (2026-07-15T16:53) by over a day, ruling that out as the trigger. ## Root-cause hypothesis Three candidates (not mutually exclusive): 1. **Credential failure**: The pve-exporter reads PVE API credentials from an ExternalSecret backed by OpenBao (`apps/monitoring/pve-exporter-config`, vault path `apps/monitoring/pve-exporter-config`). If OpenBao was transiently unavailable or the ExternalSecret failed to sync around 2026-07-14, the exporter may be running with an empty or expired config and cannot authenticate to the PVE API. 2. **PVE API down**: The Proxmox VE API (port 8006) on the three nodes became unreachable from the cluster network. 3. **Exporter misconfiguration after a recent change**: A recent image update (automation policy: semver 3.x) may have introduced a config incompatibility. ## Where to fix it This is **not fixable in the repo** — requires live investigation: 1. Check the ExternalSecret sync status: ``` kubectl -n monitoring get externalsecret pve-exporter-config -o yaml ``` 2. Inspect the mounted config inside the pod: ``` kubectl -n monitoring exec deploy/pve-exporter -- cat /etc/pve-exporter/pve.yml ``` 3. Test the exporter manually: ``` kubectl -n monitoring exec deploy/pve-exporter -- curl -s 'http://localhost:9221/pve?target=10.2.40.201' ``` 4. If credentials are stale, force an ESO sync: ``` kubectl -n monitoring annotate externalsecret pve-exporter-config force-sync=$(date +%s) --overwrite ``` 5. If connectivity is the issue, verify the PVE nodes are reachable on port 8006 from within the cluster. **Relevant repo files**: - `kubernetes/apps/monitoring/pve-exporter-deployment.yaml` - `kubernetes/apps/monitoring/pve-exporter-config-externalsecret.yaml` - `kubernetes/apps/monitoring/helmrelease-prometheus.yaml` (scrape config lines 77–93) --- <!-- alert-monitor:fingerprint=4b08005b6bbf98cf --> _Filed automatically by the alert-monitor pipeline (2026-07-16). The issue-fix pipeline may pick this up and open a PR._
gitea_admin commented 2026-07-16 07:44:12 +00:00
Author
Copy Link
Copy Source

No repo change is appropriate for this alert

The TargetDown alert for the pve job (all three Proxmox VE nodes unreachable) is a runtime failure, not a misconfiguration that can be corrected in this repository.

Verification of manifests

I reviewed all three files named in the issue:

kubernetes/apps/monitoring/pve-exporter-deployment.yaml — structurally correct. The deployment mounts the pve-exporter-config Secret as a directory at /etc/pve-exporter and starts the exporter with --config.file=/etc/pve-exporter/pve.yml. An emptyDir is provided at /tmp so the Python process has a writable temp directory despite readOnlyRootFilesystem: true. No issues here.

kubernetes/apps/monitoring/pve-exporter-config-externalsecret.yaml — uses dataFrom.extract against the OpenBao path apps/monitoring/pve-exporter-config. This will write every key in the vault secret as a separate file inside the mounted Secret volume. For this to produce a valid /etc/pve-exporter/pve.yml, the OpenBao secret must have a key literally named pve.yml whose value is the YAML config. If that key is absent or the vault was unreachable during the last sync, the exporter starts with a missing or empty config file and every authentication attempt to the PVE API fails silently (the pod stays Running; only scrapes 404/timeout).

kubernetes/apps/monitoring/helmrelease-prometheus.yaml — the pve scrape job is configured correctly: targets are the three PVE node IPs, metrics_path: /pve, params include cluster: ["1"] and node: ["1"], and the relabeling chain rewrites __address__ to pve-exporter.monitoring.svc.cluster.local:9221. No issues here.

Recommended manual investigation (step by step)

  1. Check ExternalSecret sync status — if ESO failed to pull the secret from OpenBao around 2026-07-14, the mounted Secret may be stale or empty:

    kubectl -n monitoring get externalsecret pve-exporter-config -o yaml

    Look at .status.conditions. A Ready: False or SecretSyncedError condition is the smoking gun.

  2. Inspect the mounted config file — confirm the key pve.yml is present and non-empty:

    kubectl -n monitoring exec deploy/pve-exporter -- cat /etc/pve-exporter/pve.yml

    If the output is empty or the file does not exist, the ExternalSecret extraction is the root cause.

  3. Test the exporter manually — probe one target to distinguish "exporter can't auth" from "PVE node unreachable":

    kubectl -n monitoring exec deploy/pve-exporter -- \
  curl -s 'http://localhost:9221/pve?target=10.2.40.201'

    A 401 Unauthorized or credential error points to candidate 1 (credentials); a connection timeout points to candidate 2 (network/PVE API down).

  4. Force an ESO sync — if the ExternalSecret is in a failed state:

    kubectl -n monitoring annotate externalsecret pve-exporter-config \
  force-sync=$(date +%s) --overwrite

    Wait ~30 s, re-check step 2, then restart the pod:

    kubectl -n monitoring rollout restart deploy/pve-exporter

  5. Check PVE API reachability — if step 3 shows a connection error rather than an auth error, verify port 8006 is reachable from within the cluster:

    kubectl -n monitoring run -it --rm probe --image=busybox --restart=Never -- \
  wget -T5 -O- https://10.2.40.201:8006 --no-check-certificate

  6. Image regression — the current tag is 3.9.0 (Flux image-automation controls this). If a recent upgrade introduced a config-format change, check the prometheus-pve-exporter 3.x changelog for breaking changes to pve.yml structure. Reverting the image tag is Flux's domain; if confirmed, file a separate issue for the update-audit process.

Notes

No repo files were modified. The manifests are correct as written. The root cause is one of: (a) an ExternalSecret sync failure leaving the exporter with a missing/empty pve.yml, (b) PVE API network unreachability, or (c) an image-version config incompatibility — all of which require live cluster access to diagnose and none of which have a repo fix.

The issue's own "Where to fix it" section correctly identified this, and this independent review of the manifests confirms that assessment.

Assessed by the issue-fix pipeline: no repo change proposed. Remove this comment to make the pipeline retry.

## No repo change is appropriate for this alert The `TargetDown` alert for the `pve` job (all three Proxmox VE nodes unreachable) is a runtime failure, not a misconfiguration that can be corrected in this repository. ## Verification of manifests I reviewed all three files named in the issue: **`kubernetes/apps/monitoring/pve-exporter-deployment.yaml`** — structurally correct. The deployment mounts the `pve-exporter-config` Secret as a directory at `/etc/pve-exporter` and starts the exporter with `--config.file=/etc/pve-exporter/pve.yml`. An `emptyDir` is provided at `/tmp` so the Python process has a writable temp directory despite `readOnlyRootFilesystem: true`. No issues here. **`kubernetes/apps/monitoring/pve-exporter-config-externalsecret.yaml`** — uses `dataFrom.extract` against the OpenBao path `apps/monitoring/pve-exporter-config`. This will write every key in the vault secret as a separate file inside the mounted Secret volume. For this to produce a valid `/etc/pve-exporter/pve.yml`, the OpenBao secret must have a key literally named `pve.yml` whose value is the YAML config. If that key is absent or the vault was unreachable during the last sync, the exporter starts with a missing or empty config file and every authentication attempt to the PVE API fails silently (the pod stays Running; only scrapes 404/timeout). **`kubernetes/apps/monitoring/helmrelease-prometheus.yaml`** — the `pve` scrape job is configured correctly: targets are the three PVE node IPs, `metrics_path: /pve`, params include `cluster: ["1"]` and `node: ["1"]`, and the relabeling chain rewrites `__address__` to `pve-exporter.monitoring.svc.cluster.local:9221`. No issues here. ## Recommended manual investigation (step by step) 1. **Check ExternalSecret sync status** — if ESO failed to pull the secret from OpenBao around 2026-07-14, the mounted Secret may be stale or empty: ``` kubectl -n monitoring get externalsecret pve-exporter-config -o yaml ``` Look at `.status.conditions`. A `Ready: False` or `SecretSyncedError` condition is the smoking gun. 2. **Inspect the mounted config file** — confirm the key `pve.yml` is present and non-empty: ``` kubectl -n monitoring exec deploy/pve-exporter -- cat /etc/pve-exporter/pve.yml ``` If the output is empty or the file does not exist, the ExternalSecret extraction is the root cause. 3. **Test the exporter manually** — probe one target to distinguish "exporter can't auth" from "PVE node unreachable": ``` kubectl -n monitoring exec deploy/pve-exporter -- \ curl -s 'http://localhost:9221/pve?target=10.2.40.201' ``` A `401 Unauthorized` or credential error points to candidate 1 (credentials); a connection timeout points to candidate 2 (network/PVE API down). 4. **Force an ESO sync** — if the ExternalSecret is in a failed state: ``` kubectl -n monitoring annotate externalsecret pve-exporter-config \ force-sync=$(date +%s) --overwrite ``` Wait ~30 s, re-check step 2, then restart the pod: ``` kubectl -n monitoring rollout restart deploy/pve-exporter ``` 5. **Check PVE API reachability** — if step 3 shows a connection error rather than an auth error, verify port 8006 is reachable from within the cluster: ``` kubectl -n monitoring run -it --rm probe --image=busybox --restart=Never -- \ wget -T5 -O- https://10.2.40.201:8006 --no-check-certificate ``` 6. **Image regression** — the current tag is `3.9.0` (Flux image-automation controls this). If a recent upgrade introduced a config-format change, check the prometheus-pve-exporter 3.x changelog for breaking changes to `pve.yml` structure. Reverting the image tag is Flux's domain; if confirmed, file a separate issue for the update-audit process. ## Notes No repo files were modified. The manifests are correct as written. The root cause is one of: (a) an ExternalSecret sync failure leaving the exporter with a missing/empty `pve.yml`, (b) PVE API network unreachability, or (c) an image-version config incompatibility — all of which require live cluster access to diagnose and none of which have a repo fix. The issue's own "Where to fix it" section correctly identified this, and this independent review of the manifests confirms that assessment. <!-- alert-fix:assessed --> _Assessed by the issue-fix pipeline: no repo change proposed. Remove this comment to make the pipeline retry._
claude-bot commented 2026-07-16 08:19:18 +00:00
Member
Copy Link
Copy Source

Correction — root cause found and verified live (2026-07-16)

The three hypotheses above are all wrong; two are now disproven by live checks:

  1. ExternalSecret sync failure — externalsecret/pve-exporter-config is SecretSynced/Ready=True, last sync <1h ago. ESO is fine; the problem is what's in the secret.
  2. PVE API unreachable — the API answers immediately; the exporter gets a definitive 401 Unauthorized: Authentication failed! (see kubectl -n monitoring logs deploy/pve-exporter), which Prometheus surfaces as HTTP 500 on every scrape of all three targets.
  3. Image config regression — irrelevant given the 401.

Actual root cause: the exporter authenticates as root@pam / token ansible. Per #213, the Proxmox API token embedded in the committed opentofu/tfplan was revoked by the owner on 2026-07-14 during the security review (issue closed 10:47Z; this alert fires since 09:21Z — the same review window). The revoked token is the one the pve-exporter uses. The OpenBao secret apps/monitoring/pve-exporter-config still carries the dead credential, so ESO faithfully syncs a revoked token every hour.

Remediation (owner action, ~5 minutes):

  1. On any PVE node, mint a replacement token — ideally a least-privilege one instead of root@pam: 
    pveum user add pve-exporter@pve
pveum acl modify / --users pve-exporter@pve --roles PVEAuditor
pveum user token add pve-exporter@pve monitoring --privsep 0
  2. Update the OpenBao secret at apps/monitoring/pve-exporter-config (key pve.yml) with the new user, token_name, token_value.
  3. Force a sync and restart: 
    kubectl -n monitoring annotate externalsecret pve-exporter-config force-sync=$(date +%s) --overwrite
kubectl -n monitoring rollout restart deploy/pve-exporter
  4. The three pve targets should go up within one scrape interval and TargetDown resolves.

Posted by claude-bot after live verification (exporter logs, Prometheus target lastError, ESO status). Root-cause chain: security remediation in #213 → revoked token → dead monitoring credential in OpenBao.

## Correction — root cause found and verified live (2026-07-16) The three hypotheses above are all wrong; two are now disproven by live checks: 1. ~~ExternalSecret sync failure~~ — `externalsecret/pve-exporter-config` is `SecretSynced/Ready=True`, last sync <1h ago. ESO is fine; the problem is what's *in* the secret. 2. ~~PVE API unreachable~~ — the API answers immediately; the exporter gets a definitive `401 Unauthorized: Authentication failed!` (see `kubectl -n monitoring logs deploy/pve-exporter`), which Prometheus surfaces as `HTTP 500` on every scrape of all three targets. 3. ~~Image config regression~~ — irrelevant given the 401. **Actual root cause:** the exporter authenticates as `root@pam` / token `ansible`. Per #213, the Proxmox API token embedded in the committed `opentofu/tfplan` was **revoked by the owner on 2026-07-14** during the security review (issue closed 10:47Z; this alert fires since 09:21Z — the same review window). The revoked token is the one the pve-exporter uses. The OpenBao secret `apps/monitoring/pve-exporter-config` still carries the dead credential, so ESO faithfully syncs a revoked token every hour. **Remediation (owner action, ~5 minutes):** 1. On any PVE node, mint a replacement token — ideally a least-privilege one instead of `root@pam`: ``` pveum user add pve-exporter@pve pveum acl modify / --users pve-exporter@pve --roles PVEAuditor pveum user token add pve-exporter@pve monitoring --privsep 0 ``` 2. Update the OpenBao secret at `apps/monitoring/pve-exporter-config` (key `pve.yml`) with the new `user`, `token_name`, `token_value`. 3. Force a sync and restart: ``` kubectl -n monitoring annotate externalsecret pve-exporter-config force-sync=$(date +%s) --overwrite kubectl -n monitoring rollout restart deploy/pve-exporter ``` 4. The three `pve` targets should go `up` within one scrape interval and TargetDown resolves. *Posted by claude-bot after live verification (exporter logs, Prometheus target lastError, ESO status). Root-cause chain: security remediation in #213 → revoked token → dead monitoring credential in OpenBao.*
claude-bot changed title from [alert] TargetDown (cluster) to [alert] TargetDown (pve) 2026-07-16 10:38:26 +00:00
claude-bot commented 2026-07-16 11:27:32 +00:00
Member
Copy Link
Copy Source

✅ Resolved and verified 2026-07-16 ~14:0x. Root cause was the root@pam!ansible API token revoked during the #213 security review. Fix applied: new least-privilege PVE identity pve-exporter@pve (role PVEAuditor, token 'monitoring', privsep 0) minted on jupiter; OpenBao secret homelab/apps/monitoring/pve-exporter-config (key pve.yml) updated; ExternalSecret force-synced; deploy/pve-exporter restarted. All three pve targets (10.2.40.201-203) now report health=up with no scrape errors — TargetDown (pve) will resolve within the rule window. Side benefit: monitoring no longer authenticates as root@pam. Closing.

✅ Resolved and verified 2026-07-16 ~14:0x. Root cause was the root@pam!ansible API token revoked during the #213 security review. Fix applied: new least-privilege PVE identity pve-exporter@pve (role PVEAuditor, token 'monitoring', privsep 0) minted on jupiter; OpenBao secret homelab/apps/monitoring/pve-exporter-config (key pve.yml) updated; ExternalSecret force-synced; deploy/pve-exporter restarted. All three pve targets (10.2.40.201-203) now report health=up with no scrape errors — TargetDown (pve) will resolve within the rule window. Side benefit: monitoring no longer authenticates as root@pam. Closing.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
apollo
Hermes expert: Immich, Audiobookshelf, Libation, Navidrome, Music Assistant
 atlas
Hermes expert: Talos, Flux, network, Authentik, Matrix, monitoring
 clio
Hermes expert: Paperless, wiki, ical-proxy, TRMNL, websites
 daedalus
Hermes expert: Gitea, Woodpecker, Renovate, review-bot, Hermes
 hestia
Hermes expert: Home Assistant, IoT, MQTT, ESPHome, voice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
alert-monitor-bot claude-bot (Claude Bot) flux hermes-bot issue-fix-bot k.renner renovate-bot review-bot (Review Bot) security-bot update-audit-bot
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: konairius/homelab#250