fix(monitoring): stop 3x-ingesting pve cluster metrics #294

2026-07-17T06:47:26Z

claude-bot commented

2026-07-17 06:47:26 +00:00

Root cause found during the 2026-07-17 dashboard-PR review (#287): the single pve scrape job hits all three node IPs with cluster=1, and pve-exporter returns the identical cluster-wide resource set no matter which node answers — every node/guest series is ingested 3x (same id, differing instance). Real TSDB cardinality cost; dashboards only papered over it with max by (id).

Fix per upstream prometheus-pve-exporter guidance:

pve job: cluster collectors only (cluster=1&node=0), scraped from a single node (.201). Trade-off documented in-file: cluster metrics pause if .201 is down.
new pve-node job: node-local collectors (cluster=0&node=1) from all three nodes.

Query-compatible: no dashboard filters on job=, and all pve panels already aggregate by (id) (post-#287), so they render identically with 1x data.

Verify after apply: count by (instance) (pve_up) should show one instance, and per-guest series counts drop from 3 to 1.

🤖 Generated with Claude Code

Root cause found during the 2026-07-17 dashboard-PR review (#287): the single `pve` scrape job hits all three node IPs with `cluster=1`, and pve-exporter returns the identical cluster-wide resource set no matter which node answers — every node/guest series is ingested 3x (same `id`, differing `instance`). Real TSDB cardinality cost; dashboards only papered over it with `max by (id)`. Fix per upstream prometheus-pve-exporter guidance: - `pve` job: cluster collectors only (`cluster=1&node=0`), scraped from a single node (.201). Trade-off documented in-file: cluster metrics pause if .201 is down. - new `pve-node` job: node-local collectors (`cluster=0&node=1`) from all three nodes. Query-compatible: no dashboard filters on `job=`, and all pve panels already aggregate `by (id)` (post-#287), so they render identically with 1x data. Verify after apply: `count by (instance) (pve_up)` should show one instance, and per-guest series counts drop from 3 to 1. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit 2026-07-17 06:47:26 +00:00

fix(monitoring): stop 3x-ingesting pve cluster metrics 16d4996dff

The single pve scrape job hit all three node IPs with cluster=1, and
pve-exporter returns the full cluster-wide resource set regardless of
which node answers - every node/guest series landed in the TSDB three
times (same id, differing instance). Dashboards papered over it with
max by (id); the cardinality cost stayed.

Split per upstream prometheus-pve-exporter guidance: cluster collectors
scrape one node only, node-local collectors keep scraping all three
under a new pve-node job. No dashboard queries filter on job=, and all
already aggregate by id, so this is query-compatible.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

k.renner merged commit cbf642430f into master

2026-07-17 10:59:22 +00:00

k.renner referenced this issue from a commit

2026-07-17 10:59:22 +00:00

Merge pull request 'fix(monitoring): stop 3x-ingesting pve cluster metrics' (#294) from claude-bot/pve-scrape-dedup into master

claude-bot referenced this pull request

2026-07-17 18:22:57 +00:00

fix(monitoring): scrape pve via Service VIP + node FQDNs, no hardcoded IPs #296

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: konairius/homelab#294