Rotates pve-exporter@pve!monitoring via the Proxmox API. Proxmox has no in-place secret roll, so DELETE+re-create (privsep=0), regenerate pve.yml with the new token_value, patch OpenBao. openbao-ns pattern; per-secret rotator-proxmox role reads a bootstrap privileged Proxmox token + patches pve-exporter-config. refreshInterval 1h->10m; Reloader restarts pve-exporter.
SHIPPED SUSPENDED + UNTESTED (riskiest Class-B): needs a bootstrap privileged Proxmox token at homelab/infrastructure/secret-rotators-bootstrap/proxmox, VERIFICATION that the token uses privsep=0 (else re-grant its ACL after re-create), and awareness of the brief monitoring gap during DELETE+create. Test before spec.suspend=false.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
No hardcoded IP remains in the pve scrape config: the node job now
targets jupiter/neptune/saturn.renner.uno. instance label reads the
hostname, and a renumbered node follows DNS without a config edit.
Verified: exporter returns node-local metrics when targeted at
jupiter.renner.uno from inside the cluster.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
The pve cluster-collector job hardcoded 10.2.40.201, so cluster-wide
metrics (pve_up, guest info, storage) paused whenever jupiter
specifically was down, even with the cluster healthy. Point it at the
pve Service DNS (pve.apps.svc.cluster.local) whose EndpointSlice
load-balances the PVE API across all three nodes, so the scrape
follows whichever node is up. Node-local job keeps the three explicit
targets — it needs per-node identity.
Verified: exporter returns pve_up for cluster + all nodes when
targeted at the Service DNS.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015K7gLUVVnutvDnzV2TrMRJ
The single pve scrape job hit all three node IPs with cluster=1, and
pve-exporter returns the full cluster-wide resource set regardless of
which node answers - every node/guest series landed in the TSDB three
times (same id, differing instance). Dashboards papered over it with
max by (id); the cardinality cost stayed.
Split per upstream prometheus-pve-exporter guidance: cluster collectors
scrape one node only, node-local collectors keep scraping all three
under a new pve-node job. No dashboard queries filter on job=, and all
already aggregate by id, so this is query-compatible.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every panel was dead: all 11 datasource refs pointed at the
prometheus-legacy datasource (retired 2026-05-09). Repoint to the
active prometheus datasource.
The alloy job also carries no 'host' label, so all legends and the
Agent Versions table rendered empty labels. Join via kube_pod_info
(pod -> node) so DaemonSet panels are labelled k8s-1/2/3, collapse
the eval-time histogram per pod before quantile, and tidy the
versions table to node/pod/version columns.
All queries verified against live Prometheus (3 agents, one clean
series per node).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
All 10 panels were dead: the dashboard queried datasource uid
prometheus-legacy (prometheus.srv.renner.uno:9090), which no longer
resolves (Grafana proxy: 502 dial tcp lookup i/o timeout). Gitea now
runs in-cluster and is scraped by the in-cluster Prometheus
(job="gitea", instance gitea.apps.svc.cluster.local:3000), so switch
every panel to uid prometheus.
Also wrap all targets in max(): the pre-migration instance
gitea.srv.renner.uno:3000 stopped 2026-07-12, and windows spanning
that date would otherwise render two identically-named series per
query. Stat panels additionally get instant:true so a dead exporter
shows No data instead of a stale last value.
- Historical Cost Comparison: the chained joins renamed _value columns to
_value_j2/_value (not _value_gi/_value_sp as the map expected), so the
solar ratio was always null->0 and the whole cost line was priced at full
grid price (EUR 4.67K instead of ~3.4K). Streams now pre-rename columns
to h/d/gi/ge/sp so the join is unambiguous.
- Blended price now uses counter deltas since heat-pump install (Oct 2024)
instead of device-lifetime counters that include 1.4 years of PV-only
history from May 2023 (ratio 0.369 vs 0.322 -> EUR 0.236/kWh vs 0.248).
Applied consistently to all cost/savings/ROI stats and the cost chart
(chart subtracts install-date counter baselines).
- Annual savings / payback: replace the calendar-YTD extrapolation
(12 / (month + day/30) - which overcounts elapsed time by a month and is
seasonally biased) with the actual trailing-365-day consumption spread.
- Fix stale legend 'Last' values (10734/1230/EUR 9.47K): the counters'
friendly_name tag changed 2026-03-04, splitting each series into two
tables; group()+sort() merges them before aggregateWindow.
- New oil_baseline dashboard variable (6650) replaces six hardcoded copies.
- Running Cost: neutral color (was green/yellow/red on a forever-growing
cumulative), retitled '(Since Install)'; added descriptions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- App Servers Up: job="authentik" matches nothing (real job label is
authentik-server-metrics) -> panel showed 'No data'. Also 'or vector(0)'
+ red threshold so a dead scrape shows a red 0 instead of No data.
- Task/Admin Workers, Flows/Policies Cached: raw gauges return 4 series
(2 pods x 2 pids) -> stats rendered 8 repeated numbers. Wrapped in max().
- Outposts Connected: raw per-outpost/per-pid series rendered an unreadable
grid of ~30 0/1 values (0s were per-process views, not real disconnects).
Now counts outposts with >=1 active WebSocket connection (10/10).
- Outpost Connections: legendFormat used non-existent labels {{name}}/{{uid}}
(real: outpost_name/uuid) -> every legend row was '-'. Replaced with
server-side authentik_outposts_connected as a state-timeline, one row per
outpost, red mapping for 'getrennt'; covers all 10 outposts instead of the
4 with scraped sidecar metrics.
- All stat queries switched to instant so dead sensors render honestly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Nodes Online/Down used job="node" but the scrape job is "node-exporter":
both stats showed No data. Nodes Down additionally gets OR on() vector(0)
so it renders 0 instead of No data when all nodes are up, and green-at-0/
red-at-1 thresholds (was permanently red).
- Avg Disk Usage + Disk Usage by Node filtered mountpoint="/" which does not
exist on Talos (only /var is a real xfs filesystem): both panels were empty.
- Disk I/O summed all block devices including the ~12 Rook-Ceph rbd* mappings
per node (~438 kB/s vs 8-129 kB/s physical): filter to nvme.*|vd.*.
- Network I/O filter was written for a Docker host (veth/docker/br-) and let
~40 Cilium lxc* per-pod interfaces plus kernel pseudo-devices through,
double-counting pod traffic: filter to eth.* (bond0 is down, 0 slaves).
- Legends showed instance IP:port; join node_uname_info via
on(instance) group_left(nodename) so panels label k8s-1/2/3. Both join
sides are wrapped in max by (...) so historical duplicate series from
pod restarts / the pre-2026-07-16 duplicate-scrape residue cannot break
the join or rainbow-duplicate long-window views.
- Stat panels reduce lastNotNull instead of mean: show current value, not
a time-average over the selected window.
All queries verified against live Prometheus before applying.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The dashboard was 100% dead: every panel still referenced the
prometheus-legacy datasource retired on 2026-05-09, so all panels
rendered 'No data' with datasource errors.
- Repoint all panels prometheus-legacy -> prometheus
- Remove TCP Endpoint Status panel: the blackbox-tcp Probe was
deliberately deleted 2026-05-20 (see probes-blackbox.yaml), the job
no longer exists
- Remove DNS Status panel: no blackbox-dns job has ever existed in
this Prometheus (the dns-servers job is Alloy on the DNS VMs, not a
probe)
- Replace the freed row with HTTP Status Codes (probe_http_status_code,
instant, 2xx green / 3xx orange / 4xx+ and probe-fail red)
- Convert SSL Certificate Expiry from a seconds-countdown timeseries to
a days-remaining bar gauge (instant, red <7d, orange <30d, green)
- Make status stat queries instant so dead probes render No data
instead of a stale last-known value
- Pin Uptime y-axis to 0-100% and drop the fill (10 overlapping series
at 100% rendered as a muddy block)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The pve-exporter is scraped via all 3 PVE nodes and each scrape returns the
full cluster view, so every series exists in triplicate. All stat counts were
3x too high (Online Nodes showed 39 instead of 3) and every bargauge/table
row appeared three times. Wrap all queries in max by (id) / count by (id).
- Online Nodes: count unique node/* ids with pve_up==1 (was 39, now 3);
threshold green at 3 so a lost node turns red
- Total VMs: count unique qemu guests excluding templates (was 12, now 3)
- Total LXC: count unique ids (was 9, now 3)
- Running Guests: pve_guest_info has no status label in this exporter
version (panel showed No data); derive from pve_up==1 per guest instead
- Node CPU/Memory bargauges + timeseries: filter to id=~"node/.*" — they
showed every guest (mostly lxc rows), not nodes
- Guest Status table: dedupe rows, add Status column from pve_up (color
mapped 1=Running/0=Stopped), drop useless Info column, migrate
custom.displayMode -> custom.cellOptions (schemaVersion 38 skips the
Grafana 10 migration, so gradient gauges/color background were silently
ignored)
- Node Disk I/O -> Guest Disk I/O: pve_disk_read/write metrics only exist
for qemu/lxc guests, never node/* (panel was permanently No data); use
the *_total counters with rate()
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The dashboard was entirely dead: every panel queried postgres_exporter
metrics (pg_up, pg_stat_activity_count, ...) that have never existed in
this Prometheus. Postgres runs as four CloudNativePG clusters (ak-pg,
im-pg, gf-pg, pprl-pg) whose built-in exporter on :9187 was not scraped
at all (enablePodMonitor is false and no PodMonitor existed).
- add PodMonitors for the four CNPG clusters in monitoring-rules,
labelled release=kube-prometheus-stack so the Prometheus
podMonitorSelector actually picks them up (CNPG's own
enablePodMonitor would generate unlabelled PodMonitors that this
Prometheus ignores)
- rewrite all dashboard queries to the cnpg_* metrics the exporter
actually exposes (verified against the pods' /metrics endpoints)
- dedupe multi-instance clusters (max by namespace/datname for sizes,
primary-only filter via cnpg_pg_replication_in_recovery == 0 for
per-database counters)
- replace the Active Locks stat (no lock metric in the CNPG default
queries) with Max Replication Lag
- instant queries on all stat panels so dead instances show No data
instead of a stale last value
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Healthy Hosts: job 'integrations/unix' does not exist here -> node-exporter;
thresholds now red/yellow/green(3) so a single down node shows yellow
- Unhealthy Services: 'count(up == 0)' returns empty vector when healthy ->
'or vector(0)' renders an honest 0; both stats now instant queries
- CPU: 'avg by(host)' collapsed all 3 nodes into one series (no host label) ->
avg by(instance) joined with node_uname_info for k8s-1/2/3 legend names
- Memory: legend showed raw label dumps ({{host}} does not exist) -> nodename join
- Disk: Talos has no '/' mountpoint in node_filesystem -> use /var (the
persistent data disk); retitle panel accordingly
- Cumulative Value: Self-Use Savings and Total Value series rendered nothing.
The union'd solar/export/price tables had mismatched group keys (pivot ran
per-table) AND the sparse strompreis series almost never landed in the same
aggregation window. Fix: group() before pivot + fill(usePrevious) per column.
Verified via /api/ds/query: 471 points, final value EUR 4363 == stat panel.
- Solar Power stat: was aggregateWindow(mean) over the dashboard range (~17 h
windows at 3 y), showing 3.04 kW at midnight. Now last() -> honest 0 W at night.
- Solar Installed: HA input_number stores 8140 (Wp, mislabeled kWp); with unit
kwatt the panel would read 8.14 MW. Divide by 1000 -> 8.14 kW, documented.
- Timeseries gap honesty: e3_* series have a real data hole (Nov 2025 - Feb 2026,
verified 0 points/month). createEmpty:false let Grafana draw fake decay ramps
across it. Now createEmpty:true so gaps render as gaps (panels 20/21/22).
- Units: 'kwh' is not a valid Grafana unit id (rendered lowercase 'kwh');
replaced with 'kwatth' everywhere -> proper kWh/MWh scaling.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Current by Phase: regex /smart_meter.*current/ matched nothing (smart
meter exposes no current entities) -> use verified
e380_energy_meter_e380_current_l[123]; spanNulls for sparse updates
- Energy Consumption: measurement "kWh" does not exist in the bucket
-> plot verified Wh daily counters (grid import/export, PV production,
battery charge/discharge) with readable display names
- Active Lights: light-domain data is never recorded under measurement
"%" -> replace dead table with Lights/Standby/Switches power
timeseries from verified powercalc aggregates (all_*_power)
- Top 20 Devices: add group() so sort/limit work globally (was
alphabetical, unsorted, per-table); exclude grid meters, PV/battery,
forecasts, powercalc room/floor aggregates and _device_power twins;
fresh 30-min window instead of stale last() over the full range;
rowsToFields for clean labels (was "_value xyz_power")
- Grid semantics: retitle Total Power -> Grid Power (+Import/-Export),
blue threshold for export; unstack mixed-sign phase graph; short
L1/L2/L3 legend names; taller voltage/current panels so the legend
table is not cut off
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Alarm stats: '== 1' dropped the series when false, so Armed Home/Away and
Triggered could only ever show 'No data' and never their 0-state; Disarmed
rendered 5 overlapping series. Use min()/max() aggregation over the one-hot
alarm_control_panel_state series so every tile always shows a real 0/1.
- Windows/Doors tables: replace friendly-name regexes (matched battery-low
sensors, HmIP 'Window/door is CLOSED' entities with inverted semantics,
intrusion/tamper diagnostics, _2 twins, pond-pump/fridge sensors) with a
curated entity list: one canonical is-open entity per physical sensor.
Collapse scrape labels via max by so the tables show Sensor/Name/Status.
- Motion panels: old regex required 'Bewegungs.*Occupancy' and missed the
Eingang sensor (typo 'Bewegunsmelder'), all motion-class sensors and the
mmWave Praesenzmelder. Curated list of physical detectors.
- Covers: name-based exclusion (.*Nord.*|.*Ost.* etc.) dropped 8 physical
covers incl. the garage door while letting the eg_schlafzimmer/eg_kuche
group entities through. Exclude the 5 group covers by entity instead;
label bars by entity because friendly names repeat ('Rollo' x3).
- Batteries: filter battery_notes '+' twin entities, drop PV storage SoC,
remove '> 0' (which would hide a 0% dead battery), sort ascending so the
most critical batteries are on top, instant query so stale values drop.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The template variable queried label_values(node_uname_info, host), but no
node-exporter metric in this Prometheus carries a 'host' label (that label
only exists on blackbox-probe metrics). The variable resolved to empty and
all five panels rendered blank.
- Replace $host with a visible $node variable (nodename: k8s-1/2/3) plus a
hidden chained $instance variable (nodename -> instance via node_uname_info)
- Filter Cilium pod interfaces (lxc*, cilium_*) and always-zero kernel
pseudo-devices (ip6tnl0, sit0, teql0, tunl0) out of Network I/O
- Filter Ceph RBD-mapped volumes (rbd0..11) out of Disk I/O so the panel
shows physical disks (nvme0n1, vda) only
- Exclude the rootfs bind mount (/etc/extensions.yaml) from Filesystem
Usage, leaving the meaningful /var xfs mount
All queries verified against live Prometheus before commit.
- All temperature/humidity panels: replace vendor regexes (shelly.*temp etc.)
that dragged in device-internal temperatures (pond pump 34C, power meter 60C,
relay 76C) with explicit allow-lists of verified room sensors
- Add missing sensors: shelly_blu_h_t_7203 (KG Schlafzimmer),
treppenhaus_eg_bewegungsmelder (EG Treppenhaus), vento humidity sensors,
Shelly Wall Display (DG Buero)
- OG & DG panel: regex had no og_ alternative and showed only device internals;
now honest DG-only series + description (OG deliberately has no sensor)
- Heating System / Hot Water panels were empty (domain climate/water_heater
does not exist in the C measurement) -> use vitocal_250_a_* sensors
- Current-value panels: fixed -6h window so dead sensors (EG Shelly H&T trio,
DG Meter Pro) disappear instead of showing stale values as live
- Legends: map entity_ids to room names in Flux + displayName, removes the
'_value xyz' prefix
- Align CO2 thresholds between gauge and history (800/1000/1500)
- Server node: sensor.keller_server_power is dead (UPS retired) -> Wave PM
Mini server outlet (kg_vorratskeller_wave_pm_mini_server_steckdose)
- Garden node: sensor.eg_garten_teichpumpe_power_2 is dead ->
sensor.shelly_teichpumpe_power_0 (live, peaks ~77 W)
- Home node/series: floor sums only cover submetered circuits (702 W shown
vs ~1.1 kW real); Home is now the meter balance PV + Grid + Battery, with
a new grey 'Rest' node for the non-submetered remainder
- Flow lines: Heat Pump is metered inside Keller (was drawn from EG);
Garden is on floor Aussenbereich, in no floor sum (was drawn from OG)
- Drop redundant batteryIn target (derived from the signed battery sensor);
update stale comments from the retired *_absolute sensors
- History: max by (entity) guards against duplicate-scrape residue that
doubled the summed Home series; sign hints in Battery/Grid legends;
grid query pinned to entity id instead of friendly_name
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Status panel: job label is home-assistant (hyphen), was homeassistant -> No data
- Doors & Windows / Motion: exporter has no device_class label; select curated
entity lists instead (one canonical is-open entity per physical contact,
physical motion/presence sensors without zone/magic_areas duplicates)
- Temperature: was all 61 temp series incl. device-internal electronics temps
(fridge 5C, PoE switch 75C, Shelly 58-75C); curated to room/ambient sensors
+ outdoor temp, incl. currently-dead EG H&T sensors so they reappear
- Power Consumption: exclude non-consumption series (Solcast forecasts,
battery max charge/discharge capability, derived night-load stats)
- Energy: cumulative lifetime counters (up to 80 MWh) told nothing; now
bargauge of top 12 increase() over 24h, forecast counters excluded,
max by () dedup against scrape residue
- Switches: 248 unreadable micro-tiles -> single 'Switches On' count stat
- Battery: bottomk(12) instant, battery_notes *_battery_plus twins excluded
- Entity counts: max by (entity) dedup, instant queries on all stats
- Unavailable Entities: neutral blue; ~1139 is the structural baseline
(verified against HA API), old yellow>=1/red>=5 thresholds were meaningless
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
All queries selected raw series by entity label only; over a 90d window HA pod
restarts plus the (since fixed) duplicate proxy scrape produce 30 distinct
series per entity, rendering the Außentemperatur & Puffer panel as an
unreadable 30-entry legend jumble. Wrap every query in max() so exactly one
series per entity survives — this also keeps the xychart joinByField transform
correct once heating data arrives, since the join needs unique field names.
The stat panel now says 'keine Heizdaten im Zeitraum' instead of a giant
'No data': the logging sensor is intentionally unavailable outside active
heating loading, so an empty summer dashboard is by design.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
All four panels queried raw HA metrics that still carry ~12 duplicate
series per entity in the 90d window (two scrape jobs x six pod
lifetimes; the duplicate-scrape bug was fixed 2026-07-16 but the
residue stays in range queries until it ages out). This broke the
scatter join (no points rendered), made the stat panel show 8
conflicting averages, and drew rainbow duplicates in both time series.
Wrapping every target in max by (entity) (...) collapses the residue
to one series per entity; values between the duplicate jobs were
near-identical, so max is lossless.
- Windows/Doors/Batteries tiles: literal ${__value.raw} never interpolates in
value mappings; replaced range mappings with suffix units. Counts now use
curated entity lists (one canonical 'is open' entity per physical sensor)
instead of friendly-name regexes that matched inverted *_is_closed sensors,
'Battery low' sensors and duplicate _2 entities.
- Alarm tile: old min(...==1) query could never show TRIGGERED; now
max(...{state="triggered"}) with 0=OK/1=TRIGGERED mapping.
- Power Usage: old regex summed phases A+B+C AND total_active_power AND
device_power -> ~2x real value; now reads total_active_power only.
- Floor temperature panels: select verified room sensors by entity id instead
of vendor regexes (which pulled 65-75C device-internal temps and missed the
I-9PSL sensors that live in the EG). Instant queries so dead sensors do not
render stale values. OG: no room sensor exists (owner-confirmed) -> honest
noValue; DG SwitchBot offline -> noValue 'Sensor offline'.
- Tables: dedup via max by (entity, friendly_name), drop k8s scrape label
columns, colour only the value column; battery list excludes battery_notes
*_battery_plus twin entities and no longer hides 0% batteries.
- Power Consumption: L1/L2/L3 legends, Total from total_active_power.
- Indoor Temperatures: room sensors only (y-axis no longer squashed to 80C).
- Heating row: Hot Water/Heating pointed at post-open3e-migration vitocal
sensors (old queries returned nothing resp. only the fridge); humidity/CO2
deduplicated; row renamed to Heating & Air Quality.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The union-based timeseries series were named via set(_field, ...), which Grafana's InfluxDB frontend rendered as "_value {_start=...}" instead of the intended labels (and broke the byName color overrides). Data was correct; only the legend/colors were affected.
Switch to the keep(columns:[_time,_value]) |> rename(_value -> "Name") pattern (as used in heatpump-roi), which names each series cleanly. Verified in Grafana Explore. No data/logic change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QiLCinB7NLp7YdCjGYzvfh
The 2026-07 ViCare->open3e/MQTT migration renamed the Vitocharge PV/battery entities (e3_vitocharge_03_* -> vitocharge_vx3_vx3_*) and changed some units. Both dashboards still queried the retired entities and showed No data.
power-flow (Prometheus, real-time): solar_power->pv_power, battery SoC->battery_soc; the two absolute battery-power sensors are replaced by a single signed battery_power (open3e: positive=discharge, negative=charge), so charge/discharge derive from its sign.
pv-system-roi (InfluxDB): repoint every query; fix units (PV/battery energy now Wh not kWh -> /1000; grid energy now kWh not Wh); grid power sign unchanged (negative=export). Timeseries panels UNION pre-migration e3_* with post-migration series (unit-normalized) so history back to 2023 stays unbroken. Reserved Energy recomputed as capacity x backup_discharge_limit. strompreis + the two input_number config entities survived and are kept.
Validated live via Grafana Explore: signed-battery charge/discharge union, unit normalization, pivot/value patterns (reused from merged heatpump-roi fix#271).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QiLCinB7NLp7YdCjGYzvfh
The ViCare cloud integration was replaced by open3e/MQTT (2026-07), renaming all heat-pump/PV entities (e3_vitocal_*/e3_vitocharge_03_* -> vitocal_250_a_vitocal_*/vitocharge_vx3_vx3_*). The dashboard still queried the retired IDs, and a duplicate Prometheus scrape broke the surviving-entity panels with many-to-many errors.
Rebuild every panel on InfluxDB (bucket suedstr28, authoritative + live) using current entity IDs. Where an entity was renamed, UNION the pre-migration e3_* series with the post-migration series (unit-normalized: PV Wh->kWh) so history stays unbroken. Drop all Prometheus targets. Add a grid_price variable (0.3271) replacing the retired live strompreis read.
Also rename the home-assistant-bgp Service port http->ha so the home-assistant ServiceMonitor no longer scrapes HA a second time via the LB (:8123), which duplicated homeassistant_sensor_* series cluster-wide.
Queries validated live via Grafana Explore: blended price 0.248 EUR/kWh, total consumption 14284 kWh, projected annual savings 4698 EUR, oil proration 11912 EUR, old-union-new grid-import continuous.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QiLCinB7NLp7YdCjGYzvfh
Owner directive: PVE runs ONLY the three k8s VMs. The proxmox-dns trio
(lxc 118/218/318), rustfs trio (lxc 800/801/802), fortress (vm 102)
and paperless (vm 111) are decommissioned; guest definitions are being
destroyed on PVE out-of-band.
Removed:
- opentofu: lxc_118_proxmox_dns.tf, vm_102_fortress.tf,
vm_111_paperless.tf, backup_jobs.tf (the vzdump job covered ONLY
destroyed guests — next tofu apply removes the job from PVE; k8s
data protection is Velero/CNPG)
- ansible: roles/proxmox-dns, install-proxmox-dns{,-ha}.yml,
check-chaos-testing.yml (tested failover of decommissioned DNS-HA
and postgres LXCs), proxmox_dns_vault.yml, rustfs_vault.yml,
[proxmox_dns_ha] inventory group
- monitoring: dns-servers scrape job in both the k8s HelmRelease and
the legacy prometheus-server template — resolves TargetDown
(dns-servers)
Closes#259
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
Found on the first real use of the restricted identity: assessing the
VeleroBackupFailed issue (#251) needs backups.velero.io (latest
apps-daily run is PartiallyFailed — data-mover pod unschedulable on an
unbound PVC), and the ClusterRole had neither velero.io nor CNPG.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
Both wrong assessments (#249 openbao 'sealed', #250 three untested
hypotheses while one 'kubectl logs' showed the 401) share one root
cause: the agents diagnose blind, so they narrate plausible fiction.
Labeling unverified claims (previous commit) treats the symptom; this
gives them the means to verify:
- kubernetes/apps/monitoring/ci-diagnostics-rbac.yaml: ServiceAccount
ci-diagnostics with a read-only ClusterRole (workloads, events,
pods/log, Flux/ESO/cert-manager/monitoring/ceph CRDs — NO secrets)
plus a namespaced services/proxy Role for the Prometheus and
Alertmanager HTTP APIs.
- scripts/ci/make-diag-kubeconfig.sh: mints a 1h token per run from
the admin kubeconfig (no new long-lived secret), builds
.ci-diag/kubeconfig, copies the static kubectl binary for the
jq-only ci-claude image, smoke-tests with the restricted identity,
and degrades to snapshot-only mode when the SA isn't reconciled.
- alert-monitor.yml / issue-fix.yml: agents get exactly one extra tool,
Bash(./.ci-diag/kubectl:*) — prefix match forbids pipes and any other
binary; RBAC forbids mutation and secret reads.
- prompts: verification is now mandatory, not aspirational — 'a
hypothesis you could have settled with one read-only command but
didn't is a defect in your output'; fix agent must test the issue's
claims before building on them; minimal-quote rule against dumping
logs into public issues.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV7bZzZLeoFUEdAeCSLEei
evcc, otbr, libation, external-dns (apps), monitoring (4), woodpecker:
values in OpenBao KV (homelab/apps/*, loaded 2026-07-13), manifests are
now ExternalSecrets keeping the old Secret names. libation's payload is
a binary SQLite db — stored base64 in KV with decodingStrategy: Base64
on the ES. Decryption blocks dropped from apps/evcc/otbr/libation/
monitoring KS (paths sops-free) plus the dormant one on grafana-db.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CRgBFn9m8a2ncFMLusikH6
Gitea moved in-cluster (apps ns StatefulSet); LXC 114 is stopped, so the static
`gitea` scrape target gitea.srv.renner.uno:3000 is down -> firing TargetDown.
Point it at the ClusterIP Service (gitea.apps.svc.cluster.local:3000), which
serves /metrics anonymously (verified HTTP 200 with gitea_* metrics).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HQhvu1mrQ7EgwLhrzdNyY7
Prometheus (prometheus.renner.uno, monitoring ns, :9090) had no auth of its
own. Its admin API is disabled (read-only), but the full 90d metrics DB is
an infra-recon oracle — every hostname, IP and service in the cluster
(2026-07-10 audit). Gate it the same way as alertmanager-gate (#146).
- Blueprint: prometheus-gate proxy provider (internal_host
http://kube-prometheus-stack-prometheus.monitoring.svc:9090, external_host
https://prometheus.renner.uno), application, outpost. REUSES the existing
git-owned "Monitoring Access" group — created generically in #146 for
exactly this, so no new group.
- implicit-consent flow (same rationale as alertmanager/paperless: a
frequently-checked monitoring UI; an authenticated session already closes
the exposure).
- No skip_path_regex: the Prometheus UI is a plain React SPA — verified its
index.html has no serviceWorker/manifest/webmanifest.
- Helm-generated Prometheus Ingress disabled (prometheus.ingress.enabled=
false); standalone prometheus-gate-ingress.yaml backends the outpost
ExternalName. Mirrors the alertmanager gate.
- Scoped CNPs: monitoring ingress rule allowing only the outpost -> :9090;
matching egress in the authentik namespace.
Grafana is UNAFFECTED: its Prometheus datasource targets the ClusterIP
Service (kube-prometheus-stack-prometheus.monitoring:9090), verified in the
live datasource provisioning, not the ingress hostname.
Known side effect, documented not silently shipped: three blackbox
`http_2xx` uptime probes target https://prometheus.renner.uno
(kubernetes/apps/monitoring-rules/probes-blackbox.yaml, the srv-host
ansible prometheus.yml.j2, and the ansible blackbox-exporter role). After
gating they follow the 302 to the Authentik login (200), so they stay GREEN
— but now measure Authentik reachability, not Prometheus. This is already
true for alerts.renner.uno since #146 (verified: probe_success=1,
final_status=200 via redirect-follow). No probe goes red; the monitoring
just becomes a gate-liveness check. Repointing those probes to an internal
endpoint is a separate follow-up, not done here.
Verified: kubectl kustomize builds for authentik, monitoring,
network-policies; embedded outpost survived the blueprint insert; rendered
ingress backends ak-outpost-prometheus-gate:9000; both prometheus and
alertmanager helm ingresses now disabled; egress + CNP scoped to
prometheus:9090.
Post-merge checks: only the gate Ingress serves prometheus.renner.uno (helm
one gone); unauthenticated GET -> 302 to auth.renner.uno; unauthenticated
GET /api/v1/query -> 302; outpost forwards to :9090 after login; Grafana
dashboards still render (ClusterIP datasource intact).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
claude-botandClaude Opus 4.8
k.renner