Compare commits

...
Author SHA1 Message Date
claude-botandClaude Fable 5.1 4e2303f455 feat(monitoring): Intel iGPU dashboard
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Grafana dashboard for the gpumon_* series from intel-gpu-exporter:
current Render/3D busy, GPU power and GT frequency per node as stat
tiles, then one time series per engine (Render/3D, Video, VideoEnhance,
Blitter) with node as the series, GT frequency, GPU and package power,
and memory-controller read/write bandwidth.

Node colours are fixed per node across every panel — blue, orange, aqua
for k8s-1/2/3 — rather than assigned by series order, so a node keeps
its colour whatever the $node filter shows. The trio validates for
colour-vision deficiency and normal vision on both Grafana themes
(worst all-pairs ΔE 9.4 / 20.9). Reads and writes share one axis and
differ by dash, not hue. One axis per panel throughout.

gpumon_clients_count is deliberately absent: per-client attribution
needs hostPID, which the exporter does not have, so it would read 0
forever and imply an idle GPU.

Refs #752

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 07:22:46 +02:00
k.renner 483093a104 Merge pull request 'feat(monitoring): export Intel iGPU utilisation, frequency and power' (#822) from feat/intel-gpu-exporter into master
Reviewed-on: k.renner/homelab#822
2026-09-08 05:18:43 +00:00
claude-botandClaude Fable 5.1 8d348e7f88 fix(monitoring): let intel_gpu_top pick the GPU instead of hard-coding card0
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The -device flag named /dev/dri/card0. That is the right node today on
every host, but nothing guarantees the number: a second GPU or a
different probe order and the exporter would silently watch the wrong
device or none. The plugin's monitoring resource injects every GPU on
the node and intel_gpu_top selects it on its own — verified with a probe
pod on k8s-2, which produced the same gpumon_* series without the flag.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 07:15:04 +02:00
claude-botandClaude Fable 5.1 6f9ff9d624 feat(monitoring): export Intel iGPU utilisation, frequency and power
A DaemonSet running clambin/intel-gpu-exporter (intel_gpu_top behind a
Prometheus endpoint) with a ServiceMonitor, giving gpumon_engine_usage,
gpumon_frequency, gpumon_power, gpumon_imc_* and gpumon_clients_count
per node.

Privilege model, validated with a one-shot pod before this commit:
/dev/dri comes from the device plugin's gpu.intel.com/monitoring
resource (every GPU on the node, no hostPath); the i915 PMU that
intel_gpu_top reads is a system-wide event source and the nodes run
perf_event_paranoid=3, so CAP_PERFMON is the one capability added.
No privileged, no hostPID, no hostNetwork, read-only root filesystem.
Root rather than a fixed uid because the capability has to be effective
without file capabilities on the binary.

End-to-end check on k8s-2 during a 40 s chat completion on llama-server:
Render/3D busy 98 %, actual frequency 1449 MHz, GPU power 18.8 W, while
the pod's CPU stayed near 100m — the "is it really on the GPU" question
this exists to answer.

XPU Manager was ruled out (Arc Pro / data-centre parts only); the
archived onedr0p exporter needs privileged + hostPID.

Fixes #752

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 07:14:24 +02:00
k.renner 7bf65783be Merge pull request 'feat(intel-gpu-plugin): advertise the monitoring resource' (#820) from feat/intel-gpu-monitoring-resource into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#820
2026-09-08 04:54:15 +00:00
k.renner dd9fb01359 Merge pull request 'fix(reloader): restart audiobookshelf and immich-server on ConfigMap change' (#819) from fix/reloader-configmap-annotations into master
Reviewed-on: k.renner/homelab#819
2026-09-08 04:52:32 +00:00
claude-botandClaude Fable 5.1 da8a725b32 feat(intel-gpu-plugin): advertise the monitoring resource
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Enables -enable-monitoring on the device plugin. With 0.36.0 that
advertises gpu.intel.com/monitoring, a single resource per node that
hands a pod every GPU rather than one of the shared slots; it is what a
metrics exporter requests to see the whole iGPU, and it is the only way
to get /dev/dri into a pod without a hostPath.

Nothing consumes it yet. The nodes currently advertise gpu.intel.com/i915
only — the issue assumed the chart default had this on, but the
DaemonSet here is hand-written and did not.

Refs #752

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 06:51:43 +02:00
claude-botandClaude Fable 5.1 6008d59c9f fix(reloader): restart audiobookshelf and immich-server on ConfigMap change
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Both consume their ConfigMap with envFrom, which Kubernetes does not
watch: a changed value applies in git and in the cluster while the
running container keeps the environment it started with. Paperless hit
exactly this in #739 and got the annotation in #741; these two were the
remaining envFrom consumers without one.

ftp-scanner, the third Deployment named in the issue, needs nothing: its
ConfigMap comes from a configMapGenerator that deliberately keeps the
content-hash suffix, so any change renames the ConfigMap, mutates the
pod template and rolls the pod on its own.

Reloader already watches all namespaces (kubernetes/infrastructure/reloader).

Fixes #742

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 06:46:12 +02:00
k.renner 5aa6c0609d Merge pull request 'fix(netbird-proxy-services): pin the digest the master build produced' (#818) from netbird-proxy-protocol-digest into master
ci/woodpecker/cron/e2e Pipeline failed
ci/woodpecker/cron/security-scan Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#818
2026-09-07 11:52:17 +00:00
claude-botandClaude Fable 5.1 0ebe113abb fix(netbird-proxy-services): pin the digest the master build produced
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
#817 pinned the digest from the branch build (pipeline 8452). Merging
#816 rebuilt the same date tag on master (pipeline 8457), Gitea kept
only the new manifest, and the pod has been in ImagePullBackOff since:
"failed to resolve reference ...@sha256:0d266b02...". With the Recreate
strategy no controller is running, so the services are unchanged.

The kaniko step of the master build pushed sha256:caffa308...; that is
the digest to pin. Comment states the rule.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-07 13:47:22 +02:00
k.renner 1723f73b5d Merge pull request 'feat(netbird-proxy-services): roll the PROXY-protocol build (2/2)' (#817) from netbird-proxy-protocol-deploy into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#817
2026-09-07 11:34:52 +00:00
k.renner 8d8df3b635 Merge pull request 'feat(netbird): carry the WAN client address through the passthrough edge (1/2)' (#816) from netbird-proxy-protocol into master
ci/woodpecker/push/build-ci-images Pipeline was successful
Reviewed-on: k.renner/homelab#816
2026-09-07 11:34:43 +00:00
claude-botandClaude Fable 5.1 29c91769a7 feat(netbird-proxy-services): roll the PROXY-protocol build (2/2)
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Pins the controller image by digest. The registry tag is the build
date and pipeline 8452 rebuilt today's tag in place, so the tag alone
cannot carry a new build to the cluster; the digest can.

Merge only after the Traefik change from 1/2 has reconciled: from this
build on every service target sends a PROXY protocol v2 header, and
Traefik drops such connections unless websecure trusts the pod CIDRs.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-07 13:32:06 +02:00
10 changed files with 488 additions and 1 deletions
@@ -21,6 +21,9 @@ metadata:
namespace: audiobookshelf
labels:
app.kubernetes.io/name: audiobookshelf
annotations:
# envFrom does not restart the pod when the ConfigMap changes; Reloader does.
configmap.reloader.stakater.com/reload: audiobookshelf-config
spec:
replicas: 1
strategy:
+3
View File
@@ -35,6 +35,9 @@ metadata:
namespace: immich
labels:
app.kubernetes.io/name: immich-server
annotations:
# envFrom does not restart the pod when the ConfigMap changes; Reloader does.
configmap.reloader.stakater.com/reload: immich-config
spec:
# Scaled to 0 alongside immich-machine-learning during single-node
# maintenance; see the note there. The CNPG Postgres cluster stays running,
@@ -0,0 +1,20 @@
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: intel-gpu-exporter
namespace: monitoring
labels:
app.kubernetes.io/name: intel-gpu-exporter
release: kube-prometheus-stack
spec:
selector:
matchLabels:
app.kubernetes.io/name: intel-gpu-exporter
endpoints:
- port: http-metrics
path: /metrics
interval: 30s
relabelings:
# The series would otherwise carry only the exporter pod's IP.
- sourceLabels: [__meta_kubernetes_pod_node_name]
targetLabel: node
@@ -39,4 +39,5 @@ resources:
- immich-servicemonitor.yaml
- loki-servicemonitor.yaml
- smartctl-exporter-servicemonitor.yaml
- intel-gpu-exporter-servicemonitor.yaml
- probes-blackbox.yaml
@@ -0,0 +1,368 @@
{
"title": "Intel iGPU",
"uid": "intel-gpu",
"tags": ["gpu", "intel", "i915"],
"schemaVersion": 39,
"editable": true,
"timezone": "",
"refresh": "1m",
"time": { "from": "now-6h", "to": "now" },
"templating": {
"list": [
{
"name": "node",
"label": "Node",
"type": "query",
"datasource": { "type": "prometheus", "uid": "prometheus" },
"query": {
"query": "label_values(gpumon_power, node)",
"refId": "PrometheusVariableQueryEditor-VariableQuery"
},
"definition": "label_values(gpumon_power, node)",
"includeAll": true,
"multi": true,
"refresh": 2,
"sort": 1,
"current": { "selected": true, "text": ["All"], "value": ["$__all"] }
}
]
},
"panels": [
{
"type": "stat",
"title": "Render/3D busy now",
"gridPos": { "h": 4, "w": 8, "x": 0, "y": 0 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "percent",
"decimals": 0,
"min": 0,
"max": 100,
"color": { "mode": "fixed", "fixedColor": "text" }
},
"overrides": []
},
"options": {
"colorMode": "none",
"graphMode": "none",
"justifyMode": "center",
"orientation": "horizontal",
"textMode": "value_and_name",
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }
},
"targets": [
{
"expr": "gpumon_engine_usage{attrib=\"busy\", engine=\"Render/3D\", node=~\"$node\"}",
"legendFormat": "{{node}}",
"refId": "A"
}
]
},
{
"type": "stat",
"title": "GPU power now",
"gridPos": { "h": 4, "w": 8, "x": 8, "y": 0 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "watt",
"decimals": 1,
"color": { "mode": "fixed", "fixedColor": "text" }
},
"overrides": []
},
"options": {
"colorMode": "none",
"graphMode": "none",
"justifyMode": "center",
"orientation": "horizontal",
"textMode": "value_and_name",
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }
},
"targets": [
{
"expr": "gpumon_power{type=\"gpu\", node=~\"$node\"}",
"legendFormat": "{{node}}",
"refId": "A"
}
]
},
{
"type": "stat",
"title": "GT frequency now",
"gridPos": { "h": 4, "w": 8, "x": 16, "y": 0 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "suffix: MHz",
"decimals": 0,
"color": { "mode": "fixed", "fixedColor": "text" }
},
"overrides": []
},
"options": {
"colorMode": "none",
"graphMode": "none",
"justifyMode": "center",
"orientation": "horizontal",
"textMode": "value_and_name",
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }
},
"targets": [
{
"expr": "gpumon_frequency{type=\"actual\", node=~\"$node\"}",
"legendFormat": "{{node}}",
"refId": "A"
}
]
},
{
"type": "timeseries",
"title": "Render/3D busy",
"description": "Compute and 3D. llama.cpp's Vulkan backend runs here; 0 % during an inference means the model is on the CPU.",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 4 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "percent",
"min": 0,
"max": 100,
"custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
},
"overrides": [
{ "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] }
]
},
"options": {
"legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"expr": "gpumon_engine_usage{attrib=\"busy\", engine=\"Render/3D\", node=~\"$node\"}",
"legendFormat": "{{node}}",
"refId": "A"
}
]
},
{
"type": "timeseries",
"title": "Video busy",
"description": "Fixed-function decode/encode (QuickSync).",
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 4 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "percent",
"min": 0,
"max": 100,
"custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
},
"overrides": [
{ "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] }
]
},
"options": {
"legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"expr": "gpumon_engine_usage{attrib=\"busy\", engine=\"Video\", node=~\"$node\"}",
"legendFormat": "{{node}}",
"refId": "A"
}
]
},
{
"type": "timeseries",
"title": "VideoEnhance busy",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 12 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "percent",
"min": 0,
"max": 100,
"custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
},
"overrides": [
{ "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] }
]
},
"options": {
"legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"expr": "gpumon_engine_usage{attrib=\"busy\", engine=\"VideoEnhance\", node=~\"$node\"}",
"legendFormat": "{{node}}",
"refId": "A"
}
]
},
{
"type": "timeseries",
"title": "Blitter busy",
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 12 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "percent",
"min": 0,
"max": 100,
"custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
},
"overrides": [
{ "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] }
]
},
"options": {
"legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"expr": "gpumon_engine_usage{attrib=\"busy\", engine=\"Blitter\", node=~\"$node\"}",
"legendFormat": "{{node}}",
"refId": "A"
}
]
},
{
"type": "timeseries",
"title": "GT frequency (actual)",
"description": "0 while the GT sits in RC6.",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 20 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "suffix: MHz",
"min": 0,
"custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
},
"overrides": [
{ "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] }
]
},
"options": {
"legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"expr": "gpumon_frequency{type=\"actual\", node=~\"$node\"}",
"legendFormat": "{{node}}",
"refId": "A"
}
]
},
{
"type": "timeseries",
"title": "GPU power",
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 20 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "watt",
"min": 0,
"custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
},
"overrides": [
{ "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] }
]
},
"options": {
"legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"expr": "gpumon_power{type=\"gpu\", node=~\"$node\"}",
"legendFormat": "{{node}}",
"refId": "A"
}
]
},
{
"type": "timeseries",
"title": "Package power",
"description": "Whole SoC (CPU + GPU + uncore), from the same PMU.",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 28 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "watt",
"min": 0,
"custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
},
"overrides": [
{ "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] }
]
},
"options": {
"legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"expr": "gpumon_power{type=\"pkg\", node=~\"$node\"}",
"legendFormat": "{{node}}",
"refId": "A"
}
]
},
{
"type": "timeseries",
"title": "Memory controller bandwidth",
"description": "Reads solid, writes dashed. The iGPU shares DDR5 with the CPU, so this is the whole package's traffic.",
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 28 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "MiBs",
"min": 0,
"custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
},
"overrides": [
{ "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] },
{ "matcher": { "id": "byRegexp", "options": "/write/" }, "properties": [{ "id": "custom.lineStyle", "value": { "fill": "dash", "dash": [10, 10] } }] }
]
},
"options": {
"legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"expr": "gpumon_imc_read{node=~\"$node\"}",
"legendFormat": "{{node}} read",
"refId": "A"
},
{
"expr": "gpumon_imc_write{node=~\"$node\"}",
"legendFormat": "{{node}} write",
"refId": "B"
}
]
}
]
}
@@ -77,6 +77,9 @@ configMapGenerator:
- name: dashboard-infrastructure-overview
files:
- infrastructure-overview.json
- name: dashboard-intel-gpu
files:
- intel-gpu.json
- name: dashboard-netbird-overview
files:
- netbird-overview.json
@@ -0,0 +1,81 @@
# Intel iGPU utilisation, frequency and power for every node (#752).
#
# Not privileged: /dev/dri comes from the device plugin's monitoring resource
# (every GPU on the node, no hostPath), and the i915 PMU that intel_gpu_top
# reads needs CAP_PERFMON — the one capability granted. No hostPID, no
# hostNetwork.
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: intel-gpu-exporter
namespace: monitoring
labels:
app.kubernetes.io/name: intel-gpu-exporter
spec:
selector:
matchLabels:
app.kubernetes.io/name: intel-gpu-exporter
template:
metadata:
labels:
app.kubernetes.io/name: intel-gpu-exporter
spec:
# Every node is a control plane; without this the exporter covers nothing.
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule
securityContext:
seccompProfile:
type: RuntimeDefault
containers:
- name: intel-gpu-exporter
image: ghcr.io/clambin/intel-gpu-exporter:0.7.4@sha256:d78761fef018834c759db154fe9dcf3dd9db5974703fa50ca21f6abc35221be1
args:
# No -device: intel_gpu_top picks the only GPU the plugin injects.
- -interval=5s
ports:
- name: http-metrics
containerPort: 9100
protocol: TCP
securityContext:
# Root so the PERFMON capability is effective without file
# capabilities on the binary.
runAsUser: 0
runAsNonRoot: false
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
add: ["PERFMON"]
resources:
requests:
cpu: 10m
memory: 32Mi
gpu.intel.com/monitoring: 1
limits:
memory: 128Mi
gpu.intel.com/monitoring: 1
livenessProbe:
httpGet:
path: /metrics
port: http-metrics
initialDelaySeconds: 15
periodSeconds: 30
---
apiVersion: v1
kind: Service
metadata:
name: intel-gpu-exporter
namespace: monitoring
labels:
app.kubernetes.io/name: intel-gpu-exporter
spec:
clusterIP: None
selector:
app.kubernetes.io/name: intel-gpu-exporter
ports:
- name: http-metrics
port: 9100
targetPort: http-metrics
protocol: TCP
@@ -16,6 +16,7 @@ resources:
- prometheus-gate-ingress.yaml
- ci-diagnostics-rbac.yaml
- smartctl-exporter.yaml
- intel-gpu-exporter.yaml
- dashboards/
# prometheusrule-garage.yaml lives in ./monitoring-rules/ and is applied by a
# separate Flux Kustomization that dependsOn `monitoring` — otherwise the
@@ -72,6 +72,9 @@ spec:
# they contend for it. Sized for an inference server plus room for
# a transcoder and a benchmark pod alongside it.
- -shared-dev-num=4
# Advertises gpu.intel.com/monitoring: one resource that grants a
# pod every GPU on the node instead of a slot. The exporter needs it.
- -enable-monitoring
securityContext:
seLinuxOptions:
type: "container_device_plugin_t"
@@ -76,7 +76,11 @@ spec:
type: RuntimeDefault
containers:
- name: controller
image: gitea.renner.uno/gitea_admin/netbird-proxy-services:2026-09-07
# Pinned by digest: the date tag is rebuilt on every push of the
# image directory (branch and master alike) and the registry keeps
# only the newest manifest per tag, so the digest must come from
# the MASTER build of the code change.
image: gitea.renner.uno/gitea_admin/netbird-proxy-services:2026-09-07@sha256:caffa308d802c6fcc6aea919ac94657c08b993651baa6b8d18de395bcc40523e
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true