Compare commits

...
Author SHA1 Message Date
claude-botandClaude Fable 5.1 6b0617b4a7 chore(ci): bump CI tool pins to match the deployed cluster
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/push/build-ci-images Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
From the update audit (#426): the tools CI drives the cluster with
trailed what the cluster runs.

ci/images/k8s/Containerfile
- talosctl v1.12.6 -> v1.13.9, the Talos version on the nodes. An older
  talosctl can misjudge or reject config generated for a newer machine.
- kubectl v1.36.2 -> v1.36.3, the cluster's Kubernetes version.
- flux CLI 2.9.2 -> 2.9.5, the current 2.9.x. Checksums verified against
  the upstream release assets (talos sha256sum.txt, flux checksums.txt)
  and against a local download.

.woodpecker/ci-manifests-test.yml
- woodpecker-cli v3.17.0 -> v3.18.0, the version the server and agent
  already run; the lint step was one minor behind its own server.

Merging triggers build-ci-images (path ci/images/**), which publishes
ci-k8s:<date>. The six pipelines pinned to ci-k8s:2026-08-10 move to
that tag in a follow-up once it exists.

Refs #426

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 08:58:35 +02:00
k.renner 6b7736ea58 Merge pull request 'fix(alerts): page the moment OpenBao is below quorum' (#824) from fix/openbao-alerts-immediate into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#824
2026-09-08 06:02:32 +00:00
k.renner 7ff20e5b68 Merge pull request 'feat(openbao): auto-unseal with a static seal' (#825) from feat/openbao-static-seal into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#825
2026-09-08 05:54:54 +00:00
claude-botandClaude Fable 5.1 296accd1cb feat(openbao): auto-unseal with a static seal
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Every restart sealed the replica until a human unsealed it. On
2026-09-07 a rolling talosctl upgrade rebooted all three nodes; the PDB
refused the second eviction while the first replica was sealed, Talos
waited out its 5-minute drain timeout and rebooted anyway, and raft
quorum was gone for 24 minutes — every ExternalSecret and the
Kustomizations behind them with it. A power loss would do the same
with no timeout to blame.

OpenBao 2.6.1 supports a static seal: a 32-byte key the server uses to
unseal itself. The key is generated into pass (homelab/openbao/seal-key)
and created as the openbao-seal-key Secret before this release reconciles;
the chart hands it to the server as BAO_SEAL_KEY from that Secret.

This overturns the principle in bootstrap.sh that the key protecting
OpenBao must not live in the cluster OpenBao secures. Decided by the
owner on 2026-09-08: the alternatives were a transit seal from a second
OpenBao on edge-1, which raft traffic would wear out, or a workstation
unseal script that may not be at hand when needed. The exposure is
bounded by what ESO already does — every synced secret is plaintext in
etcd (encrypted at rest) — and grows only by what lives solely in
OpenBao.

Migration is a live, per-replica procedure with the shamir key becoming
the recovery key; the StatefulSet is OnDelete, so merging this changes
nothing until each pod is restarted in turn. The bootstrap and rebuild
paths (bootstrap.sh, vmtest) follow in a separate change once the
migration is verified.

Refs #805

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 07:52:27 +02:00
claude-botandClaude Fable 5.1 2665e5ebf3 fix(alerts): page the moment OpenBao is below quorum
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
OpenBaoQuorumLost waited for: 5m. Below quorum the secret store is
down — the ClusterSecretStore goes InvalidProviderConfig and every
ExternalSecret and the Kustomizations behind them stall — and a sealed
replica never recovers on its own, so the window only delayed the page.
It fires on the first evaluation now.

OpenBaoUnavailable, the external probe, drops from 5m to 1m: two
consecutive failed probes, enough to skip a single lost probe through
Traefik, not enough to sit on a sealed store.

Both StatefulSet expressions are wrapped in max by (namespace,
statefulset). kube-state-metrics stamps instance/pod on the series, so
a rescheduled exporter starts a new series and Prometheus resets the
for: timer with it — on 2026-09-07 that made OpenBaoReplicaNotReady fire
14 minutes late. Alertmanager routes match alertname/source only, so the
dropped labels change nothing downstream.

Refs #805

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 07:40:12 +02:00
k.renner 0fe512746b Merge pull request 'feat(monitoring): Intel iGPU dashboard' (#823) from feat/intel-gpu-dashboard into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#823
2026-09-08 05:24:35 +00:00
claude-botandClaude Fable 5.1 4e2303f455 feat(monitoring): Intel iGPU dashboard
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Grafana dashboard for the gpumon_* series from intel-gpu-exporter:
current Render/3D busy, GPU power and GT frequency per node as stat
tiles, then one time series per engine (Render/3D, Video, VideoEnhance,
Blitter) with node as the series, GT frequency, GPU and package power,
and memory-controller read/write bandwidth.

Node colours are fixed per node across every panel — blue, orange, aqua
for k8s-1/2/3 — rather than assigned by series order, so a node keeps
its colour whatever the $node filter shows. The trio validates for
colour-vision deficiency and normal vision on both Grafana themes
(worst all-pairs ΔE 9.4 / 20.9). Reads and writes share one axis and
differ by dash, not hue. One axis per panel throughout.

gpumon_clients_count is deliberately absent: per-client attribution
needs hostPID, which the exporter does not have, so it would read 0
forever and imply an idle GPU.

Refs #752

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 07:22:46 +02:00
k.renner 483093a104 Merge pull request 'feat(monitoring): export Intel iGPU utilisation, frequency and power' (#822) from feat/intel-gpu-exporter into master
Reviewed-on: k.renner/homelab#822
2026-09-08 05:18:43 +00:00
claude-botandClaude Fable 5.1 8d348e7f88 fix(monitoring): let intel_gpu_top pick the GPU instead of hard-coding card0
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
The -device flag named /dev/dri/card0. That is the right node today on
every host, but nothing guarantees the number: a second GPU or a
different probe order and the exporter would silently watch the wrong
device or none. The plugin's monitoring resource injects every GPU on
the node and intel_gpu_top selects it on its own — verified with a probe
pod on k8s-2, which produced the same gpumon_* series without the flag.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 07:15:04 +02:00
claude-botandClaude Fable 5.1 6f9ff9d624 feat(monitoring): export Intel iGPU utilisation, frequency and power
A DaemonSet running clambin/intel-gpu-exporter (intel_gpu_top behind a
Prometheus endpoint) with a ServiceMonitor, giving gpumon_engine_usage,
gpumon_frequency, gpumon_power, gpumon_imc_* and gpumon_clients_count
per node.

Privilege model, validated with a one-shot pod before this commit:
/dev/dri comes from the device plugin's gpu.intel.com/monitoring
resource (every GPU on the node, no hostPath); the i915 PMU that
intel_gpu_top reads is a system-wide event source and the nodes run
perf_event_paranoid=3, so CAP_PERFMON is the one capability added.
No privileged, no hostPID, no hostNetwork, read-only root filesystem.
Root rather than a fixed uid because the capability has to be effective
without file capabilities on the binary.

End-to-end check on k8s-2 during a 40 s chat completion on llama-server:
Render/3D busy 98 %, actual frequency 1449 MHz, GPU power 18.8 W, while
the pod's CPU stayed near 100m — the "is it really on the GPU" question
this exists to answer.

XPU Manager was ruled out (Arc Pro / data-centre parts only); the
archived onedr0p exporter needs privileged + hostPID.

Fixes #752

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 07:14:24 +02:00
k.renner 7bf65783be Merge pull request 'feat(intel-gpu-plugin): advertise the monitoring resource' (#820) from feat/intel-gpu-monitoring-resource into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#820
2026-09-08 04:54:15 +00:00
k.renner dd9fb01359 Merge pull request 'fix(reloader): restart audiobookshelf and immich-server on ConfigMap change' (#819) from fix/reloader-configmap-annotations into master
Reviewed-on: k.renner/homelab#819
2026-09-08 04:52:32 +00:00
claude-botandClaude Fable 5.1 da8a725b32 feat(intel-gpu-plugin): advertise the monitoring resource
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Enables -enable-monitoring on the device plugin. With 0.36.0 that
advertises gpu.intel.com/monitoring, a single resource per node that
hands a pod every GPU rather than one of the shared slots; it is what a
metrics exporter requests to see the whole iGPU, and it is the only way
to get /dev/dri into a pod without a hostPath.

Nothing consumes it yet. The nodes currently advertise gpu.intel.com/i915
only — the issue assumed the chart default had this on, but the
DaemonSet here is hand-written and did not.

Refs #752

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 06:51:43 +02:00
claude-botandClaude Fable 5.1 6008d59c9f fix(reloader): restart audiobookshelf and immich-server on ConfigMap change
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Both consume their ConfigMap with envFrom, which Kubernetes does not
watch: a changed value applies in git and in the cluster while the
running container keeps the environment it started with. Paperless hit
exactly this in #739 and got the annotation in #741; these two were the
remaining envFrom consumers without one.

ftp-scanner, the third Deployment named in the issue, needs nothing: its
ConfigMap comes from a configMapGenerator that deliberately keeps the
content-hash suffix, so any change renames the ConfigMap, mutates the
pod template and rolls the pod on its own.

Reloader already watches all namespaces (kubernetes/infrastructure/reloader).

Fixes #742

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 06:46:12 +02:00
k.renner 5aa6c0609d Merge pull request 'fix(netbird-proxy-services): pin the digest the master build produced' (#818) from netbird-proxy-protocol-digest into master
ci/woodpecker/cron/e2e Pipeline failed
ci/woodpecker/cron/security-scan Pipeline was successful
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#818
2026-09-07 11:52:17 +00:00
claude-botandClaude Fable 5.1 0ebe113abb fix(netbird-proxy-services): pin the digest the master build produced
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
#817 pinned the digest from the branch build (pipeline 8452). Merging
#816 rebuilt the same date tag on master (pipeline 8457), Gitea kept
only the new manifest, and the pod has been in ImagePullBackOff since:
"failed to resolve reference ...@sha256:0d266b02...". With the Recreate
strategy no controller is running, so the services are unchanged.

The kaniko step of the master build pushed sha256:caffa308...; that is
the digest to pin. Comment states the rule.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-07 13:47:22 +02:00
k.renner 1723f73b5d Merge pull request 'feat(netbird-proxy-services): roll the PROXY-protocol build (2/2)' (#817) from netbird-proxy-protocol-deploy into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#817
2026-09-07 11:34:52 +00:00
k.renner 8d8df3b635 Merge pull request 'feat(netbird): carry the WAN client address through the passthrough edge (1/2)' (#816) from netbird-proxy-protocol into master
ci/woodpecker/push/build-ci-images Pipeline was successful
Reviewed-on: k.renner/homelab#816
2026-09-07 11:34:43 +00:00
15 changed files with 523 additions and 29 deletions
+1 -1
View File
@@ -158,7 +158,7 @@ steps:
# whenever it trails the deployed server. If you bump the server, bump here.
- name: woodpecker-lint
depends_on: [] # see note above `steps:`
image: woodpeckerci/woodpecker-cli:v3.17.0-alpine
image: woodpeckerci/woodpecker-cli:v3.18.0-alpine
commands:
- |
set -eu
+5 -5
View File
@@ -1,9 +1,9 @@
FROM alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d
ARG TALOS_VERSION=v1.12.6
ARG TALOS_VERSION=v1.13.9
ARG VELERO_VERSION=v1.16.0
ARG KUBECTL_VERSION=v1.36.2
ARG FLUX_VERSION=2.9.2
ARG KUBECTL_VERSION=v1.36.3
ARG FLUX_VERSION=2.9.5
RUN apk add --no-cache \
aws-cli \
@@ -18,7 +18,7 @@ RUN apk add --no-cache \
RUN curl -sfL -o /usr/local/bin/talosctl \
"https://github.com/siderolabs/talos/releases/download/${TALOS_VERSION}/talosctl-linux-amd64" \
&& echo "4c41a3b10b075292e64b182283d558e9d8e935f43239de1ed4e4b14359593efb /usr/local/bin/talosctl" | sha256sum -c - \
&& echo "7e1d4b7d5846964bdcf63a794e3c8161bb6ef2983d5ace58ea5322f3bf32a27e /usr/local/bin/talosctl" | sha256sum -c - \
&& chmod +x /usr/local/bin/talosctl
RUN curl -sfL -o /usr/local/bin/kubectl \
@@ -31,7 +31,7 @@ RUN curl -sfL -o /usr/local/bin/kubectl \
RUN curl -sfL -o /tmp/flux.tar.gz \
"https://github.com/fluxcd/flux2/releases/download/v${FLUX_VERSION}/flux_${FLUX_VERSION}_linux_amd64.tar.gz" \
&& echo "4092b367fd060097976fb7261601f79420ecf7266328417dfd8dfee27de0e6a3 /tmp/flux.tar.gz" | sha256sum -c - \
&& echo "b853df82adfd7736f580692f9f734473d571606307139f8fd20c2a80dd1ff473 /tmp/flux.tar.gz" | sha256sum -c - \
&& tar -xzf /tmp/flux.tar.gz -C /usr/local/bin flux \
&& chmod +x /usr/local/bin/flux \
&& rm -f /tmp/flux.tar.gz
@@ -21,6 +21,9 @@ metadata:
namespace: audiobookshelf
labels:
app.kubernetes.io/name: audiobookshelf
annotations:
# envFrom does not restart the pod when the ConfigMap changes; Reloader does.
configmap.reloader.stakater.com/reload: audiobookshelf-config
spec:
replicas: 1
strategy:
+3
View File
@@ -35,6 +35,9 @@ metadata:
namespace: immich
labels:
app.kubernetes.io/name: immich-server
annotations:
# envFrom does not restart the pod when the ConfigMap changes; Reloader does.
configmap.reloader.stakater.com/reload: immich-config
spec:
# Scaled to 0 alongside immich-machine-learning during single-node
# maintenance; see the note there. The CNPG Postgres cluster stays running,
@@ -0,0 +1,20 @@
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: intel-gpu-exporter
namespace: monitoring
labels:
app.kubernetes.io/name: intel-gpu-exporter
release: kube-prometheus-stack
spec:
selector:
matchLabels:
app.kubernetes.io/name: intel-gpu-exporter
endpoints:
- port: http-metrics
path: /metrics
interval: 30s
relabelings:
# The series would otherwise carry only the exporter pod's IP.
- sourceLabels: [__meta_kubernetes_pod_node_name]
targetLabel: node
@@ -39,4 +39,5 @@ resources:
- immich-servicemonitor.yaml
- loki-servicemonitor.yaml
- smartctl-exporter-servicemonitor.yaml
- intel-gpu-exporter-servicemonitor.yaml
- probes-blackbox.yaml
@@ -34,7 +34,9 @@ spec:
# query parameter, and an exact match would break silently if the
# instance label ever encodes it differently.
expr: probe_success{job="blackbox-http", instance=~"https://openbao\\.renner\\.uno/v1/sys/health.*"} == 0
for: 5m
# Two consecutive failed probes: enough to skip a single lost probe
# through Traefik, not enough to sit on a sealed store.
for: 1m
labels:
severity: critical
annotations:
@@ -31,7 +31,9 @@ spec:
# A replica that is sealed never recovers on its own, so the
# duration is only there to ignore ordinary rescheduling: a healthy
# pod moving between nodes is Ready again in well under a minute.
expr: kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"} < 3
# max by: kube-state-metrics stamps instance/pod on the series, so a
# rescheduled exporter starts a new series and resets the timer.
expr: max by (namespace, statefulset) (kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"}) < 3
for: 10m
labels:
severity: warning
@@ -53,11 +55,11 @@ spec:
# the ClusterSecretStore goes InvalidProviderConfig and no
# ExternalSecret can refresh.
#
# Shorter window than the warning above because this is an outage,
# not a degradation — but not zero, so a rolling restart that
# briefly dips does not page.
expr: kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"} < 2
for: 5m
# No window: below quorum the secret store is down and every
# ExternalSecret stalls with it. A sealed replica never recovers on
# its own, so waiting only delays the page.
expr: max by (namespace, statefulset) (kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"}) < 2
for: 0m
labels:
severity: critical
annotations:
@@ -0,0 +1,368 @@
{
"title": "Intel iGPU",
"uid": "intel-gpu",
"tags": ["gpu", "intel", "i915"],
"schemaVersion": 39,
"editable": true,
"timezone": "",
"refresh": "1m",
"time": { "from": "now-6h", "to": "now" },
"templating": {
"list": [
{
"name": "node",
"label": "Node",
"type": "query",
"datasource": { "type": "prometheus", "uid": "prometheus" },
"query": {
"query": "label_values(gpumon_power, node)",
"refId": "PrometheusVariableQueryEditor-VariableQuery"
},
"definition": "label_values(gpumon_power, node)",
"includeAll": true,
"multi": true,
"refresh": 2,
"sort": 1,
"current": { "selected": true, "text": ["All"], "value": ["$__all"] }
}
]
},
"panels": [
{
"type": "stat",
"title": "Render/3D busy now",
"gridPos": { "h": 4, "w": 8, "x": 0, "y": 0 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "percent",
"decimals": 0,
"min": 0,
"max": 100,
"color": { "mode": "fixed", "fixedColor": "text" }
},
"overrides": []
},
"options": {
"colorMode": "none",
"graphMode": "none",
"justifyMode": "center",
"orientation": "horizontal",
"textMode": "value_and_name",
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }
},
"targets": [
{
"expr": "gpumon_engine_usage{attrib=\"busy\", engine=\"Render/3D\", node=~\"$node\"}",
"legendFormat": "{{node}}",
"refId": "A"
}
]
},
{
"type": "stat",
"title": "GPU power now",
"gridPos": { "h": 4, "w": 8, "x": 8, "y": 0 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "watt",
"decimals": 1,
"color": { "mode": "fixed", "fixedColor": "text" }
},
"overrides": []
},
"options": {
"colorMode": "none",
"graphMode": "none",
"justifyMode": "center",
"orientation": "horizontal",
"textMode": "value_and_name",
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }
},
"targets": [
{
"expr": "gpumon_power{type=\"gpu\", node=~\"$node\"}",
"legendFormat": "{{node}}",
"refId": "A"
}
]
},
{
"type": "stat",
"title": "GT frequency now",
"gridPos": { "h": 4, "w": 8, "x": 16, "y": 0 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "suffix: MHz",
"decimals": 0,
"color": { "mode": "fixed", "fixedColor": "text" }
},
"overrides": []
},
"options": {
"colorMode": "none",
"graphMode": "none",
"justifyMode": "center",
"orientation": "horizontal",
"textMode": "value_and_name",
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }
},
"targets": [
{
"expr": "gpumon_frequency{type=\"actual\", node=~\"$node\"}",
"legendFormat": "{{node}}",
"refId": "A"
}
]
},
{
"type": "timeseries",
"title": "Render/3D busy",
"description": "Compute and 3D. llama.cpp's Vulkan backend runs here; 0 % during an inference means the model is on the CPU.",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 4 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "percent",
"min": 0,
"max": 100,
"custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
},
"overrides": [
{ "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] }
]
},
"options": {
"legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"expr": "gpumon_engine_usage{attrib=\"busy\", engine=\"Render/3D\", node=~\"$node\"}",
"legendFormat": "{{node}}",
"refId": "A"
}
]
},
{
"type": "timeseries",
"title": "Video busy",
"description": "Fixed-function decode/encode (QuickSync).",
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 4 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "percent",
"min": 0,
"max": 100,
"custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
},
"overrides": [
{ "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] }
]
},
"options": {
"legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"expr": "gpumon_engine_usage{attrib=\"busy\", engine=\"Video\", node=~\"$node\"}",
"legendFormat": "{{node}}",
"refId": "A"
}
]
},
{
"type": "timeseries",
"title": "VideoEnhance busy",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 12 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "percent",
"min": 0,
"max": 100,
"custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
},
"overrides": [
{ "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] }
]
},
"options": {
"legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"expr": "gpumon_engine_usage{attrib=\"busy\", engine=\"VideoEnhance\", node=~\"$node\"}",
"legendFormat": "{{node}}",
"refId": "A"
}
]
},
{
"type": "timeseries",
"title": "Blitter busy",
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 12 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "percent",
"min": 0,
"max": 100,
"custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
},
"overrides": [
{ "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] }
]
},
"options": {
"legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"expr": "gpumon_engine_usage{attrib=\"busy\", engine=\"Blitter\", node=~\"$node\"}",
"legendFormat": "{{node}}",
"refId": "A"
}
]
},
{
"type": "timeseries",
"title": "GT frequency (actual)",
"description": "0 while the GT sits in RC6.",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 20 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "suffix: MHz",
"min": 0,
"custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
},
"overrides": [
{ "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] }
]
},
"options": {
"legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"expr": "gpumon_frequency{type=\"actual\", node=~\"$node\"}",
"legendFormat": "{{node}}",
"refId": "A"
}
]
},
{
"type": "timeseries",
"title": "GPU power",
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 20 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "watt",
"min": 0,
"custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
},
"overrides": [
{ "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] }
]
},
"options": {
"legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"expr": "gpumon_power{type=\"gpu\", node=~\"$node\"}",
"legendFormat": "{{node}}",
"refId": "A"
}
]
},
{
"type": "timeseries",
"title": "Package power",
"description": "Whole SoC (CPU + GPU + uncore), from the same PMU.",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 28 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "watt",
"min": 0,
"custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
},
"overrides": [
{ "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] }
]
},
"options": {
"legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"expr": "gpumon_power{type=\"pkg\", node=~\"$node\"}",
"legendFormat": "{{node}}",
"refId": "A"
}
]
},
{
"type": "timeseries",
"title": "Memory controller bandwidth",
"description": "Reads solid, writes dashed. The iGPU shares DDR5 with the CPU, so this is the whole package's traffic.",
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 28 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": {
"defaults": {
"unit": "MiBs",
"min": 0,
"custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
},
"overrides": [
{ "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
{ "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] },
{ "matcher": { "id": "byRegexp", "options": "/write/" }, "properties": [{ "id": "custom.lineStyle", "value": { "fill": "dash", "dash": [10, 10] } }] }
]
},
"options": {
"legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"expr": "gpumon_imc_read{node=~\"$node\"}",
"legendFormat": "{{node}} read",
"refId": "A"
},
{
"expr": "gpumon_imc_write{node=~\"$node\"}",
"legendFormat": "{{node}} write",
"refId": "B"
}
]
}
]
}
@@ -77,6 +77,9 @@ configMapGenerator:
- name: dashboard-infrastructure-overview
files:
- infrastructure-overview.json
- name: dashboard-intel-gpu
files:
- intel-gpu.json
- name: dashboard-netbird-overview
files:
- netbird-overview.json
@@ -0,0 +1,81 @@
# Intel iGPU utilisation, frequency and power for every node (#752).
#
# Not privileged: /dev/dri comes from the device plugin's monitoring resource
# (every GPU on the node, no hostPath), and the i915 PMU that intel_gpu_top
# reads needs CAP_PERFMON — the one capability granted. No hostPID, no
# hostNetwork.
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: intel-gpu-exporter
namespace: monitoring
labels:
app.kubernetes.io/name: intel-gpu-exporter
spec:
selector:
matchLabels:
app.kubernetes.io/name: intel-gpu-exporter
template:
metadata:
labels:
app.kubernetes.io/name: intel-gpu-exporter
spec:
# Every node is a control plane; without this the exporter covers nothing.
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule
securityContext:
seccompProfile:
type: RuntimeDefault
containers:
- name: intel-gpu-exporter
image: ghcr.io/clambin/intel-gpu-exporter:0.7.4@sha256:d78761fef018834c759db154fe9dcf3dd9db5974703fa50ca21f6abc35221be1
args:
# No -device: intel_gpu_top picks the only GPU the plugin injects.
- -interval=5s
ports:
- name: http-metrics
containerPort: 9100
protocol: TCP
securityContext:
# Root so the PERFMON capability is effective without file
# capabilities on the binary.
runAsUser: 0
runAsNonRoot: false
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
add: ["PERFMON"]
resources:
requests:
cpu: 10m
memory: 32Mi
gpu.intel.com/monitoring: 1
limits:
memory: 128Mi
gpu.intel.com/monitoring: 1
livenessProbe:
httpGet:
path: /metrics
port: http-metrics
initialDelaySeconds: 15
periodSeconds: 30
---
apiVersion: v1
kind: Service
metadata:
name: intel-gpu-exporter
namespace: monitoring
labels:
app.kubernetes.io/name: intel-gpu-exporter
spec:
clusterIP: None
selector:
app.kubernetes.io/name: intel-gpu-exporter
ports:
- name: http-metrics
port: 9100
targetPort: http-metrics
protocol: TCP
@@ -16,6 +16,7 @@ resources:
- prometheus-gate-ingress.yaml
- ci-diagnostics-rbac.yaml
- smartctl-exporter.yaml
- intel-gpu-exporter.yaml
- dashboards/
# prometheusrule-garage.yaml lives in ./monitoring-rules/ and is applied by a
# separate Flux Kustomization that dependsOn `monitoring` — otherwise the
@@ -72,6 +72,9 @@ spec:
# they contend for it. Sized for an inference server plus room for
# a transcoder and a benchmark pod alongside it.
- -shared-dev-num=4
# Advertises gpu.intel.com/monitoring: one resource that grants a
# pod every GPU on the node instead of a slot. The exporter needs it.
- -enable-monitoring
securityContext:
seLinuxOptions:
type: "container_device_plugin_t"
@@ -77,8 +77,10 @@ spec:
containers:
- name: controller
# Pinned by digest: the date tag is rebuilt on every push of the
# image directory, so the tag alone cannot express a new build.
image: gitea.renner.uno/gitea_admin/netbird-proxy-services:2026-09-07@sha256:0d266b0288aec692c698ff5c05f76c826e247bfef365e0306a0b9a6541ebf180
# image directory (branch and master alike) and the registry keeps
# only the newest manifest per tag, so the digest must come from
# the MASTER build of the code change.
image: gitea.renner.uno/gitea_admin/netbird-proxy-services:2026-09-07@sha256:caffa308d802c6fcc6aea919ac94657c08b993651baa6b8d18de395bcc40523e
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
@@ -25,10 +25,10 @@ spec:
name: openbao
namespace: openbao
install:
# A fresh OpenBao pod is 0/1 Ready until `bao operator init` + unseal
# (readiness probe reports sealed). helm --wait would time out and
# remediation would uninstall the release, so don't wait here; the
# init/unseal procedure is manual (see kubernetes/MIGRATION-sops-openbao.md).
# A fresh OpenBao pod is 0/1 Ready until `bao operator init` (readiness
# reports sealed). helm --wait would time out and remediation would
# uninstall the release, so don't wait here; init is manual (see
# kubernetes/MIGRATION-sops-openbao.md). Unseal is automatic (static seal).
disableWait: true
remediation:
retries: 3
@@ -42,6 +42,11 @@ spec:
# the agent-injector mutating webhook is unused surface.
enabled: false
server:
# The static seal key, from the openbao-seal-key Secret (bootstrap.sh).
extraSecretEnvironmentVars:
- envName: BAO_SEAL_KEY
secretName: openbao-seal-key
secretKey: key
# Raft (not standalone/file) for two reasons: `bao operator raft
# snapshot` as a consistent backup primitive, and real HA across three
# nodes. TLS terminates at the Traefik ingress; in-cluster traffic is
@@ -53,16 +58,8 @@ spec:
# the moment one pod is sealed or rescheduled, which is worse than
# the single replica it replaced. Three tolerates one.
#
# What this buys, and what it does not: a node drain seals the replica
# on it, and raft elects a new leader from the remaining standbys
# within seconds, so the service does not go down. It does NOT remove
# the manual unseal — a sealed server can neither serve requests nor
# act as standby, so that replica stays out until someone unseals it.
#
# The consequence for rolling node maintenance is worth stating: a
# Talos upgrade drains all three nodes in turn, sealing each replica
# as it goes. Unseal each one as it comes back, or the third drain
# takes the last unsealed replica with it and quorum is lost anyway.
# A restarted replica unseals itself from the static seal and rejoins
# raft, so a rolling reboot only ever costs one replica at a time.
replicas: 3
raft:
enabled: true
@@ -111,6 +108,14 @@ spec:
service_registration "kubernetes" {}
# Auto-unseal. The key is a 32-byte AES key in the openbao-seal-key
# Secret, created from pass at bootstrap before this release. The
# former shamir key is the recovery key (#805).
seal "static" {
current_key_id = "1"
current_key = "env://BAO_SEAL_KEY"
}
audit "file" "stdout" {
description = "audit trail to pod stdout -> Loki"
options {