chore(ci): bump CI tool pins to match the deployed cluster

From the update audit (#426): the tools CI drives the cluster with trailed what the cluster runs. ci/images/k8s/Containerfile - talosctl v1.12.6 -> v1.13.9, the Talos version on the nodes. An older talosctl can misjudge or reject config generated for a newer machine. - kubectl v1.36.2 -> v1.36.3, the cluster's Kubernetes version. - flux CLI 2.9.2 -> 2.9.5, the current 2.9.x. Checksums verified against the upstream release assets (talos sha256sum.txt, flux checksums.txt) and against a local download. .woodpecker/ci-manifests-test.yml - woodpecker-cli v3.17.0 -> v3.18.0, the version the server and agent already run; the lint step was one minor behind its own server. Merging triggers build-ci-images (path ci/images/**), which publishes ci-k8s:<date>. The six pipelines pinned to ci-k8s:2026-08-10 move to that tag in a follow-up once it exists. Refs #426 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Merge pull request 'fix(alerts): page the moment OpenBao is below quorum' (#824 ) from fix/openbao-alerts-immediate into master
2026-09-08 08:58:35 +02:00 · 2026-09-08 06:02:32 +00:00 · 2026-09-08 05:54:54 +00:00 · 2026-09-08 07:52:27 +02:00 · 2026-09-08 07:40:12 +02:00 · 2026-09-08 05:24:35 +00:00
7 changed files with 407 additions and 27 deletions
@@ -158,7 +158,7 @@ steps:
  # whenever it trails the deployed server. If you bump the server, bump here.
  - name: woodpecker-lint
    depends_on: [] # see note above `steps:`
-    image: woodpeckerci/woodpecker-cli:v3.17.0-alpine
+    image: woodpeckerci/woodpecker-cli:v3.18.0-alpine
    commands:
      - |
        set -eu
@@ -1,9 +1,9 @@
 FROM alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d

-ARG TALOS_VERSION=v1.12.6
+ARG TALOS_VERSION=v1.13.9
 ARG VELERO_VERSION=v1.16.0
-ARG KUBECTL_VERSION=v1.36.2
-ARG FLUX_VERSION=2.9.2
+ARG KUBECTL_VERSION=v1.36.3
+ARG FLUX_VERSION=2.9.5

 RUN apk add --no-cache \
      aws-cli \
@@ -18,7 +18,7 @@ RUN apk add --no-cache \

 RUN curl -sfL -o /usr/local/bin/talosctl \
      "https://github.com/siderolabs/talos/releases/download/${TALOS_VERSION}/talosctl-linux-amd64" \
- && echo "4c41a3b10b075292e64b182283d558e9d8e935f43239de1ed4e4b14359593efb  /usr/local/bin/talosctl" | sha256sum -c - \
+ && echo "7e1d4b7d5846964bdcf63a794e3c8161bb6ef2983d5ace58ea5322f3bf32a27e  /usr/local/bin/talosctl" | sha256sum -c - \
 && chmod +x /usr/local/bin/talosctl

 RUN curl -sfL -o /usr/local/bin/kubectl \
@@ -31,7 +31,7 @@ RUN curl -sfL -o /usr/local/bin/kubectl \

 RUN curl -sfL -o /tmp/flux.tar.gz \
      "https://github.com/fluxcd/flux2/releases/download/v${FLUX_VERSION}/flux_${FLUX_VERSION}_linux_amd64.tar.gz" \
- && echo "4092b367fd060097976fb7261601f79420ecf7266328417dfd8dfee27de0e6a3  /tmp/flux.tar.gz" | sha256sum -c - \
+ && echo "b853df82adfd7736f580692f9f734473d571606307139f8fd20c2a80dd1ff473  /tmp/flux.tar.gz" | sha256sum -c - \
 && tar -xzf /tmp/flux.tar.gz -C /usr/local/bin flux \
 && chmod +x /usr/local/bin/flux \
 && rm -f /tmp/flux.tar.gz
@@ -34,7 +34,9 @@ spec:
          # query parameter, and an exact match would break silently if the
          # instance label ever encodes it differently.
          expr: probe_success{job="blackbox-http", instance=~"https://openbao\\.renner\\.uno/v1/sys/health.*"} == 0
-          for: 5m
+          # Two consecutive failed probes: enough to skip a single lost probe
+          # through Traefik, not enough to sit on a sealed store.
+          for: 1m
          labels:
            severity: critical
          annotations:
@@ -31,7 +31,9 @@ spec:
          # A replica that is sealed never recovers on its own, so the
          # duration is only there to ignore ordinary rescheduling: a healthy
          # pod moving between nodes is Ready again in well under a minute.
-          expr: kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"} < 3
+          # max by: kube-state-metrics stamps instance/pod on the series, so a
+          # rescheduled exporter starts a new series and resets the timer.
+          expr: max by (namespace, statefulset) (kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"}) < 3
          for: 10m
          labels:
            severity: warning
@@ -53,11 +55,11 @@ spec:
          # the ClusterSecretStore goes InvalidProviderConfig and no
          # ExternalSecret can refresh.
          #
-          # Shorter window than the warning above because this is an outage,
-          # not a degradation — but not zero, so a rolling restart that
-          # briefly dips does not page.
-          expr: kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"} < 2
-          for: 5m
+          # No window: below quorum the secret store is down and every
+          # ExternalSecret stalls with it. A sealed replica never recovers on
+          # its own, so waiting only delays the page.
+          expr: max by (namespace, statefulset) (kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"}) < 2
+          for: 0m
          labels:
            severity: critical
          annotations:
@@ -0,0 +1,368 @@
+{
+  "title": "Intel iGPU",
+  "uid": "intel-gpu",
+  "tags": ["gpu", "intel", "i915"],
+  "schemaVersion": 39,
+  "editable": true,
+  "timezone": "",
+  "refresh": "1m",
+  "time": { "from": "now-6h", "to": "now" },
+  "templating": {
+    "list": [
+      {
+        "name": "node",
+        "label": "Node",
+        "type": "query",
+        "datasource": { "type": "prometheus", "uid": "prometheus" },
+        "query": {
+          "query": "label_values(gpumon_power, node)",
+          "refId": "PrometheusVariableQueryEditor-VariableQuery"
+        },
+        "definition": "label_values(gpumon_power, node)",
+        "includeAll": true,
+        "multi": true,
+        "refresh": 2,
+        "sort": 1,
+        "current": { "selected": true, "text": ["All"], "value": ["$__all"] }
+      }
+    ]
+  },
+  "panels": [
+    {
+      "type": "stat",
+      "title": "Render/3D busy now",
+      "gridPos": { "h": 4, "w": 8, "x": 0, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percent",
+          "decimals": 0,
+          "min": 0,
+          "max": 100,
+          "color": { "mode": "fixed", "fixedColor": "text" }
+        },
+        "overrides": []
+      },
+      "options": {
+        "colorMode": "none",
+        "graphMode": "none",
+        "justifyMode": "center",
+        "orientation": "horizontal",
+        "textMode": "value_and_name",
+        "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }
+      },
+      "targets": [
+        {
+          "expr": "gpumon_engine_usage{attrib=\"busy\", engine=\"Render/3D\", node=~\"$node\"}",
+          "legendFormat": "{{node}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "type": "stat",
+      "title": "GPU power now",
+      "gridPos": { "h": 4, "w": 8, "x": 8, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "watt",
+          "decimals": 1,
+          "color": { "mode": "fixed", "fixedColor": "text" }
+        },
+        "overrides": []
+      },
+      "options": {
+        "colorMode": "none",
+        "graphMode": "none",
+        "justifyMode": "center",
+        "orientation": "horizontal",
+        "textMode": "value_and_name",
+        "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }
+      },
+      "targets": [
+        {
+          "expr": "gpumon_power{type=\"gpu\", node=~\"$node\"}",
+          "legendFormat": "{{node}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "type": "stat",
+      "title": "GT frequency now",
+      "gridPos": { "h": 4, "w": 8, "x": 16, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "suffix: MHz",
+          "decimals": 0,
+          "color": { "mode": "fixed", "fixedColor": "text" }
+        },
+        "overrides": []
+      },
+      "options": {
+        "colorMode": "none",
+        "graphMode": "none",
+        "justifyMode": "center",
+        "orientation": "horizontal",
+        "textMode": "value_and_name",
+        "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }
+      },
+      "targets": [
+        {
+          "expr": "gpumon_frequency{type=\"actual\", node=~\"$node\"}",
+          "legendFormat": "{{node}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "type": "timeseries",
+      "title": "Render/3D busy",
+      "description": "Compute and 3D. llama.cpp's Vulkan backend runs here; 0 % during an inference means the model is on the CPU.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 4 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percent",
+          "min": 0,
+          "max": 100,
+          "custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
+        },
+        "overrides": [
+          { "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
+          { "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
+          { "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] }
+        ]
+      },
+      "options": {
+        "legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
+        "tooltip": { "mode": "multi", "sort": "desc" }
+      },
+      "targets": [
+        {
+          "expr": "gpumon_engine_usage{attrib=\"busy\", engine=\"Render/3D\", node=~\"$node\"}",
+          "legendFormat": "{{node}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "type": "timeseries",
+      "title": "Video busy",
+      "description": "Fixed-function decode/encode (QuickSync).",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 4 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percent",
+          "min": 0,
+          "max": 100,
+          "custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
+        },
+        "overrides": [
+          { "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
+          { "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
+          { "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] }
+        ]
+      },
+      "options": {
+        "legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
+        "tooltip": { "mode": "multi", "sort": "desc" }
+      },
+      "targets": [
+        {
+          "expr": "gpumon_engine_usage{attrib=\"busy\", engine=\"Video\", node=~\"$node\"}",
+          "legendFormat": "{{node}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "type": "timeseries",
+      "title": "VideoEnhance busy",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 12 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percent",
+          "min": 0,
+          "max": 100,
+          "custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
+        },
+        "overrides": [
+          { "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
+          { "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
+          { "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] }
+        ]
+      },
+      "options": {
+        "legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
+        "tooltip": { "mode": "multi", "sort": "desc" }
+      },
+      "targets": [
+        {
+          "expr": "gpumon_engine_usage{attrib=\"busy\", engine=\"VideoEnhance\", node=~\"$node\"}",
+          "legendFormat": "{{node}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "type": "timeseries",
+      "title": "Blitter busy",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 12 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percent",
+          "min": 0,
+          "max": 100,
+          "custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
+        },
+        "overrides": [
+          { "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
+          { "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
+          { "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] }
+        ]
+      },
+      "options": {
+        "legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
+        "tooltip": { "mode": "multi", "sort": "desc" }
+      },
+      "targets": [
+        {
+          "expr": "gpumon_engine_usage{attrib=\"busy\", engine=\"Blitter\", node=~\"$node\"}",
+          "legendFormat": "{{node}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "type": "timeseries",
+      "title": "GT frequency (actual)",
+      "description": "0 while the GT sits in RC6.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 20 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "suffix: MHz",
+          "min": 0,
+          "custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
+        },
+        "overrides": [
+          { "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
+          { "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
+          { "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] }
+        ]
+      },
+      "options": {
+        "legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
+        "tooltip": { "mode": "multi", "sort": "desc" }
+      },
+      "targets": [
+        {
+          "expr": "gpumon_frequency{type=\"actual\", node=~\"$node\"}",
+          "legendFormat": "{{node}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "type": "timeseries",
+      "title": "GPU power",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 20 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "watt",
+          "min": 0,
+          "custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
+        },
+        "overrides": [
+          { "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
+          { "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
+          { "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] }
+        ]
+      },
+      "options": {
+        "legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
+        "tooltip": { "mode": "multi", "sort": "desc" }
+      },
+      "targets": [
+        {
+          "expr": "gpumon_power{type=\"gpu\", node=~\"$node\"}",
+          "legendFormat": "{{node}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "type": "timeseries",
+      "title": "Package power",
+      "description": "Whole SoC (CPU + GPU + uncore), from the same PMU.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 28 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "watt",
+          "min": 0,
+          "custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
+        },
+        "overrides": [
+          { "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
+          { "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
+          { "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] }
+        ]
+      },
+      "options": {
+        "legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
+        "tooltip": { "mode": "multi", "sort": "desc" }
+      },
+      "targets": [
+        {
+          "expr": "gpumon_power{type=\"pkg\", node=~\"$node\"}",
+          "legendFormat": "{{node}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "type": "timeseries",
+      "title": "Memory controller bandwidth",
+      "description": "Reads solid, writes dashed. The iGPU shares DDR5 with the CPU, so this is the whole package's traffic.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 28 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "MiBs",
+          "min": 0,
+          "custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 0, "showPoints": "never", "spanNulls": false }
+        },
+        "overrides": [
+          { "matcher": { "id": "byRegexp", "options": "/k8s-1/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#3987e5" } }] },
+          { "matcher": { "id": "byRegexp", "options": "/k8s-2/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#d95926" } }] },
+          { "matcher": { "id": "byRegexp", "options": "/k8s-3/" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "#199e70" } }] },
+          { "matcher": { "id": "byRegexp", "options": "/write/" }, "properties": [{ "id": "custom.lineStyle", "value": { "fill": "dash", "dash": [10, 10] } }] }
+        ]
+      },
+      "options": {
+        "legend": { "calcs": ["mean", "max", "lastNotNull"], "displayMode": "table", "placement": "bottom", "showLegend": true },
+        "tooltip": { "mode": "multi", "sort": "desc" }
+      },
+      "targets": [
+        {
+          "expr": "gpumon_imc_read{node=~\"$node\"}",
+          "legendFormat": "{{node}} read",
+          "refId": "A"
+        },
+        {
+          "expr": "gpumon_imc_write{node=~\"$node\"}",
+          "legendFormat": "{{node}} write",
+          "refId": "B"
+        }
+      ]
+    }
+  ]
+}
@@ -77,6 +77,9 @@ configMapGenerator:
  - name: dashboard-infrastructure-overview
    files:
      - infrastructure-overview.json
+  - name: dashboard-intel-gpu
+    files:
+      - intel-gpu.json
  - name: dashboard-netbird-overview
    files:
      - netbird-overview.json
@@ -25,10 +25,10 @@ spec:
        name: openbao
        namespace: openbao
  install:
-    # A fresh OpenBao pod is 0/1 Ready until `bao operator init` + unseal
-    # (readiness probe reports sealed). helm --wait would time out and
-    # remediation would uninstall the release, so don't wait here; the
-    # init/unseal procedure is manual (see kubernetes/MIGRATION-sops-openbao.md).
+    # A fresh OpenBao pod is 0/1 Ready until `bao operator init` (readiness
+    # reports sealed). helm --wait would time out and remediation would
+    # uninstall the release, so don't wait here; init is manual (see
+    # kubernetes/MIGRATION-sops-openbao.md). Unseal is automatic (static seal).
    disableWait: true
    remediation:
      retries: 3
@@ -42,6 +42,11 @@ spec:
      # the agent-injector mutating webhook is unused surface.
      enabled: false
    server:
+      # The static seal key, from the openbao-seal-key Secret (bootstrap.sh).
+      extraSecretEnvironmentVars:
+        - envName: BAO_SEAL_KEY
+          secretName: openbao-seal-key
+          secretKey: key
      # Raft (not standalone/file) for two reasons: `bao operator raft
      # snapshot` as a consistent backup primitive, and real HA across three
      # nodes. TLS terminates at the Traefik ingress; in-cluster traffic is
@@ -53,16 +58,8 @@ spec:
        # the moment one pod is sealed or rescheduled, which is worse than
        # the single replica it replaced. Three tolerates one.
        #
-        # What this buys, and what it does not: a node drain seals the replica
-        # on it, and raft elects a new leader from the remaining standbys
-        # within seconds, so the service does not go down. It does NOT remove
-        # the manual unseal — a sealed server can neither serve requests nor
-        # act as standby, so that replica stays out until someone unseals it.
-        #
-        # The consequence for rolling node maintenance is worth stating: a
-        # Talos upgrade drains all three nodes in turn, sealing each replica
-        # as it goes. Unseal each one as it comes back, or the third drain
-        # takes the last unsealed replica with it and quorum is lost anyway.
+        # A restarted replica unseals itself from the static seal and rejoins
+        # raft, so a rolling reboot only ever costs one replica at a time.
        replicas: 3
        raft:
          enabled: true
@@ -111,6 +108,14 @@ spec:

            service_registration "kubernetes" {}

+            # Auto-unseal. The key is a 32-byte AES key in the openbao-seal-key
+            # Secret, created from pass at bootstrap before this release. The
+            # former shamir key is the recovery key (#805).
+            seal "static" {
+              current_key_id = "1"
+              current_key    = "env://BAO_SEAL_KEY"
+            }
+
            audit "file" "stdout" {
              description = "audit trail to pod stdout -> Loki"
              options {
Author	SHA1	Message	Date
claude-botandClaude Fable 5.1	6b0617b4a7	chore(ci): bump CI tool pins to match the deployed cluster ci/woodpecker/pr/ci-manifests-test Pipeline was successful Details ci/woodpecker/pr/ci-scripts-test Pipeline was successful Details ci/woodpecker/push/build-ci-images Pipeline was successful Details ci/woodpecker/pr/pr-review Pipeline was successful Details From the update audit (#426): the tools CI drives the cluster with trailed what the cluster runs. ci/images/k8s/Containerfile - talosctl v1.12.6 -> v1.13.9, the Talos version on the nodes. An older talosctl can misjudge or reject config generated for a newer machine. - kubectl v1.36.2 -> v1.36.3, the cluster's Kubernetes version. - flux CLI 2.9.2 -> 2.9.5, the current 2.9.x. Checksums verified against the upstream release assets (talos sha256sum.txt, flux checksums.txt) and against a local download. .woodpecker/ci-manifests-test.yml - woodpecker-cli v3.17.0 -> v3.18.0, the version the server and agent already run; the lint step was one minor behind its own server. Merging triggers build-ci-images (path ci/images/**), which publishes ci-k8s:<date>. The six pipelines pinned to ci-k8s:2026-08-10 move to that tag in a follow-up once it exists. Refs #426 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>	2026-09-08 08:58:35 +02:00
k.renner	6b7736ea58	Merge pull request 'fix(alerts): page the moment OpenBao is below quorum' (#824 ) from fix/openbao-alerts-immediate into master ci/woodpecker/cron/issue-fix Pipeline was successful Details ci/woodpecker/cron/alert-monitor Pipeline was successful Details Reviewed-on: k.renner/homelab#824	2026-09-08 06:02:32 +00:00
k.renner	7ff20e5b68	Merge pull request 'feat(openbao): auto-unseal with a static seal' (#825 ) from feat/openbao-static-seal into master ci/woodpecker/cron/alert-monitor Pipeline was successful Details Reviewed-on: k.renner/homelab#825	2026-09-08 05:54:54 +00:00
claude-botandClaude Fable 5.1	296accd1cb	feat(openbao): auto-unseal with a static seal ci/woodpecker/pr/ci-scripts-test Pipeline was successful Details ci/woodpecker/pr/ci-manifests-test Pipeline was successful Details ci/woodpecker/pr/pr-review Pipeline was successful Details Every restart sealed the replica until a human unsealed it. On 2026-09-07 a rolling talosctl upgrade rebooted all three nodes; the PDB refused the second eviction while the first replica was sealed, Talos waited out its 5-minute drain timeout and rebooted anyway, and raft quorum was gone for 24 minutes — every ExternalSecret and the Kustomizations behind them with it. A power loss would do the same with no timeout to blame. OpenBao 2.6.1 supports a static seal: a 32-byte key the server uses to unseal itself. The key is generated into pass (homelab/openbao/seal-key) and created as the openbao-seal-key Secret before this release reconciles; the chart hands it to the server as BAO_SEAL_KEY from that Secret. This overturns the principle in bootstrap.sh that the key protecting OpenBao must not live in the cluster OpenBao secures. Decided by the owner on 2026-09-08: the alternatives were a transit seal from a second OpenBao on edge-1, which raft traffic would wear out, or a workstation unseal script that may not be at hand when needed. The exposure is bounded by what ESO already does — every synced secret is plaintext in etcd (encrypted at rest) — and grows only by what lives solely in OpenBao. Migration is a live, per-replica procedure with the shamir key becoming the recovery key; the StatefulSet is OnDelete, so merging this changes nothing until each pod is restarted in turn. The bootstrap and rebuild paths (bootstrap.sh, vmtest) follow in a separate change once the migration is verified. Refs #805 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>	2026-09-08 07:52:27 +02:00
claude-botandClaude Fable 5.1	2665e5ebf3	fix(alerts): page the moment OpenBao is below quorum ci/woodpecker/pr/ci-scripts-test Pipeline was successful Details ci/woodpecker/pr/ci-manifests-test Pipeline was successful Details ci/woodpecker/pr/pr-review Pipeline was successful Details OpenBaoQuorumLost waited for: 5m. Below quorum the secret store is down — the ClusterSecretStore goes InvalidProviderConfig and every ExternalSecret and the Kustomizations behind them stall — and a sealed replica never recovers on its own, so the window only delayed the page. It fires on the first evaluation now. OpenBaoUnavailable, the external probe, drops from 5m to 1m: two consecutive failed probes, enough to skip a single lost probe through Traefik, not enough to sit on a sealed store. Both StatefulSet expressions are wrapped in max by (namespace, statefulset). kube-state-metrics stamps instance/pod on the series, so a rescheduled exporter starts a new series and Prometheus resets the for: timer with it — on 2026-09-07 that made OpenBaoReplicaNotReady fire 14 minutes late. Alertmanager routes match alertname/source only, so the dropped labels change nothing downstream. Refs #805 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>	2026-09-08 07:40:12 +02:00
k.renner	0fe512746b	Merge pull request 'feat(monitoring): Intel iGPU dashboard' (#823 ) from feat/intel-gpu-dashboard into master ci/woodpecker/cron/issue-fix Pipeline was successful Details ci/woodpecker/cron/alert-monitor Pipeline was successful Details Reviewed-on: k.renner/homelab#823	2026-09-08 05:24:35 +00:00
claude-botandClaude Fable 5.1	4e2303f455	feat(monitoring): Intel iGPU dashboard ci/woodpecker/pr/ci-manifests-test Pipeline was successful Details ci/woodpecker/pr/ci-scripts-test Pipeline was successful Details ci/woodpecker/pr/pr-review Pipeline was successful Details Grafana dashboard for the gpumon_* series from intel-gpu-exporter: current Render/3D busy, GPU power and GT frequency per node as stat tiles, then one time series per engine (Render/3D, Video, VideoEnhance, Blitter) with node as the series, GT frequency, GPU and package power, and memory-controller read/write bandwidth. Node colours are fixed per node across every panel — blue, orange, aqua for k8s-1/2/3 — rather than assigned by series order, so a node keeps its colour whatever the $node filter shows. The trio validates for colour-vision deficiency and normal vision on both Grafana themes (worst all-pairs ΔE 9.4 / 20.9). Reads and writes share one axis and differ by dash, not hue. One axis per panel throughout. gpumon_clients_count is deliberately absent: per-client attribution needs hostPID, which the exporter does not have, so it would read 0 forever and imply an idle GPU. Refs #752 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>	2026-09-08 07:22:46 +02:00
k.renner	483093a104	Merge pull request 'feat(monitoring): export Intel iGPU utilisation, frequency and power' (#822 ) from feat/intel-gpu-exporter into master Reviewed-on: k.renner/homelab#822	2026-09-08 05:18:43 +00:00