Compare commits

..
Author SHA1 Message Date
claude-botandClaude Fable 5.1 6b0617b4a7 chore(ci): bump CI tool pins to match the deployed cluster
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/push/build-ci-images Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
From the update audit (#426): the tools CI drives the cluster with
trailed what the cluster runs.

ci/images/k8s/Containerfile
- talosctl v1.12.6 -> v1.13.9, the Talos version on the nodes. An older
  talosctl can misjudge or reject config generated for a newer machine.
- kubectl v1.36.2 -> v1.36.3, the cluster's Kubernetes version.
- flux CLI 2.9.2 -> 2.9.5, the current 2.9.x. Checksums verified against
  the upstream release assets (talos sha256sum.txt, flux checksums.txt)
  and against a local download.

.woodpecker/ci-manifests-test.yml
- woodpecker-cli v3.17.0 -> v3.18.0, the version the server and agent
  already run; the lint step was one minor behind its own server.

Merging triggers build-ci-images (path ci/images/**), which publishes
ci-k8s:<date>. The six pipelines pinned to ci-k8s:2026-08-10 move to
that tag in a follow-up once it exists.

Refs #426

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 08:58:35 +02:00
k.renner 6b7736ea58 Merge pull request 'fix(alerts): page the moment OpenBao is below quorum' (#824) from fix/openbao-alerts-immediate into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#824
2026-09-08 06:02:32 +00:00
k.renner 7ff20e5b68 Merge pull request 'feat(openbao): auto-unseal with a static seal' (#825) from feat/openbao-static-seal into master
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#825
2026-09-08 05:54:54 +00:00
claude-botandClaude Fable 5.1 296accd1cb feat(openbao): auto-unseal with a static seal
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
Every restart sealed the replica until a human unsealed it. On
2026-09-07 a rolling talosctl upgrade rebooted all three nodes; the PDB
refused the second eviction while the first replica was sealed, Talos
waited out its 5-minute drain timeout and rebooted anyway, and raft
quorum was gone for 24 minutes — every ExternalSecret and the
Kustomizations behind them with it. A power loss would do the same
with no timeout to blame.

OpenBao 2.6.1 supports a static seal: a 32-byte key the server uses to
unseal itself. The key is generated into pass (homelab/openbao/seal-key)
and created as the openbao-seal-key Secret before this release reconciles;
the chart hands it to the server as BAO_SEAL_KEY from that Secret.

This overturns the principle in bootstrap.sh that the key protecting
OpenBao must not live in the cluster OpenBao secures. Decided by the
owner on 2026-09-08: the alternatives were a transit seal from a second
OpenBao on edge-1, which raft traffic would wear out, or a workstation
unseal script that may not be at hand when needed. The exposure is
bounded by what ESO already does — every synced secret is plaintext in
etcd (encrypted at rest) — and grows only by what lives solely in
OpenBao.

Migration is a live, per-replica procedure with the shamir key becoming
the recovery key; the StatefulSet is OnDelete, so merging this changes
nothing until each pod is restarted in turn. The bootstrap and rebuild
paths (bootstrap.sh, vmtest) follow in a separate change once the
migration is verified.

Refs #805

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 07:52:27 +02:00
claude-botandClaude Fable 5.1 2665e5ebf3 fix(alerts): page the moment OpenBao is below quorum
ci/woodpecker/pr/ci-scripts-test Pipeline was successful
ci/woodpecker/pr/ci-manifests-test Pipeline was successful
ci/woodpecker/pr/pr-review Pipeline was successful
OpenBaoQuorumLost waited for: 5m. Below quorum the secret store is
down — the ClusterSecretStore goes InvalidProviderConfig and every
ExternalSecret and the Kustomizations behind them stall — and a sealed
replica never recovers on its own, so the window only delayed the page.
It fires on the first evaluation now.

OpenBaoUnavailable, the external probe, drops from 5m to 1m: two
consecutive failed probes, enough to skip a single lost probe through
Traefik, not enough to sit on a sealed store.

Both StatefulSet expressions are wrapped in max by (namespace,
statefulset). kube-state-metrics stamps instance/pod on the series, so
a rescheduled exporter starts a new series and Prometheus resets the
for: timer with it — on 2026-09-07 that made OpenBaoReplicaNotReady fire
14 minutes late. Alertmanager routes match alertname/source only, so the
dropped labels change nothing downstream.

Refs #805

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-08 07:40:12 +02:00
k.renner 0fe512746b Merge pull request 'feat(monitoring): Intel iGPU dashboard' (#823) from feat/intel-gpu-dashboard into master
ci/woodpecker/cron/issue-fix Pipeline was successful
ci/woodpecker/cron/alert-monitor Pipeline was successful
Reviewed-on: k.renner/homelab#823
2026-09-08 05:24:35 +00:00
5 changed files with 36 additions and 27 deletions
+1 -1
View File
@@ -158,7 +158,7 @@ steps:
# whenever it trails the deployed server. If you bump the server, bump here.
- name: woodpecker-lint
depends_on: [] # see note above `steps:`
image: woodpeckerci/woodpecker-cli:v3.17.0-alpine
image: woodpeckerci/woodpecker-cli:v3.18.0-alpine
commands:
- |
set -eu
+5 -5
View File
@@ -1,9 +1,9 @@
FROM alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d
ARG TALOS_VERSION=v1.12.6
ARG TALOS_VERSION=v1.13.9
ARG VELERO_VERSION=v1.16.0
ARG KUBECTL_VERSION=v1.36.2
ARG FLUX_VERSION=2.9.2
ARG KUBECTL_VERSION=v1.36.3
ARG FLUX_VERSION=2.9.5
RUN apk add --no-cache \
aws-cli \
@@ -18,7 +18,7 @@ RUN apk add --no-cache \
RUN curl -sfL -o /usr/local/bin/talosctl \
"https://github.com/siderolabs/talos/releases/download/${TALOS_VERSION}/talosctl-linux-amd64" \
&& echo "4c41a3b10b075292e64b182283d558e9d8e935f43239de1ed4e4b14359593efb /usr/local/bin/talosctl" | sha256sum -c - \
&& echo "7e1d4b7d5846964bdcf63a794e3c8161bb6ef2983d5ace58ea5322f3bf32a27e /usr/local/bin/talosctl" | sha256sum -c - \
&& chmod +x /usr/local/bin/talosctl
RUN curl -sfL -o /usr/local/bin/kubectl \
@@ -31,7 +31,7 @@ RUN curl -sfL -o /usr/local/bin/kubectl \
RUN curl -sfL -o /tmp/flux.tar.gz \
"https://github.com/fluxcd/flux2/releases/download/v${FLUX_VERSION}/flux_${FLUX_VERSION}_linux_amd64.tar.gz" \
&& echo "4092b367fd060097976fb7261601f79420ecf7266328417dfd8dfee27de0e6a3 /tmp/flux.tar.gz" | sha256sum -c - \
&& echo "b853df82adfd7736f580692f9f734473d571606307139f8fd20c2a80dd1ff473 /tmp/flux.tar.gz" | sha256sum -c - \
&& tar -xzf /tmp/flux.tar.gz -C /usr/local/bin flux \
&& chmod +x /usr/local/bin/flux \
&& rm -f /tmp/flux.tar.gz
@@ -34,7 +34,9 @@ spec:
# query parameter, and an exact match would break silently if the
# instance label ever encodes it differently.
expr: probe_success{job="blackbox-http", instance=~"https://openbao\\.renner\\.uno/v1/sys/health.*"} == 0
for: 5m
# Two consecutive failed probes: enough to skip a single lost probe
# through Traefik, not enough to sit on a sealed store.
for: 1m
labels:
severity: critical
annotations:
@@ -31,7 +31,9 @@ spec:
# A replica that is sealed never recovers on its own, so the
# duration is only there to ignore ordinary rescheduling: a healthy
# pod moving between nodes is Ready again in well under a minute.
expr: kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"} < 3
# max by: kube-state-metrics stamps instance/pod on the series, so a
# rescheduled exporter starts a new series and resets the timer.
expr: max by (namespace, statefulset) (kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"}) < 3
for: 10m
labels:
severity: warning
@@ -53,11 +55,11 @@ spec:
# the ClusterSecretStore goes InvalidProviderConfig and no
# ExternalSecret can refresh.
#
# Shorter window than the warning above because this is an outage,
# not a degradation — but not zero, so a rolling restart that
# briefly dips does not page.
expr: kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"} < 2
for: 5m
# No window: below quorum the secret store is down and every
# ExternalSecret stalls with it. A sealed replica never recovers on
# its own, so waiting only delays the page.
expr: max by (namespace, statefulset) (kube_statefulset_status_replicas_ready{namespace="openbao", statefulset="openbao"}) < 2
for: 0m
labels:
severity: critical
annotations:
@@ -25,10 +25,10 @@ spec:
name: openbao
namespace: openbao
install:
# A fresh OpenBao pod is 0/1 Ready until `bao operator init` + unseal
# (readiness probe reports sealed). helm --wait would time out and
# remediation would uninstall the release, so don't wait here; the
# init/unseal procedure is manual (see kubernetes/MIGRATION-sops-openbao.md).
# A fresh OpenBao pod is 0/1 Ready until `bao operator init` (readiness
# reports sealed). helm --wait would time out and remediation would
# uninstall the release, so don't wait here; init is manual (see
# kubernetes/MIGRATION-sops-openbao.md). Unseal is automatic (static seal).
disableWait: true
remediation:
retries: 3
@@ -42,6 +42,11 @@ spec:
# the agent-injector mutating webhook is unused surface.
enabled: false
server:
# The static seal key, from the openbao-seal-key Secret (bootstrap.sh).
extraSecretEnvironmentVars:
- envName: BAO_SEAL_KEY
secretName: openbao-seal-key
secretKey: key
# Raft (not standalone/file) for two reasons: `bao operator raft
# snapshot` as a consistent backup primitive, and real HA across three
# nodes. TLS terminates at the Traefik ingress; in-cluster traffic is
@@ -53,16 +58,8 @@ spec:
# the moment one pod is sealed or rescheduled, which is worse than
# the single replica it replaced. Three tolerates one.
#
# What this buys, and what it does not: a node drain seals the replica
# on it, and raft elects a new leader from the remaining standbys
# within seconds, so the service does not go down. It does NOT remove
# the manual unseal — a sealed server can neither serve requests nor
# act as standby, so that replica stays out until someone unseals it.
#
# The consequence for rolling node maintenance is worth stating: a
# Talos upgrade drains all three nodes in turn, sealing each replica
# as it goes. Unseal each one as it comes back, or the third drain
# takes the last unsealed replica with it and quorum is lost anyway.
# A restarted replica unseals itself from the static seal and rejoins
# raft, so a rolling reboot only ever costs one replica at a time.
replicas: 3
raft:
enabled: true
@@ -111,6 +108,14 @@ spec:
service_registration "kubernetes" {}
# Auto-unseal. The key is a 32-byte AES key in the openbao-seal-key
# Secret, created from pass at bootstrap before this release. The
# former shamir key is the recovery key (#805).
seal "static" {
current_key_id = "1"
current_key = "env://BAO_SEAL_KEY"
}
audit "file" "stdout" {
description = "audit trail to pod stdout -> Loki"
options {