hermes-bot
6f1a7b6d96
Folds the good parts of #1412 into this PR and addresses both review-bot reviews (#1412 review 1789, #1414 review 1791): - Everything after the OpenBao store is best-effort. Decode, reconnect, listing and revokes run in reap_superseded(); reap_summary() turns any error into "reap skipped" plus a warning, so rotate_ha_token returns Ok. A failed run would start a second pod (backoffLimit: 1) that keeps the token just stored and its successor and reaps the token consumers still hold until ESO syncs, i.e. the #1081 outage. Two unit tests cover it. - Warn when the replaced token is not named "<client_name> [<6 hex>]": no later run reaps it, so it needs a manual revoke (from #1412). - Staged image rollout (from #1412): only the three HA CronJobs carry the flux-system:secret-rotator marker; the other 17 rotators stay on 2026-07-30 until a manual HA run validates the new binary, then a follow-up adds the marker to them. rotation/kustomization.yaml records the exception. - Merged current main (a6f37e2b,1691a0ac): the Reloader annotations for zone-configurator and speech-to-phrase are already there via #1409.
125 lines
5.5 KiB
YAML
125 lines
5.5 KiB
YAML
---
|
|
# Class-A automated rotation of the mosquitto `health` broker credential — a
|
|
# self-generated random secret.
|
|
#
|
|
# FIRST of the eight mosquitto broker users to be enrolled. The credentials were
|
|
# previously minted once by bootstrap Jobs and then never changed at all: no
|
|
# schedule, no compromise response, and they existed only in the Secret object,
|
|
# outside the OpenBao snapshot. See kubernetes/apps/home-assistant/
|
|
# PROPOSAL-mosquitto-credential-rotation.md (PR #453 — not yet merged; this
|
|
# file lands with it) for the full plan and phasing.
|
|
#
|
|
# `health` deliberately goes first: it is the broker's OWN liveness/readiness
|
|
# probe credential, consumed by exec probes inside the mosquitto container via
|
|
# the same Secret that seeds the passwd file. Nothing outside the pod holds it,
|
|
# so a rotation needs no coordination with any client — the smallest possible
|
|
# blast radius for proving rotator -> OpenBao -> ESO -> Reloader end to end.
|
|
#
|
|
# Runs the secret-rotator binary (random mode): generate 32 random bytes,
|
|
# standard-base64 encode, and merge-patch into OpenBao. assert_fields_exist runs
|
|
# FIRST, so a misconfigured path fails safe (no rotation) rather than adding a
|
|
# stray field — which also means this ROTATES an existing value and never
|
|
# bootstraps one. Seeding is a one-time out-of-band step, documented in
|
|
# policies/rotator-mosquitto-health.hcl.
|
|
#
|
|
# Base64 output is safe for this credential specifically: the probes pass it as
|
|
# `-P "$HEALTH_PASSWORD"` to mosquitto_pub/sub inside the container, so the `+`,
|
|
# `/` and `=` characters are never parsed by a URL or a config file. That is NOT
|
|
# automatically true of the other seven users (sml2mqtt and open3e templates on
|
|
# the edge host, Shelly device firmware, the HA config entry), so the charset
|
|
# must be re-checked per user before enrolling each one — the old bootstrap Jobs
|
|
# generated alphanumeric-only passwords with `tr -dc 'A-Za-z0-9'`.
|
|
#
|
|
# openbao-ns pattern: the Job runs in the openbao namespace, reaching the
|
|
# OpenBao API intra-namespace — no NetworkPolicy needed. It writes OpenBao only;
|
|
# ESO syncs mosquitto-health-credentials (ns home-assistant) and Reloader
|
|
# restarts the broker so the passwd file is rebuilt from the new value.
|
|
#
|
|
# Rotation is not instant: it completes only after the ExternalSecret's
|
|
# refreshInterval elapses and the broker rolls. During that roll two broker pods
|
|
# with different passwd files sit behind the LB (maxUnavailable: 0/maxSurge: 1),
|
|
# so a probe against the outgoing pod can fail auth briefly. Self-healing.
|
|
#
|
|
# ON-DEMAND: kubectl -n openbao create job --from=cronjob/mosquitto-health-secret-rotate mosquitto-health-secret-rotate-now
|
|
apiVersion: v1
|
|
kind: ServiceAccount
|
|
metadata:
|
|
name: mosquitto-health-secret-rotator
|
|
namespace: openbao
|
|
labels:
|
|
app.kubernetes.io/name: mosquitto
|
|
app.kubernetes.io/component: secret-rotator
|
|
---
|
|
apiVersion: batch/v1
|
|
kind: CronJob
|
|
metadata:
|
|
name: mosquitto-health-secret-rotate
|
|
namespace: openbao
|
|
labels:
|
|
app.kubernetes.io/name: mosquitto
|
|
app.kubernetes.io/component: secret-rotator
|
|
spec:
|
|
# Quarterly, offset one month from paperless (which runs Jan/Apr/Jul/Oct) so
|
|
# unrelated rotations do not land in the same window and complicate diagnosis.
|
|
schedule: "0 4 1 2,5,8,11 *"
|
|
timeZone: Europe/Berlin
|
|
concurrencyPolicy: Forbid
|
|
successfulJobsHistoryLimit: 2
|
|
failedJobsHistoryLimit: 5
|
|
startingDeadlineSeconds: 3600
|
|
jobTemplate:
|
|
spec:
|
|
backoffLimit: 2
|
|
activeDeadlineSeconds: 300
|
|
ttlSecondsAfterFinished: 259200 # 3 d
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app.kubernetes.io/name: mosquitto
|
|
app.kubernetes.io/component: secret-rotator
|
|
spec:
|
|
serviceAccountName: mosquitto-health-secret-rotator
|
|
restartPolicy: Never
|
|
securityContext:
|
|
runAsNonRoot: true
|
|
runAsUser: 65534
|
|
runAsGroup: 65534
|
|
seccompProfile:
|
|
type: RuntimeDefault
|
|
containers:
|
|
- name: rotate
|
|
# Dated tag, per the convention documented in kustomization.yaml —
|
|
# every rotator here now uses the same one. Verified for this tag:
|
|
# it pulls, `random` is among its modes, and it carries the
|
|
# RANDOM_BYTES>=16 floor documented below (RANDOM_BYTES=8 exits
|
|
# "refusing to generate a weak secret").
|
|
image: gitea.renner.uno/gitea_admin/secret-rotator:2026-07-30
|
|
args: ["random"]
|
|
securityContext:
|
|
allowPrivilegeEscalation: false
|
|
readOnlyRootFilesystem: true
|
|
capabilities:
|
|
drop: ["ALL"]
|
|
env:
|
|
- name: BAO_ADDR
|
|
value: http://openbao.openbao.svc.cluster.local:8200
|
|
- name: BAO_ROLE
|
|
value: rotator-mosquitto-health
|
|
- name: BAO_PATH
|
|
value: homelab/apps/home-assistant/mosquitto-health-credentials
|
|
# Matches the key the ExternalSecret projects and the key the
|
|
# Secret already carries, so mosquitto-deployment.yaml's
|
|
# secretKeyRef is unchanged.
|
|
- name: BAO_FIELDS
|
|
value: "password"
|
|
# 32 bytes -> 44 base64 chars. The binary refuses anything below
|
|
# 16 bytes outright.
|
|
- name: RANDOM_BYTES
|
|
value: "32"
|
|
resources:
|
|
requests:
|
|
cpu: 25m
|
|
memory: 16Mi
|
|
limits:
|
|
memory: 48Mi
|